← All posts

How do you visualize a data stream?

Matthew Pagan · · 9 min read

How do you visualize a data stream?

Follow the money in real-time Follow the money in real-time

Databases capture data and provide insights into what’s happened in the past; whereas streaming data offers the ability to understand real-time information immediately. By visualizing data streams in real time, we can move beyond the limitations of databases and data lakes of the past and race into a world where we can query the future.

Imagine this: Seeing live Ethereum transactions as they happen, tracking suspicious patterns, or even tracing the flow of funds from one account to another — all in real time — gives you the power to shift from reactive to proactive decision-making.

Bringing Real-Time Analytics to Streaming Data

Traditional data pipelines, which collect and store data for future analysis, miss out on the value of immediate insights. With tools like Quine and GraphXR, detection and visualization of data relationships and patterns happen in real-time, allowing proactive responses to events that would otherwise only be known after it’s too late to act.

In a world where data is continuously flowing from various sources, being able to analyze that data in real-time provides a critical advantage. And this empowers organizations to act on insights as soon as they emerge, rather than waiting for hours or days for batch processing to complete. This is especially crucial in scenarios such as fraud detection, security monitoring, or financial markets, where every second counts. For example, with blockchain networks like Ethereum, the ability to track transactions and identify patterns as they happen can help to spot irregular behaviors or verify the authenticity of activities.

Quine and GraphXR

This blog introduces you to two technologies: Quine from thatDot and GraphXR from Kineviz. Together, Quine and GraphXR form a powerful combination: Quine ingests and forms relationships in the data in real-time, while GraphXR makes these relationships visually clear — all in real-time.

Quine acts as an event stream processor, efficiently “connecting the dots” by ingesting data from multiple sources and emitting data to downstream systems. It constructs a graph of patterns and relationships within this data, allowing for real-time pattern detection and analysis.

GraphXR complements Quine by visualizing the graph. Not only does it display the current state of the graph, but it also dynamically represents live-flowing data within the graph.

GraphXR representation of Ethereum blockchain GraphXR representation of Ethereum blockchain

Getting Started Assets

You can follow along yourself as we trace Ethereum transactions! Here are the assets that you will need to run this demo:

  1. docker-compose.yml: This docker compose file will start both Quine and GraphXR services
  2. ethereum.yaml: This is a Quine recipe that contains the Ingest Streams and Standing Queries needed to ingest the 2 sources of Ethereum data, and to stream out accounts in real-time when we wish to observe funds flowing from account to account
  3. grove-notebook.zip: This is the Grove notebook used by GraphXR to read data from Quine’s graph, and to listen for Standing Query output being emitted from Quine. Grove notebooks are built on top of Observable notebooks, giving you the power to add inputs and logic into GraphXR. The Grove notebook holds the JavaScript logic that integrates the GraphXR client with Quine.

An in-depth description of using the Ethereum recipe with Quine alone can be found here.

Ethereum Data Stream

We will be consuming data from two Server Sent Event streams.

  1. https://ethereum.demo.thatdot.com/blocks_head — Streams out block level information such as:
  • number: The block number, indicating its position in the blockchain.
  • miner: The address of the account that receives the block reward.
  • hash: This block’s hash.
  • parentHash: The hash of the parent block’s header, linking the block to its predecessor.
  1. https://ethereum.demo.thatdot.com/mined_transactions — Streams out transaction level information such as:
  • from: The address of the account sending ether
  • to: The recipient address
  • value: The amount of Ether to transfer from the sender to the recipient (in wei).
  • blockNumber: the specific block in the Ethereum blockchain where a particular transaction has been included.

We can see how this data can form a graph: sequential blocks in the Ethereum blockchain are connected to each other, and transactions are connected to the block they have been included to. We can also create relationships between accounts sending transactions between each other.

Structure for Blocks and their Transactions

Block, Block Header, and Transaction graph structure Block, Block Header, and Transaction graph structure

Structure for Transactions Between Accounts

Account-account transaction graph structure Account-account transaction graph structure

Quine Ingest Streams

To turn blockchain data into graph nodes and edges, we use Cypher, a graph query language similar to SQL but designed for graph databases.

Quine’s Ingest Streams are responsible for manifesting data into the graph using Cypher queries. We use two ingest queries in this recipe to transform the streaming Ethereum data into nodes and edges. One is for ingesting block header data and the other for ingesting transaction data.

Running Quine and GraphXR, Ethereum Visualized

Now that we understand where the Ethereum data is coming from, what it looks like, and how we will form a graph structure out of it via Quine ingest queries, let’s run Quine along with GraphXR and visualize the blockchain.

  1. First, download and place into the same directory the getting started assets that were mentioned above; The ethereum.yaml recipe, the grove-notebook.zip zip file, and the docker-compose.yml docker compose file.

  2. Run the docker compose up -d command to run the docker compose file and have it run in the background. This command starts Quine with the Ethereum recipe, and starts GraphXR with its dependencies.

  3. To double check if Quine is running, visit http://localhost:8080/ to view the running Quine instance and issue the first sample query (a recipe provided query option in the Query dropdown) to see some of the blocks in the blockchain, and their sequential relationship with each other.

  1. Visit http://localhost:9000/ and sign up for a GraphXR trial. After signing up and logging in, you will have landed on GraphXR’s main Projects page where we will Create a new project.

Create new GraphXR project Create new GraphXR project

  1. Name your project and click Confirm.

Name new project Name new project

This creates a new project and opens it. You should see a screen similar to below.

New project, Empty GraphXR graph New project, Empty GraphXR graph

  1. Next, we import the Grove notebooks for the Ethereum visualization. Click on the Project tab, click on the Extension tab, then select Grove.

Import Grove notebook Import Grove notebook

  1. To import our Grove notebook drag and drop the file onto the space where the markdown is rendered.

  2. After uploading and decompressing the zip file, we will see the Grove notebook which includes the logic that integrates GraphXR and Quine.

Upload and decompress Grove zip Upload and decompress Grove zip

Loaded Grove notebook Loaded Grove notebook

We can test to ensure our Quine/GraphXR integration is working by executing a sample query. Select the first sample query from the Sample Queries dropdown and then click Submit Query. You should now see a graph showing sequential blocks in the blockchain.

Blocks in blockchain and their sequential structure Blocks in blockchain and their sequential structure

Try running the second sample query. Now you can see events involving accounts that both sent and received transactions.

Accounts that have both sent and received ETH Accounts that have both sent and received ETH

These queries represent snapshots into the streaming graph. But there is more value in being able to visualize the changing graph in real-time. In order to leverage Quine’s capability to perform analytics on this unbounded data, we will run through a use case involving tracing funds through the chain, again, in real-time.

Follow the Money

We visualized what the current state of different sub-graphs of the blockchain look like. But what we really want is to see data in motion. We’ll follow the money in this use case, mark an account as tainted, and Quine will stream out, in real-time, transactions flowing from that account to other accounts and beyond.

Quine uses Standing Queries to incrementally MATCH a subgraph as new data enters the graph. This enables us to perform analytics the moment data is streamed into our graph, as opposed to waiting to run a query at some point in the future. The standing query included in the recipe will keep an eye out for transactions flowing from tainted accounts, and then as data flows into the graph, Quine will notify us of those moving funds, and which accounts they flowed through.

Ready to see this in action? Let’s mark some accounts as tainted and observe how the data flows through the network in real-time.

Listen to Standing Query Stream in GraphXR

One of the ways Quine streams out its standing queries is via Server Sent Events.

Server Sent Events (SSE) are a way to push real-time updates from the server to the client, which means the data you’re visualizing in GraphXR will update as it changes.

By subscribing to this stream of data in GraphXR, we can respond the moment newly tainted accounts stream out from Quine, and render them in the GraphXR graph visualization. The imported Grove notebook already contains that logic, so let’s use it.

  • We want to have GraphXR subscribe to the output of the standing query we spoke about above. To do this, click the Listen for Standing Query Output checkbox. The logic attached to this listener will render newly tainted nodes, and their relationships with other rendered nodes, into the GraphXR graph visualization.

GraphXR listening for Quine standing query results GraphXR listening for Quine standing query results

  1. Now that we are listening for standing query output, we want to query a handful of account nodes that we want to monitor for transaction activity. We’ll adjust the second sample query and query for 6 account nodes using the following Cypher query:
MATCH
 (downstream:account)<-[:to]-(tx1)-[:from]->(a:account)<-[:to]-(tx2)-[:from]->(upstream:account)
WHERE
 tx1 <> tx2
 AND upstream <> downstream
 AND upstream <> a
 AND downstream <> a
RETURN upstream LIMIT 6

This Cypher query looks for an account (a) that both sends and receives funds. We ensure that the transactions are both different, and that all accounts are unique. We then select 6 of these upstream accounts.

  1. Submit the query and you should have 6 account nodes rendered in GraphXR.

Accounts that have sent transactions that we will follow transactions from Accounts that have sent transactions that we will follow transactions from

  1. We want to taint these nodes, and set their tainted property to 0. This will allow that node to pattern match against the standing query pattern query. To do this, we will make use of Quine’s Quick Queries, integrated into GraphXR via the node context menu. Select these 6 nodes, and then right-click any of them. This will bring up the context menu. Click on Quick Queries and then click Mark as Tainted and Refresh.

  2. The moment you do this, Quine’s standing queries begin incrementally matching against those nodes that were just tainted, and look for accounts with transactions from those tainted accounts. These incremental matches will stream out and be fed into GraphXR, which will then render these accounts in real-time.

Following ethereum transactions in real-time Following ethereum transactions in real-time

This capability lets you effectively query into the future and become notified the instant matches on the query occur. Every new transaction node rendered in GraphXR represents a transaction that occurred right now. A transaction that was observed by Quine, matched on within milliseconds when a transaction occurred, and streamed out to GraphXR for visualization.

Tips and Tricks for Success

GraphXR Visual Display Options

You can customize how nodes and edges are rendered in GraphXR through display options, like node avatars, node captions, and node size. These give you a way to customize how nodes look for your visualization.

GraphXR Templates

A template lets you preserve project settings and apply them to any of your projects. Once created, templates are immediately available to all projects.

GraphXR Layouts

GraphXR provides flexibility when it comes to laying out nodes in the graph. Customize your graph to enhance your use-case visually.

Quine Recipes

Quine recipes are a fantastic way to iterate on a use case for Quine, allowing you to configure Quine’s behavior on startup. These are great when you are actively developing a streaming solution with Quine, and want to share your ideas with others.

This blog post uses a recipe that preconfigures Quine in the following ways:

  1. 2 Ingest Streams — Used to stream data in from the two Ethereum sources
  2. 1 Standing Query — Used to stream out accounts as funds flow from a tainted account
  3. 2 Sample Queries — These are two queries we can use to quickly explore the blockchain
  4. 7 Quick Queries — One of these is used to taint an account, and trace money flowing from it
  5. Many Node Appearances — These aren’t used by GraphXR, but are used by Quine’s own Exploration UI.

Conclusion

Quine and GraphXR form a powerful duo for working with streaming data, enabling real-time analysis and visualization of complex data sets like the Ethereum blockchain. By embracing real-time analytics and visualization with these tools, businesses and developers can shift from reactive to proactive strategies, gaining the upper hand in scenarios where every moment matters.

Event Stream ProcessingReal Time Streaming DataReal Time AnalyticsBlockchainData Stream Processing