DNA wire protocol

Before explaining the DNA protocol in more detail, we're going to discuss the Cursor message type. This type is used by all methods discussed later and plays a central role in how DNA works.

DNA models a blockchain as a sequence of blocks. The distance of a block from the first block in the chain (the genesis block) is known as chain height. The genesis block has height 0. Ideally, a blockchain should always build a block on top of the most recent block, but that's not always the case. For this reason, a block's height isn't enough to uniquely identify a block in the blockchain. A chain reorganization is when a chain produces blocks that are not building on top of the most recent block. As we will see later, the DNA protocol detects and handles chain reorganizations. A block that can't be part of a chain reorganization is finalized.

DNA uses a cursor to uniquely identify blocks on the chain. A cursor contains two fields:

• order_key: the block's height.
• unique_key: the block's unique identifier. Depending on the chain, it's the block hash or state root.

The Status method is used to retrieve the state of the DNA server. The request is an empty message. The response has the following fields:

• last_ingested: returns the last block ingested by the server. This is the most recent block available for streaming.
• finalized: the most recent finalized block.
• starting: the first available block. Usually this is the genesis block, but DNA server operators can prune older nodes to save on storage space.

The StreamData method is used to start a DNA stream. It accepts a StreamDataRequest message and returns an infinite stream of StreamDataResponse messages.

Request

The request message is used to configure the stream. All fields except filter are optional.

• starting_cursor: resume the stream from the provided cursor. The first block received in the stream will be the block following the provided cursor. If no cursor is provided, the stream will start from the genesis block. Notice that since starting_cursor is a cursor, the DNA server can detect if that block has been part of a chain's reorganization while the indexer was offline.
• finality: the stream contains data with at least the specified finality. Possible values are finalized (only receive finalized data), accepted (receive finalized and non-finalized blocks), and pending (receive finalized, non-finalized, and pending blocks).
• filter: a non-empty list of chain-specific data filters.
• heartbeat_interval: the stream will send an heartbeat message if there are no messages for the specified amount of time. This is useful to detect if the stream hangs. Value must be between 10 and 60 seconds.

Response

Once the server validates and accepts the request, it starts streaming data. Each stream message can be one of the following message types:

• data: receive data about a block.
• invalidate: the specified blocks don't belong to the canonical chain anymore because they were part of a chain reorganization.
• finalize: the most recent finalized block moved forward.
• heartbeat: an heartbeat message.
• system_message: used to send messages from the server to the client.

Data message

Contains the requested data for a single block. All data messages cursors are monotonically increasing, unless an Invalidate message is received.

The message contains the following fields:

• cursor: the cursor of the block before this message. If the client reconnects using this cursor, the first message will be the same as this message.
• end_cursor: this block's cursor. Reconnecting to the stream using this cursor will resume the stream.
• finality: finality status of this block.
• data: a list of encoded block data.

Notice how the data field is a list of block data. This sounds counter-intuitive since the Data message contains data about a single block. The reason is that, as we've seen in the "Request" section, the client can specify a list of filters. The data field has the same length as the request's filters field. In most cases, the client specifies a single filter and receives a single block of data. For advanced use cases (like tracking contracts deployed by a factory), the client uses multiple filters to have parallel streams of data synced on the block number.

Invalidate message

This message warns the client about a chain reorganization. It contains the following fields:

• cursor: the new chain's head. All previously received messages where the end_cursor.order_key was greater than (>) this message cursor.order_key should be considered invalid/recalled.
• removed: a list of cursors that used to belong to the canonical chain.

Finalize message

This message contains a single cursor field with the cursor of the most recent finalized block. All data at or before this block can't be part of a chain reorganization.

This message is useful to prune old data.

Heartbeat message

This message is sent at regular intervals once the stream reaches the chain's head.

Clients can detect if the stream hang by adding a timeout to the stream's receive method.

SytemMessage message

This message is used by the server to send out-of-band messages to the client. It contains text messages such as data usage, warnings about reaching the free quota, or information about upcoming system upgrades.

This section contains the protobuf definition used by the DNA server and clients. If you're implementing a new SDK for DNA, you can use this as the starting point.

syntax = "proto3";

package dna.v2.stream;

import "google/protobuf/duration.proto";

service DnaStream {
  // Stream data from the server.
  rpc StreamData(StreamDataRequest) returns (stream StreamDataResponse);
  // Get DNA server status.
  rpc Status(StatusRequest) returns (StatusResponse);
}

// A cursor over the stream content.
message Cursor {
  // Key used for ordering messages in the stream.
  //
  // This is usually the block or slot number.
  uint64 order_key = 1;
  // Key used to discriminate branches in the stream.
  //
  // This is usually the hash of the block.
  bytes unique_key = 2;
}

// Request for the `Status` method.
message StatusRequest {}

// Response for the `Status` method.
message StatusResponse {
  // The current head of the chain.
  Cursor current_head = 1;
  // The last cursor that was ingested by the node.
  Cursor last_ingested = 2;
  // The finalized block.
  Cursor finalized = 3;
  // The first block available.
  Cursor starting = 4;
}

// Request data to be streamed.
message StreamDataRequest {
  // Cursor to start streaming from.
  //
  // If not specified, starts from the genesis block.
  // Use the data's message `end_cursor` field to resume streaming.
  optional Cursor starting_cursor = 1;
  // Return data with the specified finality.
  //
  // If not specified, defaults to `DATA_FINALITY_ACCEPTED`.
  optional DataFinality finality = 2;
  // Filters used to generate data.
  repeated bytes filter = 3;
  // Heartbeat interval.
  //
  // Value must be between 10 and 60 seconds.
  // If not specified, defaults to 30 seconds.
  optional google.protobuf.Duration heartbeat_interval = 4;
}

// Contains a piece of streamed data.
message StreamDataResponse {
  oneof message {
    Data data = 1;
    Invalidate invalidate = 2;
    Finalize finalize = 3;
    Heartbeat heartbeat = 4;
    SystemMessage system_message = 5;
  }
}

// Invalidate data after the given cursor.
message Invalidate {
  // The cursor of the new chain's head.
  //
  // All data after this cursor should be considered invalid.
  Cursor cursor = 1;
  // List of blocks that were removed from the chain.
  repeated Cursor removed = 2;
}

// Move the finalized block forward.
message Finalize {
  // The cursor of the new finalized block.
  //
  // All data before this cursor cannot be invalidated.
  Cursor cursor = 1;
}

// A single block of data.
//
// If the request specified multiple filters, the `data` field will contain the
// data for each filter in the same order as the filters were specified in the
// request.
// If no data is available for a filter, the corresponding data field will be
// empty.
message Data {
  // Cursor that generated this block of data.
  optional Cursor cursor = 1;
  // Block cursor. Use this cursor to resume the stream.
  Cursor end_cursor = 2;
  // The finality status of the block.
  DataFinality finality = 3;
  // The block data.
  //
  // This message contains chain-specific data serialized using protobuf.
  repeated bytes data = 4;
}

// Sent to clients to check if stream is still connected.
message Heartbeat {}

// Message from the server to the client.
message SystemMessage {
  oneof output {
    // Output to stdout.
    string stdout = 1;
    // Output to stderr.
    string stderr = 2;
  }
}

// Data finality.
enum DataFinality {
  DATA_FINALITY_UNKNOWN = 0;
  // Data was received, but is not part of the canonical chain yet.
  DATA_FINALITY_PENDING = 1;
  // Data is now part of the canonical chain, but could still be invalidated.
  DATA_FINALITY_ACCEPTED = 2;
  // Data is finalized and cannot be invalidated.
  DATA_FINALITY_FINALIZED = 3;
}