This page contains the Apibara documentation as a single document for consumption by LLMs.
---
title: Apibara documentation
titleShort: Overview
description: "Welcome to the Apibara documentation. Find more information about the Apibara protocol."
priority: 1000
fullpage: true
---
# Welcome to the next version of Apibara
This section contains the documentation for the upcoming version of Apibara.
As we move towards a stable release, we will update this documentation to
have more information.
---
title: Installation
description: "Learn how to install and get started with Apibara."
diataxis: tutorial
updatedAt: 2024-12-03
---
# Installation
This tutorial shows how to setup an Apibara project from scratch. The goal is to
start indexing data as quickly as possible and to understand the basic structure
of a project. By the end of this tutorial, you will have a basic indexer that
streams data from two networks (Ethereum and Starknet).
## Installation
This tutorial starts with a fresh Typescript project. Let's start by creating it.
```bash [Terminal]
mkdir my-indexer
cd my-indexer
npm init -y
```
After that, you can add the Apibara NPM packages.
:::cli-command
```bash [Terminal]
npm add --save apibara@next @apibara/protocol@next @apibara/indexer@next
```
```
added 325 packages, and audited 327 packages in 11s
73 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
```
:::
## Apibara Config
Your indexers' configuration goes in the `apibara.config.ts` file. You can
leave the configuration empty for now.
```typescript [apibara.config.ts]
import { defineConfig } from "apibara/config";
export default defineConfig({});
```
## EVM Indexer
Let's create the first EVM indexer. All indexers must go in the `indexers`
directory and have a name that ends with `.indexer.ts` or `.indexer.js`.
The Apibara CLI will automatically detect the indexers in this directory and
make them available to the project.
Since Apibara is chain-agnostic, we need to add the EVM package to stream Ethereum data.
:::cli-command
```bash [Terminal]
npm add --save @apibara/evm@next
```
```
added 2 packages, and audited 329 packages in 4s
73 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
```
:::
Then you can create the first indexer.
```bash [Terminal]
mkdir indexers
touch indexers/rocket-pool.indexer.ts
```
```typescript [rocket-pool.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";
import { useLogger } from "@apibara/indexer/plugins/logger";
export default defineIndexer(EvmStream)({
streamUrl: "https://ethereum.preview.apibara.org",
startingCursor: {
orderKey: 21_000_000n
},
filter: {
logs: [{
address: "0xae78736Cd615f374D3085123A210448E74Fc6393"
}]
},
async transform({ block }) {
const logger = useLogger();
const { logs, header } = block;
logger.log(`Block number ${header?.blockNumber}`)
for (const log of logs) {
logger.log(`Log ${log.logIndex} from ${log.address} tx=${log.transactionHash}`)
}
},
});
```
Notice the following:
- The indexer file exports a single indexer.
- The `defineIndexer` function takes the stream as parameter. In this case, the
`EvmStream` is used. This is needed because Apibara supports multiple networks
with different data types.
- `streamUrl` specifies where the data comes from. You can connect to streams hosted
by us, or to self-hosted streams.
- `startingCursor` specifies from which block to start streaming.
- The `filter` specifies which data to receive. You can read more about the available
data for EVM chains in the [EVM documentation](/docs/v2/networks/evm/filter).
- The `transform` function is called for each block. It receives the block as parameter.
This is where your indexer processes the data.
- The `useLogger` hook returns an indexer-specific logger.
There are more indexer options available, you can find them in the documentation (TODO: link).
## Running the indexer
During development, you will use the `apibara` CLI to build and run indexers. For convenience,
you should add the following scripts to your `package.json` file.
```json [package.json]
{
"scripts": {
"dev": "apibara dev",
"build": "apibara build",
"start": "apibara start"
}
}
```
- `dev`: runs all indexers in development mode. Indexers are automatically
reloaded and restarted when they change.
- `build`: builds the indexers for production.
- `start`: runs a _single indexer_ in production mode. Notice you must first
build the indexers.
Now, run the indexer in development mode.
:::cli-command
```bash [Terminal]
npm run dev
```
```
> my-indexer@1.0.0 dev
> apibara dev
✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
✔ Indexers built in 3587 ms
✔ Restarting indexers
rocket-pool | log Block number 21000071
rocket-pool | log Log 239 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0xe3b7e285c02e9a1dad654ba095ee517cf4c15bf0c2c0adec555045e86ea1de89
rocket-pool | log Block number 21000097
rocket-pool | log Log 265 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool | log Log 266 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool | log Block number 21000111
rocket-pool | log Log 589 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0xa01ec6551e76364f6cf687f52823d66b1c07f7a47ce157a9cd9e441691a021f0
...
```
:::
## Starknet indexer
You can index data on different networks from the same project. Let's add an indexer for Starknet.
Like before, you need to add the package to support the network.
:::cli-command
```bash [Terminal]
npm add --save @apibara/starknet@next
```
```
added 1 package, and audited 330 packages in 5s
73 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
```
:::
After that, you can implement the indexer. In this case, the indexer listens for all events
emitted by the STRK staking contract.
```typescript [strk-staking.indexer.ts]
import { StarknetStream } from "@apibara/starknet";
import { defineIndexer } from "@apibara/indexer";
import { useLogger } from "@apibara/indexer/plugins/logger";
export default defineIndexer(StarknetStream)({
streamUrl: "https://starknet.preview.apibara.org",
startingCursor: {
orderKey: 900_000n,
},
filter: {
events: [{
address: "0x028d709c875c0ceac3dce7065bec5328186dc89fe254527084d1689910954b0a",
}],
},
async transform({ block }) {
const logger = useLogger();
const { events, header } = block;
logger.log(`Block number ${header?.blockNumber}`)
for (const event of events) {
logger.log(`Event ${event.eventIndex} tx=${event.transactionHash}`)
}
},
});
```
You can now run the indexer. In this case, you can specify which indexer you want to run
with the `--indexers` option. When the flag is omitted, all indexers are run concurrently.
:::cli-command
```bash [Terminal]
npm run dev -- --indexers strk-staking
```
```
> my-indexer@1.0.0 dev
> apibara dev --indexers=strk-staking
✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
✔ Indexers built in 3858 ms
✔ Restarting indexers
strk-staking | log Block number 929092
strk-staking | log Event 233 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 234 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 235 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 236 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Block number 929119
strk-staking | log Event 122 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
strk-staking | log Event 123 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
...
```
:::
## Production build
The `apibara build` command is used to build a production version of the indexer. There are two main
changes for the production build:
- No hot code reloading is available.
- Only one indexer is started. If your project has multiple indexers, it should start them independently.
:::cli-command
```bash [Terminal]
npm run build
```
```
> my-indexer@1.0.0 build
> apibara build
✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
◐ Building 2 indexers
✔ Build succeeded!
ℹ You can start the indexers with apibara start
```
:::
Once the indexers are built, you can run them in two (equivalent) ways:
- The `apibara start` command by specifying which indexer to run with the
`--indexer` flag. In this tutorial we are going to use this method.
- Running `.apibara/build/start.mjs` with Node. This is useful when building Docker images for your indexers.
:::cli-command
```bash [Terminal]
npm run start -- --indexer rocket-pool
```
```
> my-indexer@1.0.0 start
> apibara start --indexer rocket-pool
◐ Starting indexer rocket-pool
rocket-pool | log Block number 21000071
rocket-pool | log Log 239 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0xe3b7e285c02e9a1dad654ba095ee517cf4c15bf0c2c0adec555045e86ea1de89
rocket-pool | log Block number 21000097
rocket-pool | log Log 265 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool | log Log 266 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
...
```
:::
## Runtime configuration & presets
Apibara provides a mechanism for indexers to load their configuration from the `apibara.config.ts` file:
- Add the configuration under the `runtimeConfig` key in `apibara.config.ts`.
- Change your indexer's module to return a function that, given the runtime configuration, returns the indexer.
You can update the configuration to define values that are configurable by your indexer. This example is going to
store the DNA stream URL and contract address in the configuration.
```ts [apibara.config.ts]
import { defineConfig } from "apibara/config";
export default defineConfig({
runtimeConfig: {
streamUrl: "https://starknet.preview.apibara.org",
contractAddress: "0x028d709c875c0ceac3dce7065bec5328186dc89fe254527084d1689910954b0a",
},
});
```
Then update the indexer to return a function that returns the indexer. Your editor is going to show a type error
since the types of `config.streamUrl` and `config.contractAddress` are unknown, the next session is going to
explain how to solve that issue.
```ts [strk-staking.indexer.ts]
import { StarknetStream } from "@apibara/starknet";
import { defineIndexer } from "@apibara/indexer";
import { useLogger } from "@apibara/indexer/plugins/logger";
import { ApibaraRuntimeConfig } from "apibara/types";
export default function (config: ApibaraRuntimeConfig) {
return defineIndexer(StarknetStream)({
streamUrl: config.streamUrl,
startingCursor: {
orderKey: 900_000n,
},
filter: {
events: [{
address: config.contractAddress as `0x${string}`
}],
},
async transform({ block }) {
// Unchanged.
},
});
};
```
### Typescript & type safety
You may have noticed that the CLI generates types in `.apibara/types` before
building the indexers (both in development and production mode).
These types contain the type definition of your runtime configuration. You can
instruct Typescript to use them by adding the following `tsconfig.json` to your
project.
```json [tsconfig.json]
{
"$schema": "https://json.schemastore.org/tsconfig",
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler"
},
"include": [
"**/*.ts",
".apibara/types"
],
"exclude": [
"node_modules"
],
}
```
After restarting the Typescript language server you will have a type-safe runtime configuration
right into your indexer!
### Presets
Having a single runtime configuration is useful but not enough for real-world
indexers. The CLI provides a way to have multiple "presets" and select which
one to use at runtime. This is useful, for example, if you're deploying the
same indexers on multiple networks where only the DNA stream URL and contract
addresses change.
You can any number of presets in the configuration and use the `--preset` flag to select which one to use.
For example, you can add a `sepolia` preset that contains the URL of the Starknet Sepolia DNA stream.
If a preset doesn't specify a key, then the value from the root configuration is used.
```ts [apibara.config.ts]
import { defineConfig } from "apibara/config";
export default defineConfig({
runtimeConfig: {
streamUrl: "https://starknet.preview.apibara.org",
contractAddress: "0x028d709c875c0ceac3dce7065bec5328186dc89fe254527084d1689910954b0a" as `0x${string}`,
},
presets: {
sepolia: {
runtimeConfig: {
streamUrl: "https://starknet-sepolia.preview.apibara.org",
},
}
},
});
```
You can then run the indexer in development mode using the `sepolia` preset.
:::cli-command
```bash [Terminal]
npm run dev -- --indexers=strk-staking --preset=sepolia
```
```
> my-indexer@1.0.0 dev
> apibara dev --indexers=strk-staking --preset=sepolia
✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
✔ Indexers built in 3858 ms
✔ Restarting indexers
strk-staking | log Block number 100092
strk-staking | log Event 233 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 234 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 235 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 236 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Block number 100119
strk-staking | log Event 122 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
strk-staking | log Event 123 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
...
```
:::
## Storing data & persisting state across restarts
All indexers implemented in this tutorial are stateless. They don't store any
data to a database and if you restart them they will restart indexing from the
beginning.
You can refer to our storage section to learn more about writing data to a database
and persisting the indexer's state across restarts.
- [Drizzle with PostgreSQL](/docs/v2/storage/drizzle-pg)
---
title: Indexers
description: "Learn how to create indexers to stream and transform onchain data."
diataxis: explanation
updatedAt: 2025-01-05
---
# Building indexers
Indexers are created using the `defineIndexer` higher-order function. This function takes a _stream definition_ and returns a function to define the indexer.
The job of an indexer is to stream and process historical data (backfilling) and then switch to real-time mode. Indexers built using our SDK are designed to handle chain-reorganizations automatically.
If, for any reason, you need to receive notifications about reorgs, you can define [a custom `message:invalidate` hook](/docs/v2/getting-started/plugins#hooks) to handle them.
By default, the indexer is stateless (restarts from the beginning on restart) and does not provide any storage. You can add persistence and storage by using one of the provided storage plugins.
### Examples
The following examples show how to create indexers for the Beacon Chain, EVM (Ethereum), and Starknet.
**Beacon Chain indexer**
```ts [beaconchain.indexer.ts]
import { BeaconChainStream } from "@apibara/beaconchain";
import { defineIndexer } from "@apibara/indexer";
export default defineIndexer(BeaconChainStream)({ /* ... */ });
```
**EVM (Ethereum) indexer**
```ts [evm.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";
export default defineIndexer(EvmStream)({ /* ... */ });
```
**Starknet indexer**
```ts [starknet.indexer.ts]
import { StarknetStream } from "@apibara/starknet";
import { defineIndexer } from "@apibara/indexer";
export default defineIndexer(StarknetStream)({ /* ... */ });
```
## Indexer configuration
All indexers take the same configuration options.
- **`streamUrl`**`string`
The URL of the DNA stream to connect to.
- **`filter`**`TFilter`
The filter to apply to the DNA stream. This argument is specific to the stream definition. You should refer to the chain's filter reference for the available options (see [Beacon Chain](/docs/v2/networks/beaconchain/filter), [EVM (Ethereum)](/docs/v2/networks/evm/filter), [Starknet](/docs/v2/networks/starknet/filter)).
- **`finality`**`"finalized" | "accepted" | "pending"`
Receive data with the specified finality. Defaults to `accepted`.
- **`startingCursor`**`{ orderKey: bigint, uniqueKey?: string }`
The cursor to start the indexer from. Defaults to the genesis block. The `orderKey` represents the block number, and the `uniqueKey` represents the block hash (optional).
- **`debug`**`boolean`
Enable debug mode. This will print debug information to the console.
- **`transform`**`({ block, cursor, endCursor, finality, context }) => Promise`
The transform function called for each block received from the DNA stream.
- **`factory`**`({ block, context }) => Promise<{ filter?: TFilter }>`
The factory function used to add data filters at runtime. Useful for creating indexers for smart contracts like Uniswap V2.
- **`hooks`**`object`
The hooks to register with the indexer. Refer to the [plugins & hooks](/docs/v2/getting-started/plugins) page for more information.
- **`plugins`**`array`
The plugins to register with the indexer. Refer to the [plugins & hooks](/docs/v2/getting-started/plugins) page for more information.
### The transform function
The `transform` function is invoked for each block received from the DNA stream. This function is where you should implement your business logic.
**Arguments**
- **`block`**`TBlock`
The block received from the DNA stream. This is chain-specific (see [Beacon Chain](/docs/v2/networks/beaconchain/data), [EVM (Ethereum)](/docs/v2/networks/evm/data), [Starknet](/docs/v2/networks/starknet/data)).
- **`cursor`**`{ orderKey: bigint, uniqueKey?: string }`
The cursor of the block before the received block.
- **`endCursor`**`{ orderKey: bigint, uniqueKey?: string }`
The cursor of the current block.
- **`finality`**`"finalized" | "accepted" | "pending"`
The finality of the block.
- **`context`**`object`
The context shared between the indexer and the plugins.
The following example shows a minimal indexer that streams block headers and prints them to the console.
```ts [evm.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";
export default defineIndexer(EvmStream)({
streamUrl: "https://ethereum.preview.apibara.org",
filter: {
header: "always",
},
async transform({ block }) {
const { header } = block;
console.log(header);
},
});
```
### The factory function
The `factory` function is used to add data filters at runtime. This is useful for creating indexers for smart contracts that deploy other smart contracts like Uniswap V2 and its forks.
**Arguments**
- **`block`**`TBlock`
The block received from the DNA stream. This is chain-specific (see [Beacon Chain](/docs/v2/networks/beaconchain/data), [EVM (Ethereum)](/docs/v2/networks/evm/data), [Starknet](/docs/v2/networks/starknet/data)).
- **`context`**`object`
The context shared between the indexer and the plugins.
The following example shows a minimal indexer that streams `PairCreated` events from Uniswap V2 to detect new pools, and then streams the pool's events.
```ts [uniswap-v2.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";
export default defineIndexer(EvmStream)({
streamUrl: "https://ethereum.preview.apibara.org",
filter: {
logs: [{ /* ... */ }],
},
async factory({ block }) {
const { logs } = block;
return { /* ... */ };
},
async transform({ block }) {
const { header, logs } = block;
console.log(header);
console.log(logs);
},
});
```
---
title: Plugins & Hooks
description: "Learn how to use plugins to extend the functionality of your indexers."
diataxis: explanation
updatedAt: 2025-01-05
---
# Plugins & Hooks
Indexers are extensible through hooks and plugins. Hooks are functions that are called at specific points in the indexer's lifecycle. Plugins are components that contain reusable hooks callbacks.
## Hooks
The following hooks are available in all indexers.
- **`run:before`**`() => void`
Called before the indexer starts running.
- **`run:after`**`() => void`
Called after the indexer has finished running.
- **`connect:before`**`({ request: StreamDataRequest, options: StreamDataOptions }) => void`
Called before the indexer connects to the DNA stream. Can be used to change the request or stream options.
- **`connect:after`**`({ request: StreamDataRequest }) => void`
Called after the indexer has connected to the DNA stream.
- **`connect:factory`**`({ request: StreamDataRequest, endCursor: { orderKey: bigint, uniqueKey?: string } }) => void`
Called before the indexer reconnects to the DNA stream with a new filter (in factory mode).
- **`message`**`({ message: StreamDataResponse }) => void`
Called for each message received from the DNA stream. Additionally, message-specific hooks are available: `message:invalidate`, `message:finalize`, `message:heartbeat`, `message:systemMessage`.
- **`handler:middleware`**`({ use: MiddlewareFunction) => void }) => void`
Called to register indexer's middlewares.
## Using plugins
You can register plugins in the indexer's configuration, under the `plugins` key.
```ts [my-indexer.indexer.ts]
import { BeaconChainStream } from "@apibara/beaconchain";
import { defineIndexer } from "@apibara/indexer";
import { myAwesomePlugin } from "@/lib/my-plugin.ts";
export default defineIndexer(BeaconChainStream)({
streamUrl: "https://beaconchain.preview.apibara.org",
filter: { /* ... */ },
plugins: [myAwesomePlugin()],
async transform({ block: { header, validators } }) {
/* ... */
},
});
```
## Building plugins
Developers can create new plugins to be shared across multiple indexers or projects. Plugins use the available hooks to extend the functionality of indexers.
The main way to define a plugin is by using the `defineIndexerPlugin` function. This function takes a callback with the indexer as parameter, the plugin should register itself with the indexer's hooks.
When the runner runs the indexer, all the relevant hooks are called.
```ts [my-plugin.ts]
import type { Cursor } from "@apibara/protocol";
import { defineIndexerPlugin } from "@apibara/indexer/plugins";
export function myAwesomePlugin() {
return defineIndexerPlugin((indexer) => {
indexer.hooks.hook("connect:before", ({ request, options }) => {
// Do something before the indexer connects to the DNA stream.
});
indexer.hooks.hook("run:after", () => {
// Do something after the indexer has finished running.
});
});
}
```
## Middleware
Apibara indexers support wrapping the `transform` function in middleware. This is used, for example, to wrap all database operations in a transaction.
The middleware is registered using the `handler:middleware` hook. This hook takes a `use` argument to register the middleware with the indexer.
The argument to `use` is a function that takes the indexer's context and a `next` function to call the next middleware or the transform function.
```ts [my-plugin.ts]
import type { Cursor } from "@apibara/protocol";
import { defineIndexerPlugin } from "@apibara/indexer/plugins";
export function myAwesomePlugin() {
return defineIndexerPlugin((indexer) => {
const db = openDatabase();
indexer.hooks.hook("handler:middleware", ({ use }) => {
use(async (context, next) => {
// Start a transaction.
await db.transaction(async (txn) => {
// Add the transaction to the context.
context.db = txn;
try {
// Call the next middleware or the transform function.
await next();
} finally {
// Remove the transaction from the context.
context.db = undefined;
}
});
});
});
});
}
```
## Inline hooks
For all cases where you want to use a hook without creating a plugin, you can use the `hooks` property of the indexer.
IMPORTANT: inline hooks are the recommended way to add hooks to an indexer. If the same hook is needed in multiple indexers, it is better to create a plugin. Usually, plugins lives in the `lib` folder, for example `lib/my-plugin.ts`.
```ts [my-indexer.indexer.ts]
import { BeaconChainStream } from "@apibara/beaconchain";
import { defineIndexer } from "@apibara/indexer";
export default defineIndexer(BeaconChainStream)({
streamUrl: "https://beaconchain.preview.apibara.org",
filter: { /* ... */ },
async transform({ block: { header, validators } }) {
/* ... */
},
hooks: {
async "connect:before"({ request, options }) {
// Do something before the indexer connects to the DNA stream.
},
},
});
```
## Indexer lifecycle
The following Javascript pseudocode shows the indexer's lifecycle. This should give you a good understanding of when hooks are called.
```js
function run(indexer) {
indexer.callHook("run:before");
const { use, middleware } = registerMiddleware(indexer);
indexer.callHook("handler:middleware", { use });
// Create the request based on the indexer's configuration.
const request = Request.create({
filter: indexer.filter,
startingCursor: indexer.startingCursor,
finality: indexer.finality,
});
// Stream options.
const options = {};
indexer.callHook("connect:before", { request, options });
let stream = indexer.streamData(request, options);
indexer.callHook("connect:after");
while (true) {
const { message, done } = stream.next();
if (done) {
break;
}
indexer.callHook("message", { message });
switch (message._tag) {
case "data": {
const { block, endCursor, finality } = message.data
middleware(() => {
if (indexer.isFactoryMode()) {
// Handle the factory portion of the indexer data.
// Implementation detail is not important here.
const newFilter = indexer.factory();
const request = Request.create(/* ... */);
indexer.callHook("connect:factory", { request, endCursor });
stream = indexer.streamData(request, options);
}
indexer.transform({ block, endCursor, finality });
});
break;
}
case "invalidate": {
indexer.callHook("message:invalidate", { message });
break;
}
case "finalize": {
indexer.callHook("message:finalize", { message });
break;
}
case "heartbeat": {
indexer.callHook("message:heartbeat", { message });
break;
}
case "systemMessage": {
indexer.callHook("message:systemMessage", { message });
break;
}
}
}
indexer.callHook("run:after");
}
```
---
title: Upgrading from v1
description: "Learn how to upgrade your indexers to Apibara v2."
diataxis: how-to
updatedAt: 2024-12-04
---
# Upgrading from v1
This guide explains how to upgrade your Starknet indexers from the old Apibara
CLI experience to the new Apibara v2 experience.
At the time of writing (December 2024), Apibara v2 is still in beta. This means
that it's ready for developers to start testing it, but you should not use it in
production yet.
## Main changes
- The underlying gRPC protocol and data types have changed. You can review all
the changes [on this page](/docs/v2/networks/starknet/upgrade-from-v1).
- The old CLI has been replaced by a pure Typescript library. This means you
can now leverage the full Node ecosystem (including Bun and Deno).
- You can now extend indexers with [plugins and hooks](/docs/v2/getting-started/plugins).
## Missing (but planned) features
These features will be added before the final DNA v2 release:
- Support pending (mempool) data.
- Add missing integrations like MongoDB.
## Migration
For this guide, we'll assume an indexer like the following:
```ts [indexer.ts]
export const config = {
streamUrl: "https://mainnet.starknet.a5a.ch",
startingBlock: 800_000,
network: "starknet",
finality: "DATA_STATUS_ACCEPTED",
filter: {
header: {},
},
sinkType: "console",
sinkOptions: {},
};
export default function transform(block) {
return block;
}
```
### Step 1: initialize the Node project
Initialize the project to contain a `package.json` file:
```bash [Terminal]
npm init -y
```
Create the `indexers/` folder where all the indexers will live:
```bash [Terminal]
mkdir indexers
```
Add the dependencies needed to run the indexer. If you're using any external
dependencies, make sure to add them.
:::cli-command
```bash [Terminal]
npm add --save apibara@next @apibara/protocol@next @apibara/indexer@next @apibara/starknet@next
```
```
added 325 packages, and audited 327 packages in 11s
73 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
```
:::
### Step 2: initialize the Apibara project
Create a new file called `apibara.config.ts` in the root of your project.
```ts [apibara.config.ts]
import { defineConfig } from "apibara/config";
export default defineConfig({});
```
### Step 3: update the indexer
Now it's time to update the indexer.
- Move the indexer to the `indexers/` folder, ensuring tha the file name ends
with `.indexer.ts`.
- Wrap the indexer in a `defineIndexer(StarknetStream)({ /* ... */ })` call.
Notice that now the stream configuration and transform function live in the same
configuration object.
- `startingBlock` is now `startingCursor.orderKey`.
- `streamUrl` is the same, but the URL is different while in beta.
- `finality` is now simpler to type.
- The `filter` object changed. Please refer to the [filter documentation](/docs/v2/networks/starknet/filter) for more information.
- `sinkType` and `sinkOptions` are gone.
- The `transform` function now takes named arguments, with `block` containing the block data.
The following `git diff` shows the changes to the indexer at the beginning of the guide.
```diff
diff --git a/simple.ts b/indexers/simple.indexer.ts
index bb09fdc..701a494 100644
--- a/simple.ts
+++ b/indexers/simple.indexer.ts
@@ -1,15 +1,18 @@
-export const config = {
- streamUrl: "https://mainnet.starknet.a5a.ch",
- startingBlock: 800_000,
- network: "starknet",
- finality: "DATA_STATUS_ACCEPTED",
+import { StarknetStream } from "@apibara/starknet";
+import { defineIndexer } from "@apibara/indexer";
+import { useLogger } from "@apibara/indexer/plugins/logger";
+
+export default defineIndexer(StarknetStream)({
+ streamUrl: "https://starknet.preview.apibara.org",
+ startingCursor: {
+ orderKey: 800_000n,
+ },
+ finality: "accepted",
filter: {
- header: {},
+ header: "always",
},
- sinkType: "console",
- sinkOptions: {},
-};
-
-export default function transform(block) {
- return block;
-}
\ No newline at end of file
+ async transform({ block }) {
+ const logger = useLogger();
+ logger.info(block);
+ },
+});
\ No newline at end of file
```
---
title: Drizzle with PostgreSQL
description: "Store your indexer's data to PostgreSQL using Drizzle ORM."
diataxis: reference
updatedAt: 2024-12-15
---
# Drizzle with PostgreSQL
The Apibara Indexer SDK supports Drizzle ORM for storing data to PostgreSQL.
## Installation
To use Drizzle with PostgreSQL, you need to install the following dependencies:
```bash [Terminal]
npm install drizzle-orm pg
```
We reccomend using Drizzle Kit to manage the database schema.
```bash [Terminal]
npm install --save-dev drizzle-kit
```
## Schema configuration
Indexer's tables require storing data in a way that can handle chain reorgs. The
approach used by Apibara is to add an additional `_cursor` column that tracks
when the data was inserted or updated. This is handled transparently by the
`pgIndexerTable` function.
```ts [schema.ts]
import { pgIndexerTable } from "@apibara/indexer/sinks/drizzle";
export const transfers = pgIndexerTable("transfers", {
amount: bigint("amount", { mode: "number" }),
transactionHash: text("transaction_hash"),
});
```
When generating the database migrations with Drizzle Kit, you can see the extra `_cursor` column added to the table.
```sql [migrations/00000000000000_create_table.sql]
CREATE TABLE IF NOT EXISTS "transfers" (
"amount" bigint,
"transaction_hash" text,
"_cursor" "int8range" NOT NULL
);
```
## Writing and reading data from within the indexer
From within the indexer, use the `useSink` hook to access the current Drizzle database transaction.
Notice that all operations inside the `transform` function are executed inside the same transaction
to ensure that the data is inserted or updated atomically. The `db` object returned by the hook
provides the following methods:
- `insert`: to insert new rows into the table.
- `update`: to update existing rows in the table.
- `delete`: to delete rows from the table.
- `select`: to query the table.
All four methods work analogously to the `insert`, `update`, `delete`, and
`select` methods from the `drizzle-orm` package.
```ts [indexers/my-indexer.indexer.ts]
import pg from "pg";
import { defineIndexer, useSink } from "@apibara/indexer";
import { drizzle as drizzleSink } from "@apibara/indexer/sinks/drizzle";
import { StarknetStream } from "@apibara/starknet";
import * as schema, { transfers } from "@/lib/schema";
const connectionString = "postgres://postgres:postgres@localhost:5432/postgres";
const pool = new pg.Pool({ connectionString });
const db = drizzle(pool, { schema });
export default defineIndexer(StarknetStream)({
streamUrl: "https://starknet.preview.apibara.org",
finality: "accepted",
startingCursor,
filter,
sink: drizzleSink({ database, tables: [transfers] }),
async transform({ endCursor, block, context }) {
const { db } = useSink({ context });
const { events } = block;
const values = events.map(decodeTransferEvent);
await db.insert(transfers).values(values);
},
});
```
## Querying data from outside the indexer
When selecting data from outside the indexer, you need to add the following
condition on `_cursor` to your query. This ensure that only data that has not been
removed or updated is returned by the query.
```sql
WHERE upper_inf(_cursor)
```
## Improving performance
You should always add an index on the `_cursor` column to improve the
performance of the queries.
## Cleaning up old data
Data that cannot be removed because of reorgs is automatically removed by the
indexer as it receives updates to the finalized block. Thanks to this, your database
contains the minimal amount of data required to safely handle reorgs.
---
title: Beacon Chain
description: "Stream Beacon Chain data with Apibara."
diataxis: reference
updatedAt: 2024-12-03
---
# Beacon Chain
```
..-#%%%%+.. ....-+#%%%%%%%#*=:... ...=#%#*:..
.*@@@@@@@@@@-.:+#@@@@#*+=======+*#%@@@%*=..+@@@@@@@@@%:
-@@@@@@@@@@@@@@@%+:... ....-*%@@@@@@@@@@@@@@*.
..@@@@@@@@@@@@@#:. :%@@@@@@@@@@@@*.
.#@@@@@@@@@@@+. ..-@@@@@@@@@@@-
:%@@@@@@@@@*.. .-%@@@@@@@@#.
.%@@@@@@@@: .=@@@@@@@#
.+@@@@@@#. :%@@@@@:
-@@@@#. .*@@@:
.*@* .#@+.
.-@%. .-#@@%*:. ..:#@@@#-. ..#@=.
:@@. .-@@@@@@@@+ .*@@@@@@@%: :@%.
+@+. .=@@@@@#.+@# :@@-:%@@@@@=. .*@=
..@@. -@@@@@@++@@* .@@@=+@@@@@@:. .@%.
.:@%. +@@@@@@@@@@: ..@@@@@@@@@@:. .%@:
.-@*. :@@@@@@@@*. ..*%%%%#+. .#@@@@@@@%.. .%@:
.:@%. ..:*%@@%*.. .-@@@@@@%. :*%@@%+::.. .@@.
..%@:. .::::::::.. .. ..=@=.. .. ..::::::::. =@*.
=@*. .::::::::.. **...:@@%:...#+. ..::::::::. .@@.
.#@+ ..::::::.. .-*%#=..:=#%*-. ..::::::.. #@=.
.*@*.. ....... ......... ..%@+.
.=@@-. .=@@-
.%@@- .+@@+.
..+@@#-... ...=%@@=..
..:*@@@*=. ..:+#@@%+:.
...-*%@@@@#*+=---::::---=+*#@@@@@#+:...
..-+#%@@@@@@@@@@%#*=:
```
Apibara provides data streams for the Beacon Chain, the Ethereum consensus
layer. Notice that these stream URLs are going to change in the future when DNA
v2 is released.
**Beacon Chain Mainnet**
```txt
TODO
```
**Beacon Chain Sepolia**
```txt
https://beaconchain-sepolia.preview.apibara.org
```
### Typescript package
Types for the Beacon Chain chain are provided by the `@apibara/beaconchain` package.
```bash [Terminal]
npm install @apibara/beaconchain@next
```
---
title: Beacon Chain filter reference
description: "Beacon Chain: DNA data filter reference guide."
diataxis: reference
updatedAt: 2024-10-22
---
# Beacon Chain filter reference
This page contains reference about the available data filters for Beacon Chain
DNA streams.
### Related pages
- [Beacon Chain block data reference](/docs/v2/networks/beaconchain/data)
## Filter ID
All filters have an associated ID. When the server filters a block, it will
return a list of all filters that matched a piece of data with the data.
You can use this ID to build powerful abstractions in your indexers.
## Filter types
### Root
The root filter object contains a collection of filters.
Notice that providing an empty filter object is an error.
```ts
export type Filter = {
header?: HeaderFilter;
transactions: TransactionFilter[];
blobs: BlobFilter[];
validators: ValidatorFilter[];
};
```
### Header
The `HeaderFilter` object controls when the block header is returned to the client.
```ts
export type HeaderFilter = "always" | "on_data" | "on_data_or_on_new_block";
```
The values have the following meaning:
- `always`: Always return the header, even if no other filter matches.
- `on_data`: Return the header only if any other filter matches. This is the default value.
- `on_data_or_on_new_block`: Return the header only if any other filter matches. If no other filter matches, return the header only if the block is a new block.
### Transactions
DNA includes decoded transactions submitted to the network.
```ts
export type TransactionFilter = {
id?: number;
from?: `0x${string}`;
to?: `0x${string}`;
create?: boolean;
includeBlob?: boolean;
};
```
**Properties**
- `from`: filter by sender address. If empty, matches any sender address.
- `to`: filter by receiver address. If empty, matches any receiver address.
- `create`: filter by whether the transaction is a create transaction.
- `includeBlob`: also return all blobs included in the transaction.
**Examples**
- All blobs included in a transaction to a specific contract.
```ts
const filter = [{
transactions: [{
to: "0xff00000000000000000000000000000000074248",
includeBlob: true,
}],
}];
```
### Blobs
A blob and its content.
```ts
export type BlobFilter = {
id?: number;
includeTransaction?: boolean;
};
```
**Properties**
- `includeTransaction`: also return the transaction that included the blob.
**Examples**
- All blobs posted to the network together with the transaction that posted them.
```ts
const filter = [{
blobs: [{
includeTransaction: true,
}],
}];
```
### Validators
Validators and their historical balances.
```ts
export type ValidatorStatus =
| "pending_initialized"
| "pending_queued"
| "active_ongoing"
| "active_exiting"
| "active_slashed"
| "exited_unslashed"
| "exited_slashed"
| "withdrawal_possible"
| "withdrawal_done";
export type ValidatorFilter = {
id?: number;
validatorIndex?: number;
status?: ValidatorStatus;
};
```
**Properties**
- `validatorIndex`: filter by the validator index.
- `status`: filter by validator status.
**Examples**
- All validators that exited, both slashed and unlashed.
```ts
const filter = [{
validators: [{
status: "exited_unslashed"
}, {
status: "exited_slashed"
}],
}];
```
---
title: Beacon Chain data reference
description: "Beacon Chain: DNA data data reference guide."
diataxis: reference
updatedAt: 2024-10-22
---
# Beacon Chain data reference
This page contains reference about the available data in Beacon Chain DNA streams.
### Related pages
- [Beacon Chain data filter reference](/docs/v2/networks/beaconchain/filter)
## Filter ID
All filters have an associated ID. To help clients correlate filters with data,
the filter ID is included in the `filterIds` field of all data objects.
This field contains the list of _all filter IDs_ that matched a piece of data.
## Nullable fields
**Important**: most fields are nullable to allow evolving the protocol. You should
always assert the presence of a field for critical indexers.
## Scalar types
The `@apibara/beaconchain` package defines the following scalar types:
- `Address`: a 20-byte Ethereum address, represented as a `0x${string}` type.
- `B256`: a 32-byte Ethereum value, represented as a `0x${string}` type.
- `B384`: a 48-byte Ethereum value, represented as a `0x${string}` type.
- `Bytes`: arbitrary length bytes, represented as a `0x${string}` type.
## Data type
### Block
The root object is the `Block`.
```ts
export type Block = {
header?: BlockHeader;
transactions: Transaction[];
blobs: Blob[];
validators: Validator[];
};
```
### Header
This is the block header, which contains information about the block.
```ts
export type BlockHeader = {
slot?: bigint;
proposerIndex?: number;
parentRoot?: B256;
stateRoot?: B256;
randaoReveal?: Bytes;
depositCount?: bigint;
depositRoot?: B256;
blockHash?: B256;
graffiti?: B256;
executionPayload?: ExecutionPayload;
blobKzgCommitments: B384[];
};
export type ExecutionPayload = {
parentHash?: B256;
feeRecipient?: Address;
stateRoot?: B256;
receiptsRoot?: B256;
logsBloom?: Bytes;
prevRandao?: B256;
blockNumber?: bigint;
timestamp?: Date;
};
```
**Properties**
- `slot`: the slot number.
- `proposerIndex`: the index of the validator that proposed the block.
- `parentRoot`: the parent root.
- `stateRoot`: the state root.
- `randaoReveal`: the randao reveal.
- `depositCount`: the number of deposits.
- `depositRoot`: the deposit root.
- `blockHash`: the block hash.
- `graffiti`: the graffiti.
- `executionPayload`: the execution payload.
- `blobKzgCommitments`: the blob kzg commitments.
### Transaction
An EVM transaction.
```ts
export type Transaction = {
filterIds: number[];
transactionIndex?: number;
transactionHash?: B256;
nonce?: bigint;
from?: Address;
to?: Address;
value?: bigint;
gasPrice?: bigint;
gas?: bigint;
maxFeePerGas?: bigint;
maxPriorityFeePerGas?: bigint;
input: Bytes;
signature?: Signature;
chainId?: bigint;
accessList: AccessListItem[],
transactionType?: bigint;
maxFeePerBlobGas?: bigint;
blobVersionedHashes?: B256[];
};
export type Signature = {
r?: bigint;
s?: bigint;
v?: bigint;
yParity: boolean;
};
export type AccessListItem = {
address?: Address;
storageKeys: B256[];
};
```
**Properties**
- `transactionIndex`: the index of the transaction in the block.
- `transactionHash`: the hash of the transaction.
- `nonce`: the nonce of the transaction.
- `from`: the sender of the transaction.
- `to`: the recipient of the transaction. Empty if it's a create transaction.
- `value`: the value of the transaction, in wei.
- `gasPrice`: the gas price of the transaction.
- `gas`: the gas limit of the transaction.
- `maxFeePerGas`: the max fee per gas of the transaction.
- `maxPriorityFeePerGas`: the max priority fee per gas of the transaction.
- `input`: the input data of the transaction.
- `signature`: the signature of the transaction.
- `chainId`: the chain ID of the transaction.
- `accessList`: the access list of the transaction.
- `transactionType`: the transaction type.
- `maxFeePerBlobGas`: the max fee per blob gas of the transaction.
- `blobVersionedHashes`: the hashes of blobs posted by the transaction.
- `transactionStatus`: the status of the transaction.
**Relevant filters**
- `filter.transactions`
- `filter.blobs[].includeTransaction`
### Blob
A blob and its content.
```ts
export type Blob = {
filterIds: number[];
blobIndex?: number;
blob?: Uint8Array;
kzgCommitment?: B384;
kzgProof?: B384;
kzgCommitmentInclusionProof: B256[];
blobHash?: B256;
transactionIndex?: number;
transactionHash?: B256;
};
```
**Properties**
- `blobIndex`: the index of the blob in the block.
- `blob`: the blob content.
- `kzgCommitment`: the blob kzg commitment.
- `kzgProof`: the blob kzg proof.
- `kzgCommitmentInclusionProof`: the blob kzg commitment inclusion proof.
- `blobHash`: the hash of the blob content.
- `transactionIndex`: the index of the transaction that included the blob.
- `transactionHash`: the hash of the transaction that included the blob.
**Relevant filters**
- `filter.blobs`
- `filter.transactions[].includeBlob`
### Validator
Data about validators.
```ts
export type ValidatorStatus =
| "pending_initialized"
| "pending_queued"
| "active_ongoing"
| "active_exiting"
| "active_slashed"
| "exited_unslashed"
| "exited_slashed"
| "withdrawal_possible"
| "withdrawal_done";
export type Validator = {
filterIds: number[];
validatorIndex?: number;
balance?: bigint;
status?: ValidatorStatus;
pubkey?: B384;
withdrawalCredentials?: B256;
effectiveBalance?: bigint;
slashed?: boolean;
activationEligibilityEpoch?: bigint;
activationEpoch?: bigint;
exitEpoch?: bigint;
withdrawableEpoch?: bigint;
};
```
**Properties**
- `validatorIndex`: the index of the validator.
- `balance`: the balance of the validator.
- `status`: the status of the validator.
- `pubkey`: the validator's public key.
- `withdrawalCredentials`: the withdrawal credentials.
- `effectiveBalance`: the effective balance of the validator.
- `slashed`: whether the validator is slashed.
- `activationEligibilityEpoch`: the epoch at which the validator can be activated.
- `activationEpoch`: the epoch at which the validator was activated.
- `exitEpoch`: the epoch at which the validator exited.
- `withdrawableEpoch`: the epoch at which the validator can withdraw.
**Relevant filters**
- `filter.validators`
---
title: Ethereum EVM
description: "Stream Ethereum data with Apibara."
diataxis: reference
updatedAt: 2024-10-22
---
# Ethereum EVM
```
@
@@@
.@@@@@@
@@@@@@@@@
@@@@@@@@@@@.
.@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@.
@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@((@@@@@@@@@@@
@@@@@@@@@@((((((((@@@@@@@@@.
@@@@@@@@((((((((((((((@@@@@@@@
@@@@@@(((((((((((((((((((((@@@@@@
@@@((((((((((((((((((((((((((((@@@@
(((((((((((((((((((((((((((((((((((((,
(((((((((((((((((((((((((((((((^
*((((((((((((((((((((((((
(((((((((((((((((
@@@. (((((((((* .@@^
@@@@. *(( .@@@@@
@@@@@@@ ..@@@@@@
@@@@@@@@@. .@@@@@@@@@
^@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@
@@@@@@@@@@^
@@@@@@@
@@@@^
@
```
Apibara provides data streams for Ethereum mainnet. Notice that these
stream URLs are going to change in the future when DNA v2 is released.
**Ethereum Mainnet**
```txt
https://ethereum.preview.apibara.org
```
### Typescript package
Types for EVM chains are provided by the `@apibara/evm` package.
```bash [Terminal]
npm install @apibara/evm@next
```
---
title: EVM filter reference
description: "EVM: DNA data filter reference guide."
diataxis: reference
updatedAt: 2024-10-22
---
# EVM filter reference
This page contains reference about the available data filters for EVM DNA
streams.
### Related pages
- [EVM block data reference](/docs/v2/networks/evm/data)
## Filter ID
All filters have an associated ID. When the server filters a block, it will
return a list of all filters that matched a piece of data with the data.
You can use this ID to build powerful abstractions in your indexers.
## Usage with viem
Most types are compatible with [viem](https://viem.sh/).
For example, you can generate log filters with the following code:
```ts
import { encodeEventTopics, parseAbi } from "viem";
const abi = parseAbi([
"event Transfer(address indexed from, address indexed to, uint256 value)",
]);
const filter = {
logs: [{
topics: encodeEventTopics({
abi,
eventName: "Transfer",
args: { from: null, to: null },
}),
strict: true,
}]
}
```
## Filter types
### Root
The root filter object contains a collection of filters.
Notice that providing an empty filter object is an error.
```ts
export type Filter = {
header?: HeaderFilter;
logs?: LogFilter[];
transactions?: TransactionFilter[];
withdrawals?: WithdrawalFilter[];
};
```
### Header
The `HeaderFilter` object controls when the block header is returned to the client.
```ts
export type HeaderFilter = "always" | "on_data" | "on_data_or_on_new_block";
```
The values have the following meaning:
- `always`: Always return the header, even if no other filter matches.
- `on_data`: Return the header only if any other filter matches. This is the default value.
- `on_data_or_on_new_block`: Return the header only if any other filter matches. If no other filter matches, return the header only if the block is a new block.
## Logs
Logs are the most common type of DNA filters. Use this filter to get the logs and
their associated data like transactions, receipts, and sibling logs.
```ts
export type LogFilter = {
id?: number;
address?: `0x${string}`;
topics?: (`0x${string} | null`)[];
strict?: boolean;
transactionStatus?: "succeeded" | "reverted" | "all";
includeTransaction?: boolean;
includeReceipt?: boolean;
includeSiblings?: boolean;
};
```
**Properties**
- `address`: filter by contract address. If empty, matches any contract
address.
- `topics`: filter by topic. Use `null` to match _any_ value.
- `strict`: return logs whose topics length matches the filter. By default, the
filter does a prefix match on the topics.
- `transactionStatus`: return logs emitted by transactions with the provided
status. Defaults to `succeeded`.
- `includeTransaction`: also return the transaction that emitted the log.
- `includeReceipt`: also return the receipt of the transaction that emitted the
log.
- `includeSiblings`: also return all other logs emitted by the same transaction
that emitted the matched log.
**Examples**
- All logs in a block emitted by successful transactions.
```ts
const filter = {
logs: [{}],
};
```
- All `Transfer` events emitted by successful transactions. Notice that this will
match logs from ERC-20, ERC-721, and other contracts that emit `Transfer`.
```ts
const filter = {
logs: [{
topics: ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],
}],
};
```
- All `Transfer` events that follow the ERC-721 standard. Notice that this will
not match logs from ERC-20 since the number of indexed parameters is different.
```ts
const filter = {
logs: [{
topics: [
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
null, // from
null, // to
null, // tokenId
],
strict: true,
}],
};
```
- All logs emitted by `CONTRACT_A` OR `CONTRACT_B`.
```ts
const filter = {
logs: [{
address: CONTRACT_A,
}, {
address: CONTRACT_B,
}],
};
```
## Transactions
Request Ethereum transactions.
```ts
export type TransactionFilter = {
id?: number;
from?: `0x${string}`;
to?: `0x${string}`;
create?: true,
transactionStatus?: "succeeded" | "reverted" | "all";
includeReceipt?: boolean;
includeLogs?: boolean;
};
```
**Properties**
- `from`: filter by sender address. If empty, matches any sender address.
- `to`: filter by receiver address. If empty, matches any receiver address.
- `create`: filter by whether the transaction is a create transaction.
- `transactionStatus`: return transactions with the provided status. Defaults to
`succeeded`.
- `includeReceipt`: also return the receipt of the transaction.
- `includeLogs`: also return the logs emitted by the transaction.
**Examples**
- All transactions in a block.
```ts
const filter = {
transactions: [{}],
};
```
- All transactions from `0xAB...`.
```ts
const filter = {
transactions: [{
from: "0xAB...",
}],
};
```
- All create transactions.
```ts
const filter = {
transactions: [{
create: true,
}],
};
```
## Withdrawals
Request Ethereum withdrawals.
```ts
export type WithdrawalFilter = {
id?: number;
validatorIndex?: number;
address?: string;
};
```
**Properties**
- `validatorIndex`: filter by validator's index. If empty, matches any
validator's index.
- `address`: filter by withdrawal address. If empty, matches any withdrawal
address.
**Examples**
- All withdrawals
```ts
const filter = {
withdrawals: [{}],
};
```
- All withdrawals from validator with index `1234`.
```ts
const filter = {
withdrawals: [{
validatorIndex: 1234,
}],
};
```
- All withdrawals from validators with index `1234` OR `7890`.
```ts
const filter = {
withdrawals: [{
validatorIndex: 1234,
}, {
validatorIndex: 7890,
}],
};
```
- All withdrawals to address `0xAB...`.
```ts
const filter = {
withdrawals: [{
address: "0xAB...",
}],
};
```
---
title: EVM data reference
description: "EVM: DNA data data reference guide."
diataxis: reference
updatedAt: 2024-10-22
---
# EVM data reference
This page contains reference about the available data in EVM DNA streams.
### Related pages
- [EVM data filter reference](/docs/v2/networks/evm/filter)
## Filter ID
All filters have an associated ID. To help clients correlate filters with data,
the filter ID is included in the `filterIds` field of all data objects.
This field contains the list of _all filter IDs_ that matched a piece of data.
## Nullable fields
**Important**: most fields are nullable to allow evolving the protocol. You should
always assert the presence of a field for critical indexers.
## Scalar types
The `@apibara/evm` package defines the following scalar types:
- `Address`: a 20-byte Ethereum address, represented as a `0x${string}` type.
- `B256`: a 32-byte Ethereum value, represented as a `0x${string}` type.
- `Bytes`: arbitrary length bytes, represented as a `0x${string}` type.
## Usage with viem
Most types are compatible with [viem](https://viem.sh/).
For example, you can decode logs with the following code:
```ts
import type { B256 } from "@apibara/evm";
import { decodeEventLog, parseAbi } from "viem";
const abi = parseAbi([
"event Transfer(address indexed from, address indexed to, uint256 value)",
]);
// Somewhere in your indexer...
for (const log of logs) {
const { args, eventName } = decodeEventLog({
abi,
topics: log.topics as [B256, ...B256[]],
data: log.data,
});
}
```
## Data type
### Block
The root object is the `Block`.
```ts
export type Block = {
header?: BlockHeader;
logs: Log[];
transactions: Transaction[];
receipts: TransactionReceipt[];
withdrawals: Withdrawal[];
};
```
### Header
This is the block header, which contains information about the block.
```ts
export type Bloom = Bytes;
export type BlockHeader = {
blockNumber?: bigint;
blockHash?: B256,
parentBlockHash?: B256;
unclesHash?: B256;
miner?: Address;
stateRoot?: B256;
transactionsRoot?: B256;
receiptsRoot?: B256;
logsBloom?: Bloom;
difficulty?: bigint;
gasLimit?: bigint;
gasUsed?: bigint;
timestamp?: Date;
extraData?: Bytes;
mixHash?: B256;
nonce?: bigint;
baseFeePerGas?: bigint;
withdrawalsRoot?: B256;
totalDifficulty?: bigint;
blobGasUsed?: bigint;
excessBlobGas?: bigint;
parentBeaconBlockRoot?: B256;
};
```
**Properties**
- `blockNumber`: the block number.
- `blockHash`: the block hash.
- `parentBlockHash`: the block hash of the parent block.
- `unclesHash`: the block hash of the uncles.
- `miner`: the address of the miner.
- `stateRoot`: the state root.
- `transactionsRoot`: the transactions root.
- `receiptsRoot`: the receipts root.
- `logsBloom`: the logs bloom.
- `difficulty`: the block difficulty.
- `gasLimit`: the block gas limit.
- `gasUsed`: the gas used by transactions in the block.
- `timestamp`: the block timestamp.
- `extraData`: extra bytes data picked by the miner.
- `mixHash`: the mix hash.
- `nonce`: the nonce.
- `baseFeePerGas`: the base fee per gas.
- `withdrawalsRoot`: the withdrawals root.
- `totalDifficulty`: the total difficulty.
- `blobGasUsed`: the gas used by transactions posting blob data in the block.
- `excessBlobGas`: the excess blob gas.
- `parentBeaconBlockRoot`: the parent beacon block root.
### Log
An EVM log. It comes together with the essential information about the transaction
that emitted the log.
```ts
export type Log = {
filterIds: number[];
address?: Address;
topics: B256[];
data: Bytes;
logIndex: number;
transactionIndex: number;
transactionHash: B256;
transactionStatus: "succeeded" | "reverted";
};
```
**Properties**
- `address`: the address of the contract that emitted the log.
- `topics`: the topics of the log.
- `data`: the data of the log.
- `logIndex`: the index of the log in the block.
- `transactionIndex`: the index of the transaction that emitted the log.
- `transactionHash`: the hash of the transaction that emitted the log.
- `transactionStatus`: the status of the transaction that emitted the log.
**Relevant filters**
- `filter.logs`
- `filter.logs[].includeSiblings`
- `filter.transactions[].includeLogs`
### Transaction
An EVM transaction.
```ts
export type Transaction = {
filterIds: number[];
transactionIndex?: number;
transactionHash?: B256;
nonce?: bigint;
from?: Address;
to?: Address;
value?: bigint;
gasPrice?: bigint;
gas?: bigint;
maxFeePerGas?: bigint;
maxPriorityFeePerGas?: bigint;
input: Bytes;
signature?: Signature;
chainId?: bigint;
accessList: AccessListItem[],
transactionType?: bigint;
maxFeePerBlobGas?: bigint;
blobVersionedHashes?: B256[];
transactionStatus: "succeeded" | "reverted";
};
export type Signature = {
r?: bigint;
s?: bigint;
v?: bigint;
yParity: boolean;
};
export type AccessListItem = {
address?: Address;
storageKeys: B256[];
};
```
**Properties**
- `transactionIndex`: the index of the transaction in the block.
- `transactionHash`: the hash of the transaction.
- `nonce`: the nonce of the transaction.
- `from`: the sender of the transaction.
- `to`: the recipient of the transaction. Empty if it's a create transaction.
- `value`: the value of the transaction, in wei.
- `gasPrice`: the gas price of the transaction.
- `gas`: the gas limit of the transaction.
- `maxFeePerGas`: the max fee per gas of the transaction.
- `maxPriorityFeePerGas`: the max priority fee per gas of the transaction.
- `input`: the input data of the transaction.
- `signature`: the signature of the transaction.
- `chainId`: the chain ID of the transaction.
- `accessList`: the access list of the transaction.
- `transactionType`: the transaction type.
- `maxFeePerBlobGas`: the max fee per blob gas of the transaction.
- `blobVersionedHashes`: the hashes of blobs posted by the transaction.
- `transactionStatus`: the status of the transaction.
**Relevant filters**
- `filter.transactions`
- `filter.logs[].includeTransaction`
### Transaction Receipt
Information about the transaction's execution.
```ts
export type TransactionReceipt = {
filterIds: number[];
transactionIndex?: number;
transactionHash?: B256;
cumulativeGasUsed?: bigint;
gasUsed?: bigint;
effectiveGasPrice?: bigint;
from?: Address;
to?: Address;
contractAddress?: Address;
logsBloom?: Bloom;
transactionType?: bigint;
blobGasUsed?: bigint;
blobGasPrice?: bigint;
transactionStatus: "succeeded" | "reverted";
};
```
**Properties**
- `transactionIndex`: the transaction index in the block.
- `transactionHash`: the hash of the transaction.
- `cumulativeGasUsed`: the cumulative gas used by the transactions.
- `gasUsed`: the gas used by the transaction.
- `effectiveGasPrice`: the effective gas price of the transaction.
- `from`: the sender of the transaction.
- `to`: the recipient of the transaction. Empty if it's a create transaction.
- `contractAddress`: the address of the contract created by the transaction.
- `logsBloom`: the logs bloom of the transaction.
- `transactionType`: the transaction type.
- `blobGasUsed`: the gas used by the transaction posting blob data.
- `blobGasPrice`: the gas price of the transaction posting blob data.
- `transactionStatus`: the status of the transaction.
**Relevant filters**
- `filter.transactions[].includeReceipt`
- `filter.logs[].includeReceipt`
### Withdrawal
A withdrawal from the Ethereum network.
```ts
export type Withdrawal = {
filterIds: number[];
withdrawalIndex?: number;
index?: bigint;
validatorIndex?: number;
address?: Address;
amount?: bigint;
};
```
**Properties**
- `withdrawalIndex`: the index of the withdrawal in the block.
- `index`: the global index of the withdrawal.
- `validatorIndex`: the index of the validator that created the withdrawal.
- `address`: the destination address of the withdrawal.
- `amount`: the amount of the withdrawal, in wei.
**Relevant filters**
- `filter.withdrawals`
---
title: Starknet
description: "Stream Starknet data with Apibara."
diataxis: reference
updatedAt: 2024-10-22
---
# Starknet
```
,//(//,.
(@. .#@@@@@@@@@@@@@@@@@#.
(@@&. (@@@@@@@@@@@@@@@@@@@@@@@@&.
/%@@@@@@@@@%. ,@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@,
.@@# *@@@@@@@@@@@@@@@@@@@@@@@&&&&&&&@@@%
*% ,@@@@@@@@@@@@@@@@@@@@#((((((((((((,
.@@@@@@@@@@@@@@@@@@@%((((((((((((,
.@@@@@@@@@@@@@@@@@@@&((((((((((((.
.@@@@@@@@@@@@@@@@@@@@%(((((((((((.
*@@@@@@@@@@@@@@@@@@@@&(((((((((((,
.%@@@@@@@@@@@@@@@@@@@@@@#((((((((((*
@@@@@&&&@@@@@@@@@@@@@@@@@@@@@@@@@@@@#((((((((((/
%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#(((((((((((.
,&@@@@@@@@@@@@@@@@@@@@@@@@@@@#((((((((((((.
,#@@@@@@@@@@@@@@@@@@@@@&#((((((((((((* .,**.
./(#%@@@@@@@@@@@&#((((((((((((((/ *(((((((/
*(((((((((((((((((((((((/. (((((((((
.*/(((((((((((/*. /((((((.
```
Apibara provides data streams for all Starknet networks. Notice that these
stream URLs are going to change in the future when DNA v2 is released.
**Starknet Mainnet**
```txt
https://starknet.preview.apibara.org
```
## Starknet appchains
You can ingest data from Starknet appchains and serve it using our open-source
DNA service. Please get in touch with the team if you'd like a managed solution.
---
title: Starknet filter reference
description: "Starknet: DNA data filter reference guide."
diataxis: reference
updatedAt: 2024-10-23
---
# Starknet filter reference
This page contains reference about the available data filters for Starknet DNA
streams.
### Related pages
- [Starknet block data reference](/docs/v2/networks/starknet/data)
## Filter ID
All filters have an associated ID. When the server filters a block, it will
return a list of all filters that matched a piece of data with the data.
You can use this ID to build powerful abstractions in your indexers.
## Field elements
Apibara represents Starknet field elements as hex-encode strings.
```ts
export type FieldElement = `0x${string}`;
```
## Filter types
### Root
The root filter object contains a collection of filters.
Notice that providing an empty filter object is an error.
```ts
export type Filter = {
header?: HeaderFilter;
transactions?: TransactionFilter[];
events?: EventFilter[];
messages?: MessageToL1Filter[];
storageDiffs?: StorageDiffFilter[];
contractChanges?: ContractChangeFilter[];
nonceUpdates?: NonceUpdateFilter[];
};
```
### Header
The `HeaderFilter` object controls when the block header is returned to the client.
```ts
export type HeaderFilter = "always" | "on_data" | "on_data_or_on_new_block";
```
The values have the following meaning:
- `always`: Always return the header, even if no other filter matches.
- `on_data`: Return the header only if any other filter matches. This is the default value.
- `on_data_or_on_new_block`: Return the header only if any other filter matches. If no other filter matches, return the header only if the block is a new block.
### Events
Events are the most common filter used by Apibara users. You can filter by smart contract or event selector.
```ts
export type EventFilter = {
id?: number;
address?: FieldElement;
keys?: (FieldElement | null)[];
strict?: boolean;
transactionStatus?: "succeeded" | "reverted" | "all";
includeTransaction?: boolean;
includeReceipt?: boolean;
includeMessages?: boolean;
includeSiblings?: boolean;
};
```
**Properties**
- `address`: filter by contract address. If empty, matches any contract
address.
- `keys`: filter by keys. Use `null` to match _any_ value. The server will
filter based only the first four elements of the array.
- `strict`: return events whose keys length matches the filter. By default, the
filter does a prefix match on the keys.
- `transactionStatus`: return events emitted by transactions with the provided
status. Defaults to `succeeded`.
- `includeTransaction`: also return the transaction that emitted the event.
- `includeReceipt`: also return the receipt of the transaction that emitted the
event.
- `includeMessages`: also return all messages to L1 sent by the transaction that
emitted the event.
- `includeSiblings`: also return all other events emitted by the same transaction
that emitted the matched event.
**Examples**
- All events from a specific smart contract.
```ts
const filter = {
events: [{ address: MY_CONTRACT }],
};
```
- Multiple events from the same smart contract.
```ts
const filter = {
events: [
{
address: MY_CONTRACT,
keys: [APPROVE_SELECTOR],
},
{
address: MY_CONTRACT,
keys: [TRANSFER_SELECTOR],
},
],
};
```
- Multiple events from different smart contracts.
```ts
const filter = {
events: [
{
address: CONTRACT_A,
keys: [TRANSFER_SELECTOR],
},
{
address: CONTRACT_B,
keys: [TRANSFER_SELECTOR],
includeReceipt: false,
},
{
address: CONTRACT_C,
keys: [TRANSFER_SELECTOR],
},
],
};
```
### Transactions
Transactions on Starknet can be of different type (invoke, declare contract, deploy contract or account, handle L1 message). Clients can request all transactions or filter by transaction type.
```ts
export type InvokeTransactionV0Filter = {
_tag: "invokeV0";
invokeV0: {};
};
export type InvokeTransactionV1Filter = {
_tag: "invokeV1";
invokeV1: {};
};
export type InvokeTransactionV3Filter = {
_tag: "invokeV3";
invokeV3: {};
};
export type DeployTransactionFilter = {
_tag: "deploy";
deploy: {};
};
export type DeclareV0TransactionFilter = {
_tag: "declareV0";
declareV0: {};
};
export type DeclareV1TransactionFilter = {
_tag: "declareV1";
declareV1: {};
};
export type DeclareV2TransactionFilter = {
_tag: "declareV2";
declareV2: {};
};
export type DeclareV3TransactionFilter = {
_tag: "declareV3";
declareV3: {};
};
export type L1HandlerTransactionFilter = {
_tag: "l1Handler";
l1Handler: {};
};
export type DeployAccountV1TransactionFilter = {
_tag: "deployAccountV1";
deployAccountV1: {};
};
export type DeployAccountV3TransactionFilter = {
_tag: "deployAccountV3";
deployAccountV3: {};
};
export type TransactionFilter = {
id?: number;
transactionStatus?: "succeeded" | "reverted" | "all";
includeReceipt?: boolean;
includeMessages?: boolean;
includeEvents?: boolean;
transactionType?:
| InvokeTransactionV0Filter
| InvokeTransactionV1Filter
| InvokeTransactionV3Filter
| DeployTransactionFilter
| DeclareV0TransactionFilter
| DeclareV1TransactionFilter
| DeclareV2TransactionFilter
| DeclareV3TransactionFilter
| L1HandlerTransactionFilter
| DeployAccountV1TransactionFilter
| DeployAccountV3TransactionFilter;
};
```
**Properties**
- `transactionStatus`: return transactions with the provided status. Defaults to
`succeeded`.
- `includeReceipt`: also return the receipt of the transaction.
- `includeMessages`: also return the messages to L1 sent by the transaction.
- `includeEvents`: also return the events emitted by the transaction.
- `transactionType`: filter by transaction type.
**Examples**
- Request all transactions in a block. Notice the empty transaction filter object, this filter will match _any_ transaction.
```ts
const filter = { transactions: [{}] };
```
- Request all transactions of a specific type, e.g. deploy account. In this case we specify the `deployAccountV3` variant.
```ts
const filter = {
transactions: [{
transactionType: { _tag: "deployAccountV3", deployAccountV3: {} }
}]
};
```
### Messages
Filter messages from L1 to Starknet.
```ts
export type MessageToL1Filter = {
id?: number;
fromAddress?: FieldElement;
toAddress?: FieldElement;
transactionStatus?: "succeeded" | "reverted" | "all";
includeTransaction?: boolean;
includeReceipt?: boolean;
includeEvents?: boolean;
};
```
**Properties**
- `fromAddress`: filter by sender address. If empty, matches any sender address.
- `toAddress`: filter by receiver address. If empty, matches any receiver address.
- `transactionStatus`: return messages with the provided status. Defaults to
`succeeded`.
- `includeTransaction`: also return the transaction that sent the message.
- `includeReceipt`: also return the receipt of the transaction that sent the
message.
- `includeEvents`: also return the events emitted by the transaction that sent
the message.
### Storage diff
Request changes to the storage of one or more contracts.
```ts
export type StorageDiffFilter = {
id?: number;
contractAddress?: FieldElement;
};
```
**Properties**
- `contractAddress`: filter by contract address. If empty, matches any contract
address.
### Contract change
Request changes to the declared or deployed contracts.
```ts
export type DeclaredClassFilter = {
_tag: "declaredClass";
declaredClass: {};
};
export type ReplacedClassFilter = {
_tag: "replacedClass";
replacedClass: {};
};
export type DeployedContractFilter = {
_tag: "deployedContract";
deployedContract: {};
};
export type ContractChangeFilter = {
id?: number;
change?: DeclaredClassFilter | ReplacedClassFilter | DeployedContractFilter;
};
```
**Properties**
- `change`: filter by change type.
+ `declaredClass`: receive declared classes.
+ `replacedClass`: receive replaced classes.
+ `deployedContract`: receive deployed contracts.
### Nonce update
Request changes to the nonce of one or more contracts.
```ts
export type NonceUpdateFilter = {
id?: number;
contractAddress?: FieldElement;
};
```
**Properties**
- `contractAddress`: filter by contract address. If empty, matches any contract.
---
title: Starknet data reference
description: "Starknet: DNA data data reference guide."
diataxis: reference
updatedAt: 2024-10-23
---
# Starknet data reference
This page contains reference about the available data in Starknet DNA streams.
### Related pages
- [Starknet data filter reference](/docs/v2/networks/starknet/filter)
## Filter ID
All filters have an associated ID. To help clients correlate filters with data,
the filter ID is included in the `filterIds` field of all data objects.
This field contains the list of _all filter IDs_ that matched a piece of data.
## Nullable fields
**Important**: most fields are nullable to allow evolving the protocol. You should
always assert the presence of a field for critical indexers.
## Field elements
Apibara represents Starknet field elements as hex-encode strings.
```ts
export type FieldElement = `0x${string}`;
```
## Data types
### Block
The root object is the `Block`.
```ts
export type Block = {
header?: BlockHeader;
transactions: Transaction[];
receipts: TransactionReceipt[];
events: Event[];
messages: MessageToL1[];
storageDiffs: StorageDiff[];
contractChanges: ContractChange[];
nonceUpdates: NonceUpdate[];
};
```
### Block header
This is the block header, which contains information about the block.
```ts
export type BlockHeader = {
blockHash?: FieldElement;
parentBlockHash?: FieldElement;
blockNumber?: bigint;
sequencerAddress?: FieldElement;
newRoot?: FieldElement;
timestamp?: Date;
starknetVersion?: string;
l1GasPrice?: ResourcePrice;
l1DataGasPrice?: ResourcePrice;
l1DataAvailabilityMode?: "blob" | "calldata";
};
export type ResourcePrice = {
priceInFri?: FieldElement;
priceInWei?: FieldElement;
};
```
**Properties**
- `blockHash`: the block hash.
- `parentBlockHash`: the block hash of the parent block.
- `blockNumber`: the block number.
- `sequencerAddress`: the sequencer address.
- `newRoot`: the new state root.
- `timestamp`: the block timestamp.
- `starknetVersion`: the Starknet version.
- `l1GasPrice`: the L1 gas price.
- `l1DataGasPrice`: the L1 data gas price.
- `l1DataAvailabilityMode`: the L1 data availability mode.
- `priceInFri`: the price of L1 gas in the block, in units of fri (10^-18 $STRK).
- `priceInWei`: the price of L1 gas in the block, in units of wei (10^-18 $ETH).
### Event
An event is emitted by a transaction.
```ts
export type Event = {
filterIds: number[];
address?: FieldElement;
keys: FieldElement[];
data: FieldElement[];
eventIndex?: number;
transactionIndex?: number;
transactionHash?: FieldElement;
transactionStatus?: "succeeded" | "reverted";
};
```
**Properties**
- `address`: the address of the contract that emitted the event.
- `keys`: the keys of the event.
- `data`: the data of the event.
- `eventIndex`: the index of the event in the block.
- `transactionIndex`: the index of the transaction that emitted the event.
- `transactionHash`: the hash of the transaction that emitted the event.
- `transactionStatus`: the status of the transaction that emitted the event.
**Relevant filters**
- `filter.events`
- `filter.transactions[].includeEvents`
- `filter.events[].includeSiblings`
- `filter.messages[].includeEvents`
### Transaction
Starknet has different types of transactions, all of them are grouped together
in the `Transaction` type. Common transaction information is accessible in the
`meta` field.
```ts
export type TransactionMeta = {
transactionIndex?: number;
transactionHash?: FieldElement;
transactionStatus?: "succeeded" | "reverted";
};
export type Transaction = {
filterIds: number[];
meta?: TransactionMeta;
transaction?:
| InvokeTransactionV0
| InvokeTransactionV1
| InvokeTransactionV3
| L1HandlerTransaction
| DeployTransaction
| DeclareTransactionV0
| DeclareTransactionV1
| DeclareTransactionV2
| DeclareTransactionV3
| DeployAccountTransactionV1
| DeployAccountTransactionV3;
};
```
**Properties**
- `meta`: transaction metadata.
- `transaction`: the transaction type.
- `meta.transactionIndex`: the index of the transaction in the block.
- `meta.transactionHash`: the hash of the transaction.
- `meta.transactionStatus`: the status of the transaction.
**Relevant filters**
- `filter.transactions`
- `filter.events[].includeTransaction`
- `filter.messages[].includeTransaction`
```ts
export type InvokeTransactionV0 = {
_tag: "invokeV0";
invokeV0: {
maxFee?: FieldElement;
signature: FieldElement[];
contractAddress?: FieldElement;
entryPointSelector?: FieldElement;
calldata: FieldElement[];
};
}
```
**Properties**
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `contractAddress`: the address of the contract that will receive the call.
- `entryPointSelector`: the selector of the function that will be called.
- `calldata`: the calldata of the transaction.
```ts
export type InvokeTransactionV1 = {
_tag: "invokeV1";
invokeV1: {
senderAddress?: FieldElement;
calldata: FieldElement[];
maxFee?: FieldElement;
signature: FieldElement[];
nonce?: FieldElement;
};
};
```
**Properties**
- `senderAddress`: the address of the account that will send the transaction.
- `calldata`: the calldata of the transaction.
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `nonce`: the nonce of the transaction.
```ts
export type ResourceBounds = {
maxAmount?: bigint;
maxPricePerUnit?: bigint;
}
export type ResourceBoundsMapping = {
l1Gas?: ResourceBounds;
l2Gas?: ResourceBounds;
};
export type InvokeTransactionV3 = {
_tag: "invokeV3";
invokeV3: {
senderAddress?: FieldElement;
calldata: FieldElement[];
signature: FieldElement[];
nonce?: FieldElement;
resourceBounds?: ResourceBoundsMapping;
tip?: bigint;
paymasterData: FieldElement[];
accountDeploymentData: FieldElement[];
nonceDataAvailabilityMode?: "l1" | "l2";
feeDataAvailabilityMode?: "l1" | "l2";
};
};
```
**Properties**
- `senderAddress`: the address of the account that will send the transaction.
- `calldata`: the calldata of the transaction.
- `signature`: the signature of the transaction.
- `nonce`: the nonce of the transaction.
- `resourceBounds`: the resource bounds of the transaction.
- `tip`: the tip of the transaction.
- `paymasterData`: the paymaster data of the transaction.
- `accountDeploymentData`: the account deployment data of the transaction.
- `nonceDataAvailabilityMode`: the nonce data availability mode of the transaction.
- `feeDataAvailabilityMode`: the fee data availability mode of the transaction.
```ts
export type L1HandlerTransaction = {
_tag: "l1Handler";
l1Handler: {
contractAddress?: FieldElement;
entryPointSelector?: FieldElement;
calldata: FieldElement[];
nonce?: bigint;
};
}
```
**Properties**
- `contractAddress`: the address of the contract that will receive the call.
- `entryPointSelector`: the selector of the function that will be called.
- `calldata`: the calldata of the transaction.
- `nonce`: the nonce of the transaction.
```ts
export type DeployTransaction = {
_tag: "deploy";
deploy: {
contractAddressSalt?: FieldElement;
constructorCalldata: FieldElement[];
classHash?: FieldElement;
};
}
```
**Properties**
- `contractAddressSalt`: the salt used to compute the contract address.
- `constructorCalldata`: the calldata used to initialize the contract.
- `classHash`: the class hash of the contract.
```ts
export type DeclareTransactionV0 = {
_tag: "declareV0";
declareV0: {
senderAddress?: FieldElement;
maxFee?: FieldElement;
signature: FieldElement[];
classHash?: FieldElement;
};
}
```
**Properties**
- `senderAddress`: the address of the account that will send the transaction.
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `classHash`: the class hash of the contract.
```ts
export type DeclareTransactionV1 = {
_tag: "declareV1";
declareV1: {
senderAddress?: FieldElement;
maxFee?: FieldElement;
signature: FieldElement[];
classHash?: FieldElement;
nonce?: FieldElement;
};
}
```
**Properties**
- `senderAddress`: the address of the account that will send the transaction.
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `classHash`: the class hash of the contract.
- `nonce`: the nonce of the transaction.
```ts
export type DeclareTransactionV2 = {
_tag: "declareV2";
declareV2: {
senderAddress?: FieldElement;
maxFee?: FieldElement;
signature: FieldElement[];
classHash?: FieldElement;
nonce?: FieldElement;
compiledClassHash?: FieldElement;
};
}
```
**Properties**
- `senderAddress`: the address of the account that will send the transaction.
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `classHash`: the class hash of the contract.
- `nonce`: the nonce of the transaction.
- `compiledClassHash`: the compiled class hash of the contract.
```ts
export type ResourceBounds = {
maxAmount?: bigint;
maxPricePerUnit?: bigint;
}
export type ResourceBoundsMapping = {
l1Gas?: ResourceBounds;
l2Gas?: ResourceBounds;
};
export type DeclareTransactionV3 = {
_tag: "declareV3";
declareV3: {
senderAddress?: FieldElement;
maxFee?: FieldElement;
signature: FieldElement[];
classHash?: FieldElement;
nonce?: FieldElement;
compiledClassHash?: FieldElement;
resourceBounds?: ResourceBoundsMapping;
tip?: bigint;
paymasterData: FieldElement[];
accountDeploymentData: FieldElement[];
nonceDataAvailabilityMode?: "l1" | "l2";
feeDataAvailabilityMode?: "l1" | "l2";
};
}
```
**Properties**
- `senderAddress`: the address of the account that will send the transaction.
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `classHash`: the class hash of the contract.
- `nonce`: the nonce of the transaction.
- `compiledClassHash`: the compiled class hash of the contract.
- `resourceBounds`: the resource bounds of the transaction.
- `tip`: the tip of the transaction.
- `paymasterData`: the paymaster data of the transaction.
- `accountDeploymentData`: the account deployment data of the transaction.
- `nonceDataAvailabilityMode`: the nonce data availability mode of the transaction.
- `feeDataAvailabilityMode`: the fee data availability mode of the transaction.
```ts
export type DeployAccountTransactionV1 = {
_tag: "deployAccountV1";
deployAccountV1: {
maxFee?: FieldElement;
signature: FieldElement[];
nonce?: FieldElement;
contractAddressSalt?: FieldElement;
constructorCalldata: FieldElement[];
classHash?: FieldElement;
};
}
```
**Properties**
- `maxFee`: the maximum fee for the transaction.
- `signature`: the signature of the transaction.
- `nonce`: the nonce of the transaction.
- `contractAddressSalt`: the salt used to compute the contract address.
- `constructorCalldata`: the calldata used to initialize the contract.
- `classHash`: the class hash of the contract.
```ts
export type DeployAccountTransactionV3 = {
_tag: "deployAccountV3";
deployAccountV3: {
signature: FieldElement[];
nonce?: FieldElement;
contractAddressSalt?: FieldElement;
constructorCalldata: FieldElement[];n
classHash?: FieldElement;
resourceBounds?: ResourceBoundsMapping;
tip?: bigint;
paymasterData: FieldElement[];
nonceDataAvailabilityMode?: "l1" | "l2";
feeDataAvailabilityMode?: "l1" | "l2";
};
};
```
**Properties**
- `signature`: the signature of the transaction.
- `nonce`: the nonce of the transaction.
- `contractAddressSalt`: the salt used to compute the contract address.
- `constructorCalldata`: the calldata used to initialize the contract.
- `classHash`: the class hash of the contract
- `resourceBounds`: the resource bounds of the transaction.
- `tip`: the tip of the transaction.
- `paymasterData`: the paymaster data of the transaction.
- `nonceDataAvailabilityMode`: the nonce data availability mode of the transaction.
- `feeDataAvailabilityMode`: the fee data availability mode of the transaction.
### Transaction receipt
The receipt of a transaction contains information about the execution of the transaction.
```ts
export type TransactionReceipt = {
filterIds: number[];
meta?: TransactionReceiptMeta;
receipt?:
| InvokeTransactionReceipt
| L1HandlerTransactionReceipt
| DeclareTransactionReceipt
| DeployTransactionReceipt
| DeployAccountTransactionReceipt;
};
export type TransactionReceiptMeta = {
transactionIndex?: number;
transactionHash?: FieldElement;
actualFee?: FeePayment;
executionResources?: ExecutionResources;
executionResult?: ExecutionSucceeded | ExecutionReverted;
};
export type InvokeTransactionReceipt = {
_tag: "invoke";
invoke: {}
};
export type L1HandlerTransactionReceipt = {
_tag: "l1Handler";
l1Handler: {
messageHash?: Uint8Array;
};
};
export type DeclareTransactionReceipt = {
_tag: "declare";
declare: {};
};
export type DeployTransactionReceipt = {
_tag: "deploy";
deploy: {
contractAddress?: FieldElement;
};
};
export type DeployAccountTransactionReceipt = {
_tag: "deployAccount";
deployAccount: {
contractAddress?: FieldElement;
};
};
export type ExecutionSucceeded = {
_tag: "succeeded";
succeeded: {};
};
export type ExecutionReverted = {
_tag: "reverted";
reverted: {
reason: string;
};
};
export type FeePayment = {
amount?: FieldElement;
unit?: "wei" | "strk";
};
```
**Relevant filters**
- `filter.transactions[].includeReceipt`
- `filter.events[].includeReceipt`
- `filter.messages[].includeReceipt`
### Message to L1
A message to L1 is sent by a transaction.
```ts
export type MessageToL1 = {
filterIds: number[];
fromAddress?: FieldElement;
toAddress?: FieldElement;
payload: FieldElement[];
messageIndex?: number;
transactionIndex?: number;
transactionHash?: FieldElement;
transactionStatus?: "succeeded" | "reverted";
};
```
**Properties**
- `fromAddress`: the address of the contract that sent the message.
- `toAddress`: the address of the contract that received the message.
- `payload`: the payload of the message.
- `messageIndex`: the index of the message in the block.
- `transactionIndex`: the index of the transaction that sent the message.
- `transactionHash`: the hash of the transaction that sent the message.
- `transactionStatus`: the status of the transaction that sent the message.
**Relevant filters**
- `filter.messages`
- `filter.transactions[].includeMessages`
- `filter.events[].includeMessages`
### Storage diff
A storage diff is a change to the storage of a contract.
```ts
export type StorageDiff = {
filterIds: number[];
contractAddress?: FieldElement;
storageEntries: StorageEntry[];
};
export type StorageEntry = {
key?: FieldElement;
value?: FieldElement;
};
```
**Properties**
- `contractAddress`: the contract whose storage changed.
- `storageEntries`: the storage entries that changed.
- `key`: the key of the storage entry that changed.
- `value`: the new value of the storage entry that changed.
**Relevant filters**
- `filter.storageDiffs`
### Contract change
A change in the declared or deployed contracts.
```ts
export type ContractChange = {
filterIds: number[];
change?: DeclaredClass | ReplacedClass | DeployedContract;
};
export type DeclaredClass = {
_tag: "declaredClass";
declaredClass: {
classHash?: FieldElement;
compiledClassHash?: FieldElement;
};
};
export type ReplacedClass = {
_tag: "replacedClass";
replacedClass: {
contractAddress?: FieldElement;
classHash?: FieldElement;
};
};
export type DeployedContract = {
_tag: "deployedContract";
deployedContract: {
contractAddress?: FieldElement;
classHash?: FieldElement;
};
};
```
**Relevant filters**
- `filter.contractChanges`
### Nonce update
A change in the nonce of a contract.
```ts
export type NonceUpdate = {
filterIds: number[];
contractAddress?: FieldElement;
nonce?: FieldElement;
};
```
**Properties**
- `contractAddress`: the address of the contract whose nonce changed.
- `nonce`: the new nonce of the contract.
**Relevant filters**
- `filter.nonceUpdates`
---
title: Upgrading from v1
description: "This page explains how to upgrade from DNA v1 to DNA v2."
diataxis: how-to
updatedAt: 2024-11-06
---
# Upgrading from v1
This page contains a list of changes between DNA v1 and DNA v2.
## @apibara/starknet package
This package now works in combination with `@apibara/protocol` to provide a DNA stream
that automatically encodes and decodes the Protobuf data. This means tha field elements
are automatically converted to `0x${string}` values.
Notice that the data stream is now unary.
```js
import { createClient } from "@apibara/protocol";
import { Filter, StarknetStream } from "@apibara/starknet";
const client = createClient(StarknetStream, process.env.STREAM_URL);
const filter = Filter.make({
events: [{
address:
"0x049d36570d4e46f48e99674bd3fcc84644ddd6b96f7c741b1562b82f9e004dc7",
}],
});
const request = StarknetStream.Request.make({
filter: [filter],
finality: "accepted",
startingCursor: {
orderKey: 800_000n,
},
});
for await (const message of client.streamData(request)) {
switch (message._tag) {
case "data": {
break;
}
case "invalidate": {
break;
}
default: {
break;
}
}
}
```
### Reconnecting on error
The client now doesn't automatically reconnect on error. This is because the reconnection step
is very delicate and depends on your indexer's implementation.
The recommended approach is to wrap your indexer's main loop in a `try/catch` block.
```ts
import { createClient, type ClientError, type Status } from "@apibara/protocol";
import { Filter, StarknetStream } from "@apibara/starknet";
const client = createClient(StarknetStream, process.env.STREAM_URL);
const filter = Filter.make({
events: [{
address:
"0x049d36570d4e46f48e99674bd3fcc84644ddd6b96f7c741b1562b82f9e004dc7",
}],
});
while (true) {
try {
const startingCursor = await loadCursorFromDatabase();
const request = StarknetStream.Request.make({
filter: [filter],
finality: "accepted",
startingCursor,
});
for await (const message of client.streamData(request)) {
}
} catch (err) {
if (err instanceof ClientError) {
// It's a gRPC error.
if (err.status !== Status.INTERNAL) {
// NON-INTERNAL errors are not recoverable.
throw err;
}
// INTERNAL errors are caused by a disconnection.
// Sleep and reconnect.
await new Promise(r => setTimeout(r, 2_000));
}
}
}
```
## Filter
### Header
- The `header` field is now an enum. See the [dedicated section](/docs/v2/networks/starknet/filter#header)
in the filter documentation for more information.
### Events
- `fromAddress` is now `address`.
- The `keys` field accepts `null` values to match any key at that position.
- The `data` field was removed.
- Use `transactionStatus: "all"` instead of `includeReverted` to include reverted transactions.
- `includeReceipt` and `includeTransaction` are now `false` by default.
### Transactions
- Now you can only filter by transaction type.
- We will add transaction-specific filters in the future.
- Use `transactionStatus: "all"` instead of `includeReverted` to include reverted transactions.
- `includeReceipt` is now `false` by default.
### Messages
- Can now filter by `fromAddress` and `toAddress`.
- Use `transactionStatus: "all"` instead of `includeReverted` to include reverted transactions.
- `includeReceipt` and `includeTransaction` are now `false` by default.
### State Update
- State update has been split into separate filters for storage diffs, contract
changes, and nonce updates.
- Declared and deployed contracts, declared classes, and replaced classes are now
a single `contractChanges` filter.
## Block data
- Block data has been _"flattened"_. Use the `*Index` field to access related data.
For example, the following code iterates over all events and looks up their
transactions.
```js
for (const event of block.events) {
const transaction = block.transactions.find(tx => tx.transactionIndex === event.transactionIndex);
}
```
### Events
- `fromAddress` is now `address`.
- `index` is now `eventIndex`.
- Events now include `transactionIndex`, `transactionHash`, and `transactionStatus`.
### Transactions
- `TransactionMeta` now includes `transactionIndex`, `transactionHash`, and `transactionStatus`.
- The transaction type is now an enum using the `_tag` field as discriminator.
- For other minor changes, see the [transaction documentation](/docs/v2/networks/starknet/data#transaction).
### Receipts
- Transaction receipts are now transaction-specific.
- For other minor changes, see the [receipts documentation](/docs/v2/networks/starknet/data#transaction-receipt).
### Messages
- `index` is now `messageIndex`.
- Messages now include `transactionIndex`, `transactionHash`, and `transactionStatus`.
---
title: DNA protocol & architecture
description: "Learn about the low-level DNA streaming protocol to access onchain data."
diataxis: explanation
updatedAt: 2024-09-20
---
# DNA protocol & architecture
This section describes the internals of DNA v2.
- [Wire protocol](/docs/v2/dna/protocol): describes the gRPC streaming
protocol. This page is useful if you're connecting directly to the stream or
are adding support for a new programming language.
- [Architecture](/docs/v2/dna/architecture): describes the high-level components of DNA v2.
- [Adding a new chain](/docs/v2/dna/add-new-chain): describes what you
need to do to bring DNA to a new chain. It digs deeper into anything
chain-specific like storage and filters.
---
title: DNA v2 architecture
description: "Discover how DNA achieves best-in-class performance for indexing onchain data."
diataxis: explanation
updatedAt: 2024-09-20
---
# DNA v2 architecture
This page describes in detail the architecture of DNA v2.
At a high-level, the goals for DNA v2 are:
- serve onchain data through a protocol that's optimized for building indexers.
- provide a scalable and cost-efficient way to access onchain data.
- decouple compute from storage.
This is achieved by building a _cloud native_ service that ingests onchain data
from an archive node and stores it into Object Storage (for example Amazon S3,
Cloudflare R2). Data is served by stateless workers that read and filter data
from Object Storage before sending it to the indexers. The diagram below shows
all the high-level components that make a production deployment of DNA v2.
Communication between components is done through etcd.
```txt
┌─────────────────────────────────────────────┐
│ Archive Node │░
└─────────────────────────────────────────────┘░
░░░░░░░░░░░░░░░░░░░░░░│░░░░░░░░░░░░░░░░░░░░░░░░
│
│
╔═ DNA Cluster ═══════════════════════╬══════════════════════════════════════╗
║ │ ║░
║ ┌──────┐ ▼ ┌──────┐ ║░
║ │ │ ┌─────────────────────────────────────────────┐ │ │ ║░
║ │ │ │ │ │ │ ║░
║ │ │◀────│ Ingestion Service │────▶│ │ ║░
║ │ │ │ │ │ │ ║░
║ │ │ └─────────────────────────────────────────────┘ │ │ ║░
║ │ │ ┌─────────────────────────────────────────────┐ │ │ ║░
║ │ │ │ │ │ │ ║░
║ │ │◀────│ Compaction Service │────▶│ │ ║░
║ │ │ │ │ │ │ ║░
║ │ │ └─────────────────────────────────────────────┘ │ │ ║░
║ │ S3 │ ┌─────────────────────────────────────────────┐ │ etcd │ ║░
║ │ │ │ │ │ │ ║░
║ │ │◀────│ Pruning Service │────▶│ │ ║░
║ │ │ │ │ │ │ ║░
║ │ │ └─────────────────────────────────────────────┘ │ │ ║░
║ │ │ ┌───────────────────────────────────────────┐ │ │ ║░
║ │ │ │┌──────────────────────────────────────────┴┐ │ │ ║░
║ │ │ ││┌──────────────────────────────────────────┴┐ │ │ ║░
║ │ │ │││ │ │ │ ║░
║ │ │ │││ Stream │ │ │ ║░
║ │ │◀────┤││ ├────▶│ │ ║░
║ │ │ │││ Service │ │ │ ║░
║ └──────┘ └┤│ │ └──────┘ ║░
║ └┤ │ ║░
║ └───────────────────────────────────────────┘ ║░
║ ║░
╚════════════════════════════════════════════════════════════════════════════╝░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
```
## DNA service
The DNA service is comprised of several components:
- ingestion service: listens for new blocks on the network and stores them into
Object Storage.
- compaction service: combines multiple blocks together into _segments_.
Segments are grouped by data type (like logs, transactions, and receipts).
- pruner service: removes blocks that have been compacted to reduce storage cost.
- stream service: receives streaming requests from clients (indexers) and serves
onchain data by filtering objects stored on S3.
### Ingestion service
The ingestion service fetches blocks from the network and stores them into
Object Storage. This service is the only chain-specific service in DNA, all
other components work on generic data-structures.
Serving onchain data requires serving a high-volume of data filtered by a
relatively small number of columns. When designing DNA, we took a few decisions
to make this process as efficient as possible:
- data is stored as pre-serialized protobuf messages to avoid wasting CPU
cycles serializing the same data over and over again.
- filtering is entirely done using indices to reduce reads.
- joins (for example include logs' transactions) are also achieved with indices.
The ingestion service is responsible for creating this data and indices.
Data is grouped into _blocks_. Blocks are comprised of _fragments_, that
is groups of related data. All fragments have an unique numerical id used
to identify them. There are four different types of fragments:
- index: a collection of indices, the fragment id is `0`.
Indices are grouped by the fragment they index.
- join: a collection of join indices, the fragment id is `254`.
Join indices are also grouped by the source fragment index.
- header: the block header, the fragment id is `1`. Header are stored as
pre-serialized protobuf messages.
- body: the chain-specific block data, grouped by fragment id.
Note that we call block number + hash a _cursor_ since it uniquely identifies a
block in the chain.
```txt
╔═ Block ══════════════════════════════════════════════════════════════╗
║ ┌─ Index ──────────────────────────────────────────────────────────┐ ║░
║ │ ┌─ Fragment 0 ─────────────────────────────────────────────────┐ │ ║░
║ │ │┌────────────────────────────────────────────────────────────┐│ │ ║░
║ │ ││ Index 0 ││ │ ║░
║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
║ │ ││ Index 1 ││ │ ║░
║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
║ │ │ │ │ ║░
║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
║ │ ││ Index N ││ │ ║░
║ │ │└────────────────────────────────────────────────────────────┘│ │ ║░
║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
║ └──────────────────────────────────────────────────────────────────┘ ║░
║ ┌─ Join ───────────────────────────────────────────────────────────┐ ║░
║ │ ┌─ Fragment 0 ─────────────────────────────────────────────────┐ │ ║░
║ │ │┌────────────────────────────────────────────────────────────┐│ │ ║░
║ │ ││ Fragment 1 ││ │ ║░
║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
║ │ ││ Fragment 2 ││ │ ║░
║ │ │└────────────────────────────────────────────────────────────┘│ │ ║░
║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
║ └──────────────────────────────────────────────────────────────────┘ ║░
║ ┌─ Body ───────────────────────────────────────────────────────────┐ ║░
║ │ ┌──────────────────────────────────────────────────────────────┐ │ ║░
║ │ │ │ │ ║░
║ │ │ Fragment 0 │ │ ║░
║ │ │ │ │ ║░
║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
║ │ ┌──────────────────────────────────────────────────────────────┐ │ ║░
║ │ │ │ │ ║░
║ │ │ Fragment 1 │ │ ║░
║ │ │ │ │ ║░
║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
║ └──────────────────────────────────────────────────────────────────┘ ║░
╚══════════════════════════════════════════════════════════════════════╝░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
```
On supported networks, the ingestion service is also responsible for
periodically refreshing the mempool (pending) block data and uploading it into
Object Storage. This works exactly as for all other blocks.
The ingestion service tracks the canonical chain and uploads it to Object Storage.
This data is used by the stream service to track online and offline chain
reorganizations.
The ingestion service stores its data on etcd. Stream services subscribe to
etcd updates to receive notifications about new blocks ingested and other
changes to the chain (for example changes to the finalized block).
Finally, the ingestion service is _fault tolerant_. When the ingestion service
starts, it acquires a distributed lock from etcd to ensure only one instance is
running at the same time. If running multiple deployments of DNA, all other
instances will wait for the lock to be released (following a service restart or
crash) and will try to take over the ingestion.
### Compaction service
The compaction service groups together data from several blocks (usually 100 or
1000) into _segments_. Segments only contain data for one fragment type (for example
headers, indices, and transactions).
In other words, the compaction service groups `N` blocks into `M` segments.
Only data that has been finalized is compacted into segments.
The compaction service also creates block-level indices called _groups_. Groups
combine indices from multiple blocks/segments to quickly look up which blocks
contain specific data. This type of index is very useful to increase performance
on sparse datasets.
```txt
╔═ Index Segment ═══════════════════════╗ ╔═ Transaction Segment ═════════════════╗
║ ┌─ Block ───────────────────────────┐ ║░ ║ ┌─ Block ───────────────────────────┐ ║░
║ │ ┌─ Fragment 0 ──────────────────┐ │ ║░ ║ │ ┌───────────────────────────────┐ │ ║░
║ │ │┌─────────────────────────────┐│ │ ║░ ║ │ │ │ │ ║░
║ │ ││ Index 0 ││ │ ║░ ║ │ │ │ │ ║░
║ │ │├─────────────────────────────┤│ │ ║░ ║ │ │ Fragment 2 │ │ ║░
║ │ ││ Index 1 ││ │ ║░ ║ │ │ │ │ ║░
║ │ │├─────────────────────────────┤│ │ ║░ ║ │ │ │ │ ║░
║ │ │ │ │ ║░ ║ │ └───────────────────────────────┘ │ ║░
║ │ │├─────────────────────────────┤│ │ ║░ ║ └───────────────────────────────────┘ ║░
║ │ ││ Index N ││ │ ║░ ║ ┌─ Block ───────────────────────────┐ ║░
║ │ │└─────────────────────────────┘│ │ ║░ ║ │ ┌───────────────────────────────┐ │ ║░
║ │ └───────────────────────────────┘ │ ║░ ║ │ │ │ │ ║░
║ └───────────────────────────────────┘ ║░ ║ │ │ │ │ ║░
║ ┌─ Block ───────────────────────────┐ ║░ ║ │ │ Fragment 2 │ │ ║░
║ │ ┌─ Fragment 0 ──────────────────┐ │ ║░ ║ │ │ │ │ ║░
║ │ │┌─────────────────────────────┐│ │ ║░ ║ │ │ │ │ ║░
║ │ ││ Index 0 ││ │ ║░ ║ │ └───────────────────────────────┘ │ ║░
║ │ │├─────────────────────────────┤│ │ ║░ ║ └───────────────────────────────────┘ ║░
║ │ ││ Index 1 ││ │ ║░ ║ ┌─ Block ───────────────────────────┐ ║░
║ │ │├─────────────────────────────┤│ │ ║░ ║ │ ┌───────────────────────────────┐ │ ║░
║ │ │ │ │ ║░ ║ │ │ │ │ ║░
║ │ │├─────────────────────────────┤│ │ ║░ ║ │ │ Fragment 2 │ │ ║░
║ │ ││ Index N ││ │ ║░ ║ │ │ │ │ ║░
║ │ │└─────────────────────────────┘│ │ ║░ ║ │ │ │ │ ║░
║ │ └───────────────────────────────┘ │ ║░ ║ │ └───────────────────────────────┘ │ ║░
║ └───────────────────────────────────┘ ║░ ║ └───────────────────────────────────┘ ║░
╚═══════════════════════════════════════╝░ ╚═══════════════════════════════════════╝░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
```
### Pruner service
The pruner service cleans up block data that has been included in segments.
This is done to reduce the storage used by DNA.
### Object hierarchy
We now have all elements to understand the objects uploaded to Object Storage by the
ingestion service.
If you run DNA pointing it to your bucket, you can eventually see a folder structure
that looks like the following.
```txt
my-chain
├── blocks
│ ├── 000020908017
│ │ └── 0xc137607affd53bd9e857af372429762f77eaff0fe32f0e49224e9fc0e439118d
│ │ ├── pending-0
│ │ ├── pending-1
│ │ └── pending-2
│ ├── 000020908018
│ │ └── ... same as above
│ └── 000020908019
│ └── ... same as above
├── chain
│ ├── recent
│ ├── z-000020906000
│ ├── z-000020907000
│ └── z-000020908000
├── groups
│ └── 000020905000
│ └── index
└── segments
├── 000020906000
│ ├── header
│ ├── index
│ ├── join
│ ├── log
│ ├── receipt
│ └── transaction
├── 000020907000
│ └── ... same as above
└── 000020908000
└── ... same as above
```
### Stream service
The stream service is responsible for serving data to clients. The raw onchain
data stored in Object Storage is filtered by the stream service before being
sent over the network, this results in lower egress fees compared to solutions
that filter data on the client.
Upon receiving a stream request, the service validates and compiles the request
into a _query_. A query is simply a list of index lookup requests that are
applied to each block.
The stream service loops then keeps repeating the following steps:
- check if it should send a new block of data or inform the client of a chain
reorganization.
- load the indices from the segment or the block and use them to compute what
data to send the client.
- load the pre-serialized protobuf messages and copy them to the output stream.
One critical aspect of the stream service is how it loads blocks and segments.
Reading from Object Storage has virtually unlimited throughput, but also high
latency. The service is also very likely to access data closer to the chain's
tip more frequently, and we should cache Object Storage requests to avoid
unnecessarily increase our cloud spending.
We achieve all of this (and more!) by using an hybrid cache that stores
frequently accessed data in memory and _on disk_. This may come as a surprise
since isn't the point of DNA to avoid expensive disks and rely on cheap Object
Storage? The reasons this design still makes sense are multiple:
- we can use cheaper and higher performance temporary NVMe disks attached
directly to our server.
- we can quickly scale horizontally the stream service without re-indexing all
data.
- we can use disks that are much smaller than the full chain's data. The cache
dynamically stores the most frequently accessed data. Unused or rarely used
data lives on Object Storage.
The following table, inspired [by the table in this article by
Vantage](https://www.vantage.sh/blog/ebs-vs-nvme-pricing-performance), shows the
difference in performance and price between an AWS EC2 instance using
(temporary) NVMe disks and two using EBS (one with a general purpose `gp3`
volume, and one with higher performance `io1` volume). All prices as of April
2024, US East, 1 year reserved with no upfront payment.
| Metric | EBS (gp3) | EBS (io1) | NVMe |
| --------------------- | ----------- | ----------- | ----------------- |
| Instance Type | r6i.8xlarge | r6i.8xlarge | i4i.8xlarge |
| vCPU | 32 | 32 | 32 |
| Memory (GiB) | 256 | 256 | 245 |
| Network (Gibps) | 12.50 | 12.50 | 18.75 |
| Storage (GiB) | 7500 | 7500 | 2x3750 |
| IOPS (read/write) | 16,000 | 40,000 | 800,000 / 440,000 |
| Cost - Compute ($/mo) | 973 | 973 | 1,300 |
| Cost - Storage ($/mo) | 665 | 3,537 | 0 |
| Cost - Total ($/mo) | 1,638 | 4,510 | 1,300 |
Notice how the NVMe instance has 30-50x the IOPS per dollar. This price
difference means that Apibara users benefit from lower costs and/or higher
performance.
---
title: DNA wire protocol
description: "DNA is a protocol built on top of gRPC to stream onchain data."
diataxis: explanation
updatedAt: 2024-10-10
---
# DNA wire protocol
## `Cursor` message
Before explaining the DNA protocol in more detail, we're going to discuss the
`Cursor` message type. This type is used by all methods discussed later and
plays a central role in how DNA works.
DNA models a blockchain as a sequence of blocks. The distance of a block from
the first block in the chain (the genesis block) is known as chain height. The
genesis block has height `0`. Ideally, a blockchain should always build a block
on top of the most recent block, but that's not always the case. For this
reason, a block's height isn't enough to uniquely identify a block in the
blockchain. A _chain reorganization_ is when a chain produces blocks that are
not building on top of the most recent block. As we will see later, the DNA
protocol detects and handles chain reorganizations.
A block that can't be part of a chain reorganization is _finalized_.
DNA uses a _cursor_ to uniquely identify blocks on the chain. A cursor contains
two fields:
- `order_key`: the block's height.
- `unique_key`: the block's unique identifier. Depending on the chain, it's
the block hash or state root.
## `Status` method
The `Status` method is used to retrieve the state of the DNA server. The request
is an empty message. The response has the following fields:
- `last_ingested`: returns the last block ingested by the server. This is the most
recent block available for streaming.
- `finalized`: the most recent finalized block.
- `starting`: the first available block. Usually this is the genesis block,
but DNA server operators can prune older nodes to save on storage space.
## `StreamData` method
The `StreamData` method is used to start a DNA stream. It accepts a `StreamDataRequest`
message and returns an infinite stream of `StreamDataResponse` messages.
### Request
The request message is used to configure the stream. All fields except `filter`
are optional.
- `starting_cursor`: resume the stream from the provided cursor. The first
block received in the stream will be the block following the provided
cursor. If no cursor is provided, the stream will start from the genesis
block. Notice that since `starting_cursor` is a cursor, the DNA server can
detect if that block has been part of a chain's reorganization while the
indexer was offline.
- `finality`: the stream contains data with at least the specified finality.
Possible values are _finalized_ (only receive finalized data), _accepted_
(receive finalized and non-finalized blocks), and _pending_ (receive
finalized, non-finalized, and pending blocks).
- `filter`: a non-empty list of chain-specific data filters.
- `heartbeat_interval`: the stream will send an heartbeat message if there are
no messages for the specified amount of time. This is useful to detect if
the stream hangs. Value must be between 10 and 60 seconds.
### Response
Once the server validates and accepts the request, it starts streaming data.
Each stream message can be one of the following message types:
- `data`: receive data about a block.
- `invalidate`: the specified blocks don't belong to the canonical chain
anymore because they were part of a chain reorganization.
- `finalize`: the most recent finalized block moved forward.
- `heartbeat`: an heartbeat message.
- `system_message`: used to send messages from the server to the client.
#### `Data` message
Contains the requested data for a single block. All data messages cursors are
monotonically increasing, unless an `Invalidate` message is received.
The message contains the following fields:
- `cursor`: the cursor of the block before this message. If the client
reconnects using this cursor, the first message will be the same as this
message.
- `end_cursor`: this block's cursor. Reconnecting to the stream using this
cursor will resume the stream.
- `finality`: finality status of this block.
- `data`: a list of encoded block data.
Notice how the `data` field is a _list of block data_. This sounds
counter-intuitive since the `Data` message contains data about a _single
block_. The reason is that, as we've seen in the _"Request"_ section,
the client can specify a list of filters. The `data` field has the same
length as the request's `filters` field.
In most cases, the client specifies a single filter and receives a single block
of data. For advanced use cases (like tracking contracts deployed by a
factory), the client uses multiple filters to have parallel streams of data
synced on the block number.
#### `Invalidate` message
This message warns the client about a chain reorganization. It contains the
following fields:
- `cursor`: the new chain's head. All previously received messages where the
`end_cursor.order_key` was greater than (`>`) this message
`cursor.order_key` should be considered invalid/recalled.
- `removed`: a list of cursors that used to belong to the canonical chain.
#### `Finalize` message
This message contains a single `cursor` field with the cursor of the most
recent finalized block. All data at or before this block can't be part of a
chain reorganization.
This message is useful to prune old data.
#### `Heartbeat` message
This message is sent at regular intervals once the stream reaches the chain's head.
Clients can detect if the stream hang by adding a timeout to the stream's _receive_
method.
#### `SytemMessage` message
This message is used by the server to send out-of-band messages to the client.
It contains text messages such as data usage, warnings about reaching the free
quota, or information about upcoming system upgrades.
## protobuf definition
This section contains the protobuf definition used by the DNA server and clients.
If you're implementing a new SDK for DNA, you can use this as the starting point.
```proto
syntax = "proto3";
package dna.v2.stream;
import "google/protobuf/duration.proto";
service DnaStream {
// Stream data from the server.
rpc StreamData(StreamDataRequest) returns (stream StreamDataResponse);
// Get DNA server status.
rpc Status(StatusRequest) returns (StatusResponse);
}
// A cursor over the stream content.
message Cursor {
// Key used for ordering messages in the stream.
//
// This is usually the block or slot number.
uint64 order_key = 1;
// Key used to discriminate branches in the stream.
//
// This is usually the hash of the block.
bytes unique_key = 2;
}
// Request for the `Status` method.
message StatusRequest {}
// Response for the `Status` method.
message StatusResponse {
// The current head of the chain.
Cursor current_head = 1;
// The last cursor that was ingested by the node.
Cursor last_ingested = 2;
// The finalized block.
Cursor finalized = 3;
// The first block available.
Cursor starting = 4;
}
// Request data to be streamed.
message StreamDataRequest {
// Cursor to start streaming from.
//
// If not specified, starts from the genesis block.
// Use the data's message `end_cursor` field to resume streaming.
optional Cursor starting_cursor = 1;
// Return data with the specified finality.
//
// If not specified, defaults to `DATA_FINALITY_ACCEPTED`.
optional DataFinality finality = 2;
// Filters used to generate data.
repeated bytes filter = 3;
// Heartbeat interval.
//
// Value must be between 10 and 60 seconds.
// If not specified, defaults to 30 seconds.
optional google.protobuf.Duration heartbeat_interval = 4;
}
// Contains a piece of streamed data.
message StreamDataResponse {
oneof message {
Data data = 1;
Invalidate invalidate = 2;
Finalize finalize = 3;
Heartbeat heartbeat = 4;
SystemMessage system_message = 5;
}
}
// Invalidate data after the given cursor.
message Invalidate {
// The cursor of the new chain's head.
//
// All data after this cursor should be considered invalid.
Cursor cursor = 1;
// List of blocks that were removed from the chain.
repeated Cursor removed = 2;
}
// Move the finalized block forward.
message Finalize {
// The cursor of the new finalized block.
//
// All data before this cursor cannot be invalidated.
Cursor cursor = 1;
}
// A single block of data.
//
// If the request specified multiple filters, the `data` field will contain the
// data for each filter in the same order as the filters were specified in the
// request.
// If no data is available for a filter, the corresponding data field will be
// empty.
message Data {
// Cursor that generated this block of data.
optional Cursor cursor = 1;
// Block cursor. Use this cursor to resume the stream.
Cursor end_cursor = 2;
// The finality status of the block.
DataFinality finality = 3;
// The block data.
//
// This message contains chain-specific data serialized using protobuf.
repeated bytes data = 4;
}
// Sent to clients to check if stream is still connected.
message Heartbeat {}
// Message from the server to the client.
message SystemMessage {
oneof output {
// Output to stdout.
string stdout = 1;
// Output to stderr.
string stderr = 2;
}
}
// Data finality.
enum DataFinality {
DATA_FINALITY_UNKNOWN = 0;
// Data was received, but is not part of the canonical chain yet.
DATA_FINALITY_PENDING = 1;
// Data is now part of the canonical chain, but could still be invalidated.
DATA_FINALITY_ACCEPTED = 2;
// Data is finalized and cannot be invalidated.
DATA_FINALITY_FINALIZED = 3;
}
```
---
title: Adding a new chain
description: "Learn how to bring DNA to your chain, giving developers access to the best indexing platform on the market."
diataxis: how-to
updatedAt: 2024-09-22
---
# Adding a new chain
This page explains how to add support for a new chain to the DNA protocol. It's recommended that you're familiar with the high-level [DNA architecture](/docs/v2/dna/architecture) and the [DNA streaming protocol](/docs/v2/dna/protocol) before reading this page.
## Overview
Adding a new chain is relatively straightforward. Most of the code you need to write is describing the type of data stored on your chain.
The guide is split in the following sections:
- **gRPC Protocol**: describes how to augment the gRPC protocol with filters and data types specific to the new chain.
- **Storage**: describes how data is stored on disk and S3.
- **Data filtering**: describes how to filter data based on the client's request.
## gRPC Protocol
The first step is to define the root `Filter` and `Block` protobuf messages.
There are a few hard requirements on the messages:
- The `header` field of the block must have tag `1`.
- All other fields can have any tag.
- Add one message type for each chain's resource (transactions, receipts, logs, etc.).
- Each resource must have a `filter_ids` field with tag `1`.
- Add a `Filter.id: uint32` property. Indexers use this to know which filters matched a specific piece of data and is used to populate the `filter_ids` field.
The following items are optional:
- Add an option to the `Filter` to request all block headers. Users use this to debug their indexer.
- Think how users are going to use the data. For example, developers often access the transaction's hash of a log, for this reason we include the transaction hash in the `Log` message.
- Avoid excessive nesting of messages.
## Storage
The goal of the ingestion service is to fetch data from the chain (using the chain's RPC protocol), preprocess and index it, and then store it into the object storage.
DNA stores block data as pre-serialized protobuf messages. This is done to send data to clients by
copying bytes directly, without expensive serialization and deserialization.
Since DNA doesn't know about the chain, it needs a way to filter data without scanning the entire block.
This is done with _indices_.
The chain-specific ingestion service is responsible for creating these indices. The next section goes into
detail how indices work, the important part is that:
- Indices are grouped by the type of data they index (for example transactions, logs, and traces).
- For each type of data, there can be multiple indices.
- Indices point to one or more pre-serialized protobuf messages.
## Data filtering
As mentioned in the previous section, the DNA server uses indices to lookup data without scanning the entire block.
This is done by compiling the protobuf filter sent by the client into a special representation.
This `Filter` specifies:
- What resource to filter (for example transactions, logs, and traces).
- The list of conditions to match.
A _condition_ is a tuple with the filter id and the lookup key.