Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions docs/v2/advanced/custom_network.mdx
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
---
title: Custom networks
sidebar:
order: 5
---

Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / preprod / preview).
Connecting to a network other than mainnet, testnet, preprod, or preview? The `[chain]` block
gives Oura the parameters it needs to make sense of that network.

## Context

Expand All @@ -14,7 +17,7 @@ Since `mainnet`, `testnet`, `preprod` and `preview` are well-known, heavily used

By adding a `[chain]` section in the daemon configuration file, users can provide the information required by Oura to connect to a custom network.

The `[chain]` section has the following propoerties:
The `[chain]` section has the following properties:

| Name | DataType | Description |
| :------------------- | :------- | :------------------------------------------------------------ |
Expand All @@ -35,9 +38,11 @@ The `[chain]` section has the following propoerties:

### Chain information for mainnet

This example configuration shows the values for mainnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like.
This example shows the values for mainnet. Since these are hardcoded in Oura's release you'd
never enter them by hand, but they're a good illustration of what a `[chain]` section looks
like.

```toml
```toml title="daemon.toml"
[chain]
type = "custom"
magic = 764824073
Expand Down
23 changes: 15 additions & 8 deletions docs/v2/advanced/finalize_options.mdx
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
---
title: Finalize Options
sidebar:
order: 2
---

Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can instruct Oura to stop syncing once it reaches a specific block hash, slot, or block count.
*Finalizing* means telling Oura to **stop and exit gracefully** once it reaches a given point —
handy when you only want to sync a specific section of the chain.

There is no top-level `[finalize]` block in v2. Finalization is configured through the [Work Stats filter](/oura/v2/filters/work_stats), which counts blocks as they pass through and stops the pipeline once a configured condition is met.
:::note
There's no top-level `[finalize]` block in v2. Finalization is configured through the
[Work Stats filter](/oura/v2/filters/work_stats), which counts blocks as they pass and stops
the pipeline once a condition is met.
:::

## Configuration

Add a `WorkStats` filter to the `[[filters]]` chain in your `daemon.toml` and set any of the stopping conditions:
Add a `WorkStats` filter to your `[[filters]]` chain and set any of the stopping conditions:

```toml
```toml title="daemon.toml"
[[filters]]
type = "WorkStats"
until_hash = <BlockHash>
Expand All @@ -19,16 +26,16 @@ max_block_quantity = <BlockCount>
```

- `until_hash` (optional): stop once the block with this hash has been processed.
- `max_block_slot` (optional): stop once a block at or beyond this slot number has been processed.
- `max_block_slot` (optional): stop once a block at or beyond this slot has been processed.
- `max_block_quantity` (optional): stop after this many blocks have been processed.

The pipeline stops as soon as any configured condition is met.
The pipeline stops as soon as **any** configured condition is met.

## Examples

The following example shows how to configure Oura to stop syncing on the Byron era
Stop at the end of the Byron era:

```toml
```toml title="daemon.toml"
[[filters]]
type = "WorkStats"
until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
Expand Down
19 changes: 19 additions & 0 deletions docs/v2/advanced/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
title: Advanced Features
sidebar:
label: Overview
order: 0
---

Beyond the source, filters, and sink, a `daemon.toml` can carry a handful of **optional
configuration blocks** that tune how a pipeline behaves. Add only the ones you need — every
one of them has a sensible default.

| Block / feature | Configures | Page |
| :-------------- | :--------- | :--- |
| `[intersect]` | where on the chain a pipeline starts reading | [Intersect options](/oura/v2/advanced/intersect_options) |
| `WorkStats` filter | where a pipeline stops (finalization) | [Finalize options](/oura/v2/advanced/finalize_options) |
| `[cursor]` | persisting progress so restarts resume where they left off | [Stateful cursor](/oura/v2/advanced/stateful_cursor) |
| `[retries]` | how failures are retried and backed off | [Retry policy](/oura/v2/advanced/retry_policy) |
| `[chain]` | connecting to a non-standard network | [Custom networks](/oura/v2/advanced/custom_network) |
| `[metrics]` | exposing a Prometheus metrics endpoint | [Pipeline metrics](/oura/v2/advanced/pipeline_metrics) |
39 changes: 23 additions & 16 deletions docs/v2/advanced/intersect_options.mdx
Original file line number Diff line number Diff line change
@@ -1,40 +1,47 @@
---
title: Intersect Options
sidebar:
order: 1
---

Advanced options for instructing Oura from which point in the chain to start reading from.
The `[intersect]` block tells Oura **where on the chain to start reading**. By default it picks
up from the tip; change it when you need to (re)process from a specific point or from the
chain's origin.

## Feature
## Strategies

When running in daemon mode, Oura provides 4 different strategies for finding the intersection point within the chain sync process.
In daemon mode, Oura supports four ways to find its starting point:

- `Tip`: Oura will start reading from the current tip of the chain.
- `Origin`: Oura will start reading from the beginning of the chain.
- `Point`: Oura will start reading from a particular point [slot, hash] in the chain.
- `Breadcrumbs`: Oura will start reading from a some points [[slot, hash],[slot, hash]] in the chain.
- `Tip` _(default)_: start from the current tip of the chain.
- `Origin`: start from the very beginning of the chain.
- `Point`: start from a specific `[slot, hash]`.
- `Breadcrumbs`: start from a set of points `[[slot, hash], [slot, hash], …]` (useful for
recovering across possible rollbacks).

The default strategy use by Oura is `Tip`, unless an alternative option is specified via configuration.

You can also have Oura stop reading from the chain and exit gracefully once it reaches a given block hash, slot, or block count. This is configured through the [Work Stats filter](/oura/v2/filters/work_stats); see [finalize options](/oura/v2/advanced/finalize_options) for details.
:::note
To stop reading at a given point — rather than start — use the
[Work Stats filter](/oura/v2/filters/work_stats); see [finalize options](/oura/v2/advanced/finalize_options).
:::

## Configuration

To modify the default behaviour used by the daemon mode, a section named `[intersect]` needs to be added in the `daemon.toml` file.
Add an `[intersect]` section to your `daemon.toml`:

```toml
```toml title="daemon.toml"
[intersect]
type = <Type>
value = <Value>
```

- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Breadcrumbs`. Default value is `Tip`.
- `value`: Either a point or an array of points to be used as argument for the selected strategy.
- `type` (optional, default = `Tip`): the strategy — `Origin`, `Tip`, `Point`, or
`Breadcrumbs`.
- `value`: a point, or array of points, used as the argument for the chosen strategy.

## Examples

The following example show how to configure Oura to use a intersection point.
Starting from a specific point:

```toml
```toml title="daemon.toml"
[intersect]
type = "Point"
value = [
Expand Down
24 changes: 16 additions & 8 deletions docs/v2/advanced/pipeline_metrics.mdx
Original file line number Diff line number Diff line change
@@ -1,18 +1,26 @@
---
title: Pipeline Metrics
sidebar:
order: 6
---

The _metrics_ features allows operators to track the progress and performance of long-running Oura sessions.
The `[metrics]` feature lets operators track the progress and performance of long-running Oura
sessions over a Prometheus endpoint.

## Context

Some use-cases require Oura to be running either continuosuly or for prolonged periods of time. Keeping a sink updated in real-time requires 24/7 operation. Dumping historical chain-data from origin might take several hours.
Some use-cases keep Oura running continuously or for long stretches: keeping a sink updated in
real time means 24/7 operation, and dumping historical chain data from origin can take hours.

These scenarios require an understanding of the internal state of the pipeline to facilitate monitoring and troubleshooting of the system. In a data-processing pipeline such as this, two important aspects need to observable: progress and performance.
These scenarios call for visibility into the pipeline's internal state to make monitoring and
troubleshooting possible. Two aspects matter most: **progress** and **performance**.

## Feature

Oura provides an optional `/metrics` HTTP endpoint that uses Prometheus format to expose real-time opertional metrics of the pipeline. Each stage (source / sink) is responsible for notifying their progress as they process each event. This notifications are then aggregated via counters & gauges and exposed via HTTP using the well-known Prometheus encoding.
Oura provides an optional `/metrics` HTTP endpoint that exposes real-time operational metrics
in Prometheus format. Each stage (source / sink) reports its progress as it processes events;
these notifications are aggregated into counters and gauges and served over HTTP using the
well-known Prometheus encoding.

The following metrics are available:

Expand All @@ -28,9 +36,7 @@ The following metrics are available:

The _metrics_ feature is a configurable setting available when running in daemon mode. A top level `[metrics]` section of the daemon toml file controls the feature:

```toml
# daemon.toml file

```toml title="daemon.toml"
[metrics]
address = "0.0.0.0:9186"
```
Expand Down Expand Up @@ -68,7 +74,9 @@ source_current_slot 2839340
source_event_count 2277715
```

Regardless of the above mechanism, the inteded approach for tracking Oura's metrics is to use a monitoring infrastructure compatible with Prometheus format. Setting up and managing the monitoring stack is outside the scope of Oura. If you don't have any infrastructure in place, we recommend checking out some of the more commons stacks:
The browser check above is handy, but the intended approach is to point a Prometheus-compatible
monitoring stack at the endpoint. Setting up and running that stack is outside Oura's scope; if
you don't already have one, these are common choices:

- Prometheus Server + Grafana
- Metricbeat + Elasticsearch + Kibana
Expand Down
37 changes: 24 additions & 13 deletions docs/v2/advanced/retry_policy.mdx
Original file line number Diff line number Diff line change
@@ -1,23 +1,34 @@
---
title: Retry Policy
sidebar:
order: 4
---

Advanced options for instructing Oura how to deal with failed attempts.
The `[retries]` block controls how Oura reacts when a stage fails — how many times it retries,
how long it waits between attempts, and whether a persistent failure should bring the pipeline
down or be skipped.

## Configuration

To modify the default behavior, a section named `[retries]` needs to be added to the `daemon.toml` file.

```toml
```toml title="daemon.toml"
[retries]
max_retries = 3
backoff_unit_sec = 10
backoff_factor = 3
max_backoff_sec = 10
dismissible = true
max_retries = 20
backoff_unit_sec = 1
backoff_factor = 2
max_backoff_sec = 60
dismissible = false
```

- `max_retries`: the max number of retries before failing the whole pipeline.
- `backoff_unit_sec`: the delay expressed in seconds between each retry.
- `backoff_factor`: the amount to increase the backoff delay after each attempt.
- `max_backoff_sec`: the longest possible delay in seconds.
- `max_retries` (optional, default = `20`): how many times to retry before giving up.
- `backoff_unit_sec` (optional, default = `1`): the base delay, in seconds, between retries.
- `backoff_factor` (optional, default = `2`): the multiplier applied to the delay after each
attempt (exponential backoff).
- `max_backoff_sec` (optional, default = `60`): the longest the delay is ever allowed to grow.
- `dismissible` (optional, default = `false`): when `false`, exhausting the retries tears the
pipeline down. When `true`, the failing unit of work is dismissed and the pipeline continues.

:::note[How the backoff grows]
Each retry waits `backoff_unit_sec × backoff_factor^attempt`, capped at `max_backoff_sec`. With
the values above, the delays are **1s, 2s, 4s, 8s, 16s, 32s**, then **60s** for every attempt
after that.
:::
89 changes: 46 additions & 43 deletions docs/v2/advanced/stateful_cursor.mdx
Original file line number Diff line number Diff line change
@@ -1,62 +1,62 @@
---
title: Stateful Cursor
sidebar:
order: 3
---

The _cursor_ feature provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts.
import { Tabs, TabItem } from '@astrojs/starlight/components';

## Context
The **cursor** persists the pipeline's position so it survives restarts. Without it, a restart
makes Oura start over from its configured intersect point.

When running in daemon mode, a restart of the process will make Oura start crawling from the initially configured point. This default behavior can be problematic depending on the use-case of the operator.
## Why it matters

An scenario where continuous "coverage" of all the processed blocks is important (eg: building a stateful view of the chain), it's prohibitive to have "gaps" while procesing. If Oura is configure to start from the "tip" of the chain, a restart of the process might miss some blocks during the bootstrap procedure.
Some use-cases — like building a stateful view of the chain — can't tolerate gaps. If Oura
starts from the chain `Tip`, a restart could miss the blocks produced during bootstrap. You
*could* work around this by always starting from a fixed point and re-processing (safe if your
sink is idempotent), but if that point is far back, catching up can take hours.

A valid workaround to the above problem is to configure Oura to start from a fixed point in the chain. If a restart occurs, the pipeline will re-process the blocks, ensuring that each block was processed at least once. When working with sinks that implement idempotency when processing an event, receiving data from the same block multiple times should not impose a problem.
The cursor solves this directly: the sink stage reports its position as it works, and at a
regular interval that position is saved to a backend. On restart, Oura loads the saved position
and resumes the source from there.

Although valid, the workaround described above is very inefficient. If the fixed point at which the pipeline starts is too far behind, catching up could take several hours, wasting time and resource.

## Feature

Oura implements an optional stateful cursor that receives notifications from the sink stage of the pipeline to continuously track the current position of the chain. At certain checkpoints (every 10 secs by default), the position is persisted onto the configured backend.
## Configuration

Assuming that a restart occurs and the cursor feature is enabled, the process will attempt to locate and load the persisted value and instruct the source stage to begin reading chain data from the last known position.
Add a top-level `[cursor]` section to enable the feature, and pick a backend with `type`:

## Configuration
<Tabs syncKey="cursor-backend">
<TabItem label="Memory">
Keeps the cursor in memory only — the position is **not** persisted across restarts. This is
the default when no `[cursor]` section is present, and takes no extra fields.

The _cursor_ feature is a configurable setting available when running in daemon mode. A top level `[cursor]` section of the daemon toml file controls the feature:
```toml title="daemon.toml"
[cursor]
type = "Memory"
```
</TabItem>
<TabItem label="File">
Stores the cursor in a JSON file on the local file system.

```toml
```toml title="daemon.toml"
[cursor]
type = "File"
path = "./cursor.json"
max_breadcrumbs = 10
flush_interval = 10
```

- `[cursor]`: The presence of this section in the toml file indicates Oura to enable the _cursor_ feature.
- `type`: The type of persistence backend to use for storing the state of the cursor. Available options are `Memory` (the default if no `[cursor]` section is present), `File`, and `Redis`.

### Backend: `Memory`

Keeps the cursor breadcrumbs in memory only; the position is **not** persisted across restarts. This is the default behavior when no `[cursor]` section is configured. It takes no extra fields:

```toml
[cursor]
type = "Memory"
```

### Backend: `File`

Stores the cursor in a JSON file on the local file system.

- `path` (optional): The location of the cursor file within the file system. Default value is `./cursor.json` (relative to the working directory).
- `max_breadcrumbs` (optional): The number of recent chain positions to retain (used to recover across rollbacks). Default value is `10`.
- `flush_interval` (optional): How often, in seconds, the position is flushed to disk. Default value is `10`.

### Backend: `Redis`

Stores the cursor in a Redis instance (requires the `redis` feature).

```toml
- `path` (optional, default = `./cursor.json`): where the cursor file lives, relative to the
working directory.
- `max_breadcrumbs` (optional, default = `10`): how many recent positions to retain, used to
recover across rollbacks.
- `flush_interval` (optional, default = `10`): how often, in seconds, the position is written
to disk.
</TabItem>
<TabItem label="Redis">
Stores the cursor in a Redis instance — handy when running multiple environments or ephemeral
containers.

```toml title="daemon.toml"
[cursor]
type = "Redis"
url = "redis://localhost:6379"
Expand All @@ -65,7 +65,10 @@ max_breadcrumbs = 10
flush_interval = 10
```

- `url`: The Redis connection URL.
- `key`: The Redis key under which the cursor state is stored.
- `max_breadcrumbs` (optional): The number of recent chain positions to retain. Default value is `10`.
- `flush_interval` (optional): How often, in seconds, the position is flushed to Redis. Default value is `10`.
- `url` (required): the Redis connection URL.
- `key` (required): the Redis key the cursor state is stored under.
- `max_breadcrumbs` (optional, default = `10`): how many recent positions to retain.
- `flush_interval` (optional, default = `10`): how often, in seconds, the position is flushed
to Redis.
</TabItem>
</Tabs>
Loading
Loading