diff --git a/docs/v2/advanced/custom_network.mdx b/docs/v2/advanced/custom_network.mdx index 478e5a14..55675b18 100644 --- a/docs/v2/advanced/custom_network.mdx +++ b/docs/v2/advanced/custom_network.mdx @@ -1,8 +1,11 @@ --- title: Custom networks +sidebar: + order: 5 --- -Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / preprod / preview). +Connecting to a network other than mainnet, testnet, preprod, or preview? The `[chain]` block +gives Oura the parameters it needs to make sense of that network. ## Context @@ -14,7 +17,7 @@ Since `mainnet`, `testnet`, `preprod` and `preview` are well-known, heavily used By adding a `[chain]` section in the daemon configuration file, users can provide the information required by Oura to connect to a custom network. -The `[chain]` section has the following propoerties: +The `[chain]` section has the following properties: | Name | DataType | Description | | :------------------- | :------- | :------------------------------------------------------------ | @@ -35,9 +38,11 @@ The `[chain]` section has the following propoerties: ### Chain information for mainnet -This example configuration shows the values for mainnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like. +This example shows the values for mainnet. Since these are hardcoded in Oura's release you'd +never enter them by hand, but they're a good illustration of what a `[chain]` section looks +like. -```toml +```toml title="daemon.toml" [chain] type = "custom" magic = 764824073 diff --git a/docs/v2/advanced/finalize_options.mdx b/docs/v2/advanced/finalize_options.mdx index 383877b4..3655db32 100644 --- a/docs/v2/advanced/finalize_options.mdx +++ b/docs/v2/advanced/finalize_options.mdx @@ -1,16 +1,23 @@ --- title: Finalize Options +sidebar: + order: 2 --- -Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can instruct Oura to stop syncing once it reaches a specific block hash, slot, or block count. +*Finalizing* means telling Oura to **stop and exit gracefully** once it reaches a given point — +handy when you only want to sync a specific section of the chain. -There is no top-level `[finalize]` block in v2. Finalization is configured through the [Work Stats filter](/oura/v2/filters/work_stats), which counts blocks as they pass through and stops the pipeline once a configured condition is met. +:::note +There's no top-level `[finalize]` block in v2. Finalization is configured through the +[Work Stats filter](/oura/v2/filters/work_stats), which counts blocks as they pass and stops +the pipeline once a condition is met. +::: ## Configuration -Add a `WorkStats` filter to the `[[filters]]` chain in your `daemon.toml` and set any of the stopping conditions: +Add a `WorkStats` filter to your `[[filters]]` chain and set any of the stopping conditions: -```toml +```toml title="daemon.toml" [[filters]] type = "WorkStats" until_hash = @@ -19,16 +26,16 @@ max_block_quantity = ``` - `until_hash` (optional): stop once the block with this hash has been processed. -- `max_block_slot` (optional): stop once a block at or beyond this slot number has been processed. +- `max_block_slot` (optional): stop once a block at or beyond this slot has been processed. - `max_block_quantity` (optional): stop after this many blocks have been processed. -The pipeline stops as soon as any configured condition is met. +The pipeline stops as soon as **any** configured condition is met. ## Examples -The following example shows how to configure Oura to stop syncing on the Byron era +Stop at the end of the Byron era: -```toml +```toml title="daemon.toml" [[filters]] type = "WorkStats" until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de" diff --git a/docs/v2/advanced/index.mdx b/docs/v2/advanced/index.mdx new file mode 100644 index 00000000..f29b0e2d --- /dev/null +++ b/docs/v2/advanced/index.mdx @@ -0,0 +1,19 @@ +--- +title: Advanced Features +sidebar: + label: Overview + order: 0 +--- + +Beyond the source, filters, and sink, a `daemon.toml` can carry a handful of **optional +configuration blocks** that tune how a pipeline behaves. Add only the ones you need — every +one of them has a sensible default. + +| Block / feature | Configures | Page | +| :-------------- | :--------- | :--- | +| `[intersect]` | where on the chain a pipeline starts reading | [Intersect options](/oura/v2/advanced/intersect_options) | +| `WorkStats` filter | where a pipeline stops (finalization) | [Finalize options](/oura/v2/advanced/finalize_options) | +| `[cursor]` | persisting progress so restarts resume where they left off | [Stateful cursor](/oura/v2/advanced/stateful_cursor) | +| `[retries]` | how failures are retried and backed off | [Retry policy](/oura/v2/advanced/retry_policy) | +| `[chain]` | connecting to a non-standard network | [Custom networks](/oura/v2/advanced/custom_network) | +| `[metrics]` | exposing a Prometheus metrics endpoint | [Pipeline metrics](/oura/v2/advanced/pipeline_metrics) | diff --git a/docs/v2/advanced/intersect_options.mdx b/docs/v2/advanced/intersect_options.mdx index b91ca0d8..6f958935 100644 --- a/docs/v2/advanced/intersect_options.mdx +++ b/docs/v2/advanced/intersect_options.mdx @@ -1,40 +1,47 @@ --- title: Intersect Options +sidebar: + order: 1 --- -Advanced options for instructing Oura from which point in the chain to start reading from. +The `[intersect]` block tells Oura **where on the chain to start reading**. By default it picks +up from the tip; change it when you need to (re)process from a specific point or from the +chain's origin. -## Feature +## Strategies -When running in daemon mode, Oura provides 4 different strategies for finding the intersection point within the chain sync process. +In daemon mode, Oura supports four ways to find its starting point: -- `Tip`: Oura will start reading from the current tip of the chain. -- `Origin`: Oura will start reading from the beginning of the chain. -- `Point`: Oura will start reading from a particular point [slot, hash] in the chain. -- `Breadcrumbs`: Oura will start reading from a some points [[slot, hash],[slot, hash]] in the chain. +- `Tip` _(default)_: start from the current tip of the chain. +- `Origin`: start from the very beginning of the chain. +- `Point`: start from a specific `[slot, hash]`. +- `Breadcrumbs`: start from a set of points `[[slot, hash], [slot, hash], …]` (useful for + recovering across possible rollbacks). -The default strategy use by Oura is `Tip`, unless an alternative option is specified via configuration. - -You can also have Oura stop reading from the chain and exit gracefully once it reaches a given block hash, slot, or block count. This is configured through the [Work Stats filter](/oura/v2/filters/work_stats); see [finalize options](/oura/v2/advanced/finalize_options) for details. +:::note +To stop reading at a given point — rather than start — use the +[Work Stats filter](/oura/v2/filters/work_stats); see [finalize options](/oura/v2/advanced/finalize_options). +::: ## Configuration -To modify the default behaviour used by the daemon mode, a section named `[intersect]` needs to be added in the `daemon.toml` file. +Add an `[intersect]` section to your `daemon.toml`: -```toml +```toml title="daemon.toml" [intersect] type = value = ``` -- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Breadcrumbs`. Default value is `Tip`. -- `value`: Either a point or an array of points to be used as argument for the selected strategy. +- `type` (optional, default = `Tip`): the strategy — `Origin`, `Tip`, `Point`, or + `Breadcrumbs`. +- `value`: a point, or array of points, used as the argument for the chosen strategy. ## Examples -The following example show how to configure Oura to use a intersection point. +Starting from a specific point: -```toml +```toml title="daemon.toml" [intersect] type = "Point" value = [ diff --git a/docs/v2/advanced/pipeline_metrics.mdx b/docs/v2/advanced/pipeline_metrics.mdx index 94fdc2b3..8d7bb5f8 100644 --- a/docs/v2/advanced/pipeline_metrics.mdx +++ b/docs/v2/advanced/pipeline_metrics.mdx @@ -1,18 +1,26 @@ --- title: Pipeline Metrics +sidebar: + order: 6 --- -The _metrics_ features allows operators to track the progress and performance of long-running Oura sessions. +The `[metrics]` feature lets operators track the progress and performance of long-running Oura +sessions over a Prometheus endpoint. ## Context -Some use-cases require Oura to be running either continuosuly or for prolonged periods of time. Keeping a sink updated in real-time requires 24/7 operation. Dumping historical chain-data from origin might take several hours. +Some use-cases keep Oura running continuously or for long stretches: keeping a sink updated in +real time means 24/7 operation, and dumping historical chain data from origin can take hours. -These scenarios require an understanding of the internal state of the pipeline to facilitate monitoring and troubleshooting of the system. In a data-processing pipeline such as this, two important aspects need to observable: progress and performance. +These scenarios call for visibility into the pipeline's internal state to make monitoring and +troubleshooting possible. Two aspects matter most: **progress** and **performance**. ## Feature -Oura provides an optional `/metrics` HTTP endpoint that uses Prometheus format to expose real-time opertional metrics of the pipeline. Each stage (source / sink) is responsible for notifying their progress as they process each event. This notifications are then aggregated via counters & gauges and exposed via HTTP using the well-known Prometheus encoding. +Oura provides an optional `/metrics` HTTP endpoint that exposes real-time operational metrics +in Prometheus format. Each stage (source / sink) reports its progress as it processes events; +these notifications are aggregated into counters and gauges and served over HTTP using the +well-known Prometheus encoding. The following metrics are available: @@ -28,9 +36,7 @@ The following metrics are available: The _metrics_ feature is a configurable setting available when running in daemon mode. A top level `[metrics]` section of the daemon toml file controls the feature: -```toml -# daemon.toml file - +```toml title="daemon.toml" [metrics] address = "0.0.0.0:9186" ``` @@ -68,7 +74,9 @@ source_current_slot 2839340 source_event_count 2277715 ``` -Regardless of the above mechanism, the inteded approach for tracking Oura's metrics is to use a monitoring infrastructure compatible with Prometheus format. Setting up and managing the monitoring stack is outside the scope of Oura. If you don't have any infrastructure in place, we recommend checking out some of the more commons stacks: +The browser check above is handy, but the intended approach is to point a Prometheus-compatible +monitoring stack at the endpoint. Setting up and running that stack is outside Oura's scope; if +you don't already have one, these are common choices: - Prometheus Server + Grafana - Metricbeat + Elasticsearch + Kibana diff --git a/docs/v2/advanced/retry_policy.mdx b/docs/v2/advanced/retry_policy.mdx index 89a8781e..62f94950 100644 --- a/docs/v2/advanced/retry_policy.mdx +++ b/docs/v2/advanced/retry_policy.mdx @@ -1,23 +1,34 @@ --- title: Retry Policy +sidebar: + order: 4 --- -Advanced options for instructing Oura how to deal with failed attempts. +The `[retries]` block controls how Oura reacts when a stage fails — how many times it retries, +how long it waits between attempts, and whether a persistent failure should bring the pipeline +down or be skipped. ## Configuration -To modify the default behavior, a section named `[retries]` needs to be added to the `daemon.toml` file. - -```toml +```toml title="daemon.toml" [retries] -max_retries = 3 -backoff_unit_sec = 10 -backoff_factor = 3 -max_backoff_sec = 10 -dismissible = true +max_retries = 20 +backoff_unit_sec = 1 +backoff_factor = 2 +max_backoff_sec = 60 +dismissible = false ``` -- `max_retries`: the max number of retries before failing the whole pipeline. -- `backoff_unit_sec`: the delay expressed in seconds between each retry. -- `backoff_factor`: the amount to increase the backoff delay after each attempt. -- `max_backoff_sec`: the longest possible delay in seconds. +- `max_retries` (optional, default = `20`): how many times to retry before giving up. +- `backoff_unit_sec` (optional, default = `1`): the base delay, in seconds, between retries. +- `backoff_factor` (optional, default = `2`): the multiplier applied to the delay after each + attempt (exponential backoff). +- `max_backoff_sec` (optional, default = `60`): the longest the delay is ever allowed to grow. +- `dismissible` (optional, default = `false`): when `false`, exhausting the retries tears the + pipeline down. When `true`, the failing unit of work is dismissed and the pipeline continues. + +:::note[How the backoff grows] +Each retry waits `backoff_unit_sec × backoff_factor^attempt`, capped at `max_backoff_sec`. With +the values above, the delays are **1s, 2s, 4s, 8s, 16s, 32s**, then **60s** for every attempt +after that. +::: diff --git a/docs/v2/advanced/stateful_cursor.mdx b/docs/v2/advanced/stateful_cursor.mdx index a5baeaad..fe401d41 100644 --- a/docs/v2/advanced/stateful_cursor.mdx +++ b/docs/v2/advanced/stateful_cursor.mdx @@ -1,30 +1,43 @@ --- title: Stateful Cursor +sidebar: + order: 3 --- -The _cursor_ feature provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts. +import { Tabs, TabItem } from '@astrojs/starlight/components'; -## Context +The **cursor** persists the pipeline's position so it survives restarts. Without it, a restart +makes Oura start over from its configured intersect point. -When running in daemon mode, a restart of the process will make Oura start crawling from the initially configured point. This default behavior can be problematic depending on the use-case of the operator. +## Why it matters -An scenario where continuous "coverage" of all the processed blocks is important (eg: building a stateful view of the chain), it's prohibitive to have "gaps" while procesing. If Oura is configure to start from the "tip" of the chain, a restart of the process might miss some blocks during the bootstrap procedure. +Some use-cases — like building a stateful view of the chain — can't tolerate gaps. If Oura +starts from the chain `Tip`, a restart could miss the blocks produced during bootstrap. You +*could* work around this by always starting from a fixed point and re-processing (safe if your +sink is idempotent), but if that point is far back, catching up can take hours. -A valid workaround to the above problem is to configure Oura to start from a fixed point in the chain. If a restart occurs, the pipeline will re-process the blocks, ensuring that each block was processed at least once. When working with sinks that implement idempotency when processing an event, receiving data from the same block multiple times should not impose a problem. +The cursor solves this directly: the sink stage reports its position as it works, and at a +regular interval that position is saved to a backend. On restart, Oura loads the saved position +and resumes the source from there. -Although valid, the workaround described above is very inefficient. If the fixed point at which the pipeline starts is too far behind, catching up could take several hours, wasting time and resource. - -## Feature - -Oura implements an optional stateful cursor that receives notifications from the sink stage of the pipeline to continuously track the current position of the chain. At certain checkpoints (every 10 secs by default), the position is persisted onto the configured backend. +## Configuration -Assuming that a restart occurs and the cursor feature is enabled, the process will attempt to locate and load the persisted value and instruct the source stage to begin reading chain data from the last known position. +Add a top-level `[cursor]` section to enable the feature, and pick a backend with `type`: -## Configuration + + +Keeps the cursor in memory only — the position is **not** persisted across restarts. This is +the default when no `[cursor]` section is present, and takes no extra fields. -The _cursor_ feature is a configurable setting available when running in daemon mode. A top level `[cursor]` section of the daemon toml file controls the feature: +```toml title="daemon.toml" +[cursor] +type = "Memory" +``` + + +Stores the cursor in a JSON file on the local file system. -```toml +```toml title="daemon.toml" [cursor] type = "File" path = "./cursor.json" @@ -32,31 +45,18 @@ max_breadcrumbs = 10 flush_interval = 10 ``` -- `[cursor]`: The presence of this section in the toml file indicates Oura to enable the _cursor_ feature. -- `type`: The type of persistence backend to use for storing the state of the cursor. Available options are `Memory` (the default if no `[cursor]` section is present), `File`, and `Redis`. - -### Backend: `Memory` - -Keeps the cursor breadcrumbs in memory only; the position is **not** persisted across restarts. This is the default behavior when no `[cursor]` section is configured. It takes no extra fields: - -```toml -[cursor] -type = "Memory" -``` - -### Backend: `File` - -Stores the cursor in a JSON file on the local file system. - -- `path` (optional): The location of the cursor file within the file system. Default value is `./cursor.json` (relative to the working directory). -- `max_breadcrumbs` (optional): The number of recent chain positions to retain (used to recover across rollbacks). Default value is `10`. -- `flush_interval` (optional): How often, in seconds, the position is flushed to disk. Default value is `10`. - -### Backend: `Redis` - -Stores the cursor in a Redis instance (requires the `redis` feature). - -```toml +- `path` (optional, default = `./cursor.json`): where the cursor file lives, relative to the + working directory. +- `max_breadcrumbs` (optional, default = `10`): how many recent positions to retain, used to + recover across rollbacks. +- `flush_interval` (optional, default = `10`): how often, in seconds, the position is written + to disk. + + +Stores the cursor in a Redis instance — handy when running multiple environments or ephemeral +containers. + +```toml title="daemon.toml" [cursor] type = "Redis" url = "redis://localhost:6379" @@ -65,7 +65,10 @@ max_breadcrumbs = 10 flush_interval = 10 ``` -- `url`: The Redis connection URL. -- `key`: The Redis key under which the cursor state is stored. -- `max_breadcrumbs` (optional): The number of recent chain positions to retain. Default value is `10`. -- `flush_interval` (optional): How often, in seconds, the position is flushed to Redis. Default value is `10`. +- `url` (required): the Redis connection URL. +- `key` (required): the Redis key the cursor state is stored under. +- `max_breadcrumbs` (optional, default = `10`): how many recent positions to retain. +- `flush_interval` (optional, default = `10`): how often, in seconds, the position is flushed + to Redis. + + diff --git a/docs/v2/examples/index.mdx b/docs/v2/examples/index.mdx index e0b6bdf0..a9e3a616 100644 --- a/docs/v2/examples/index.mdx +++ b/docs/v2/examples/index.mdx @@ -21,10 +21,11 @@ Running from the repository root would scatter relative output and state (`./out `./snapshot`, cursor and database files) in the wrong place and break the path-based examples. -Examples that target an optional sink or filter (`kafka`, `redis`, `rabbitmq`, `zeromq`, -`elasticsearch`, the AWS and GCP sinks, `SqlDb`, `WasmPlugin`, …) require _Oura_ to be built -with the matching cargo feature. See [Install from source](/oura/v2/installation/from_source) -for the feature flags, or use a Docker image that bundles them. +Most examples run on the default binary as-is. Only a few target an integration that isn't +bundled — the `kafka` and `zeromq` sinks, the `mithril` source, and the `WasmPlugin` filter — +and those need a build with the matching cargo feature. See +[Install from source](/oura/v2/installation/from_source) for the feature flags, or use a Docker +image that bundles them. ## Companion files & setup diff --git a/docs/v2/filters/index.mdx b/docs/v2/filters/index.mdx new file mode 100644 index 00000000..a772a44b --- /dev/null +++ b/docs/v2/filters/index.mdx @@ -0,0 +1,42 @@ +--- +title: Filters +sidebar: + label: Overview + order: 0 +--- + +**Filters** sit between the source and the sink. Each one receives the events produced by the +previous stage, then transforms, decodes, or drops them before passing the result along. You +can chain as many as you like — they run top to bottom, in the order they appear in your +config. (For where filters fit in the bigger picture, see [How it works](/oura/v2/how_it_works).) + +## The typical pipeline shape + +A source emits raw CBOR blocks. From there, most pipelines compose a few filters to get the +data into a useful shape. A very common chain is: + +``` +SplitBlock → ParseCbor → Select → sink +``` + +That is: break each block into its transactions, decode them into structured records, then +keep only the ones you care about. + +## Which filter does what? + +| Filter | What it does | Notes | +| :----- | :----------- | :---- | +| [Split Block](/oura/v2/filters/split_block) | breaks a block into one event per transaction | run it first when you want tx-level processing | +| [Parse CBOR](/oura/v2/filters/parse_cbor) | decodes raw CBOR into structured blocks/txs ([UTxORPC](https://utxorpc.org/cardano)) | required before `Select` and most downstream logic | +| [Select](/oura/v2/filters/select) | keeps only events matching a predicate (address, asset, datum, metadata…) | needs `ParseCbor` earlier in the chain | +| [Into JSON](/oura/v2/filters/into_json) | converts any record into generic JSON | handy for sinks that just want JSON | +| [Legacy V1](/oura/v2/filters/legacy_v1) | reshapes records into the Oura v1 event schema | for compatibility with v1 consumers | +| [Rollback Buffer](/oura/v2/filters/rollback_buffer) | holds blocks until they're _N_ deep, absorbing shallow rollbacks | trades a little latency for fewer rollback events | +| [Work Stats](/oura/v2/filters/work_stats) | tracks progress and can stop the pipeline at a target | this is how [finalization](/oura/v2/advanced/finalize_options) works in v2 | +| [Wasm](/oura/v2/filters/wasm) | runs your own plugin compiled to WebAssembly | for custom logic in any WASM language | + +:::tip +The shape of the `record` each filter produces is documented in the +[Data Dictionary](/oura/v2/reference/data_dictionary) — a quick way to know what a sink will +actually receive given your filter chain. +::: diff --git a/docs/v2/filters/into_json.mdx b/docs/v2/filters/into_json.mdx index e751b0bd..6482c6e8 100644 --- a/docs/v2/filters/into_json.mdx +++ b/docs/v2/filters/into_json.mdx @@ -2,17 +2,20 @@ title: Into JSON filter sidebar: label: Into JSON + order: 4 --- -The `into_json` filter converts each record flowing through the pipeline into a generic JSON record. It is useful when a downstream sink expects JSON regardless of the original record type (raw CBOR, parsed transaction/block, or a legacy v1 event). +The `IntoJson` filter converts whatever record is flowing through the pipeline into a generic +JSON record. Use it when a downstream sink just wants JSON, regardless of the original record +type (raw CBOR, a parsed block/tx, or a legacy v1 event). -The JSON produced mirrors the serde representation of the incoming record, so the exact shape depends on the filters placed before it (for example, running `ParseCbor` first yields the UTxO RPC parsed representation, while `LegacyV1` yields the v1 event schema). +The JSON mirrors the serde representation of the incoming record, so its exact shape depends on +the filters before it — run [ParseCbor](/oura/v2/filters/parse_cbor) first for the UTxORPC +parsed structure, or [LegacyV1](/oura/v2/filters/legacy_v1) for the v1 event schema. ## Configuration -Adding the following section to the daemon config file will enable the filter as part of the pipeline: - -```toml +```toml title="daemon.toml" [[filters]] type = "IntoJson" ``` diff --git a/docs/v2/filters/legacy_v1.mdx b/docs/v2/filters/legacy_v1.mdx index 85ffa1b6..3a1453e7 100644 --- a/docs/v2/filters/legacy_v1.mdx +++ b/docs/v2/filters/legacy_v1.mdx @@ -1,16 +1,18 @@ --- title: Legacy V1 filter -sidebar: +sidebar: label: Legacy V1 + order: 5 --- -Transforms the block data to the Oura V1 data structure. This filter is ideal for compatibility with systems that are using Oura V1 and want to migrate without much work. +The `LegacyV1` filter reshapes records into the Oura **v1** event schema. It's the easy path +for systems already built against Oura v1: point them at v2 and add this filter, and they keep +receiving the events they expect. For the full v1 event catalog, see +[Legacy V1 events](/oura/v2/reference/legacy_v1_events). ## Configuration -Adding the following section to the daemon config file will enable the filter as part of the pipeline: - -```toml +```toml title="daemon.toml" [[filters]] type = "LegacyV1" include_block_end_events = false @@ -20,18 +22,22 @@ include_block_details = false include_block_cbor = false ``` -### Section `LegacyV1` - -- `type`: the literal value `LegacyV1`. -- `include_block_end_events`: if enabled, the source will output an event signaling the end of a block, duplicating all of the data already sent in the corresponding block start event. Default value is `false`. -- `include_transaction_details`: if enabled, each transaction event payload will contain an nested version of all of the details of the transaction (inputs, outputs, mint, assets, metadata, etc). Useful when the pipeline needs to process the tx as a unit, instead of handling each sub-object as an independent event. Default value is `false`. -- `include_transaction_end_events`: if enabled, the source will output an event signaling the end of a transaction, duplicating all of the data already sent in the corresponding transaction start event. Defaul value is `false`. -- `include_block_cbor`: if enabled, the block event will include the raw, unaltered cbor content received from the node, formatted as an hex string. Useful when some custom cbor decoding is required. Default value is `false`. -- `include_block_details`: If enabled, will be added the basic details of each transaction. Default value is `false`. +- `type` (required): the literal value `LegacyV1`. +- `include_block_end_events` (optional, default = `false`): emit an event marking the end of a + block, duplicating the data already sent in the block-start event. +- `include_transaction_details` (optional, default = `false`): nest the full transaction + details (inputs, outputs, mint, assets, metadata, …) inside each transaction event. Useful + when you want to process a tx as a single unit rather than as separate sub-events. +- `include_transaction_end_events` (optional, default = `false`): emit an event marking the end + of a transaction, duplicating the data already sent in the transaction-start event. +- `include_block_details` (optional, default = `false`): include the basic details of each + transaction in the block event. +- `include_block_cbor` (optional, default = `false`): include the raw, unaltered block CBOR as + a hex string. Useful when you need to do custom CBOR decoding. ## Examples -Below is an example of the data that will be sent to the sink with all settings disabled. +With every option disabled, the next stage receives events like this: ```json { diff --git a/docs/v2/filters/parse_cbor.mdx b/docs/v2/filters/parse_cbor.mdx index 2863f073..a4aefee9 100644 --- a/docs/v2/filters/parse_cbor.mdx +++ b/docs/v2/filters/parse_cbor.mdx @@ -1,27 +1,35 @@ --- title: Parse CBOR filter -sidebar: +sidebar: label: Parse CBOR + order: 2 --- -The `parse_cbor` filter aims to map cbor blocks to structured blocks and cbor transactions to structured transactions. +The `ParseCbor` filter decodes raw CBOR into structured records, so downstream stages get +something far easier to work with than a hex blob. It's a prerequisite for most real +processing — notably the [Select](/oura/v2/filters/select) filter. The records follow the +[UTxORPC](https://utxorpc.org/cardano) schema. -- When the record received in the stage is CborBlock, the filter will decode and map it to ParsedBlock, which is passed to the next stage. -- When the record received in the stage is CborTx, the filter will decode and map it to ParsedTx, which is passed to the next stage. This will only happen when the record received in the stage is CborTx and therefore requires to enable the [split_cbor](/oura/v2/filters/split_block) filter before for the stage to receive the CborTx format. -- Else, parse_cbor will ignore and pass the record to the next stage. +It adapts to whatever it receives: -## Configuration +- a **CborBlock** is decoded into a structured `Block`. +- a **CborTx** is decoded into a structured `Tx`. This requires the + [SplitBlock](/oura/v2/filters/split_block) filter earlier in the chain to produce the + `CborTx` records. +- anything else is passed through untouched. -Adding the following section to the daemon config file will enable the filter as part of the pipeline: +## Configuration -```toml +```toml title="daemon.toml" [[filters]] type = "ParseCbor" ``` +This filter takes no additional options. + ## Examples -Below is an example of the data that will be sent to the sink when the filter received a CborBlock record. +From a `CborBlock`, the next stage receives a structured block: ```json { @@ -37,7 +45,8 @@ Below is an example of the data that will be sent to the sink when the filter re } ``` -Below is an example of the data that will be sent to the sink when the filter received a CborTx record. A block can contain many transactions, so the sink will receive an event for each transaction. +From a `CborTx` (when `SplitBlock` ran first), it receives one structured transaction per +event: ```json { diff --git a/docs/v2/filters/rollback_buffer.mdx b/docs/v2/filters/rollback_buffer.mdx index 024c85ee..c890620e 100644 --- a/docs/v2/filters/rollback_buffer.mdx +++ b/docs/v2/filters/rollback_buffer.mdx @@ -1,37 +1,43 @@ --- title: Rollback Buffer +sidebar: + order: 6 --- -The "rollback buffer" feature provides a way to mitigate the impact of chain rollbacks in downstream stages of the data-processing pipeline. +The `RollbackBuffer` filter softens the impact of chain rollbacks on downstream stages. It +holds blocks in memory until they reach a configured depth, only forwarding the ones that are +unlikely to be rolled back. Reach for it when your sink writes to storage where undoing +orphaned data is expensive or awkward. -## Context +## Why it helps -Handling rollbacks in a persistent storage requires clearing the orphaned data / blocks before adding new records. The complexity of this process may vary by concrete storage engine, but it always has an associated impact on performance. In some scenarios, it might even be prohibitive to process events without a reasonable level of confidence about the immutability of the record. +Rollbacks happen routinely, but a block's chance of being orphaned drops sharply the deeper it +sits in the chain. Without buffering, every rollback forces your sink to clear the orphaned +records before writing new ones — the cost of which varies by storage engine, and can be +prohibitive. -Rollbacks occur frequently under normal conditions, but the chances of a block becoming orphaned diminishes as the depth of the block increases. Some Oura use-cases may benefit from this property, some pipelines might prefer lesser rollback events, even if it means waiting for a certain number of confirmations. +With the buffer in place, only blocks past the `min_depth` threshold are sent downstream: -## Feature - -Oura provides a "rollback buffer" that will hold blocks in memory until they reach a certain depth. Only blocks above a min depth threshold will be sent down the pipeline. If a rollback occurs and the intersection is within the scope of the buffer, the rollback operation will occur within memory, totally transparent to the subsequent stages of the pipeline. - -If a rollback occurs and the intersection is outside of the scope of the buffer, Oura will fallback to the original behaviour and publish a RollbackEvent so that the "sink" stages may handle the rollback procedure manually. - -## Trade-off - -There's an obvious trade-off to this approach: latency. A pipeline will not process any events until the buffer fills up. Once the initial wait is over, the throughput of the whole pipeline should be equivalent to having no buffer at all (due to Oura's "pipelining" nature). If a rollback occurs, an extra delay will be required to fill the buffer again. - -Notice that even if the throughput isn't affected, the latency (measured as the delta between the timestamp at which the event reaches the "sink" stage and the original timestamp of the block) will always be affected by a fixed value proportional to the size of the buffer. - -## Implementation Details - -The buffer logic is implemented in pallas-miniprotocols library. It works by keeping a VecDeque data structure of chain "points", where roll-forward operations accumulate at the end of the deque and retrieving confirmed points means popping from the front of the deque. +- If a rollback lands **within** the buffer, it's resolved in memory and your sink never sees + it. +- If a rollback lands **outside** the buffer, Oura falls back to the normal behavior and emits + a rollback event for the sink to handle manually. ## Configuration -The min depth is a configurable setting available when running in daemon mode. Higher min_depth values will lower the chances of experiencing a rollback event, at the cost of adding more latency. A config would look like this: - -```toml +```toml title="daemon.toml" [[filters]] type = "RollbackBuffer" min_depth = 6 ``` + +- `type` (required): the literal value `RollbackBuffer`. +- `min_depth` (required): how many confirmations a block needs before it's forwarded. Higher + values mean fewer rollback events, at the cost of more latency. + +:::caution[Trade-off: latency] +The pipeline won't emit anything until the buffer fills, and each event reaches the sink a +fixed delay later (proportional to `min_depth`). Throughput is unaffected once the buffer is +primed, thanks to Oura's pipelining — but a rollback empties the buffer and adds the warm-up +delay again. +::: diff --git a/docs/v2/filters/select.mdx b/docs/v2/filters/select.mdx index ec8daad0..fe98e544 100644 --- a/docs/v2/filters/select.mdx +++ b/docs/v2/filters/select.mdx @@ -1,57 +1,75 @@ --- title: Select Filter -sidebar: +sidebar: label: Select + order: 3 --- -The `select` filter makes it possible to filter data that make sense for your cases. +The `Select` filter keeps only the events that match a **predicate** and drops the rest — so a +sink downstream sees just the transactions you care about (by address, asset, datum, or +metadata). It's the workhorse for "tell me when _X_ happens on chain". -The select filter requires [ParseCbor](/oura/v2/filters/parse_cbor) filter enabled to work. +:::note +`Select` works on structured records, so it needs the +[ParseCbor](/oura/v2/filters/parse_cbor) filter earlier in the chain (usually preceded by +[SplitBlock](/oura/v2/filters/split_block) for tx-level matching). +::: ## Configuration -Adding the following section to the daemon config file will enable the filter as part of the pipeline: - -```toml +```toml title="daemon.toml" [[filters]] type = "Select" skip_uncertain = true predicate = ``` -## Examples +- `type` (required): the literal value `Select`. +- `skip_uncertain` (required): when `true`, events that can't be conclusively evaluated against + the predicate are dropped rather than kept. +- `predicate` (required): the match expression — see the patterns below. + +## Simple predicates + +The simplest predicate is a single string that matches any transaction touching a given +entity. -Match any tx that interacts with this particular address +Match a transaction that interacts with an address: ```toml predicate = "addr1qx2fxv2umyhttkxyxp8x0dlpdt3k6cwng5pxj3jhsydzer3n0d3vllmyqwsx5wktcd8cc3sq835lu7drv2xwl2wywfgse35a3x" ``` -Match any tx that interacts with this particular stake address +Match a stake address: ```toml predicate = "stake178phkx6acpnf78fuvxn0mkew3l0fd058hzquvz7w36x4gtcccycj5" ``` -Match any tx that interacts with this particular asset +Match an asset: ```toml predicate = "asset17jd78wukhtrnmjh3fngzasxm8rck0l2r4hhyyt" ``` -Match any tx that holds a particular datum +Match a datum: ```toml predicate = "datum1httkxyxp8x0dlpdt3k6cwng5pxj3j" ``` -Match any tx that holds a particular metadata label +Match a metadata label (prefix the label with `#`): ```toml predicate = "#127" ``` -Match any tx that interacts with any of these particular address +## Combining predicates + +Use `any` (match if _any_ entry matches) or `all` (match only if _every_ entry matches) to +compose simple predicates. + +Match a transaction touching **any** of these addresses: ```toml [filters.predicate] @@ -62,7 +80,7 @@ any = [ ] ``` -Match any tx that interacts with all of these particular address simultaneously +Match a transaction touching **all** of these at once: ```toml [filters.predicate] @@ -72,7 +90,8 @@ all = [ ] ``` -Match any tx that simultanously interacts with a particlar address, holds a particular asset and present a particular metadata label +Mix entity types — here, an address that simultaneously holds an asset and carries a metadata +label: ```toml [filters.predicate] @@ -83,7 +102,7 @@ all = [ ] ``` -Match tx that has an output that simultaneously points to a particular address and contains a particular asset +Match against a single output's contents (address, assets, and datum together): ```toml [filters.predicate.match.output] @@ -92,22 +111,21 @@ assets = ["asset17jd78wukhtrnmjh3fngzasxm8rck0l2r4hhyyt"] datum = "datum1httkxyxp8x0dlpdt3k6cwng5pxj3j" ``` -## Metadata Filtering +## Metadata patterns -Match any tx that holds a particular metadata label +Match any transaction carrying a metadata label: ```toml predicate = "#674" ``` -Match transactions with metadata containing a regex pattern (recursively searches arrays and maps — including map keys and values — and matches only text metadatum) +Match on the *content* of metadata with a regular expression. The match recurses through +arrays and maps (including map keys and values) and applies only to text metadata: ```toml [filters.predicate.match.metadata] label = 674 [filters.predicate.match.metadata.value.text] -regex = "(?i)hello.*world" # Case-insensitive +regex = "(?i)hello.*world" # case-insensitive ``` - - diff --git a/docs/v2/filters/split_block.mdx b/docs/v2/filters/split_block.mdx index ef31a075..6d584cf6 100644 --- a/docs/v2/filters/split_block.mdx +++ b/docs/v2/filters/split_block.mdx @@ -1,25 +1,28 @@ --- title: Split CBOR Block filter -sidebar: +sidebar: label: Split Block + order: 1 --- -The `split_block` filter will decode the Cbor block and map each transaction within the block to the CborTx format. - -Therefore the next stage will receive each transaction that was in the block as an event in the pipeline, i.e. if the block contains 5 transactions the next stage will receive 5 events with each transaction, in CborTx format. +The `SplitBlock` filter decodes a CBOR block and emits each transaction inside it as its own +event (in `CborTx` format). If a block holds 5 transactions, the next stage receives 5 events. +Add it first whenever you want to work at the transaction level rather than the block level — +for example before [ParseCbor](/oura/v2/filters/parse_cbor) and +[Select](/oura/v2/filters/select). ## Configuration -Adding the following section to the daemon config file will enable the filter as part of the pipeline: - -```toml +```toml title="daemon.toml" [[filters]] type = "SplitBlock" ``` +This filter takes no additional options. + ## Examples -Below is an example of the data that will be sent to the sink. A block can contain many transactions, so the sink will receive an event for each transaction. +Each event the next stage receives carries one transaction's raw CBOR: ```json { diff --git a/docs/v2/filters/wasm.mdx b/docs/v2/filters/wasm.mdx index 4c56cb7a..fabef783 100644 --- a/docs/v2/filters/wasm.mdx +++ b/docs/v2/filters/wasm.mdx @@ -1,18 +1,32 @@ --- title: Wasm +sidebar: + order: 8 --- -The `wasm` or `webassembly` filter will allow a wasm binary to be executed as a stage within the pipeline. Therefore, it's possible to write a stage in any language that supports wasm. +The `WasmPlugin` filter runs a WebAssembly binary as a stage in the pipeline. Each event is +handed to your plugin, which can transform it, drop it, or expand it into several events — so +you can write custom filter logic in any language that compiles to WASM (Go, Rust, Assembly +Script, …) without forking Oura. -## Configuration +:::caution[Requires a custom build] +The WASM plugin runtime isn't in the default binary (it's large). Build Oura with the `wasm` +feature — see [Install from source](/oura/v2/installation/from_source). +::: -Adding the following section to the daemon config file will enable the filter as part of the pipeline: +## Configuration -```toml +```toml title="daemon.toml" [[filters]] type = "WasmPlugin" path = "./extract_fee/plugin.wasm" ``` -Follow this example to build a plugin wasm in golang [example](https://github.com/txpipe/oura/blob/main/examples/wasm_basic/README.md) +- `type` (required): the literal value `WasmPlugin`. +- `path` (required): the path to the compiled `.wasm` plugin to load. + +## Building a plugin +The [`wasm_basic` example](https://github.com/txpipe/oura/blob/main/examples/wasm_basic/README.md) +walks through building a plugin in Go, including the expected entry point and how records are +passed in and out. diff --git a/docs/v2/filters/work_stats.mdx b/docs/v2/filters/work_stats.mdx index 204c7108..e484b071 100644 --- a/docs/v2/filters/work_stats.mdx +++ b/docs/v2/filters/work_stats.mdx @@ -2,15 +2,20 @@ title: Work Stats filter sidebar: label: Work Stats + order: 7 --- -The `work_stats` filter tracks pipeline progress as events pass through it and can optionally finalize (stop) the pipeline once a configured stopping point is reached. It passes events through unchanged, so it can be placed anywhere in the filter chain. +The `WorkStats` filter tracks pipeline progress as events pass through, and can optionally stop +the pipeline once a target is reached. It passes events through unchanged, so you can drop it +anywhere in the filter chain. -This filter is how finalization is configured in v2 (see [finalize options](/oura/v2/advanced/finalize_options) for the conceptual overview): the pipeline stops as soon as any configured condition is met. +This is also how **finalization** is configured in v2 (see +[finalize options](/oura/v2/advanced/finalize_options) for the concept): the pipeline exits +gracefully as soon as any configured condition is met. ## Configuration -```toml +```toml title="daemon.toml" [[filters]] type = "WorkStats" until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de" diff --git a/docs/v2/guides/cardano_2_kafka.mdx b/docs/v2/guides/cardano_2_kafka.mdx index 816698b3..1f8878c4 100644 --- a/docs/v2/guides/cardano_2_kafka.mdx +++ b/docs/v2/guides/cardano_2_kafka.mdx @@ -2,52 +2,54 @@ title: Cardano => Kafka --- -This guide shows how to leverage _Oura_ to stream data from a Cardano node into a _Kafka_ topic. +import { Steps } from '@astrojs/starlight/components'; -## About Kafka +This guide walks through streaming data from a Cardano node into a [Kafka](https://kafka.apache.org/) +topic with Oura — a typical setup for feeding chain data into an existing stream-processing +platform. -> Apache Kafka is a framework implementation of a software bus using stream-processing. It is an open-source software platform developed by the Apache Software Foundation written in Scala and Java. - -Find [more info](https://en.wikipedia.org/wiki/Apache_Kafka) about _Kafka_ in wikipedia or visit _Kafka's_ official [website](https://kafka.apache.org/) +:::caution[Kafka needs a custom build] +The Kafka sink isn't in the default binary. Build Oura with the `kafka` feature first — see +[Install from source](/oura/v2/installation/from_source). +::: ## Prerequisites -This examples assumes the following prerequisites: - -- A running Cardano node locally accesible via a unix socket. -- A Kafka cluster accesible through the network. -- An already existing Kafka topic where to output events -- _Oura_ binary release installed in local system +- A running Cardano node, reachable locally via a unix socket. +- A Kafka cluster reachable over the network, with the destination topic already created. +- Oura installed with the `kafka` feature enabled. ## Instructions -### 1. Create an Oura configuration file `cardano2kafka.toml` + -```toml -[source] -type = "N2C" -socket_path = "/opt/cardano/cnode/sockets/node0.socket" +1. **Create a config file** named `cardano2kafka.toml`: -[sink] -type = "Kafka" -brokers = ["kafka-broker-0:9092"] -topic = "cardano-events" -``` + ```toml title="cardano2kafka.toml" + [source] + type = "N2C" + socket_path = "/opt/cardano/cnode/sockets/node0.socket" -Some comments regarding the above configuration: + [sink] + type = "Kafka" + brokers = ["kafka-broker-0:9092"] + topic = "cardano-events" + ``` -- the `[source]` section indicates _Oura_ from where to pull chain data. - - the `N2C` source type value tells _Oura_ to get the data from a Cardano node using Node-to-Client mini-protocols (chain-sync instantiated to full blocks). - - the `socket_path` field indicates that we should connect via `Unix` socket at the specified path. This value should match the location of your local node socket path. -- the `[sink]` section tells _Oura_ where to send the information it gathered. - - the `Kafka` sink type value indicates that _Oura_ should use a _Kafka_ producer client as the output - - the `brokers` field indicates the location of the _Kafka_ brokers within the network. Several hostname:port pairs can be added to the array for a "cluster" scenario. - - the `topic` fields indicates which _Kafka_ topic to used for the outbound messages. + What this configures: + - **`[source]`** pulls chain data from your node. + - `N2C` reads from the node over Node-to-Client mini-protocols (chain-sync in full-blocks + mode). + - `socket_path` is the path to your local node's unix socket. + - **`[sink]`** sends the data on. + - `Kafka` uses a Kafka producer client as the output. + - `brokers` lists the Kafka brokers; add several `hostname:port` pairs for a cluster. + - `topic` is the topic to publish messages to. -### 2. Run _Oura_ in `daemon` mode +2. **Run Oura in `daemon` mode** against the config: -Run the following command from your terminal to start the daemon process: + ```sh + RUST_LOG=info oura daemon --config cardano2kafka.toml + ``` -```sh -RUST_LOG=info oura daemon --config cardano2kafka.toml -``` + diff --git a/docs/v2/how_it_works.mdx b/docs/v2/how_it_works.mdx new file mode 100644 index 00000000..63ca41c8 --- /dev/null +++ b/docs/v2/how_it_works.mdx @@ -0,0 +1,79 @@ +--- +title: How it works +sidebar: + order: 5 +--- + +_Oura_ is a streaming pipeline. It connects to a Cardano node, reads the chain one block at a +time, runs each event through a chain of filters, and hands the result to a sink. Every stage +is independent and runs on its own thread, so data flows through continuously rather than in +batches. + +```mermaid +flowchart LR + node([Cardano node]) -->|Ouroboros mini-protocols| source[Source] + source --> f1[Filter] --> f2[Filter] --> sink[Sink] + sink --> dest([Destination]) + sink -.reports position.-> cursor[(Cursor)] + cursor -.restores on restart.-> source +``` + +## The pipeline + +A pipeline is just four pieces, wired together in your `daemon.toml`: + +- **[Source](/oura/v2/sources)** — connects to the chain and produces a stream of events. It can + read from a local node (`N2C`), a remote relay (`N2N`), a [UTxO RPC](/oura/v2/sources/utxorpc) + endpoint, and [more](/oura/v2/sources). +- **[Intersect](/oura/v2/advanced/intersect_options)** — tells the source *where on the chain to + start*. The default is the current tip; you can also start from the origin or a specific point + to backfill history. +- **[Filters](/oura/v2/filters)** — a sequence of transformations applied to each event, in + order. Filters decode the raw block, split it into transactions, keep only what matches a + pattern, reshape it into JSON, and so on. +- **[Sink](/oura/v2/sinks)** — the destination. It takes the final event and sends it somewhere: + your terminal, a file, a message broker, a database, an HTTP endpoint, a cloud service. + +## Events and rollbacks + +The Cardano chain isn't append-only — the tip can **reorganize**, abandoning recently-seen +blocks in favor of a competing fork. Oura surfaces this directly: every event carries an +**action** describing what happened to the block. + +- `apply` — a block was added to the chain. The normal case. +- `undo` — a block was rolled back. If your sink already acted on it, you need to reverse that. +- `reset` — the pipeline jumped to a new position (for example, on startup or after recovering + across a gap). It carries a `point`, not a record. + +This is the one concept worth getting right before you build a consumer, because **how you +handle `undo` is up to you**. There are three common approaches: + +- **Handle it** — react to `undo` events and reverse whatever the matching `apply` did. Most + precise, but your downstream logic has to support it. +- **Buffer it away** — add the [RollbackBuffer filter](/oura/v2/filters/rollback_buffer) to hold + blocks until they're deep enough to be unlikely to roll back, so your sink rarely sees an + `undo` at all (it trades a little latency for simplicity). +- **Stay idempotent** — design the sink so reprocessing the same block is harmless, and combine + it with a [cursor](/oura/v2/advanced/stateful_cursor) so a restart resumes cleanly instead of + re-reading from scratch. + +For the exact JSON shape of an event — the `event` / `point` / `record` envelope — see the +[Data Dictionary](/oura/v2/reference/data_dictionary). + +## How records change shape + +The `record` inside each event isn't fixed — its shape depends on which filters ran before the +sink. A source emits the raw block as CBOR; filters progressively refine it: + +``` +CborBlock ──SplitBlock──▶ CborTx ──ParseCbor──▶ parsed Tx ──IntoJson──▶ JSON +``` + +So a sink reading straight from a source sees raw block CBOR, while one sitting after +`ParseCbor` sees a structured transaction. You compose the chain to produce whatever shape your +destination needs. The canonical pipeline — `SplitBlock → ParseCbor → Select → sink` — is a good +default starting point. + +See the [Filters overview](/oura/v2/filters) for the full set, and the record-variant table in +the [Data Dictionary](/oura/v2/reference/data_dictionary) for exactly which filter produces which +shape. diff --git a/docs/v2/index.mdx b/docs/v2/index.mdx index 669dd64a..7c61a0c0 100644 --- a/docs/v2/index.mdx +++ b/docs/v2/index.mdx @@ -6,9 +6,19 @@ sidebar: ## Motivation -We have tools to "explore" the Cardano blockchain, which are useful when you know what you're looking for. We argue that there's a different, complementary use-case which is to "observe" the blockchain and react to particular event patterns. +We already have tools to "explore" the Cardano blockchain — great when you know what you're +looking for. _Oura_ fills the complementary need: to "observe" the chain as it moves and react +to particular event patterns. -_Oura_ is a rust-native implementation of a pipeline that connects to the tip of a Cardano node through a combination of _Ouroboros_ mini-protocol (using either a unix socket or tcp bearer), filters the events that match a particular pattern and then submits a succinct, self-contained payload to pluggable observers called "sinks". +_Oura_ is a Rust-native pipeline. It connects to the tip of a Cardano node over the _Ouroboros_ +mini-protocols (via a unix socket or TCP), filters the events that match a pattern you define, +and forwards a succinct, self-contained payload to pluggable observers called **sinks** — a +terminal, a file, a message broker, a database, and more. + +:::tip[New to Oura?] +Jump to the [Quick Start](/oura/v2/quickstart) to see live chain data in a couple of minutes, or +read [How it works](/oura/v2/how_it_works) for the pipeline model. +::: ## Etymology @@ -16,9 +26,12 @@ The name of the tool is inspired by the `tail` command available in unix-like sy ## Under the Hood -All the heavy lifting required to communicate with the Cardano node is done by the [Pallas](https://github.com/txpipe/pallas) library, which provides an implementation of the Ouroboros multiplexer and a few of the required mini-protocol state-machines (ChainSync and LocalState in particular). +The heavy lifting of talking to the Cardano node is handled by the [Pallas](https://github.com/txpipe/pallas) +library, which implements the Ouroboros multiplexer and the mini-protocol state machines Oura +relies on (notably ChainSync and LocalState). -The data pipeline makes heavy use (maybe a bit too much) of multi-threading and mpsc channels provided by Rust's `std::sync` library. +The data pipeline itself leans heavily on multi-threading and mpsc channels from Rust's +`std::sync` library, which keeps each stage independent and the footprint small. ## Use Cases @@ -44,5 +57,7 @@ If the available out-of-the-box features don't satisfy your particular use-case, For ready-to-run configurations covering every source, filter and sink — plus end-to-end pipeline recipes — see the [Examples](/oura/v2/examples) section. -## (Experimental) Windows Support -_Oura_ Windows support is currently __experimental__, Windows build supports only [Node-to-Node](/oura/v2/sources/n2n) source with tcp socket bearer. +:::caution[Windows support is experimental] +On Windows, Oura currently supports only the [Node-to-Node](/oura/v2/sources/n2n) source over a +TCP socket bearer. +::: diff --git a/docs/v2/installation/binary_release.mdx b/docs/v2/installation/binary_release.mdx index 92212bb2..08ccc096 100644 --- a/docs/v2/installation/binary_release.mdx +++ b/docs/v2/installation/binary_release.mdx @@ -2,56 +2,58 @@ title: Binary Releases --- -Oura can be run as a standalone executable. Follow the instructions for your particular OS / Platform to install a local copy from our Github pre-built releases. +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +Oura ships as a standalone executable. Grab a pre-built release for your platform below — no +toolchain or compilation required. ## What's included -The pre-built binaries are **batteries-included**: every integration that builds cleanly across all release targets ships by default, so you rarely need to compile a custom build. +The pre-built binaries are **batteries-included**: every integration that builds cleanly +across all release targets ships by default, so you rarely need a custom build. - **Sources:** node-to-node (`n2n`), node-to-client (`n2c`), UTxO RPC (`u5c`), Hydra (`hydra`), AWS S3 (`aws`) - **Sinks:** terminal, stdout, file rotate, webhook, Elasticsearch, Redis, RabbitMQ, SQL (SQLite / Postgres), AWS (SQS / Lambda / S3), GCP (Pub/Sub / Cloud Functions) - **Filters:** all built-in filters (select, split block, JSON, legacy v1, etc.) -A few integrations are left out to keep the binary lean and free of system/C dependencies, and require a [source build](/v2/installation/from_source): +A few integrations are left out to keep the binary lean and free of system/C dependencies. They +require a [source build](/oura/v2/installation/from_source): - **`kafka`** — its only TLS backend is OpenSSL; bundling it would pull a vendored OpenSSL build into the otherwise pure-Rust-TLS binary - **`zeromq`** — dynamically links a system `libzmq` at runtime, so it can't ship in a portable prebuilt binary - **`mithril`** — pulls a heavy GMP/blst C-crypto toolchain that significantly increases binary size and build time - **`wasm`** — the WASM plugin runtime (large, and itself a plugin host) -## Install via shell script - -You can run the following command line script to install Oura on supported systems (Mac / Linux) - -```sh -curl --proto '=https' --tlsv1.2 -LsSf https://github.com/txpipe/oura/releases/latest/download/oura-installer.sh | sh -``` - -## Install via powershell script - -You can use Powershell to install Oura on Windows systems. - -```sh -powershell -c "irm https://github.com/txpipe/oura/releases/latest/download/oura-installer.ps1 | iex" -``` - -## Install via Homebrew - -You can use Homebrew to install the latest version of Oura in supported systems (Mac / Linux) - -```sh -brew install txpipe/tap/oura -``` - -## Install via NPM - -You can use NPM to install the latest version of Oura in supported systems (Mac / Linux) - -```sh -npm install -g @txpipe/oura -``` - -## Download Binaries +## Install + +Pick the method that suits your platform: + + + + ```sh + curl --proto '=https' --tlsv1.2 -LsSf https://github.com/txpipe/oura/releases/latest/download/oura-installer.sh | sh + ``` + + + ```sh + powershell -c "irm https://github.com/txpipe/oura/releases/latest/download/oura-installer.ps1 | iex" + ``` + + + ```sh + brew install txpipe/tap/oura + ``` + + + ```sh + npm install -g @txpipe/oura + ``` + + + +## Download binaries + +Prefer to grab the archive yourself? Download the build for your platform: | File | Platform | |--------|----------| diff --git a/docs/v2/installation/docker.mdx b/docs/v2/installation/docker.mdx index 956c623b..7b7dbbbe 100644 --- a/docs/v2/installation/docker.mdx +++ b/docs/v2/installation/docker.mdx @@ -47,9 +47,13 @@ Images are published to `ghcr.io/txpipe/oura` under several tags, each following ## Versioned Images -It is highly recommended to use a fixed image version in production environments to avoid the effects of new features being included in each release. +:::caution +Pin a fixed image version in production. `latest` tracks the `main` branch (not releases), so +it can change under you on any push — avoid it outside of local experimentation. +::: -To use a versioned image, replace the `latest` tag by the desired version with the `v` prefix. For example, to pin the `v2.1.0` release, use the following image: +To use a versioned image, replace the `latest` tag with the desired version, prefixed with +`v`. For example, to pin the `v2.1.0` release: ``` ghcr.io/txpipe/oura:v2.1.0 diff --git a/docs/v2/installation/from_source.mdx b/docs/v2/installation/from_source.mdx index 7b9beabf..0c85c903 100644 --- a/docs/v2/installation/from_source.mdx +++ b/docs/v2/installation/from_source.mdx @@ -16,7 +16,7 @@ cd oura cargo install --path . ``` -This builds the same [batteries-included](/v2/installation/binary_release) plugin set as the pre-built releases. +This builds the same [batteries-included](/oura/v2/installation/binary_release) plugin set as the pre-built releases. ## Enabling the remaining integrations diff --git a/docs/v2/installation/kubernetes.mdx b/docs/v2/installation/kubernetes.mdx index a88356b6..fe97be1e 100644 --- a/docs/v2/installation/kubernetes.mdx +++ b/docs/v2/installation/kubernetes.mdx @@ -2,13 +2,23 @@ title: Kubernetes --- -_Oura_ running in `daemon` mode can be installed can be deployed in Kubernetes cluster, Depending on your needs, we recommend any of the following approaches. +import { Tabs, TabItem } from '@astrojs/starlight/components'; -## Approach #1: Sidecar Container +Oura runs well in Kubernetes in `daemon` mode. There are two common ways to deploy it, +depending on where your Cardano node lives: -_Oura_ can be loaded as a sidecar container in the same pod as your Cardano node. Since containers in a pod share the same file-system layer, it's easy to point _Oura_ to the unix-socket of the node. +- **Sidecar** — run Oura in the same pod as your node, sharing the unix socket. Simplest when + the node is already in your cluster. +- **Standalone** — run Oura as its own `Deployment`. Use this when the node lives outside the + cluster, or when you want to keep it isolated from sidecar access. -In the following example yaml, we show a redacted version of a Cardano node resource defined a s Kubernetes STS. Pay attention on the extra container and to the volume mounts to share the unix socket. + + +Load Oura as a second container in your node's pod. Because containers in a pod share a +file-system layer, Oura can read the node's unix socket directly. + +The example below is a redacted Cardano node `StatefulSet`. Note the extra container and the +volume mounts that share the unix socket. ```yaml apiVersion: apps/v1 @@ -49,7 +59,7 @@ spec: volumes: - # REDACTED: here goes any required volume for you normal cardano node setup + # REDACTED: here goes any required volume for your normal cardano node setup # empty-dir volume to share the unix socket between containers - name: unix-socket @@ -60,13 +70,10 @@ spec: configMap: name: oura-config ``` - - -## Approach #2: Standalone Deployment - -_Oura_ can be implemented as a standalone Kubernetes `deployment` resource. This is useful if your Cardano node is not part of your Kubernetes cluster or if you want to keep your node strictly isolated from the access of a sidecard pod. - -Please note that the amount of replicas is set to `1`. _Oura_ doesn't have any kind of "coordination" between instances. Adding more than one replica will just create extra pipelines duplicating the same work. + + +Run Oura as its own `Deployment`, with its config supplied by a `ConfigMap`. Use this when the +node isn't part of your cluster, or you want it isolated from sidecar access. ```yaml apiVersion: v1 @@ -123,3 +130,10 @@ spec: configMap: name: oura ``` + + + +:::caution[Keep replicas at 1] +Oura instances don't coordinate with each other. Running more than one replica just creates +parallel pipelines duplicating the same work — keep `replicas: 1`. +::: diff --git a/docs/v2/quickstart.mdx b/docs/v2/quickstart.mdx new file mode 100644 index 00000000..875d2816 --- /dev/null +++ b/docs/v2/quickstart.mdx @@ -0,0 +1,75 @@ +--- +title: Quick Start +sidebar: + order: 15 +--- + +import { Steps } from '@astrojs/starlight/components'; + +This page takes you from nothing to live Cardano data in a couple of minutes. You don't need +your own node — the first step connects to a public relay. + + + +1. **Install Oura.** Grab the pre-built binary for your platform (no toolchain required): + + ```sh + curl --proto '=https' --tlsv1.2 -LsSf https://github.com/txpipe/oura/releases/latest/download/oura-installer.sh | sh + ``` + + Other methods (Homebrew, npm, Windows, manual download) are on the + [Binary Releases](/oura/v2/installation/binary_release) page. + +2. **Watch live chain data.** Point `oura watch` at a public mainnet relay and see events scroll + by in your terminal: + + ```sh + oura watch backbone.mainnet.cardanofoundation.org:3001 --bearer tcp + ``` + + That's the fastest way to confirm Oura works. `watch` is a read-only, throttled view meant + for eyeballing the chain — see the [watch command](/oura/v2/usage/watch) for its options. + +3. **Run a real pipeline.** For anything beyond watching, you write a `daemon.toml` describing a + source, filters, and a sink. This one reads from the same relay, decodes each block into + transactions, and prints them to stdout: + + ```toml title="daemon.toml" + [source] + type = "N2N" + peers = ["backbone.mainnet.cardanofoundation.org:3001"] + + [intersect] + type = "Tip" + + [[filters]] + type = "SplitBlock" + + [[filters]] + type = "ParseCbor" + + [sink] + type = "Stdout" + ``` + + Then run it in daemon mode: + + ```sh + oura daemon --config daemon.toml + ``` + + + +:::tip +Starting from `Tip` means you only see *new* blocks as they're produced. To replay history +instead, change the `[intersect]` block — see [Intersect options](/oura/v2/advanced/intersect_options). +::: + +## Where to go next + +- **[How it works](/oura/v2/how_it_works)** — the pipeline model, events, and rollbacks in one + page. Worth reading before you build a consumer. +- **[Sinks](/oura/v2/sinks)** — send events to a file, message broker, database, or cloud + service instead of stdout. +- **[Examples](/oura/v2/examples)** — ready-to-run configurations for every source, filter, and + sink. diff --git a/docs/v2/reference/data_dictionary.mdx b/docs/v2/reference/data_dictionary.mdx index 340ad68e..b382112b 100644 --- a/docs/v2/reference/data_dictionary.mdx +++ b/docs/v2/reference/data_dictionary.mdx @@ -1,5 +1,7 @@ --- title: Data Dictionary +sidebar: + order: 1 --- _Oura_ follows a Cardano chain and outputs events. Each event contains data about itself and about the _context_ in which it occurred. @@ -8,6 +10,9 @@ A consumer aggregating a sequence of multiple events will notice redundant / dup ## Pipeline Data Model +This page is the static schema reference — the exact shape of an event. For the moving picture +(how events flow and how rollbacks are surfaced), see [How it works](/oura/v2/how_it_works). + Internally, every unit that flows through an Oura v2 pipeline is a `ChainEvent` that wraps a chain `point` and (except for resets) a `Record`. When a sink serializes an event to JSON, it uses the following envelope: ```json @@ -34,7 +39,7 @@ The `record` is one of the following variants, determined by the source and the | Parsed block | [ParseCbor](/oura/v2/filters/parse_cbor) | a [UTxORPC](https://utxorpc.org/cardano) `Block`. | | Parsed tx | [ParseCbor](/oura/v2/filters/parse_cbor) | a [UTxORPC](https://utxorpc.org/cardano) `Tx`. | | Generic JSON | [Into JSON](/oura/v2/filters/into_json) | an arbitrary JSON value. | -| Legacy v1 | [LegacyV1](/oura/v2/filters/legacy_v1) | a v1-style event (see _Available Events (Legacy V1 Filter)_ below). | +| Legacy v1 | [LegacyV1](/oura/v2/filters/legacy_v1) | a v1-style event (see [Legacy V1 events](/oura/v2/reference/legacy_v1_events)). | ## Available Events (ParseCbor Filter) @@ -42,593 +47,6 @@ When the [ParseCbor](/oura/v2/filters/parse_cbor) filter is enabled, records use ## Available Events (Legacy V1 Filter) -The following list represent the already implemented events. These data structures are represented as an `enum` at the code level. - -### `RollBack` Event - -Data on chain rollback(The result of the local node switching to the consensus chains). - -| Name | DataType | Description | -| :--------- | :-------------- | :----------------------------------------- | -| block_slot | u64 | Slot of the rolled back block. | -| block_hash | Option\ | Block hash. Hash of the rolled back block. | - -
-
-
- -### `Block` Event - -Data on an issued block. - -| Name | DataType | Description | -| :---------- | :------- | :------------------------------------ | -| body_size | usize | Size of the block. | -| issuer_vkey | String | Block issuer Public verification key. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | - -
-
-
- -### `Transaction` Event - -Data on a transaction. - -| Name | DataType | Description | -| :---------------------- | :----------- | :------------------------------------- | -| fee | u64 | Transaction fees in lovelace. | -| ttl | Option\ | Transaction time to live. | -| validity_interval_start | Option\ | Start of transaction validity interval | -| network_id | Option\ | Network ID. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `TxInput` Event - -Data on a transaction input. - -| Name | DataType | Description | -| :---- | :------- | :------------------------------------ | -| tx_id | String | Transaction ID. | -| index | u64 | Index of input in transaction inputs. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| input_idx | Option\ | Input ID. | - -
-
-
- -### `TxOutput` Event - -Data on a transaction output (UTXO). - -| Name | DataType | Description | -| :------ | :------- | :-------------------------- | -| address | String | Address of UTXO. | -| amount | u64 | Amount of lovelace in UTXO. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| output_idx | Option\ | Output ID. | - -
-
-
- -### `OutputAsset` Event - -Data on a non-ADA asset in a UTXO. - -| Name | DataType | Description | -| :----- | :------- | :----------------------- | -| policy | String | Minting policy of asset. | -| asset | String | Asset ID. | -| amount | u64 | Amount of asset. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| output_idx | Option\ | Output ID. | - -
-
-
- -### `Metadata` Event - -| Name | DataType | Description | -| :--------------- | :----------------- | :------------ | -| label | String | Metada label. | -| map_json (\*) | Option\ | Json map. | -| array_json (\*) | Option\ | Json array. | -| int_scalar (\*) | Option\ | Text. | -| bytes_hex (\*) | Option\ | Bytes. | - -(\*) Only one of these options will be used. - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `VkeyWitness` Event - -| Name | DataType | Description | -| :------------ | :------- | :---------- | -| vkey_hex | String | | -| signature_hex | String | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | - -
-
-
- -### `NativeWitness` Event - -| Name | DataType | Description | -| :---------- | :-------- | :---------- | -| policy_id | String | | -| script_json | JsonValue | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `PlutusWitness` Event - -| Name | DataType | Description | -| :---------- | :------- | :---------- | -| script_hash | String | | -| script_hex | String | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `PlutusRedeemer` Event - -| Name | DataType | Description | -| :------------- | :-------- | :---------- | -| purpose | String | | -| ex_units_mem | u32 | | -| ex_units_steps | u64 | | -| input_idx | u32 | | -| plutus_data | JsonValue | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `PlutusDatum` Event - -| Name | DataType | Description | -| :---------- | :-------- | :---------- | -| datum_hash | String | | -| plutus_data | JsonValue | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `CIP25Asset` Event - -| Name | DataType | Description | -| :---------- | :-------------- | :--------------------------------------------------- | -| version | String | [version](https://cips.cardano.org/cips/cip25/#cddl) | -| policy | String | | -| asset | String | | -| name | Option\ | | -| image | Option\ | | -| media_type | Option\ | | -| description | Option\ | | -| raw_json | JsonValue | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `CIP15Asset` Event - -| Name | DataType | Description | -| :------------- | :-------- | :---------- | -| voting_key | String | | -| stake_pub | String | | -| reward_address | String | | -| nonce | i64 | | -| raw_json | JsonValue | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `Mint` Event - -Data on the minting of a non-ADA asset. - -| Name | DataType | Description | -| :------- | :------- | :------------------------ | -| policy | String | Minting policy of asset. | -| asset | String | Asset ID. | -| quantity | i64 | Quantity of asset minted. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `Collateral` Event - -Data on [collateral inputs](https://docs.cardano.org/plutus/collateral-mechanism). - -| Name | DataType | Description | -| :---- | :------- | :------------------------------------ | -| tx_id | String | Transaction ID. | -| index | u64 | Index of transaction input in inputs. | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `NativeScript` Event - -| Name | DataType | Description | -| :-------- | :-------- | :---------- | -| policy_id | String | | -| script | JsonValue | | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | - -
-
-
- -### `PlutusScript` Event - -| Name | DataType | Description | -| :--- | :------- | :---------- | -| hash | String | .... | -| data | String | .... | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `StakeRegistration` Event - -Data on stake registration event. - -| Name | DataType | Description | -| :--------- | :--------------------------------------------------------------------------------- | :------------------- | -| credential | [StakeCredential](https://docs.rs/oura/1.4.1/oura/model/enum.StakeCredential.html) | Staking credentials. | - -**Context** - -| Name | DataType | Description | -| :-------------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| certificate_idx | Option\ | | - -
-
-
- -### `StakeDeregistration` Event - -Data on stake deregistration event. - -| Name | DataType | Description | -| :--------- | :--------------------------------------------------------------------------------- | :------------------- | -| credential | [StakeCredential](https://docs.rs/oura/1.4.1/oura/model/enum.StakeCredential.html) | Staking credentials. | - -**Context** - -| Name | DataType | Description | -| :-------------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| certificate_idx | Option\ | | - -
-
-
- -### `StakeDelegation` Event - -Data on [stake delegation](https://docs.cardano.org/core-concepts/delegation) event. - -| Name | DataType | Description | -| :--------- | :--------------------------------------------------------------------------------- | :--------------------- | -| credential | [StakeCredential](https://docs.rs/oura/1.4.1/oura/model/enum.StakeCredential.html) | Stake credentials. | -| pool_hash | String | Hash of stake pool ID. | - -**Context** - -| Name | DataType | Description | -| :-------------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| certificate_idx | Option\ | | - -
-
-
- -### `PoolRegistration` Event - -Data on the stake [registration event](https://developers.cardano.org/docs/stake-pool-course/handbook/register-stake-pool-metadata/). - -| Name | DataType | Description | -| :------------- | :-------------- | :-------------------------------------- | -| operator | String | Stake pool operator ID. | -| vrf_keyhash | String | Kehash of node VRF operational key. | -| pledge | u64 | Stake pool pledge (lovelace). | -| cost | u64 | Operational costs per epoch (lovelace). | -| margin | f64 | Operator margin. | -| reward_account | String | Account to receive stake pool rewards. | -| pool_owners | Vec\ | Stake pool owners. | -| relays | Vec\ | .... | -| pool_metadata | Option\ | .... | - -**Context** - -| Name | DataType | Description | -| :-------------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| certificate_idx | Option\ | | - -
-
-
- -### `PoolRetirement` Event - -Data on [stake pool retirement](https://cardano-foundation.gitbook.io/stake-pool-course/stake-pool-guide/stake-pool/retire_stakepool) event. - -| Name | DataType | Description | -| :---- | :------- | :------------- | -| pool | String | Pool ID. | -| epoch | u64 | Current epoch. | - -**Context** - -| Name | DataType | Description | -| :-------------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | -| certificate_idx | Option\ | | - -
-
-
- -### `GenesisKeyDelegation` Event - -Data on genesis key delegation. - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Current slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | - -
-
-
- -### `MoveInstantaneousRewardsCert` Event - -| Name | DataType | Description | -| :------------------- | :--------------------------------------- | :---------- | -| from_reserves | bool | .... | -| from_treasury | bool | .... | -| to_stake_credentials | Option\> | .... | -| to_other_pot | Option\ | .... | - -**Context** - -| Name | DataType | Description | -| :----------- | :-------------- | :---------------------------- | -| block_number | Option\ | Height of block from genesis. | -| block_hash | Option\ | Block hash. | -| slot | Option\ | Blockchain slot. | -| timestamp | Option\ | Timestamp. | -| tx_idx | Option\ | Transaction Index. | -| tx_hash | Option\ | Transaction hash. | +When the [LegacyV1](/oura/v2/filters/legacy_v1) filter is enabled, records take the shape of +the original Oura v1 events. That catalog is long, so it lives on its own page: +**[Legacy V1 events](/oura/v2/reference/legacy_v1_events)**. diff --git a/docs/v2/reference/legacy_v1_events.mdx b/docs/v2/reference/legacy_v1_events.mdx new file mode 100644 index 00000000..8b06f95e --- /dev/null +++ b/docs/v2/reference/legacy_v1_events.mdx @@ -0,0 +1,507 @@ +--- +title: Legacy V1 Events +sidebar: + label: Legacy V1 Events + order: 2 +--- + +When the [LegacyV1](/oura/v2/filters/legacy_v1) filter is enabled, records are reshaped into +the Oura **v1** event schema. This page is the catalog of those events — each is a variant of +an `enum` at the code level. Every event carries its own fields plus a **Context** describing +where in the chain it occurred. + +For the structured (non-legacy) data model, see the +[Data Dictionary](/oura/v2/reference/data_dictionary). + +### `RollBack` Event + +Data on chain rollback (the result of the local node switching to the consensus chain). + +| Name | DataType | Description | +| :--------- | :-------------- | :----------------------------------------- | +| block_slot | u64 | Slot of the rolled back block. | +| block_hash | Option\ | Block hash. Hash of the rolled back block. | + +### `Block` Event + +Data on an issued block. + +| Name | DataType | Description | +| :---------- | :------- | :------------------------------------ | +| body_size | usize | Size of the block. | +| issuer_vkey | String | Block issuer Public verification key. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | + +### `Transaction` Event + +Data on a transaction. + +| Name | DataType | Description | +| :---------------------- | :----------- | :------------------------------------- | +| fee | u64 | Transaction fees in lovelace. | +| ttl | Option\ | Transaction time to live. | +| validity_interval_start | Option\ | Start of transaction validity interval | +| network_id | Option\ | Network ID. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `TxInput` Event + +Data on a transaction input. + +| Name | DataType | Description | +| :---- | :------- | :------------------------------------ | +| tx_id | String | Transaction ID. | +| index | u64 | Index of input in transaction inputs. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| input_idx | Option\ | Input ID. | + +### `TxOutput` Event + +Data on a transaction output (UTXO). + +| Name | DataType | Description | +| :------ | :------- | :-------------------------- | +| address | String | Address of UTXO. | +| amount | u64 | Amount of lovelace in UTXO. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| output_idx | Option\ | Output ID. | + +### `OutputAsset` Event + +Data on a non-ADA asset in a UTXO. + +| Name | DataType | Description | +| :----- | :------- | :----------------------- | +| policy | String | Minting policy of asset. | +| asset | String | Asset ID. | +| amount | u64 | Amount of asset. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| output_idx | Option\ | Output ID. | + +### `Metadata` Event + +| Name | DataType | Description | +| :--------------- | :------------------ | :------------ | +| label | String | Metada label. | +| map_json (\*) | Option\ | Json map. | +| array_json (\*) | Option\ | Json array. | +| int_scalar (\*) | Option\ | Number. | +| text_scalar (\*) | Option\ | Text. | +| bytes_hex (\*) | Option\ | Bytes. | + +(\*) Only one of these options will be used. + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `VkeyWitness` Event + +| Name | DataType | Description | +| :------------ | :------- | :---------- | +| vkey_hex | String | | +| signature_hex | String | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | + +### `NativeWitness` Event + +| Name | DataType | Description | +| :---------- | :-------- | :---------- | +| policy_id | String | | +| script_json | JsonValue | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `PlutusWitness` Event + +| Name | DataType | Description | +| :---------- | :------- | :---------- | +| script_hash | String | | +| script_hex | String | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `PlutusRedeemer` Event + +| Name | DataType | Description | +| :------------- | :-------- | :---------- | +| purpose | String | | +| ex_units_mem | u32 | | +| ex_units_steps | u64 | | +| input_idx | u32 | | +| plutus_data | JsonValue | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `PlutusDatum` Event + +| Name | DataType | Description | +| :---------- | :-------- | :---------- | +| datum_hash | String | | +| plutus_data | JsonValue | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `CIP25Asset` Event + +| Name | DataType | Description | +| :---------- | :-------------- | :--------------------------------------------------- | +| version | String | [version](https://cips.cardano.org/cips/cip25/#cddl) | +| policy | String | | +| asset | String | | +| name | Option\ | | +| image | Option\ | | +| media_type | Option\ | | +| description | Option\ | | +| raw_json | JsonValue | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `CIP15Asset` Event + +| Name | DataType | Description | +| :------------- | :-------- | :---------- | +| voting_key | String | | +| stake_pub | String | | +| reward_address | String | | +| nonce | i64 | | +| raw_json | JsonValue | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `Mint` Event + +Data on the minting of a non-ADA asset. + +| Name | DataType | Description | +| :------- | :------- | :------------------------ | +| policy | String | Minting policy of asset. | +| asset | String | Asset ID. | +| quantity | i64 | Quantity of asset minted. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `Collateral` Event + +Data on [collateral inputs](https://docs.cardano.org/plutus/collateral-mechanism). + +| Name | DataType | Description | +| :---- | :------- | :------------------------------------ | +| tx_id | String | Transaction ID. | +| index | u64 | Index of transaction input in inputs. | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `NativeScript` Event + +| Name | DataType | Description | +| :-------- | :-------- | :---------- | +| policy_id | String | | +| script | JsonValue | | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | + +### `PlutusScript` Event + +| Name | DataType | Description | +| :--- | :------- | :---------- | +| hash | String | .... | +| data | String | .... | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `StakeRegistration` Event + +Data on stake registration event. + +| Name | DataType | Description | +| :--------- | :--------------------------------------------------------------------------------- | :------------------- | +| credential | [StakeCredential](https://docs.rs/oura/1.4.1/oura/model/enum.StakeCredential.html) | Staking credentials. | + +**Context** + +| Name | DataType | Description | +| :-------------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| certificate_idx | Option\ | | + +### `StakeDeregistration` Event + +Data on stake deregistration event. + +| Name | DataType | Description | +| :--------- | :--------------------------------------------------------------------------------- | :------------------- | +| credential | [StakeCredential](https://docs.rs/oura/1.4.1/oura/model/enum.StakeCredential.html) | Staking credentials. | + +**Context** + +| Name | DataType | Description | +| :-------------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| certificate_idx | Option\ | | + +### `StakeDelegation` Event + +Data on [stake delegation](https://docs.cardano.org/core-concepts/delegation) event. + +| Name | DataType | Description | +| :--------- | :--------------------------------------------------------------------------------- | :--------------------- | +| credential | [StakeCredential](https://docs.rs/oura/1.4.1/oura/model/enum.StakeCredential.html) | Stake credentials. | +| pool_hash | String | Hash of stake pool ID. | + +**Context** + +| Name | DataType | Description | +| :-------------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| certificate_idx | Option\ | | + +### `PoolRegistration` Event + +Data on the stake [registration event](https://developers.cardano.org/docs/stake-pool-course/handbook/register-stake-pool-metadata/). + +| Name | DataType | Description | +| :------------- | :-------------- | :-------------------------------------- | +| operator | String | Stake pool operator ID. | +| vrf_keyhash | String | Kehash of node VRF operational key. | +| pledge | u64 | Stake pool pledge (lovelace). | +| cost | u64 | Operational costs per epoch (lovelace). | +| margin | f64 | Operator margin. | +| reward_account | String | Account to receive stake pool rewards. | +| pool_owners | Vec\ | Stake pool owners. | +| relays | Vec\ | .... | +| pool_metadata | Option\ | .... | + +**Context** + +| Name | DataType | Description | +| :-------------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| certificate_idx | Option\ | | + +### `PoolRetirement` Event + +Data on [stake pool retirement](https://cardano-foundation.gitbook.io/stake-pool-course/stake-pool-guide/stake-pool/retire_stakepool) event. + +| Name | DataType | Description | +| :---- | :------- | :------------- | +| pool | String | Pool ID. | +| epoch | u64 | Current epoch. | + +**Context** + +| Name | DataType | Description | +| :-------------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | +| certificate_idx | Option\ | | + +### `GenesisKeyDelegation` Event + +Data on genesis key delegation. + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Current slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | + +### `MoveInstantaneousRewardsCert` Event + +| Name | DataType | Description | +| :------------------- | :--------------------------------------- | :---------- | +| from_reserves | bool | .... | +| from_treasury | bool | .... | +| to_stake_credentials | Option\> | .... | +| to_other_pot | Option\ | .... | + +**Context** + +| Name | DataType | Description | +| :----------- | :-------------- | :---------------------------- | +| block_number | Option\ | Height of block from genesis. | +| block_hash | Option\ | Block hash. | +| slot | Option\ | Blockchain slot. | +| timestamp | Option\ | Timestamp. | +| tx_idx | Option\ | Transaction Index. | +| tx_hash | Option\ | Transaction hash. | diff --git a/docs/v2/sinks/aws_lambda.mdx b/docs/v2/sinks/aws_lambda.mdx index 6e824401..99bc12b0 100644 --- a/docs/v2/sinks/aws_lambda.mdx +++ b/docs/v2/sinks/aws_lambda.mdx @@ -1,38 +1,36 @@ --- title: AWS Lambda +sidebar: + order: 41 --- -A sink that invokes an AWS Lambda function for each received event. Each event is json-encoded and sent to a configurable function using AWS API endpoints. +Invokes an [AWS Lambda](https://aws.amazon.com/lambda/) function for each event, sending the +json-encoded payload through the AWS API. Use it to trigger custom serverless logic — alerts, +enrichment, downstream writes — whenever a matching event appears on chain. -The sink will process each incoming event in sequence, invoke the specified function and wait for the response. +The sink processes events in sequence: it invokes the function, waits for the response, then +moves on to the next event. -A retry mechanism is available for failures to dispatch the call, but not for failures within the execution of the function. Regardless of the success or not of the function, the sink will advance and continue with the following event. - -Authentication against AWS is built-in in the SDK library and follows the common chain of providers (env vars, ~/.aws, etc). +:::note +Authentication uses the standard AWS provider chain — no credentials go in the config. See +[AWS credentials](/oura/v2/sinks/#aws-credentials) for the supported options. +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "AwsLambda" region = "us-east-1" function_name = "my-lambda" ``` -### Section: `sink` - -- `type`: the literal value `AwsLambda`. -- `region`: The AWS region where the function is located. -- `function_name`: The ARN of the function we wish to invoke. - -## AWS Credentials - -The sink needs valid AWS credentials to interact with the cloud service. The majority of the SDKs and libraries that interact with AWS follow the same approach to access these credentials from a chain of possible providers: - -- Credentials stored as the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. -- A Web Identity Token credentials from the environment or container (including EKS) -- ECS credentials (IAM roles for tasks) -- As entries in the credentials file in the .aws directory in your home directory (~/.aws/) -- From the EC2 Instance Metadata Service (IAM Roles attached to an instance) +- `type` (required): the literal value `AwsLambda`. +- `region` (required): the AWS region where the function lives. +- `function_name` (required): the ARN of the function to invoke. -Oura, by mean of the Rust AWS SDK lib, will honor the above chain of providers. Use any of the above that fits your particular scenario. Please refer to AWS' documentation for more detail. +:::caution[Retries cover dispatch, not execution] +Oura retries failures to *dispatch* the call, but not failures *inside* the function. Whether +the function succeeds or not, the sink advances to the next event — so the function itself must +handle its own errors and idempotency. +::: diff --git a/docs/v2/sinks/aws_s3.mdx b/docs/v2/sinks/aws_s3.mdx index 3941f7b3..38328987 100644 --- a/docs/v2/sinks/aws_s3.mdx +++ b/docs/v2/sinks/aws_s3.mdx @@ -1,18 +1,21 @@ --- title: AWS S3 +sidebar: + order: 42 --- -A sink that saves the CBOR content of the blocks as S3 object. +Stores the raw CBOR of each block as an object in an [AWS S3](https://aws.amazon.com/s3/) +bucket. It's the go-to sink for archiving chain data: write blocks to a bucket once, then +replay them later with the [S3 source](/oura/v2/sources/s3) — no node required. -The sink will process each input event in sequence and according to the configuration in daemon.toml it will save in S3 the block in CBOR format. - -The location where the object will be saved can be configured by adding value in the prefix field, for example `mainnet/` - -Authentication against AWS is built-in in the SDK library and follows the common chain of providers (env vars, ~/.aws, etc). +:::note +Authentication uses the standard AWS provider chain — no credentials go in the config. See +[AWS credentials](/oura/v2/sinks/#aws-credentials) for the supported options. +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "AwsS3" region = "us-west-2" @@ -20,31 +23,15 @@ bucket = "my-bucket" prefix = "mainnet/" ``` -### Section: `sink` - -- `type`: the literal value `AwsS3`. -- `region`: The AWS region where the bucket is located. -- `bucket`: The name of the bucket to store the blocks. -- `prefix`: A prefix to prepend on each object's key. - -IMPORTANT: Only the cbor block format is supported. - -## Naming Convention - -The name of the object and the slot number in which it was processed. - -## Content Encoding - -- `application/cbor` - -## AWS Credentials - -The sink needs valid AWS credentials to interact with the cloud service. The majority of the SDKs and libraries that interact with AWS follow the same approach to access these credentials from a chain of possible providers: +- `type` (required): the literal value `AwsS3`. +- `region` (required): the AWS region where the bucket lives. +- `bucket` (required): the bucket to store blocks in. +- `prefix` (optional): a prefix prepended to each object's key, e.g. `mainnet/`. -- Credentials stored as the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. -- A Web Identity Token credentials from the environment or container (including EKS) -- ECS credentials (IAM roles for tasks) -- As entries in the credentials file in the .aws directory in your home directory (~/.aws/) -- From the EC2 Instance Metadata Service (IAM Roles attached to an instance) +Each object is keyed by the slot number of the block it contains (under `prefix`, if set) and +stored with content type `application/cbor`. -Oura, by mean of the Rust AWS SDK lib, will honor the above chain of providers. Use any of the above that fits your particular scenario. Please refer to AWS' documentation for more detail. +:::caution[CBOR blocks only] +This sink stores the raw CBOR block, so the pipeline must feed it CBOR blocks — don't place +filters like `ParseCbor` or `IntoJson` before it. +::: diff --git a/docs/v2/sinks/aws_sqs.mdx b/docs/v2/sinks/aws_sqs.mdx index 31453f9e..34f36cb7 100644 --- a/docs/v2/sinks/aws_sqs.mdx +++ b/docs/v2/sinks/aws_sqs.mdx @@ -1,18 +1,21 @@ --- title: AWS SQS +sidebar: + order: 40 --- -A sink that sends each event as a message to an AWS SQS queue. Each event is json-encoded and sent to a configurable SQS queue using AWS API endpoints. +Sends each event as a json-encoded message to an [AWS SQS](https://aws.amazon.com/sqs/) queue. +The sink submits a `SendMessage` request per event and waits for the queue to acknowledge it +before moving on. It supports both Standard and FIFO queues — see the trade-off below. -The sink will process each incoming event in sequence and submit the corresponding `SendMessage` request to the SQS API. Once the queue acknowledges reception of the message, the sink will advance and continue with the following event. - -The sink support both FIFO and Standard queues. The sink configuration will determine which logic to apply. In case of FIFO, it is necessary to enable `content-based deduplication` and the group id is determined by an explicit configuration value or `oura-sink` by default. - -Authentication against AWS is built-in in the SDK library and follows the common chain of providers (env vars, ~/.aws, etc). +:::note +Authentication uses the standard AWS provider chain — no credentials go in the config. See +[AWS credentials](/oura/v2/sinks/#aws-credentials) for the supported options. +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "AwsSqs" region = "us-west-2" @@ -20,35 +23,33 @@ queue_url = "https://sqs.us-west-2.amazonaws.com/xxxxxx/my-queue.fifo" group_id = "my_group" ``` -### Section: `sink` - -- `type`: the literal value `AwsSqs`. -- `region`: The AWS region where the queue is located. -- `queue_url`: The SQS queue URL provided by AWS (not to be confused with the ARN). -- `group_id`: A fixed group id to be used when sending messages to a FIFO queue. - -## AWS Credentials - -The sink needs valid AWS credentials to interact with the cloud service. The majority of the SDKs and libraries that interact with AWS follow the same approach to access these credentials from a chain of possible providers: - -- Credentials stored as the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. -- A Web Identity Token credentials from the environment or container (including EKS) -- ECS credentials (IAM roles for tasks) -- As entries in the credentials file in the .aws directory in your home directory (~/.aws/) -- From the EC2 Instance Metadata Service (IAM Roles attached to an instance) - -Oura, by mean of the Rust AWS SDK lib, will honor the above chain of providers. Use any of the above that fits your particular scenario. Please refer to AWS' documentation for more detail. - -## FIFO vs Standard Queues +- `type` (required): the literal value `AwsSqs`. +- `region` (required): the AWS region where the queue lives. +- `queue_url` (required): the queue URL provided by AWS (not the ARN). +- `group_id` (optional, default = `oura-sink`): the message group id used for FIFO queues. -Oura processes messages maintaining the sequence of the blocks and respecting the order of the transactions within the block. An AWS SQS FIFO queue provides the consumer with a guarantee that the order in which messages are consumed matches the order in which they were sent. +For a FIFO queue, also enable **content-based deduplication** on the queue itself. -Connecting Oura with a FIFO queue would provide the consumer with the guarantee that the events received follow the same order as they appeared in the blockchain. This might be useful, for example, in scenarios where the processing of an event requires a reference of a previous state of the chain. +## FIFO vs Standard queues -Please note that rollback events might happen upstream, at the blockchain level, which need to be handled by the consumer to unwind any side-effects of the processing of the newly orphaned blocks. This problem can be mitigated by using Oura's [rollback buffer](/oura/v2/filters/rollback_buffer) feature. +Oura processes events in chain order, preserving the sequence of blocks and the order of +transactions within each block. -If each event can be processed in isolation, if the process is idempotent or if the order doesn't affect the outcome, the recommended approach is to use a Standard queue which provides "at least once" processing guarantees, relaxing the constraints and improving the overall performance. +- A **FIFO** queue carries that ordering guarantee through to your consumer — the events come + out in the same order they appeared on chain. This matters when processing one event depends + on the state left by an earlier one. +- A **Standard** queue offers "at least once" delivery without strict ordering. If each event + can be processed in isolation, or your processing is idempotent, a Standard queue relaxes + the constraints and gives you better throughput. -## Payload Size Limitation +:::note +Rollbacks can still happen upstream at the chain level, and your consumer needs to unwind the +side-effects of orphaned blocks. The [rollback buffer](/oura/v2/filters/rollback_buffer) +filter mitigates this by only forwarding blocks once they're sufficiently deep. +::: -AWS SQS service has a 256kb payload size limit. This is more than enough for individual events, but it might be too little for pipelines where the `include_cbor_hex` option is enabled. If your goal of your pipeline is to access the raw CBOR content, we recommend taking a look at the [AWS S3 Sink](/oura/v2/sinks/aws_s3) that provides a direct way for storing CBOR block in an S3 bucket. +:::caution[256 KB payload limit] +SQS caps messages at 256 KB. That's plenty for individual events, but may be too small if your +pipeline enables `include_cbor_hex`. If you need the raw CBOR, use the +[AWS S3 sink](/oura/v2/sinks/aws_s3) instead, which stores CBOR blocks directly in a bucket. +::: diff --git a/docs/v2/sinks/elasticsearch.mdx b/docs/v2/sinks/elasticsearch.mdx index 739cefc1..0d13f4ba 100644 --- a/docs/v2/sinks/elasticsearch.mdx +++ b/docs/v2/sinks/elasticsearch.mdx @@ -1,12 +1,16 @@ --- title: Elasticsearch +sidebar: + order: 51 --- -A sink that outputs events into an Elasticsearch server. Each event is json-encoded and sent as a message to an index or data stream. +Indexes each event as a json document into an [Elasticsearch](https://www.elastic.co/) +index or data stream. Reach for it when you want chain data searchable and ready for Kibana +dashboards. ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "ElasticSearch" url = "https://localhost:9200" @@ -18,27 +22,32 @@ username = "oura123" password = "my very secret stuff" ``` -### Section: `sink` - -- `type`: the literal value `ElasticSearch`. -- `url`: the location of the Elasticsearch's API -- `index`: the name of the index (or data stream) to store the event documents -- `idempotency` (optional): flag that if enabled will force idempotent calls to ES (to avoid duplicates) +- `type` (required): the literal value `ElasticSearch`. +- `url` (required): the Elasticsearch API endpoint. +- `index` (required): the index (or data stream) to store documents in. +- `idempotency` (optional): when enabled, forces idempotent writes to avoid duplicates (see + below). ### Section: `sink.credentials` -This section configures the auth mechanism against Elasticsearch. You can remove the whole section from the configuration if you want to skip authentication altogether (maybe private cluster without auth?). - -We currently only implement _basic_ auth, other mechanisms will be implemented at some point. +Configures authentication against Elasticsearch. Omit the whole section to skip auth (e.g. a +private cluster without authentication). -- `type`: the mechanism to use for authentication, only `Basic` is currently implemented -- `username`: username of the user with access to Elasticsearch -- `password`: password of the user with access to Elasticsearch +- `type` (required): the auth mechanism. Only `Basic` is currently implemented. +- `username` (required): the Elasticsearch user. +- `password` (required): that user's password. ## Idempotency -In services and API calls, _idempotency_ refers to a property of the system where the execution of multiple "equivalent" requests have the same effect as a single request. In other words, "idempotent" calls can be triggered multiple times without problem. - -If Oura restarts without having a cursor or if the same block is processed for any reason, repeated events will present the same ID and Elasticsearch will reject them and Oura will continue with the following event. This mechanism provides a strong guarantee that our index won't contain duplicate data. - -If the flag is disabled, each document will be generated using a random ID, ensuring that it will be indexed regardless. +An *idempotent* write can run multiple times with the same effect as running it once. With +`idempotency` enabled, the sink derives each document's ID deterministically from the event, +so a block that gets reprocessed — for example after a restart without a +[cursor](/oura/v2/advanced/stateful_cursor) — produces the same IDs and Elasticsearch rejects +the duplicates. Oura then continues with the next event. This guarantees the index won't +accumulate duplicate data. + +:::tip +If your pipeline can't guarantee exactly-once delivery (no cursor, at-least-once source, etc.), +enable `idempotency`. With it disabled, each document gets a random ID and is always indexed — +so reprocessing a block creates duplicates. +::: diff --git a/docs/v2/sinks/file_rotate.mdx b/docs/v2/sinks/file_rotate.mdx index 5c610af4..db90ce25 100644 --- a/docs/v2/sinks/file_rotate.mdx +++ b/docs/v2/sinks/file_rotate.mdx @@ -1,14 +1,16 @@ --- title: File Rotate +sidebar: + order: 20 --- -A sink that saves events into the file system. Each event is json-encoded and appended to the of a text file. Files are rotated once they reach a certain size. Optionally, old files can be automatically compressed once they have rotated. +Saves events to the local file system as JSON, one event per line. Files are rotated once they +reach a size limit, and old files can be compressed automatically. Reach for it when you want +a durable, on-disk log of chain data without standing up a database or broker. ## Configuration -Example sink section config - -```toml +```toml title="daemon.toml" [sink] type = "FileRotate" output_path = "/var/oura/mainnet" @@ -18,11 +20,15 @@ max_total_files = 10 compress_files = true ``` -### Section: `sink` +- `type` (required): the literal value `FileRotate`. +- `output_path` (required): the path-like prefix for the output log files. +- `output_format` (optional): the serialization format. The only option at the moment is + `JSONL` (JSON, one object per line). +- `max_bytes_per_file` (optional): the size a file may reach before it's rotated. +- `max_total_files` (optional): how many files to keep before the oldest are deleted. +- `compress_files` (optional): when `true`, rotated files are gzip-compressed. -- `type`: the literal value `FileRotate`. -- `output_path`: the path-like prefix for the output log files -- `output_format` (optional): specified the type of syntax to use for the serialization of the events. Only available option at the moment is `JSONL` (json + line break) -- `max_bytes_per_file` (optional): the max amount of bytes to add in a file before rotating it -- `max_total_files` (optional): the max amount of files to keep in the file system before start deleting the old ones -- `compress_files` (optional): a boolean indicating if the rotated files should be compressed. +:::caution +On a busy network this sink can produce a lot of data. Set `max_total_files` (and ideally +`compress_files`) so the logs are bounded — otherwise they'll grow until they fill the disk. +::: diff --git a/docs/v2/sinks/gcp_cloudfunction.mdx b/docs/v2/sinks/gcp_cloudfunction.mdx index fd6883a0..96b9bc00 100644 --- a/docs/v2/sinks/gcp_cloudfunction.mdx +++ b/docs/v2/sinks/gcp_cloudfunction.mdx @@ -2,13 +2,22 @@ title: Google Cloud Functions sidebar: label: Gcp Functions + order: 44 --- -A sink that sends each event to a cloud function. Each event is json-encoded and sent as a POST request. +Sends each event as a json-encoded `POST` request to a +[Google Cloud Function](https://cloud.google.com/functions). Use it to trigger custom +serverless logic on GCP whenever a matching event appears on chain. + +:::note +When `authentication` is enabled, the sink uses the standard Google Cloud credential chain to +mint an identity token — no credentials go in the config. See +[Google Cloud credentials](/oura/v2/sinks/#google-cloud-credentials). +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "GcpCloudFunction" url = "https://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION_NAME" @@ -20,16 +29,9 @@ extra_header_1 = "abc" extra_header_2 = "123" ``` -### Section: `sink` - -- `type`: the literal value `GcpCloudFunction` -- `url`: Your function url -- `timeout` (optional): the timeout value for the HTTP response in milliseconds. Default value is `30000`. -- `authentication` (optional): a flag that, when set to `true`, attaches a GCP identity token as the 'Authorization' HTTP header (see _GCP Authentication_ below). Default value is `false`. -- `headers` (optional): key-value map of extra headers to pass in each HTTP call - -### GCP Authentication - -The GCP authentication process relies on the following conventions: - -- If the `GOOGLE_APPLICATION_CREDENTIALS` environmental variable is specified, the value will be used as the file path to retrieve the JSON file with the credentials. +- `type` (required): the literal value `GcpCloudFunction`. +- `url` (required): your function's URL. +- `timeout` (optional, default = `30000`): the HTTP response timeout, in milliseconds. +- `authentication` (optional, default = `false`): when `true`, attaches a GCP identity token + as the `Authorization` header. +- `headers` (optional): a key-value map of extra headers to send with each call. diff --git a/docs/v2/sinks/gcp_pubsub.mdx b/docs/v2/sinks/gcp_pubsub.mdx index 0e15be07..455fe31b 100644 --- a/docs/v2/sinks/gcp_pubsub.mdx +++ b/docs/v2/sinks/gcp_pubsub.mdx @@ -2,26 +2,24 @@ title: Google Cloud PubSub sidebar: label: Gcp PubSub + order: 43 --- -A sink that sends each event as a message to a PubSub topic. Each event is json-encoded and sent to a configurable PubSub topic. +Publishes each event as a json-encoded message to a [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) +topic. A good fit when your services on GCP consume events through Pub/Sub subscriptions. + +:::note +Authentication uses the standard Google Cloud credential chain — no credentials go in the +config. See [Google Cloud credentials](/oura/v2/sinks/#google-cloud-credentials). +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "GcpPubSub" topic = "test" ``` -### Section: `sink` - -- `type`: the literal value `GcpPubSub`. -- `topic`: the short name of the topic to send message to. - -### GCP Authentication - -The GCP authentication process relies on the following conventions: - -- If the `GOOGLE_APPLICATION_CREDENTIALS` environmental variable is specified, the value will be used as the file path to retrieve the JSON file with the credentials. -- If the server is running on GCP, the credentials will be retrieved from the metadata server. +- `type` (required): the literal value `GcpPubSub`. +- `topic` (required): the short name of the topic to publish to. diff --git a/docs/v2/sinks/index.mdx b/docs/v2/sinks/index.mdx new file mode 100644 index 00000000..c1d45d6f --- /dev/null +++ b/docs/v2/sinks/index.mdx @@ -0,0 +1,86 @@ +--- +title: Sinks +sidebar: + label: Overview + order: 0 +--- + +A **sink** is the last stage of a pipeline — it takes each processed event and delivers it +somewhere. Which sink you pick comes down to where your data needs to land: a terminal, a +file, a message broker, a cloud service, or a database. Most sinks json-encode each event and +send it on; the exceptions are noted below. + +## Which sink should I use? + +**Console** — quick, no dependencies. + +| Sink | What it does | +| :--- | :----------- | +| [Stdout](/oura/v2/sinks/stdout) | writes raw events to standard output, ready to pipe into other tools | +| [Terminal](/oura/v2/sinks/terminal) | prints a human-readable, colorized feed | + +**Files** + +| Sink | What it does | +| :--- | :----------- | +| [File Rotate](/oura/v2/sinks/file_rotate) | appends events to rotating, optionally compressed, files on disk | + +**Messaging & streaming** + +| Sink | What it does | +| :--- | :----------- | +| [Kafka](/oura/v2/sinks/kafka) | publishes to a Kafka topic _(custom build)_ | +| [Redis](/oura/v2/sinks/redis) | appends to a Redis Stream | +| [RabbitMQ](/oura/v2/sinks/rabbitmq) | publishes to a RabbitMQ exchange | +| [ZeroMQ](/oura/v2/sinks/zeromq) | pushes over a ZeroMQ socket _(custom build)_ | + +**Cloud** + +| Sink | What it does | +| :--- | :----------- | +| [AWS SQS](/oura/v2/sinks/aws_sqs) | enqueues a message per event (FIFO or Standard) | +| [AWS Lambda](/oura/v2/sinks/aws_lambda) | invokes a Lambda function per event | +| [AWS S3](/oura/v2/sinks/aws_s3) | stores raw CBOR blocks as S3 objects | +| [GCP Pub/Sub](/oura/v2/sinks/gcp_pubsub) | publishes to a Pub/Sub topic | +| [GCP Cloud Functions](/oura/v2/sinks/gcp_cloudfunction) | calls a Cloud Function per event | + +**Databases & search** + +| Sink | What it does | +| :--- | :----------- | +| [SQL Database](/oura/v2/sinks/sql_db) | runs a templated SQL statement per event (SQLite / Postgres / MySQL) | +| [Elasticsearch](/oura/v2/sinks/elasticsearch) | indexes events into an Elasticsearch index or data stream | + +:::note +Every sink above ships in the default binary except **Kafka** and **ZeroMQ**, which need a +[custom build](/oura/v2/installation/from_source). +::: + +## Cloud credentials + +The AWS and GCP sinks don't take credentials in their config. Instead they rely on the +standard credential discovery built into each cloud's SDK, so the same setup works whether +you run Oura on your laptop, in a container, or on a managed instance. + +### AWS credentials + +The AWS sinks ([SQS](/oura/v2/sinks/aws_sqs), [Lambda](/oura/v2/sinks/aws_lambda), +[S3](/oura/v2/sinks/aws_s3)) honor the standard AWS SDK provider chain. Oura uses the first +source that resolves: + +- the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables +- a Web Identity Token from the environment or container (including EKS) +- ECS container credentials (IAM roles for tasks) +- entries in `~/.aws/` (the shared credentials file) +- the EC2 Instance Metadata Service (IAM roles attached to an instance) + +Use whichever fits your deployment. See the [AWS credentials documentation](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) +for the full details. + +### Google Cloud credentials + +The GCP sinks ([Pub/Sub](/oura/v2/sinks/gcp_pubsub), +[Cloud Functions](/oura/v2/sinks/gcp_cloudfunction)) resolve credentials in this order: + +- if `GOOGLE_APPLICATION_CREDENTIALS` is set, the JSON key file at that path is used +- otherwise, when running on GCP, credentials are read from the metadata server diff --git a/docs/v2/sinks/kafka.mdx b/docs/v2/sinks/kafka.mdx index bbcdc500..f767bd20 100644 --- a/docs/v2/sinks/kafka.mdx +++ b/docs/v2/sinks/kafka.mdx @@ -1,12 +1,22 @@ --- title: Kafka +sidebar: + order: 30 --- -A sink that implements a _Kafka_ producer. Each event is json-encoded and sent as a message to a single Kafka topic. +Publishes each event as a json-encoded message to an [Apache Kafka](https://kafka.apache.org/) +topic. A solid choice when you're already running Kafka and want chain data flowing into your +existing stream-processing setup. For a full walkthrough, see the +[Cardano → Kafka guide](/oura/v2/guides/cardano_2_kafka). + +:::caution[Requires a custom build] +Kafka isn't in the default binary (its TLS backend pulls in OpenSSL). Build Oura with the +`kafka` feature — see [Install from source](/oura/v2/installation/from_source). +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "Kafka" brokers = ["kafka-broker-0:9092"] @@ -15,8 +25,15 @@ ack_timeout_secs = 5 paritioning = "Random" ``` -- `type`: the literal value `Kafka`. -- `brokers`: indicates the location of the _Kafka_ brokers within the network. Several hostname:port pairs can be added to the array for a "cluster" scenario. -- `topic` this field indicates which _Kafka_ topic to use to send the outbound messages. -- `ack_timeout_secs` (optional): the timeout, in seconds, to wait for the broker to acknowledge a produced message. -- `paritioning` (optional): the strategy used to assign each message to a partition. Available values are `ByBlock` (messages from the same block share a partition key) and `Random`. Default value is `Random`. (Note: the config key is spelled `paritioning` to match the field name in the codebase.) +- `type` (required): the literal value `Kafka`. +- `brokers` (required): the Kafka brokers to connect to. Add several `hostname:port` pairs for + a cluster. +- `topic` (required): the Kafka topic to publish messages to. +- `ack_timeout_secs` (optional): how long, in seconds, to wait for the broker to acknowledge a + message. +- `paritioning` (optional, default = `Random`): how messages are assigned to partitions — + `ByBlock` (messages from the same block share a partition key) or `Random`. + +:::note +The config key really is spelled `paritioning`, to match the field name in the codebase. +::: diff --git a/docs/v2/sinks/rabbitmq.mdx b/docs/v2/sinks/rabbitmq.mdx index 8b18c242..01b82adf 100644 --- a/docs/v2/sinks/rabbitmq.mdx +++ b/docs/v2/sinks/rabbitmq.mdx @@ -1,12 +1,16 @@ --- title: RabbitMq +sidebar: + order: 32 --- -A sink that implements a rabbitmq publisher. Each event is json-encoded and sent as a message to a rabbitmq exchange. +Publishes each event as a json-encoded message to a [RabbitMQ](https://www.rabbitmq.com/) +exchange. A good fit when your downstream consumers already speak AMQP and you want RabbitMQ +to handle routing and delivery. ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "Rabbitmq" uri = "amqp://rabbitmq:rabbitmq@127.0.0.1:5672" @@ -14,7 +18,7 @@ exchange = "events.exchange" routing_key = "" ``` -- `type`: the literal value `Rabbitmq`. -- `uri`: uri to connect on rabbitmq server. -- `exchange` field with the name of the exchange where the cardano event will be published. -- `routing_key` field with cardano event routing key configuration. +- `type` (required): the literal value `Rabbitmq`. +- `uri` (required): the connection URI of the RabbitMQ server. +- `exchange` (required): the exchange to publish events to. +- `routing_key` (optional): the routing key attached to each published message. diff --git a/docs/v2/sinks/redis.mdx b/docs/v2/sinks/redis.mdx index c340b390..78d32c28 100644 --- a/docs/v2/sinks/redis.mdx +++ b/docs/v2/sinks/redis.mdx @@ -2,19 +2,21 @@ title: Redis Streams sidebar: label: Redis + order: 31 --- -A sink that outputs events into _Redis Stream_. - -_Redis Streams_ works as an append-only log where multiple consumers can read from the same queue while keeping independent offsets (as opposed to a PubSub topic where one subscriber affect the other). You can learn more about the _Streams_ feature in the official [Redis Documentation](https://redis.io/docs/manual/data-types/streams). - -This sink will process incoming events and send a JSON-encoded message of the payload for each one using the `XADD` command. The Redis instance can be local or remote. +Appends each event to a [Redis Stream](https://redis.io/docs/manual/data-types/streams) using +the `XADD` command. A Redis Stream is an append-only log that multiple consumers can read from +at independent offsets (unlike a PubSub topic, where one subscriber doesn't affect another) — +handy when several services need to consume the same chain data at their own pace. The Redis +instance can be local or remote. ## Configuration -Example configuration that sends all events into a single stream named `mystream` of a Redis instance running in port 6379 of the localhost. +This example sends every event into a single stream named `mystream` on a Redis instance +running on `localhost:6379`: -```toml +```toml title="daemon.toml" [sink] type = "Redis" url = "redis://localhost:6379" @@ -22,20 +24,21 @@ stream_name = "mystream" stream_max_length = 100000 ``` -### Section: `sink` - -- `type`: the literal value `Redis`. -- `url`: the redis server in the format `redis://[][:]@[:port][/]` -- `stream_name` (optional): the name of the redis stream, default is `oura-sink` if not specified -- `stream_max_length` (optional): if set, caps the stream to this many entries (using `XADD ... MAXLEN`), discarding the oldest entries as new ones arrive. If omitted, the stream grows unbounded. +- `type` (required): the literal value `Redis`. +- `url` (required): the Redis server, as `redis://[][:]@[:port][/]`. +- `stream_name` (optional, default = `oura-sink`): the name of the stream to append to. +- `stream_max_length` (optional): caps the stream to this many entries (via `XADD … MAXLEN`), + discarding the oldest as new ones arrive. Omit it to let the stream grow unbounded. ## Conventions -It is possible to send all Event to a single stream or create multiple streams, one for each event type. By appling the [select](/oura/v2/filters/select) filter it is possible to define the streams which should be created. - -The sink uses the default Redis convention to define the unique entry ID for each message sent to the stream ( `-`). +You can send every event to a single stream, or split events into one stream per type by +adding a [Select](/oura/v2/filters/select) filter upstream to define which streams get +created. -Messages in Redis Streams are required to be `hashes` (maps between the string fields and the string values). This sink will serialize the event into a single-entry map with the following parts: +Each entry's ID uses the default Redis convention, `-`. +Because Redis Stream entries must be hashes (string-to-string maps), the sink serializes each +event into a single-entry map: - `key`: the event type name. - `value`: the json-encoded payload of the event. diff --git a/docs/v2/sinks/sql_db.mdx b/docs/v2/sinks/sql_db.mdx index b7504570..0225a720 100644 --- a/docs/v2/sinks/sql_db.mdx +++ b/docs/v2/sinks/sql_db.mdx @@ -2,21 +2,24 @@ title: SQL Database sidebar: label: SQL Database + order: 50 --- -A sink that executes a SQL statement against a relational database for each event. It is built on top of [sqlx](https://github.com/launchbadge/sqlx)'s `Any` driver, so it can target SQLite, PostgreSQL or MySQL depending on the connection string. +Runs a SQL statement against a relational database for each event. It's built on +[sqlx](https://github.com/launchbadge/sqlx)'s `Any` driver, so the same sink targets SQLite, +PostgreSQL, or MySQL depending on the connection string. Reach for it when you want chain data +landing directly in tables you control. -For every event, the sink renders a [Handlebars](https://handlebarsjs.com/) template into a SQL statement and executes it. There are three independent templates, one for each kind of chain event: +For every event, the sink renders a [Handlebars](https://handlebarsjs.com/) template into a +SQL statement and executes it. There's one template per kind of chain event: - `apply`: rendered when a block/tx is applied to the chain. - `undo`: rendered when a block/tx is rolled back. -- `reset`: rendered when the pipeline resets to a given point (no record is available, only the point). - -This sink requires Oura to be built with the `sql` feature. +- `reset`: rendered when the pipeline resets to a point (no record is available, only the point). ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "SqlDb" connection = "sqlite::memory:" @@ -25,17 +28,19 @@ undo_template = "DELETE FROM events WHERE slot = {{point.slot}}" reset_template = "DELETE FROM events WHERE slot > {{point.slot}}" ``` -### Section: `sink` - -- `type`: the literal value `SqlDb`. -- `connection`: the database connection string passed to sqlx (e.g. `sqlite::memory:`, `sqlite://./oura.db`, `postgres://user:pass@host/db`). -- `apply_template`: a Handlebars template rendered to a SQL statement for `apply` events. -- `undo_template`: a Handlebars template rendered to a SQL statement for `undo` events. -- `reset_template`: a Handlebars template rendered to a SQL statement for `reset` events. +- `type` (required): the literal value `SqlDb`. +- `connection` (required): the sqlx connection string (e.g. `sqlite::memory:`, + `sqlite://./oura.db`, `postgres://user:pass@host/db`). +- `apply_template` (required): the Handlebars template rendered for `apply` events. +- `undo_template` (required): the Handlebars template rendered for `undo` events. +- `reset_template` (required): the Handlebars template rendered for `reset` events. -## Template Data +## Template data -Each template is rendered with the following data context: +Each template is rendered with the following context: -- `point`: the chain point. For a specific point this is an object with `slot` (number) and `hash` (hex string). For the chain origin it is `null`. -- `record`: the event payload. For `reset` events this value is `null`. The shape for non-reset events depends on the record type produced by the preceding filters (e.g. raw CBOR hex, a parsed transaction, or a legacy v1 event). +- `point`: the chain point. For a specific point, an object with `slot` (number) and `hash` + (hex string); for the chain origin, `null`. +- `record`: the event payload. `null` for `reset` events. For other events its shape depends + on the filters before it (raw CBOR hex, a parsed transaction, a legacy v1 event, …) — see + the [Data Dictionary](/oura/v2/reference/data_dictionary). diff --git a/docs/v2/sinks/stdout.mdx b/docs/v2/sinks/stdout.mdx index dcc02240..c88eb9d4 100644 --- a/docs/v2/sinks/stdout.mdx +++ b/docs/v2/sinks/stdout.mdx @@ -1,14 +1,18 @@ --- title: Stdout +sidebar: + order: 10 --- -A sink that outputs each event data into the terminal through stdout. +Writes each event to standard output, one per line. It's the simplest sink — great for piping +Oura into other shell tools (`grep`, `jq`, a file, another process) or for a quick look at +what a pipeline produces. ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "Stdout" ``` -- `type`: the literal value `Stdout`. +- `type` (required): the literal value `Stdout`. diff --git a/docs/v2/sinks/terminal.mdx b/docs/v2/sinks/terminal.mdx index a054b3ee..ffc4de34 100644 --- a/docs/v2/sinks/terminal.mdx +++ b/docs/v2/sinks/terminal.mdx @@ -1,12 +1,16 @@ --- title: Terminal +sidebar: + order: 11 --- -A sink that outputs each event into the terminal through stdout using fancy coloring 💅. +Prints each event to the terminal with color-coded, human-readable formatting (the same look +as `oura watch`). Use it when you want to *watch* a pipeline live rather than feed events into +another program. ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "Terminal" throttle_min_span_millis = 500 @@ -14,6 +18,11 @@ wrap = true ``` - `type` (required): the literal value `Terminal`. -- `throttle_min_span_millis` (optional, default = `500`): the amount of time (milliseconds) to wait between printing each event into the console. This is used to facilitate the reading for human following the output. -- `wrap` (optional, default = `false`): a true value indicates that long output text should break and continue in the following line. If false, lines will be truncated to fit in the available terminal width. -- `adahandle_policy` (optional): the minting policy id (hex) of [ADA Handle](https://adahandle.com/). When set, the sink resolves and displays handles found in outputs for a friendlier rendering. If omitted, handle resolution is disabled. +- `throttle_min_span_millis` (optional, default = `500`): minimum delay, in milliseconds, + between printed events. Throttling keeps the output readable instead of dumping a whole + block's events at once. +- `wrap` (optional, default = `false`): when `true`, long lines wrap; otherwise they're + truncated to the terminal width. +- `adahandle_policy` (optional): the minting policy id (hex) of [ADA Handle](https://adahandle.com/). + When set, the sink resolves and displays handles found in outputs for a friendlier rendering. + Omit it to disable handle resolution. diff --git a/docs/v2/sinks/webhook.mdx b/docs/v2/sinks/webhook.mdx index e55c1cd9..5e00f7ee 100644 --- a/docs/v2/sinks/webhook.mdx +++ b/docs/v2/sinks/webhook.mdx @@ -1,14 +1,19 @@ --- title: Webhook +sidebar: + order: 34 --- -A sink that outputs each event as an HTTP call to a remote endpoint. Each event is json-encoded and sent as the body of a request using `POST` method. +Sends each event as an HTTP `POST` to a remote endpoint, with the json-encoded event as the +request body. A flexible, dependency-free way to push chain data into any service that can +accept an HTTP call. -The sink expect a 200 reponse code for each HTTP call. If found, the process will continue with the next message. If an error occurs (either at the tcp or http level), the sink will apply the corresponding retry logic as specified in the configuration. +The sink expects a `200` response for each call: on success it moves to the next event; on +failure (at the TCP or HTTP level) it applies the configured [retry policy](/oura/v2/advanced/retry_policy). ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "WebHook" url = "https://endpoint:5000/events" @@ -20,12 +25,10 @@ extra_header_1 = "abc" extra_header_2 = "123" ``` -### Section: `sink` - -- `type`: the literal value `WebHook`. -- `url`: url of your remote endpoint (needs to accept POST method) -- `authorization` (optional): value to add as the 'Authorization' HTTP header -- `headers` (optional): key-value map of extra headers to pass in each HTTP call -- `allow_invalid_certs` (optional): a flag to skip TLS cert validation (usually for self-signed certs). -- `timeout` (optional): the timeout value for the HTTP response in milliseconds. Default value is `30000`. - +- `type` (required): the literal value `WebHook`. +- `url` (required): your endpoint's URL (must accept `POST`). +- `authorization` (optional): a value to send as the `Authorization` header. +- `headers` (optional): a key-value map of extra headers to send with each call. +- `allow_invalid_certs` (optional): skip TLS certificate validation — handy for self-signed + certs. +- `timeout` (optional, default = `30000`): the HTTP response timeout, in milliseconds. diff --git a/docs/v2/sinks/zeromq.mdx b/docs/v2/sinks/zeromq.mdx index a2834b48..ad50fe7a 100644 --- a/docs/v2/sinks/zeromq.mdx +++ b/docs/v2/sinks/zeromq.mdx @@ -2,21 +2,27 @@ title: ZeroMQ sidebar: label: ZeroMQ + order: 33 --- -A sink that publishes chain events over a [ZeroMQ](https://zeromq.org/) socket. The sink opens a `PUSH` socket and connects it to the configured address; each event that carries a record is json-encoded and sent as a single message. Events without a record (such as resets) are skipped and not published. A downstream `PULL` socket can then consume the stream of events. +Publishes chain events over a [ZeroMQ](https://zeromq.org/) socket. The sink opens a `PUSH` +socket, connects it to the configured address, and sends each event that carries a record as a +single json-encoded message; events without a record (such as resets) are skipped. A +downstream `PULL` socket consumes the stream. Useful for lightweight, broker-less fan-out to +your own processes. -This sink requires Oura to be built with the `zeromq` feature. +:::caution[Requires a custom build] +ZeroMQ isn't in the default binary (it links a system `libzmq` at runtime). Build Oura with +the `zeromq` feature — see [Install from source](/oura/v2/installation/from_source). +::: ## Configuration -```toml +```toml title="daemon.toml" [sink] type = "Zeromq" url = "tcp://localhost:5555" ``` -### Section: `sink` - -- `type`: the literal value `Zeromq`. -- `url`: the ZeroMQ endpoint the `PUSH` socket connects to (e.g. `tcp://localhost:5555`). +- `type` (required): the literal value `Zeromq`. +- `url` (required): the endpoint the `PUSH` socket connects to (e.g. `tcp://localhost:5555`). diff --git a/docs/v2/sources/hydra.mdx b/docs/v2/sources/hydra.mdx index df9ce40b..b3b4b03e 100644 --- a/docs/v2/sources/hydra.mdx +++ b/docs/v2/sources/hydra.mdx @@ -2,24 +2,20 @@ title: Hydra sidebar: label: Hydra + order: 4 --- -The Hydra fetch blocks and receive blocks from hydra. - -This source requires building oura using the feature `hydra` enabled. +The Hydra source connects to a [Hydra](https://hydra.family/head-protocol/) head over a +**WebSocket** and streams its events into the pipeline. Use it to observe activity on a Hydra +Layer 2 head with the same tooling you'd use for the main chain. ## Configuration -The following snippet shows an example of how to set up a Hydra source: - -```toml +```toml title="daemon.toml" [source] type = "Hydra" ws_url = "ws://127.0.0.1:4001" ``` -### Section `source`: - -- `type`: this field must be set to the literal value `Hydra` -- `ws_url`: websocket hydra server - +- `type` (required): the literal value `Hydra`. +- `ws_url` (required): the WebSocket URL of the Hydra node's API. diff --git a/docs/v2/sources/index.mdx b/docs/v2/sources/index.mdx new file mode 100644 index 00000000..6b625169 --- /dev/null +++ b/docs/v2/sources/index.mdx @@ -0,0 +1,33 @@ +--- +title: Sources +sidebar: + label: Overview + order: 0 +--- + +A **source** is the first stage of every pipeline — it's where Oura reads chain data from +and turns it into a stream of events. Most pipelines connect straight to a Cardano node; the +rest cover more specialized cases like fast bootstrapping, archived blocks, or Layer 2. + +## Which source should I use? + +| Source | Connects to… | Reach for it when… | +| :----- | :----------- | :----------------- | +| [N2N](/oura/v2/sources/n2n) | a node over a TCP socket (a relay/peer) | you want to follow the chain without running your own node — the most common choice | +| [N2C](/oura/v2/sources/n2c) | a local node over a unix socket | you run the node yourself and want the lowest-overhead connection | +| [UtxoRPC](/oura/v2/sources/utxorpc) | a Dolos / Demeter gRPC endpoint | you'd rather consume a hosted data service than operate a node | +| [Hydra](/oura/v2/sources/hydra) | a Hydra head over a WebSocket | you're observing events on a Hydra Layer 2 head | +| [Mithril](/oura/v2/sources/mithril) | a Mithril aggregator snapshot | you need to bootstrap historical chain data quickly | +| [S3](/oura/v2/sources/s3) | block objects stored in an AWS S3 bucket | you're replaying blocks you previously archived | + +:::note +N2N and N2C are bundled in the default binary. Mithril needs a custom build — see the +note on its page. +::: + +## How sources fit together + +Whatever the source, it hands the rest of the pipeline a stream of `apply` / `undo` events +carrying raw CBOR blocks by default. [Filters](/oura/v2/filters) then decode and reshape that +data before a [sink](/oura/v2/sinks) delivers it. To control *where on the chain* a source +starts (and optionally stops), see [Intersect options](/oura/v2/advanced/intersect_options). diff --git a/docs/v2/sources/mithril.mdx b/docs/v2/sources/mithril.mdx index bf2eb25c..693f76e0 100644 --- a/docs/v2/sources/mithril.mdx +++ b/docs/v2/sources/mithril.mdx @@ -2,17 +2,22 @@ title: Mithril sidebar: label: Mithril + order: 5 --- -Connect to Mithril Server and download a snapshot of the chain. +The Mithril source downloads a certified snapshot of the chain from a +[Mithril](https://mithril.network/) aggregator and replays it into the pipeline. It's the +fastest way to bootstrap historical chain data — instead of syncing block by block from a +node, you fetch a verified snapshot in one go. -This source requires building oura using the feature `mithril` enabled. +:::caution[Requires a custom build] +Mithril isn't in the default binary (it pulls a heavy C-crypto toolchain). Build Oura with +the `mithril` feature — see [Install from source](/oura/v2/installation/from_source). +::: ## Configuration -The following snippet shows an example of how to set up a Mithril source: - -```toml +```toml title="daemon.toml" [source] type = "Mithril" aggregator = "https://aggregator.pre-release-preview.api.mithril.network/aggregator" @@ -21,11 +26,15 @@ snapshot_download_dir = "./snapshot" skip_validation = false ``` -### Section `source`: - -- `type`: this field must be set to the literal value `Mithril` -- `aggregator`: url to fetch data from the aggregator -- `genesis_key`: genesis verification key -- `snapshot_download_dir`: the directory to persist snapshot, must have read/write access to it -- `skip_validation`: validate the chain snapshot - +- `type` (required): the literal value `Mithril`. +- `aggregator` (required): the URL of the Mithril aggregator to fetch from. +- `genesis_key` (required): the genesis verification key used to validate the snapshot. +- `snapshot_download_dir` (required): a writable directory where the snapshot is stored. +- `skip_validation` (optional, default = `false`): set to `true` to skip verifying the + snapshot against the certificate chain — faster, but you lose Mithril's integrity guarantee. + +:::tip +A Mithril snapshot is a fast way to *catch up*, not to follow the tip. A common pattern is to +bootstrap from a snapshot and then keep following the chain from a node — see the +[`mithril` example](/oura/v2/examples) for a `Breadcrumbs` intersect that does exactly that. +::: diff --git a/docs/v2/sources/n2c.mdx b/docs/v2/sources/n2c.mdx index f3789ab6..057a3328 100644 --- a/docs/v2/sources/n2c.mdx +++ b/docs/v2/sources/n2c.mdx @@ -5,28 +5,26 @@ sidebar: order: 2 --- -The Node-to-Client (N2C) source uses Ouroboros mini-protocols to connect to a local Cardano node through a unix socket bearer and fetches block data using the ChainSync mini-protocol instantiated to "full blocks". +The Node-to-Client (N2C) source connects to a **local** Cardano node over a unix socket, +using the Ouroboros ChainSync mini-protocol in "full blocks" mode. Because it talks to the +node directly over a socket, it's the lowest-overhead option when you run the node yourself. ## Configuration -The following snippet shows an example of how to set up a typical N2C source: - -```toml +```toml title="daemon.toml" [source] type = "N2C" socket_path = "" ``` -### Section `source`: - -- `type`: this field must be set to the literal value `N2C`. -- `socket_path`: the location of the socket file. +- `type` (required): the literal value `N2C`. +- `socket_path` (required): the path to the node's unix socket file. ## Examples -Connecting to a local Cardano node through unix sockets: +Connecting to a local Cardano node over its unix socket: -```toml +```toml title="daemon.toml" [source] type = "N2C" socket_path = "/opt/cardano/cnode/sockets/node0.socket" diff --git a/docs/v2/sources/n2n.mdx b/docs/v2/sources/n2n.mdx index f77b93e7..9cb99ff5 100644 --- a/docs/v2/sources/n2n.mdx +++ b/docs/v2/sources/n2n.mdx @@ -5,28 +5,27 @@ sidebar: order: 1 --- -The Node-to-Node (N2N) source uses Ouroboros mini-protocols to connect to a local or remote Cardano node through a tcp socket bearer and fetches block data using the ChainSync mini-protocol instantiated to "headers only" and the BlockFetch mini-protocol for retrieval of the actual block payload. +The Node-to-Node (N2N) source connects to a Cardano node over a **TCP socket** — local or +remote — using the Ouroboros mini-protocols. It runs ChainSync in "headers only" mode and +fetches the actual block payloads with BlockFetch. Reach for N2N when you want to follow the +chain through a relay without running your own node. ## Configuration -The following snippet shows an example of how to set up a typical N2N source: - -```toml +```toml title="daemon.toml" [source] type = "N2N" peers = [""] ``` -### Section `source`: - -- `type`: this field must be set to the literal value `N2N` -- `peers`: the location of the tcp endpoint It must be specified as a string with hostname and port number. +- `type` (required): the literal value `N2N`. +- `peers` (required): the TCP endpoints to connect to, each as a `hostname:port` string. ## Examples -Connecting to a remote Cardano node through tcp sockets: +Connecting to a remote Cardano node over TCP: -```toml +```toml title="daemon.toml" [source] type = "N2N" peers = ["backbone.mainnet.cardanofoundation.org:3001"] @@ -34,8 +33,15 @@ peers = ["backbone.mainnet.cardanofoundation.org:3001"] ### Public relays -**Mainnet** `backbone.mainnet.cardanofoundation.org:3001` +If you don't run your own node, these public relays are a convenient starting point: -**Preprod** `preprod-node.world.dev.cardano.org:30000` +| Network | Relay | +| :------ | :---- | +| Mainnet | `backbone.mainnet.cardanofoundation.org:3001` | +| Preprod | `preprod-node.world.dev.cardano.org:30000` | +| Preview | `preview-node.world.dev.cardano.org:30002` | -**Preview** `preview-node.world.dev.cardano.org:30002` +:::tip +Connecting to anything other than mainnet, preprod, or preview? Tell Oura which network it is +with a [`[chain]` section](/oura/v2/advanced/custom_network). +::: diff --git a/docs/v2/sources/s3.mdx b/docs/v2/sources/s3.mdx index 4ffc8c04..d45e4acf 100644 --- a/docs/v2/sources/s3.mdx +++ b/docs/v2/sources/s3.mdx @@ -2,24 +2,29 @@ title: Aws S3 Bucket sidebar: label: S3 + order: 6 --- -Download block object from S3 storage. - -This source requires building oura using the feature `aws` enabled. +The S3 source reads block objects from an **AWS S3 bucket** and replays them through the +pipeline. It pairs naturally with the [AWS S3 sink](/oura/v2/sinks/aws_s3), which writes raw +CBOR blocks to a bucket — letting you archive chain data once and reprocess it later without +touching a node. ## Configuration -The following snippet shows an example of how to set up a S3 source: - -```toml +```toml title="daemon.toml" [source] type = "S3" bucket = "bucket-name" items_per_batch = 10 ``` -the envs below need to be set if oura is not running in the aws environment configured +- `type` (required): the literal value `S3`. +- `bucket` (required): the name of the bucket to read from. +- `items_per_batch` (optional): the maximum number of object keys fetched per request. + +Set the following environment variables unless Oura is already running in a configured AWS +environment: ```sh AWS_ACCESS_KEY_ID= @@ -27,9 +32,7 @@ AWS_SECRET_ACCESS_KEY= AWS_DEFAULT_REGION=us-west-2 ``` -### Section `source`: - -- `type`: this field must be set to the literal value `S3` -- `bucket`: bucket name -- `items_per_batch`: sets the maximum number of keys returned in the response - +:::note +Credentials follow the standard AWS provider chain — see +[AWS credentials](/oura/v2/sinks/#aws-credentials) for all the supported options. +::: diff --git a/docs/v2/sources/utxorpc.mdx b/docs/v2/sources/utxorpc.mdx index 51ef70b6..78120472 100644 --- a/docs/v2/sources/utxorpc.mdx +++ b/docs/v2/sources/utxorpc.mdx @@ -5,44 +5,38 @@ sidebar: order: 3 --- -The UtxoRPC(U5C) source uses gRPC to fetch blocks and receive blocks from a Dolos node. - -This source requires building oura using the feature `u5c` enabled. - -```sh -cargo build --release --features=u5c -``` +The UtxoRPC (U5C) source fetches blocks over **gRPC** from a [Dolos](https://github.com/txpipe/dolos) +node or a hosted endpoint like [Demeter](https://demeter.run/). It's a good fit when you'd +rather consume a managed data service than run and maintain a Cardano node yourself. ## Configuration -The following snippet shows an example of how to set up a typical UtxoRPC source: - -```toml +```toml title="daemon.toml" [source] type = "U5C" url = "https://" use_parsed_blocks = false # optional [source.metadata] # optional -"key": "value" +"key" = "value" ``` -### Section `source`: - -- `type`: this field must be set to the literal value `U5C` -- `url`: A string contains Dolos gRPC url -- `use_parsed_blocks`: An optional field, enable parsing blocks from bytes to struct directly in the source. -- `source.metadata`: An optional field, hash map that will be included in gRPC metadata connection. +- `type` (required): the literal value `U5C`. +- `url` (required): the gRPC URL of the Dolos/U5C endpoint. +- `use_parsed_blocks` (optional, default = `false`): parse blocks into structured records + directly in the source, instead of emitting raw CBOR. +- `[source.metadata]` (optional): a key-value map added to the gRPC connection metadata — + useful for API keys and similar headers. ## Examples -Connecting to a remote Dolos node in preprod through gRPC using Demeter: +Connecting to a hosted preprod endpoint on Demeter, authenticating with an API key: -```toml +```toml title="daemon.toml" [source] type = "U5C" url = "https://preprod.utxorpc-v0.demeter.run" -[source.metadata] # optional +[source.metadata] "dmtr-api-key" = "YOUR DEMETER API KEY" ``` diff --git a/docs/v2/troubleshooting.mdx b/docs/v2/troubleshooting.mdx new file mode 100644 index 00000000..8bca2916 --- /dev/null +++ b/docs/v2/troubleshooting.mdx @@ -0,0 +1,82 @@ +--- +title: Troubleshooting +sidebar: + order: 90 +--- + +Common symptoms and what usually causes them. Each entry links to the page with the full +details. + +## Oura runs but no events appear + +By default the `[intersect]` block starts at the chain **`Tip`**, so you only see *new* blocks +as the network produces them — if the tip is momentarily quiet, nothing prints. This is normal. + +To process existing history, start from a specific point or the chain origin instead: + +```toml title="daemon.toml" +[intersect] +type = "Origin" +``` + +See [Intersect options](/oura/v2/advanced/intersect_options) for `Point`, `Origin`, and +`Breadcrumbs`. + +## Can't connect to the node socket / "connection refused" + +For a local [`N2C`](/oura/v2/sources/n2c) source, check, in order: + +- The `socket_path` points at the node's actual socket file. +- The node is running and has finished opening the socket. +- The Oura process can read the socket — node sockets are often owned by another user, so a + permissions mismatch is the most common cause. +- The bearer matches the endpoint: a unix socket needs `--bearer unix` (or the `N2C` source), + while a `host:port` relay needs `--bearer tcp` (or the [`N2N`](/oura/v2/sources/n2n) source). + +## Immediate disconnect, or addresses look wrong + +Almost always a **network-magic mismatch** — Oura is configured for a different network than the +node it's talking to. Make sure `--magic` (in `watch`) or `[chain].type` (in `daemon.toml`) +matches the node's network (`mainnet`, `preprod`, `preview`, `testnet`). + +For a network that isn't one of the well-known ones, supply the parameters explicitly — see +[Custom networks](/oura/v2/advanced/custom_network). + +## The pipeline restarts from scratch every time + +Without a cursor, Oura starts over from its configured intersect point on every restart (the +default backend is `Memory`, which keeps nothing across restarts). Add a persistent +[`[cursor]`](/oura/v2/advanced/stateful_cursor) backed by a file or Redis so it resumes where it +left off: + +```toml title="daemon.toml" +[cursor] +type = "File" +path = "./cursor.json" +``` + +## Duplicate or orphaned records after a rollback or restart + +The chain rolls back, and a restart can replay recently-seen blocks. Pick the strategy that fits +your sink (see [How it works](/oura/v2/how_it_works) for the full picture): + +- React to `undo` events and reverse the matching `apply`. +- Add a [RollbackBuffer filter](/oura/v2/filters/rollback_buffer) so your sink rarely sees a + rollback at all. +- Make writes idempotent and pair them with a [cursor](/oura/v2/advanced/stateful_cursor) so a + restart resumes cleanly. + +## Build fails for `kafka`, or `libzmq` not found + +These integrations aren't in the default binary and need build prerequisites: + +- **`kafka`** builds a vendored OpenSSL — install a C compiler and Perl. +- **`zeromq`** links a system `libzmq` — install it via your OS package manager. + +See [Install from source](/oura/v2/installation/from_source) for the full feature/prerequisite +table. + +## Webhook fails with a TLS certificate error + +If your endpoint uses a self-signed certificate, the [WebHook sink](/oura/v2/sinks/webhook) +rejects it by default. Set `allow_invalid_certs = true` to skip certificate validation. diff --git a/docs/v2/usage/daemon.mdx b/docs/v2/usage/daemon.mdx index 6d57cb13..ebbc32d7 100644 --- a/docs/v2/usage/daemon.mdx +++ b/docs/v2/usage/daemon.mdx @@ -2,38 +2,47 @@ title: Daemon --- -_Oura's_ `daemon` mode processes data in the background, without any live output. This mode is used in scenarios where you need to continuously bridge blockchain data with other persistence mechanisms or to trigger an automated process in response to a particular event pattern. +import { Steps } from '@astrojs/starlight/components'; -## Start Daemon Mode +`daemon` mode runs Oura in the background with no live output. It's the mode you use in +production — to continuously bridge chain data into another system, or to trigger an automated +action whenever a particular event pattern shows up. -To start _Oura_ in _daemon mode_, use the following command: +If you're just getting started, the [Quick Start](/oura/v2/quickstart) walks through a minimal +daemon config end to end. -``` -oura daemon -``` +## Getting started -Available options: + -- `--config`: path of a custom toml configuration file to use. If not specified, configuration will be loaded from `/etc/oura/daemon.toml`. +1. **Write a config file** describing your pipeline — which source, filters, and sink to use + (see [Configuration](#configuration) below). -Example of starting daemon mode with default config file: +2. **Start the daemon**, pointing it at your config: -```sh -# config will be loaded from /etc/oura/daemon.toml -oura daemon -``` + ```sh + oura daemon --config my_config.toml + ``` -Example of starting daemon mode with a custom config file at `my_config.toml`: + With no `--config`, Oura loads `/etc/oura/daemon.toml` by default: -```sh -oura daemon --config my_config.toml -``` + ```sh + oura daemon + ``` + + + +:::tip +The [Examples](/oura/v2/examples) section has ready-to-run `daemon.toml` files for every +source, filter, and sink — a good starting point to copy and adapt. +::: ## Configuration -The configuration file needs to specify the source, intersect, filters and sink to use in a particular pipeline. The following toml represent the typical skeleton of an _Oura_ config file: +A config file describes one pipeline: a `source`, where to `intersect` the chain, a chain of +`filters`, and a `sink`. Here's the typical skeleton: -```toml +```toml title="daemon.toml" [source] type = "X" # the type of source to use @@ -42,7 +51,7 @@ foo = "abc" bar = "xyz" [intersect] -type = "W" # the type of source intersect chain +type = "W" # where to start reading the chain [[filters]] type = "Y" # the type of filter to use @@ -52,35 +61,29 @@ foo = "123" bar = "789" [sink] -# the type of sink to use -type = "Z" +type = "Z" # the type of sink to use # custom config fields for this sink type foo = "123" bar = "789" ``` -### The `source` section - -This section specifies the origin of the data. The special `type` field must always be present and containing a value matching any of the available built-in sources. The rest of the fields in the section will depend on the selected `type`. See the sources section for a list of available options and their corresponding config values. - -### The `intersect` section - -Advanced options for instructing Oura from which point in the chain to start reading from. You can read more in [intersect advanced](/oura/v2/advanced/intersect_options) - -### The `filters` section - -This section specifies a collection of filters that are applied in sequence to each event. The special `type` field must always be present and containing a value matching any of the available built-in filters. Notice that this section of the config is an array, allowing multiple filter sections per config file. See the filters section for a list of available options and their corresponding config values. - -### The `sink` section +Every section has a `type` field selecting one of the built-in components; the remaining fields +depend on the type you choose. -This section specifies the destination of the data. The special `type` field must always be present and containing a value matching any of the available built-in sinks. The rest of the fields in the section will depend on the selected `type`. See the sinks section for a list of available options. +- **`[source]`** — where the data comes from. See [Sources](/oura/v2/sources) for the options + and their fields. +- **`[intersect]`** — where on the chain to start reading. See + [Intersect options](/oura/v2/advanced/intersect_options). +- **`[[filters]]`** — a sequence of transformations applied to each event, in order. This is an + array, so you can list several. See [Filters](/oura/v2/filters). +- **`[sink]`** — where the data goes. See [Sinks](/oura/v2/sinks). -### Full Example +### Full example -Here's an example configuration file that uses a Node-to-Node source and output the events into a Kafka sink: +A pipeline reading from a Node-to-Node source and publishing to a Kafka topic: -```toml +```toml title="daemon.toml" [source] type = "N2N" peers = ["backbone.mainnet.cardanofoundation.org:3001"] diff --git a/docs/v2/usage/dump.mdx b/docs/v2/usage/dump.mdx index 2436f07f..766a3608 100644 --- a/docs/v2/usage/dump.mdx +++ b/docs/v2/usage/dump.mdx @@ -25,6 +25,13 @@ oura dump [OPTIONS] - `--since ,`: an option to specify from which point in the chain _Oura_ should start reading from. The point is referenced by passing the slot of the block followed by a comma and the hash of the block (`,`). If omitted, _Oura_ will start reading from the tail (tip) of the node. - `--output `: an option to specify an output file prefix for storing the log files. Logs are rotated, so a timestamp will be added as a suffix to the final filename. If omitted, data will be sent to stdout. +:::tip +`--magic` accepts `mainnet`, `preprod`, `preview`, `testnet`, or a raw numeric magic. For +`--since`, find a block's `,` in any chain explorer — or omit it to start from the +tip. See the [watch command](/oura/v2/usage/watch) for the same options with a live, colorized +view. +::: + ## Examples ### Dump Data From A Remote Relay Node into Stdout diff --git a/docs/v2/usage/library.mdx b/docs/v2/usage/library.mdx index 8d77fa69..891419ff 100644 --- a/docs/v2/usage/library.mdx +++ b/docs/v2/usage/library.mdx @@ -2,22 +2,27 @@ title: Library --- -Oura allows users to build custom pipeline stages using the Oura library in Rust. If none of the default filters or sinks meet your needs, you can implement your own stages in Rust to filter data based on your business logic and output it wherever you choose. +When the built-in filters and sinks don't fit your use-case, you can embed Oura as a Rust +library and build your own pipeline stages — filtering on your own business logic and sending +events wherever you need. Each Oura component (sources, filters, sinks) is self-contained, so +you can reuse the stock pieces and only write the parts that are unique to you. ## Installing -It's possible to install oura lib using crate or directly from github. +Add Oura to your project, either from crates.io: -Using crate -``` +```sh cargo add oura ``` -Using github add this dependence in `Cargo.toml` -``` +Or directly from GitHub, in your `Cargo.toml`: + +```toml title="Cargo.toml" oura = { git = "https://github.com/txpipe/oura.git" } ``` -### Example +## Example -Follow the example code and the dependencies used to build a custom filter and sink. [Example Code](https://github.com/txpipe/oura/tree/main/examples/lib). +The [`examples/lib`](https://github.com/txpipe/oura/tree/main/examples/lib) project shows a +complete custom pipeline — a custom filter plus a custom sink that persists events into SQLite +— along with the dependencies it uses. Its README covers the `sqlx-cli` setup. diff --git a/docs/v2/usage/watch.mdx b/docs/v2/usage/watch.mdx index 46f16ac7..ff8395b0 100644 --- a/docs/v2/usage/watch.mdx +++ b/docs/v2/usage/watch.mdx @@ -24,6 +24,12 @@ oura watch [OPTIONS] - `--throttle`: milliseconds to wait between output lines (for easier reading). - `--wrap`: indicates that long output text should break and continue in the following line. If omitted, lines will be truncated to fit in the available terminal width. +:::tip +`--magic` accepts `mainnet`, `preprod`, `preview`, `testnet`, or a raw numeric magic for a +custom network. For `--since`, the `,` of any block is shown by chain explorers +(e.g. [Cardanoscan](https://cardanoscan.io/)) — or just omit it to start from the tip. +::: + ## Examples ### Watch Live Data From A Remote Relay Node