Data Processor

Unlocking the Full Potential of On-Chain Data with Verifiable Compute

Herodotus Data Processor (HDP) is a powerful solution for defining extensive sets of on-chain data and running complex computations over them in a fully sound and proven environment using STARKs and Storage Proofs. It enables developers to delegate intensive computations on authenticated on-chain data to an off-chain service.

HDP consists of two primary components:

• Data

• Compute

Data

HDP provides seamless access to historical on-chain data from multiple blockchains through the Storage Proofs API. To ensure data correctness, HDP uses Storage Proofs. Before processing any on-chain data, such as an account balance, HDP verifies its authenticity through a Storage Proof. This approach guarantees that you access accurate information that cannot be manipulated.

Compute

After data authentication, HDP performs computations on the verified data. Developers can specify the data scope and the functions to apply. For example, you might want to calculate your average balance over the past two weeks or determine your profit and loss (P&L).

HDP allows developers to perform these computations on verified on-chain data with certainty.

Use Cases

Practical Examples

1. Calculating Time-Weighted Average Price (TWAP)

   TWAP smooths out short-term price fluctuations by averaging prices over a specified period. This results in a more stable and representative value of an asset pair, which is particularly valuable for financial applications like options pricing.

2. Verifying Average Balance

   To prove an account maintained an average balance of 1 ETH over 1000 blocks, you would set up a data lake to fetch the account balance for these blocks. Using the avg function, you can compute the average balance, verifying the account's balance consistency over time.

3. Identifying Balance Fluctuations

   To count how often an account's average balance drops below 50 ETH, you could use the count_if function. This helps in assessing the frequency of significant balance reductions, providing insights into account activity.

4. Ensuring Compliance with OFAC Sanctions

   Ensuring Compliance with Sanctions If you want to restrict access to your dApp to addresses that have never sent funds to a sanctioned address. HDP enables this capability efficiently in a trustless way due to the ability to analyze historical on-chain data.

5. Predicting User's Balance

   HDP can perform sophisticated computations, such as linear regression. Given a set of historical ETH balances for an address, you can predict future balances using linear regression within HDP.

Structured vs. Unstructured Data

Many use cases require specific data access patterns, such as retrieving all ETH or USDC balances of an account within a given timeframe. These structured data access patterns are commonly referred to as data lakes.

Structured data access patterns provide a clear and organized structure for the resulting data set. However, some scenarios require more complex approaches.

In certain use cases, you might need to define logic where, if the average balance exceeds a certain value, you then check the number and frequency of transactions for that address. This involves accessing various unpredictable data points from the blockchain, such as the account's nonce, all transactions associated with that nonce, their timestamps, and more.

These scenarios require handling unstructured data, which is inherently more complex and less predictable than structured data.

Modularized Compute

HDP encapsulates computations into modules. These modules can be either predefined or user-defined functions that perform calculations on authenticated data.

For common computations like calculating average balances, you can use a predefined data lake with appropriate parameters and a predefined module, such as the average function. This approach simplifies the process for standard computations.

However, developers may need to execute custom logic within HDP that doesn't fit into predefined functions like min, max, or average. Instead, it may involve custom conditions and specific data access patterns.

To address these needs, HDP allows developers to define their own modules. These user-defined modules can be Turing complete, meaning they can access any on-chain data and perform any required computations.

How HDP is Built

HDP is built on the robust foundation of the CairoVM and the underlying STARK proof system. These technologies provide the powerful backend that drives HDP.

However, using HDP doesn't require familiarity with Cairo. Currently, custom modules in HDP can be written in Cairo1, with future support planned for other languages like Solidity.

Using HDP

Developers can use HDP through a Solidity or Cairo SDK. The process is straightforward:

Specify which function to run Define the data to process Provide the context of the computation

This request is then processed, either synchronously or asynchronously, by the HDP operator. The result is delivered on-chain, allowing smart contracts to query it.

Efficiency Through Batching

HDP can batch multiple tasks together, significantly reducing proving and verification costs for all users.

Batching is crucial because, in Cairo, the efficiency of proving remains constant for 2^n clock cycles. For example, proving 6 clock cycles takes the same time as proving 8 clock cycles. Therefore, batching tasks maximizes efficiency.

Moreover, when batching, data between runs can be reused if there's a coincidence of needs. If multiple users request similar computations or access the same data, HDP can optimize the process by reusing data, further reducing costs and improving performance.

Accessible Data Types in HDP

Data Processor enables access to various types of on-chain data, making it a powerful tool for developers. Here's a breakdown of the data types you can access:

• Block Headers: These are summaries of each block on the blockchain, containing metadata such as the previous block hash, timestamp, and more.

• Account Balances: This is the amount of ETH held by an account at any given time.

• Account Nonces: A nonce is a counter that tracks the number of transactions sent from an account. It helps in preventing double-spending and replay attacks.

• Account Code: This refers to the smart contract code associated with an account. Smart contracts are self-executing contracts with the terms directly written into code.

• Account Storage: This is the storage space where smart contracts can save data on the blockchain.

• Transaction Receipts: These are records that contain the outcome of transactions, including logs of events and the status of the transaction (success or failure).

• Transactions: These are the individual operations sent from one account to another, including the transfer of funds or execution of smart contracts.

• Blobs: In this context, blobs are large, arbitrary chunks of data stored on the blockchain, often used in advanced applications such as rollups, they have been introduced with EIP4844.

• Beacon Chain State: This refers to the state of the Beacon Chain, which is part of the Ethereum Beacon chain and responsible for managing the proof-of-stake protocol.

Data Processor provides developers with the ability to access and utilize these diverse data types, enabling a wide range of applications and computations on the Ethereum blockchain.

HDP Modules

HDP Modules enable developers to define custom computations they want to perform on the provided data scope. Modules receive authenticated on-chain data as input, process it according to their defined logic, and produce a result that is verified using STARK proofs. HDP offers both predefined Modules(Aggregation Functions) for common computations and allows developers to create custom modules for specific, or more complex tasks. Currently, modules can only be written in Cairo1.

All HDP modules are identified by their corresponding code hash. A registry manages these modules, and when a developer uploads a module to the registry, it becomes callable on-chain. This system allows for easy utilization and sharing of modules within the HDP ecosystem.

Private HDP Modules

Additionally, HDP supports private modules, where only the prover, registry maintainer, or deployer knows the module's function. This feature enables use cases such as trustless yet non-gameable credit scoring, where computation details remain private while ensuring result integrity.

Data Processor Features

	Today	Future Releases
On-chain data access	✅	✅
Storage Proofs	✅	✅
Aggregating functions	✅	✅
Multi-chain deployment	✅	✅
Unrestricted access patterns	❌	✅
Parallelization	❌	✅
Caching	❌	✅
Composition	❌	✅
Ability to pull data from multiple chains at once	❌	✅
Custom modules	❌	✅
Private Modules	❌	✅