Blockchain Crawler with several types of Database as Backend

Crawlori is a powerful crawler developed by Functori. It allows to crawl Tezos blockchain in a very fast and modular way.

A strong and rich filter system allows you to crawl only the data that you want like monitoring an account or a smart contract, etc.

Moreover, a callback mechanism allows some customization of the data collected by the crawler. When an operation hits your filters, you can decide whatever you want to do with it (display it, add it to another database, etc.).

Quickstart

You'll first need to install one of the databases you want to use among:

Installation

Docker

TODO

Installation

OCaml switch

Follow the online instructions to install opam.

Initialize your local switch, using opam:

opam switch create <switch_name> 4.12.0

Library dependencies

Depending on the database backend you chose, you will need to setup some some configuration and install optional ocaml dependencies.

Postgresql

Before installing the OCaml binding of postgresql, you need to install postgresql itself by following the online documentation.

To use postgresql with ocaml, you will to setup a user and configure it (requiring superuser).

make pg-user

After the configuration, you can install PGOCaml, the OCaml binding of postgres, in addition to other project dependencies. This can be achieved with:

make pg-deps

LMDB

You only need to install the ocaml binding for lmdb, in addition to other project dependencies. This can be achieved with:

make lmdb-deps

CAQTI

You only need to install the caqti-lwt and the caqti driver of your choice. This can be achieved with:

make caqti-deps

Indexedb

You only need to install the ocaml binding for indexedDB, in addition to other project dependencies. This can be achieved with:

make idb-deps

Building/Installing the binaries and library

If everything went well, you should be able to build crawlori with:

make

and install it in the opam switch with:

make install

Setup the Configuration File

First, you need to setup a configuration file to crawl the Tezos blockchain. A minimal example is provided in config.json file.

{
  "nodes": <string list> ,    // required, list of Tezos node: for example [ "http://tz.functori.com" ]
  "start": <int>,             // optional, the start block from which you want to start crawling
  "db_kind": <string>,        // optional, the database backend you want to use ("pg", "lmdb" or "idb"), defaul value is "pg"
  "step_forward": <int>,      // optional, number of block chunks you want to crawl, default value is 100
  "sleep" : <float>,          // optional, sleep time for the head watcher
  "forward" : <int>,          // optional, the level until you want to crawl forward
  "confirmations" : <int>,    // optional, depth at which you want to acknowledge that a block is assured, default is 10
  "verbose" : <int>,          // optional, verbose level, default is 0
  "accounts" : <string list>, // optional, list of accounts you want to filter during the crawl, all operation non-related to these 
accounts will not be registered.
  "register" : <"block" | "operation"> list, // optional, specify if you want to register block operation, both or none, default is both
}

The only required field is the nodes. All other fields are optional with some default values.

How to Start

To start the binary, you can run the following command:

_bin/crawlori config.json

To use crawlori as a lib, you may want to have a look at file src/crawler/unix/main.ml. To get access to the crawler's data in this case, you should provide an implementation for the following hooks as follows:

Hooks.set_operation <operation_handler> ;
Hooks.set_block <block_handler> ;
Hooks.set_main <main_handler> ;

The first one is a callback for (relevant) operations encountered by the crawler, the second one is for blocks, and the last one is used to set/unset blocks that are considered to be in the main chain.

Build your Docker Image

You can build a docker image containing crawlori with:

make docker-build IMAGE=<crawlori-image-name>

Too keep the database in a docker first create a docker volume:

docker create volume <crawlori-volume-name>

And run the image with:

docker run -v </path/to/config.json>:$PWD/config.json -v <crawlori-volume-name>:$PWD/data -p <local-port>:5432 <crawlori-image-name>

Troubleshooting

If you encouter some issues with pgocaml, you may need to disable the sandbox mode of OPAM and reinstall it.

opam init --reinit --disable-sandboxing --bare
opam reinstall pgocaml pgocaml_ppx ez_pgocaml

License

Copyright © 2021, Functori contact@functori.com. Released under the MIT License.