Skip to main content

Running an Archive Server

Please see The Data Waterfall for an overview of the different sources of archive data. This page is dedicated to operational details on running an Archive Server.

For anyone looking to reliably serve historical transactional data, it is recommended to run multiple Archive Servers, fed by multiple full nodes running the Archive Writer process.

Suggested configuration:

  • 2 Archive Servers running MongoDB + monad-indexer
  • 2 Archive Writer nodes, aka full node + monad-archiver
  • Many full nodes serving RPC requests, connected to both Archive Servers for historical transactional data
  • CPU: 16 cores
  • RAM: minimum 64GB; prefer >512GB for best performance
  • Storage: minimum recommended 16TB; preferred 32TB+
    • NVMe SSD
    • RAID 10
    • Minimum 10K IOPS sustained
  • Network: >1GbE, scale higher as needed when serving more RPC servers

Fresh installation​

These instructions assume an existing full node (for Archive Writer) and a new host (for Archive Server).

On your Archive Server:​

  1. Add the following section to your ~/.env

    # Note: systemd service files do not perform substitution, so you must re-type these
    # in the connection strings below
    DB_USER="<your db username>"
    DB_PWD="<your db pwd>"
    DB_VOLUME="~/archive-db-data"
    

    # Note: source and sink should generally be the same local MongoDB.
    # Indexer reads from:
    BLOCK_DATA_SOURCE="mongodb mongodb://<your db username>:<your db pwd>@0.0.0.0:27017 archive-db"
    # Indexer writes to:
    ARCHIVE_SINK="mongodb mongodb://<your db username>:<your db pwd>@0.0.0.0:27017 archive-db"
    OTEL_ENDPOINT="http://0.0.0.0:4317"

  2. Run mongodb. Instructions show how to run in Docker, but operators can run however they choose.

    archive-db:
      image: mongo:latest
      command: mongod --bind_ip 0.0.0.0
      networks:
        - host
      environment:
        MONGO_INITDB_ROOT_USERNAME: ${DB_USER}
        MONGO_INITDB_ROOT_PASSWORD: ${DB_PWD}
      ports:
        - "27017:27017"
      volumes:
        - ${DB_VOLUME}:/data/db
      logging:
        driver: journald
        options:
          tag: "mongo"

  3. Start the db

    # create directory from .env db volume
    source ~/.env
    sudo mkdir -p $DB_VOLUME
    sudo chown -R monad:monad $DB_VOLUME
    sudo chmod 700 $DB_VOLUME
    

    docker compose up archive-db -d

On your Archive Writer host:​

  1. Add the following section to your ~/.env

    ###############################################################
    ################### Vars that do not change ###################
    ###############################################################
    ## Where block data gets written
    ARCHIVE_SINK="mongodb mongodb://<your db username>:<your db pwd>@<your db hostname>:27017 archive-db"
    ## Change this if your ledger folder is in a different location
    BFT_BLOCK_PATH=~/monad-bft/ledger
    OTEL_ENDPOINT=http://0.0.0.0:4317
    ## This can be significantly higher during backfill
    MAX_CONCURRENT_BLOCKS=50
    

    

    ###############################################################
    ####### Backfill from Genesis (initial configuration) #########
    ###############################################################
    BLOCK_DATA_SOURCE="aws testnet-ltu-032-0 50"
    ## Alternatively you can use your other ArchiveDB if it has already been backfilled
    #BLOCK_DATA_SOURCE="mongodb mongodb://<your other db username>:<your other db pwd>@<your other db hostname>:27017 archive-db"
    FALLBACK_BLOCK_DATA_SOURCE="aws testnet-ltu-032-0 50"
    

    ###############################################################
    ###################### Normal Operation #######################
    ###############################################################
    ## Archiver checks triedb first for block data
    BLOCK_DATA_SOURCE="triedb /dev/triedb 5000"
    

    ### FALLBACK_BLOCK_DATA_SOURCE ###
    ## This is used whenever data is missing from BLOCK_DATA_SOURCE
    ## Normally used after state-sync
    FALLBACK_BLOCK_DATA_SOURCE="mongodb mongodb://<your other db username>:<your other db pwd>@<your other db hostname>:27017 archive-db"
    

    ## Alternatively you can use category-labs aws bucket
    ## Note: YOU pay for S3 egress costs if pulling from this bucket
    #FALLBACK_BLOCK_DATA_SOURCE="aws testnet-ltu-032-0 50"

  2. Smart backfilling:

    sudo systemctl start monad-archiver
    # Should show it running
    # Double check the arguments look correct based on the above ^
    systemctl status monad-archiver
    # Expect many:
    # > INFO: Successfully archived block block_num=X
    journalctl -u monad-archiver -o cat

  3. Once monad-archiver has caught up to the chain tip you will see:

    INFO: Nothing to process

  4. This means backfilling is done and you should

    1. Comment out BACKFILL section in your .env and uncomment Normal Operation
    2. Restart: systemctl restart monad-archiver
    3. Verify status and logs as above

On your ArchiveDB host​

  1. Start monad-indexer

    sudo systemctl start monad-indexer
    systemctl status monad-indexer
    # Expect many:
    # > INFO: Indexing block...
    # > INFO: Index spot-check successful
    journalctl -u monad-indexer -o cat

[Optional] Set up monad-archive-checker​

Typically only 1 checker is run per organization, but more can be run if desired

  1. Add the following systemd override to monad-archive-checker

    sudo systemctl edit monad-archive-checker
    [Service]
    ExecStart=
    ExecStart=/usr/local/bin/monad-archive-checker \
        # Example --bucket my-org-mainnet-checker
        # Note: storing checker state in local mongo or filesystem
        #       will be supported in a future release
        --bucket <S3 Bucket for storing checker state>
        checker
        # Example of comparing a local self-hosted mongo against a Category Labs aws bucket
        # --init-replicas "<aws mainnet-deu-009-0 50,mongodb mongodb://<username>:<pwd>@<db hostname>:27017 archive-db>,mongodb mongodb://<username>:<pwd>@<second db hostname>:27017 archive-db>"
        --init-replicas "<ArchiveDB 1>,<ArchiveDB 2>"

  2. Reload and Restart

    sudo systemctl daemon-reload
    sudo systemctl restart monad-archive-checker
    systemctl status monad-archive-checker
    

    # Check for errors
    journalctl -u monad-archive-checker -o cat -f