Skip to main content

Execution Events and Websocket Setup

Summary​

The Execution Events and Websocket features were designed to work together to make Monad even faster for high volume applications. Execution Events is a low-level system, and WebSocket support is one specific usage of that system.

Execution Events​

Execution Events offers developers the highest performance option for listening to real-time data from the Monad blockchain.

  • Execution Events uses a shared memory communication system that requires additional setup, which is described here. This setup is not part of the default instructions; it's only needed if you run real-time data consumers that use the Execution Events feature.

  • The "shared memory" nature of the communication means that consumers of execution events must run directly on the same host as the Monad node, so they can observe real-time data in the host's RAM

  • Monad's RPC server can optionally use execution events for better performance, and to support certain features, namely, the eth_subscribe JSON-RPC call

note

Here is an overview of all real-time data offerings in Monad. Here is a tutorial on writing programs that consume execution events.

WebSockets​

In Monad's JSON-RPC server, WebSockets have two uses:

  • Creating a persistent connection to make JSON-RPC requests
  • The ability to call the eth_subscribe API, which will "push" new real-time data as it happens, so you do not need to poll for new events

WebSocket support must be explicitly enabled in the RPC server with command-line flag --ws-enabled. When --ws-enabled is passed, then the host must be configured to support execution events, otherwise RPC will exit with an error.

An user guide to WebSockets on Monad is here.

Requirements​

  • A running Monad full node (setup instructions)
  • A hugetlbfs filesystem mount
    • This can be set up using the hugeadm utility; see below for an example
  • Custom (aka "override") systemd unit files for both RPC and Execution
    • Examples are both below

Setup a hugetlbfs mount using hugeadm​

General prerequisites​

Install the required package:

sudo apt install libhugetlbfs-bin

Execution events SDK prerequisites​

There are two additional steps (one required, one optional) if you want to consume real-time data in your own software using the execution events SDK. If you only need to enable WebSocket support in the RPC server, you do not need these additional steps, and can skip this section.

The required SDK step is to install these packages:

sudo apt install libhugetlbfs-dev libhugetlbfs0 libzstd-dev

The optional step is to run a preview release of the node software, which records more kinds of events. This version has support for:

  • EVM call frame recording, similar to the debug_traceTransaction RPC call (event type TXN_CALL_FRAME)

  • Access and modification records for all storage read/written by a transaction; there are several event types involved: ACCOUNT_ACCESS_LIST_HEADER, ACCOUNT_ACCESS, and STORAGE_ACCESS

If you do not need these events (i.e., if you only need EVM logs and basic transaction information) then you do not need to run this preview release. If you do want these events, run:

sudo apt install monad=0.12.0-rc-exec-events

CLI onetime setup​

warning

This is for onetime testing and will NOT persist after a reboot

# NOTE: here we use `monad` but if you are running as a custom user, that should be set here
$ sudo hugeadm --create-user-mounts monad

Sample systemd unit file​

This makes the mounts persistent after a reboot.

File location:

/etc/systemd/system/events-hugepages-mounts.service

File contents (as viewed when using the override editor):

### Anything between here and the comment below will become the contents of the drop-in file


# NOTE: as mentioned above, you can change the `monad` user to your custom user (if needed)
[Unit]
Description=Create hugepage mounts for monad
After=local-fs.target


[Service]
Type=oneshot
ExecStart=/usr/bin/hugeadm --create-user-mounts monad
RemainAfterExit=yes


[Install]
WantedBy=multi-user.target


### Edits below this comment will be discarded

Configure the systemd overrides​

Important items​

systemd:

  • As a reminder, if you installed Monad via apt, the systemd unit files live in: /usr/lib/systemd/sytem
  • This means we need to create a systemd override
  • systemd overrides for ExecStart (and other additive settings) require two blocks. The first "clears" the original value and the second sets the new value.
  • You will need to do a systemctl daemon-reload after the changes

events + websockets:

  • RPC + websockets has a HARD dependency on Execution runnning with events

websockets specific:

  • You will need to open a port in your firewall
  • The default is 8081
  • If you want to use a custom port, the --ws-port <PORT> for RPC allows you to set the port of your choosing

Configure the Execution override for systemd​

This override enables the events sub-component for Execution

warning

If RPC + websockets is started without events being enabled for Execution, RPC will start and then crash

You can launch the override editor via:

sudo systemctl edit monad-execution

Which will make a new file at /etc/systemd/system/monad-execution.service.d/override.conf

File contents (as viewed when using the override editor):

### Anything between here and the comment below will become the contents of the drop-in file


# NOTE: BOTH ExecStarts are REQUIRED


[Service]
ExecStart=
ExecStart=/usr/local/bin/monad \
    --chain "$CHAIN" \
    --db /dev/triedb \
    --block_db /home/monad/monad-bft/ledger \
    --statesync /home/monad/monad-bft/statesync.sock \
    --exec-event-ring /var/lib/hugetlbfs/user/monad/pagesize-2MB/event-rings/monad-exec-events \
    --ro_sq_thread_cpu 5 \
    --sq_thread_cpu 6 \
    --log_level INFO


### Edits below this comment will be discarded

Do reload:

sudo systemctl daemon-reload

Restart Execution​

Adding this step here to ensure that Execution is restarted with the events enabled

sudo systemctl restart monad-execution

Configure the RPC override for systemd​

This override enables the websockets sub-component for RPC

You can launch the override editor via:

sudo systemctl edit monad-rpc

Which will make a new file at /etc/systemd/system/monad-rpc.service.d/override.conf

File contents:

# NOTE: BOTH ExecStarts are REQUIRED


[Service]
ExecStart=
ExecStart=/usr/local/bin/monad-rpc \
    --ipc-path /home/monad/monad-bft/mempool.sock \
    --triedb-path /dev/triedb \
    --otel-endpoint "http://0.0.0.0:4317" \
    --allow-unprotected-txs \
    --node-config /home/monad/monad-bft/config/node.toml \
    --exec-event-path /var/lib/hugetlbfs/user/monad/pagesize-2MB/event-rings/monad-exec-events \
    --ws-enabled

Restart RPC​

sudo systemctl restart monad-rpc

Checking the connectivity​

A quick way to check if WebSocket connectivity is working is to use a general purpose command-line tool that can act as WebSocket client, such as websocat. This is a powerful command-line "swiss army knife" tool, like nc or the original socat. It is not officially packaged for Debian/Ubuntu yet, but precompiled binaries can be downloaded or installed via cargo install websocat (it is a Rust program).

Here is an example of running it in verbose mode (-v), with the WebSocket service hosted on default port 8081:

$ websocat -v ws://localhost:8081
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::net_peer] Connected to TCP 127.0.0.1:8081
[INFO  websocat::ws_client_peer] Connected to ws
[INFO  websocat::ws_peer] Received WebSocket ping

To subscribe, type the subscription JSON RPC call for eth_subscribe into your terminal's stdin and press enter:

{ "id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newHeads"] }

Every half-second or so, you should see updates about new blocks.