tornadocontrib/ web3-proxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
194 Commits 1 Branch 0 Tags 12 MiB
Rust 97%
Shell 2.4%
Dockerfile 0.5%
Go to file
Bryan Stitt 5beb8b271a archive nodes 2022-05-27 16:49:55 +00:00
.cargo keep RUSTFLAGS in one place 2022-05-17 16:36:42 +00:00
.vscode minimal app served its purpose 2022-05-20 02:01:02 +00:00
bin helper scripts 2022-05-27 16:47:33 +00:00
config flatten cache code 2022-05-22 18:39:06 +00:00
linkedhashmap add upstream readme 2022-05-17 04:08:33 +00:00
redis-cell-client flatten cache code 2022-05-22 18:39:06 +00:00
redis-cell-server start adding redis-cell for rate limits 2022-05-21 20:40:22 +00:00
web3-proxy off by one 2022-05-27 16:34:42 +00:00
wrk move data files 2022-05-06 01:40:43 +00:00
.dockerignore start adding redis-cell for rate limits 2022-05-21 20:40:22 +00:00
.gitignore move data files 2022-05-06 01:40:43 +00:00
Cargo.lock use redis-cell instead of governor 2022-05-21 23:34:05 +00:00
Cargo.toml start adding redis-cell for rate limits 2022-05-21 20:40:22 +00:00
Dockerfile keep RUSTFLAGS in one place 2022-05-17 16:36:42 +00:00
LICENSE add LICENSE 2022-03-04 19:56:05 -08:00
README.md Update README.md 2022-05-19 19:50:22 -07:00
TODO.md start adding redis-cell for rate limits 2022-05-21 20:40:22 +00:00
docker-compose.common.yml flatten cache code 2022-05-22 18:39:06 +00:00
docker-compose.prod.yml archive nodes 2022-05-27 16:49:55 +00:00
docker-compose.yml flatten cache code 2022-05-22 18:39:06 +00:00

README.md

web3-proxy

Web3-proxy is a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers.

Signed transactions (eth_sendRawTransaction) are sent in parallel to the configured private RPCs (eden, ethermine, flashbots, etc.).

All other requests are sent to an RPC server on the latest block (alchemy, moralis, rivet, your own node, or one of many other providers). If multiple servers are in sync, they are prioritized by active_requests/soft_limit. Note that this means that the fastest server is most likely to serve requests and slow servers are unlikely to ever get any requests.

Each server has different limits to configure. The soft_limit is the number of parallel active requests where a server starts to slow down. The hard_limit is where a server starts giving rate limits or other errors.

$ cargo run --release -p web3-proxy -- --help

   Compiling web3-proxy v0.1.0 (/home/bryan/src/web3-proxy/web3-proxy)
    Finished release [optimized + debuginfo] target(s) in 17.69s
     Running `target/release/web3-proxy --help`
Usage: web3-proxy [--port <port>] [--workers <workers>] [--config <config>]

Web3-proxy is a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers.

Options:
  --port            what port the proxy should listen on
  --workers         number of worker threads
  --config          path to a toml of rpc servers
  --help            display usage information

Start the server with the defaults (listen on http://localhost:8544 and use ./config/example.toml which proxies to a local websocket on 8546 and ankr's public ETH node):

cargo run --release -p web3-proxy

Check that the proxy is working:

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"web3_clientVersion","id":1}' 127.0.0.1:8544

You can copy config/example.toml to config/production-$CHAINNAME.toml and then run docker-compose up --build -d start a proxies for many chains.

Flame Graphs

Flame graphs make finding slow code painless:

$ cat /proc/sys/kernel/kptr_restrict
1
$ echo 0 | sudo tee /proc/sys/kernel/kptr_restrict
0
$ CARGO_PROFILE_RELEASE_DEBUG=true cargo flamegraph

GDB

Run the proxy under gdb for advanced debugging:

cargo build --release && RUST_LOG=web3_proxy=debug rust-gdb --args target/debug/web3-proxy --listen-port 7503 --rpc-config-path ./config/production-eth.toml

Load Testing

Test the proxy:

wrk -s ./data/wrk/getBlockNumber.lua -t12 -c400 -d30s --latency http://127.0.0.1:8544
wrk -s ./data/wrk/getLatestBlockByNumber.lua -t12 -c400 -d30s --latency http://127.0.0.1:8544

Test geth:

wrk -s ./data/wrk/getBlockNumber.lua -t12 -c400 -d30s --latency http://127.0.0.1:8545
wrk -s ./data/wrk/getLatestBlockByNumber.lua -t12 -c400 -d30s --latency http://127.0.0.1:8545

Test erigon:

wrk -s ./data/wrk/getBlockNumber.lua -t12 -c400 -d30s --latency http://127.0.0.1:8945
wrk -s ./data/wrk/getLatestBlockByNumber.lua -t12 -c400 -d30s --latency http://127.0.0.1:8945

Note: Testing with getLatestBlockByNumber.lua is not great because the latest block changes and so one run is likely to be very different than another.