web3-proxy/TODO.md

Todo

MVP

These are roughly in order of completition

• simple proxy
• better locking. when lots of requests come in, we seem to be in the way of block updates
• load balance between multiple RPC servers
• support more than just ETH
• option to disable private rpc and send everything to primary
• support websocket clients
  • we support websockets for the backends already, but we need them for the frontend too
• health check nodes by block height
• Dockerfile
• docker-compose.yml
• after connecting to a server, check that it gives the expected chainId
• the ethermine rpc is usually fastest. but its in the private tier. since we only allow synced rpcs, we are going to not have an rpc a lot of the time
• if not backends. return a 502 instead of delaying?
• move from warp to axum
• handle websocket disconnect and reconnect
• eth_sendRawTransaction should return the most common result, not the first
• use redis and redis-cell for rate limits
• it works for a few seconds and then gets stuck on something.
  • its working with one backend node, but multiple breaks. something to do with pending transactions
  • dashmap entry api is easy to deadlock! be careful with it!
• the web3proxyapp object gets cloned for every call. why do we need any arcs inside that? shouldn't they be able to connect to the app's? can we just use static lifetimes
• refactor Connection::spawn. have it return a handle to the spawned future of it running with block and transaction subscriptions
• refactor Connections::spawn. have it return a handle that is selecting on those handles?
• some production configs are occassionally stuck waiting at 100% cpu
  • they stop processing new blocks. i'm guessing 2 blocks arrive at the same time, but i thought our locks would handle that
  • even after removing a bunch of the locks, the deadlock still happens. i can't reliably reproduce. i just let it run for awhile and it happens.
  • running gdb shows the thread at tokio tungstenite thread is spinning near 100% cpu and none of the rest of the program is proceeding
  • fixed by https://github.com/gakonst/ethers-rs/pull/1287
• when sending with private relays, brownie's tx.wait can think the transaction was dropped. smarter retry on eth_getTransactionByHash and eth_getTransactionReceipt (maybe only if we sent the transaction ourselves)
• if web3 proxy gets an http error back, retry another node
• endpoint for health checks. if no synced servers, give a 502 error
• rpc errors propagate too far. one subscription failing ends the app. isolate the providers more (might already be fixed)
• incoming rate limiting (by ip)
• connection pool for redis
• automatically route to archive server when necessary
  • originally, no processing was done to params; they were just serde_json::RawValue. this is probably fastest, but we need to look for "latest" and count elements, so we have to use serde_json::Value
  • when getting the next server, filtering on "archive" isn't going to work well. need to check inner instead
• if the requested block is ahead of the best block, return without querying any backend servers
• http servers should check block at the very start
• subscription id should be per connection, not global
• when under load, i'm seeing "http interval lagging!". sometimes it happens when not loaded.
  • we were skipping our delay interval when block hash wasn't changed. so if a block was ever slow, the http provider would get the same hash twice and then would try eth_getBlockByNumber a ton of times
• inspect any jsonrpc errors. if its something like "header not found" or "block with id $x not found" retry on another node (and add a negative score to that server)
  • this error seems to happen when we use load balanced backend rpcs like pokt and ankr
• RESPONSE_CACHE_CAP in bytes instead of number of entries
• if we don't cache errors, then in-flight request caching is going to bottleneck
  • i think now that we retry header not found and similar, caching errors should be fine
• RESPONSE_CACHE_CAP from config
• web3_sha3 rpc command
• test that launches anvil and connects the proxy to it and does some basic queries
  • need to have some sort of shutdown signaling. doesn't need to be graceful at this point, but should be eventually
• if the fastest server has hit rate limits, we won't be able to serve any traffic until another server is synced.
  • thundering herd problem if we only allow a lag of 0 blocks
  • we can improve this by only publishing the synced connections once a threshold of total available soft and hard limits is passed. how can we do this without hammering redis? at least its only once per block per server
  • instead of tracking pending_synced_connections, have a mapping of where all connections are individually. then each change, re-check for consensus.
• synced connections swap threshold set to 1 so that it always serves something
• cli tool for creating new users
• incoming rate limiting by api key
• sort forked blocks by total difficulty like geth does
• refactor result type on active handlers to use a cleaner success/error so we can use the try operator
• give users different rate limits looked up from the database
• Add a "weight" key to the servers. Sort on that after block. keep most requests local
• cache db query results for user data. db is a big bottleneck right now
• allow blocking public requests
• Got warning: "WARN subscribe_new_heads:send_block: web3_proxy::connection: unable to get block from https://rpc.ethermine.org: Deserialization Error: expected value at line 1 column 1. Response: error code: 1015". this is cloudflare rate limiting on fetching a block, but this is a private rpc. why is there a block subscription?
• im seeing ethspam occasionally try to query a future block. something must be setting the head block too early
  • we were sorting best block the wrong direction. i flipped a.cmp(b) to b.cmp(a) so that the largest would be first, but then i used 'max_by' which looks at the end of the list
• HTTP GET to the websocket endpoints should redirect instead of giving an ugly error
• load the redirected page from config
• prettier output for create_user command. need the key in hex
• drop redis-cell in favor of a simpler (and faster) implementation.
  • redis-cell was giving me weird errors and it isn't worth debugging it right now.
• create user script should allow setting the api key
• disable redis persistence in dev
• attach a request id to every web request
• attach user id (not IP!) to each request
• fantom_1 | 2022-08-10T22:19:43.522465Z WARN web3_proxy::jsonrpc: forwarding error err=missing field jsonrpc at line 1 column 60
  • i think the server isn't following the spec. we need a context attached to more errors so we know which one
  • make jsonrpc default to "2.0" (including the custom deserializer that handles the RawValues)
• if the eth_call (or similar) params include a block, we can cache for that
• when block subscribers receive blocks, store them in a block_map
• eth_blockNumber without a backend request
• if we send a transaction to private rpcs and then people query it on public rpcs things, some interfaces might think the transaction is dropped (i saw this happen in a brownie script of mine). how should we handle this?
  • send getTransaction rpc requests to the private rpc tier
• I'm hitting infura rate limits very quickly. I feel like that means something is very inefficient
  • whenever blocks were slow, we started checking as fast as possible
• create user script should allow setting requests per minute
• cache api keys that are not in the database
• improve consensus block selection. Our goal is to find the highest work chain with a block over a minimum threshold of sum_soft_limit.
  • i saw a fork of like 300 blocks. probably just because a node was restarted and had fallen behind. need some checks to ignore things that are far behind. this improvement should fix this problem
  • A new block arrives at a connection.
  • It checks that it isn't the same that it already has (which is a problem with polling nodes)
  • If its new to this node...
    • if the block does not have total work, check our cache. otherwise, query the node
    • save the block num and hash so that http polling doesn't send duplicates
    • send the deduped block through a channel to be handled by the connections grouping.
  • The connections group...
    • input = rpc, new_block
    • adds the block and rpc to it's internal maps
      • connection_heads: HashMap<rpc_name, blockhash>
      • block_map: DashMap<blockhash, Arc>
      • block_num: DashMap<U64, H256>
      • blockchain: DiGraphMap<blockhash, ?>
    • iterate the rpc_map to find the highest_work_block
    • update synced connections
    • send the block through new head_block_sender
  • rewrite cannonical_block to work as long as there are no forks
  • rewrite cannonical_block (again) and related functions to handle forks
    • got a very large number of possible heads here. i think maybe a server was very far out of sync. we should drop servers behind by too much eth_1 | 2022-08-10T23:26:06.377129Z WARN web3_proxy::connections: chain is forked! 261 possible heads. 1/2/5/5 rpcs have 0xd403…3c5d eth_1 | 2022-08-10T23:26:08.917603Z WARN web3_proxy::connections: chain is forked! 262 possible heads. 1/2/5/5 rpcs have 0x0538…bfff eth_1 | 2022-08-10T23:26:10.195014Z WARN web3_proxy::connections: chain is forked! 262 possible heads. 1/2/5/5 rpcs have 0x0538…bfff eth_1 | 2022-08-10T23:26:10.195658Z WARN web3_proxy::connections: chain is forked! 262 possible heads. 2/3/5/5 rpcs have 0x0538…bfff
    • todo!("handle equal") and also less and greater
    • "chain is forked" message is wrong. it includes nodes just being on different heights of the same chain. need a smarter check
      • i think there is also a bug because i've seen "server not synced" a couple times
• bug around eth_getBlockByHash sometimes causes tokio to lock up
  • i keep a mapping of blocks so that i can go from hash -> block. it has some consistent hashing it does to split them up across multiple maps each with their own lock. so a lot of the time reads dont block writes because they are in different internal maps. this was fine. but after changing my fork detection logic to use the same rules as erigon, i discovered that when you get blocks from a websocket subscription in erigon and geth, theres a missing field (https://github.com/ledgerwatch/erigon/issues/5190). so i added a query to get the block that includes the missing field.
  • but i did this in a way where i was holding the write lock open while doing the query. the "new" block that has the missing field ends up in the same bucket and it also wants a write lock. oops. entry api has very sharp edges. don't ever await inside a match on DashMap::entry
• requests for "Get transactions receipts" are routed to the private_rpcs and not the balanced_rpcs. do this better.
  • quick fix, send to balanced_rpcs for now. we will just live with errors on new transactions.
  • this was intentional so that recently confirmed transactions go to a server that is more likely to have the tx.
  • but under heavy load, we hit their rate limits. need a "retry_until_success" function that goes to balanced_rpcs. or maybe store in redis the txids that we broadcast privately and use that to route.
• some of the DashMaps grow unbounded! Make/find a "SizedDashMap" that cleans up old rows with some garbage collection task
  • moka has all the features that we need and more
• if block data limit is 0, say Unknown in Debug output
• basic request method stats (using the user_id and other fields that are in the tracing frame)
• refactor from_anyhow_error to have consistent error codes and http codes. maybe implement the Error trait
• improve rpc weights. i think theres still a potential thundering herd
• improved logging with useful instrumentation
• right now the block_map is unbounded. move this to redis and do some calculations to be sure about RAM usage
• synced connections swap threshold should come from config
• right now we send too many getTransaction queries to the private rpc tier and i are being rate limited by some of them. change to be serial and weight by hard/soft limit.
• ip blocking gives a 500 and not the proper error code
• need a reconnect that doesn't unwrap
• need a retrying_reconnect that is used everywhere reconnect is. have exponential backoff here
• it looks like our reconnect logic is not always firing. we need to make reconnect more robust!
  • i am pretty sure that this is actually servers that fail to connect on initial setup (maybe the rpcs that are on the wrong chain are just timing out and they aren't set to reconnect?)
• chain rolled back 1/1/1 con_head=15510065 (0xa4a3…d2d8) rpc_head=15510065 (0xa4a3…d2d8) rpc=local_erigon_archive
  • include the old head number and block in the log
• exponential backoff when reconnecting a connection
• once the merge happens, we don't want to use total difficulty and instead just care about the number
• rewrite rate limiting to have a tiered cache. do not put redis in the hot path
  • instead, we should check a local cache for the current rate limit (+1) and spawn an update to the local cache from redis in the background.
  • when there are a LOT of concurrent requests, we see errors. i thought that was a problem with redis cell, but it happens with my simpler rate limit. now i think the problem is actually with bb8
  • https://docs.rs/redis/latest/redis/aio/struct.ConnectionManager.html or https://crates.io/crates/deadpool-redis?
  • WARN http_request: redis_rate_limit::errors: redis error err=Response was of incompatible type: "Response type not string compatible." (response was int(500237)) id=01GC6514JWN5PS1NCWJCGJTC94 method=POST
• web3_proxy_error_count{path = "backend_rpc/request"} is inflated by a bunch of reverts. do not log reverts as warn.
  • erigon gives method=eth_call reqid=986147 t=1.151551ms err="execution reverted"
• database migration to change user_keys.requests_per_minute to bigunsigned (max of 18446744073709551615)
• change user creation script to have a "unlimited requests per minute" flag that sets it to u64::MAX (18446744073709551615)
• in /status, block hashes has a lower count than block numbers. how is that possible?
  • we weren't calling sync. now we are
• opt-in debug mode that inspects responses for reverts and saves the request to the database for the user.
• Api keys need option to lock to IP, cors header, referer, user agent, etc
• /user/logout to clear bearer token and jwt
• bearer tokens should expire
• login endpoint needs its own rate limiter
  • we don't want an rpc request limit of 0 to block logins
  • for security, we want these limits low.
• user login should return the bearer token and the user keys
• use siwe messages and signatures for sign up and login
• check for bearer token on /rpc
• ip blocking logs a warn. we don't need that
• Ulid instead of Uuid for user keys
  • https://discord.com/channels/873880840487206962/900758376164757555/1012942974608474142
  • since users are actively using our service, we will need to support both
• get to /, when not serving a websocket, should have a simple welcome page. maybe with a button to update your wallet.
• instead of giving a rate limit error code, delay the connection's response at the start. reject if incoming requests is super high?
  • did this by checking a key/ip-specific semaphore before checking rate limits
• emit user stat on cache hit
• emit user stat on cache miss
• have migration use tokio instead of async-std
• user create script should allow a description field
• change stats to using the database
• emit user stat on retry
• improve web3_proxy_cli check_config
  • print out warnings if important settings are missing
• if unknown config items, error
  • unknown configs are almost always a mistake. usually from me changing config parsing on my side and old fields not being updated to the new way
  • also need to change how we disable rpcs since i was using an unknown field
• paginate responses
• graceful shutdown. stop taking new requests and don't stop until all outstanding queries are handled
  • https://github.com/tokio-rs/mini-redis/blob/master/src/shutdown.rs
  • we need this because we need to be sure all the queries are saved in the db. maybe put stuff in Drop
  • need an flume::watch on unflushed stats that we can subscribe to. wait for it to flip to true
• don't use unix timestamps for response_millis since leap seconds will confuse it
• config to allow origins even on the anonymous endpoints
• send logs to sentry
• login should return the user id
• when we show keys, also show the key's id
• add config for concurrent requests from public requests
• new endpoints for users (not totally sure about the exact paths, but these features are all needed):
  • sign in
    • login should include the key id, not just the key ULID
  • sign out
  • GET profile endpoint
  • POST profile endpoint
  • GET stats endpoint
    • display distribution of methods per api key (eth_call, eth_getLogs, etc.) (only with authentication!)
  • get aggregate stats endpoint
    • display requests per second per api key (only with authentication!)
  • POST key endpoint
    • generate a new key from a web endpoint
    • modifying key settings such as private relay, revert logging, ip/origin/etc checks
  • GET logged reverts on an endpoint that requires authentication.
• endpoint to list keys without having to sign a message to log in again
• rename user_key to rpc_key
  • in code
  • in database with a migration
• instead of requests_per_minute on every key, have a "user_tier" that gets joined
• document url params with examples
• improve "docs/http routes.txt"
• remove request per minute and concurrency limits from the keys. those are on the user tiers now.
• revertLogs db table should have rpc_key_id on it
• the relation in Relation is wrong now. it is called user_key_id, but point to the rpc key table
• instruments are missing. maybe that is why sentry had broken traces
• description should default to an empty string instead of being nullable
• include if archive query or not in the stats
• fix test not shutting down
• proper authentication on rpc_key_id
  • we have bearer token auth for user_id, but rpc_key_id needs more code
• use rpc_key_id instead of user_id in the redirect
• /status should include the server weights
• improve rate limiting anon ips
• nullable rpc_key_id on revert log
• attach origin to revert_log
  • opt-in origin logging
• test that runs check_config against example.toml
• improve sorting servers by weight. don't force to lower weights, still have a probability that smaller weights might be
• flamegraphs show 52% of the time to be in tracing. replace with simpler logging
• add optional display name to rpc configs
• add locking around running migrations
• cli tool for checking config
• web3_proxy_cli command should read database settings from config
• cli command to change user_tier by key
• cache the status page for a second
• request accounting for websockets
• database merge scripts
• test that sets up a Web3Rpc and asks "has_block" for old and new blocks
• test that sets up Web3Rpcs with 2 nodes. one behind by several blocks. and see what the "next" server shows as
• ethspam on bsc and polygon gives 1/4 errors. fix whatever is causing this
  • bugfix! we were using the whole connection list instead of just the synced connection list when picking servers. oops!
• actually block unauthenticated requests instead of emitting warning of "allowing without auth during development!"
• smarter reconnection logic
• if a websocket connection hasn't received a new block in a while, do a reconnect or just query the block. its possible that the node was syncing when the proxy started
• on web3-proxy start, if a node fails to connect, it can hold up listening on 8544
  • need to do all the connections in parallel with spawns
• add block timestamp to the /status page
  • be sure to save the timestamp in a way that our request routing logic can make use of it
• node selection still needs improvements. we still send to syncing nodes if they are close
  • try consensus heads first! only if that is empty should we try others. and we should try them sorted by block height and then randomly chosen from there
• logging of "bad response!" is way too verbose
• i think our "best" server picking is incorrect somehow.
  • we upgraded erigon to a version with a broken websocket
  • that made it clear we still route to the lagged server sometimes. this is bad, but retries keep it from giving users bad data.
• more trace logging
• on ETH, we no longer need total difficulty
• cli for creating and editing a user's first api key
• benchmarks of the different Cache implementations (futures vs dash)
  • futures is better
• if archive servers are added to the rotation while they are still syncing, they might get requests too soon. keep archive servers out of the configs until they are done syncing. full nodes should be fine to add to the configs even while syncing, though its a wasted connection
• subscribing to transactions should be configurable per server. listening to paid servers can get expensive
• status page leaks our urls which contain secrets. change that to use names
• for easier errors in the axum code, i think we need to have our own type that wraps anyhow::Result+Error
• hit counts seem wrong. how are we hitting the backend so much more than the frontend? retries on disconnect don't seem to fit that web3_proxy_hit_count{path = "app/proxy_web3_rpc_request"} 857270 web3_proxy_hit_count{path = "backend_rpc/request"} 1396127
  • this was because backend server ordering was including servers that were still syncing from too long ago
• keep it working without redis and a database
• manually tune database and redis connection pool size

V1

These are not yet ordered. There might be duplicates. We might not actually need all of these.

• cache user stats in redis and with headers
• optional read-only database connection
• put display name into our prod configs
• sometimes when fetching a txid through the proxy it fails, but fetching from the backends works fine
  • check flashprofits logs for examples
  • we were caching too aggressively
• BUG! if sending transactions gets "INTERNAL_ERROR: existing tx with same hash", create a success message
  • we just want to be sure that the server has our tx and in this case, it does.
  • ERROR http_request:request:try_send_all_upstream_servers: web3_proxy::rpcs::request: bad response! err=JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32000, message: "INTERNAL_ERROR: existing tx with same hash", data: None })) method=eth_sendRawTransaction rpc=local_erigon_alpha_archive id=01GF4HV03Y4ZNKQV8DW5NDQ5CG method=POST authorized_request=User(Some(SqlxMySqlPoolConnection), AuthorizedKey { ip: 10.11.12.15, origin: None, user_key_id: 4, log_revert_chance: 0.0000 }) self=Web3Rpcs { conns: {"local_erigon_alpha_archive_ws": Web3Rpc { name: "local_erigon_alpha_archive_ws", blocks: "all", .. }, "local_geth_ws": Web3Rpc { name: "local_geth_ws", blocks: 64, .. }, "local_erigon_alpha_archive": Web3Rpc { name: "local_erigon_alpha_archive", blocks: "all", .. }}, .. } authorized_request=Some(User(Some(SqlxMySqlPoolConnection), AuthorizedKey { ip: 10.11.12.15, origin: None, user_key_id: 4, log_revert_chance: 0.0000 })) request=JsonRpcRequest { id: RawValue(39), method: "eth_sendRawTransaction", .. } request_metadata=Some(RequestMetadata { datetime: 2022-10-11T22:14:57.406829095Z, period_seconds: 60, request_bytes: 633, backend_requests: 0, no_servers: 0, error_response: false, response_bytes: 0, response_millis: 0 }) block_needed=None
• serde collect unknown fields in config instead of crash
• upgrade user tier by address
• all_backend_connections skips syncing servers
• change weight back to tier
• fix multiple origin and referer checks
• ip detection needs work so that everything doesnt show up as 172.x.x.x
  • i think this was done, but am not positive.
• if private txs are disabled, only send trasactions to some of our servers. we were DOSing ourselves with transactions and slowing down sync
• retry if we get "the method X is not available"
• remove weight. we don't use it anymore. tiers are what we use now
• make deadlock feature optional
• standalone healthcheck daemon (sentryd)
• status page should show version
• combine the proxy and cli into one bin
• improve rate limiting on websockets
• retry another server if we get a jsonrpc response error about rate limits
• major refactor to only use backup servers when absolutely necessary
• remove allowed lag
• configurable gas buffer. default to the larger of 25k or 25% on polygon to work around erigon bug
• public is 3900, but free is 360. free should be at least 3900 but probably more
• add --max-wait to wait_for_sync
• add automatic compare urls to wait_for_sync
• send panics to pagerduty
• enable lto on release builds
• less logs for backup servers
• use channels instead of arcswap
  • this will let us easily wait for a new head or a new synced connection
• broadcast transactions to more servers
• send sentryd errors to pagerduty
• improve handling of unknown methods
• don't send pagerduty alerts for websocket panics
• improve waiting for sync when rate limited
• improve pager duty errors for smarter deduping
• add create_key cli command
• short lived cache on /health
• cache /status for longer
• sort connections during eth_sendRawTransaction
• block all admin_ rpc commands
• remove the "metered" crate now that we save aggregate queries?
• add archive depth to app config
• use from_block and to_block so that eth_getLogs is routed correctly
• improve eth_sendRawTransaction server selection
• don't cache methods that are usually very large
• use http provider when available
• per-chain rpc rate limits
• canonical block checks giving weird errors. change healthcheck to use block number [2023-02-21T02:58:06Z DEBUG web3_proxy::rpcs::request] error response from blastapi! method=eth_getCode params=(0xa9a8760b8333efae8c9c751e6695a11938ae4b90, 0x73a627f588338804e6dc880154728484f7e0373c29057408c6674d75bdc29d12) err=JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32603, message: "hash 73a627f588338804e6dc880154728484f7e0373c29057408c6674d75bdc29d12 is not currently canonical", data: None })) [2023-02-21T02:58:06Z DEBUG web3_proxy::rpcs::one] blastapi failed health check query! Error { context: "ProviderError from the backend", source: JsonRpcClientError( JsonRpcError( JsonRpcError { code: -32603, message: "hash 73a627f588338804e6dc880154728484f7e0373c29057408c6674d75bdc29d12 is not currently canonical", data: None, }, ), ), }
• add a "failover" tier that is only used if balanced_rpcs has "no servers synced"
  • use this tier (and private tier) to check timestamp on latest block. if we are behind that by more than a few seconds, something is wrong
• cli flag to set prometheus port
• eth_getLogs is going to unsynced nodes because it only checks start block and not the end block
• have multiple providers on each backend rpc. one websocket for newHeads. and then http providers for handling requests
  • erigon only streams the JSON over HTTP. that code isn't enabled for websockets. so this should save memory on the erigon servers
  • i think this also means we don't need to worry about changing the id that the user gives us.
• eth_getLogs is going to unsynced nodes because it only checks start block and not the end block
• fix caching getLogs with blockhash
• fix trying to send signed transactions to an empty list of private_rpcs
• improve logging around consensus head.
  • it was "num in best synced tier"/num rpc connected/num rpc known.
  • it should be "num with best head in best synced tier/num with best head in any tier/num rpcs connected/num rpcs known
• add /debug/:rpckey endpoint that logs requests and responses to kafka
• refactor so configs can change while running
  • this will probably be a rather large change, but is necessary when we have autoscaling
  • create the app without applying any config to it
  • have a blocking future watching the config file and calling app.apply_config() on first load and on change
  • work started on this in the "config_reloads" branch. because of how we pass channels around during spawn, this requires a larger refactor.
• [-] if we subscribe to a server that is syncing, it gives us null block_data_limit. when it catches up, we don't ever send queries to it. we need to recheck block_data_limit
• don't use new_head_provider anywhere except new head subscription
• remove the "metered" crate now that we save aggregate queries?
• don't use systemtime. use chrono
• graceful shutdown
  • frontend needs to shut down first. this will stop serving requests on /health and so new requests should quickly stop being routed to us
  • when frontend has finished, tell all the other tasks to stop
  • stats buffer needs to flush to both the database and influxdb
• rpc_accounting script
• period_datetime should always round to the start of the minute. this will ensure aggregations use as few rows as possible
• weighted random choice should still prioritize non-archive servers
  • maybe shuffle randomly and then sort by (block_limit, random_index)?
  • maybe sum available_requests grouped by archive/non-archive. only limit to non-archive if they have enough?
• if we subscribe to a server that is syncing, it gives us null block_data_limit. when it catches up, we don't ever send queries to it. we need to recheck block_data_limit
• add a "backup" tier that is only used if balanced_rpcs has "no servers synced"
  • use this tier to check timestamp on latest block. if we are behind that by more than a few seconds, something is wrong
• change_user_tier_by_address script
• emit stats for user's successes, retries, failures, with the types of requests, chain, rpc
• add caching to speed up stat queries
• config parsing is strict right now. this makes it hard to deploy on git push since configs need to change along with it
  • changed to only emit a warning if there is an unknown configuration key
• make the "not synced" error more verbose
• short lived cache on /health
• cache /status for longer
• sort connections during eth_sendRawTransaction
• block all admin_ rpc commands
• remove the "metered" crate now that we save aggregate queries?
• add archive depth to app config
• improve "archive_needed" boolean. change to "block_depth"
• keep score of new_head timings for all rpcs
• having the whole block in /status is very verbose. trim it down
• maybe we shouldn't route eth_getLogs to syncing nodes. serving queries slows down sync significantly
  • change the send_best function to only include servers that are at least close to fully synced
• [-] proxy mode for benchmarking all backends
• [-] proxy mode for sending to multiple backends
• [-] let users choose a % of reverts to log (or maybe x/second). someone like curve logging all reverts will be a BIG database very quickly
  • this must be opt-in and spawned in the background since it will slow things down and will make their calls less private
  • automatic pruning of old revert logs once too many are collected
  • we currently default to 0.0 and don't expose a way to edit it. we have a database row, but we don't use it
• [-] add configurable size limits to all the Caches
  • instead of configuring each cache with MB sizes, have one value for total memory footprint and then percentages for each cache
  • https://github.com/moka-rs/moka/issues/201
• all anyhow::Results need to be replaced with FrontendErrorResponse.
  • rename FrontendErrorResponse to Web3ProxyError
  • almost all the anyhows should be Web3ProxyError::BadRequest
  • as is, these errors are seen as 500 errors and so haproxy keeps retrying them
• change premium concurrency limit to be against ip+rpckey
  • then sites like curve.fi don't have to worry about their user count
  • it does mean we will have a harder time capacity planning from the number of keys
• have the healthcheck get the block over http. if it errors, or doesn't match what the websocket says, something is wrong (likely a deadlock in the websocket code)
• has_block_data is too simple. it needs to know what kind of data is being requested
  • all nodes have all blocks
  • most nodes have all receipts
  • only archives have old state
• don't use new_head_provider anywhere except new head subscription
• add support for http basic auth
• [-] enable mev protected transactions with either a /protect/ url (instead of /private/) or the database (when on /rpc/)
• a lot got done that wasn't included in this todo list. go through commits and update this
• eth_sendRawTransaction should only forward if the chain_id matches what we are running
• cli for adding rpc keys to an existing user
• rename "private" to "mev protected" to avoid confusion about private transactions being public once they are mined
• allow restricting an rpc key to specific chains
• [-] writes to median_request_latency should be handled by a background task so they don't slow down the request
• keep re-broadcasting transactions until they are confirmed
• if mev protection is disabled, we should send to both balanced_rpcs and private_rps
• if mev protection is enabled, we should sent to only private_rpcs
• rate limiting/throttling on query_user_stats
• web3rpc configs should have a max_concurrent_requests
  • will probably want a tool for calculating a safe value for this. too low and we could kill our performance
• rename "concurrent" requests to "parallel" requests
• minimum allowed query_start on query_user_stats
• setting request limits to None is broken. it does maxu64 and then internal deferred rate limiter counts try to *99/100
• if kafka fails to connect at the start, automatically reconnect
• during shutdown, mark the proxy unhealthy and send unsubscribe responses for any open websocket subscriptions
• setting request limits to None is broken. it does maxu64 and then internal deferred rate limiter counts overflows when it does to x*99/100
• during shutdown, send unsubscribe responses for any open websocket subscriptions
• some chains still use total_difficulty. have total_difficulty be used only if the chain needs it
  • if total difficulty is not on the block and we aren't on ETH, fetch the full block instead of just the header
  • if total difficulty is set and non-zero, use it for consensus instead of just the number
• query_user_stats cache hit rate
• need debounce on reconnect. websockets are closing on us and then we reconnect twice. locks on ProviderState need more thought
• having the whole block in /status is very verbose. trim it down
• we have our hard rate limiter set up with a period of 60. but most providers have period of 1- [ ] two servers running will confuse rpc_accounting!
  • it won't happen with users often because they should be sticky to one proxy, but unauthenticated users will definitely hit this
  • one option: we need the insert to be an upsert, but how do we merge historgrams?
• don't use systemtime. use chrono
• soft limit needs more thought
  • it should be the min of total_sum_soft_limit (from only non-lagged servers) and min_sum_soft_limit
  • otherwise it won't track anything and will just give errors.
  • but if web3 proxy has just started, we should give some time otherwise we will thundering herd the first server that responds
• connection pool for websockets. use tokio-tungstenite directly. no need for ethers providers since serde_json is enough for us
  • this should also get us closer to being able to do our own streaming json parser where we can
• figure out if "could not get block from params" is a problem worth logging
  • maybe it was an ots request?
• change redirect_rpc_key_url to match the newest url scheme
• implement filters
• implement remaining subscriptions
  • would be nice if our subscriptions had better gaurentees than geth/erigon do, but maybe simpler to just setup a broadcast channel and proxy all the respones to a backend instead
• tests should use test-env-log = "0.2.8"
• eth_sendRawTransaction should only forward if the chain_id matches what we are running
• weighted random choice should still prioritize non-archive servers
  • maybe shuffle randomly and then sort by (block_limit, random_index)?
  • maybe sum available_requests grouped by archive/non-archive. only limit to non-archive if they have enough?
• some places we call it "accounting" others a "stat". be consistent
• cli commands to search users by key
• flamegraphs show 25% of the time to be in moka-housekeeper. tune that
• config parsing is strict right now. this makes it hard to deploy on git push since configs need to change along with it
• when displaying the user's data, they just see an opaque id for their tier. We should join that data
• refactor so configs can change while running
  • this will probably be a rather large change, but is necessary when we have autoscaling
  • create the app without applying any config to it
  • have a blocking future watching the config file and calling app.apply_config() on first load and on change
  • work started on this in the "config_reloads" branch. because of how we pass channels around during spawn, this requires a larger refactor.
• when displaying the user's data, they just see an opaque id for their tier. We should join that data so they see the tier name and limits
• add indexes to speed up stat queries
• the public rpc is rate limited by ip and the authenticated rpc is rate limit by key
  • this means if a dapp uses the authenticated RPC on their website, they could get rate limited more easily
• take an option to set a non-default role when creating a user
• different prune levels for free tiers
• have a test that runs ethspam and versus
• status page show git hash of running version
• Email confirmation
  • we'll need a pretty template email that the backend will send.
  • That will link them to a a page on llamanodes.com
  • There, they click "confirm" (or JavaScript does it for them automatically) to POST to this new endpoint
• test in the migration repo that sets up a sqlite database that runs up and down
• unbounded queues are risky. add limits
• after running for a while, https://eth-ski.llamanodes.com/status is only at 157 blocks and hashes. i thought they would be near 10k after running for a while
  • adding uptime to the status should help
  • i think this is already in our todo list
• write a test that uses the cli to create a user and modifies their key
• Uuid/Ulid instead of big_unsigned for database ids
  • might have to use Uuid in sea-orm and then convert to Ulid on display
  • https://www.kostolansky.sk/posts/how-to-migrate-to-uuid/
• emit stdandard deviation?
• emit global stat on retry
• emit global stat on no servers synced
• emit global stat on error (maybe just use sentry, but graphs are handy)
  • if we wait until the error handler to emit the stat, i don't think we have access to the authorized_request
• endpoint (and cli script) to rotate api key
• if no bearer token found in redis (likely because it expired), send 401 unauthorized
• user create script should allow multiple keys per user
• somehow the proxy thought latest was hours behind. need internal health check that forces reconnect if this happens
• display concurrent requests per api key (only with authentication!)
• change "remember me" to last until 4 weeks of no use, rather than 4 weeks since login? that will be a lot more database writes
• BUG? WARN http_request:request: web3_proxy::block_number: could not get block from params err=unexpected params length id=01GF4HTRKM4JV6NX52XSF9AYMW method=POST authorized_request=User(Some(SqlxMySqlPoolConnection), AuthorizedKey { ip: 10.11.12.15, origin: None, user_key_id: 4, log_revert_chance: 0.0000 })
  • why is it failing to get the block from params when its set to None? That should be the simple case
• BUG: i think if all backend servers stop, the server doesn't properly reconnect. It appears to stop listening on 8854, but not shut down.
• if user-specific caches have evictions that aren't from timeouts, log a warning
• make sure the email address is valid. probably have a "verified" column in the database
• if invalid user id given, we give a 500. should be a different error code instead
  • WARN http_request: web3_proxy::errors: anyhow err=UserKey was not a ULID or UUID id=01GER4VBTS0FDHEBR96D1JRDZF method=POST
• admin-only endpoint for seeing a user's stats for support requests
• from what i thought, /status should show hashes > numbers!
  • but block numbers count is maxed out (10k)
  • and block hashes count is tiny (83)
  • what is going on? when the server fist launches they are in sync
  • related BUG? WARN web3_proxy::rpcs::blockchain: Missing connection_head_block in block_hashes. Fetching now connection_head_hash=0x4b7a…14b5 conn_name=local_erigon_alpha_archive rpc=local_erigon_alpha_archive
  • i see this a lot more than expected. why is it happening so much? better logs needed
• after adding semaphores (or maybe something else), CPU load seems a lot higher. investigate
• proper support for Finalized and Safe block queries
• admin-only page for viewing user stat pages
• geth sometimes gives an empty response instead of an error response. figure out a good way to catch this and not serve it
• GET balance endpoint
• POST balance endpoint
• EIP1271 for siwe
• Limited throughput during high traffic
• instead of Option<...> in our frontend function signatures, use result and then the try operator so that we get our errors wrapped in json
• revert logs should have a maximum age and a maximum count to keep the database from being huge
• user login should also return a jwt (jsonwebtoken rust crate should make it easy)
• script that looks at config and estimates max memory used by caches
• favicon
  • eth_1 | 2022-09-07T17:10:48.431536Z WARN web3_proxy::jsonrpc: forwarding error err=nothing to see here
  • use the one on https://staging.llamanodes.com/
• warn if no servers have transaction subscriptions
  • if no servers have transaction subscriptions, and a user tries to subscribe, make sure the error is user friendly
• only allow transaction and full block subscriptions if the user is registered?
• eth_subscribe logs (https://geth.ethereum.org/docs/rpc/pubsub)
• make private transactions opt in (its already in the database, but not our code)
• write a function for receipts that tries balanced_rpcs and only if they all error should it try private relays
  • automatic retries with a timeout or until all the servers have been tried.
    • i had the websocket die on me in the middle of a long test. only one in-flight request failed because of it. the rest delayed. figure out how to catch these ones since websocket fails sadly seem common
• nice output when cargo doc is run
• cache more things locally or in redis
• stats when forks are resolved (and what chain they were on?)
• Only subscribe to transactions when someone is listening and if the server has opted in to it
• When sending eth_sendRawTransaction, retry errors
• If we need an archive server and no servers in sync, exit immediately with an error instead of waiting 60 seconds
• 120 second timeout is too short. Maybe do that for free tier and larger timeout for paid. Problem is that some queries can take over 1000 seconds
• when handling errors from axum parsing the Json...Enum in the function signature, the errors don't get wrapped in json. i think we need a axum::Layer
• don't "unwrap" anywhere. give proper errors
• handle log subscriptions
  • probably as a paid feature
• relevant erigon changelogs: add pendingTransactionWithBody subscription method (#5675)
• change_user_tier_by_key should not show the rpc key id. that way our ansible playbook won't expose it
• make sure all our responses follow the spec: https://www.jsonrpc.org/specification#examples
• min_sum_soft_limit should be automatic based on the app's average rps plus a buffer.
  • add a rate counter to the balanced_rpcs
  • every time a block is found, update min_sum_soft_limit
  • add a min_sum_soft_limit_safety
    • keeps the automaticly calculated limit from going so high that we stop serving requests
  • add a min_sum_soft_limit_max_wait that advances the consensus block even if mins not met yet
• a script for load testing a server and calculating its hard and soft limits
• use https://github.com/dherman/esprit or similar to parse https://github.com/DefiLlama/chainlist/blob/main/constants/extraRpcs.js
• update example.toml
  • might need to make changes so the influxdb stuff is optional. david said it stopped right after starting
• i'm seeing a bunch of errors with eth_getLogs.
  • i think maybe my block number rewriting is causing problems. but maybe its just a user doing bad queries
• Use "is_fresh" instead of our atomic bool
  • moka 0.10 - Add entry and entry_by_ref APIs to sync and future caches (#193): They allow users to perform more complex operations on a cache entry. At this point, the following operations (methods) are provided: or_default or_insert or_insert_with or_insert_with_if or_optionally_insert_with or_try_insert_with The above methods return Entry type, which provides is_fresh method to check if the value was freshly computed or already existed in the cache.
• lag message always shows on first response
  • http interval on blastapi lagging by 1!
• change scoring for rpcs again. "p2c ewma"
  • weighted random sort: (soft_limit - ewma active requests * num web3_proxy servers)
    • 2. soft_limit
  • pick 2 servers from the random sort.
    • exponential weighted moving average for block subscriptions of time behind the first server (works well for ws but not http)

V2

These are not ordered. I think some rows also accidently got deleted here. Check git history.

• less Arc (and more pin?). we use arcs on a lot of things where i think a &self should work fine.
• automatically tune database and redis connection pool size
• if db is down, keep logins cached longer. at least only new logins will have trouble then
• handle user payments
  • separate daemon (or users themselves) call POST /users/process_transaction
    • checks a transaction to see if it modifies a user's balance. records results in a sql database
    • we will have our own event subscriber watching for "deposit" events, but sometimes events get missed and users might incorrectly "transfer" the tokens directly to an address instead of using the dapp
• if a rpc fails to connect at start, retry later instead of skipping it forever (need config hot reloads first)
• jwt auth so people can easily switch from infura
• automated soft limit
  • look at average request time for getBlock? i'm not sure how good a proxy that will be for serving eth_call, but its a start
  • https://crates.io/crates/histogram-sampler
• interval for http subscriptions should be based on block time. load from config is easy, but better to query. currently hard coded to 13 seconds
• check code to keep us from going backwards. maybe that is causing outages
• min_backup_rpcs seperate from min_synced_rpcs

in another repo: event subscriber

• watch for transfer events to our contract and submit them to /payment/$tx_hash
• cli tool that support can run to manually check and submit a transaction

"Maybe some day" and other Miscellaneous Things

• tool to revoke bearer tokens that clears redis
• eth_getBlockByNumber and similar calls served from the block map
  • will need all Block and Block in caches or fetched efficiently
  • so maybe we don't want this. we can just use the general request cache for these. they will only require 1 request and it means requests won't get in the way as much on writes as new blocks arrive.
  • after looking at my request logs, i think its worth doing this. no point hitting the backends with requests for blocks multiple times. will also help with cache hit rates since we can keep recent blocks in a separate cache
• Public bsc server got “0” for block data limit (ninicoin)
• cli tool for resetting api keys
• Advanced load testing scripts so we can find optimal cost servers
  • benchmarks from https://github.com/llamafolio/llamafolio-api/
  • benchmarks from ethspam and versus
  • benchmarks from other things
  • quick script that calls all the curve-api endpoints once and checks for success, then calls wrk to hammer it
    • https://github.com/curvefi/curve-api
    • test /api/getGaugesmethod
      • usually times out after vercel's 60 second timeout
      • one time got: Error invalid Json response ""
• page that prints a graphviz dotfile of the blockchain
• search for all the "TODO" and todo!(...) items in the code and move them here
• add the backend server to the header?
• have a low-latency option that always tries at least two servers in parallel and then returns the first success?
  • this doubles our request load though. maybe only if the first one doesn't respond very quickly?
• zero downtime deploys
• are we using Acquire/Release/AcqRel properly? or do we need other modes?
• use https://github.com/ledgerwatch/interfaces to talk to erigon directly instead of through erigon's rpcdaemon (possible example code which uses ledgerwatch/interfaces: https://github.com/akula-bft/akula/tree/master)
• subscribe to pending transactions and build an intelligent gas estimator
• flashbots specific methods
  • flashbots protect fast mode or not? probably fast matches most user's needs, but no reverts is nice.
  • https://docs.flashbots.net/flashbots-auction/searchers/advanced/rpc-endpoint#authentication maybe have per-user keys. or pass their header on if its set
• if no redis set, but public rate limits are set, exit with an error
• i saw "WebSocket connection closed unexpectedly" but no log about reconnecting
  • need better logs on this because afaict it did reconnect
• better document load tests: docker run --rm --name spam shazow/ethspam --rpc http://$LOCAL_IP:8544 | versus --concurrency=100 --stop-after=10000 http://$LOCAL_IP:8544; docker stop spam
• if the call is something simple like "symbol" or "decimals", cache that too. though i think this could bite us.
• add a subscription that returns the head block number and hash but nothing else
• if chain split detected, what should we do? don't send transactions?
• archive check works well for local servers, but public nodes (especially on other chains) seem to give unreliable results. likely because of load balancers.
  • configurable block data limit until better checks
• https://docs.rs/derive_builder/latest/derive_builder/
• Detect orphaned transactions
• https://crates.io/crates/reqwest-middleware easy retry with exponential back off
  • Though I think we want retries that go to other backends instead
• Some of the pub things should probably be "pub(crate)"
• Maybe storing pending txs on receipt in a dashmap is wrong. We want to store in a timer_heap (or similar) when we actually send. This way there's no lock contention until the race is over.
• Support "safe" block height. It's planned for eth2 but we can kind of do it now but just doing head block num-3
• Archive check on BSC gave “archive” when it isn’t. and FTM gave 90k for all servers even though they should be archive
• cache eth_getLogs in a database?
• stats for "read amplification". how many backend requests do we send compared to frontend requests we received?
• fully test retrying when "header not found"
  • i saw "header not found" on a simple eth_getCode query to a public load balanced bsc archive node on block 1
• weird flapping fork could have more useful logs. like, howd we get to 1/1/4 and fork. geth changed its mind 3 times?
  • should we change our code to follow the same consensus rules as geth? our first seen still seems like a reasonable choice
  • other chains might change all sorts of things about their fork choice rules 2022-07-22T23:52:18.593956Z WARN block_receiver: web3_proxy::connections: chain is forked! 1 possible heads. 1/1/4 rpcs have 0xa906…5bc1 rpc=Web3Rpc { url: "ws://127.0.0.1:8546", data: 64, .. } new_block_num=15195517 2022-07-22T23:52:18.983441Z WARN block_receiver: web3_proxy::connections: chain is forked! 1 possible heads. 1/1/4 rpcs have 0x70e8…48e0 rpc=Web3Rpc { url: "ws://127.0.0.1:8546", data: 64, .. } new_block_num=15195517 2022-07-22T23:52:19.350720Z WARN block_receiver: web3_proxy::connections: chain is forked! 2 possible heads. 1/2/4 rpcs have 0x70e8…48e0 rpc=Web3Rpc { url: "ws://127.0.0.1:8549", data: "archive", .. } new_block_num=15195517 2022-07-22T23:52:26.041140Z WARN block_receiver: web3_proxy::connections: chain is forked! 2 possible heads. 2/4/4 rpcs have 0x70e8…48e0 rpc=Web3Rpc { url: "http://127.0.0.1:8549", data: "archive", .. } new_block_num=15195517
  • threshold should check actual available request limits (if any) instead of just the soft limit
• foreign key on_update and on_delete
• database creation timestamps
• better error handling. we warn too often for validation errors and use the same error code for most every request
• use &str more instead of String. lifetime annotations get really annoying though
• tarpit instead of reject requests (unless theres a lot)
• archive servers should be lowest priority
• docker build context is really big. we must be including target or something
• fix ip detection when running in dev
• PR to add this to sea orm prelude:

  #[cfg(feature = "with-uuid")]
  pub use uuid::Builder as UuidBuilder;
  

• rate limit thoughts:
  • if someone subscribes to all pending transactions, how should that count against rate limits
  • when those rate limits are hit, what should happen?
  • missing pending transactions might be okay, but not missing confirmed blocks
• sea-orm brings in async-std, but we are using tokio. benchmark switching
• this query always times out, but erigon can serve it quickly: curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["latest"],"id":1}' 127.0.0.1:8544' 127.0.0.1:8544 {"jsonrpc":"2.0","id":null,"error":{"code":-32099,"message":"deadline has elapsed"}}
  • figure out rate limits for private rpcs. eden v1 gives 500 error instead of a code for rate limits
• https://gitlab.com/moka-labs/tiered-cache-example
• web3connection3.block(...) might wait forever. be sure to do it safely
• search for all "todo!"
• when using a bunch of slow public servers, i see "no servers in sync" even when things should be right
  • maybe iterate connection heads by total weight? i still think we need to include parent hashes
• i see "No block found" sometimes for a single server's block. Not sure why since reads should happen after writes
• better handling for offline http servers
  • if we get a connection refused, we should remove the server's block info so it is taken out of rotation
• how should we handle reverting transactions? they won't confirm for a while after we send them
• allow configuration of the expiration time of bearer tokens. currently defaults to 4 weeks
• emit stat when an IP/key goes over rate limits
• readme command should run create_user commands via docker-compose
• helper for UUID <-> ULID
• Wrapping extractors in Result makes them optional and gives you the reason the extraction failed
• at concurrency 100, ethspam is getting 400 and 422 errors. figure out why. probably something with redis or mysql, but maybe its something else like spawning
• emit per-key stats for latency of semaphore awaits. if this starts to grow, people will know they are hitting limits and need a higher tier
• need a status page for your wallet's rpc. show head block information with age
• replace serde_json::Value with https://lib.rs/crates/ijson (more memory efficient)
• have a log all option? instead of just reverts, log all request/responses? can be very useful for debugging but would flood our database. maybe better for them to do that on their client side
• failsafe. if no blocks or transactions in some time, warn and reset the connection
• WARN http_request:request: web3_proxy::block_number: could not get block from params err=unexpected params length id=01GF4HTRKM4JV6NX52XSF9AYMW method=POST authorized_request=User(Some(SqlxMySqlPoolConnection), AuthorizedKey { ip: 10.11.12.15, origin: None, user_key_id: 4, log_revert_chance: 0.0000 })
• having tons of worker threads can actually make us slower if they keep waking to steal work from eachother. need benchmarks
• change the wrk data to log requests and errors to a file
• if redis is not set and login page is visited, users get a 502. should be 501
• allow passing the authorization header to the anonymous rpc endpoint
• sentry profiling
• support alchemy_minedTransactions
• debug print of user::Model's address is a big vec of numbers. make that hex somehow
• make it so you can put a string like "LN arbitrum" into the create_user script, and have it automatically turn it into 0x4c4e20617262697472756d000000000000000000.
  • if --address not given, use the --description
  • if it is too long, (the last 4 bytes must be zero), give an error so descriptions like this stand out
• we need to use docker-compose's proper environment variable handling. because now if someone tries to start dev containers in their prod, remove orphans stops and removes them
• change invite codes to set the user_tier
• some cli commands should use the replica if possible
• some third party rpcs have limits on the size of eth_getLogs. include those limits in server config
• some internal requests should go through app.proxy_rpc_request so that they get caching!
  • be careful not to make an infinite loop
• request timeout messages should include the request id
• have an upgrade tier that queries multiple backends at once. returns on first Ok result, collects errors. if no Ok, find the most common error and then respond with that
• give public_recent_ips_salt a better, more general, name
• include tier in the head block logs?
• i think i use FuturesUnordered when a try_join_all might be better
• since we are read-heavy on our configs, maybe we should use a cache
  • "using a thread local storage and explicit types" https://docs.rs/arc-swap/latest/arc_swap/cache/struct.Cache.html
• tests for config reloading
• use pin instead of arc for a bunch of things?
  • https://fasterthanli.me/articles/pin-and-suffering
• calculate archive depth automatically based on block_data_limits