Commit Graph

152 Commits

Author SHA1 Message Date
Bryan Stitt
e8d992168c more logs and retries 2023-06-28 18:36:17 -07:00
Bryan Stitt
df865292a7 add another layer of retries 2023-06-28 15:12:16 -07:00
Bryan Stitt
eab3aa7943 that simpler param doesn't work everywhere 2023-06-27 17:39:37 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
a440d51377 test fixture 2023-06-24 21:17:08 -07:00
Bryan Stitt
30828ccae2 prioritize block. queries to syncing nodes ends up being slower 2023-06-24 00:41:30 -07:00
Bryan Stitt
fd661689a3
Tracing (#144)
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
2023-06-23 16:28:45 -07:00
Bryan Stitt
3c2f95de10
Improve max wait and timeouts (#137)
* pass max wait with more functions

* move rpc_head_str higher up to use it in all the logs

* refresh consensus at double the block time

* new pricing that also includes archive check

* lint

* don't clone method

* put caches into one list

they serialize with their name, so it will still be easy to tell them apart

* more consistent waiting. still waits more than I want right now

* clean up default max waits

* more timeouts
2023-06-20 14:22:14 -07:00
Bryan Stitt
825ba006f1 move floats and durations around for latency tracking 2023-06-18 10:46:22 -07:00
Bryan Stitt
2f0e6103ac Revert "tier more important than backup"
This reverts commit bca22537c633822e5e6759fa2840712a837211fd.
2023-06-18 10:09:21 -07:00
Bryan Stitt
bca22537c6 tier more important than backup
we filter backups when they aren't needed, so this should be okay
2023-06-18 10:06:26 -07:00
Bryan Stitt
e54299beca rolling median for choosing tier 2023-06-18 09:53:36 -07:00
Bryan Stitt
8b22c9861b Revert "temp fix while filtering is improved"
This reverts commit 5367c511953fcea1af3da1d905de06d9e76711ba.
2023-06-17 12:15:40 -07:00
Bryan Stitt
5367c51195 temp fix while filtering is improved 2023-06-17 12:09:04 -07:00
Bryan Stitt
745870d119 drop a bunch of log levels 2023-06-16 23:46:20 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129)
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
2023-06-16 00:46:27 -07:00
Bryan Stitt
b8f429d70a improvements to popularity_contesst 2023-06-13 11:51:19 -07:00
Bryan Stitt
f5a1ac274a refactor send_head_block_result 2023-06-13 10:00:08 -07:00
Bryan Stitt
1473ccb085 gentler parsing of new head blocks 2023-06-13 09:31:22 -07:00
Bryan Stitt
a7fb12bd72 less random is fine for shuffling 2023-06-12 22:42:22 -07:00
Bryan Stitt
55e8471b19 interval instead of watch_blocks 2023-06-12 22:26:10 -07:00
Bryan Stitt
bba7ccf7eb change divisor for tier calculation 2023-06-12 20:44:52 -07:00
Bryan Stitt
c771479d94 use WyRand instead of Xoshiro 2023-06-12 19:46:36 -07:00
Bryan Stitt
1a8f799969
we need our own Reconnect (#124)
* first pass at handling different return data limits

* put ws_provider in an arcswap

* add min max_latency

* add min max_latency

* subscribe with reconnect

* better logging around reconnect

* select on both watches

* subscribe to the correct watch
2023-06-12 18:58:51 -07:00
Bryan Stitt
bd87fcb13c move sort and shuffle for loadbalancing into proper functions 2023-06-09 13:09:58 -07:00
Bryan Stitt
1f3040f6c7 automated tiers 2023-06-09 12:21:55 -07:00
Bryan Stitt
40bee30730 back to moka 2023-06-07 17:30:05 -07:00
Bryan Stitt
fd8bba4bdd most of these are for the whole app, not just the frontend 2023-05-30 21:26:11 -07:00
Bryan Stitt
cebe8ed1ea
Request refactor (#99)
* wip

* AsRef finally works like i wanted

* actually return the block

* start adding async trait

* remove stale import

* include id in the error response when possible

* remove stale comments
2023-05-30 16:32:34 -07:00
Bryan Stitt
bb50efb7f9
quick cache ttl max item size, better function names, and tests (#97)
* Revert "remove cache on /status and /health for now"

This reverts commit 166b0d810cd25b895560d50d00383a57e1269ad4.

* Revert "remove cache on backups needed, too"

This reverts commit 4597967def2f1daa71a277ec6731db4739ead640.

* add tests

* add max_item_weight

* comment

* add some helper functions, max weights, and tests
2023-05-29 15:48:22 -07:00
Bryan Stitt
dd347fd916 add units to serialized variables 2023-05-23 14:57:24 -07:00
Bryan Stitt
77c9b0ab12 peak latency instead of head latency 2023-05-23 14:52:45 -07:00
Bryan Stitt
78f247fc6c split http and ws provider 2023-05-23 14:40:34 -07:00
Bryan Stitt
3ac15558e3 fix decay time 2023-05-22 15:43:39 -07:00
Bryan Stitt
91eeee23e2 use ether's reconnects instead of our own which need a lock (wip) 2023-05-22 15:32:27 -07:00
Bryan Stitt
5aae318620 logs and comments 2023-05-18 15:11:29 -07:00
Bryan Stitt
b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt
89961331af refactor wait_for_block into should_wait_for_block 2023-05-16 22:29:36 -07:00
Bryan Stitt
c66eb6d864 head latency instead of peak latency for now 2023-05-16 12:18:59 -07:00
Bryan Stitt
2080739865
Quick cache (#70)
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343fc51338a8a4282dcc229b50097835b.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
2023-05-13 11:13:02 -07:00
Bryan Stitt
c3cdf7ef43 fix consensus finding bug 2023-05-12 23:40:15 -07:00
Bryan Stitt
c54970da0a change ordering and move fetch_add and fetch_sub 2023-05-12 23:00:03 -07:00
Bryan Stitt
914c3e03a8 remove stale comments 2023-05-12 22:22:21 -07:00
Bryan Stitt
3cea4c3123 update peak_ewma to use request latency and fix off by one 2023-05-12 22:21:11 -07:00
Bryan Stitt
510612d343
use kanal instead of flume or tokio channels (#68) 2023-05-12 15:47:01 -07:00
Bryan Stitt
8a097dabbe
Bryan devel 2023-05-12 (#67)
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
2023-05-12 15:15:32 -07:00
Bryan Stitt
36cc884112 Merge remote-tracking branch 'origin/main' into devel 2023-05-11 20:05:47 -07:00
Bryan Stitt
6e8536d477 add support for http basic auth 2023-05-11 13:21:12 -07:00
Rory Trent
ec11e210ee
Peak ewma (#63)
* use peak-ewma instead of head for latency calculation

* Implement some suggested changes from PR

* move latency to new package in workspace root

* fix unit tests which now require peak_latency on Web3Rpc

* Switch to atomics for peak-ewma

This change is to avoid locking from tokio::sync::watch.

* add decay calculation to latency reads in peak-ewma

* Add some tests for peak-ewma

* Sensible latency defaults and not blocking on full

* Cleanup and a couple additional comments
2023-05-11 13:09:15 -07:00
Bryan Stitt
96a9a39789 add support for http basic auth 2023-05-11 13:07:38 -07:00
Bryan Stitt
56fdf48129
Send transactions to protected and public rpcs (#57)
* move protected transactions into their own function and dry stats sending

* cargo upgrade

* comments

* time to live instead of time to idle

* minor workaround for eth_chainId

* cargo upgrade
2023-04-24 11:00:12 -07:00
Bryan Stitt
4889c3e1ce Merge branch 'main' into devel 2023-04-18 20:38:43 -07:00
Bryan Stitt
e921d02eb2 remove stale todo 2023-04-17 17:10:09 -07:00
Bryan Stitt
5ff28943da fix incorrect context 2023-04-14 03:44:32 -07:00
Bryan Stitt
2652f88f52 attach more contexts 2023-04-14 03:43:49 -07:00
Bryan Stitt
8ed71e1cf1 more goerli fixes 2023-04-14 00:36:46 -07:00
Bryan Stitt
d035049c8f add /backups_needed endpoint for easy alerts 2023-04-13 17:15:01 -07:00
Bryan Stitt
7250a0c155 upgrade ethers to 2.0.2 2023-04-10 15:01:36 -07:00
Bryan Stitt
b460f1701a rename revert_handler to error_handler 2023-04-05 15:34:28 -07:00
Bryan Stitt
7994c894bb cargo clippy and fmt 2023-04-05 15:15:20 -07:00
Bryan Stitt
1a160a8c7d Merge remote-tracking branch 'rorytrent/Web3ProxyError' into devel 2023-04-05 15:08:57 -07:00
yenicelik
0069e76040 latest changes from bryan 2023-04-04 14:40:22 +02:00
Rory Neithinger
f0d8a0c8c9 Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-04-01 00:04:44 -07:00
Rory Neithinger
ffdf25787f Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-31 23:23:30 -07:00
yenicelik
0bb3a2dc06 simple changes around bryans comments 2023-03-31 12:43:41 +01:00
yenicelik
345fa6c513 git status 2023-03-30 13:42:56 +01:00
Bryan Stitt
2205a472fd add optional tokio-console and improve reconnect logic 2023-03-22 18:43:13 -07:00
Bryan Stitt
7b223efa4d improve reconnect logic 2023-03-22 16:16:15 -07:00
Rory Neithinger
beac7ee017 better error handling for proxy_web3_rpc() 2023-03-19 18:52:28 -07:00
Rory Neithinger
f3fc4924dc Refactor FrontendErrorResponse into Web3ProxyError
Renamed FrontendResponse to Web3ProxyResponse and introduced
a new generic type alias Web3ProxyResult.

Fixed a few noisy cargo warnings.
2023-03-16 22:21:20 -07:00
Bryan Stitt
dbd7860416 wip influxdb2 2023-03-04 20:54:04 +01:00
Bryan Stitt
eb4d05a520 stats v2
rebased all my commits and squashed them down to one
2023-03-04 20:48:49 +01:00
Bryan Stitt
d1b955275d clean up wait_for_ functions 2023-03-02 18:27:32 +00:00
Bryan Stitt
907a147afa wait_for_provider helper function 2023-03-02 18:14:17 +00:00
Bryan Stitt
0d07e20d7c make it compile 2023-03-02 17:38:42 +00:00
Bryan Stitt
258e889f69 wait for a provider 2023-03-02 16:52:28 +00:00
Bryan Stitt
2e9803de72 remove arbitrary sleep timer 2023-03-01 20:56:00 +00:00
Bryan Stitt
b87c988439 lint 2023-03-01 11:23:59 -08:00
Bryan Stitt
20851af51b log seconds instead of milliseconds 2023-02-28 11:09:49 -08:00
Bryan Stitt
d83a1f0169 use Web3Rpc instead of the name as a key 2023-02-28 11:01:34 -08:00
Bryan Stitt
c5e2402d6e disconnect is noisy but it works 2023-02-27 13:29:07 -08:00
Bryan Stitt
6067369ee3 add inotify and rpc disconnect 2023-02-27 12:50:57 -08:00
Bryan Stitt
f8f5e7a1c8 add thread for config file watching and run clippy lint 2023-02-26 23:08:00 -08:00
Bryan Stitt
e96f09a9c4 refactors to make configs partially reloadable 2023-02-25 23:52:33 -08:00
Bryan Stitt
91b853d21e archive instead of Some(u64::MAX) in the logs 2023-02-21 21:14:49 -08:00
Bryan Stitt
6c33b1ae1e block number instead of hash during health check 2023-02-21 20:25:54 -08:00
Bryan Stitt
738815244d change ewma calculation 2023-02-16 00:26:58 -08:00
Bryan Stitt
4916188d5c theres a perf regression in here somewhere 2023-02-15 21:54:07 -08:00
Bryan Stitt
cb2b5c16ad lower log level 2023-02-15 21:30:42 -08:00
Bryan Stitt
b7dfec9c63 fix params in health check query 2023-02-15 21:13:28 -08:00
Bryan Stitt
3c9576c13b yes health, no channel 2023-02-15 15:52:42 -08:00
Bryan Stitt
63adbc0587 remove health for now? 2023-02-15 15:42:27 -08:00
Bryan Stitt
268ad3a7c5 dont hold a lock open 2023-02-15 15:37:43 -08:00
Bryan Stitt
056670d29a health check and latency recordings 2023-02-15 15:31:59 -08:00
Bryan Stitt
1fb4dd6ccc i think it works 2023-02-15 12:33:43 -08:00
Bryan Stitt
c65ffc9ce0 getting close 2023-02-14 17:41:40 -08:00
Bryan Stitt
824a6fa1f0 well it compiles. doesnt work yet. but it compiles 2023-02-14 12:14:50 -08:00
Bryan Stitt
70105bc7bb variable rename 2023-02-12 10:22:20 -08:00
Bryan Stitt
c008f50943 use http and ws providers in a single config 2023-02-12 01:40:01 -08:00
Bryan Stitt
c959110986 better sorting of connections 2023-02-10 20:24:20 -08:00