tornadocontrib/web3-proxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,297 Commits 1 Branch 0 Tags 12 MiB
d50a7a06d9
Commit Graph

566 Commits

Author SHA1 Message Date
Bryan Stitt
36eb520f84 rename block_ to block_and_rpc_ where applicable 2023-08-16 13:20:22 -07:00
Bryan Stitt
b742a25729 dont return max depth so we can do cannonical block faster 2023-08-15 16:12:40 -07:00
Bryan Stitt
bc8040f61d more yields and less timeouts 2023-08-04 18:54:16 -07:00
Bryan Stitt
aed98ac034 try yielding around all selects 2023-08-04 18:37:46 -07:00
Bryan Stitt
0bc523ab9f improve hard limits 
rate limit per server or per endpoint and with any period
 2023-08-03 12:24:28 -07:00
Bryan Stitt
690601643d improve eth_sendRawTransaction and other timeouts 2023-08-03 11:39:48 -07:00
Bryan Stitt
74224977b7 Revert "Revert "remove max_tries for now"" 
This reverts commit 5fbcb75157cd012b723c9d2b37bb7947e5e05aa5.
 2023-08-03 09:15:31 -07:00
Bryan Stitt
ca802751c1 if block fails, try other rpcs 2023-08-03 00:29:13 -07:00
Bryan Stitt
5747caec71 break if no max_wait 2023-08-03 00:23:44 -07:00
Bryan Stitt
5fbcb75157 Revert "remove max_tries for now" 
This reverts commit e0d7d11398163bb8554cd6736d1dbc7f55caaa24.
 2023-08-03 00:22:45 -07:00
Bryan Stitt
e0d7d11398 remove max_tries for now 
i think this is causing problematic looping
 2023-08-02 22:07:28 -07:00
Bryan Stitt
8aac4779bf clean up some selects 2023-08-02 21:27:56 -07:00
Bryan Stitt
32a73a0c53 cargo upgrade --incompatible 2023-07-27 12:29:10 -07:00
Bryan Stitt
b1f447c5c8
add mass_grant_credits helper script and other prep for premium (#196) 
* add mass_grant_credits helper script

* include error in the rate limit log

* allow granting 0 credits

* migration to change default user_tier to premium
 2023-07-26 07:33:00 -07:00
Bryan Stitt
a68d0d60b7 only health check if theres a block and rpc sender 2023-07-21 13:28:00 -07:00
Bryan Stitt
ae8a12047e tarpit strange methods 2023-07-20 14:32:52 -07:00
Bryan Stitt
e7905b4344 Revert "First pass implementation of async serialize for Web3Rps (#192)" 
This reverts commit 269aa32260a4c3e94d7ada55fee6c8e62d975c87.
 2023-07-19 23:12:49 -07:00
Bryan Stitt
264508c454 treat serde errors as rate limits (because all of them seem to be) 2023-07-14 21:43:37 -07:00
Bryan Stitt
61126bb2ac one more Cow 2023-07-14 18:35:40 -07:00
Bryan Stitt
020dff0233 lint 2023-07-14 18:31:42 -07:00
Bryan Stitt
efa189d101 less anyhow and fix tests 2023-07-14 18:31:00 -07:00
Bryan Stitt
0046a02a4a
Global db (#191) 
* dont migrate on start. just connect

* it compiles

* that log is loud

* get transaction from a local clone

* import the trait

* put the test app into a thread instead of tokio::spawn

* fix one test

* try db before rpcs

* db connection is too slow. need to wait for it

* do db setup once while spawning
 2023-07-14 18:30:01 -07:00
Bryan Stitt
269aa32260
First pass implementation of async serialize for Web3Rps (#192) 
Co-authored-by: Rory Neithinger <rorytrent@gmail.com>
 2023-07-14 18:29:48 -07:00
Bryan Stitt
3c2d06324e faster rpc. backups always lose 2023-07-13 18:14:13 -07:00
Bryan Stitt
dce3540ee3 less authorization and remove pending txs for now 2023-07-13 11:01:16 -07:00
Bryan Stitt
df2f3d340f
More balance tests (#182) 
* fix popularity contest

* more info in the Debug for Web3Rpc

* add frontend_requests and cache_misses to the Balance query

* add more to balance and stats flushing and improved test coverage

* it compiles

* deserializer for Ulid to Uuid

I think a wrapper type on Ulid that implements sea_orm::Value is probably better

* rename variable to match struct name

* add deserializer for Address -> Vec<u8>

* sql sum returns a Decimal. need to convert to u64

* assert more

* one log and assert more

* log more

* use a helper to get the user's rpc provider

* this should be 2 now that we have a public and authed call

* this should be zero. the public has the cache miss

* instrument cu calcs

* trace the value we took, not the default that replaced it

* move usd_per_chain into config

* remove some extra logging

* use Arc::into_inner to maybe avoid a race

* off by 1

* pass paid credits used instead of returning it

this lets us use it to write to our user balance cache first. importantly, this keeps us from holding a write lock while writing to mysql

* no cache misses expected in this test

* actually check the admin

* put the balance checks back now that the rest of the test works

* archive request is being set incorrectly

* wow howd we manage flipping the greater than sign on archive depth

* move latest_balance and premium_credits_used to before any stats are emitted

* lint

* and build undoes the linting. fun

i didnt even want to lint them in the first place, so this is fine

* missed incrementing total_spent when not incrementing total_spent_paid_credits

* use the credits on self

* use the credits on self (pt 2)

* fix type for 10 cu query

* convert the requestmetadata on the other side of the channel

* logs

* viewing stats is allowed even without a balance

* move paid_credits_used to AuthorizationChecks

* wip

* test_sum_credits_used finally passes

* UserBalanceCache::get_or_insert

* re-enable rpc_secret_key_cache

* move invalidate to a helper function

and always call it **after** the db is commited

* fix PartialEq and Eq on RpcSecretKey

* cargo upgrade
 2023-07-12 00:35:07 -07:00
Bryan Stitt
5f215facab fix typo 2023-07-11 13:18:19 -07:00
Bryan Stitt
8746cc1ae6 catch the new error format for returndata limit 2023-07-10 23:38:23 -07:00
Bryan Stitt
0f7e370c92 drop flume. just use tokio 
i dont know if this is the problem, but this will also get compile times down
 2023-07-10 23:08:06 -07:00
Bryan Stitt
3059d08674 error on max_head_block_age and keep the sender alive longer 2023-07-10 22:17:15 -07:00
Bryan Stitt
04825be99d error if health checks are failing 2023-07-10 20:59:17 -07:00
Bryan Stitt
fe39ad1e1e Revert "Revert "fixes for disconnect and config reloads"" 
This reverts commit 3cfbc3cbf124caf9154ffe6a444082cb0c3e20a1.
 2023-07-10 20:59:03 -07:00
Bryan Stitt
3cfbc3cbf1 Revert "fixes for disconnect and config reloads" 
This reverts commit 50fed0b0e79778d17bb40df96a11e44341dde0d1.
 2023-07-10 20:30:10 -07:00
Bryan Stitt
50fed0b0e7 fixes for disconnect and config reloads 2023-07-10 19:41:14 -07:00
Bryan Stitt
31f772591b put rpcs in a rwlock again 2023-07-10 18:09:58 -07:00
Bryan Stitt
0181347b42 one less clone 2023-07-10 15:41:55 -07:00
Bryan Stitt
e80d1bd23a add more to Web3Rpcs debug 2023-07-10 15:31:10 -07:00
Bryan Stitt
a7f1c043c5 sync before Serialize 2023-07-10 15:26:00 -07:00
Bryan Stitt
9ef08c80e1 remove parking_lot 2023-07-10 11:05:07 -07:00
Bryan Stitt
78c6179f02 clean up rate limit logging 2023-07-07 15:56:46 -07:00
Bryan Stitt
e0c57a22ae derivative and serde didn't work together like we'd hoped 2023-07-06 03:51:39 -07:00
Bryan Stitt
0e3fdfa0ea cache eth_getLogs fromBlock properly 2023-07-05 23:13:42 -07:00
Bryan Stitt
eb7b98fdbe
Split errors (#158) 
* add migration for splitting errors

* fix type from u32 to u64

* update entities to match migrations

* no migration needed. these are only in stats

* add user_error_response to influx

* only if detailed

* set error_response and user_error_response

* 0 cost error responses

* only 33 migrations now

* put macros back

* get the stat buffer sender to the TestApp helper

* fixes
 2023-07-05 18:18:10 -07:00
Bryan Stitt
4a03d2e331 loop until it works 2023-06-29 23:15:57 -07:00
Bryan Stitt
5da334fcb7 start adding tests that need docker for mysql management 2023-06-29 22:38:35 -07:00
Bryan Stitt
335480cec9 more tracing logs 2023-06-29 10:15:12 -07:00
Bryan Stitt
280e2075d3
Fix Issue 65 Pt. 2 (#153) 
* change subuser balance logic and add message to access denied errors

* clearer import

* comments

* comments
 2023-06-28 23:00:34 -07:00
Bryan Stitt
0c389e43b4 don't dedupe rpc 2023-06-28 21:34:51 -07:00
Bryan Stitt
fba41b9bc2 lots of tracing improvements 2023-06-28 21:30:00 -07:00
Bryan Stitt
25b04db3b5 lower log level and optional to 2023-06-28 21:08:43 -07:00
Bryan Stitt
d11d6c1f8a more logs 2023-06-28 18:44:54 -07:00
Bryan Stitt
e8d992168c more logs and retries 2023-06-28 18:36:17 -07:00
Bryan Stitt
3cfbc5baa4 improve cache keys and allow failures while parsing params 2023-06-28 17:42:49 -07:00
Bryan Stitt
df865292a7 add another layer of retries 2023-06-28 15:12:16 -07:00
Bryan Stitt
eab3aa7943 that simpler param doesn't work everywhere 2023-06-27 17:39:37 -07:00
Bryan Stitt
ec7c8f0434 todo 2023-06-27 17:08:49 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
8e98eabebe lower error log when starting 2023-06-27 10:05:07 -07:00
Bryan Stitt
b35ad330c9 save last provider error 2023-06-26 23:04:56 -07:00
Bryan Stitt
917dfc914f handle more jsonrpc errors as HTTP 200 2023-06-26 22:40:00 -07:00
Bryan Stitt
0d1e629567 more tracing and error handling 2023-06-26 21:11:43 -07:00
Bryan Stitt
a440d51377 test fixture 2023-06-24 21:17:08 -07:00
Bryan Stitt
8cfbbfe1b3 use default derive helpers 2023-06-24 20:39:46 -07:00
Bryan Stitt
efe1e30791 better handling of 0 port and fix test port 2023-06-24 20:38:06 -07:00
Bryan Stitt
0c10994835 why wont that import? 2023-06-24 16:45:07 -07:00
Bryan Stitt
94d0a25aaf test logs 2023-06-24 16:43:11 -07:00
Bryan Stitt
1d6e1ef9af polish error logs 2023-06-24 15:24:46 -07:00
Bryan Stitt
fef03f089f less verbose db_conn and db_replica helpers 2023-06-24 11:11:07 -07:00
Bryan Stitt
30828ccae2 prioritize block. queries to syncing nodes ends up being slower 2023-06-24 00:41:30 -07:00
Bryan Stitt
ac4b7e46b4 actually import info level 2023-06-23 22:54:02 -07:00
Bryan Stitt
6edf1a5c1b info on new block 2023-06-23 22:52:42 -07:00
Bryan Stitt
fd661689a3
Tracing (#144) 
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
 2023-06-23 16:28:45 -07:00
Bryan Stitt
8f1b6fe6c0 lint 2023-06-21 10:02:02 -07:00
Bryan Stitt
4dad743fc3 tuple, not list 2023-06-21 10:01:51 -07:00
Bryan Stitt
f75f3f8d2f serialize caches in a list 2023-06-21 10:00:05 -07:00
Bryan Stitt
3c2f95de10
Improve max wait and timeouts (#137) 
* pass max wait with more functions

* move rpc_head_str higher up to use it in all the logs

* refresh consensus at double the block time

* new pricing that also includes archive check

* lint

* don't clone method

* put caches into one list

they serialize with their name, so it will still be easy to tell them apart

* more consistent waiting. still waits more than I want right now

* clean up default max waits

* more timeouts
 2023-06-20 14:22:14 -07:00
Bryan Stitt
b7935f77fc spawn recording latency 2023-06-19 17:46:58 -07:00
Bryan Stitt
c3ae4ded2b move data to the right places 2023-06-18 21:42:16 -07:00
Bryan Stitt
58cc129837 forgot to convert tier_sec_size from ms. 1 was a very large default 2023-06-18 17:57:21 -07:00
Bryan Stitt
ba410cf126 better log message 2023-06-18 16:41:01 -07:00
Bryan Stitt
825ba006f1 move floats and durations around for latency tracking 2023-06-18 10:46:22 -07:00
Bryan Stitt
2f0e6103ac Revert "tier more important than backup" 
This reverts commit bca22537c633822e5e6759fa2840712a837211fd.
 2023-06-18 10:09:21 -07:00
Bryan Stitt
bca22537c6 tier more important than backup 
we filter backups when they aren't needed, so this should be okay
 2023-06-18 10:06:26 -07:00
Bryan Stitt
e54299beca rolling median for choosing tier 2023-06-18 09:53:36 -07:00
Bryan Stitt
8b22c9861b Revert "temp fix while filtering is improved" 
This reverts commit 5367c511953fcea1af3da1d905de06d9e76711ba.
 2023-06-17 12:15:40 -07:00
Bryan Stitt
5367c51195 temp fix while filtering is improved 2023-06-17 12:09:04 -07:00
Bryan Stitt
3b65074f38 cleaner duration multiplication 2023-06-17 11:33:33 -07:00
Bryan Stitt
eab86c7ee8 automatic max_head_block_age 2023-06-17 10:13:40 -07:00
Bryan Stitt
bf13e86a04 warn if no max_head_block_age 2023-06-17 09:19:05 -07:00
Bryan Stitt
3b9314a76f comments 2023-06-17 08:44:18 -07:00
Bryan Stitt
745870d119 drop a bunch of log levels 2023-06-16 23:46:20 -07:00
Bryan Stitt
86f9e7358a
move balance onto AuthorizationChecks (#131) 
* move balance onto AuthorizationChecks

* todone

* bigger default

* flat cost for testing
 2023-06-16 23:14:43 -07:00
Bryan Stitt
3928496be5 start tiers at 1 2023-06-16 10:52:13 -07:00
Bryan Stitt
af7759e491 by_name needs a lock to prevent races during disconnects 2023-06-16 10:40:02 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129) 
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
 2023-06-16 00:46:27 -07:00
Bryan Stitt
b8f429d70a improvements to popularity_contesst 2023-06-13 11:51:19 -07:00
Bryan Stitt
f5a1ac274a refactor send_head_block_result 2023-06-13 10:00:08 -07:00
Bryan Stitt
1473ccb085 gentler parsing of new head blocks 2023-06-13 09:31:22 -07:00
Bryan Stitt
a7fb12bd72 less random is fine for shuffling 2023-06-12 22:42:22 -07:00
Bryan Stitt
55e8471b19 interval instead of watch_blocks 2023-06-12 22:26:10 -07:00
Bryan Stitt
c192e03fa2 count all the tiers at once 2023-06-12 21:03:38 -07:00
Bryan Stitt
bba7ccf7eb change divisor for tier calculation 2023-06-12 20:44:52 -07:00
Bryan Stitt
c771479d94 use WyRand instead of Xoshiro 2023-06-12 19:46:36 -07:00
Bryan Stitt
1a8f799969
we need our own Reconnect (#124) 
* first pass at handling different return data limits

* put ws_provider in an arcswap

* add min max_latency

* add min max_latency

* subscribe with reconnect

* better logging around reconnect

* select on both watches

* subscribe to the correct watch
 2023-06-12 18:58:51 -07:00
Bryan Stitt
f1176685dd pause time on more tests 2023-06-12 11:08:33 -07:00
Bryan Stitt
4a9c1a0ce6 move user registration into a dedicated function 2023-06-09 18:31:47 -07:00
Bryan Stitt
41950c886c improvements for web3-this-then-that 2023-06-09 16:35:14 -07:00
Bryan Stitt
bb900c61a5 todo 2023-06-09 13:31:46 -07:00
Bryan Stitt
8bf0ee473d loud log on rollback 2023-06-09 13:30:00 -07:00
Bryan Stitt
bd87fcb13c move sort and shuffle for loadbalancing into proper functions 2023-06-09 13:09:58 -07:00
Bryan Stitt
1f3040f6c7 automated tiers 2023-06-09 12:21:55 -07:00
Bryan Stitt
c37b320286 add cache names and actually serialize 2023-06-07 17:55:34 -07:00
Bryan Stitt
40bee30730 back to moka 2023-06-07 17:30:05 -07:00
Bryan Stitt
c2a3a4a0a9 Merge remote-tracking branch 'origin/main' into devel 2023-06-07 13:29:12 -07:00
Bryan Stitt
91d288315f add queue lengths to status page 2023-06-07 13:05:15 -07:00
Bryan Stitt
b50579ea3a add block caches to web3rpcs serializer 2023-06-07 12:04:39 -07:00
Bryan Stitt
aa57dd2fcf serialize the caches on /status 2023-06-07 10:58:30 -07:00
Bryan Stitt
f0a9f03076 smaller caches 2023-06-07 10:57:43 -07:00
Bryan Stitt
c4b2ce5266 smaller caches 2023-06-07 10:57:01 -07:00
Bryan Stitt
2b814e7a4d serialize the caches on /status 2023-06-07 10:48:55 -07:00
Bryan Stitt
7947cb95ff use bloom filters and support transactions with multiple deposit events 2023-05-31 11:20:17 -07:00
Bryan Stitt
fd8bba4bdd most of these are for the whole app, not just the frontend 2023-05-30 21:26:11 -07:00
Bryan Stitt
2273637a07 move cli into a separate packge and clean up dependencies 2023-05-30 20:58:24 -07:00
Bryan Stitt
cebe8ed1ea
Request refactor (#99) 
* wip

* AsRef finally works like i wanted

* actually return the block

* start adding async trait

* remove stale import

* include id in the error response when possible

* remove stale comments
 2023-05-30 16:32:34 -07:00
Bryan Stitt
c1eba556a5 fix missing ttl send and add name to cache 2023-05-29 17:19:05 -07:00
Bryan Stitt
bb50efb7f9
quick cache ttl max item size, better function names, and tests (#97) 
* Revert "remove cache on /status and /health for now"

This reverts commit 166b0d810cd25b895560d50d00383a57e1269ad4.

* Revert "remove cache on backups needed, too"

This reverts commit 4597967def2f1daa71a277ec6731db4739ead640.

* add tests

* add max_item_weight

* comment

* add some helper functions, max weights, and tests
 2023-05-29 15:48:22 -07:00
Bryan Stitt
b05d866f8a Merge remote-tracking branch 'origin/main' into devel 2023-05-29 09:42:21 -07:00
Bryan Stitt
dc6e0559f7 improve will_work_eventually 2023-05-29 09:41:19 -07:00
Bryan Stitt
84990bb7b6 check hard_limit_until when finding if an rpc will work 2023-05-29 09:41:19 -07:00
Bryan Stitt
08859ba04d wait for consensus rpcs to change before trying again 2023-05-29 09:41:19 -07:00
Bryan Stitt
7cb07cc49f debug -> trace for some verbose logs 2023-05-27 01:53:49 -07:00
Bryan Stitt
b1a0bcac57 add max wait to fix tests 2023-05-23 20:46:27 -07:00
Bryan Stitt
55f9c57827 include error in warning 2023-05-23 15:03:16 -07:00
Bryan Stitt
dd347fd916 add units to serialized variables 2023-05-23 14:57:24 -07:00
Bryan Stitt
77c9b0ab12 peak latency instead of head latency 2023-05-23 14:52:45 -07:00
Bryan Stitt
651f0fcad3 lint 2023-05-23 14:51:34 -07:00
Bryan Stitt
78f247fc6c split http and ws provider 2023-05-23 14:40:34 -07:00
Bryan Stitt
3ac15558e3 fix decay time 2023-05-22 15:43:39 -07:00
Bryan Stitt
91eeee23e2 use ether's reconnects instead of our own which need a lock (wip) 2023-05-22 15:32:27 -07:00
Bryan Stitt
c443f76623 refactor tiers again 2023-05-19 14:09:03 -07:00
Bryan Stitt
5aae318620 logs and comments 2023-05-18 15:11:29 -07:00
Bryan Stitt
9c584354d9 add to skip list earlier 2023-05-18 13:51:28 -07:00
Bryan Stitt
a92c93706b dont check heads while waiting 
this isn't great. but should work better than what we have
 2023-05-18 13:47:00 -07:00
Bryan Stitt
b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt
1c61390ad1 check if watch_consensus_rpcs.changed has failed 
i dont think it ever will, but better safe than sorry
 2023-05-16 23:42:42 -07:00
Bryan Stitt
89961331af refactor wait_for_block into should_wait_for_block 2023-05-16 22:29:36 -07:00
Bryan Stitt
a07da30042 keep usable_rpcs_by_tier_and_head_number around longer 2023-05-16 18:32:33 -07:00
Bryan Stitt
0f79d2e60d more loud logs 2023-05-16 18:29:56 -07:00
Bryan Stitt
a0d8218eb1 best block num has to check has_block_data 2023-05-16 18:18:56 -07:00
Bryan Stitt
30ea532c70 display, not debug 2023-05-16 16:13:56 -07:00