Commit Graph

165 Commits

Author SHA1 Message Date
Bryan Stitt
ba7eba41db allow_unhealthy should also skip checking block heights 2023-10-13 20:39:34 -07:00
Bryan Stitt
8067c33079 add todo 2023-10-13 00:46:46 -07:00
Bryan Stitt
fc3d2420bd bring back head_delay 2023-10-12 23:33:25 -07:00
Bryan Stitt
93533eabc7 Revert "try a RwLock (more should be done if this works)"
This reverts commit 9aa33dae2d3869965ba131e8e561338f2298fd56.
2023-10-12 23:33:25 -07:00
Bryan Stitt
3b45256c9a always have some kind of health check 2023-10-12 21:05:17 -07:00
Bryan Stitt
51d86ade59 handle empty futures 2023-10-12 20:48:37 -07:00
Bryan Stitt
3bf7f6c34e replace subscribe_stop_rx with aborts 2023-10-12 20:17:40 -07:00
Bryan Stitt
9786fb58b2 use select_all more 2023-10-12 19:23:22 -07:00
Bryan Stitt
9aa33dae2d try a RwLock (more should be done if this works) 2023-10-11 10:55:45 -07:00
Bryan Stitt
e43309abc0 move permits around and start rpcs slower 2023-10-11 04:18:53 -07:00
Bryan Stitt
072c52050f add another minimum sleep and set missed tick behavior 2023-10-11 01:42:55 -07:00
Bryan Stitt
15216b7d33 no waiting and simpler rpc looping 2023-10-11 01:23:08 -07:00
Bryan Stitt
91a9d62ce1 pass head_block to internal_request 2023-10-11 00:48:08 -07:00
Bryan Stitt
57f8c4bd40 split web3_proxy/src/jsonrpc.rs into multiple files and DRY it up 2023-10-11 00:12:20 -07:00
Bryan Stitt
9ba2045dbd improve health checking 2023-10-10 15:46:11 -07:00
Bryan Stitt
78edfee6b9 timeout and server selection fixes
add a short connect timeout separate from the overall request timeout.

also fix a bug when only 1 server was in the rpc list causing a very tight loop that made tokio sad
2023-10-09 23:30:17 -07:00
Bryan Stitt
36ce2f6524 derp 2023-10-07 16:16:42 -07:00
Bryan Stitt
7e0b6c3034 next_available should not be reversed and other bugs 2023-10-07 16:12:51 -07:00
Bryan Stitt
5a1a87a314 stream should count as success. lagged return the rpc 2023-10-07 15:18:55 -07:00
Bryan Stitt
9ed0c70ecd wait for lagged rpcs. v1.43.19 2023-10-07 11:58:47 -07:00
Bryan Stitt
349be463f3 initial sort on median latency instead of weighted peak 2023-10-07 10:31:33 -07:00
Bryan Stitt
cc2eb9e862 no more overflow on the pending txid firehose 2023-10-07 03:41:11 -07:00
Bryan Stitt
36363c8921 lint 2023-10-07 00:35:22 -07:00
Bryan Stitt
d38a35903e cargo update 2023-10-07 00:35:22 -07:00
Bryan Stitt
97f7e1673e finish todo for waiting for head_block_sender 2023-10-07 00:35:22 -07:00
Bryan Stitt
159fef9e8d include ws_url in hash 2023-10-07 00:35:22 -07:00
Bryan Stitt
949c3eeb5d DRY throttling 2023-10-07 00:35:22 -07:00
Bryan Stitt
9a7777e3f9 fix >= to < 2023-10-07 00:35:22 -07:00
Bryan Stitt
c1df05f8b6 fix sort order (hopefully) 2023-10-07 00:35:22 -07:00
Bryan Stitt
0dcc324b61 disable tests that need refactor to match new code 2023-10-07 00:35:22 -07:00
Bryan Stitt
6756559ec2 wow this got big 2023-10-07 00:35:22 -07:00
Bryan Stitt
9258d36e6a refactor rpc selection 2023-10-07 00:35:22 -07:00
Bryan Stitt
e917a11d6c
Suprisingly large refactor to get ids everywhere (#222)
* cargo upgrade --incompatible and update

* first draft at suprisingly_large_refactor_to_get_ids_everywhere

* put app in a task_local

* ref cleanup

* use a OnceLock instead of a tokio local

* test more methods

* APP isn't set in all tests

* it compiles. tests fail. todos still open

* use the app only when necessary

* more tests. less panic

* less verbose debug impl

* short enum names

* move kafka and secrets to their own files

* main tests pass

* add debug chain block time

* helper for stats that ignores internal stats

* Update Jenkinsfile (#223)

* more tests

---------

Co-authored-by: Pewxz <124064710+pewxz@users.noreply.github.com>
2023-10-03 13:46:27 -07:00
Bryan Stitt
f0b6465069
streaming responses 2 (#219)
* streaming responses compile, still small TODOs & cleanup

* refactor to allow short curcuits on uncacheable requests (#220)

* refactor to allow short curcuits on uncacheable requests

* handle error inside an Arc from moka

* arc instead of box

* lint

* more lint

* add bonus rate limits and keep all semaphores after rate limits (#221)

* add bonus rate limits and keep all semaphores after rate limits

* remove stale todos

* better function names

* cargo upgrade and update

* make some panics warn. more todo

* pass request_metadata through so streaming responses include their final size

* remove stale TODO

* remove more stale todos

* add file i missed

* make to_json_string work well enough

* check config for free_subscriptions

---------

Co-authored-by: Rory Neithinger <rory@llamanodes.com>
2023-09-26 18:18:06 -07:00
Bryan Stitt
9b2f27862d remove unnecessary yield_nows 2023-09-21 11:22:20 -07:00
Bryan Stitt
fc287d24ea borrow_and_update instead of borrow
i think this won't matter and the update is probably a waste, but it feels wrong to leave it in a changed state
2023-09-18 23:29:01 -07:00
Bryan Stitt
357dae108a loop, dont crash 2023-09-14 12:51:40 -07:00
Bryan Stitt
8e5a27a184 use Ordering::Relaxed more 2023-09-13 12:35:58 -07:00
Bryan Stitt
1fd8f6f383
Deduped broadcast channel (#209)
* upgrade and start adding deduped_broadcast

* basic test

* accept arg for cache ttl

* first pass at bringing the tx firehose back

* try_send instead of send since there aren't always receivers

* deduped broadcaster needs the try send

* track counts and include in /status

* use config for enable subscribe_txs
2023-09-13 12:05:47 -07:00
Bryan Stitt
ceb202b53c allow the string 'Archive' when setting block data limit 2023-09-12 13:35:51 -07:00
Bryan Stitt
bc8040f61d more yields and less timeouts 2023-08-04 18:54:16 -07:00
Bryan Stitt
0bc523ab9f improve hard limits
rate limit per server or per endpoint and with any period
2023-08-03 12:24:28 -07:00
Bryan Stitt
74224977b7 Revert "Revert "remove max_tries for now""
This reverts commit 5fbcb75157cd012b723c9d2b37bb7947e5e05aa5.
2023-08-03 09:15:31 -07:00
Bryan Stitt
5fbcb75157 Revert "remove max_tries for now"
This reverts commit e0d7d11398163bb8554cd6736d1dbc7f55caaa24.
2023-08-03 00:22:45 -07:00
Bryan Stitt
e0d7d11398 remove max_tries for now
i think this is causing problematic looping
2023-08-02 22:07:28 -07:00
Bryan Stitt
a68d0d60b7 only health check if theres a block and rpc sender 2023-07-21 13:28:00 -07:00
Bryan Stitt
61126bb2ac one more Cow 2023-07-14 18:35:40 -07:00
Bryan Stitt
0046a02a4a
Global db (#191)
* dont migrate on start. just connect

* it compiles

* that log is loud

* get transaction from a local clone

* import the trait

* put the test app into a thread instead of tokio::spawn

* fix one test

* try db before rpcs

* db connection is too slow. need to wait for it

* do db setup once while spawning
2023-07-14 18:30:01 -07:00
Bryan Stitt
dce3540ee3 less authorization and remove pending txs for now 2023-07-13 11:01:16 -07:00
Bryan Stitt
df2f3d340f
More balance tests (#182)
* fix popularity contest

* more info in the Debug for Web3Rpc

* add frontend_requests and cache_misses to the Balance query

* add more to balance and stats flushing and improved test coverage

* it compiles

* deserializer for Ulid to Uuid

I think a wrapper type on Ulid that implements sea_orm::Value is probably better

* rename variable to match struct name

* add deserializer for Address -> Vec<u8>

* sql sum returns a Decimal. need to convert to u64

* assert more

* one log and assert more

* log more

* use a helper to get the user's rpc provider

* this should be 2 now that we have a public and authed call

* this should be zero. the public has the cache miss

* instrument cu calcs

* trace the value we took, not the default that replaced it

* move usd_per_chain into config

* remove some extra logging

* use Arc::into_inner to maybe avoid a race

* off by 1

* pass paid credits used instead of returning it

this lets us use it to write to our user balance cache first. importantly, this keeps us from holding a write lock while writing to mysql

* no cache misses expected in this test

* actually check the admin

* put the balance checks back now that the rest of the test works

* archive request is being set incorrectly

* wow howd we manage flipping the greater than sign on archive depth

* move latest_balance and premium_credits_used to before any stats are emitted

* lint

* and build undoes the linting. fun

i didnt even want to lint them in the first place, so this is fine

* missed incrementing total_spent when not incrementing total_spent_paid_credits

* use the credits on self

* use the credits on self (pt 2)

* fix type for 10 cu query

* convert the requestmetadata on the other side of the channel

* logs

* viewing stats is allowed even without a balance

* move paid_credits_used to AuthorizationChecks

* wip

* test_sum_credits_used finally passes

* UserBalanceCache::get_or_insert

* re-enable rpc_secret_key_cache

* move invalidate to a helper function

and always call it **after** the db is commited

* fix PartialEq and Eq on RpcSecretKey

* cargo upgrade
2023-07-12 00:35:07 -07:00