Commit Graph

481 Commits

Author SHA1 Message Date
Bryan Stitt
f0b6465069
streaming responses 2 (#219)
* streaming responses compile, still small TODOs & cleanup

* refactor to allow short curcuits on uncacheable requests (#220)

* refactor to allow short curcuits on uncacheable requests

* handle error inside an Arc from moka

* arc instead of box

* lint

* more lint

* add bonus rate limits and keep all semaphores after rate limits (#221)

* add bonus rate limits and keep all semaphores after rate limits

* remove stale todos

* better function names

* cargo upgrade and update

* make some panics warn. more todo

* pass request_metadata through so streaming responses include their final size

* remove stale TODO

* remove more stale todos

* add file i missed

* make to_json_string work well enough

* check config for free_subscriptions

---------

Co-authored-by: Rory Neithinger <rory@llamanodes.com>
2023-09-26 18:18:06 -07:00
Bryan Stitt
9b2f27862d remove unnecessary yield_nows 2023-09-21 11:22:20 -07:00
Bryan Stitt
f0be8b992b always wait for watch_ranked_rpcs change 2023-09-19 14:42:31 -07:00
Bryan Stitt
41314df25b move another yield_now 2023-09-18 23:42:54 -07:00
Bryan Stitt
fc287d24ea borrow_and_update instead of borrow
i think this won't matter and the update is probably a waste, but it feels wrong to leave it in a changed state
2023-09-18 23:29:01 -07:00
Bryan Stitt
a68a98ce41 cargo upgrade and update
moka 0.12 required some migration help: https://github.com/moka-rs/moka/blob/main/MIGRATION-GUIDE.md
2023-09-18 11:04:22 -07:00
Bryan Stitt
357dae108a loop, dont crash 2023-09-14 12:51:40 -07:00
Bryan Stitt
8e5a27a184 use Ordering::Relaxed more 2023-09-13 12:35:58 -07:00
Bryan Stitt
1fd8f6f383
Deduped broadcast channel (#209)
* upgrade and start adding deduped_broadcast

* basic test

* accept arg for cache ttl

* first pass at bringing the tx firehose back

* try_send instead of send since there aren't always receivers

* deduped broadcaster needs the try send

* track counts and include in /status

* use config for enable subscribe_txs
2023-09-13 12:05:47 -07:00
Bryan Stitt
ceb202b53c allow the string 'Archive' when setting block data limit 2023-09-12 13:35:51 -07:00
Bryan Stitt
633abc0d60 make wait_for_best_rpc work for private/4337/groups without head subscriptions 2023-09-02 10:59:07 -07:00
Bryan Stitt
2cf688cc68 fix one syntax error 2023-09-02 10:38:26 -07:00
Bryan Stitt
428dfdd835 improve 4337 bundler support and retries 2023-09-02 10:19:43 -07:00
Bryan Stitt
a487ce5149 skip backups unless backups are needed for ranked_rpcs to exist 2023-09-01 17:01:59 -07:00
Bryan Stitt
8b4aba592c cargo fmt 2023-08-22 10:49:04 -07:00
Bryan Stitt
36eb520f84 rename block_ to block_and_rpc_ where applicable 2023-08-16 13:20:22 -07:00
Bryan Stitt
b742a25729 dont return max depth so we can do cannonical block faster 2023-08-15 16:12:40 -07:00
Bryan Stitt
bc8040f61d more yields and less timeouts 2023-08-04 18:54:16 -07:00
Bryan Stitt
aed98ac034 try yielding around all selects 2023-08-04 18:37:46 -07:00
Bryan Stitt
0bc523ab9f improve hard limits
rate limit per server or per endpoint and with any period
2023-08-03 12:24:28 -07:00
Bryan Stitt
690601643d improve eth_sendRawTransaction and other timeouts 2023-08-03 11:39:48 -07:00
Bryan Stitt
74224977b7 Revert "Revert "remove max_tries for now""
This reverts commit 5fbcb75157cd012b723c9d2b37bb7947e5e05aa5.
2023-08-03 09:15:31 -07:00
Bryan Stitt
ca802751c1 if block fails, try other rpcs 2023-08-03 00:29:13 -07:00
Bryan Stitt
5747caec71 break if no max_wait 2023-08-03 00:23:44 -07:00
Bryan Stitt
5fbcb75157 Revert "remove max_tries for now"
This reverts commit e0d7d11398163bb8554cd6736d1dbc7f55caaa24.
2023-08-03 00:22:45 -07:00
Bryan Stitt
e0d7d11398 remove max_tries for now
i think this is causing problematic looping
2023-08-02 22:07:28 -07:00
Bryan Stitt
8aac4779bf clean up some selects 2023-08-02 21:27:56 -07:00
Bryan Stitt
32a73a0c53 cargo upgrade --incompatible 2023-07-27 12:29:10 -07:00
Bryan Stitt
b1f447c5c8
add mass_grant_credits helper script and other prep for premium (#196)
* add mass_grant_credits helper script

* include error in the rate limit log

* allow granting 0 credits

* migration to change default user_tier to premium
2023-07-26 07:33:00 -07:00
Bryan Stitt
a68d0d60b7 only health check if theres a block and rpc sender 2023-07-21 13:28:00 -07:00
Bryan Stitt
ae8a12047e tarpit strange methods 2023-07-20 14:32:52 -07:00
Bryan Stitt
e7905b4344 Revert "First pass implementation of async serialize for Web3Rps (#192)"
This reverts commit 269aa32260a4c3e94d7ada55fee6c8e62d975c87.
2023-07-19 23:12:49 -07:00
Bryan Stitt
264508c454 treat serde errors as rate limits (because all of them seem to be) 2023-07-14 21:43:37 -07:00
Bryan Stitt
61126bb2ac one more Cow 2023-07-14 18:35:40 -07:00
Bryan Stitt
020dff0233 lint 2023-07-14 18:31:42 -07:00
Bryan Stitt
efa189d101 less anyhow and fix tests 2023-07-14 18:31:00 -07:00
Bryan Stitt
0046a02a4a
Global db (#191)
* dont migrate on start. just connect

* it compiles

* that log is loud

* get transaction from a local clone

* import the trait

* put the test app into a thread instead of tokio::spawn

* fix one test

* try db before rpcs

* db connection is too slow. need to wait for it

* do db setup once while spawning
2023-07-14 18:30:01 -07:00
Bryan Stitt
269aa32260
First pass implementation of async serialize for Web3Rps (#192)
Co-authored-by: Rory Neithinger <rorytrent@gmail.com>
2023-07-14 18:29:48 -07:00
Bryan Stitt
3c2d06324e faster rpc. backups always lose 2023-07-13 18:14:13 -07:00
Bryan Stitt
dce3540ee3 less authorization and remove pending txs for now 2023-07-13 11:01:16 -07:00
Bryan Stitt
df2f3d340f
More balance tests (#182)
* fix popularity contest

* more info in the Debug for Web3Rpc

* add frontend_requests and cache_misses to the Balance query

* add more to balance and stats flushing and improved test coverage

* it compiles

* deserializer for Ulid to Uuid

I think a wrapper type on Ulid that implements sea_orm::Value is probably better

* rename variable to match struct name

* add deserializer for Address -> Vec<u8>

* sql sum returns a Decimal. need to convert to u64

* assert more

* one log and assert more

* log more

* use a helper to get the user's rpc provider

* this should be 2 now that we have a public and authed call

* this should be zero. the public has the cache miss

* instrument cu calcs

* trace the value we took, not the default that replaced it

* move usd_per_chain into config

* remove some extra logging

* use Arc::into_inner to maybe avoid a race

* off by 1

* pass paid credits used instead of returning it

this lets us use it to write to our user balance cache first. importantly, this keeps us from holding a write lock while writing to mysql

* no cache misses expected in this test

* actually check the admin

* put the balance checks back now that the rest of the test works

* archive request is being set incorrectly

* wow howd we manage flipping the greater than sign on archive depth

* move latest_balance and premium_credits_used to before any stats are emitted

* lint

* and build undoes the linting. fun

i didnt even want to lint them in the first place, so this is fine

* missed incrementing total_spent when not incrementing total_spent_paid_credits

* use the credits on self

* use the credits on self (pt 2)

* fix type for 10 cu query

* convert the requestmetadata on the other side of the channel

* logs

* viewing stats is allowed even without a balance

* move paid_credits_used to AuthorizationChecks

* wip

* test_sum_credits_used finally passes

* UserBalanceCache::get_or_insert

* re-enable rpc_secret_key_cache

* move invalidate to a helper function

and always call it **after** the db is commited

* fix PartialEq and Eq on RpcSecretKey

* cargo upgrade
2023-07-12 00:35:07 -07:00
Bryan Stitt
5f215facab fix typo 2023-07-11 13:18:19 -07:00
Bryan Stitt
8746cc1ae6 catch the new error format for returndata limit 2023-07-10 23:38:23 -07:00
Bryan Stitt
0f7e370c92 drop flume. just use tokio
i dont know if this is the problem, but this will also get compile times down
2023-07-10 23:08:06 -07:00
Bryan Stitt
3059d08674 error on max_head_block_age and keep the sender alive longer 2023-07-10 22:17:15 -07:00
Bryan Stitt
04825be99d error if health checks are failing 2023-07-10 20:59:17 -07:00
Bryan Stitt
fe39ad1e1e Revert "Revert "fixes for disconnect and config reloads""
This reverts commit 3cfbc3cbf124caf9154ffe6a444082cb0c3e20a1.
2023-07-10 20:59:03 -07:00
Bryan Stitt
3cfbc3cbf1 Revert "fixes for disconnect and config reloads"
This reverts commit 50fed0b0e79778d17bb40df96a11e44341dde0d1.
2023-07-10 20:30:10 -07:00
Bryan Stitt
50fed0b0e7 fixes for disconnect and config reloads 2023-07-10 19:41:14 -07:00
Bryan Stitt
31f772591b put rpcs in a rwlock again 2023-07-10 18:09:58 -07:00