Commit Graph

127 Commits

Author SHA1 Message Date
Bryan Stitt
1fd8f6f383
Deduped broadcast channel (#209)
* upgrade and start adding deduped_broadcast

* basic test

* accept arg for cache ttl

* first pass at bringing the tx firehose back

* try_send instead of send since there aren't always receivers

* deduped broadcaster needs the try send

* track counts and include in /status

* use config for enable subscribe_txs
2023-09-13 12:05:47 -07:00
Bryan Stitt
ceb202b53c allow the string 'Archive' when setting block data limit 2023-09-12 13:35:51 -07:00
Bryan Stitt
bc8040f61d more yields and less timeouts 2023-08-04 18:54:16 -07:00
Bryan Stitt
0bc523ab9f improve hard limits
rate limit per server or per endpoint and with any period
2023-08-03 12:24:28 -07:00
Bryan Stitt
74224977b7 Revert "Revert "remove max_tries for now""
This reverts commit 5fbcb75157cd012b723c9d2b37bb7947e5e05aa5.
2023-08-03 09:15:31 -07:00
Bryan Stitt
5fbcb75157 Revert "remove max_tries for now"
This reverts commit e0d7d11398163bb8554cd6736d1dbc7f55caaa24.
2023-08-03 00:22:45 -07:00
Bryan Stitt
e0d7d11398 remove max_tries for now
i think this is causing problematic looping
2023-08-02 22:07:28 -07:00
Bryan Stitt
a68d0d60b7 only health check if theres a block and rpc sender 2023-07-21 13:28:00 -07:00
Bryan Stitt
61126bb2ac one more Cow 2023-07-14 18:35:40 -07:00
Bryan Stitt
0046a02a4a
Global db (#191)
* dont migrate on start. just connect

* it compiles

* that log is loud

* get transaction from a local clone

* import the trait

* put the test app into a thread instead of tokio::spawn

* fix one test

* try db before rpcs

* db connection is too slow. need to wait for it

* do db setup once while spawning
2023-07-14 18:30:01 -07:00
Bryan Stitt
dce3540ee3 less authorization and remove pending txs for now 2023-07-13 11:01:16 -07:00
Bryan Stitt
df2f3d340f
More balance tests (#182)
* fix popularity contest

* more info in the Debug for Web3Rpc

* add frontend_requests and cache_misses to the Balance query

* add more to balance and stats flushing and improved test coverage

* it compiles

* deserializer for Ulid to Uuid

I think a wrapper type on Ulid that implements sea_orm::Value is probably better

* rename variable to match struct name

* add deserializer for Address -> Vec<u8>

* sql sum returns a Decimal. need to convert to u64

* assert more

* one log and assert more

* log more

* use a helper to get the user's rpc provider

* this should be 2 now that we have a public and authed call

* this should be zero. the public has the cache miss

* instrument cu calcs

* trace the value we took, not the default that replaced it

* move usd_per_chain into config

* remove some extra logging

* use Arc::into_inner to maybe avoid a race

* off by 1

* pass paid credits used instead of returning it

this lets us use it to write to our user balance cache first. importantly, this keeps us from holding a write lock while writing to mysql

* no cache misses expected in this test

* actually check the admin

* put the balance checks back now that the rest of the test works

* archive request is being set incorrectly

* wow howd we manage flipping the greater than sign on archive depth

* move latest_balance and premium_credits_used to before any stats are emitted

* lint

* and build undoes the linting. fun

i didnt even want to lint them in the first place, so this is fine

* missed incrementing total_spent when not incrementing total_spent_paid_credits

* use the credits on self

* use the credits on self (pt 2)

* fix type for 10 cu query

* convert the requestmetadata on the other side of the channel

* logs

* viewing stats is allowed even without a balance

* move paid_credits_used to AuthorizationChecks

* wip

* test_sum_credits_used finally passes

* UserBalanceCache::get_or_insert

* re-enable rpc_secret_key_cache

* move invalidate to a helper function

and always call it **after** the db is commited

* fix PartialEq and Eq on RpcSecretKey

* cargo upgrade
2023-07-12 00:35:07 -07:00
Bryan Stitt
0f7e370c92 drop flume. just use tokio
i dont know if this is the problem, but this will also get compile times down
2023-07-10 23:08:06 -07:00
Bryan Stitt
3059d08674 error on max_head_block_age and keep the sender alive longer 2023-07-10 22:17:15 -07:00
Bryan Stitt
04825be99d error if health checks are failing 2023-07-10 20:59:17 -07:00
Bryan Stitt
fe39ad1e1e Revert "Revert "fixes for disconnect and config reloads""
This reverts commit 3cfbc3cbf124caf9154ffe6a444082cb0c3e20a1.
2023-07-10 20:59:03 -07:00
Bryan Stitt
3cfbc3cbf1 Revert "fixes for disconnect and config reloads"
This reverts commit 50fed0b0e79778d17bb40df96a11e44341dde0d1.
2023-07-10 20:30:10 -07:00
Bryan Stitt
50fed0b0e7 fixes for disconnect and config reloads 2023-07-10 19:41:14 -07:00
Bryan Stitt
9ef08c80e1 remove parking_lot 2023-07-10 11:05:07 -07:00
Bryan Stitt
78c6179f02 clean up rate limit logging 2023-07-07 15:56:46 -07:00
Bryan Stitt
280e2075d3
Fix Issue 65 Pt. 2 (#153)
* change subuser balance logic and add message to access denied errors

* clearer import

* comments

* comments
2023-06-28 23:00:34 -07:00
Bryan Stitt
0c389e43b4 don't dedupe rpc 2023-06-28 21:34:51 -07:00
Bryan Stitt
fba41b9bc2 lots of tracing improvements 2023-06-28 21:30:00 -07:00
Bryan Stitt
25b04db3b5 lower log level and optional to 2023-06-28 21:08:43 -07:00
Bryan Stitt
d11d6c1f8a more logs 2023-06-28 18:44:54 -07:00
Bryan Stitt
e8d992168c more logs and retries 2023-06-28 18:36:17 -07:00
Bryan Stitt
df865292a7 add another layer of retries 2023-06-28 15:12:16 -07:00
Bryan Stitt
eab3aa7943 that simpler param doesn't work everywhere 2023-06-27 17:39:37 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
a440d51377 test fixture 2023-06-24 21:17:08 -07:00
Bryan Stitt
30828ccae2 prioritize block. queries to syncing nodes ends up being slower 2023-06-24 00:41:30 -07:00
Bryan Stitt
fd661689a3
Tracing (#144)
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
2023-06-23 16:28:45 -07:00
Bryan Stitt
3c2f95de10
Improve max wait and timeouts (#137)
* pass max wait with more functions

* move rpc_head_str higher up to use it in all the logs

* refresh consensus at double the block time

* new pricing that also includes archive check

* lint

* don't clone method

* put caches into one list

they serialize with their name, so it will still be easy to tell them apart

* more consistent waiting. still waits more than I want right now

* clean up default max waits

* more timeouts
2023-06-20 14:22:14 -07:00
Bryan Stitt
825ba006f1 move floats and durations around for latency tracking 2023-06-18 10:46:22 -07:00
Bryan Stitt
2f0e6103ac Revert "tier more important than backup"
This reverts commit bca22537c633822e5e6759fa2840712a837211fd.
2023-06-18 10:09:21 -07:00
Bryan Stitt
bca22537c6 tier more important than backup
we filter backups when they aren't needed, so this should be okay
2023-06-18 10:06:26 -07:00
Bryan Stitt
e54299beca rolling median for choosing tier 2023-06-18 09:53:36 -07:00
Bryan Stitt
8b22c9861b Revert "temp fix while filtering is improved"
This reverts commit 5367c511953fcea1af3da1d905de06d9e76711ba.
2023-06-17 12:15:40 -07:00
Bryan Stitt
5367c51195 temp fix while filtering is improved 2023-06-17 12:09:04 -07:00
Bryan Stitt
745870d119 drop a bunch of log levels 2023-06-16 23:46:20 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129)
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
2023-06-16 00:46:27 -07:00
Bryan Stitt
b8f429d70a improvements to popularity_contesst 2023-06-13 11:51:19 -07:00
Bryan Stitt
f5a1ac274a refactor send_head_block_result 2023-06-13 10:00:08 -07:00
Bryan Stitt
1473ccb085 gentler parsing of new head blocks 2023-06-13 09:31:22 -07:00
Bryan Stitt
a7fb12bd72 less random is fine for shuffling 2023-06-12 22:42:22 -07:00
Bryan Stitt
55e8471b19 interval instead of watch_blocks 2023-06-12 22:26:10 -07:00
Bryan Stitt
bba7ccf7eb change divisor for tier calculation 2023-06-12 20:44:52 -07:00
Bryan Stitt
c771479d94 use WyRand instead of Xoshiro 2023-06-12 19:46:36 -07:00
Bryan Stitt
1a8f799969
we need our own Reconnect (#124)
* first pass at handling different return data limits

* put ws_provider in an arcswap

* add min max_latency

* add min max_latency

* subscribe with reconnect

* better logging around reconnect

* select on both watches

* subscribe to the correct watch
2023-06-12 18:58:51 -07:00
Bryan Stitt
bd87fcb13c move sort and shuffle for loadbalancing into proper functions 2023-06-09 13:09:58 -07:00