Commit Graph

200 Commits

Author SHA1 Message Date
Bryan Stitt
b80f994e90 timestamps instead of tags for cardinality reduction 2023-07-25 21:45:01 -07:00
Bryan Stitt
439ac49495 jsonrpc formatted errors for JsonRejection 2023-07-24 16:25:58 -07:00
Bryan Stitt
4091e05f3d influx id from config. default to the hostname or fallback to ulid 2023-07-22 00:26:02 -07:00
Bryan Stitt
185ce7a3b4 flush tsdb every 10 2023-07-21 16:39:27 -07:00
Bryan Stitt
b08a279b60 add another field to try to handle improper de-duplication 2023-07-21 16:29:10 -07:00
Bryan Stitt
620f2d9d00 30 second flushes for db too 2023-07-21 16:12:43 -07:00
Bryan Stitt
0a3698de0f 30 seconds for tsdb everywhere 2023-07-21 15:51:32 -07:00
Bryan Stitt
7fa88e7251 keep the cache size consistent 2023-07-20 22:34:32 -07:00
Bryan Stitt
cb5c1c6582 no limits on rpc_secret_key_cache and user_balance_cache 2023-07-20 19:04:32 -07:00
Bryan Stitt
7678cb7389 caches aren't actualy optional on buffer 2023-07-20 18:21:49 -07:00
Bryan Stitt
ae8a12047e tarpit strange methods 2023-07-20 14:32:52 -07:00
Bryan Stitt
e7905b4344 Revert "First pass implementation of async serialize for Web3Rps (#192)"
This reverts commit 269aa32260a4c3e94d7ada55fee6c8e62d975c87.
2023-07-19 23:12:49 -07:00
Bryan Stitt
ac2d9049be improve flushing and buffering of stats 2023-07-19 22:55:14 -07:00
Bryan Stitt
600c1bafb4
David/multiple proxy test (#187)
* add test_multiple_proxies_stats_add_up

* make a premium user using both proxies

* added a couple clones, must add constraints now to run for multiple-proxies (check arithmetic)

* lint and code review

* fix comment

* fix tests (now fails at todo)

* will introduce endpoint to fetch rpc stats from mysql

* added influxdb to tests, should next do asserst in stats collected by influx, and mysql for multi-proxy

* created test where influx and mysql have separate data, should help with debugging

* forgot to drop influx

* tests pass except multi-proxy

* test passes, will check out nothing broke elswhere

* go back to numbers

* some linting

* linting

* removed redundant info!

* responding to PR comments

* ULID as instance-hash for the tag in influx (for anti-dup)

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
2023-07-17 13:31:41 -07:00
Bryan Stitt
61126bb2ac one more Cow 2023-07-14 18:35:40 -07:00
Bryan Stitt
0046a02a4a
Global db (#191)
* dont migrate on start. just connect

* it compiles

* that log is loud

* get transaction from a local clone

* import the trait

* put the test app into a thread instead of tokio::spawn

* fix one test

* try db before rpcs

* db connection is too slow. need to wait for it

* do db setup once while spawning
2023-07-14 18:30:01 -07:00
Bryan Stitt
269aa32260
First pass implementation of async serialize for Web3Rps (#192)
Co-authored-by: Rory Neithinger <rorytrent@gmail.com>
2023-07-14 18:29:48 -07:00
Bryan Stitt
37e3d0be2b you cant do a runtime in a drop like this apparently 2023-07-14 01:19:27 -07:00
Bryan Stitt
5d5e65ed40 close the db connection on exit 2023-07-13 15:44:07 -07:00
Bryan Stitt
a893a41c90 rename get to connect 2023-07-13 15:44:07 -07:00
Bryan Stitt
dce3540ee3 less authorization and remove pending txs for now 2023-07-13 11:01:16 -07:00
Bryan Stitt
df2f3d340f
More balance tests (#182)
* fix popularity contest

* more info in the Debug for Web3Rpc

* add frontend_requests and cache_misses to the Balance query

* add more to balance and stats flushing and improved test coverage

* it compiles

* deserializer for Ulid to Uuid

I think a wrapper type on Ulid that implements sea_orm::Value is probably better

* rename variable to match struct name

* add deserializer for Address -> Vec<u8>

* sql sum returns a Decimal. need to convert to u64

* assert more

* one log and assert more

* log more

* use a helper to get the user's rpc provider

* this should be 2 now that we have a public and authed call

* this should be zero. the public has the cache miss

* instrument cu calcs

* trace the value we took, not the default that replaced it

* move usd_per_chain into config

* remove some extra logging

* use Arc::into_inner to maybe avoid a race

* off by 1

* pass paid credits used instead of returning it

this lets us use it to write to our user balance cache first. importantly, this keeps us from holding a write lock while writing to mysql

* no cache misses expected in this test

* actually check the admin

* put the balance checks back now that the rest of the test works

* archive request is being set incorrectly

* wow howd we manage flipping the greater than sign on archive depth

* move latest_balance and premium_credits_used to before any stats are emitted

* lint

* and build undoes the linting. fun

i didnt even want to lint them in the first place, so this is fine

* missed incrementing total_spent when not incrementing total_spent_paid_credits

* use the credits on self

* use the credits on self (pt 2)

* fix type for 10 cu query

* convert the requestmetadata on the other side of the channel

* logs

* viewing stats is allowed even without a balance

* move paid_credits_used to AuthorizationChecks

* wip

* test_sum_credits_used finally passes

* UserBalanceCache::get_or_insert

* re-enable rpc_secret_key_cache

* move invalidate to a helper function

and always call it **after** the db is commited

* fix PartialEq and Eq on RpcSecretKey

* cargo upgrade
2023-07-12 00:35:07 -07:00
Bryan Stitt
0f7e370c92 drop flume. just use tokio
i dont know if this is the problem, but this will also get compile times down
2023-07-10 23:08:06 -07:00
Bryan Stitt
3059d08674 error on max_head_block_age and keep the sender alive longer 2023-07-10 22:17:15 -07:00
Bryan Stitt
31f772591b put rpcs in a rwlock again 2023-07-10 18:09:58 -07:00
Bryan Stitt
b234265458
Premium tier changes (#170)
* lower premium concurrency in preparation for key+IP limits

* include the ip in the user semaphore

* 3, not 5

this is our current limit for free

* per user_id+ip rate limiting
2023-07-07 15:15:41 -07:00
Bryan Stitt
57f640765a remove a level of retries 2023-07-07 14:06:44 -07:00
Bryan Stitt
3bbbdd5596
Flush stats (#161)
* create buffer if mysql OR influx is set up

* this one flush should error

* it helps to set up the db in the db migration test

* comment
2023-07-05 19:24:21 -07:00
Bryan Stitt
eb7b98fdbe
Split errors (#158)
* add migration for splitting errors

* fix type from u32 to u64

* update entities to match migrations

* no migration needed. these are only in stats

* add user_error_response to influx

* only if detailed

* set error_response and user_error_response

* 0 cost error responses

* only 33 migrations now

* put macros back

* get the stat buffer sender to the TestApp helper

* fixes
2023-07-05 18:18:10 -07:00
Bryan Stitt
a64e32b085 remove bearer token semaphore. it isn't really necesary
we can come back to rate limiting authenticated endpoints later
2023-07-03 16:54:25 -07:00
Bryan Stitt
357701bccc no need to check the caches, we have latest block 2023-06-29 00:06:58 -07:00
Bryan Stitt
bfc663d678 fix eth_blockNumber formatting 2023-06-28 23:00:48 -07:00
Bryan Stitt
280e2075d3
Fix Issue 65 Pt. 2 (#153)
* change subuser balance logic and add message to access denied errors

* clearer import

* comments

* comments
2023-06-28 23:00:34 -07:00
Bryan Stitt
ebceb0d7d6 descriptive text on NotImplemented errors 2023-06-28 21:47:06 -07:00
Bryan Stitt
fba41b9bc2 lots of tracing improvements 2023-06-28 21:30:00 -07:00
Bryan Stitt
2a714acc30 deref 2023-06-28 19:19:36 -07:00
Bryan Stitt
c6930dcb4c saturating sub 2023-06-28 19:19:14 -07:00
Bryan Stitt
3cfbc5baa4 improve cache keys and allow failures while parsing params 2023-06-28 17:42:49 -07:00
Bryan Stitt
df865292a7 add another layer of retries 2023-06-28 15:12:16 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
b35ad330c9 save last provider error 2023-06-26 23:04:56 -07:00
Bryan Stitt
009fdbe2cf make references compile 2023-06-26 21:38:41 -07:00
Bryan Stitt
f68964f63d add 3 tries 2023-06-26 21:08:32 -07:00
Bryan Stitt
efe1e30791 better handling of 0 port and fix test port 2023-06-24 20:38:06 -07:00
Bryan Stitt
fef03f089f less verbose db_conn and db_replica helpers 2023-06-24 11:11:07 -07:00
Bryan Stitt
fd661689a3
Tracing (#144)
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
2023-06-23 16:28:45 -07:00
Bryan Stitt
3c2f95de10
Improve max wait and timeouts (#137)
* pass max wait with more functions

* move rpc_head_str higher up to use it in all the logs

* refresh consensus at double the block time

* new pricing that also includes archive check

* lint

* don't clone method

* put caches into one list

they serialize with their name, so it will still be easy to tell them apart

* more consistent waiting. still waits more than I want right now

* clean up default max waits

* more timeouts
2023-06-20 14:22:14 -07:00
Bryan Stitt
825ba006f1 move floats and durations around for latency tracking 2023-06-18 10:46:22 -07:00
Bryan Stitt
eab86c7ee8 automatic max_head_block_age 2023-06-17 10:13:40 -07:00
Bryan Stitt
bf13e86a04 warn if no max_head_block_age 2023-06-17 09:19:05 -07:00