tornadocontrib/web3-proxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,253 Commits 1 Branch 0 Tags 12 MiB
b5d34b0fad
Commit Graph

1530 Commits

Author SHA1 Message Date
yenicelik
af45f78ff5 added test for admin deposit endpoint, fixed bugs 2023-07-06 08:19:33 -04:00
Bryan Stitt
fc9b1dd69e fix feature name 2023-07-06 04:37:49 -07:00
Bryan Stitt
ffef36dafe more tracing cleanup 2023-07-06 04:36:02 -07:00
Bryan Stitt
a0b256cbd2 panic on deadlock 2023-07-06 04:18:48 -07:00
Bryan Stitt
5d3751ea88 another test 2023-07-06 04:04:12 -07:00
Bryan Stitt
0e51a7ca2e roundabout Default 2023-07-06 04:02:30 -07:00
Bryan Stitt
e0c57a22ae derivative and serde didn't work together like we'd hoped 2023-07-06 03:51:39 -07:00
Bryan Stitt
0e3fdfa0ea cache eth_getLogs fromBlock properly 2023-07-05 23:13:42 -07:00
Bryan Stitt
4ae4d7a75e include default sample rate 2023-07-05 22:34:10 -07:00
David
8d31680ff1
David/referral test new (#162) 
* working on test for referral

* refactored logic in tests to helper functions

* added getbalance helper

* added more helper functions to test for referral

* gotta be able to submit to rpc now

* adding provider instead

* referral tests are not passing, may be because we need for stats to be updated

* balance doesnt decrease

* will look into deposits in the meantime

* finding some first bugs

* seems to have fixed a bunch of stuff

* feedback, lint, and make it compile

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
 2023-07-05 22:18:39 -07:00
Bryan Stitt
cc0ce1611a cargo upgrade 2023-07-05 21:02:30 -07:00
Bryan Stitt
04d58a68dd fix version number to match migration count 2023-07-05 20:58:24 -07:00
David
659fce0e33
Deposit history for stripe and admin (endpoints) (#160) 
* added endpoints for stripe and admin deposits

* modified comments
 2023-07-05 19:26:03 -07:00
Bryan Stitt
3bbbdd5596
Flush stats (#161) 
* create buffer if mysql OR influx is set up

* this one flush should error

* it helps to set up the db in the db migration test

* comment
 2023-07-05 19:24:21 -07:00
Bryan Stitt
eb7b98fdbe
Split errors (#158) 
* add migration for splitting errors

* fix type from u32 to u64

* update entities to match migrations

* no migration needed. these are only in stats

* add user_error_response to influx

* only if detailed

* set error_response and user_error_response

* 0 cost error responses

* only 33 migrations now

* put macros back

* get the stat buffer sender to the TestApp helper

* fixes
 2023-07-05 18:18:10 -07:00
Bryan Stitt
f8d824330b cargo upgrade 2023-07-05 11:55:17 -07:00
Bryan Stitt
97d3faa3d7 wait up to 35 seconds to start frontend 2023-07-05 11:29:56 -07:00
Bryan Stitt
21b59bb953 rename stripe_api_key to stripe_whsec_key 2023-07-05 11:26:39 -07:00
Bryan Stitt
477f69c6e9 use derivative+serde default instead of just serde default 2023-07-03 20:38:39 -07:00
Bryan Stitt
4fa28608b1 cargo upgrade 2023-07-03 16:55:44 -07:00
Bryan Stitt
a64e32b085 remove bearer token semaphore. it isn't really necesary 
we can come back to rate limiting authenticated endpoints later
 2023-07-03 16:54:25 -07:00
Bryan Stitt
b119dc8a92 comment cleanup 2023-07-03 16:03:06 -07:00
Bryan Stitt
f0cdb3b234 flatten routes 2023-07-03 15:59:06 -07:00
Bryan Stitt
865c6d498e db helper and fix test never timing out 2023-07-03 15:54:22 -07:00
Bryan Stitt
d613969c3e Merge remote-tracking branch 'origin/devel' into david/test-admin 2023-07-03 14:07:43 -07:00
Bryan Stitt
ee6b725a1b lower log level on payload and headers 2023-07-03 13:34:16 -07:00
Bryan Stitt
d9c6da74c2 fix case on currency check 2023-07-03 13:30:37 -07:00
Bryan Stitt
8c8080e01d less async-stripe features 2023-07-03 13:12:12 -07:00
Bryan Stitt
d8512d50de just pass the whole string through 
based on this test (which i think async-stripe forked): https://docs.rs/stripe-rust/0.12.3/src/stripe/resources/event.rs.html#355

and then based on the struct that they deserialize into: https://docs.rs/async-stripe/latest/src/stripe/resources/generated/event.rs.html#16
 2023-07-03 12:57:52 -07:00
Bryan Stitt
64505953ee cargo upgrade 2023-07-03 12:28:07 -07:00
Bryan Stitt
b13e5b2440 move more logs to tracing and possible stripe payment fix 
maybe RawValue does need to_string instead of just get
 2023-07-03 12:17:30 -07:00
Bryan Stitt
832432e9d4 cargo upgrade 2023-07-02 16:21:10 -07:00
yenicelik
6cf7e4d621 problem executing the last bit 2023-07-01 15:38:56 -04:00
yenicelik
458f7d6514 could call subcommand 2023-07-01 15:26:42 -04:00
Bryan Stitt
377cd8bcf3 one more stripe option 
need to check if case matters
 2023-07-01 09:12:50 -07:00
Bryan Stitt
7044edc63c check multiple posible stripe headers 2023-07-01 09:10:53 -07:00
yenicelik
17c42243d0 wont shutdown 2023-06-30 22:24:48 -04:00
Bryan Stitt
ee626bfa61 lint 2023-06-30 13:52:09 -07:00
Bryan Stitt
501ec21442 drop workspace-hack for now 
it doesn't work well with features and so we were installing things like kafka even if the feature was off
 2023-06-30 13:18:37 -07:00
Bryan Stitt
c26d57fe5e test log in and out 2023-06-30 12:53:21 -07:00
Bryan Stitt
645fa7328b cargo upgrade tokio 2023-06-29 23:16:31 -07:00
Bryan Stitt
4a03d2e331 loop until it works 2023-06-29 23:15:57 -07:00
Bryan Stitt
5da334fcb7 start adding tests that need docker for mysql management 2023-06-29 22:38:35 -07:00
Bryan Stitt
31c611f4ff add todo about siwe 2023-06-29 14:42:01 -07:00
Bryan Stitt
581af0d596
stripe integration first mvp (replaces #140) (#154) 
* stripe integration first mvp

* addressing some points from #140

* stripe endpoint needs testing

* addressed some comments

* added Json(payload)

* update toolchain

* lint (well, most of it)

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
 2023-06-29 14:32:05 -07:00
Bryan Stitt
f6c8172a13 always do connectinfo for now 2023-06-29 13:12:35 -07:00
Bryan Stitt
ffc5a46dc4 add some stub tests 2023-06-29 12:41:21 -07:00
Bryan Stitt
294acbd712 sqlite::memory: doesn't work. investigate later 2023-06-29 12:30:03 -07:00
Bryan Stitt
c0a658c6b4 move subcommands so integration tests are easier to write 2023-06-29 12:24:36 -07:00
Bryan Stitt
b4f02b24f3 pricing for ots_ 2023-06-29 10:24:05 -07:00
Bryan Stitt
da13542789 lower sample rate 2023-06-29 10:18:32 -07:00
Bryan Stitt
335480cec9 more tracing logs 2023-06-29 10:15:12 -07:00
Bryan Stitt
357701bccc no need to check the caches, we have latest block 2023-06-29 00:06:58 -07:00
Bryan Stitt
bfc663d678 fix eth_blockNumber formatting 2023-06-28 23:00:48 -07:00
Bryan Stitt
280e2075d3
Fix Issue 65 Pt. 2 (#153) 
* change subuser balance logic and add message to access denied errors

* clearer import

* comments

* comments
 2023-06-28 23:00:34 -07:00
Bryan Stitt
ebceb0d7d6 descriptive text on NotImplemented errors 2023-06-28 21:47:06 -07:00
Bryan Stitt
d0e77237fa raise log level 2023-06-28 21:38:57 -07:00
Bryan Stitt
0c389e43b4 don't dedupe rpc 2023-06-28 21:34:51 -07:00
Bryan Stitt
fba41b9bc2 lots of tracing improvements 2023-06-28 21:30:00 -07:00
Bryan Stitt
25b04db3b5 lower log level and optional to 2023-06-28 21:08:43 -07:00
Bryan Stitt
342adb528d add sentry tag for env and chain_id 2023-06-28 20:38:33 -07:00
Bryan Stitt
2a714acc30 deref 2023-06-28 19:19:36 -07:00
Bryan Stitt
c6930dcb4c saturating sub 2023-06-28 19:19:14 -07:00
Bryan Stitt
d11d6c1f8a more logs 2023-06-28 18:44:54 -07:00
Bryan Stitt
e8d992168c more logs and retries 2023-06-28 18:36:17 -07:00
Bryan Stitt
3cfbc5baa4 improve cache keys and allow failures while parsing params 2023-06-28 17:42:49 -07:00
Bryan Stitt
df865292a7 add another layer of retries 2023-06-28 15:12:16 -07:00
Bryan Stitt
eab3aa7943 that simpler param doesn't work everywhere 2023-06-27 17:39:37 -07:00
Bryan Stitt
ec7c8f0434 todo 2023-06-27 17:08:49 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
272f7ce800 add cargo hakari and chef to build process 2023-06-27 11:59:49 -07:00
Bryan Stitt
46ee5bf277 that loop waits forever 2023-06-27 11:02:12 -07:00
Bryan Stitt
25e5caaab8 simple sleep for now. will fix with better waits later 2023-06-27 10:50:15 -07:00
Bryan Stitt
86f9560fc3 less serde features 
if we don't use them its just wasted compile time
 2023-06-27 10:18:32 -07:00
Bryan Stitt
8e98eabebe lower error log when starting 2023-06-27 10:05:07 -07:00
Bryan Stitt
1cf18b78a9 cargo upgrade 2023-06-27 10:04:48 -07:00
Bryan Stitt
b35ad330c9 save last provider error 2023-06-26 23:04:56 -07:00
Bryan Stitt
917dfc914f handle more jsonrpc errors as HTTP 200 2023-06-26 22:40:00 -07:00
Bryan Stitt
009fdbe2cf make references compile 2023-06-26 21:38:41 -07:00
Bryan Stitt
469b55fa9a sort and 0 cost test method 2023-06-26 21:38:08 -07:00
Bryan Stitt
0d1e629567 more tracing and error handling 2023-06-26 21:11:43 -07:00
Bryan Stitt
6f7783d260 add block stats to top of status data 2023-06-26 21:11:16 -07:00
Bryan Stitt
1a7d4c9cbb handle subscriptions and unimplemented 
this also means we handle new methods that haven't been added to our list as only 2 cu
 2023-06-26 21:10:32 -07:00
Bryan Stitt
8cb57e2f88 add EXTRA_RUST_LOG helper 2023-06-26 21:08:48 -07:00
Bryan Stitt
f68964f63d add 3 tries 2023-06-26 21:08:32 -07:00
Bryan Stitt
54107a390b cargo upgrade 2023-06-26 21:07:25 -07:00
Bryan Stitt
8815b5c8cd lint 2023-06-25 21:26:28 -07:00
yenicelik
627c359645 mixed up directions 2023-06-25 21:19:13 -04:00
yenicelik
bbdff1d916 some code got lost in all the commits, deploying again 2023-06-25 21:02:07 -04:00
David
0f4d12e2a5
Merge pull request #149 from yenicelik/devel 
Devel
 2023-06-25 17:55:55 -04:00
Bryan Stitt
0aa94d9c87 don't escape newlines 2023-06-25 14:34:19 -07:00
Bryan Stitt
fb76fd2d1c remove unused ewma crate now that we have our own 2023-06-25 14:32:45 -07:00
yenicelik
88617dc72b added more tracing for balance 2023-06-25 17:25:35 -04:00
yenicelik
32156e981d adding balances 2023-06-25 17:19:31 -04:00
yenicelik
992860a46d forgot to allow aggregate global stats 2023-06-25 16:18:09 -04:00
yenicelik
4c157cfcf7 made paymentrequired error code instead of introducing a new one 2023-06-25 15:05:27 -04:00
yenicelik
935681fde7 stats subuser + premium access logic 2023-06-25 14:29:54 -04:00
Bryan Stitt
fee1a423a1 simpler type when no args 2023-06-24 22:30:57 -07:00
Bryan Stitt
83123f5cdd wait up to a second for it to sync 2023-06-24 21:51:58 -07:00
Bryan Stitt
e7baeb1ba4 rearrange so maybe time works better 2023-06-24 21:41:18 -07:00
Bryan Stitt
c2917f1093 try sleeping 2023-06-24 21:35:36 -07:00
Bryan Stitt
ba702c7647 tests pass local, but jenkins isn't as fast 2023-06-24 21:27:11 -07:00
Bryan Stitt
b5ae11b7f7 lint 2023-06-24 21:21:51 -07:00
Bryan Stitt
a440d51377 test fixture 2023-06-24 21:17:08 -07:00
Bryan Stitt
8cfbbfe1b3 use default derive helpers 2023-06-24 20:39:46 -07:00
Bryan Stitt
efe1e30791 better handling of 0 port and fix test port 2023-06-24 20:38:06 -07:00
Bryan Stitt
ab3d4a91a8 json body instead of params 2023-06-24 18:17:51 -07:00
Bryan Stitt
0c10994835 why wont that import? 2023-06-24 16:45:07 -07:00
Bryan Stitt
94d0a25aaf test logs 2023-06-24 16:43:11 -07:00
Bryan Stitt
38b266156a consistent route name 2023-06-24 15:49:26 -07:00
Bryan Stitt
1d6e1ef9af polish error logs 2023-06-24 15:24:46 -07:00
Bryan Stitt
fef03f089f less verbose db_conn and db_replica helpers 2023-06-24 11:11:07 -07:00
Bryan Stitt
9115419ec0 handle addresses consistently 2023-06-24 10:20:24 -07:00
Bryan Stitt
39a505c0ff POST instead of GET and use more txs 2023-06-24 09:59:08 -07:00
Bryan Stitt
8f76d9320d support EIP1271 login 2023-06-24 09:48:31 -07:00
Bryan Stitt
91cbce6ce8 start adding stat for CU 2023-06-24 00:48:56 -07:00
Bryan Stitt
4e6a80caa2 cargo upgrade 2023-06-24 00:41:48 -07:00
Bryan Stitt
30828ccae2 prioritize block. queries to syncing nodes ends up being slower 2023-06-24 00:41:30 -07:00
Bryan Stitt
ac4b7e46b4 actually import info level 2023-06-23 22:54:02 -07:00
Bryan Stitt
6edf1a5c1b info on new block 2023-06-23 22:52:42 -07:00
Bryan Stitt
89330cfba0 upstream ethers again 2023-06-23 22:23:51 -07:00
Bryan Stitt
fd661689a3
Tracing (#144) 
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
 2023-06-23 16:28:45 -07:00
Bryan Stitt
3fac4248d6 Revert "just use Ulid instead of RpcSecretKey type" 
It did work

This reverts commit 377e8356a57d80dd17b746dd05a62b475d9e7d07.
 2023-06-23 01:29:11 -07:00
Bryan Stitt
377e8356a5 just use Ulid instead of RpcSecretKey type 2023-06-23 01:28:47 -07:00
Bryan Stitt
68ab9b6d1d always serialize RpcSecretKey as a ULID 2023-06-23 01:07:54 -07:00
Bryan Stitt
6b6fac396f lower default web3_proxy::rpcs::consensus level 2023-06-22 22:45:43 -07:00
Bryan Stitt
b1b8346db9
#123 role to rpc keys (#142) 
* added roles to rpc keys

* dont unwrap

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
 2023-06-22 12:56:19 -07:00
Bryan Stitt
c79b787e13
allow setting referral code if not already set (#141) 2023-06-22 12:41:05 -07:00
Bryan Stitt
376b9f474d use short syntax for json data in logs 2023-06-22 12:10:23 -07:00
Bryan Stitt
f593667169 fix lifetimes 2023-06-22 10:16:58 -07:00
Bryan Stitt
66212b5cf5 cargo upgrade --incompatible -p itertools 2023-06-22 09:26:57 -07:00
Bryan Stitt
dffff65950 allow no referrer_record 2023-06-22 09:25:12 -07:00
Bryan Stitt
c560a59ef0 add more headers for debug endpoints and use more refs 2023-06-21 22:11:26 -07:00
Bryan Stitt
095a505017 cargo upgrade 2023-06-21 17:57:53 -07:00
Bryan Stitt
662bbb296f add header debug endpoint 2023-06-21 11:28:33 -07:00
Bryan Stitt
63c4f9e591 allow no ip 
this shouldn't be needed, but is a workaround while we debug staging
 2023-06-21 11:28:22 -07:00
Bryan Stitt
7f4da8d43c lint 2023-06-21 10:40:55 -07:00
Bryan Stitt
d962b749a2 use find_with_related, not find_also_related 
find_also_related is 1:1, but we want 1:Many
 2023-06-21 10:37:01 -07:00
Bryan Stitt
1cbb58e8e0 comment 2023-06-21 10:25:10 -07:00
Bryan Stitt
06ce95c386 let serde_json do more of the work 2023-06-21 10:18:36 -07:00
Bryan Stitt
5504062f47 dont error if no referrals 2023-06-21 10:18:06 -07:00
Bryan Stitt
8f1b6fe6c0 lint 2023-06-21 10:02:02 -07:00
Bryan Stitt
4dad743fc3 tuple, not list 2023-06-21 10:01:51 -07:00
Bryan Stitt
f75f3f8d2f serialize caches in a list 2023-06-21 10:00:05 -07:00
Bryan Stitt
3c2f95de10
Improve max wait and timeouts (#137) 
* pass max wait with more functions

* move rpc_head_str higher up to use it in all the logs

* refresh consensus at double the block time

* new pricing that also includes archive check

* lint

* don't clone method

* put caches into one list

they serialize with their name, so it will still be easy to tell them apart

* more consistent waiting. still waits more than I want right now

* clean up default max waits

* more timeouts
 2023-06-20 14:22:14 -07:00
Bryan Stitt
78efad5453 cargo upgrade strum 2023-06-19 17:48:46 -07:00
Bryan Stitt
6d25c41faf
Compute units (#136) 
* wip

* add basic compute unit calculator

* calculate costs with compute units
 2023-06-19 17:47:38 -07:00
Bryan Stitt
b7935f77fc spawn recording latency 2023-06-19 17:46:58 -07:00
Bryan Stitt
6038351cb8 change default tracking in prep for premium-only keys 2023-06-19 13:00:57 -07:00
Bryan Stitt
b35cd58a76 rate limit user_balance_post by bearer OR ip 2023-06-19 10:21:48 -07:00
Bryan Stitt
c3ae4ded2b move data to the right places 2023-06-18 21:42:16 -07:00
Bryan Stitt
58cc129837 forgot to convert tier_sec_size from ms. 1 was a very large default 2023-06-18 17:57:21 -07:00
Bryan Stitt
ba410cf126 better log message 2023-06-18 16:41:01 -07:00
Bryan Stitt
89009dcd42 use web3_context for more errors 2023-06-18 16:40:17 -07:00
Bryan Stitt
af8543c3bf some defaults are needed maybe? 2023-06-18 16:36:23 -07:00
Bryan Stitt
2f78e64c5d create empty balance entry if none exists 2023-06-18 16:25:25 -07:00
Bryan Stitt
6deeabe1e1 pretty print errors 2023-06-18 16:14:06 -07:00
Bryan Stitt
9dd0bae40a text instead of string payload 2023-06-18 16:13:55 -07:00
Bryan Stitt
825ba006f1 move floats and durations around for latency tracking 2023-06-18 10:46:22 -07:00
Bryan Stitt
2f0e6103ac Revert "tier more important than backup" 
This reverts commit bca22537c633822e5e6759fa2840712a837211fd.
 2023-06-18 10:09:21 -07:00
Bryan Stitt
bca22537c6 tier more important than backup 
we filter backups when they aren't needed, so this should be okay
 2023-06-18 10:06:26 -07:00
Bryan Stitt
e54299beca rolling median for choosing tier 2023-06-18 09:53:36 -07:00
Bryan Stitt
8b22c9861b Revert "temp fix while filtering is improved" 
This reverts commit 5367c511953fcea1af3da1d905de06d9e76711ba.
 2023-06-17 12:15:40 -07:00
Bryan Stitt
5367c51195 temp fix while filtering is improved 2023-06-17 12:09:04 -07:00
Bryan Stitt
3b65074f38 cleaner duration multiplication 2023-06-17 11:33:33 -07:00
Bryan Stitt
eab86c7ee8 automatic max_head_block_age 2023-06-17 10:13:40 -07:00
Bryan Stitt
bf13e86a04 warn if no max_head_block_age 2023-06-17 09:19:05 -07:00
Bryan Stitt
3b9314a76f comments 2023-06-17 08:44:18 -07:00
Bryan Stitt
9b9e5341cb lint 2023-06-17 08:44:07 -07:00
Bryan Stitt
6bfa5a69ca latest_balance is part of AuthorizationChecks now 2023-06-17 08:43:51 -07:00
Bryan Stitt
8ffd7d2e1b enable more advanced cost calculation 2023-06-16 23:55:41 -07:00
Bryan Stitt
ba67151706 lint 2023-06-16 23:48:16 -07:00
Bryan Stitt
745870d119 drop a bunch of log levels 2023-06-16 23:46:20 -07:00
Bryan Stitt
74d2333039 try this version of influxdb2 2023-06-16 23:41:12 -07:00
David
d6513cde74
Forgot to handle Option unwrap (#132) 
* made the query much faster

* aggregate queries should be fast now too

* forgot to handle an option unwrap
 2023-06-16 23:18:51 -07:00
Bryan Stitt
86f9e7358a
move balance onto AuthorizationChecks (#131) 
* move balance onto AuthorizationChecks

* todone

* bigger default

* flat cost for testing
 2023-06-16 23:14:43 -07:00
Bryan Stitt
5d9365449f simplify max item weight by having the weigher send MAX for big things 2023-06-16 16:14:11 -07:00
Bryan Stitt
1ec0f14144 upgrade to influxdb 1e5577e14150797584f5ed0ea7aba0bd68f0a678 2023-06-16 14:48:03 -07:00
Bryan Stitt
cb71de7d14 pin influxdb2 to 6d3cf64d61967f41fafb57b3f4553206a8f3f2a3 2023-06-16 13:30:35 -07:00
Bryan Stitt
d9a865fd1b cargo upgrade 2023-06-16 10:58:09 -07:00
Bryan Stitt
3928496be5 start tiers at 1 2023-06-16 10:52:13 -07:00
Bryan Stitt
af7759e491 by_name needs a lock to prevent races during disconnects 2023-06-16 10:40:02 -07:00
Bryan Stitt
7c876ac561 delete stale code 
this was rewritten and put into its own crate
 2023-06-16 10:02:14 -07:00
Bryan Stitt
7707729f77 Merge remote-tracking branch 'origin/main' into devel 2023-06-16 10:01:17 -07:00
David
4772166e97
Make aggregate queries faster as well (#130) 
* made the query much faster

* aggregate queries should be fast now too
 2023-06-16 08:27:46 -07:00
Bryan Stitt
fe8675ddf2 cargo upgrade 2023-06-16 00:48:22 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129) 
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
 2023-06-16 00:46:27 -07:00
Bryan Stitt
6040ca297f fix eth_unsubscribe 2023-06-15 11:43:24 -07:00
Bryan Stitt
49c60ac1b5 improve websocket error handling 2023-06-15 09:50:21 -07:00
David
5859cd8a8d
made the query much faster (#127) 2023-06-15 09:38:51 -07:00
Bryan Stitt
957ba161b5 move table data around 2023-06-13 12:08:47 -07:00
Bryan Stitt
b8f429d70a improvements to popularity_contesst 2023-06-13 11:51:19 -07:00
Bryan Stitt
f5a1ac274a refactor send_head_block_result 2023-06-13 10:00:08 -07:00
Bryan Stitt
1473ccb085 gentler parsing of new head blocks 2023-06-13 09:31:22 -07:00
Bryan Stitt
602b0117c4 panic if influxdb_bucket is not set when influxdb_host is set 2023-06-13 09:30:41 -07:00
Bryan Stitt
45543be256 add more caches to /status 2023-06-12 23:03:38 -07:00
Bryan Stitt
a7fb12bd72 less random is fine for shuffling 2023-06-12 22:42:22 -07:00
Bryan Stitt
55e8471b19 interval instead of watch_blocks 2023-06-12 22:26:10 -07:00
Bryan Stitt
6324d9c7d9 short circuit if no requests 2023-06-12 21:54:33 -07:00
Bryan Stitt
18835e09da allow empty batches 2023-06-12 21:51:08 -07:00
Bryan Stitt
c192e03fa2 count all the tiers at once 2023-06-12 21:03:38 -07:00
Bryan Stitt
bba7ccf7eb change divisor for tier calculation 2023-06-12 20:44:52 -07:00
Bryan Stitt
c771479d94 use WyRand instead of Xoshiro 2023-06-12 19:46:36 -07:00
Bryan Stitt
1a8f799969
we need our own Reconnect (#124) 
* first pass at handling different return data limits

* put ws_provider in an arcswap

* add min max_latency

* add min max_latency

* subscribe with reconnect

* better logging around reconnect

* select on both watches

* subscribe to the correct watch
 2023-06-12 18:58:51 -07:00
Bryan Stitt
62bda636b8 return the response instead of panicing 2023-06-12 16:38:57 -07:00
Bryan Stitt
c456432cc3 cargo upgrade 2023-06-12 11:44:51 -07:00
Bryan Stitt
c3a7d1646c lint 2023-06-12 11:41:10 -07:00
Bryan Stitt
70d5c09d3e clean up redirect url 2023-06-12 11:38:34 -07:00
Bryan Stitt
f1176685dd pause time on more tests 2023-06-12 11:08:33 -07:00
David
093ca19454
added subuser RPC keys into key access control, as well as fetching RPC keys (#115) 2023-06-12 08:59:23 -07:00
David
0cc4557e8d
Use rpc id instead of key (#118) 
* added logic to check e-mails thru SMPT server, very basic, no validation e-mails etc.

* Revert "added logic to check e-mails thru SMPT server, very basic, no validation e-mails etc."

This reverts commit 022a13cfd9ca24aad9e05f75489396936f14784a.

* mixed up branches
 2023-06-11 19:14:02 -07:00
David
81aa4788b4
David/60 add basic email verification (#119) 
* added logic to check e-mails thru SMPT server, very basic, no validation e-mails etc.

* added rpc key logic to use db numbers instead of UUIDs

* Revert "added rpc key logic to use db numbers instead of UUIDs"

This reverts commit 056771d02725f90ed5f1745b0372e13e630a19ac.
 2023-06-11 19:10:10 -07:00
yenicelik
0b26ae3bad made modify-subuser post 2023-06-11 18:59:27 +02:00
yenicelik
2333b8bbb6 influx query is not 20-40% faster 2023-06-11 17:57:02 +02:00
Bryan Stitt
4a9c1a0ce6 move user registration into a dedicated function 2023-06-09 18:31:47 -07:00
Bryan Stitt
71d3d63524 put ip rate limiting back 2023-06-09 16:35:44 -07:00
Bryan Stitt
41950c886c improvements for web3-this-then-that 2023-06-09 16:35:14 -07:00
Bryan Stitt
bcf4f33543 add payment_factory_address to /status 2023-06-09 14:02:29 -07:00
Bryan Stitt
bb900c61a5 todo 2023-06-09 13:31:46 -07:00
Bryan Stitt
8bf0ee473d loud log on rollback 2023-06-09 13:30:00 -07:00
Bryan Stitt
bd87fcb13c move sort and shuffle for loadbalancing into proper functions 2023-06-09 13:09:58 -07:00
Bryan Stitt
1f3040f6c7 automated tiers 2023-06-09 12:21:55 -07:00
Bryan Stitt
b6ff14210d cargo upgrade 2023-06-08 22:35:14 -07:00
David
d91713e09e
this brings back balance back to the stats request (#112) 
* removed bloom filter temporarily, added some fixes with decimals in payment.rs

* balance is inside the influx query again

* fixed aggregate
 2023-06-08 15:57:53 -07:00
Bryan Stitt
68f73ec0b1 more things should be BadRequest instead of 500 2023-06-08 13:42:45 -07:00
Bryan Stitt
50ec653557 lower log level 2023-06-08 11:46:38 -07:00
Bryan Stitt
74280bc329 lint 2023-06-08 11:33:16 -07:00
Bryan Stitt
334b1c21b3 improve logs 2023-06-08 11:30:48 -07:00
Bryan Stitt
1d05bc5e89 lower log levels 2023-06-08 11:26:09 -07:00
David
3f76b08364
Test balance and referral accounting (#111) 
* removed bloom filter temporarily, added some fixes with decimals in payment.rs

* balance accounting for a single user (paid and free tier) seems to be ok

* fixed some balance accounting

* compute_cost reduced back to 0
 2023-06-08 10:08:29 -07:00
Bryan Stitt
c37b320286 add cache names and actually serialize 2023-06-07 17:55:34 -07:00
Bryan Stitt
5744b459ee lint 2023-06-07 17:43:10 -07:00
Bryan Stitt
6cf738261b fix test 2023-06-07 17:42:34 -07:00
Bryan Stitt
bfaff81626 cargo upgrade 2023-06-07 17:31:29 -07:00
Bryan Stitt
5c1569f9c2 clippy lint 2023-06-07 17:31:15 -07:00
Bryan Stitt
40bee30730 back to moka 2023-06-07 17:30:05 -07:00
David
de7d8919d3
Change balance to record total spend and total deposits (#109) 
* lets test total deposit and total spent

* removed referrer from cache for performance reasons
 2023-06-07 14:45:57 -07:00
Bryan Stitt
4f7144abc6 Arc instead of Box for better caching 2023-06-07 13:57:38 -07:00
Bryan Stitt
c2a3a4a0a9 Merge remote-tracking branch 'origin/main' into devel 2023-06-07 13:29:12 -07:00
Bryan Stitt
91d288315f add queue lengths to status page 2023-06-07 13:05:15 -07:00
Bryan Stitt
b50579ea3a add block caches to web3rpcs serializer 2023-06-07 12:04:39 -07:00
Bryan Stitt
8249b0aefe cache isnt on main yet 2023-06-07 11:18:36 -07:00
Bryan Stitt
aa57dd2fcf serialize the caches on /status 2023-06-07 10:58:30 -07:00
Bryan Stitt
f0a9f03076 smaller caches 2023-06-07 10:57:43 -07:00
Bryan Stitt
c4b2ce5266 smaller caches 2023-06-07 10:57:01 -07:00
Bryan Stitt
fb97ef5180 cargo clippy lints 2023-06-07 10:55:07 -07:00
Bryan Stitt
f6b2b0deab cargo upgrade 2023-06-07 10:50:34 -07:00
Bryan Stitt
2b814e7a4d serialize the caches on /status 2023-06-07 10:48:55 -07:00
David
63499c1564
David/66 downgrade balance logic (#89) 
* tried to make the balance_entry atomic, asking for feedback

* added locked select everywhere and inside the txn, it should be atomic now

* forgot about overwrite (referee vs sender entities were the same, overwriting each other), fixed it

* will now trigger refreshing cache when balance thresholds are met

* will also invalidate cache if balance gets too low (to out of funds)

* added separate balance cache with shorter TTL. need to know if i should modify it (rn its not atomic), or if low TTL is good enough, and we just fetch the value every now and then from the Database

* removed UserTier struct bcs duplicate with database

* removed subuser premium requirement (which would have lead to a merge conflict later on)

* removed user_tier artefact

* replaced cache to use AtomicF64

* a bunch of small changes

* some changes

* will merge with devel

* changed AtomicF64 to RwLock<Decimal>

* changed AtomicF64 to RwLock<Decimal>

* downgrading user when at 0.1$ or if sum_credits_used is very low

* changed caches to be more aggressive in being non-empty

* replaced Arc::clone() by reference
 2023-06-07 09:39:30 -07:00
David
fd69e6acdd
added stats page for referrals (shared, and received) (#95) 
* added stats page for referrals (shared, and received)

* removed referrals address, and return ok response if no referrals used

* merged from devel

* changed unwrap for context

* changes from PR
 2023-06-07 09:38:19 -07:00
Bryan Stitt
172baabb4e even more fields onto popularity contest 2023-06-05 16:40:49 -07:00
Bryan Stitt
640bcd1378 update popularity_contest script to match new status page 2023-06-05 16:40:49 -07:00
David
f66edd961b
Making Balance Changes Atomic (Read and Write) (#84) 
* tried to make the balance_entry atomic, asking for feedback

* added locked select everywhere and inside the txn, it should be atomic now

* forgot about overwrite (referee vs sender entities were the same, overwriting each other), fixed it

* will now trigger refreshing cache when balance thresholds are met

* Revert "will now trigger refreshing cache when balance thresholds are met"

This reverts commit b1215ddc43a1358f6086bddbd106996165431165.

* will not modify userTier in database anymore

* made payment atomic

* made updates mostly atomic, locking only the referral table (bcs more complex logic)

* latest changes
 2023-06-04 09:32:53 -07:00
Bryan Stitt
79e52f4cdb Merge remote-tracking branch 'origin/main' into devel 2023-05-31 17:12:59 -07:00
Bryan Stitt
633bf1f53b need a decimal 0 2023-05-31 17:12:32 -07:00
Bryan Stitt
cb580d4ea8 Merge remote-tracking branch 'origin/main' into devel 2023-05-31 17:02:34 -07:00
Bryan Stitt
1a26260351 if no credits used, return early 2023-05-31 17:02:24 -07:00
Bryan Stitt
8709737d74 lower log level 2023-05-31 16:07:13 -07:00
Bryan Stitt
fbe0ecfbff rate limit on websockets 2023-05-31 16:05:44 -07:00
Bryan Stitt
cadab50692 lint 2023-05-31 15:54:32 -07:00
Bryan Stitt
56ec9439c3 insert on conflict update for balance table 2023-05-31 14:07:25 -07:00
David
88526c3702
Missing subuser logic (#96) 
* will push it for feedback loop

* filtering also to only show rpc keys that the user is an owner of

* revert logs also returning rpc keys

* merged with devel
 2023-05-31 13:46:59 -07:00
Bryan Stitt
62997774b1 use abigen types for decoding the logs 2023-05-31 12:44:02 -07:00
Bryan Stitt
7dcc2b4d93 use MultiAbigen 2023-05-31 11:52:57 -07:00
Bryan Stitt
7947cb95ff use bloom filters and support transactions with multiple deposit events 2023-05-31 11:20:17 -07:00
Bryan Stitt
b9f0824dfe start using the internal provider 2023-05-30 23:52:12 -07:00
Bryan Stitt
d64761403b Cow instead of String 2023-05-30 23:17:05 -07:00
Bryan Stitt
f1636d3b85 unlimited localhost 2023-05-30 22:31:35 -07:00
Bryan Stitt
fcc1843af0 put it back. duplication between cargo.toml isnt worth it 2023-05-30 21:37:58 -07:00
Bryan Stitt
fd8bba4bdd most of these are for the whole app, not just the frontend 2023-05-30 21:26:11 -07:00
Bryan Stitt
0b62ffcdfc no need to move the semaphore around like this 2023-05-30 21:08:21 -07:00
Bryan Stitt
2273637a07 move cli into a separate packge and clean up dependencies 2023-05-30 20:58:24 -07:00
Bryan Stitt
cebe8ed1ea
Request refactor (#99) 
* wip

* AsRef finally works like i wanted

* actually return the block

* start adding async trait

* remove stale import

* include id in the error response when possible

* remove stale comments
 2023-05-30 16:32:34 -07:00
Bryan Stitt
4246342806 lower log levels 2023-05-29 17:26:32 -07:00
Bryan Stitt
c1eba556a5 fix missing ttl send and add name to cache 2023-05-29 17:19:05 -07:00
Bryan Stitt
bb50efb7f9
quick cache ttl max item size, better function names, and tests (#97) 
* Revert "remove cache on /status and /health for now"

This reverts commit 166b0d810cd25b895560d50d00383a57e1269ad4.

* Revert "remove cache on backups needed, too"

This reverts commit 4597967def2f1daa71a277ec6731db4739ead640.

* add tests

* add max_item_weight

* comment

* add some helper functions, max weights, and tests
 2023-05-29 15:48:22 -07:00
David
5f0c99530c
referral code does not have llamanodes- prefix any longer (#90) 2023-05-29 14:41:15 -07:00
Bryan Stitt
9e935e2c44 cargo upgrade 2023-05-29 09:43:51 -07:00
Bryan Stitt
b05d866f8a Merge remote-tracking branch 'origin/main' into devel 2023-05-29 09:42:21 -07:00
Bryan Stitt
a275e2dee4 add blocktime for binance 2023-05-29 09:41:19 -07:00
Bryan Stitt
dc6e0559f7 improve will_work_eventually 2023-05-29 09:41:19 -07:00
Bryan Stitt
84990bb7b6 check hard_limit_until when finding if an rpc will work 2023-05-29 09:41:19 -07:00
Bryan Stitt
08859ba04d wait for consensus rpcs to change before trying again 2023-05-29 09:41:19 -07:00
David
48905f0235
David/76 bring back docs (#91) 
* bring back docs

* added some more rudimentary documentation
 2023-05-29 09:24:41 -07:00
David
ccb3a559ad
removed requirement for subuser logic to be admin (#94) 2023-05-29 08:35:24 -07:00
Bryan Stitt
4597967def remove cache on backups needed, too 2023-05-28 09:39:24 -07:00
Bryan Stitt
166b0d810c remove cache on /status and /health for now 2023-05-28 09:38:51 -07:00
Bryan Stitt
7cb07cc49f debug -> trace for some verbose logs 2023-05-27 01:53:49 -07:00
Bryan Stitt
0e7bd1fbf3 add missing anyhow::Context 2023-05-26 17:00:39 -07:00
Bryan Stitt
4aa34740a8 default balance to 0 2023-05-26 16:54:52 -07:00
David
9a5e9d46a9
Merge branch 'devel' into david/devel/debug-graph 2023-05-24 13:36:45 +02:00
yenicelik
b6cab88c4e removed joined_on variable that was not used 2023-05-24 13:35:10 +02:00
yenicelik
47f07261ed detailed request must come with a bearer token 2023-05-24 13:34:50 +02:00
Bryan Stitt
0d15d5baf7 bump versions to match number of migrations 2023-05-23 20:58:29 -07:00
Bryan Stitt
b1a0bcac57 add max wait to fix tests 2023-05-23 20:46:27 -07:00
Bryan Stitt
dafb69fae1 move timeout deeper 2023-05-23 15:50:29 -07:00
Bryan Stitt
55f9c57827 include error in warning 2023-05-23 15:03:16 -07:00
Bryan Stitt
4c7a7b87cb cargo upgrade 2023-05-23 14:58:13 -07:00
Bryan Stitt
dd347fd916 add units to serialized variables 2023-05-23 14:57:24 -07:00
Bryan Stitt
77c9b0ab12 peak latency instead of head latency 2023-05-23 14:52:45 -07:00
Bryan Stitt
651f0fcad3 lint 2023-05-23 14:51:34 -07:00
Bryan Stitt
d83e2f7015 Merge remote-tracking branch 'origin/devel' into devel 2023-05-23 14:44:40 -07:00
Bryan Stitt
78f247fc6c split http and ws provider 2023-05-23 14:40:34 -07:00
David
27333a0028
David/devel/debug graph (#82) 
* now should also group by archive_needed (turns out this was wanted)

* influxdb query also groups by archive_needed now

* updated error_response

* updated error_response
 2023-05-22 17:08:35 -07:00
David
ddfd1dbff1
Added admin balance increase, and also test (#72) 
* added admin balance increase, and also test

* slightly hacky, but does do an update expression

* added note column to entities

* removed redundant printlns

* flattened items

* removed user tier logic from admin increase balance completely. admin will have to manually increase the user tier
 2023-05-22 17:07:17 -07:00
Bryan Stitt
05f862f3e4 cargo upgrade 2023-05-22 15:50:00 -07:00
Bryan Stitt
3ac15558e3 fix decay time 2023-05-22 15:43:39 -07:00
Bryan Stitt
91eeee23e2 use ether's reconnects instead of our own which need a lock (wip) 2023-05-22 15:32:27 -07:00
yenicelik
59634139c9 updated error_response 2023-05-21 20:27:52 -04:00
yenicelik
29678b025e updated error_response 2023-05-21 20:27:46 -04:00
yenicelik
46afe8b1e4 influxdb query also groups by archive_needed now 2023-05-21 17:13:15 -04:00
yenicelik
795548580a now should also group by archive_needed (turns out this was wanted) 2023-05-21 16:40:42 -04:00
Bryan Stitt
c443f76623 refactor tiers again 2023-05-19 14:09:03 -07:00
Bryan Stitt
5aae318620 logs and comments 2023-05-18 15:11:29 -07:00
Bryan Stitt
9c584354d9 add to skip list earlier 2023-05-18 13:51:28 -07:00
Bryan Stitt
a92c93706b dont check heads while waiting 
this isn't great. but should work better than what we have
 2023-05-18 13:47:00 -07:00
Bryan Stitt
8898ffe578 cargo upgrade 2023-05-18 13:36:35 -07:00
Bryan Stitt
b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt
24439c5143 disable balance join for now 2023-05-18 00:43:33 -07:00
Bryan Stitt
0f367d9035 more quick cache ttl 2023-05-18 00:04:30 -07:00
Bryan Stitt
1c61390ad1 check if watch_consensus_rpcs.changed has failed 
i dont think it ever will, but better safe than sorry
 2023-05-16 23:42:42 -07:00
Bryan Stitt
c818a8fcd7 default balance to zero instead of panicing 2023-05-16 23:40:32 -07:00
Bryan Stitt
89961331af refactor wait_for_block into should_wait_for_block 2023-05-16 22:29:36 -07:00
Bryan Stitt
a07da30042 keep usable_rpcs_by_tier_and_head_number around longer 2023-05-16 18:32:33 -07:00
Bryan Stitt
0f79d2e60d more loud logs 2023-05-16 18:29:56 -07:00
Bryan Stitt
58b7799cc5 for now, always return 0 for cost 2023-05-16 18:25:56 -07:00
Bryan Stitt
a0d8218eb1 best block num has to check has_block_data 2023-05-16 18:18:56 -07:00
Bryan Stitt
30ea532c70 display, not debug 2023-05-16 16:13:56 -07:00
Bryan Stitt
5d64524aa6 add name to web3rpcs 2023-05-16 16:04:17 -07:00
Bryan Stitt
efaaf1bb5f send the correct content type 2023-05-16 15:27:18 -07:00
Bryan Stitt
525f6d4a34 better variable names 2023-05-16 14:58:00 -07:00
Bryan Stitt
40b15579ab Merge remote-tracking branch 'origin/devel' into quick_cache_ttl 2023-05-16 14:47:43 -07:00
Bryan Stitt
f4cebde53f use ttl quick cache 2023-05-16 14:46:05 -07:00
Bryan Stitt
978c385b3c improve wait for block 2023-05-16 14:02:37 -07:00
Bryan Stitt
c66eb6d864 head latency instead of peak latency for now 2023-05-16 12:18:59 -07:00
Bryan Stitt
685c1d039a cargo upgrade 2023-05-13 15:00:24 -07:00
Bryan Stitt
2080739865
Quick cache (#70) 
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343fc51338a8a4282dcc229b50097835b.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
 2023-05-13 11:13:02 -07:00
Bryan Stitt
c3cdf7ef43 fix consensus finding bug 2023-05-12 23:40:15 -07:00
Bryan Stitt
4b7f88367d fix serializing /status 2023-05-12 23:22:20 -07:00
Bryan Stitt
74a602bc9c wip 2023-05-12 23:20:59 -07:00
Bryan Stitt
a3b97fda9b lint 2023-05-12 23:04:56 -07:00
Bryan Stitt
c54970da0a change ordering and move fetch_add and fetch_sub 2023-05-12 23:00:03 -07:00
Bryan Stitt
914c3e03a8 remove stale comments 2023-05-12 22:22:21 -07:00
Bryan Stitt
3cea4c3123 update peak_ewma to use request latency and fix off by one 2023-05-12 22:21:11 -07:00
Bryan Stitt
0c990b0755 use a cancel-safe channel for stats 2023-05-12 16:02:43 -07:00
Bryan Stitt
510612d343
use kanal instead of flume or tokio channels (#68) 2023-05-12 15:47:01 -07:00
Bryan Stitt
8a097dabbe
Bryan devel 2023-05-12 (#67) 
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
 2023-05-12 15:15:32 -07:00
David
34ed450fab
User Balance + Referral Logic (#44) 
* will implement balance topup endpoint

* will quickly fix other PR reviews

* merging from master

* will finish up godmoe

* will finish up login

* added logic to top up balance (first iteration)

* should implement additional columns soon (currency, amount, tx-hash), as well as a new table for spend

* updated migrations, will account for spend next

* get back to this later

* will merge PR from stats-v2

* stats v2

rebased all my commits and squashed them down to one

* cargo upgrade

* added migrtation for spend in accounting table. will run test-deposit next

* trying to get request from polygon

* first iteration /user/balance/:tx_hash works, needs to add accepted tokens next

* creating the referral code seems to work

* will now check if spending enough credits will lead to both parties receiving credits

* rpcstats takes care of accounting for spend data

* removed track spend from table

* Revert "removed track spend from table"

This reverts commit a50802d6ae75f786864c5ec42d0ceb2cb27124ed.

* Revert "rpcstats takes care of accounting for spend data"

This reverts commit 1cec728bf241e4cfd24351134637ed81c1a5a10b.

* removed rpc request table entity

* updated referral code to use ulid s

* credits used are aggregated

* added a bunch of fields to referrer

* added database logic whenever an aggregate stats is added. will have to iterate over this a couple times i think. go to (1) detecting accepted stables next, (2) fix influxdb bug and (3) start to write test

* removed track spend as this will occur in the database

* will first work on "balance", then referral. these should really be treated as two separate PRs (although already convoluted)

* balance logic initial commit

* breaking WIP, changing the RPC call logic functions

* will start testing next

* got rid of warnings & lint

* will proceed with subtracting / adding to balance

* added decimal points, balance tracking seems to work

* will beautify code a bit

* removed deprecated dependency, and added topic + deposit contract to app.yaml

* brownie test suite does not rely on local contract files it pulls all from polygonscan

* will continue with referral

* should perhaps (in a future revision) recordhow much the referees got for free. marking referrals seems to work rn

* user is upgraded to premium if they deposit more than 10$. we dont accept more than $10M in a single tx

* will start PR, referral seems to be fine so far, perhaps up to some numbers that still may need tweaking

* will start PR

* removed rogue comments, cleaned up payments a bit

* changes before PR

* apply stats

* added unique constraint

* some refactoring such that the user file is not too bloated

* compiling

* progress with subusers, creating a table entry seems to work

* good response type is there as well now, will work on getters from primary user and secondary user next

* subuser logic also seems fine now

* downgrade logic

* fixed bug influxdb does not support different types in same query (which makes sense)

* WIP temporary commit

* merging with PR

* Delete daemon.rs

there are multiple daemons now, so this was moved to `proxyd`

* will remove request clone to &mut

* multiple request handles for payment

* making requests still seem fine

* removed redundant commented out bits

* added deposit endpoint, added deposit amount and deposit user, untested yet

* small bug with downgrade tier id

* will add authorization so balance can be received for users

* balance history should be set now too

* will check balance over time again

* subususer can see rpc key balance if admin or owner

* stats also seems to work fine now with historical balance

* things seem to be building and working

* removed clone from  OpenRequestHandle

* removed influxdb from workspace members

* changed config files

* reran sea-orm generate entities, added a foreign key, should be proper now

* removed contract from commit

* made deposit contract optional

* added topic in polygon dev

* changed deposit contract to deposit factory contract

* added selfrelation on user_tier

* added payment required

* changed chain id to u64

* add wss in polygon llamarpc

* removed origin and method from the table

* added onchain transactions naming (and forgot to add a migration before)

* changed foreign key to be the referrer (id), not the code itself

* forgot to add id as the target foreign key

* WIP adding cache to update role

* fixed merge conflicts

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
Co-authored-by: Bryan Stitt <bryan@stitthappens.com>
 2023-05-12 09:45:15 -07:00
Bryan Stitt
36cc884112 Merge remote-tracking branch 'origin/main' into devel 2023-05-11 20:05:47 -07:00
Bryan Stitt
fc8ca4ba4f less max lag 2023-05-11 17:04:33 -07:00
Bryan Stitt
2a242fe37f cargo upgrade --incompatible 2023-05-11 13:25:55 -07:00
Bryan Stitt
6e8536d477 add support for http basic auth 2023-05-11 13:21:12 -07:00
Rory Trent
ec11e210ee
Peak ewma (#63) 
* use peak-ewma instead of head for latency calculation

* Implement some suggested changes from PR

* move latency to new package in workspace root

* fix unit tests which now require peak_latency on Web3Rpc

* Switch to atomics for peak-ewma

This change is to avoid locking from tokio::sync::watch.

* add decay calculation to latency reads in peak-ewma

* Add some tests for peak-ewma

* Sensible latency defaults and not blocking on full

* Cleanup and a couple additional comments
 2023-05-11 13:09:15 -07:00
Bryan Stitt
96a9a39789 add support for http basic auth 2023-05-11 13:07:38 -07:00
Bryan Stitt
0531e2f8dd add 'quota usage' rate limit substring 2023-05-08 11:25:27 -06:00
Bryan Stitt
4d64bf9580 cargo upgrade 2023-05-08 10:44:01 -06:00
Bryan Stitt
e7da6cd3d6 clone for now. this will change to use refs soon 2023-05-08 10:42:03 -06:00
Bryan Stitt
c2e3d71080 if protected rpcs, only use protected rpcs 2023-05-08 10:39:06 -06:00
Bryan Stitt
8916533a2a set archive_request to true on transaction retrying 2023-05-08 10:38:38 -06:00
Bryan Stitt
aff53ce36a automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash 
thanks for the report Lefteris @ Rotki
 2023-05-08 10:38:23 -06:00
Bryan Stitt
ffb554d0b3 online editor isn't actually nice when it misses syntax errors :( 2023-04-29 18:50:26 -07:00
Bryan Stitt
91acbcaf75
Fix sort order 2023-04-29 18:38:31 -07:00
Bryan Stitt
93f243696a
simple match statement 2023-04-27 11:15:45 -07:00
Bryan Stitt
a236f7364d
DRY stats (#58) 
* DRY functions for saving stats and batch tsdb stats

* move comments
 2023-04-25 07:42:00 -07:00
Bryan Stitt
cc3c9462a3 cargo upgrade 2023-04-25 05:39:09 -07:00
Bryan Stitt
1fab1b1d46 oops. lower this log level 2023-04-24 11:21:53 -07:00
Bryan Stitt
56fdf48129
Send transactions to protected and public rpcs (#57) 
* move protected transactions into their own function and dry stats sending

* cargo upgrade

* comments

* time to live instead of time to idle

* minor workaround for eth_chainId

* cargo upgrade
 2023-04-24 11:00:12 -07:00
Bryan Stitt
7a087415a2 hash instead of block_hash 2023-04-20 19:55:18 -07:00
Bryan Stitt
4d2c0dae53 no sasl 2023-04-20 18:08:53 -07:00
Bryan Stitt
684b14c8a7 enable sasl for kafka 2023-04-20 14:48:15 -07:00
Bryan Stitt
77a317fdf8 default window to 60 seconds 2023-04-20 13:35:59 -07:00
Bryan Stitt
4ce372e57b lint 2023-04-20 10:38:10 -07:00
Bryan Stitt
03c4990dc3 remove commented out code 2023-04-20 10:36:56 -07:00
Bryan Stitt
9929e59f04 lower log level 2023-04-20 09:17:18 -07:00
Bryan Stitt
30eb6e91bf cargo upgrade 2023-04-19 21:09:17 -07:00
Bryan Stitt
03ef27baa4 Merge branch 'main' into devel 2023-04-19 15:31:08 -07:00
Bryan Stitt
b0653c0e9f add log for connecting to kafka 2023-04-19 15:30:01 -07:00
Bryan Stitt
10d2d46512 more packages for kafka+ssl 2023-04-19 15:22:27 -07:00
Bryan Stitt
f5775e5577 add security.protocol to search_kafka script 2023-04-19 14:28:28 -07:00
Bryan Stitt
decdac4b6d set protocol 2023-04-19 10:42:57 -07:00
Bryan Stitt
db3b157a47 really. shut up. i mean it 2023-04-18 21:19:55 -07:00
Bryan Stitt
9118117fe7 tell ethers to shut up 2023-04-18 20:50:35 -07:00
Bryan Stitt
4889c3e1ce Merge branch 'main' into devel 2023-04-18 20:38:43 -07:00
Bryan Stitt
e921d02eb2 remove stale todo 2023-04-17 17:10:09 -07:00
Bryan Stitt
5ff28943da fix incorrect context 2023-04-14 03:44:32 -07:00
Bryan Stitt
2652f88f52 attach more contexts 2023-04-14 03:43:49 -07:00
Bryan Stitt
fffd645acf remove copypasta comment 2023-04-14 02:10:49 -07:00
Bryan Stitt
c2710858e1 only warn if rpc_configs is too short 2023-04-14 00:41:51 -07:00
Bryan Stitt
df19619b77 add context to apply_server_configs 2023-04-14 00:38:49 -07:00
Bryan Stitt
8ed71e1cf1 more goerli fixes 2023-04-14 00:36:46 -07:00
Bryan Stitt
3621d71037 if not watching heads, send to any server 2023-04-14 00:15:27 -07:00
Bryan Stitt
f3435bc6e0 add bundler_4337_rpcs 2023-04-14 00:04:35 -07:00
Bryan Stitt
d035049c8f add /backups_needed endpoint for easy alerts 2023-04-13 17:15:01 -07:00
Bryan Stitt
3c5f973107 more stats 2023-04-11 12:04:16 -07:00
Bryan Stitt
b1120e635a change some error codes jsonrpc error handling 2023-04-10 22:29:02 -07:00
Bryan Stitt
b5ed0c4710 sometimes watch_for_block instead of returning an error 2023-04-10 22:28:31 -07:00
Bryan Stitt
ed5ca0575c logging improvements 2023-04-10 16:05:53 -07:00
Bryan Stitt
29c015508a fix warnings in cargo doc 2023-04-10 15:06:40 -07:00
Bryan Stitt
7250a0c155 upgrade ethers to 2.0.2 2023-04-10 15:01:36 -07:00
Bryan Stitt
55924ff840 cargo upgrade the easy things 2023-04-10 14:27:06 -07:00
Bryan Stitt
767920c055 remove log based debug lines 2023-04-10 14:19:13 -07:00
Bryan Stitt
b460f1701a rename revert_handler to error_handler 2023-04-05 15:34:28 -07:00
Bryan Stitt
072119901e remove stale comment 2023-04-05 15:26:39 -07:00
Bryan Stitt
7994c894bb cargo clippy and fmt 2023-04-05 15:15:20 -07:00
Bryan Stitt
1a160a8c7d Merge remote-tracking branch 'rorytrent/Web3ProxyError' into devel 2023-04-05 15:08:57 -07:00
Bryan Stitt
2b30422b84 fixes for NULL and UNIQUE to work together 2023-04-05 14:55:37 -07:00
Bryan Stitt
e90175c638 remove unused atomics 2023-04-05 14:37:59 -07:00
Bryan Stitt
09dacb45e2 remove stale comment 2023-04-05 14:37:48 -07:00
Bryan Stitt
2e934a41bf cargo upgrade 2023-04-05 12:30:52 -07:00
Bryan Stitt
05db94177d cargo fmt 2023-04-05 12:19:03 -07:00
yenicelik
0069e76040 latest changes from bryan 2023-04-04 14:40:22 +02:00
Rory Neithinger
f0d8a0c8c9 Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-04-01 00:04:44 -07:00
Rory Neithinger
ffdf25787f Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-31 23:23:30 -07:00
yenicelik
0bb3a2dc06 simple changes around bryans comments 2023-03-31 12:43:41 +01:00
yenicelik
345fa6c513 git status 2023-03-30 13:42:56 +01:00
yenicelik
05463c7ee5 migration seems to work for the most part 2023-03-30 12:54:01 +01:00
yenicelik
7390bb7910 non-lossy data transfer 2023-03-26 15:53:17 +02:00
yenicelik
cef26e66cc will create a PR for some feedback 2023-03-25 19:26:23 +01:00
yenicelik
53c7541fed seems to wait until everything is flushed. I will have to debug some stuff that isnt saved properly in the mysql (and then check influx) 2023-03-25 17:56:45 +01:00
Bryan Stitt
2205a472fd add optional tokio-console and improve reconnect logic 2023-03-22 18:43:13 -07:00
Bryan Stitt
7b223efa4d improve reconnect logic 2023-03-22 16:16:15 -07:00
Bryan Stitt
86b4f39a75 string hostname 2023-03-22 15:36:40 -07:00
Bryan Stitt
b4ca0fb62d better log when servers skipped 2023-03-22 15:19:09 -07:00
Bryan Stitt
c0afc295ca add hostname to status 2023-03-22 15:18:54 -07:00
Bryan Stitt
a69737db98 serialize best_rpcs 2023-03-22 14:23:14 -07:00
Bryan Stitt
8eff48611c more direct consensus finding code 
this hopefully has less bugs. speed isn't super important since this isn't on the host path.
 2023-03-22 13:10:15 -07:00
Bryan Stitt
9fae137e45 shorten log message 2023-03-22 02:06:26 -07:00
Rory Neithinger
e9c1d019fc fix some incorrect error codes and log levels 2023-03-21 22:52:46 -07:00
Rory Neithinger
b012f18b2b Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-21 22:40:39 -07:00
Bryan Stitt
5465d927b4 remove a debug log 2023-03-21 21:22:16 -07:00
Bryan Stitt
d992bb2dc9 skip, don't error 2023-03-21 16:00:34 -07:00
yenicelik
1d72a3cd44 will continue after bryan response 2023-03-21 12:49:36 +01:00
yenicelik
64505102ef creating a CLI endpoint for the migration 2023-03-21 12:07:21 +01:00
yenicelik
9beddc43a4 will get and inser some migration data 2023-03-21 10:11:35 +01:00
yenicelik
fbe97c12b2 will continue mtr 2023-03-20 22:00:49 +01:00
Rory Neithinger
fe2a760c8a more conversions to Web3ProxyError 2023-03-20 13:45:21 -07:00
Rory Neithinger
847e961eb0 remove more anyhows from app module 
the only ones left should be in the top level spawn functions.
 2023-03-20 13:07:24 -07:00
Rory Neithinger
60c1a6d382 remove anyhow from rest of frontend module 2023-03-20 12:47:57 -07:00
Rory Neithinger
3479bf9d06 add context type which preserves status code 2023-03-20 11:38:54 -07:00
yenicelik
0a6ccf28b5 will continue with migration 2023-03-20 12:22:30 +01:00
Rory Neithinger
1493d73386 better error handling for ws 2023-03-19 19:14:46 -07:00
Rory Neithinger
beac7ee017 better error handling for proxy_web3_rpc() 2023-03-19 18:52:28 -07:00
Rory Neithinger
c32d12b5e0 better error handling for ip_is_authorized() 2023-03-19 15:50:25 -07:00
yenicelik
c925ec9c27 about to merge stats into it 2023-03-18 10:15:21 +01:00
Rory Neithinger
f3fc4924dc Refactor FrontendErrorResponse into Web3ProxyError 
Renamed FrontendResponse to Web3ProxyResponse and introduced
a new generic type alias Web3ProxyResult.

Fixed a few noisy cargo warnings.
 2023-03-16 22:21:20 -07:00
yenicelik
ca41cb5c9e will push this for short review 2023-03-14 20:08:16 +01:00
yenicelik
65ca628cc7 will modify query so we can make queries more easily 2023-03-14 16:32:19 +01:00
yenicelik
9417961d78 for some reason values output is always zero 2023-03-12 16:09:20 +01:00
yenicelik
8c8c183e65 trying to debug returning multiple files 2023-03-10 20:26:15 +01:00
Bryan Stitt
b0c5d2b0eb remove debugging headers 2023-03-10 09:45:55 -08:00