Commit Graph

925 Commits

Author SHA1 Message Date
Bryan Stitt 05f862f3e4 cargo upgrade 2023-05-22 15:50:00 -07:00
Bryan Stitt 3ac15558e3 fix decay time 2023-05-22 15:43:39 -07:00
Bryan Stitt 91eeee23e2 use ether's reconnects instead of our own which need a lock (wip) 2023-05-22 15:32:27 -07:00
yenicelik 59634139c9 updated error_response 2023-05-21 20:27:52 -04:00
yenicelik 29678b025e updated error_response 2023-05-21 20:27:46 -04:00
yenicelik 46afe8b1e4 influxdb query also groups by archive_needed now 2023-05-21 17:13:15 -04:00
yenicelik 795548580a now should also group by archive_needed (turns out this was wanted) 2023-05-21 16:40:42 -04:00
Bryan Stitt c443f76623 refactor tiers again 2023-05-19 14:09:03 -07:00
Bryan Stitt 5aae318620 logs and comments 2023-05-18 15:11:29 -07:00
Bryan Stitt 9c584354d9 add to skip list earlier 2023-05-18 13:51:28 -07:00
Bryan Stitt a92c93706b dont check heads while waiting
this isn't great. but should work better than what we have
2023-05-18 13:47:00 -07:00
Bryan Stitt 8898ffe578 cargo upgrade 2023-05-18 13:36:35 -07:00
Bryan Stitt b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt 24439c5143 disable balance join for now 2023-05-18 00:43:33 -07:00
Bryan Stitt 0f367d9035 more quick cache ttl 2023-05-18 00:04:30 -07:00
Bryan Stitt 1c61390ad1 check if watch_consensus_rpcs.changed has failed
i dont think it ever will, but better safe than sorry
2023-05-16 23:42:42 -07:00
Bryan Stitt c818a8fcd7 default balance to zero instead of panicing 2023-05-16 23:40:32 -07:00
Bryan Stitt 89961331af refactor wait_for_block into should_wait_for_block 2023-05-16 22:29:36 -07:00
Bryan Stitt a07da30042 keep usable_rpcs_by_tier_and_head_number around longer 2023-05-16 18:32:33 -07:00
Bryan Stitt 0f79d2e60d more loud logs 2023-05-16 18:29:56 -07:00
Bryan Stitt 58b7799cc5 for now, always return 0 for cost 2023-05-16 18:25:56 -07:00
Bryan Stitt a0d8218eb1 best block num has to check has_block_data 2023-05-16 18:18:56 -07:00
Bryan Stitt 30ea532c70 display, not debug 2023-05-16 16:13:56 -07:00
Bryan Stitt 5d64524aa6 add name to web3rpcs 2023-05-16 16:04:17 -07:00
Bryan Stitt efaaf1bb5f send the correct content type 2023-05-16 15:27:18 -07:00
Bryan Stitt 525f6d4a34 better variable names 2023-05-16 14:58:00 -07:00
Bryan Stitt 40b15579ab Merge remote-tracking branch 'origin/devel' into quick_cache_ttl 2023-05-16 14:47:43 -07:00
Bryan Stitt f4cebde53f use ttl quick cache 2023-05-16 14:46:05 -07:00
Bryan Stitt 978c385b3c improve wait for block 2023-05-16 14:02:37 -07:00
Bryan Stitt c66eb6d864 head latency instead of peak latency for now 2023-05-16 12:18:59 -07:00
Bryan Stitt 685c1d039a cargo upgrade 2023-05-13 15:00:24 -07:00
Bryan Stitt 2080739865
Quick cache (#70)
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
2023-05-13 11:13:02 -07:00
Bryan Stitt c3cdf7ef43 fix consensus finding bug 2023-05-12 23:40:15 -07:00
Bryan Stitt 4b7f88367d fix serializing /status 2023-05-12 23:22:20 -07:00
Bryan Stitt 74a602bc9c wip 2023-05-12 23:20:59 -07:00
Bryan Stitt a3b97fda9b lint 2023-05-12 23:04:56 -07:00
Bryan Stitt c54970da0a change ordering and move fetch_add and fetch_sub 2023-05-12 23:00:03 -07:00
Bryan Stitt 914c3e03a8 remove stale comments 2023-05-12 22:22:21 -07:00
Bryan Stitt 3cea4c3123 update peak_ewma to use request latency and fix off by one 2023-05-12 22:21:11 -07:00
Bryan Stitt 0c990b0755 use a cancel-safe channel for stats 2023-05-12 16:02:43 -07:00
Bryan Stitt 510612d343
use kanal instead of flume or tokio channels (#68) 2023-05-12 15:47:01 -07:00
Bryan Stitt 8a097dabbe
Bryan devel 2023-05-12 (#67)
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
2023-05-12 15:15:32 -07:00
David 34ed450fab
User Balance + Referral Logic (#44)
* will implement balance topup endpoint

* will quickly fix other PR reviews

* merging from master

* will finish up godmoe

* will finish up login

* added logic to top up balance (first iteration)

* should implement additional columns soon (currency, amount, tx-hash), as well as a new table for spend

* updated migrations, will account for spend next

* get back to this later

* will merge PR from stats-v2

* stats v2

rebased all my commits and squashed them down to one

* cargo upgrade

* added migrtation for spend in accounting table. will run test-deposit next

* trying to get request from polygon

* first iteration /user/balance/:tx_hash works, needs to add accepted tokens next

* creating the referral code seems to work

* will now check if spending enough credits will lead to both parties receiving credits

* rpcstats takes care of accounting for spend data

* removed track spend from table

* Revert "removed track spend from table"

This reverts commit a50802d6ae75f786864c5ec42d0ceb2cb27124ed.

* Revert "rpcstats takes care of accounting for spend data"

This reverts commit 1cec728bf241e4cfd24351134637ed81c1a5a10b.

* removed rpc request table entity

* updated referral code to use ulid s

* credits used are aggregated

* added a bunch of fields to referrer

* added database logic whenever an aggregate stats is added. will have to iterate over this a couple times i think. go to (1) detecting accepted stables next, (2) fix influxdb bug and (3) start to write test

* removed track spend as this will occur in the database

* will first work on "balance", then referral. these should really be treated as two separate PRs (although already convoluted)

* balance logic initial commit

* breaking WIP, changing the RPC call logic functions

* will start testing next

* got rid of warnings & lint

* will proceed with subtracting / adding to balance

* added decimal points, balance tracking seems to work

* will beautify code a bit

* removed deprecated dependency, and added topic + deposit contract to app.yaml

* brownie test suite does not rely on local contract files it pulls all from polygonscan

* will continue with referral

* should perhaps (in a future revision) recordhow much the referees got for free. marking referrals seems to work rn

* user is upgraded to premium if they deposit more than 10$. we dont accept more than $10M in a single tx

* will start PR, referral seems to be fine so far, perhaps up to some numbers that still may need tweaking

* will start PR

* removed rogue comments, cleaned up payments a bit

* changes before PR

* apply stats

* added unique constraint

* some refactoring such that the user file is not too bloated

* compiling

* progress with subusers, creating a table entry seems to work

* good response type is there as well now, will work on getters from primary user and secondary user next

* subuser logic also seems fine now

* downgrade logic

* fixed bug influxdb does not support different types in same query (which makes sense)

* WIP temporary commit

* merging with PR

* Delete daemon.rs

there are multiple daemons now, so this was moved to `proxyd`

* will remove request clone to &mut

* multiple request handles for payment

* making requests still seem fine

* removed redundant commented out bits

* added deposit endpoint, added deposit amount and deposit user, untested yet

* small bug with downgrade tier id

* will add authorization so balance can be received for users

* balance history should be set now too

* will check balance over time again

* subususer can see rpc key balance if admin or owner

* stats also seems to work fine now with historical balance

* things seem to be building and working

* removed clone from  OpenRequestHandle

* removed influxdb from workspace members

* changed config files

* reran sea-orm generate entities, added a foreign key, should be proper now

* removed contract from commit

* made deposit contract optional

* added topic in polygon dev

* changed deposit contract to deposit factory contract

* added selfrelation on user_tier

* added payment required

* changed chain id to u64

* add wss in polygon llamarpc

* removed origin and method from the table

* added onchain transactions naming (and forgot to add a migration before)

* changed foreign key to be the referrer (id), not the code itself

* forgot to add id as the target foreign key

* WIP adding cache to update role

* fixed merge conflicts

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
Co-authored-by: Bryan Stitt <bryan@stitthappens.com>
2023-05-12 09:45:15 -07:00
Bryan Stitt 36cc884112 Merge remote-tracking branch 'origin/main' into devel 2023-05-11 20:05:47 -07:00
Bryan Stitt fc8ca4ba4f less max lag 2023-05-11 17:04:33 -07:00
Bryan Stitt 2a242fe37f cargo upgrade --incompatible 2023-05-11 13:25:55 -07:00
Bryan Stitt 6e8536d477 add support for http basic auth 2023-05-11 13:21:12 -07:00
Rory Trent ec11e210ee
Peak ewma (#63)
* use peak-ewma instead of head for latency calculation

* Implement some suggested changes from PR

* move latency to new package in workspace root

* fix unit tests which now require peak_latency on Web3Rpc

* Switch to atomics for peak-ewma

This change is to avoid locking from tokio::sync::watch.

* add decay calculation to latency reads in peak-ewma

* Add some tests for peak-ewma

* Sensible latency defaults and not blocking on full

* Cleanup and a couple additional comments
2023-05-11 13:09:15 -07:00
Bryan Stitt 96a9a39789 add support for http basic auth 2023-05-11 13:07:38 -07:00
Bryan Stitt 0531e2f8dd add 'quota usage' rate limit substring 2023-05-08 11:25:27 -06:00
Bryan Stitt 4d64bf9580 cargo upgrade 2023-05-08 10:44:01 -06:00
Bryan Stitt e7da6cd3d6 clone for now. this will change to use refs soon 2023-05-08 10:42:03 -06:00
Bryan Stitt c2e3d71080 if protected rpcs, only use protected rpcs 2023-05-08 10:39:06 -06:00
Bryan Stitt 8916533a2a set archive_request to true on transaction retrying 2023-05-08 10:38:38 -06:00
Bryan Stitt aff53ce36a automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash
thanks for the report Lefteris @ Rotki
2023-05-08 10:38:23 -06:00
Bryan Stitt ffb554d0b3 online editor isn't actually nice when it misses syntax errors :( 2023-04-29 18:50:26 -07:00
Bryan Stitt 91acbcaf75
Fix sort order 2023-04-29 18:38:31 -07:00
Bryan Stitt 93f243696a
simple match statement 2023-04-27 11:15:45 -07:00
Bryan Stitt a236f7364d
DRY stats (#58)
* DRY functions for saving stats and batch tsdb stats

* move comments
2023-04-25 07:42:00 -07:00
Bryan Stitt cc3c9462a3 cargo upgrade 2023-04-25 05:39:09 -07:00
Bryan Stitt 1fab1b1d46 oops. lower this log level 2023-04-24 11:21:53 -07:00
Bryan Stitt 56fdf48129
Send transactions to protected and public rpcs (#57)
* move protected transactions into their own function and dry stats sending

* cargo upgrade

* comments

* time to live instead of time to idle

* minor workaround for eth_chainId

* cargo upgrade
2023-04-24 11:00:12 -07:00
Bryan Stitt 7a087415a2 hash instead of block_hash 2023-04-20 19:55:18 -07:00
Bryan Stitt 4d2c0dae53 no sasl 2023-04-20 18:08:53 -07:00
Bryan Stitt 684b14c8a7 enable sasl for kafka 2023-04-20 14:48:15 -07:00
Bryan Stitt 77a317fdf8 default window to 60 seconds 2023-04-20 13:35:59 -07:00
Bryan Stitt 4ce372e57b lint 2023-04-20 10:38:10 -07:00
Bryan Stitt 03c4990dc3 remove commented out code 2023-04-20 10:36:56 -07:00
Bryan Stitt 9929e59f04 lower log level 2023-04-20 09:17:18 -07:00
Bryan Stitt 30eb6e91bf cargo upgrade 2023-04-19 21:09:17 -07:00
Bryan Stitt 03ef27baa4 Merge branch 'main' into devel 2023-04-19 15:31:08 -07:00
Bryan Stitt b0653c0e9f add log for connecting to kafka 2023-04-19 15:30:01 -07:00
Bryan Stitt 10d2d46512 more packages for kafka+ssl 2023-04-19 15:22:27 -07:00
Bryan Stitt f5775e5577 add security.protocol to search_kafka script 2023-04-19 14:28:28 -07:00
Bryan Stitt decdac4b6d set protocol 2023-04-19 10:42:57 -07:00
Bryan Stitt db3b157a47 really. shut up. i mean it 2023-04-18 21:19:55 -07:00
Bryan Stitt 9118117fe7 tell ethers to shut up 2023-04-18 20:50:35 -07:00
Bryan Stitt 4889c3e1ce Merge branch 'main' into devel 2023-04-18 20:38:43 -07:00
Bryan Stitt e921d02eb2 remove stale todo 2023-04-17 17:10:09 -07:00
Bryan Stitt 5ff28943da fix incorrect context 2023-04-14 03:44:32 -07:00
Bryan Stitt 2652f88f52 attach more contexts 2023-04-14 03:43:49 -07:00
Bryan Stitt fffd645acf remove copypasta comment 2023-04-14 02:10:49 -07:00
Bryan Stitt c2710858e1 only warn if rpc_configs is too short 2023-04-14 00:41:51 -07:00
Bryan Stitt df19619b77 add context to apply_server_configs 2023-04-14 00:38:49 -07:00
Bryan Stitt 8ed71e1cf1 more goerli fixes 2023-04-14 00:36:46 -07:00
Bryan Stitt 3621d71037 if not watching heads, send to any server 2023-04-14 00:15:27 -07:00
Bryan Stitt f3435bc6e0 add bundler_4337_rpcs 2023-04-14 00:04:35 -07:00
Bryan Stitt d035049c8f add /backups_needed endpoint for easy alerts 2023-04-13 17:15:01 -07:00
Bryan Stitt 3c5f973107 more stats 2023-04-11 12:04:16 -07:00
Bryan Stitt b1120e635a change some error codes jsonrpc error handling 2023-04-10 22:29:02 -07:00
Bryan Stitt b5ed0c4710 sometimes watch_for_block instead of returning an error 2023-04-10 22:28:31 -07:00
Bryan Stitt ed5ca0575c logging improvements 2023-04-10 16:05:53 -07:00
Bryan Stitt 29c015508a fix warnings in cargo doc 2023-04-10 15:06:40 -07:00
Bryan Stitt 7250a0c155 upgrade ethers to 2.0.2 2023-04-10 15:01:36 -07:00
Bryan Stitt 55924ff840 cargo upgrade the easy things 2023-04-10 14:27:06 -07:00
Bryan Stitt 767920c055 remove log based debug lines 2023-04-10 14:19:13 -07:00
Bryan Stitt b460f1701a rename revert_handler to error_handler 2023-04-05 15:34:28 -07:00
Bryan Stitt 072119901e remove stale comment 2023-04-05 15:26:39 -07:00
Bryan Stitt 7994c894bb cargo clippy and fmt 2023-04-05 15:15:20 -07:00
Bryan Stitt 1a160a8c7d Merge remote-tracking branch 'rorytrent/Web3ProxyError' into devel 2023-04-05 15:08:57 -07:00
Bryan Stitt 2b30422b84 fixes for NULL and UNIQUE to work together 2023-04-05 14:55:37 -07:00
Bryan Stitt e90175c638 remove unused atomics 2023-04-05 14:37:59 -07:00
Bryan Stitt 09dacb45e2 remove stale comment 2023-04-05 14:37:48 -07:00
Bryan Stitt 2e934a41bf cargo upgrade 2023-04-05 12:30:52 -07:00
Bryan Stitt 05db94177d cargo fmt 2023-04-05 12:19:03 -07:00
yenicelik 0069e76040 latest changes from bryan 2023-04-04 14:40:22 +02:00
Rory Neithinger f0d8a0c8c9 Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-04-01 00:04:44 -07:00
Rory Neithinger ffdf25787f Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-31 23:23:30 -07:00
yenicelik 0bb3a2dc06 simple changes around bryans comments 2023-03-31 12:43:41 +01:00
yenicelik 345fa6c513 git status 2023-03-30 13:42:56 +01:00
yenicelik 05463c7ee5 migration seems to work for the most part 2023-03-30 12:54:01 +01:00
yenicelik 7390bb7910 non-lossy data transfer 2023-03-26 15:53:17 +02:00
yenicelik cef26e66cc will create a PR for some feedback 2023-03-25 19:26:23 +01:00
yenicelik 53c7541fed seems to wait until everything is flushed. I will have to debug some stuff that isnt saved properly in the mysql (and then check influx) 2023-03-25 17:56:45 +01:00
Bryan Stitt 2205a472fd add optional tokio-console and improve reconnect logic 2023-03-22 18:43:13 -07:00
Bryan Stitt 7b223efa4d improve reconnect logic 2023-03-22 16:16:15 -07:00
Bryan Stitt 86b4f39a75 string hostname 2023-03-22 15:36:40 -07:00
Bryan Stitt b4ca0fb62d better log when servers skipped 2023-03-22 15:19:09 -07:00
Bryan Stitt c0afc295ca add hostname to status 2023-03-22 15:18:54 -07:00
Bryan Stitt a69737db98 serialize best_rpcs 2023-03-22 14:23:14 -07:00
Bryan Stitt 8eff48611c more direct consensus finding code
this hopefully has less bugs. speed isn't super important since this isn't on the host path.
2023-03-22 13:10:15 -07:00
Bryan Stitt 9fae137e45 shorten log message 2023-03-22 02:06:26 -07:00
Rory Neithinger e9c1d019fc fix some incorrect error codes and log levels 2023-03-21 22:52:46 -07:00
Rory Neithinger b012f18b2b Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-21 22:40:39 -07:00
Bryan Stitt 5465d927b4 remove a debug log 2023-03-21 21:22:16 -07:00
Bryan Stitt d992bb2dc9 skip, don't error 2023-03-21 16:00:34 -07:00
yenicelik 1d72a3cd44 will continue after bryan response 2023-03-21 12:49:36 +01:00
yenicelik 64505102ef creating a CLI endpoint for the migration 2023-03-21 12:07:21 +01:00
yenicelik 9beddc43a4 will get and inser some migration data 2023-03-21 10:11:35 +01:00
yenicelik fbe97c12b2 will continue mtr 2023-03-20 22:00:49 +01:00
Rory Neithinger fe2a760c8a more conversions to Web3ProxyError 2023-03-20 13:45:21 -07:00
Rory Neithinger 847e961eb0 remove more anyhows from app module
the only ones left should be in the top level spawn functions.
2023-03-20 13:07:24 -07:00
Rory Neithinger 60c1a6d382 remove anyhow from rest of frontend module 2023-03-20 12:47:57 -07:00
Rory Neithinger 3479bf9d06 add context type which preserves status code 2023-03-20 11:38:54 -07:00
yenicelik 0a6ccf28b5 will continue with migration 2023-03-20 12:22:30 +01:00
Rory Neithinger 1493d73386 better error handling for ws 2023-03-19 19:14:46 -07:00
Rory Neithinger beac7ee017 better error handling for proxy_web3_rpc() 2023-03-19 18:52:28 -07:00
Rory Neithinger c32d12b5e0 better error handling for ip_is_authorized() 2023-03-19 15:50:25 -07:00
yenicelik c925ec9c27 about to merge stats into it 2023-03-18 10:15:21 +01:00
Rory Neithinger f3fc4924dc Refactor FrontendErrorResponse into Web3ProxyError
Renamed FrontendResponse to Web3ProxyResponse and introduced
a new generic type alias Web3ProxyResult.

Fixed a few noisy cargo warnings.
2023-03-16 22:21:20 -07:00
yenicelik ca41cb5c9e will push this for short review 2023-03-14 20:08:16 +01:00
yenicelik 65ca628cc7 will modify query so we can make queries more easily 2023-03-14 16:32:19 +01:00
yenicelik 9417961d78 for some reason values output is always zero 2023-03-12 16:09:20 +01:00
yenicelik 8c8c183e65 trying to debug returning multiple files 2023-03-10 20:26:15 +01:00
Bryan Stitt b0c5d2b0eb remove debugging headers 2023-03-10 09:45:55 -08:00
Bryan Stitt 684c37ef6c cargo upgrade 2023-03-10 09:45:47 -08:00
Bryan Stitt 89853e24e5 all strings for json 2023-03-09 22:53:01 -08:00
Bryan Stitt babd215e69 add an endpoint for debugging headers 2023-03-09 22:51:23 -08:00
Bryan Stitt 4203c61a59 use pretty address in logs 2023-03-09 11:25:57 -08:00
Bryan Stitt 7b9bcf6881 include head block in the log 2023-03-09 11:22:40 -08:00