Commit Graph

200 Commits

Author SHA1 Message Date
Bryan Stitt
86f9e7358a
move balance onto AuthorizationChecks (#131)
* move balance onto AuthorizationChecks

* todone

* bigger default

* flat cost for testing
2023-06-16 23:14:43 -07:00
Bryan Stitt
5d9365449f simplify max item weight by having the weigher send MAX for big things 2023-06-16 16:14:11 -07:00
Bryan Stitt
af7759e491 by_name needs a lock to prevent races during disconnects 2023-06-16 10:40:02 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129)
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
2023-06-16 00:46:27 -07:00
Bryan Stitt
b8f429d70a improvements to popularity_contesst 2023-06-13 11:51:19 -07:00
Bryan Stitt
f5a1ac274a refactor send_head_block_result 2023-06-13 10:00:08 -07:00
Bryan Stitt
602b0117c4 panic if influxdb_bucket is not set when influxdb_host is set 2023-06-13 09:30:41 -07:00
Bryan Stitt
6324d9c7d9 short circuit if no requests 2023-06-12 21:54:33 -07:00
Bryan Stitt
c771479d94 use WyRand instead of Xoshiro 2023-06-12 19:46:36 -07:00
Bryan Stitt
62bda636b8 return the response instead of panicing 2023-06-12 16:38:57 -07:00
Bryan Stitt
4a9c1a0ce6 move user registration into a dedicated function 2023-06-09 18:31:47 -07:00
Bryan Stitt
c37b320286 add cache names and actually serialize 2023-06-07 17:55:34 -07:00
Bryan Stitt
40bee30730 back to moka 2023-06-07 17:30:05 -07:00
Bryan Stitt
4f7144abc6 Arc instead of Box for better caching 2023-06-07 13:57:38 -07:00
David
63499c1564
David/66 downgrade balance logic (#89)
* tried to make the balance_entry atomic, asking for feedback

* added locked select everywhere and inside the txn, it should be atomic now

* forgot about overwrite (referee vs sender entities were the same, overwriting each other), fixed it

* will now trigger refreshing cache when balance thresholds are met

* will also invalidate cache if balance gets too low (to out of funds)

* added separate balance cache with shorter TTL. need to know if i should modify it (rn its not atomic), or if low TTL is good enough, and we just fetch the value every now and then from the Database

* removed UserTier struct bcs duplicate with database

* removed subuser premium requirement (which would have lead to a merge conflict later on)

* removed user_tier artefact

* replaced cache to use AtomicF64

* a bunch of small changes

* some changes

* will merge with devel

* changed AtomicF64 to RwLock<Decimal>

* changed AtomicF64 to RwLock<Decimal>

* downgrading user when at 0.1$ or if sum_credits_used is very low

* changed caches to be more aggressive in being non-empty

* replaced Arc::clone() by reference
2023-06-07 09:39:30 -07:00
David
f66edd961b
Making Balance Changes Atomic (Read and Write) (#84)
* tried to make the balance_entry atomic, asking for feedback

* added locked select everywhere and inside the txn, it should be atomic now

* forgot about overwrite (referee vs sender entities were the same, overwriting each other), fixed it

* will now trigger refreshing cache when balance thresholds are met

* Revert "will now trigger refreshing cache when balance thresholds are met"

This reverts commit b1215ddc43a1358f6086bddbd106996165431165.

* will not modify userTier in database anymore

* made payment atomic

* made updates mostly atomic, locking only the referral table (bcs more complex logic)

* latest changes
2023-06-04 09:32:53 -07:00
Bryan Stitt
fbe0ecfbff rate limit on websockets 2023-05-31 16:05:44 -07:00
Bryan Stitt
7947cb95ff use bloom filters and support transactions with multiple deposit events 2023-05-31 11:20:17 -07:00
Bryan Stitt
d64761403b Cow instead of String 2023-05-30 23:17:05 -07:00
Bryan Stitt
f1636d3b85 unlimited localhost 2023-05-30 22:31:35 -07:00
Bryan Stitt
fd8bba4bdd most of these are for the whole app, not just the frontend 2023-05-30 21:26:11 -07:00
Bryan Stitt
cebe8ed1ea
Request refactor (#99)
* wip

* AsRef finally works like i wanted

* actually return the block

* start adding async trait

* remove stale import

* include id in the error response when possible

* remove stale comments
2023-05-30 16:32:34 -07:00
Bryan Stitt
c1eba556a5 fix missing ttl send and add name to cache 2023-05-29 17:19:05 -07:00
Bryan Stitt
bb50efb7f9
quick cache ttl max item size, better function names, and tests (#97)
* Revert "remove cache on /status and /health for now"

This reverts commit 166b0d810cd25b895560d50d00383a57e1269ad4.

* Revert "remove cache on backups needed, too"

This reverts commit 4597967def2f1daa71a277ec6731db4739ead640.

* add tests

* add max_item_weight

* comment

* add some helper functions, max weights, and tests
2023-05-29 15:48:22 -07:00
Bryan Stitt
dafb69fae1 move timeout deeper 2023-05-23 15:50:29 -07:00
Bryan Stitt
78f247fc6c split http and ws provider 2023-05-23 14:40:34 -07:00
Bryan Stitt
b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt
0f367d9035 more quick cache ttl 2023-05-18 00:04:30 -07:00
Bryan Stitt
30ea532c70 display, not debug 2023-05-16 16:13:56 -07:00
Bryan Stitt
5d64524aa6 add name to web3rpcs 2023-05-16 16:04:17 -07:00
Bryan Stitt
f4cebde53f use ttl quick cache 2023-05-16 14:46:05 -07:00
Bryan Stitt
2080739865
Quick cache (#70)
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343fc51338a8a4282dcc229b50097835b.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
2023-05-13 11:13:02 -07:00
Bryan Stitt
c54970da0a change ordering and move fetch_add and fetch_sub 2023-05-12 23:00:03 -07:00
Bryan Stitt
0c990b0755 use a cancel-safe channel for stats 2023-05-12 16:02:43 -07:00
Bryan Stitt
510612d343
use kanal instead of flume or tokio channels (#68) 2023-05-12 15:47:01 -07:00
Bryan Stitt
8a097dabbe
Bryan devel 2023-05-12 (#67)
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
2023-05-12 15:15:32 -07:00
David
34ed450fab
User Balance + Referral Logic (#44)
* will implement balance topup endpoint

* will quickly fix other PR reviews

* merging from master

* will finish up godmoe

* will finish up login

* added logic to top up balance (first iteration)

* should implement additional columns soon (currency, amount, tx-hash), as well as a new table for spend

* updated migrations, will account for spend next

* get back to this later

* will merge PR from stats-v2

* stats v2

rebased all my commits and squashed them down to one

* cargo upgrade

* added migrtation for spend in accounting table. will run test-deposit next

* trying to get request from polygon

* first iteration /user/balance/:tx_hash works, needs to add accepted tokens next

* creating the referral code seems to work

* will now check if spending enough credits will lead to both parties receiving credits

* rpcstats takes care of accounting for spend data

* removed track spend from table

* Revert "removed track spend from table"

This reverts commit a50802d6ae75f786864c5ec42d0ceb2cb27124ed.

* Revert "rpcstats takes care of accounting for spend data"

This reverts commit 1cec728bf241e4cfd24351134637ed81c1a5a10b.

* removed rpc request table entity

* updated referral code to use ulid s

* credits used are aggregated

* added a bunch of fields to referrer

* added database logic whenever an aggregate stats is added. will have to iterate over this a couple times i think. go to (1) detecting accepted stables next, (2) fix influxdb bug and (3) start to write test

* removed track spend as this will occur in the database

* will first work on "balance", then referral. these should really be treated as two separate PRs (although already convoluted)

* balance logic initial commit

* breaking WIP, changing the RPC call logic functions

* will start testing next

* got rid of warnings & lint

* will proceed with subtracting / adding to balance

* added decimal points, balance tracking seems to work

* will beautify code a bit

* removed deprecated dependency, and added topic + deposit contract to app.yaml

* brownie test suite does not rely on local contract files it pulls all from polygonscan

* will continue with referral

* should perhaps (in a future revision) recordhow much the referees got for free. marking referrals seems to work rn

* user is upgraded to premium if they deposit more than 10$. we dont accept more than $10M in a single tx

* will start PR, referral seems to be fine so far, perhaps up to some numbers that still may need tweaking

* will start PR

* removed rogue comments, cleaned up payments a bit

* changes before PR

* apply stats

* added unique constraint

* some refactoring such that the user file is not too bloated

* compiling

* progress with subusers, creating a table entry seems to work

* good response type is there as well now, will work on getters from primary user and secondary user next

* subuser logic also seems fine now

* downgrade logic

* fixed bug influxdb does not support different types in same query (which makes sense)

* WIP temporary commit

* merging with PR

* Delete daemon.rs

there are multiple daemons now, so this was moved to `proxyd`

* will remove request clone to &mut

* multiple request handles for payment

* making requests still seem fine

* removed redundant commented out bits

* added deposit endpoint, added deposit amount and deposit user, untested yet

* small bug with downgrade tier id

* will add authorization so balance can be received for users

* balance history should be set now too

* will check balance over time again

* subususer can see rpc key balance if admin or owner

* stats also seems to work fine now with historical balance

* things seem to be building and working

* removed clone from  OpenRequestHandle

* removed influxdb from workspace members

* changed config files

* reran sea-orm generate entities, added a foreign key, should be proper now

* removed contract from commit

* made deposit contract optional

* added topic in polygon dev

* changed deposit contract to deposit factory contract

* added selfrelation on user_tier

* added payment required

* changed chain id to u64

* add wss in polygon llamarpc

* removed origin and method from the table

* added onchain transactions naming (and forgot to add a migration before)

* changed foreign key to be the referrer (id), not the code itself

* forgot to add id as the target foreign key

* WIP adding cache to update role

* fixed merge conflicts

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
Co-authored-by: Bryan Stitt <bryan@stitthappens.com>
2023-05-12 09:45:15 -07:00
Bryan Stitt
e7da6cd3d6 clone for now. this will change to use refs soon 2023-05-08 10:42:03 -06:00
Bryan Stitt
c2e3d71080 if protected rpcs, only use protected rpcs 2023-05-08 10:39:06 -06:00
Bryan Stitt
8916533a2a set archive_request to true on transaction retrying 2023-05-08 10:38:38 -06:00
Bryan Stitt
aff53ce36a automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash
thanks for the report Lefteris @ Rotki
2023-05-08 10:38:23 -06:00
Bryan Stitt
56fdf48129
Send transactions to protected and public rpcs (#57)
* move protected transactions into their own function and dry stats sending

* cargo upgrade

* comments

* time to live instead of time to idle

* minor workaround for eth_chainId

* cargo upgrade
2023-04-24 11:00:12 -07:00
Bryan Stitt
03ef27baa4 Merge branch 'main' into devel 2023-04-19 15:31:08 -07:00
Bryan Stitt
b0653c0e9f add log for connecting to kafka 2023-04-19 15:30:01 -07:00
Bryan Stitt
10d2d46512 more packages for kafka+ssl 2023-04-19 15:22:27 -07:00
Bryan Stitt
decdac4b6d set protocol 2023-04-19 10:42:57 -07:00
Bryan Stitt
4889c3e1ce Merge branch 'main' into devel 2023-04-18 20:38:43 -07:00
Bryan Stitt
fffd645acf remove copypasta comment 2023-04-14 02:10:49 -07:00
Bryan Stitt
df19619b77 add context to apply_server_configs 2023-04-14 00:38:49 -07:00
Bryan Stitt
f3435bc6e0 add bundler_4337_rpcs 2023-04-14 00:04:35 -07:00
Bryan Stitt
ed5ca0575c logging improvements 2023-04-10 16:05:53 -07:00
Bryan Stitt
29c015508a fix warnings in cargo doc 2023-04-10 15:06:40 -07:00
Bryan Stitt
7994c894bb cargo clippy and fmt 2023-04-05 15:15:20 -07:00
Bryan Stitt
1a160a8c7d Merge remote-tracking branch 'rorytrent/Web3ProxyError' into devel 2023-04-05 15:08:57 -07:00
yenicelik
0069e76040 latest changes from bryan 2023-04-04 14:40:22 +02:00
Rory Neithinger
f0d8a0c8c9 Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-04-01 00:04:44 -07:00
Rory Neithinger
ffdf25787f Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-31 23:23:30 -07:00
yenicelik
0bb3a2dc06 simple changes around bryans comments 2023-03-31 12:43:41 +01:00
yenicelik
345fa6c513 git status 2023-03-30 13:42:56 +01:00
yenicelik
05463c7ee5 migration seems to work for the most part 2023-03-30 12:54:01 +01:00
yenicelik
53c7541fed seems to wait until everything is flushed. I will have to debug some stuff that isnt saved properly in the mysql (and then check influx) 2023-03-25 17:56:45 +01:00
Bryan Stitt
86b4f39a75 string hostname 2023-03-22 15:36:40 -07:00
Bryan Stitt
c0afc295ca add hostname to status 2023-03-22 15:18:54 -07:00
Rory Neithinger
b012f18b2b Merge branch 'stats_v3' of github.com:yenicelik/web3-proxy into Web3ProxyError 2023-03-21 22:40:39 -07:00
Rory Neithinger
fe2a760c8a more conversions to Web3ProxyError 2023-03-20 13:45:21 -07:00
Rory Neithinger
847e961eb0 remove more anyhows from app module
the only ones left should be in the top level spawn functions.
2023-03-20 13:07:24 -07:00
Rory Neithinger
1493d73386 better error handling for ws 2023-03-19 19:14:46 -07:00
Rory Neithinger
beac7ee017 better error handling for proxy_web3_rpc() 2023-03-19 18:52:28 -07:00
Rory Neithinger
f3fc4924dc Refactor FrontendErrorResponse into Web3ProxyError
Renamed FrontendResponse to Web3ProxyResponse and introduced
a new generic type alias Web3ProxyResult.

Fixed a few noisy cargo warnings.
2023-03-16 22:21:20 -07:00
yenicelik
8c8c183e65 trying to debug returning multiple files 2023-03-10 20:26:15 +01:00
Bryan Stitt
b480e92259 don't panic if param is not set 2023-03-09 09:32:30 -08:00
Bryan Stitt
eb4d05a520 stats v2
rebased all my commits and squashed them down to one
2023-03-04 20:48:49 +01:00
Bryan Stitt
4c18657387 more kafka things 2023-03-03 06:58:45 -08:00
Bryan Stitt
2eb8df22ec kafka helper script 2023-03-03 02:54:52 -08:00
Bryan Stitt
cbdd6d7c1e oops. didn't mean to commit this script yet 2023-03-02 21:12:35 -08:00
Bryan Stitt
3098791ad9 add optional kafka feature 2023-03-03 01:39:50 +00:00
Bryan Stitt
499d50b5fc handle empty private_rpcs 2023-03-01 23:21:09 +00:00
Bryan Stitt
f68886fafd add simple test for all connections and include more rpcs 2023-03-01 22:32:01 +00:00
Bryan Stitt
d83a1f0169 use Web3Rpc instead of the name as a key 2023-02-28 11:01:34 -08:00
Bryan Stitt
6067369ee3 add inotify and rpc disconnect 2023-02-27 12:50:57 -08:00
Bryan Stitt
475e521918 cache getLogs with blockhash 2023-02-26 23:08:00 -08:00
Bryan Stitt
f8f5e7a1c8 add thread for config file watching and run clippy lint 2023-02-26 23:08:00 -08:00
Bryan Stitt
e96f09a9c4 refactors to make configs partially reloadable 2023-02-25 23:52:33 -08:00
Bryan Stitt
c9e5661c5b wip 2023-02-25 09:55:28 -08:00
Bryan Stitt
0ab7738393 per-chain rpc rate limits 2023-02-21 21:10:23 -08:00
Bryan Stitt
f2cfe2956a the macro is easier 2023-02-15 21:19:24 -08:00
Bryan Stitt
02bada3006 properly encode number responses as hex 2023-02-15 21:16:33 -08:00
Bryan Stitt
df668a5dfd add time to idle to more caches 2023-02-15 11:42:25 -08:00
Bryan Stitt
c65ffc9ce0 getting close 2023-02-14 17:41:40 -08:00
Bryan Stitt
824a6fa1f0 well it compiles. doesnt work yet. but it compiles 2023-02-14 12:14:50 -08:00
Bryan Stitt
c008f50943 use http and ws providers in a single config 2023-02-12 01:40:01 -08:00
Bryan Stitt
f1bc00082a include to_block more places 2023-02-10 20:45:57 -08:00
Bryan Stitt
c959110986 better sorting of connections 2023-02-10 20:24:20 -08:00
Bryan Stitt
4e63e69c97 actually use app.response_cache_max_bytes 2023-02-09 11:55:39 -08:00
Bryan Stitt
5ffe2aa72a eth_sendRawTransactions should use backups more often 2023-02-09 11:47:56 -08:00
Bryan Stitt
ff4cb9abce cargo upgrade 2023-02-08 11:11:16 -08:00
Bryan Stitt
0f280ce483 cargo upgrade and shorten variable names
also begin adding a latency tracker for rpc stats
2023-02-06 09:55:27 -08:00
Bryan Stitt
f2d35ba5eb remove metered in favor of influxdb stats 2023-02-05 18:17:26 -08:00
Bryan Stitt
6d959e2c1f add archive depth to app config 2023-02-05 17:58:03 -08:00
Bryan Stitt
1d749ed33d block all admin_ commands 2023-02-03 10:56:19 -08:00