Commit Graph

141 Commits

Author SHA1 Message Date
Bryan Stitt
06f48ecbbc lint (which captures fdlimit error) 2023-11-09 18:21:32 -08:00
Bryan Stitt
da7bb4b07c 1.43.55 2023-10-20 12:30:54 -07:00
Bryan Stitt
c12ab1fa3e remove extra dep 2023-10-16 12:46:03 -07:00
Bryan Stitt
27ba7adea2 rustls is faster 2023-10-10 23:04:42 -07:00
Bryan Stitt
4f6e75e109 cargo upgrade and update 2023-10-09 09:46:43 -07:00
Bryan Stitt
aeb30787dd more openssl and no rmp
i like rustls, but we can't use it on everything and so our compile times are bad
2023-10-07 00:35:22 -07:00
Bryan Stitt
f0b6465069
streaming responses 2 (#219)
* streaming responses compile, still small TODOs & cleanup

* refactor to allow short curcuits on uncacheable requests (#220)

* refactor to allow short curcuits on uncacheable requests

* handle error inside an Arc from moka

* arc instead of box

* lint

* more lint

* add bonus rate limits and keep all semaphores after rate limits (#221)

* add bonus rate limits and keep all semaphores after rate limits

* remove stale todos

* better function names

* cargo upgrade and update

* make some panics warn. more todo

* pass request_metadata through so streaming responses include their final size

* remove stale TODO

* remove more stale todos

* add file i missed

* make to_json_string work well enough

* check config for free_subscriptions

---------

Co-authored-by: Rory Neithinger <rory@llamanodes.com>
2023-09-26 18:18:06 -07:00
Bryan Stitt
d732b8a171 cargo upgrade and update
ordered-float went from v3 to v4
2023-09-25 10:02:04 -07:00
Bryan Stitt
a68a98ce41 cargo upgrade and update
moka 0.12 required some migration help: https://github.com/moka-rs/moka/blob/main/MIGRATION-GUIDE.md
2023-09-18 11:04:22 -07:00
Bryan Stitt
5913064809 add high concurrency tier 2023-09-11 11:32:24 -07:00
Bryan Stitt
0dcd368ac7 v1.42.18 2023-09-06 13:17:35 -07:00
Bryan Stitt
ec533c8b30 cargo upgrade and update 2023-08-30 20:59:06 -07:00
Bryan Stitt
6be05bab4b cargo upgrade 2023-08-17 13:49:45 -07:00
Bryan Stitt
aa71a406bb cargo upgrade 2023-08-15 15:39:13 -07:00
Bryan Stitt
5257329559 cargo upgrade 2023-08-04 16:19:43 -07:00
Bryan Stitt
32a73a0c53 cargo upgrade --incompatible 2023-07-27 12:29:10 -07:00
Bryan Stitt
d2f0d45dd2 reduce out of funds tier to match free config and bump to version 1 2023-07-26 16:40:08 -07:00
Bryan Stitt
9bc1e5a783 update for the latest contracts 2023-07-26 16:34:21 -07:00
Bryan Stitt
4dfef7a9f7 let it fail 2023-07-26 11:08:55 -07:00
Bryan Stitt
ca9a863138 actually add the file 2023-07-26 09:40:29 -07:00
Bryan Stitt
4d8836538d drop old fk 2023-07-26 09:33:15 -07:00
Bryan Stitt
b1f447c5c8
add mass_grant_credits helper script and other prep for premium (#196)
* add mass_grant_credits helper script

* include error in the rate limit log

* allow granting 0 credits

* migration to change default user_tier to premium
2023-07-26 07:33:00 -07:00
Bryan Stitt
05e618f74c
Anon rpc accounting (#194)
* cargo generate migrations didnt work without this

* anon rpc accounting

* Revert "anon rpc accounting"

This reverts commit f1b8cbcdeca0d5f9545da32f11dda6a90854f97a.

* 0 for anon in mysql

* only set approximate_balance_remaining on owned stats

* only set self.approximate_balance_remaining when adding if some

* todo complete
2023-07-24 22:02:33 -07:00
Bryan Stitt
6f28f7c337 lint 2023-07-14 00:22:26 -07:00
David
31f840432d
Add DateCreated fields to admin and on-chain deposit (#190)
* tests seem to be passing

* bump version

* bump version
2023-07-13 19:25:14 -07:00
Bryan Stitt
46127149b9 lint 2023-07-13 18:12:07 -07:00
David
d8e4115d5a
fix default timestamp for stripe table (#189)
* fix default timestamp for stripe table

* changed time local to time utc

* keep chrono for now
2023-07-13 13:38:56 -07:00
Bryan Stitt
a560d47c48 lint 2023-07-09 20:13:03 -07:00
Bryan Stitt
b527f5d0d6
David/remove balance logic non destructive (#173)
* WIP will work on locks

* added concurrent test for referral, numbers rn still add up

* will change balance table to be a view instead of a table

* merged in devel. will proceed with migrations for rpc_stats_v2 and referral

* WIP

* WIP

* WIP compiling

* gotta add the balance_v2 view

* about to add balance view

* stupid missing closing param

* compiles again, now i will just have to implement the raw sql

* need to paste the real query and play with it

* ok now on to testing

* addresses many comments in PR

* app works for simple unauthorized access. will look into tests

* some tests pass, referral logic tests fail

* not sure why the test is failing

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
2023-07-09 19:23:32 -07:00
Bryan Stitt
b234265458
Premium tier changes (#170)
* lower premium concurrency in preparation for key+IP limits

* include the ip in the user semaphore

* 3, not 5

this is our current limit for free

* per user_id+ip rate limiting
2023-07-07 15:15:41 -07:00
Bryan Stitt
04d58a68dd fix version number to match migration count 2023-07-05 20:58:24 -07:00
Bryan Stitt
eb7b98fdbe
Split errors (#158)
* add migration for splitting errors

* fix type from u32 to u64

* update entities to match migrations

* no migration needed. these are only in stats

* add user_error_response to influx

* only if detailed

* set error_response and user_error_response

* 0 cost error responses

* only 33 migrations now

* put macros back

* get the stat buffer sender to the TestApp helper

* fixes
2023-07-05 18:18:10 -07:00
Bryan Stitt
501ec21442 drop workspace-hack for now
it doesn't work well with features and so we were installing things like kafka even if the feature was off
2023-06-30 13:18:37 -07:00
Bryan Stitt
645fa7328b cargo upgrade tokio 2023-06-29 23:16:31 -07:00
Bryan Stitt
581af0d596
stripe integration first mvp (replaces #140) (#154)
* stripe integration first mvp

* addressing some points from #140

* stripe endpoint needs testing

* addressed some comments

* added Json(payload)

* update toolchain

* lint (well, most of it)

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
2023-06-29 14:32:05 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
272f7ce800 add cargo hakari and chef to build process 2023-06-27 11:59:49 -07:00
Bryan Stitt
6038351cb8 change default tracking in prep for premium-only keys 2023-06-19 13:00:57 -07:00
Bryan Stitt
891fb7b0b3 copypasta 2023-06-18 16:15:15 -07:00
Bryan Stitt
9dd0bae40a text instead of string payload 2023-06-18 16:13:55 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129)
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
2023-06-16 00:46:27 -07:00
David
3f76b08364
Test balance and referral accounting (#111)
* removed bloom filter temporarily, added some fixes with decimals in payment.rs

* balance accounting for a single user (paid and free tier) seems to be ok

* fixed some balance accounting

* compute_cost reduced back to 0
2023-06-08 10:08:29 -07:00
Bryan Stitt
5c1569f9c2 clippy lint 2023-06-07 17:31:15 -07:00
David
de7d8919d3
Change balance to record total spend and total deposits (#109)
* lets test total deposit and total spent

* removed referrer from cache for performance reasons
2023-06-07 14:45:57 -07:00
Bryan Stitt
9e935e2c44 cargo upgrade 2023-05-29 09:43:51 -07:00
Bryan Stitt
0d15d5baf7 bump versions to match number of migrations 2023-05-23 20:58:29 -07:00
David
ddfd1dbff1
Added admin balance increase, and also test (#72)
* added admin balance increase, and also test

* slightly hacky, but does do an update expression

* added note column to entities

* removed redundant printlns

* flattened items

* removed user tier logic from admin increase balance completely. admin will have to manually increase the user tier
2023-05-22 17:07:17 -07:00
Bryan Stitt
2080739865
Quick cache (#70)
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343fc51338a8a4282dcc229b50097835b.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
2023-05-13 11:13:02 -07:00
Bryan Stitt
8a097dabbe
Bryan devel 2023-05-12 (#67)
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
2023-05-12 15:15:32 -07:00
David
34ed450fab
User Balance + Referral Logic (#44)
* will implement balance topup endpoint

* will quickly fix other PR reviews

* merging from master

* will finish up godmoe

* will finish up login

* added logic to top up balance (first iteration)

* should implement additional columns soon (currency, amount, tx-hash), as well as a new table for spend

* updated migrations, will account for spend next

* get back to this later

* will merge PR from stats-v2

* stats v2

rebased all my commits and squashed them down to one

* cargo upgrade

* added migrtation for spend in accounting table. will run test-deposit next

* trying to get request from polygon

* first iteration /user/balance/:tx_hash works, needs to add accepted tokens next

* creating the referral code seems to work

* will now check if spending enough credits will lead to both parties receiving credits

* rpcstats takes care of accounting for spend data

* removed track spend from table

* Revert "removed track spend from table"

This reverts commit a50802d6ae75f786864c5ec42d0ceb2cb27124ed.

* Revert "rpcstats takes care of accounting for spend data"

This reverts commit 1cec728bf241e4cfd24351134637ed81c1a5a10b.

* removed rpc request table entity

* updated referral code to use ulid s

* credits used are aggregated

* added a bunch of fields to referrer

* added database logic whenever an aggregate stats is added. will have to iterate over this a couple times i think. go to (1) detecting accepted stables next, (2) fix influxdb bug and (3) start to write test

* removed track spend as this will occur in the database

* will first work on "balance", then referral. these should really be treated as two separate PRs (although already convoluted)

* balance logic initial commit

* breaking WIP, changing the RPC call logic functions

* will start testing next

* got rid of warnings & lint

* will proceed with subtracting / adding to balance

* added decimal points, balance tracking seems to work

* will beautify code a bit

* removed deprecated dependency, and added topic + deposit contract to app.yaml

* brownie test suite does not rely on local contract files it pulls all from polygonscan

* will continue with referral

* should perhaps (in a future revision) recordhow much the referees got for free. marking referrals seems to work rn

* user is upgraded to premium if they deposit more than 10$. we dont accept more than $10M in a single tx

* will start PR, referral seems to be fine so far, perhaps up to some numbers that still may need tweaking

* will start PR

* removed rogue comments, cleaned up payments a bit

* changes before PR

* apply stats

* added unique constraint

* some refactoring such that the user file is not too bloated

* compiling

* progress with subusers, creating a table entry seems to work

* good response type is there as well now, will work on getters from primary user and secondary user next

* subuser logic also seems fine now

* downgrade logic

* fixed bug influxdb does not support different types in same query (which makes sense)

* WIP temporary commit

* merging with PR

* Delete daemon.rs

there are multiple daemons now, so this was moved to `proxyd`

* will remove request clone to &mut

* multiple request handles for payment

* making requests still seem fine

* removed redundant commented out bits

* added deposit endpoint, added deposit amount and deposit user, untested yet

* small bug with downgrade tier id

* will add authorization so balance can be received for users

* balance history should be set now too

* will check balance over time again

* subususer can see rpc key balance if admin or owner

* stats also seems to work fine now with historical balance

* things seem to be building and working

* removed clone from  OpenRequestHandle

* removed influxdb from workspace members

* changed config files

* reran sea-orm generate entities, added a foreign key, should be proper now

* removed contract from commit

* made deposit contract optional

* added topic in polygon dev

* changed deposit contract to deposit factory contract

* added selfrelation on user_tier

* added payment required

* changed chain id to u64

* add wss in polygon llamarpc

* removed origin and method from the table

* added onchain transactions naming (and forgot to add a migration before)

* changed foreign key to be the referrer (id), not the code itself

* forgot to add id as the target foreign key

* WIP adding cache to update role

* fixed merge conflicts

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
Co-authored-by: Bryan Stitt <bryan@stitthappens.com>
2023-05-12 09:45:15 -07:00