Commit Graph

121 Commits

Author SHA1 Message Date
Bryan Stitt
e6dc64f369 add timeout as a the tower layer 2023-07-19 18:41:26 -07:00
Bryan Stitt
600c1bafb4
David/multiple proxy test (#187)
* add test_multiple_proxies_stats_add_up

* make a premium user using both proxies

* added a couple clones, must add constraints now to run for multiple-proxies (check arithmetic)

* lint and code review

* fix comment

* fix tests (now fails at todo)

* will introduce endpoint to fetch rpc stats from mysql

* added influxdb to tests, should next do asserst in stats collected by influx, and mysql for multi-proxy

* created test where influx and mysql have separate data, should help with debugging

* forgot to drop influx

* tests pass except multi-proxy

* test passes, will check out nothing broke elswhere

* go back to numbers

* some linting

* linting

* removed redundant info!

* responding to PR comments

* ULID as instance-hash for the tag in influx (for anti-dup)

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
2023-07-17 13:31:41 -07:00
Bryan Stitt
fe672b5189 dont disable on_failure. the log breadcrumbs can be helpful still 2023-07-14 23:04:24 -07:00
Bryan Stitt
0046a02a4a
Global db (#191)
* dont migrate on start. just connect

* it compiles

* that log is loud

* get transaction from a local clone

* import the trait

* put the test app into a thread instead of tokio::spawn

* fix one test

* try db before rpcs

* db connection is too slow. need to wait for it

* do db setup once while spawning
2023-07-14 18:30:01 -07:00
David
c1a3a2dfbb
Remove stripe duplicate endpoint (#186) 2023-07-12 19:19:15 -07:00
Bryan Stitt
2dfc5dcdf3 set http keep alive to a longer time 2023-07-10 18:37:25 -07:00
Bryan Stitt
386363e295 add request id to tracing 2023-07-08 11:57:53 -07:00
David
659fce0e33
Deposit history for stripe and admin (endpoints) (#160)
* added endpoints for stripe and admin deposits

* modified comments
2023-07-05 19:26:03 -07:00
Bryan Stitt
f0cdb3b234 flatten routes 2023-07-03 15:59:06 -07:00
Bryan Stitt
581af0d596
stripe integration first mvp (replaces #140) (#154)
* stripe integration first mvp

* addressing some points from #140

* stripe endpoint needs testing

* addressed some comments

* added Json(payload)

* update toolchain

* lint (well, most of it)

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
2023-06-29 14:32:05 -07:00
Bryan Stitt
f6c8172a13 always do connectinfo for now 2023-06-29 13:12:35 -07:00
Bryan Stitt
efe1e30791 better handling of 0 port and fix test port 2023-06-24 20:38:06 -07:00
Bryan Stitt
38b266156a consistent route name 2023-06-24 15:49:26 -07:00
Bryan Stitt
1d6e1ef9af polish error logs 2023-06-24 15:24:46 -07:00
Bryan Stitt
39a505c0ff POST instead of GET and use more txs 2023-06-24 09:59:08 -07:00
Bryan Stitt
fd661689a3
Tracing (#144)
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
2023-06-23 16:28:45 -07:00
Bryan Stitt
662bbb296f add header debug endpoint 2023-06-21 11:28:33 -07:00
Bryan Stitt
3c2f95de10
Improve max wait and timeouts (#137)
* pass max wait with more functions

* move rpc_head_str higher up to use it in all the logs

* refresh consensus at double the block time

* new pricing that also includes archive check

* lint

* don't clone method

* put caches into one list

they serialize with their name, so it will still be easy to tell them apart

* more consistent waiting. still waits more than I want right now

* clean up default max waits

* more timeouts
2023-06-20 14:22:14 -07:00
Bryan Stitt
ba67151706 lint 2023-06-16 23:48:16 -07:00
Bryan Stitt
745870d119 drop a bunch of log levels 2023-06-16 23:46:20 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129)
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
2023-06-16 00:46:27 -07:00
yenicelik
0b26ae3bad made modify-subuser post 2023-06-11 18:59:27 +02:00
Bryan Stitt
41950c886c improvements for web3-this-then-that 2023-06-09 16:35:14 -07:00
Bryan Stitt
c37b320286 add cache names and actually serialize 2023-06-07 17:55:34 -07:00
Bryan Stitt
40bee30730 back to moka 2023-06-07 17:30:05 -07:00
David
fd69e6acdd
added stats page for referrals (shared, and received) (#95)
* added stats page for referrals (shared, and received)

* removed referrals address, and return ok response if no referrals used

* merged from devel

* changed unwrap for context

* changes from PR
2023-06-07 09:38:19 -07:00
Bryan Stitt
fd8bba4bdd most of these are for the whole app, not just the frontend 2023-05-30 21:26:11 -07:00
Bryan Stitt
c1eba556a5 fix missing ttl send and add name to cache 2023-05-29 17:19:05 -07:00
Bryan Stitt
bb50efb7f9
quick cache ttl max item size, better function names, and tests (#97)
* Revert "remove cache on /status and /health for now"

This reverts commit 166b0d810cd25b895560d50d00383a57e1269ad4.

* Revert "remove cache on backups needed, too"

This reverts commit 4597967def2f1daa71a277ec6731db4739ead640.

* add tests

* add max_item_weight

* comment

* add some helper functions, max weights, and tests
2023-05-29 15:48:22 -07:00
Bryan Stitt
d83e2f7015 Merge remote-tracking branch 'origin/devel' into devel 2023-05-23 14:44:40 -07:00
Bryan Stitt
78f247fc6c split http and ws provider 2023-05-23 14:40:34 -07:00
David
ddfd1dbff1
Added admin balance increase, and also test (#72)
* added admin balance increase, and also test

* slightly hacky, but does do an update expression

* added note column to entities

* removed redundant printlns

* flattened items

* removed user tier logic from admin increase balance completely. admin will have to manually increase the user tier
2023-05-22 17:07:17 -07:00
Bryan Stitt
b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt
efaaf1bb5f send the correct content type 2023-05-16 15:27:18 -07:00
Bryan Stitt
525f6d4a34 better variable names 2023-05-16 14:58:00 -07:00
Bryan Stitt
f4cebde53f use ttl quick cache 2023-05-16 14:46:05 -07:00
Bryan Stitt
2080739865
Quick cache (#70)
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343fc51338a8a4282dcc229b50097835b.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
2023-05-13 11:13:02 -07:00
Bryan Stitt
8a097dabbe
Bryan devel 2023-05-12 (#67)
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
2023-05-12 15:15:32 -07:00
David
34ed450fab
User Balance + Referral Logic (#44)
* will implement balance topup endpoint

* will quickly fix other PR reviews

* merging from master

* will finish up godmoe

* will finish up login

* added logic to top up balance (first iteration)

* should implement additional columns soon (currency, amount, tx-hash), as well as a new table for spend

* updated migrations, will account for spend next

* get back to this later

* will merge PR from stats-v2

* stats v2

rebased all my commits and squashed them down to one

* cargo upgrade

* added migrtation for spend in accounting table. will run test-deposit next

* trying to get request from polygon

* first iteration /user/balance/:tx_hash works, needs to add accepted tokens next

* creating the referral code seems to work

* will now check if spending enough credits will lead to both parties receiving credits

* rpcstats takes care of accounting for spend data

* removed track spend from table

* Revert "removed track spend from table"

This reverts commit a50802d6ae75f786864c5ec42d0ceb2cb27124ed.

* Revert "rpcstats takes care of accounting for spend data"

This reverts commit 1cec728bf241e4cfd24351134637ed81c1a5a10b.

* removed rpc request table entity

* updated referral code to use ulid s

* credits used are aggregated

* added a bunch of fields to referrer

* added database logic whenever an aggregate stats is added. will have to iterate over this a couple times i think. go to (1) detecting accepted stables next, (2) fix influxdb bug and (3) start to write test

* removed track spend as this will occur in the database

* will first work on "balance", then referral. these should really be treated as two separate PRs (although already convoluted)

* balance logic initial commit

* breaking WIP, changing the RPC call logic functions

* will start testing next

* got rid of warnings & lint

* will proceed with subtracting / adding to balance

* added decimal points, balance tracking seems to work

* will beautify code a bit

* removed deprecated dependency, and added topic + deposit contract to app.yaml

* brownie test suite does not rely on local contract files it pulls all from polygonscan

* will continue with referral

* should perhaps (in a future revision) recordhow much the referees got for free. marking referrals seems to work rn

* user is upgraded to premium if they deposit more than 10$. we dont accept more than $10M in a single tx

* will start PR, referral seems to be fine so far, perhaps up to some numbers that still may need tweaking

* will start PR

* removed rogue comments, cleaned up payments a bit

* changes before PR

* apply stats

* added unique constraint

* some refactoring such that the user file is not too bloated

* compiling

* progress with subusers, creating a table entry seems to work

* good response type is there as well now, will work on getters from primary user and secondary user next

* subuser logic also seems fine now

* downgrade logic

* fixed bug influxdb does not support different types in same query (which makes sense)

* WIP temporary commit

* merging with PR

* Delete daemon.rs

there are multiple daemons now, so this was moved to `proxyd`

* will remove request clone to &mut

* multiple request handles for payment

* making requests still seem fine

* removed redundant commented out bits

* added deposit endpoint, added deposit amount and deposit user, untested yet

* small bug with downgrade tier id

* will add authorization so balance can be received for users

* balance history should be set now too

* will check balance over time again

* subususer can see rpc key balance if admin or owner

* stats also seems to work fine now with historical balance

* things seem to be building and working

* removed clone from  OpenRequestHandle

* removed influxdb from workspace members

* changed config files

* reran sea-orm generate entities, added a foreign key, should be proper now

* removed contract from commit

* made deposit contract optional

* added topic in polygon dev

* changed deposit contract to deposit factory contract

* added selfrelation on user_tier

* added payment required

* changed chain id to u64

* add wss in polygon llamarpc

* removed origin and method from the table

* added onchain transactions naming (and forgot to add a migration before)

* changed foreign key to be the referrer (id), not the code itself

* forgot to add id as the target foreign key

* WIP adding cache to update role

* fixed merge conflicts

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
Co-authored-by: Bryan Stitt <bryan@stitthappens.com>
2023-05-12 09:45:15 -07:00
Bryan Stitt
4889c3e1ce Merge branch 'main' into devel 2023-04-18 20:38:43 -07:00
Bryan Stitt
d035049c8f add /backups_needed endpoint for easy alerts 2023-04-13 17:15:01 -07:00
Bryan Stitt
29c015508a fix warnings in cargo doc 2023-04-10 15:06:40 -07:00
yenicelik
0069e76040 latest changes from bryan 2023-04-04 14:40:22 +02:00
yenicelik
345fa6c513 git status 2023-03-30 13:42:56 +01:00
Bryan Stitt
8eff48611c more direct consensus finding code
this hopefully has less bugs. speed isn't super important since this isn't on the host path.
2023-03-22 13:10:15 -07:00
Bryan Stitt
b0c5d2b0eb remove debugging headers 2023-03-10 09:45:55 -08:00
Bryan Stitt
babd215e69 add an endpoint for debugging headers 2023-03-09 22:51:23 -08:00
Bryan Stitt
eb4d05a520 stats v2
rebased all my commits and squashed them down to one
2023-03-04 20:48:49 +01:00
Bryan Stitt
3098791ad9 add optional kafka feature 2023-03-03 01:39:50 +00:00
yenicelik
fb5c9c8770 typo login should have been logout 2023-02-27 10:56:06 +01:00