Commit Graph

305 Commits

Author SHA1 Message Date
Bryan Stitt
5da334fcb7 start adding tests that need docker for mysql management 2023-06-29 22:38:35 -07:00
Bryan Stitt
31c611f4ff add todo about siwe 2023-06-29 14:42:01 -07:00
Bryan Stitt
581af0d596
stripe integration first mvp (replaces #140) (#154)
* stripe integration first mvp

* addressing some points from #140

* stripe endpoint needs testing

* addressed some comments

* added Json(payload)

* update toolchain

* lint (well, most of it)

---------

Co-authored-by: yenicelik <david.yenicelik@gmail.com>
2023-06-29 14:32:05 -07:00
Bryan Stitt
f6c8172a13 always do connectinfo for now 2023-06-29 13:12:35 -07:00
Bryan Stitt
cdea61cb6b one list for ranked rpcs 2023-06-27 15:55:29 -07:00
Bryan Stitt
272f7ce800 add cargo hakari and chef to build process 2023-06-27 11:59:49 -07:00
Bryan Stitt
86f9560fc3 less serde features
if we don't use them its just wasted compile time
2023-06-27 10:18:32 -07:00
Bryan Stitt
1cf18b78a9 cargo upgrade 2023-06-27 10:04:48 -07:00
Bryan Stitt
54107a390b cargo upgrade 2023-06-26 21:07:25 -07:00
Bryan Stitt
fb76fd2d1c remove unused ewma crate now that we have our own 2023-06-25 14:32:45 -07:00
Bryan Stitt
a440d51377 test fixture 2023-06-24 21:17:08 -07:00
Bryan Stitt
94d0a25aaf test logs 2023-06-24 16:43:11 -07:00
Bryan Stitt
1d6e1ef9af polish error logs 2023-06-24 15:24:46 -07:00
Bryan Stitt
8f76d9320d support EIP1271 login 2023-06-24 09:48:31 -07:00
Bryan Stitt
4e6a80caa2 cargo upgrade 2023-06-24 00:41:48 -07:00
Bryan Stitt
89330cfba0 upstream ethers again 2023-06-23 22:23:51 -07:00
Bryan Stitt
fd661689a3
Tracing (#144)
* tracing

* add tracing and features to match

* use just one hostname crate

* cargo upgrade

* set up pretty logger and sentry layer
2023-06-23 16:28:45 -07:00
Bryan Stitt
66212b5cf5 cargo upgrade --incompatible -p itertools 2023-06-22 09:26:57 -07:00
Bryan Stitt
095a505017 cargo upgrade 2023-06-21 17:57:53 -07:00
Bryan Stitt
78efad5453 cargo upgrade strum 2023-06-19 17:48:46 -07:00
Bryan Stitt
9dd0bae40a text instead of string payload 2023-06-18 16:13:55 -07:00
Bryan Stitt
e54299beca rolling median for choosing tier 2023-06-18 09:53:36 -07:00
Bryan Stitt
74d2333039 try this version of influxdb2 2023-06-16 23:41:12 -07:00
Bryan Stitt
5d9365449f simplify max item weight by having the weigher send MAX for big things 2023-06-16 16:14:11 -07:00
Bryan Stitt
1ec0f14144 upgrade to influxdb 1e5577e14150797584f5ed0ea7aba0bd68f0a678 2023-06-16 14:48:03 -07:00
Bryan Stitt
cb71de7d14 pin influxdb2 to 6d3cf64d61967f41fafb57b3f4553206a8f3f2a3 2023-06-16 13:30:35 -07:00
Bryan Stitt
d9a865fd1b cargo upgrade 2023-06-16 10:58:09 -07:00
Bryan Stitt
fe8675ddf2 cargo upgrade 2023-06-16 00:48:22 -07:00
Bryan Stitt
a083bc652d
Handle uncle transactions (#129)
* add more columns to handle uncled transactions

* handle payment uncles

* put relations back

* include all the new columns

* lower log levels

* improve block caching

if we have a block with a number, its canonical. uncles don't get returned

* improve disconnect logic

* lint

* clear first changed for new_top_config_receiver

* better logs around config changing

* i guess we do want one apply top_config at the start

* check correct variable for data limits
2023-06-16 00:46:27 -07:00
Bryan Stitt
c771479d94 use WyRand instead of Xoshiro 2023-06-12 19:46:36 -07:00
Bryan Stitt
c456432cc3 cargo upgrade 2023-06-12 11:44:51 -07:00
David
81aa4788b4
David/60 add basic email verification (#119)
* added logic to check e-mails thru SMPT server, very basic, no validation e-mails etc.

* added rpc key logic to use db numbers instead of UUIDs

* Revert "added rpc key logic to use db numbers instead of UUIDs"

This reverts commit 056771d02725f90ed5f1745b0372e13e630a19ac.
2023-06-11 19:10:10 -07:00
Bryan Stitt
4a9c1a0ce6 move user registration into a dedicated function 2023-06-09 18:31:47 -07:00
Bryan Stitt
1f3040f6c7 automated tiers 2023-06-09 12:21:55 -07:00
Bryan Stitt
b6ff14210d cargo upgrade 2023-06-08 22:35:14 -07:00
Bryan Stitt
bfaff81626 cargo upgrade 2023-06-07 17:31:29 -07:00
Bryan Stitt
40bee30730 back to moka 2023-06-07 17:30:05 -07:00
Bryan Stitt
f6b2b0deab cargo upgrade 2023-06-07 10:50:34 -07:00
Bryan Stitt
7dcc2b4d93 use MultiAbigen 2023-05-31 11:52:57 -07:00
Bryan Stitt
7947cb95ff use bloom filters and support transactions with multiple deposit events 2023-05-31 11:20:17 -07:00
Bryan Stitt
fcc1843af0 put it back. duplication between cargo.toml isnt worth it 2023-05-30 21:37:58 -07:00
Bryan Stitt
2273637a07 move cli into a separate packge and clean up dependencies 2023-05-30 20:58:24 -07:00
Bryan Stitt
cebe8ed1ea
Request refactor (#99)
* wip

* AsRef finally works like i wanted

* actually return the block

* start adding async trait

* remove stale import

* include id in the error response when possible

* remove stale comments
2023-05-30 16:32:34 -07:00
Bryan Stitt
9e935e2c44 cargo upgrade 2023-05-29 09:43:51 -07:00
Bryan Stitt
0d15d5baf7 bump versions to match number of migrations 2023-05-23 20:58:29 -07:00
Bryan Stitt
4c7a7b87cb cargo upgrade 2023-05-23 14:58:13 -07:00
Bryan Stitt
78f247fc6c split http and ws provider 2023-05-23 14:40:34 -07:00
Bryan Stitt
05f862f3e4 cargo upgrade 2023-05-22 15:50:00 -07:00
Bryan Stitt
3ac15558e3 fix decay time 2023-05-22 15:43:39 -07:00
Bryan Stitt
c443f76623 refactor tiers again 2023-05-19 14:09:03 -07:00
Bryan Stitt
8898ffe578 cargo upgrade 2023-05-18 13:36:35 -07:00
Bryan Stitt
b61675e928 replace all moka with quick_cache_ttl 2023-05-18 13:34:22 -07:00
Bryan Stitt
f4cebde53f use ttl quick cache 2023-05-16 14:46:05 -07:00
Bryan Stitt
685c1d039a cargo upgrade 2023-05-13 15:00:24 -07:00
Bryan Stitt
2080739865
Quick cache (#70)
* quick cache and allocate less

* improve /status cache

* prepare to cache raw transaction hashes so we dont dos our backends

* simple benchmark for /health and /status

* mut not needed with atomics

* DRY all the status pages

* use u64 instead of bytes for subscriptions

* fix setting earliest_retry_at and improve logs

* Revert "use kanal instead of flume or tokio channels (#68)"

This reverts commit 510612d343fc51338a8a4282dcc229b50097835b.

* fix automatic retries

* put relaxed back

* convert error message time to seconds

* assert instead of debug_assert while we debug

* ns instead of seconds

* disable peak_latency for now

* null is the default

* cargo fmt

* comments

* remove request caching for now

* log on exit

* unit weigher for now

* make cache smaller. we need a weigher for prod. just debugging

* oops. we need async

* add todo

* no need for to_string on a RawValue
2023-05-13 11:13:02 -07:00
Bryan Stitt
510612d343
use kanal instead of flume or tokio channels (#68) 2023-05-12 15:47:01 -07:00
Bryan Stitt
8a097dabbe
Bryan devel 2023-05-12 (#67)
* add minor todo

* BadRequest instead of web3_context

* more bad request error codes

* use tokio-uring for the tcp listener

* clear block instead of panic

* clone earlier

* more watch channels instead of rwlocks

* drop uring for now (its single threaded) and combine get/post/put routes

* clean up iter vs into_iter and unnecessary collect

* arcswap instead of rwlock for Web3Rpcs.by_name

* cargo upgrade

* uuid fast-rng and alphabetize

* if protected rpcs, only use protected rpcs

* listenfd

* make connectinfo optional

* try_get_with_by_ref instead of try_get_with

* anyhow ensure. and try_get_with_as_ref isn't actually needed

* fix feature flags

* more refs and less clone

* automatic retry for eth_getTransactionReceipt and eth_getTransactionByHash

thanks for the report Lefteris @ Rotki

* ArcSwap for provider

* set archive_request to true on transaction retrying

* merge durable stats

* Revert "ArcSwap for provider"

This reverts commit 166d77f204cde9fa7722c0cefecbb27008749d47.

* comments

* less clones

* more refs

* fix test

* add optional mimalloc feature

* remove stale dependency

* sort

* cargo upgrade

* lint constants

* add todo

* another todo

* lint

* anyhow::ensure instead of panic

* allow rpc_accounting_v2 entries for requests without an rpc key
2023-05-12 15:15:32 -07:00
David
34ed450fab
User Balance + Referral Logic (#44)
* will implement balance topup endpoint

* will quickly fix other PR reviews

* merging from master

* will finish up godmoe

* will finish up login

* added logic to top up balance (first iteration)

* should implement additional columns soon (currency, amount, tx-hash), as well as a new table for spend

* updated migrations, will account for spend next

* get back to this later

* will merge PR from stats-v2

* stats v2

rebased all my commits and squashed them down to one

* cargo upgrade

* added migrtation for spend in accounting table. will run test-deposit next

* trying to get request from polygon

* first iteration /user/balance/:tx_hash works, needs to add accepted tokens next

* creating the referral code seems to work

* will now check if spending enough credits will lead to both parties receiving credits

* rpcstats takes care of accounting for spend data

* removed track spend from table

* Revert "removed track spend from table"

This reverts commit a50802d6ae75f786864c5ec42d0ceb2cb27124ed.

* Revert "rpcstats takes care of accounting for spend data"

This reverts commit 1cec728bf241e4cfd24351134637ed81c1a5a10b.

* removed rpc request table entity

* updated referral code to use ulid s

* credits used are aggregated

* added a bunch of fields to referrer

* added database logic whenever an aggregate stats is added. will have to iterate over this a couple times i think. go to (1) detecting accepted stables next, (2) fix influxdb bug and (3) start to write test

* removed track spend as this will occur in the database

* will first work on "balance", then referral. these should really be treated as two separate PRs (although already convoluted)

* balance logic initial commit

* breaking WIP, changing the RPC call logic functions

* will start testing next

* got rid of warnings & lint

* will proceed with subtracting / adding to balance

* added decimal points, balance tracking seems to work

* will beautify code a bit

* removed deprecated dependency, and added topic + deposit contract to app.yaml

* brownie test suite does not rely on local contract files it pulls all from polygonscan

* will continue with referral

* should perhaps (in a future revision) recordhow much the referees got for free. marking referrals seems to work rn

* user is upgraded to premium if they deposit more than 10$. we dont accept more than $10M in a single tx

* will start PR, referral seems to be fine so far, perhaps up to some numbers that still may need tweaking

* will start PR

* removed rogue comments, cleaned up payments a bit

* changes before PR

* apply stats

* added unique constraint

* some refactoring such that the user file is not too bloated

* compiling

* progress with subusers, creating a table entry seems to work

* good response type is there as well now, will work on getters from primary user and secondary user next

* subuser logic also seems fine now

* downgrade logic

* fixed bug influxdb does not support different types in same query (which makes sense)

* WIP temporary commit

* merging with PR

* Delete daemon.rs

there are multiple daemons now, so this was moved to `proxyd`

* will remove request clone to &mut

* multiple request handles for payment

* making requests still seem fine

* removed redundant commented out bits

* added deposit endpoint, added deposit amount and deposit user, untested yet

* small bug with downgrade tier id

* will add authorization so balance can be received for users

* balance history should be set now too

* will check balance over time again

* subususer can see rpc key balance if admin or owner

* stats also seems to work fine now with historical balance

* things seem to be building and working

* removed clone from  OpenRequestHandle

* removed influxdb from workspace members

* changed config files

* reran sea-orm generate entities, added a foreign key, should be proper now

* removed contract from commit

* made deposit contract optional

* added topic in polygon dev

* changed deposit contract to deposit factory contract

* added selfrelation on user_tier

* added payment required

* changed chain id to u64

* add wss in polygon llamarpc

* removed origin and method from the table

* added onchain transactions naming (and forgot to add a migration before)

* changed foreign key to be the referrer (id), not the code itself

* forgot to add id as the target foreign key

* WIP adding cache to update role

* fixed merge conflicts

---------

Co-authored-by: Bryan Stitt <bryan@llamanodes.com>
Co-authored-by: Bryan Stitt <bryan@stitthappens.com>
2023-05-12 09:45:15 -07:00
Bryan Stitt
36cc884112 Merge remote-tracking branch 'origin/main' into devel 2023-05-11 20:05:47 -07:00
Bryan Stitt
2a242fe37f cargo upgrade --incompatible 2023-05-11 13:25:55 -07:00
Rory Trent
ec11e210ee
Peak ewma (#63)
* use peak-ewma instead of head for latency calculation

* Implement some suggested changes from PR

* move latency to new package in workspace root

* fix unit tests which now require peak_latency on Web3Rpc

* Switch to atomics for peak-ewma

This change is to avoid locking from tokio::sync::watch.

* add decay calculation to latency reads in peak-ewma

* Add some tests for peak-ewma

* Sensible latency defaults and not blocking on full

* Cleanup and a couple additional comments
2023-05-11 13:09:15 -07:00
Bryan Stitt
4d64bf9580 cargo upgrade 2023-05-08 10:44:01 -06:00
Bryan Stitt
cc3c9462a3 cargo upgrade 2023-04-25 05:39:09 -07:00
Bryan Stitt
56fdf48129
Send transactions to protected and public rpcs (#57)
* move protected transactions into their own function and dry stats sending

* cargo upgrade

* comments

* time to live instead of time to idle

* minor workaround for eth_chainId

* cargo upgrade
2023-04-24 11:00:12 -07:00
Bryan Stitt
4d2c0dae53 no sasl 2023-04-20 18:08:53 -07:00
Bryan Stitt
684b14c8a7 enable sasl for kafka 2023-04-20 14:48:15 -07:00
Bryan Stitt
30eb6e91bf cargo upgrade 2023-04-19 21:09:17 -07:00
Bryan Stitt
03ef27baa4 Merge branch 'main' into devel 2023-04-19 15:31:08 -07:00
Bryan Stitt
10d2d46512 more packages for kafka+ssl 2023-04-19 15:22:27 -07:00
Bryan Stitt
7250a0c155 upgrade ethers to 2.0.2 2023-04-10 15:01:36 -07:00
Bryan Stitt
55924ff840 cargo upgrade the easy things 2023-04-10 14:27:06 -07:00
Bryan Stitt
072119901e remove stale comment 2023-04-05 15:26:39 -07:00
Bryan Stitt
2e934a41bf cargo upgrade 2023-04-05 12:30:52 -07:00
yenicelik
345fa6c513 git status 2023-03-30 13:42:56 +01:00
Bryan Stitt
2205a472fd add optional tokio-console and improve reconnect logic 2023-03-22 18:43:13 -07:00
Bryan Stitt
c0afc295ca add hostname to status 2023-03-22 15:18:54 -07:00
Bryan Stitt
684c37ef6c cargo upgrade 2023-03-10 09:45:47 -08:00
yenicelik
5a54fc5e99 lets modify web3 proxyd manually copy pasta 2023-03-04 21:19:39 +01:00
Bryan Stitt
5b48d471bb fix influxdb derive macro
https://github.com/aprimadi/influxdb2/issues/17
2023-03-04 20:54:24 +01:00
Bryan Stitt
dbd7860416 wip influxdb2 2023-03-04 20:54:04 +01:00
Bryan Stitt
3f217b930a cargo upgrade 2023-03-04 20:49:21 +01:00
Bryan Stitt
eb4d05a520 stats v2
rebased all my commits and squashed them down to one
2023-03-04 20:48:49 +01:00
Bryan Stitt
cbdd6d7c1e oops. didn't mean to commit this script yet 2023-03-02 21:12:35 -08:00
Bryan Stitt
3098791ad9 add optional kafka feature 2023-03-03 01:39:50 +00:00
Bryan Stitt
1da12f5844 bump version to match number of migrations 2023-02-28 23:35:20 -08:00
Bryan Stitt
11ee0aafe9 inotify is a pain cross platform. just check file hashes 2023-02-27 13:13:18 -08:00
Bryan Stitt
6067369ee3 add inotify and rpc disconnect 2023-02-27 12:50:57 -08:00
Bryan Stitt
f8f5e7a1c8 add thread for config file watching and run clippy lint 2023-02-26 23:08:00 -08:00
Bryan Stitt
c47c9136a1 cargo upgrade 2023-02-25 00:47:16 -08:00
Bryan Stitt
8506cd6bdc cargo upgrade 2023-02-15 11:40:55 -08:00
Bryan Stitt
824a6fa1f0 well it compiles. doesnt work yet. but it compiles 2023-02-14 12:14:50 -08:00
Bryan Stitt
0e2e45cf90 cargo upgrade 2023-02-12 01:47:06 -08:00
Bryan Stitt
c008f50943 use http and ws providers in a single config 2023-02-12 01:40:01 -08:00
Bryan Stitt
e062975b13 cargo upgrade 2023-02-09 11:56:58 -08:00
Bryan Stitt
ff4cb9abce cargo upgrade 2023-02-08 11:11:16 -08:00
Bryan Stitt
0f280ce483 cargo upgrade and shorten variable names
also begin adding a latency tracker for rpc stats
2023-02-06 09:55:27 -08:00
Bryan Stitt
f2d35ba5eb remove metered in favor of influxdb stats 2023-02-05 18:17:26 -08:00
Bryan Stitt
a3f2eff512 simple request counts 2023-02-02 13:45:05 -08:00
Bryan Stitt
6c23b73f2d cargo upgrade 2023-01-30 15:57:06 -08:00
Bryan Stitt
f80390c88a upgrade sentry and fix pagerduty features so we do not need openssl 2023-01-25 14:11:20 -08:00
Bryan Stitt
07bac8618e cargo upgrade --workspace 2023-01-24 22:46:58 -08:00
Bryan Stitt
c9b8e5dfb9 wrote our own panic handler 2023-01-24 05:27:07 -08:00
Bryan Stitt
17c446b68c why did cargo upgrade miss all these? 2023-01-24 03:25:12 -08:00
Bryan Stitt
776bcd149f send panics to pagerduty 2023-01-24 00:05:31 -08:00
Bryan Stitt
c1e81089b1 sort dependencies 2023-01-23 21:50:59 -08:00
Bryan Stitt
86e3f2991f use watch instead of arcswap 2023-01-22 22:02:08 -08:00
Bryan Stitt
e89126cff8 cargo upgrade --workspace 2023-01-20 15:51:19 -08:00
Bryan Stitt
501cb4a1b9 cargo upgrade --workspace for some perf improvements 2023-01-19 21:30:24 -08:00
Bryan Stitt
17d8ea0b7f make free tier even better and improve migration locking 2023-01-19 21:28:33 -08:00
Bryan Stitt
274778cd12 cargo upgrade --workspace 2023-01-19 02:21:45 -08:00
Bryan Stitt
053947de40 one bin for everything 2023-01-18 13:56:23 -08:00
Bryan Stitt
7d0ac54019 upgrade hashbrown 2023-01-15 12:02:33 -08:00
Bryan Stitt
364097dcec cargo upgrade 2023-01-11 15:19:21 -08:00
Bryan Stitt
99d095bc33 upgrade after revert 2023-01-06 15:59:36 -08:00
Bryan Stitt
fb62243c3e Revert "cargo upgrade and use trailing slash route extra"
This reverts commit 5bfba3a87afbf4157c2b0bd36273e3dcc4075578.
2023-01-06 15:58:52 -08:00
Bryan Stitt
5bfba3a87a cargo upgrade and use trailing slash route extra 2023-01-06 14:22:57 -08:00
Bryan Stitt
221f2c1651 upgrade tokio
its a change for windows so doesn't really matter
2023-01-04 14:34:58 -08:00
Bryan Stitt
f701295439 upgrade arc-swap 2023-01-02 10:43:49 -08:00
Bryan Stitt
853ee354c9 cargo upgrade 2022-12-27 22:10:49 -08:00
Bryan Stitt
9885bdfd8c minor update 2022-12-22 12:04:58 -08:00
Bryan Stitt
263f98ced5 minor updates 2022-12-19 10:58:49 -08:00
Bryan Stitt
a3a3c78039 boring upgrades 2022-12-16 20:06:53 -08:00
Bryan Stitt
081873e937 move pending logins to the database 2022-12-13 18:34:08 -08:00
Bryan Stitt
4c2d3634c5 default to less detailed stats 2022-12-11 20:39:54 -08:00
Bryan Stitt
ed505165ca upgrade things except axum 2022-12-05 14:18:47 -08:00
Bryan Stitt
eb4b487aae better block saving 2022-12-02 21:31:03 -08:00
Bryan Stitt
4dfb7a5451 fix archive node detection when nodes used to be archive but than changed flags 2022-11-25 00:48:17 +00:00
Bryan Stitt
0cffa73d41 cargo upgrade serde_json 2022-11-22 22:46:47 +00:00
Bryan Stitt
ff3c5de0aa start adding user_export and user_import scripts 2022-11-21 22:52:47 +00:00
Bryan Stitt
4018121e51 cargo upgrade 2022-11-21 17:49:57 +00:00
Bryan Stitt
4d5ed64ac3 cargo upgrade 2022-11-16 20:17:50 +00:00
Bryan Stitt
50a572bc00 upgrade uuid 2022-11-16 07:27:16 +00:00
Bryan Stitt
ea5f791560 simple lock around database migrations 2022-11-14 18:24:52 +00:00
Bryan Stitt
0e1cf5767c cut out tracing for now 2022-11-12 08:24:32 +00:00
Bryan Stitt
9ae2337d1d thread fast rng 2022-11-12 06:11:58 +00:00
Bryan Stitt
1329084621 cargo upgrade 2022-11-11 18:40:52 +00:00
Bryan Stitt
c35dd96cfb save rpc_key_id or origin. needs some testing 2022-11-11 06:19:14 +00:00
Bryan Stitt
b800b862ee bump version 2022-11-07 23:06:14 +00:00
Bryan Stitt
cb69a2d132 cargo upgrade (requires rust 1.65+) 2022-11-07 22:23:55 +00:00
Bryan Stitt
bde240c18a more auth when only rpc_key_id is set 2022-11-04 22:58:15 +00:00
Bryan Stitt
e8f2a13f5d better stats aggregations 2022-11-02 23:14:16 +00:00
Bryan Stitt
f2268dbb1b this works, but its not super fast 2022-11-01 20:51:33 +00:00
Bryan Stitt
f859ed56fc period instead of minute in variable names 2022-11-01 19:24:29 +00:00
Bryan Stitt
234289965d cargo upgrade 2022-10-31 19:34:29 +00:00
Bryan Stitt
366f2c8f84 rename api_key to rpc_key 2022-10-27 00:12:42 +00:00
Bryan Stitt
0c3194f445 upgrades and todo cleanup 2022-10-26 22:38:04 +00:00
Bryan Stitt
8f3e5c0146 user post endpoint 2022-10-26 21:39:26 +00:00
Bryan Stitt
8230172366 more user endpoints 2022-10-25 21:11:19 +00:00
Bryan Stitt
4ccbcb8af4 update sea-orm and entities 2022-10-25 03:41:59 +00:00
Bryan Stitt
a0056ca881 optional tracing-sentry 2022-10-24 21:07:29 +00:00