Commit Graph

100 Commits

Author SHA1 Message Date
github-actions[bot]
efe1879957 Version Packages 2023-02-23 01:53:04 +00:00
dependabot[bot]
2fc790aeda chore(deps): bump github.com/prometheus/client_golang in /proxyd
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.11.0 to 1.11.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.11.0...v1.11.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-15 20:01:27 +00:00
Matthew Slipper
2427bfeadf Merge pull request #4897 from ethereum-optimism/feat/remove-logs
proxyd: Remove logging
2023-02-15 10:11:05 -07:00
Matthew Slipper
336c3a7c54 proxyd: Remove logging 2023-02-15 10:07:44 -07:00
github-actions[bot]
4037ef8b28 Version Packages 2023-02-15 16:36:28 +00:00
Matthew Slipper
c17bcc9b83 proxyd: Add global flag to method overrides
This allows us to set global limits on individual RPCs that ignore any origin/user agent exemption.
2023-02-15 00:44:19 -07:00
Matthew Slipper
454bc10e44 Merge pull request #4797 from ethereum-optimism/bugfix/include-nonce-in-rate-limit
proxyd: Add nonce to sender-based rate limit
2023-01-27 12:42:30 -07:00
Matthew Slipper
d7c80a5a8c proxyd: Add nonce to sender-based rate limit 2023-01-27 12:36:20 -07:00
inphi
2a87efb68a proxyd: Add missing req_id to log entry 2023-01-27 13:23:20 -05:00
github-actions[bot]
868d622e4f Version Packages 2023-01-23 19:35:31 +00:00
Matthew Slipper
aa3a5c859a proxyd: Add sender-based rate limiter
This PR adds a sender-based rate limiter to proxyd. This rate limiter throttles eth_sendRawTransaction requests based on sender in order to prevent abusive behavior on the sequencer.

Integration tests have been added to test this functionality.
2023-01-23 10:59:40 -07:00
github-actions[bot]
f1b582c385 Version Packages 2022-10-17 23:21:20 +00:00
mergify[bot]
019f800e90 Merge branch 'develop' into feat/proxyd-allow-backend-lim-disable 2022-10-17 15:53:44 +00:00
Matthew Slipper
fe8a89e61d proxyd: Support pattern matching in origin and user agent
Adds support for pattern matchin in exempt origins. This should help with some of the issues Synthetix and others are seeing.
2022-10-17 07:08:33 -06:00
Matthew Slipper
fd5e974bd5 fix tests 2022-10-15 09:59:11 -05:00
Zach Howard
24a3668bc8 adds log level conf to proxyd (#3704) 2022-10-14 13:56:27 -05:00
Matthew Slipper
8a624bb2ca proxyd: Allow disabling backend rate limiting
The backend rate limiter is in place to protect upstreams like the sequencer. However, in many cases it isn't needed and it causes unnecessary requests to Redis. This PR allows this to be disabled, and disables this by default.
2022-10-13 15:02:35 -05:00
github-actions[bot]
053ce4ddfd Version Packages 2022-10-13 15:32:03 +00:00
Matthew Slipper
3f19d26ea8 Merge pull request #3681 from ethereum-optimism/10-09-proxyd_Integrate_custom_rate_limiter
proxyd: Integrate custom rate limiter
2022-10-12 15:34:14 -05:00
Matthew Slipper
7fd63c6c6c Merge pull request #3679 from ethereum-optimism/10-09-proxyd_Custom_rate_limiter_implementation
proxyd: Custom rate limiter implementation
2022-10-12 15:34:02 -05:00
Matthew Slipper
f737002baa proxyd: Integrate custom rate limiter
Integrates the custom rate limiter in the previous PR into the rest of the application. Also takes the opportunity to clean up how we instantiate Redis clients so that we can share them among multiple different services.

There are some config changes in this PR. Specifically, you must specify a `base_rate` and `base_interval` in the rate limit config.
2022-10-11 17:20:23 -05:00
Matthew Slipper
fa7425683a proxyd: Custom rate limiter implementation
Our current proxyd deployment does not share rate limit state across multiple servers within a backend group. This means that rate limits on the public endpoint are artifically high.

This PR adds a Redis-based rate limiter to fix this problem. While our current rate limiting library (github.com/sethvargo/go-limiter) _does_ support Redis, the client library it uses is not type safe, is less performant, and would require us to update the other places we use Redis. To avoid these issues, I created a simple rate limiting interface with both Redis and memory backend.

Note that this PR only adds the new implementations - it does not integrate them with the rest of the codebase. I'll do that in a separate PR to make review easier.
2022-10-11 17:20:23 -05:00
Chris Wessels
15a59bad99 fix(proxyd): Fix compliance with JSON-RPC 2.0 spec by adding optional RPCError.Data (#3683)
* fix: add optional data field to RPCError struct

* fix: formatting lint

* feat(proxyd): add changeset
2022-10-11 08:51:02 -06:00
github-actions[bot]
09552fddea Version Packages 2022-09-29 01:37:00 +00:00
Matthew Slipper
537717610e proxyd: Add batch size metric and configurable max (#3545)
* proxyd: Add batch size metric and configurable max

The max batch size will be overwritten if it is over `MaxBatchRPCCallsHardLimit`.

Builds on https://github.com/ethereum-optimism/optimism/pull/3544.

* changeset

* fix lint

* fix test
2022-09-23 15:06:02 -06:00
Matthew Slipper
485258d3c2 proxyd: Add customizable whitelist error message (#3544)
* proxyd: Add customizable whitelist error message

Alchemy is asking for this so that we can include their affiliate link.

* add missing test case

* fix error message

* goimports
2022-09-23 14:21:12 -06:00
github-actions[bot]
70557fdafd Version Packages 2022-09-20 15:21:51 +00:00
Matthew Slipper
d228a55bd7 proxyd: Fix error message in tests (#3503)
* proxyd: Fix error message in tests

The custom error message wasn't being carried over in batch responses.

* review fixes

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-09-19 16:44:08 +00:00
Matthew Slipper
3fb4debfe1 proxyd: Add logging for origin and remote IP (#3482)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
2022-09-16 17:28:56 +00:00
github-actions[bot]
320d6ebeea Version Packages 2022-09-15 08:51:21 +00:00
Matthew Slipper
ccf0934459 proxyd: Support per-RPC rate limits (#3471)
* proxyd: Support per-RPC rate limits

* add log

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-09-15 08:46:11 +00:00
Joshua Gutow
8e92c91e6d Last unwrapped errors
These where missed in local lints but when a full lint was run these
popped up.
2022-09-09 04:56:59 -07:00
protolambda
9a4626856a style(batch-submitter,bss-core,proxyd): Fix lint Go (#3328)
* style(batch-submitter): fix lint

* style(bss-core): fix lint

* chore(proxyd): use io and os instead of deprecated ioutil methods, fixes lint
2022-08-26 19:39:30 -06:00
github-actions[bot]
fc9487bf83 Version Packages 2022-08-04 18:48:19 +00:00
Matthew Slipper
7a2b4bdf45 proxyd: Add rate limit logging (#3173) 2022-08-04 18:44:46 +00:00
github-actions[bot]
7c599f758f Version Packages 2022-08-04 17:40:22 +00:00
Matthew Slipper
f3d3492a81 proxyd: Add frontend rate limiting (#3166)
* proxyd: Add frontend rate limiting

To give us more flexibiltiy with rate limiting, proxyd now supports rate limiting of client (frontend) requests in addition to upstream (backend) requests. This PR also gives us the ability to exempt certain user agents/origins from rate limiting.

* lint
2022-08-04 11:34:43 -06:00
Matthew Slipper
4ea6a054c3 proxyd: Unwrap single RPC batches (#3165)
* proxyd: Unwrap single RPC batches

* Update backend.go
2022-08-04 01:36:07 +00:00
Matthew Slipper
cd0afa3176 proxyd: Parameterize full RPC request logging (#3110)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-07-27 17:12:47 +00:00
Matthew Slipper
614366aa5b ci: Upgrade golangci-lint version to one that handles Go 1.18 (#2841)
* ci: Upgrade golangci-lint version to one that handles Go 1.18

* Fix proxyd lint
2022-06-23 04:45:26 +00:00
github-actions[bot]
5da617a5ba Version Packages 2022-06-18 02:48:41 +00:00
Matthew Slipper
e2e3a622ce proxyd: Use canned response for eth_accounts (#2801)
We never want to expose Geth's accounts to the public internet, so proxyd will now return `[]` for `eth_accounts` RPC calls without hitting the backend.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-16 19:02:39 +00:00
github-actions[bot]
c31d7501ea Version Packages 2022-06-08 20:51:10 +00:00
Matthew Slipper
e41cfc1d94 proxyd: Fix concurrent WS write panic (#2711)
Fixes a panic in the websocket proxyd logic. Normally, the `clientPump` and `backendPump` methods in `WSProxier` send data in one direction. However, when the client sends an invalid RPC, the `clientPump` will send a response _directly to the client_ in order to avoid unnecessary roundtrips to the backend. This could be interleaved with concurrent writes to the client's WS in `backendPump`, and would cause a panic in the WS library.

To test this, this PR includes a dedicated integration test that reliably triggers the issue. In addition, this PR adds additional testing for WS functionality.
2022-06-08 15:09:32 +00:00
Murphy Law
69f189c0ea proxyd: Handle unexpected JSON-RPC responses (#2628)
This fixes a bug where the infura backend would be labeled offline because it
returns an unexpected JSON-RPC response. Unexpected, but well-formed,
JSON-RPC response are handled specially. Such errors are surfaced up to
the backend proxier so failover still occurs.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-08 13:56:24 +00:00
github-actions[bot]
c43d2c708d Version Packages 2022-06-01 21:25:41 +00:00
dependabot[bot]
7d77581725 chore(deps): bump github.com/ethereum/go-ethereum in /proxyd (#2607)
Bumps [github.com/ethereum/go-ethereum](https://github.com/ethereum/go-ethereum) from 1.10.16 to 1.10.17.
- [Release notes](https://github.com/ethereum/go-ethereum/releases)
- [Commits](https://github.com/ethereum/go-ethereum/compare/v1.10.16...v1.10.17)

---
updated-dependencies:
- dependency-name: github.com/ethereum/go-ethereum
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-26 23:35:13 +00:00
Matthew Slipper
5019c2f69f Bedrock -> Develop (#2563)
* opnode: Reconcile epochs and handle reorgs (#309)

* opnode: Switch to uint64 in FindL2Heads function

* opnode: Properly initialize state loop

This properly sets the L1 head, L2 Head, and L2 Unsafe Head during
`state.Start()` in normal operation and when the current L1 head
has not reached the L1 genesis.

* opnode: Rename output functions

The old names where not good.
In addition, this standardizes the order of arguements and extends the
returns. The returns are not fully used, but will be.

* opnode: Add sanity check to L1 window in insertEpoch

This make sure that the L1 window starts at the correct block.

* opnode: Split out loop

Now each action occurs in a seperate function. The state loop is
still responsible for sequencing follow-up actions, but is not
responsible for setting state.

* opnode: Better name for batch prepare functions

This name indicates that it provides missing batches.

* opnode: Verify epochs as sequencer

Sequencers now verify epochs and will reorg if they see that verified
epochs do not match what they created.

* opnode: Mute driver test

* opnode: Set L2 Geth forkchoice on reorg

This is required to handle L1 reorgs (rather than just missing
batches).

* opnode: Bump op-geth version

Needed to fix a bug where the L2 geth node was missing the `mint` field
in deposit transactions when returning them over the JSON RPC server.

* opnode: Don't incorrectly advance safehead

There is some L2 block that is possible to fully derive from L1;
however, if it is ahead of the current state's safe head, do not
advance the current state's safehead because the chain has not
been verified up to that point.

* opnode: Don't rebuild L1 Genesis Block ID

Co-authored-by: Diederik Loerakker <proto@protolambda.com>

* opnode: More comments

Co-authored-by: Diederik Loerakker <proto@protolambda.com>

* opnode: Don't cast slice

Co-authored-by: Diederik Loerakker <proto@protolambda.com>

* opnode: PR comments

* opnode: Better error reason in attributesMatch

Include the details of the mismatched fields in addition to what
field it was. Will make diagnosing errors much easier.

Co-authored-by: Diederik Loerakker <proto@protolambda.com>

* go.mod: update to squashed l2 geth diff d5e1fc1a74bda3cbe5715d5732621460f9e00908

* opnode/rollup: use L2 block height in L1-info deposit tx block-height field (tx field, not info field) for uniqueness of deposit tx hash on L2

* feat: DepositFeed receive function

It is convenient for users to be able to deposit funds
directly to L2 by simply doing an ETH send to an address.
The prevents needing an ABI.

* Update slither db

* Add solidity test on df.receive() function

* Add warning message on DepositFeed.receive()

* specs: Add Withdrawals specification

* contracts: Add withdrawor and start of tests

* ops: Add withdrawor predeploy to docker setup

ops: Add contract code to Makefile and Dockerfile

* Add gaslimit to withdrawal encoding

* test(contracts): Withdrawal creation and proof gen

* deps: install entire goddamn monorepo using forge.

I needed the trie contracts over here, so now the whole monorepo
is a submodule of the specs repo. Happy now @norswap?

Why, you ask, did I do this rather than the alternatives available to
me?
1. Copying and pasting is gross.
2. `yarn add` would have installed a smaller dep, but it's also gross.
3. git submodules, while also gross in their own right, is at least
   consistent with how we're currently handling other dependencies, so
   I went with that.

* Add output root inclusion proof

Adds a check to ensure that withdrawal root is included in the
l2 output root.

* Add storage inclusion proof

Adds a check to ensure that the withdrawal message hash is included
  in the mapping of withdrawals in the withdrawer contract

* Improve testing against L2 node

* Add finalizationWindow var to WithdrawalVerifier

* Add WithdrawalVerified event

* Remove unused constructor argument

* Clean up Forge test of WithdrawalVerifier

* ops: Docker get l2 genesis hash on startup

Docker fixes

* contracts: ignore dirty submodules

* Don't run hardhat tests in CI

* Create Optimism Portal contract

* Move receive function to OptimismPortal

* specs: Add overview page (#286)

* Convert WithdrawalVerifier to a library

This changes the WithdrawalVerifier to be stateless, and moves all
checks into the OptimismPortal

* contracts: Rename Withdrawor to Withdrawer

* specs: define timestamp bounds of derivation process, clear up block derivation, define missing noTxPool field

* opnode/rollup: implement new timestamp bounds on block derivation batch filtering and filling

* opnode: fix sequencer L1 origin advancing, and fix e2e genesis time

* contracts: Add burn function to the Withdrawer contract

* contracts: Add natspec comments

* contracts: Remove timestamp from OutputRootProof

* contracts: undoL1ToL2Alias on withdrawals from a contract

Also uses AddressAliasHelper lib for aliasing instead of reimplementing
it.

* specs/rollup-node: improve spec based on review suggestions from @norswap

* opnode/rollup: implement review suggestions from @trianglesphere

* specs/rollup-node: document timestamp invariant between L2 and L1 origin

* opnode/rollup/driver: continue sequencing on current epoch if next L1 origin timestamp is not reached yet

* opnode/rollup: fix maxL2Time excl bound, and return adjusted maxL2time in long L1 gap case

* contracts: Add withdrawer tests

fixup! contracts: Add burn function to the Withdrawer contract

* Rename Finalization Window to Finalization Period

* contracts: Validate output root proof in portal

Moves the comparison into the Portal contract, and changes the
WithdrawalVerifier function from _verifyWithdrawerStorageRoot to
_deriveOutputRoot.

* contracts: Use solidity custom errors

* contracts: Use unchecked math when safe

* contracts: Add section dividers to contract layout

* specs: Update withdrawals spec to match impl

* contracts: Fix ETH burning in the withdrawer contract

* specs: Address review comments

* ops: Address review comments

* Update Dockerfile.opnode

* ops: Fix devnet

- Fixes the devnet config to correctly generate rollup/genesis files.
- Fixes a bug that causes the rollup node to get stuck when there's a large timestamp delta between genesis and the current time, as is the case in the devnet.

* Update opnode/rollup/driver/state.go

Co-authored-by: Diederik Loerakker <proto@protolambda.com>

* Update rollup.json

* Update predeploy addresses

The new predeploy addresses are incremented by 1 in order to avoid
conflicts with the system addresses.

* tests: Add tests for batch requests when the parent request fails

This test makes sure that we do not retry when the parent request fails. We don't, but it wasn't immediately clear.

I also removed a duplicate assertion lib since we're using `testify` everywhere else.

* opnode: Don't start sequencing blocks until past the L1 genesis (#325)

This has to be reverted to ensure that the sequencer starts producing
blocks at the same location as the verifier. Sometimes the previous
code would cause reorgs on the sequencer (expected), and sometimes
it would cause a chain split (unexpected and unexplained as of
writing this commit message 4/6/22).

* opnode: update to latest L2 geth

* readme: fix typos and break lines (#327)

* Update ref optimistic geth image (#328)

* ci: Improve timestamp management in devnet (#329)

This change sets timestamps to the current timestamp on fresh devnet deploys. This avoids the devnet having to fill in a bunch of blocks on startup. The timestamp is preserved between runs; `make devnet-clean` will erase the timestamp data if you want to start anew.

* contracts: Move withdrawal logic into abstract contract

* contracts: Move withdrawal hashing logic into library

* contracts: Update .gas-snapshot

* contracts: silence compiler warnings

* feat: expose debug http api (#334)

* opnode: Use l2 safe head instead of l2 head

Epochs should be inserted on top of the L2 Safe Head, not the
unsafe head. This was causing problems when a reorg would occur
and the node would start building in the wrong spot.

* opnode: Clear l1WindowBuf on insertEpoch failure

This is done to handle the case that the l1WindowBuf does not start
in the correct location (direct check is difficult with context).
The correct response to that is to clear the window and initialize
it from the correct block.

* hardhat: clean up the hardhat deposit task

* fix: oe => optimism (#338)

* opnode: Rework state test to build on safe head

This also got caught up in the unsafe/safe head confusion.

* opnode: fix setL1BlockValues selector used for deposits (#339)

Solidity ABI only looks at the function name and types, but does not
consider the name of the function parameters.

* rollup: fix address of the L1 Block Attributes contracts

* fix: typo in l1InfoPredeployAddress

* fix: gofmt and goimports

CI was asking for goimport'd when all it needed was gofmt.

* opnode: Import constants

* ops: fix formatting, add docker buildkit support

* ops: prune volumes on devnet-down

* fix: replace fraud with fault (#337)

* opnode: Fix bug in block creation (#345)

The NoTxPool flag was incorrectly set to falses in the InsertEpoch
function.

* opnode: Don't allow arbitrary mints (#346)

The ordering of `mint, value` was switched to be `value, mint`
when parsing emitted logs. This allows users to control their
mint rather than using `msg.value` which is being fed directly
into the event.

Thanks to Mofi (Inphi) for finding this.

* opnode,specs: deposit tx-hash uniqueness based on L1

* opnode/l1: verify critical receipt/log info we get from RPC

* opnode/test: reconstruct deposit hash in e2e test, fetch l2 deposit receipt

* opnode/l1: fix panic on channel send after close, sema channel can stay open

* go.mod: l2 geth update with improved deposit tx hash

* README: remove alpha branch + update description (#332)

* allow building everything from top-level makefile

* allow testing from top level makefile

* Update Makefile

Co-authored-by: Joshua Gutow <jgutow@optimism.io>

* opnode/l1: wrap errors and fmt test imports

* opnode: don't expose types.Block to derive or driver packages, prefer l2.ExecutionPayload

* opnode: add comments, fix genesis base case to get l1 origin

* init: integration tests (#344)

Co-authored-by: Javed Khan <tuxcanfly@gmail.com>

* opnode: Fix possible panic in sequencer batch creation (#342)

Previously we assumed that every deposit would be included in the L2
block. However, the L2 execution engine removes depoists that are
not valid. As such the number of deposits submitted to the execution
engine could be greater than the number of transactions returned
causing a panic.

This fixes the issue by counting the number of deposits returned
and using that to slice the payload transactions for batch submission.
(Deposits are not submitted in batches and instead read directly from L1).

This also does more sanity checks on the transactions included in the L2
block and rejects blocks with invalid deposit tx configurations. This is
done prior to marking the payload canonical with the FCU.

* opnode: Test Utilities (#319)

* opnode: Deduplicate fake chain source

The fake chain source is an easy way to create and L1 and L2 chains from
a simple string defition. There were previously two implementations and
now they are in a shared testutils package.

* opnode: Dedicated wait functions

This deduplicates wait for transactions on L1 and L2.

* opnode: New e2e test system

This splits out the initialization from the actual tests to eanble
multiple e2e tests. Some of the config is still not as dedpulicated
as I would like, but it is more centralized than it previously was.

Aims to move the setup from the tests and enable easier e2e tests.

* opnode: Silence FindSyncStartTest

* opnode: Add test for missing batches

This ensures the the sequencer follows the chain that is derived
from L1 rather than what it sequenced in the case that it misses
batch submission.

* opnode: Pull out system config to default cfg

This reduces duplication and makes tests easier to understand.

* opnode: Address PR comments

* opnode: Reduce flakiness in missing batch test

The proper way to do this is to wait for a safe block, but the API
is not yet ready for that.

* Fix sync-start bug + improve sync-start comments (#355)

Fix bug (L1 origin head with non-canonical ancestors)

This happened in the scenario where `l2ahead == true` (L1 origin of start is
ahead of know L1 head) and `start` had an ancestor whose L1 origin was known not
to be canonical.

Example: L1 head is at block X. `start` has an L1 origin with number X + 1 but its
parent L2 block has an L1 origin with number X whose blockhash is not canonical.
In this case, the algorithm still returned `start` as the unsafe L1 head, which is
incorrect.

This also touches a bunch of comments to generally improve understanbility.

* feat: deposit tx js helpers (#352)

* feat: js deposit tx

Hardhat task now lists the tx hash

* rollup: new deposit hashing

* nice api!

* fix: dockerfile

* devnet: Hardcode deposit feed bytecode (#359)

* opnode,l2os: Better lifecycle management (#358)

* opnode: Properly wait for limit client to shut down

We removed closing the semaphore channel because it would cause a
panic as we did not wait for in flight requests (and new requests)
to stop before closing the L1 source.

* l2os: Change shutdown procedure

This is now blocking, but also reuces the amount of error logs.

* opnode: Shutdown rollup nodes before clients/geth nodes

Also actually shuts down the L2 clients in the rollup node.

* opnode: Properly shut everything down

* style: clean up and comments for state.go (#356)

* opnode: Parameterize deposit contract address (#361)

The contract can be deployed to any address. It is set in the
rollup config file.

* opnode/test: Add reverted deposit with mint test

* fix(ci): use Foundry GHA (#367)

Nightly Foundry releases are kept around for 3 days. This means we cannot be pinning the download link on CI to a specific commit.

Instead, we use the Foundry Toolchain GHA which always uses the latest nightly version.

* opnode/test: check nonce increment in deposit fail

* integration-tests: End-to-end withdrawal test (#362)

Includes several fixes to make this work:

- Modifies the devnet to deploy the L1 contracts rather than include them in genesis.
- Changes the deployment of L2OutputOracle to support a custom start timestamp.
- Updates UserDeposits to properly filter out non-TransactionDeposited events.
- Updates the L2 output submitter to post the current L1 block with each output rather than the L2 checkpoint block
- Updates the opnode API to allow all vhosts and CORS domains.

* go.mod: bump l2geth version

* opnode: Update and deploy contracts (#360)

This is rather than hardcoding the contract bytecode. It slows down the e2e
tests (by having to wait 1 L1 block for contracts to be ready), but it
is more accurate to what occurs (and is now required as we initialize
actions).

This also pins the L2OutputOracle to v0.8.10 of solidity.

* specs: Update output root derivation (#357)

* specs: Update output root derivation

* specs: Separate version from payload

* opnode: libp2p setup

* p2p flags, config, and libp2p + discv5 setup

* specs: initial rollup node p2p spec

* specs: Fix definition of output root

* solidity: fail if updated gas snapshot isn't included

A new gas snapshot can be generated by running
`forge snapshot`

* p2p setup: implement review suggestions, fix toc, fix lint

* opnode/p2p: test with require instead of assert

* contracts: Fix reentrancy attack in WithdrawalsRelay (#378)

* contracts: Output oracle improvements (#370)

* contracts: Use correct uppercase for immutables

* itest: shorten submission interval for faster devnet tests

* contracts: Ensure submission interval is multiple of block time

* itest: Simplify withdrawals spec

* itest: Increase withdrawal finalization timeout

* itest: Bump withdrawal timeout to +5 minutes

* Bump timeout

* Bump again

* Add gas snap

Co-authored-by: Matthew Slipper <me@matthewslipper.com>

* Clean up withdrawals itest (#372)

* contracts: Use correct uppercase for immutables

* itest: shorten submission interval for faster devnet tests

* contracts: Ensure submission interval is multiple of block time

* itest: Simplify withdrawals spec

* itest: Increase withdrawal finalization timeout

* itest: Bump withdrawal timeout to +5 minutes

* itest: Extract logic into getTargetOutput

* Update 000_withdrawals.spec.ts

Co-authored-by: Matthew Slipper <me@matthewslipper.com>

* fix dead anchor link

* higher difficulty rather than number

* 'L1 attributes block' is not a thing + add link

* try to align sentences with lines

* ci: Add automated docker builds for opnode and l2 output submitter (#369)

* contracts: Add deleteL2Output function

* specs: Add deleteL2Output

* ci: Update Docker version in CircleCi (#381)

We ran into this on the monorepo too. Alpine 3.14 removes the `faccessat2` syscall. This caused "operation not permitted" errors while building. Upgrading Circle's Docker version fixes this. See https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2 for more information.

* contracts: reuse code (#380)

* contracts: reuse the library to compute withdrawal hash

The `WithdrawalVerifier` has a method to compute the
withdrawal hash as part of a library. Use this library
in both the L1 and L2 contracts so that code can be reused.

* contracts: gas snapshot

* opnode: Better parsing of the TransactionDepositedEvent (#382)

This now asserts that the offset for the bytes field of the event
is the correct value rather than the previous incorrect check. This
check is not fully necessary, but good to have to validate that the
data is well formed.

* Makefile,ops: docker compose up instead of run to make devnet-down work, and fix makefile label (#385)

* opnode: Do not drop all deposits on parsing error (#383)

* Change deposit gasLimit from uint256 to uint64

This prevents the user from setting a gas limit that the rollup node
is not able to parse.

* opnode: Do not drop all deposits on parsing error

Only drop the affected deposit. This does mean loss of funds, but
this is also a case that should never happen. In addition, halting
the chain means that while funds are safe, we are exposed to a
denial of service attack if it would be possible to cause a parsing
error from the deposit feed contract.

* opnode/test: Provide flag to access geth logs (#389)

This enables a more verbose output without having to modify the test
itself. Provide the flag `gethlogs` to see logs from geth.

* opnode: Fix L1 Info Transactions

There are three issues being fixed:
1. Set the `from` field to the correct magic value (found by ToB)
2. Update the L2 EE to provide gas to deposits. Otherwise the
   deposit transactions immediatley out of gas and fail.
3. Update the manual ABI encoding to match what solidity expects.
   The previous version was similar to packed encodeding, but as
   such could not be parsed by solidity.

This also includes a regression test that the parsing of the L1 info
tx and what is recorded in the state for each block matches.

* contracts: import messengers (#393)

* chore: copy L1 and L2 Messengers from Monorepo

* test: Add Messenger test files

Monorepo test cases are copied in as comments from the monorepo ts tests.

* chore: Add OZ upgradable contracts

New remappings were also added in order to avoid excessively

long import statements.

* chore: Importe OZ and OP contracts as node_modules

Necessary because hardhat does not support remappings!?!

y u no?

* refactor: Remove replayMessage()

We no longer need this function as it was only necessary when the CTC
a maximum gas limit per epoch concept. In order to remove the function
I had to copy in the L1xDM interface rather than import from the
node_modules.

* chore: Remove unused files

The ts test file is made redundant by an itest in the itest package.
The sol test file was not being used for anything

* refactor: Remove Address Manager and Resolver

Instead the CTC and SCC are state variables

* refactor: Replace CTC with OptimismPortal

* refactor: Remove SCC

We don't need to replace it with the L2OutputOracle in the L1xDM,
because the verification is now done in the OptimismPortal itself.

* forge install: forge-std

* forge install: ds-test

* refactor: Move boilerplate test code into CommonTest

* test: Add sendMessage and pause tests for L1xDM

* test: L1CrossDomainMessenger sendMessage and pause

* refactor: replace L2ToL1MessagePasser with Withdrawer contract

Also adds a lib with constant values for new bedrock predeploys.

* chore: Make functions external, and reorder for CEI

For whatever reason a bunch of functions on the messengers were public,
when they could have been external. I fixed that, and removed the
slither annotations. Where possible (in the sendMessage functions), I
also reordered the events and calls to respect
Checks-Effects-Interactions. There was no risk previously, but this
removes any question, and quiets slither.

* refactor: Reorganize ts helpers

Move helpers/index into utils.ts, and add other files which are exported in

the new index.ts.

* test: Add mock proof generation script and helpers

* test: Add L1xDM relayMessageSucceeds tests

* test: Add proof generation scripts and helpers

* refactor: Add l2Sender check in L2xDM

* refactor: Copy in the L1 and L2 standard bridge

At this point they are simply verbatim.

* refactor: Bridges - fix import paths

* refactor: Token Bridge - replace messengers with Portal
Also remove the CrossDomainEnabled lib.

refactor: Token Bridge - replace messengers with Portal
Also remove the CrossDomainEnabled lib.

* style: Address/remove some slither disable comments

style: Address/remove some slither disable comments

* refactor: extract l2Sender check into a modifier

* refactor: Support deposits of ETH in L2 Bridge

This copies in the IL2ERC20Bridge interface so that payable can be added.

In the case that the L2 token address matches OVM_ETH, the value of the

call will be forwarded.

* interface: add IWithdrawer.sol

* contracts: add comments to L1 contracts

* contracts: use unchecked

* contracts: fix imports in common test

* test: L1CrossDomainMessenger

* test: L2CrossDomainMessenger

* contracts: fix typo

* tests: bridge tests

* contracts: remove extra message assignment

* contracts: update gas snapshot

* forge install: solmate

* contracts: remove usage of OVM_ETH

All `OVM_ETH` will be migrated to `ETH` with the upgrade
to bedrock. We do not want to allow for the creation of new
`OVM_ETH` by depositing `ETH` into the bridge and have it
create `OVM_ETH` on L2.

* contracts: add in L2StandardERC20

* contracts: add in token factory

The token factory will deploy tokens on L2
that correspond to tokens on L1. This allows for
easy deposits through the bridge.

* contracts: test rlp lib for computing contract addrs

This library lets you compute the contract address
based on the deployment account and nonce.

h/t @t11s

* contracts: test infra for bridge

* contracts: add note to self

* contracts: fix compiler warnings

* contracts: update snapshot

* contracts: add IDepositFeed

* contracts: type cast uint256 to uint64 in messenger

* test: fix merge

* contracts: modify paths to compile with hardhat

* hardhat: update config

* contracts: fix build

* forge tests: first yarn install

* contracts: lint

* contracts: update snapshot

Co-authored-by: Maurelian <maurelian@protonmail.ch>

* contracts: fix abi encoding in proof verification (#395)

Use `abi.encode` instead of `abi.encodePacked` to ensure
a constant serialization. `abi.encode` will be sure to
pad the value to its size while `abi.encodePacked` will not
when operating on integers. There should not have been a bug here
because it was being called with a `bytes32`, which should
always be 32 bytes when returned from `abi.encodePacked`.

* contracts: don't unalias on L2->L1

Co-authored-by: smartcontracts <kelvin@optimism.io>

* opnode: attach p2p host and discovery to rollup node, update e2e test (#388)

* deps: update forge-std

* contracts: fix build

* build: fix bindings build

* fix: builds

* forge install: ds-test

* build: fix for good

* l2os: update bindings

* opnode: add seq nr to l1-info to fix driver bug

* feat: create opnode/predeploy package

This resolves an import cycle when trying to use
WithdrawalContractAddress in opnode/node. Otherwise the compiler
complains that opnode imports opnode/node which imports opnode.

* opnode: refactor RPC server / config, prototype batch serving

* feat: enforce size contraints on batch bundles

This algorithm is modeled mostly off the one used in the existing batch
submitter. The additional complexity enables us to configure a min and
max value such that max - min < max_l2_tx_size. Applied strictly,
submitting a large L2 tx would cause the batch construct to halt, since
the batch constructed without including batch `i` can be under the min,
while including batch `i` resulting in exceeding the maximum. The
additional complexity in the pruning algorithm is used to identify this
case and submit an undersized batch when it cannot be avoided. In the
general case, however, this strategy increases our expected
profitibability since the average batch size is closer to the desired
maximum.

* feat: add BundleBuilder helper class

* fix: typo in l2os config docs

* feat: add GetBatchBundle method to rollupclient.RollupClient

* feat: simplify l2os rollupclient init

* feat: reduce BSS poll interval

The current setup assumes the batches are submitted very soon after the
block is created on L2. Increasing the poll interval ensures that we
discover and publish new batches in response. Without modifying this
value, the subsequent changes will still fail even though the code is
correct.

* feat: add sequencer history db

* feat: add sequencer BSS driver to submit batches

* feat: add bss configuration

* feat: add LOG_TERMINAL flag to l2os, use non-global log instance

Also modifies the shutdown logic of l2os.Service to abort if stuck
publishing a transaction.

* feat: enable standalone bss, disable simple bss

* feat: remove simple bss

* feat: add bss to devnet

* fix: ci build bindings (#406)

* ci: build bindings in ci

* ci: install abigen

* ci: fix geth install

* bindings: regenerate

* contracts: no metadata hash in contracts

* bindings: regenerate

* opnode: Avoid busy-waiting while L2 head is behind L1

In the state select loop, we immediately request for new L2 blocks whenever
the latest L1 origin is behind L1. This can be an issue as we attempt to
re-request the latest L2 block without any delay. This is a DoS hazard
particularly when an L2 block cannot be retrieved because either the `L1Chain`
or `L2Chain` backends have errors.

Another problem with this is that other, more useful events in the state
select loop, are less likely to be scheduled by the Go runtime due to
the busy-wait.

Adding a small delay, before calling `reql2BlockCreation` of about 10ms
should be enough to prevent issues.

* opnode: Fix Incorrect error handling when creating an L2 block (#391)

* opnode: Fix Incorrect error handling when creating an L2 block

This is an issue ToB identified (issue #8). What happens is that
the ethereum.NotFound error would never be returned, but in general
if there is an error in the lookup, the state loop should keep
going.

* opnode: Properly wrap errors in l1/source.go

* opnode: Fuzz manual ABI parsing (#384)

* opnode: Add OptimismPortal deployed bytecode

This is important for fuzzing

* opnode: Add differential fuzzing test for deposit events

fixup: Proper tests

* build: Add fuzz target

* opnode: Fix ineffctual error assignment

* opnode: More fuzzing cleanup

The purpose of this is to make it more likely that the deposit
succeeds and that we fuzz what we actually want to fuzz (the
parsing).

* fix: DepositFeed.sol link

* opnode: gossip blocks topic validator and subscriber

* opnode/l2: ExecutionPayload SSZ encoding/decoding

* opnode/l2: check payload block hash consistency

* opnode/p2p,specs: update p2p block gossip validation

* opnode: refactor node initialization, change l1-head fan out, implement l2 payload driver receiver, setup block signing, update config / cli / e2e

* opnode/l2,opnode/rollup/driver: process incoming unsafe L2 blocks

* specs/rollup-node-p2p: fix toc

* opnode/p2p: compress published messages

* specs/rollup-node-p2p: clarify hardfork version start

* opnode/node: fix l1 head notify timeout bug

* opnode: fix receive p2p payload timeout bug, fix publishing buffer missing reset

* opnode: fix/improve new loggers

* opnode: fix p2p block distribution

* opnode/test: connect peers only after starting p2p application-layer protocols

* ops,opnode/p2p: fix opnode start up in devnet

* opnode/l2: use common.Big0 instead of big.NewInt(0), thx mark

* opnode: Add debug tracer to geth in e2e test (#418)

* Update reference geth entrypoint with variable parameters

* opnode: fix imports and code newlines style

* opnode,specs: gossip blocks validation seen cache fixes

* opnode/rollup/driver: no heuristics in driver, rely more on engine to handle reorgs

* opnode: tracer to watch node events during testing

* opnode,specs: clarify gossip params

* opnode/l2/ssz: fix lousy copy range

* opnode/rollup/driver/state: fix doc typo

Co-authored-by: Joshua Gutow <jgutow@optimism.io>

* Add version RPC (#432)

Meta:

- Fixes ENG-2200

* refactor: new messengers (#421)

* contracts: consolidate OptimismPortal

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: delete abstracts

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: base `CrossDomainMessenger` + L1 and L2

Implement the `L1CrossDomainMessenger` and
`L2CrossDomainMessenger` based on the base
`CrossDomainMessenger`. This makes the interfaces
the same on both sides.

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: `StandardBridge`

Also implement the `L1StandardBridge` and
`L2StandardBridge` based off of the base
`StandardBridge`

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: L2OutputOracle

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: standard bridge tests

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: cross domain hashing lib + tests

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: tests for cross domain messengers

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: L2ToL1MessagePasser

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: safe call lib

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: optimism mintable erc20

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: common test setup

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: optimism mintable token factory

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: update forge-std

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: test OptimismPortal

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: L1Block attributes

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: update libraries

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: update foundry.toml

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: Burner

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: delete dead code

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: update hh deploy scripts

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: update gas snapshot

Co-authored-by: smartcontracts <kelvin@optimism.io>

* integration-tests: update for new messengers

Co-authored-by: smartcontracts <kelvin@optimism.io>

* ops: devnet up script new genesis

Co-authored-by: smartcontracts <kelvin@optimism.io>

* opnode: new bindings

Co-authored-by: smartcontracts <kelvin@optimism.io>

* l2os: new bindings

Co-authored-by: smartcontracts <kelvin@optimism.io>

* contracts: update gas snapshot

Co-authored-by: smartcontracts <kelvin@optimism.io>

* specs: L2ToL1MessagePasser

* specs: messengers

* specs: bridges

* specs: fix linting issues

* contracts: better contract for is optimism mintable

* contracts: fix some comments

* contracts: assert finalization window has passed

* bindings: regenerate

* contracts: address comments

Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>

* opnode: p2p RPC, fix static-peers

* StateViz: Visualize rollup state changes (#419)

* StateViz: Visualize rollup state changes

* remove DEBUGMEs

* Add L1WindowBuf to viz

* move stateviz to opnode/cmd

* Add stateviz to docker-compose

* html lint

* Add SRI

* Fix linter

* Merge fixes

* Fixes from code review

* Imports

* Fix space

Co-authored-by: Matthew Slipper <me@matthewslipper.com>

* contracts: add backup logic for deposits

Introduces backup logic for deposits that prevents users from
accidentally making bad deposits. Deposits that don't complete
successfully will be returned back to the layer where the deposits were
made.

* contracts: add base gas to sent messages

Introduces base gas to the CrossDomainMessenger's sendMessage function.
Base gas is used to guarantee that all messages sent between messengers
will at the very least be able to store the message hash on the other
chain and therefore be replayable. Base gas scales dynamically with the
size of the message.

* contracts: use clearer message encoding

Uses the clearer abi.encodeWithSelector within CrossDomainMessenger's
sendMessage function. Doing this makes it much more apparent that a call
to the relayMessage function is being triggered.

* contracts: add L1BlockNumber predeploy

Re-introduces the L1BlockNumber predeployed contract for backwards
compatibility with the previous OVM_L1BlockNumber contract.

* opnode: p2p rpc client bindings, test update, minor p2p flag update

* opnode/p2p: set up connection notification before starting host B

* opnode/p2p: bundle p2p components into p2p node for separate testing

* opnode/p2p: minor style fix + rename

* opnode/p2p: add method to get peer info about self

* Fix SRI on stateviz (#440)

* Fix L2 Output Timestamps (#416)

There was an off by one error in the L2 Output Oracle contract in the timestamp to
block number conversion. In addition, the L2 Output Submitter (op proposer) did
not recognize that there was a mismatch between the timestamp/block number in
the header and the timestamp/block number from the contract.

This bug causes problems when trying to do withdrawals.


Fixes ENG-2128

* opnode: Withdrawal E2E test in go (#423)

This does the following:
- Adds withdrawal utilities (to opnode/withdrawals)
- Adds an end to end test in go of withdrawals
- Adds the L2 withdrawer contract
- Updates to a newer version of reference-go-ethereum


Fixes ENG-2202

* Adopt go.work, rename modules to prep for monorepo (#441)

Co-Authored-By: Matthew Slipper <me@matthewslipper.com>

Co-authored-by: protolambda <proto@protolambda.com>

* remove git submodules

* optimistic-specs: monorepo merge, mv into protocol dir

* bedrock contracts dependencies

* monorepo merge: fix Go env

* monorepo merge: move contracts-bedrock into non-Lerna package

* monorepo merge: bring back CI

Known issues:

- There are broken links that `lychee` picks up on. These still need to be updated.
- Slither returns errors, both here and in `optimistic-specs`.
- `go-bip39` was updated to a newer version. The newer version broke a `bss-core` test, which had to be fixed.
- Forge is not compatible with Lerna. As a result, the `contracts-bedrock` package had to be moved out of the `packages` hierarchy.
- The devnet itests don't work because the Go modules aren't on the default branch. We need to decide if we merge to develop, or stay on a feature branch before fixing this.

* outline bedrock dirs, rm stale protocol repo files (#2562)

* monorepo merge: re-run yarn

Co-authored-by: Joshua Gutow <jgutow@optimism.io>
Co-authored-by: Diederik Loerakker <proto@protolambda.com>
Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
Co-authored-by: Maurelian <maurelian@protonmail.ch>
Co-authored-by: smartcontracts <kelvin@optimism.io>
Co-authored-by: Murphy Law <Inphi@users.noreply.github.com>
Co-authored-by: norswap <norswap@gmail.com>
Co-authored-by: Javed Khan <tuxcanfly@gmail.com>
Co-authored-by: Murphy Law <mlaw2501@gmail.com>
Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>
Co-authored-by: Conner Fromknecht <conner@alum.mit.edu>
Co-authored-by: Luca Donno <30298476+lucadonnoh@users.noreply.github.com>
Co-authored-by: Ben Wilson <bwilson@optimism.io>
Co-authored-by: Ben Wilson <82120899+optimisticben@users.noreply.github.com>
2022-05-16 21:43:01 +00:00
github-actions[bot]
038c936db5 Version Packages 2022-05-05 18:26:32 +00:00
Matthew Slipper
67a4016fdf maint: Move Go packages into root of repo, adopt go.work (#2524)
- Adopts Go workspaces for future compatibility with the Bedrock move into the monorepo
- Moves Go packages to the root of the repo in order to fix import paths
- Rewrites existing Go import paths
- Removes Stackman, since it's not needed anymore

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-04 21:51:24 +00:00