this case failed randomly in github CI, the ratio of around 20%
but can not be reproduced locally by:
```
go test -p 1 -v -run TestSnapSyncWithBlobs -count 100
```
there could be a potential risk between `runEthPeer` and `runSnapExtension`,as
they shared the same handler.peers.
Add extra wait time for `runEthPeer` to resolve it.
And same for another UT: testBroadcastBlock
* core: use finalized block as the chain freeze indicator (#28683)
* core/rawdb: use max(finality, head-90k) as chain freezing threshold
* core: impl multi database for block data
* core: fix db inspect total size bug
* core: add tips for user who use multi-database
* core: adapt some cmd for multi-database
* core: adapter blockBlobSidecars
* core: fix freezer readHeader bug
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* ci: temp enable blobtx branch ci run;
* Switch ON blobpool & ensure Cancun hardfork can occur (#2223)
* feat: support blob storage & miscs; (#2229)
* chainconfig: use cancun fork for BSC;
* feat: fill WithdrawalsHash when BSC enable cancun fork;
* rawdb: support to CRUD blobs;
* freezer: support to freeze block blobs;
* blockchain: add blob cache & blob query helper;
* freezer: refactor addition table logic, add uts;
* blobexpiry: add more extra expiry time, and logs;
* parlia: implement IsDataAvailable function;
* blob: refactor blob transfer logic;
* blob: support config blob extra reserve;
* blockchian: support to import block with blob & blobGasFee; (#2260)
* blob: implement min&max gas price logic;
* blockchian: support import side chain;
* blobpool: reject the banned address;
* blockchain: add chasing head for DA check;
* params: update blob related config;
* blockchain: opt data available checking performance;
* params: modify blob related params;
* gasprice: support BEP-336 blob gas price calculate;
* blobTx: mining + brodcasting (#2253)
* blobtx mining pass (#2282)
* Sidecar fetching changes for 4844 (#2283)
* ci: temp enable blobtx branch ci run;
* Switch ON blobpool & ensure Cancun hardfork can occur (#2223)
* feat: support blob storage & miscs; (#2229)
* chainconfig: use cancun fork for BSC;
feat: fill WithdrawalsHash when BSC enable cancun fork;
* rawdb: support to CRUD blobs;
* freezer: support to freeze block blobs;
* blockchain: add blob cache & blob query helper;
* freezer: refactor addition table logic, add uts;
* blobexpiry: add more extra expiry time, and logs;
* parlia: implement IsDataAvailable function;
* blob: refactor blob transfer logic;
* blob: support config blob extra reserve;
* blockchian: support to import block with blob & blobGasFee; (#2260)
* blob: implement min&max gas price logic;
* blockchian: support import side chain;
* blobpool: reject the banned address;
* blockchain: add chasing head for DA check;
* params: update blob related config;
* blockchain: opt data available checking performance;
* params: modify blob related params;
* gasprice: support BEP-336 blob gas price calculate;
* fix failed check for WithdrawalsHash (#2276)
* eth: include sidecars in fitering of body
* core: refactor sidecars name
* eth: sidecars type refactor
* core: remove extra from bad merge
* eth: fix handlenewblock test after merge
* Implement eth_getBlobSidecars && eth_getBlobSidecarByTxHash (#2286)
* execution: add blob gas fee reward to system;
* syncing: support blob syncing & DA checking;
* naming: rename blobs to sidecars;
* fix the semantics of WithXXX (#2293)
* config: reduce sidecar cache to 1024 and rename (#2297)
* fix: Withdrawals turn into empty from nil when BlockBody has Sidecars (#2301)
* internal/api_test: add test case for eth_getBlobSidecars && eth_getBlobSidecarByTxHash (#2300)
* consensus/misc: rollback CalcBlobFee (#2306)
* flags: add new flags to override blobs' params;
* freezer: fix blob ancient save error;
* blobsidecar: add new sidecar struct with metadata; (#2315)
* core/rawdb: optimize write block with sidecars (#2318)
* core: more check for validity of sidecars
* mev: add TxIndex for mev bid (#2325)
* remove useless Config() (#2326)
* fix WithSidecars (#2327)
* fix: fix mined block sidecar issue; (#2328)
* fix WithSidecars (#2329)
---------
Co-authored-by: GalaIO <GalaIO@users.noreply.github.com>
Co-authored-by: buddho <galaxystroller@gmail.com>
Co-authored-by: Satyajit Das <emailtovamos@gmail.com>
Co-authored-by: Eric <45141191+zlacfzy@users.noreply.github.com>
Co-authored-by: zzzckck <152148891+zzzckck@users.noreply.github.com>
* internal/jsre: format receipt.{blobGasPrice,blobGasUsed} to int
Signed-off-by: jsvisa <delweng@gmail.com>
* internal/jsre: format tx.maxFeePerBlobGas to int
Signed-off-by: jsvisa <delweng@gmail.com>
* internal/jsre: format blob* in block
Signed-off-by: jsvisa <delweng@gmail.com>
---------
Signed-off-by: jsvisa <delweng@gmail.com>
* core/txpool: no need to run rotate if no local txs
Signed-off-by: jsvisa <delweng@gmail.com>
* Revert "core/txpool: no need to run rotate if no local txs"
This reverts commit 17fab173883168c586d57ca9c05dfcbd9e7831b4.
Signed-off-by: jsvisa <delweng@gmail.com>
* use Debug if todo is empty
Signed-off-by: jsvisa <delweng@gmail.com>
---------
Signed-off-by: jsvisa <delweng@gmail.com>
* make blobpool reject blob transactions with fee below the minimum
* core/txpool: some minot nitpick polishes and unified error formats
* core/txpool: do less big.Int constructions with the min blob cap
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This PR enhances eth_createAccessList RPC call to support scenarios where the node is launched with an unlimited gas cap (--rpc.gascap 0). The eth_createAccessList RPC call returns failure if user doesn't explicitly set a gas limit.
eth_call and debug_traceCall allow users to override various block fields, among them base fee. However the overriden base fee was not considered for computing the effective gas price of that message, and instead base fee of the base block was used. This has been fixed in this commit.
* remove i386 linux tests
* test: fix building of tests
* return empty logs instead of nil in receipts
* keep InitialBaseFee same with geth, so not break a ton of cases
* fix eth dir
* fix subdir of core
* fix subdir of eth
* fix cmd/geth dir
* fix ethtest by adding UpgradeStatusMsg when handshake
* fix ethclient_test.go
* fix ethclient/simulated
* fix internal
* fix graphql
* fix consensus
* fix accounts
* fix log
* fix p2p
* fix metrics
* fix tests dir
* fix golangci-lint
---------
Co-authored-by: Matus Kysel <matus.kysel@bnbchain.org>
This PR fixes an overflow which can could happen if inconsistent blockchain rules were configured. Additionally, it tries to prevent such inconsistencies from occurring by making sure that merge cannot be enabled unless previous fork(s) are also enabled.
* core/txpool, miner: speed up blob pool pending retrievals
* miner: fix test merge issue
* eth: same same
* core/txpool/blobpool: speed up blobtx creation in benchmark a bit
* core/txpool/blobpool: fix linter
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This pull request fixes a flaw in ethstats which can lead to node crash
A panic could happens when the local blockchain is reorging which causes the original head block not to be reachable (since number->hash canonical mapping is deleted). In order to prevent the panic, the block nilness is now checked in ethstats.
Improving two things here:
On hive, where we look at these tests, the Go code comment above the test
is not visible. When there is a failure, it's not obvious what the test is actually
expecting. I have converted the comments in to printed log messages to
explain the test more.
Second, I noticed that besu is failing some tests because it happens to request
a header when we want it to send transactions. Trying the minimal fix here to
serve the headers.
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
This change makes the legacy transaction pool use of `uint256.Int` instead of `big.Int`. The changes are made primarily only on the internal functions of legacypool.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
As mentioned in #26621, the block index format for era1 is not in line with the regular era block index. This change modifies the index so all relative offsets are based against the beginning of the block index record.
This change adds support for blob-transaction in certain API-endpoints, e.g. eth_fillTransaction. A follow-up PR will add support for signing such transactions.
* all: implement era format, add history importer/export
* internal/era/e2store: refactor e2store to provide ReadAt interface
* internal/era/e2store: export HeaderSize
* internal/era: refactor era to use ReadAt interface
* internal/era: elevate anonymous func to named
* cmd/utils: don't store entire era file in-memory during import / export
* internal/era: better abstraction between era and e2store
* cmd/era: properly close era files
* cmd/era: don't let defers stack
* cmd/geth: add description for import-history
* cmd/utils: better bytes buffer
* internal/era: error if accumulator has more records than max allowed
* internal/era: better doc comment
* internal/era/e2store: rm superfluous reader, rm superfluous testcases, add fuzzer
* internal/era: avoid some repetition
* internal/era: simplify clauses
* internal/era: unexport things
* internal/era,cmd/utils,cmd/era: change to iterator interface for reading era entries
* cmd/utils: better defer handling in history test
* internal/era,cmd: add number method to era iterator to get the current block number
* internal/era/e2store: avoid double allocation during write
* internal/era,cmd/utils: fix lint issues
* internal/era: add ReaderAt func so entry value can be read lazily
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/era: improve iterator interface
* internal/era: fix rlp decode of header and correctly read total difficulty
* cmd/era: fix rebase errors
* cmd/era: clearer comments
* cmd,internal: fix comment typos
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth, miner: fix enforcing the minimum miner tip
* ethclient/simulated: fix failing test due the min tip change
* accounts/abi/bind: fix simulater gas tip issue
* core/txpool/blobpool: clean up resurrected junk after a crash
* core/txpool/blobpool: track transaction insertions and rejections
* core/txpool/blobpool: linnnnnnnt
* eth/downloader: fix skeleton cleanup
* eth/downloader: short circuit if nothing to delete
* eth/downloader: polish the logic in cleanup
* eth/downloader: address comments
At some point, `ForkchoiceUpdatedV2` stopped working for `PayloadAttributesV1` while `paris` was active. This was causing a few failures in hive. This PR fixes that, and also adds a gate in `ForkchoiceUpdatedV1` to disallow `PayloadAttributesV3`.
GetPayloadVX should only return payloads which match its version. GetPayloadV2 is a special snowflake that supports v1 and v2 payloads. This change uses a a version-specific prefix within in the payload id, basically a namespace for the version number.
This PR fixes an issues in the new simulated backend. The root cause is the fact that the transaction pool has an internal reset operation that runs on a background thread.
When a new transaction is added to the pool via the RPC, the transaction is added to a non-executable queue and will be moved to its final location on a background thread. If the machine is overloaded (or simply due to timing issues), it can happen that the simulated backend will try to produce the next block, whilst the pool has not yet marked the newly added transaction executable. This will cause the block to not contain the transaction. This is an issue because we want determinism from the simulator: add a tx, mine a block. It should be in there.
The PR fixes it by adding a Sync function to the txpool, which waits for the current reset operation (if any) to finish, and then runs an entire round of reset on top. The new round is needed because resets are only triggered by new head events, so newly added transactions will not trigger the outer resets that we can wait on. The transaction pool would eventually internally do a reset even on transaction addition, but there's no easy way to wait on that and there's no meaningful reason to bubble that across everything. A clean outer reset will at worse be a small noop goroutine.
This PR introduces a few changes with respect to payload verification in fcu and new payload requests:
* First of all, it undoes the `verifyPayloadAttributes(..)` simplification I attempted in #27872.
* Adds timestamp validation to fcu payload attributes [as required](https://github.com/ethereum/execution-apis/blob/main/src/engine/cancun.md#specification-1) (section 2) by the Engine API spec.
* For the new payload methods, I also update the verification of the executable data. For `newPayloadV2`, it does not currently ensure that cancun values are `nil`. Which could make it possible to submit cancun payloads through it.
* On `newPayloadV3` the same types of checks are added. All shanghai and cancun related fields in the executable data must be non-nil, with the addition that the timestamp is _only_ with cancun.
* Finally it updates a newly failing catalyst test to call the correct fcu and new payload methods depending on the fork.
This change switches from using the `Hasher` interface to add/query the bloomfilter to implementing it as methods.
This significantly reduces the allocations for Search and Rebloom.
This change makes use of uin256 to represent balance in state. It touches primarily upon statedb, stateobject and state processing, trying to avoid changes in transaction pools, core types, rpc and tracers.
This change simplifies the logic for indexing transactions and enhances the UX when transaction is not found by returning more information to users.
Transaction indexing is now considered as a part of the initial sync, and `eth.syncing` will thus be `true` if transaction indexing is not yet finished. API consumers can use the syncing status to determine if the node is ready to serve users.
The code to compute a versioned hash was duplicated a couple times, and also had a small
issue: if we ever change params.BlobTxHashVersion, it will most likely also cause changes
to the actual hash computation. So it's a bit useless to have this constant in params.
EIP-4844 adds a new transaction type for blobs. Users can submit such transactions via `eth_sendRawTransaction`. In this PR we refrain from adding support to `eth_sendTransaction` and in fact it will fail if the user passes in a blob hash.
However since the chain can handle such transactions it makes sense to allow simulating them. E.g. an L2 operator should be able to simulate submitting a rollup blob and updating the L2 state. Most methods that take in a transaction object should recognize blobs. The change boils down to adding `blobVersionedHashes` and `maxFeePerBlobGas` to `TransactionArgs`. In summary:
- `eth_sendTransaction`: will fail for blob txes
- `eth_signTransaction`: will fail for blob txes
The methods that sign txes does not, as of this PR, add support the for new EIP-4844 transaction types. Resuming the summary:
- `eth_sendRawTransaction`: can send blob txes
- `eth_fillTransaction`: will fill in a blob tx. Note: here we simply fill in normal transaction fields + possibly `maxFeePerBlobGas` when blobs are present. One can imagine a more elaborate set-up where users can submit blobs themselves and we fill in proofs and commitments and such. Left for future PRs if desired.
- `eth_call`: can simulate blob messages
- `eth_estimateGas`: blobs have no effect here. They have a separate unit of gas which is not tunable in the transaction.
In the tracing tests, the base fee was generally set to nil. This commit changes this to pass the proper base instead, and fixes the few tests which become broken by the change.
Given the discussions around deprecating pending (see #28623 or ethereum/execution-apis#495), we can move away from using the pending block internally, and use latest instead
* accounts, ethclient: minor tweaks on the new simulated backend
* ethclient/simulated: add an initial batch of gas options
* accounts, ethclient: remove mandatory gasLimit constructor param
* accounts, ethclient: minor option naming tweaks
* cmd, core: resolve scheme from a read-write database
* cmd, core, eth: move the scheme check in the ethereum constructor
* cmd/geth: dump should in ro mode
* cmd: reverts
This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces
which is backed by a simulated blockchain. It was getting annoying to maintain the old
version of the simulated backend feature because there was a lot of code duplication with
the main client.
The new version is built using parts that we already have: an in-memory geth node instance
running in developer mode provides the chain, while the Go API is provided by ethclient.
A backwards-compatibility wrapper is provided, but the simulated backend has also moved to
a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
When managing geth, it is sometimes desirable to do a partial wipe; deleting state but retaining freezer data. A partial wipe can be somewhat tricky to accomplish.
This change implements the ability to perform partial wipe by making it possible to run geth removedb non-interactive, using command line options instead.
This PR will allow a previously underpriced transaction back in after a timeout
of 5 minutes. This will block most transaction spam but allow for transactions to
be re-broadcasted on networks with less transaction flow.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
# eth/fetcher/tx_fetcher.go
This is a minor refactor in preparation of changes to range verifier. This PR contains no intentional functional changes but moves (and renames) the light.NodeSet
* cmd, les, tests: remove light client code
This commit removes the light client (LES) code.
Since the merge the light client has been broken and
it is hard to maintain it alongside the normal client.
We decided it would be best to remove it for now and
maybe rework and reintroduce it in the future.
* cmd, eth: remove some more mentions of light mode
* cmd: re-add flags and mark as deprecated
* cmd: warn the user about deprecated flags
* eth: better error message
* triedb/pathdb: fix async node buffer diskroot mismatches when journaling
* triedb: check whether the async flush is done
* fix: generate new eth config
---------
Co-authored-by: VM <arimas@foxmail.com>
This pull request improves the condition to check if path state scheme is in use.
Originally, root node presence was used as the indicator if path scheme is used or not. However due to fact that root node will be deleted during the initial snap sync, this condition is no longer useful.
If PersistentStateID is present, it shows that we've already configured for path scheme.
Original problem was caused by #28595, where we made it so that as soon as we start to sync, the root of the disk layer is deleted. That is not wrong per se, but another part of the code uses the "presence of the root" as an init-check for the pathdb. And, since the init-check now failed, the code tried to re-initialize it which failed since a sync was already ongoing.
The total impact being: after a state-sync has begun, if the node for some reason is is shut down, it will refuse to start up again, with the error message: `Fatal: Failed to register the Ethereum service: waiting for sync.`.
This change also modifies how `geth removedb` works, so that the user is prompted for two things: `state data` and `ancient chain`. The former includes both the chaindb aswell as any state history stored in ancients.
---------
Co-authored-by: Martin HS <martin@swende.se>
Here we update the eth and snap protocol test suites with a new test chain,
created by the hivechain tool. The new test chain uses proof-of-stake. As such,
tests using PoW block propagation in the eth protocol are removed. The test suite
now connects to the node under test using the engine API in order to make it
accept transactions.
The snap protocol test suite has been rewritten to output test descriptions and
log requests more verbosely.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This is primarily to make lint work again on macOS 14. The older version of golangci-lint kept crashing.
Also included is a fix for a goroutine leak in the recently-introduced function MustRunCommandWithOutput.
This is the fix to issue #27483. A new hiddenBytes() is introduced to calculate the byte size of hidden items in the freezer table. When reporting the size of the freezer table, size of the hidden items will be subtracted from the total size.
---------
Co-authored-by: Yifan <Yifan Wang>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change fixes a problem with our non-core binaries: evm, clef, bootnode.
First of all, they failed to convert from legacy loglevels 1 to 5, to the new slog loglevels -4 to 4.
Secondly, the logging was actually setup in the init phase, and then overridden in the main. This is not needed for evm, since it used the same flag name as the main geth verbosity. Better to let the flags/internal handle the logging init.
Certain flags, such as `--rpc.txfeecap` currently do not have an env-var auto-generated for them. This change adds three missing cli flag types to the auto env-var helper function to fix this.
* p2p/discover: add liveness check in collectTableNodes
* p2p/discover: fix test
* p2p/discover: rename to appendLiveNodes
* p2p/discover: add dedup logic back
* p2p/discover: simplify
* p2p/discover: fix issue found by test
* core: LoadChainConfig return predefined config for built-in net firstly
* cmd/geth: add a warn message for chain config in the configuration file
* consensus/parlia: change chain config log level when New parlia
* core: fix code style
* cmd, core, params: add support for the Holesky testnet
* cmd/devp2p: add support for holesky for the dns crawler
# Conflicts:
# cmd/devp2p/nodesetcmd.go
# cmd/geth/main.go
# cmd/utils/flags.go
# core/genesis.go
# params/bootnodes.go
# params/config.go
This fixes a database corruption issue that could occur during state healing.
When sync is aborted while certain modifications were already committed, and a
reorg occurs, the database would contain incorrect trie nodes stored by path.
These nodes need to detected/deleted in order to obtain a complete and fully correct state
after state healing.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change implements CommitteeChain which is a key component of the beacon light client. It is a passive data structure that can validate, hold and update a chain of beacon light sync committees and updates, starting from a checkpoint that proves the starting committee through a beacon block hash, header and corresponding state. Once synced to the current sync period, CommitteeChain can also validate signed beacon headers.
The dump after state-test didn't work, the problem was an error, "Already committed", which was silently ignored.
This change re-initialises the state, so the dumping works again.
This change
- Removes interface `log.Format`,
- Removes method `log.FormatFunc`,
- unexports `TerminalHandler.TerminalFormat` formatting methods (renamed to `TerminalHandler.format`)
- removes the notion of `log.Lazy` values
The lazy handler was useful in the old log package, since it
could defer the evaluation of costly attributes until later in the
log pipeline: thus, if the logging was done at 'Trace', we could
skip evaluation if logging only was set to 'Info'.
With the move to slog, this way of deferring evaluation is no longer
needed, since slog introduced 'Enabled': the caller can thus do
the evaluate-or-not decision at the callsite, which is much more
straight-forward than dealing with lazy reflect-based evaluation.
Also, lazy evaluation would not work with 'native' slog, as in, these
two statements would be evaluated differently:
```golang
log.Info("foo", "my lazy", lazyObj)
slog.Info("foo", "my lazy", lazyObj)
```
* cmd/geth: standardize the action of importing bls account
* rename bLSAccountPasswordFileFlag to blsAccountPasswordFileFlag
* use password of wallet as account password when new an account or importing a raw key
These changes improves the performance of the non-coloured terminal formatting, _quite a lot_.
```
name old time/op new time/op delta
TerminalHandler-8 10.2µs ±15% 5.4µs ± 9% -47.02% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
TerminalHandler-8 2.17kB ± 0% 0.40kB ± 0% -81.46% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
TerminalHandler-8 33.0 ± 0% 5.0 ± 0% -84.85% (p=0.008 n=5+5)
```
I tried to _somewhat_ organize the commits, but the it might still be a bit chaotic. Some core insights:
- The function `terminalHandler.Handl` uses a mutex, and writes all output immediately to 'upstream'. Thus, it can reuse a scratch-buffer every time.
- This buffer can be propagated internally, making all the internal formatters either write directly to it,
- OR, make use of the `tmp := buf.AvailableBuffer()` in some cases, where a byte buffer "extra capacity" can be temporarily used.
- The `slog` package uses `Attr` by value. It makes sense to minimize operating on them, since iterating / collecting into a new slice, iterating again etc causes copy-on-heap. Better to operate on them only once.
- If we want to do padding, it's better to copy from a constant `space`-buffer than to invoke `bytes.Repeat` every single time.
Add read locking of db lock around access to dirties cache in hashdb.Database to prevent
data race versus hashdb.Database.dereference which can modify the dirities map by deleting
an item.
Fixes#28541
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.
Main changes are as follows:
* removes any log handlers that were unused in the Geth codebase.
* Json, logfmt, and terminal formatters are now slog handlers.
* Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options.
* `--log.backtraceat` and `--log.debug` are removed.
The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged.
A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed.
----
For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did
```golang
log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true))))
```
You now instead need to do
```golang
log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true)))
```
See more about reasoning here: https://github.com/ethereum/go-ethereum/issues/28558#issuecomment-1820606613
* eth/gasestimator: early exit for plain transfer and error allowance
* core, eth/gasestimator: hard guess at a possible required gas
* internal/ethapi: update estimation tests with the error ratio
* eth/gasestimator: I hate you linter
* graphql: fix gas estimation test
---------
Co-authored-by: Oren <orenyomtov@users.noreply.github.com>
This change fixes two type-inconsistencies in the JS tracer:
- In most places we return byte arrays as a `Uint8Array` to the tracer. However it seems we missed doing the conversion for `ctx` fields which are passed to the tracer during `result`. They are passed as simple arrays. I think Uint8Arrays are more suitable and we should change this inconsistency. Note: this will be a breaking-change. But I believe the effect is small. If we look at our tracers we see that these fields (`ctx.from`, `ctx.to`, etc.) are used in 2 ways. Passed to `toHex` which takes both array or buffer. Or the length was measured which is the same for both types.
- The `slice` taking in `int, int` params versus `memory.slice` taking `int64, int64` params. I suggest changing `slice` types to `int64`. This should have no effect almost in any case.
There were several problems related to dumping state.
- If a preimage was missing, even if we had set the `OnlyWithAddresses` to `false`, to export them anyway, the way the mapping was constructed (using `common.Address` as key) made the entries get lost anyway. Concerns both state- and blockchain tests.
- Blockchain test execution was not configured to store preimages.
This changes makes it so that the block test executor takes a callback, just like the state test executor already does. This callback can be used to examine the post-execution state, e.g. to aid debugging of test failures.
* cmd, les, tests: remove light client code
This commit removes the light client (LES) code.
Since the merge the light client has been broken and
it is hard to maintain it alongside the normal client.
We decided it would be best to remove it for now and
maybe rework and reintroduce it in the future.
* cmd, eth: remove some more mentions of light mode
* cmd: re-add flags and mark as deprecated
* cmd: warn the user about deprecated flags
* eth: better error message
Adds a subcommand: `geth snapshot export-preimages`, to export preimages of every hash found during a snapshot enumeration: that is, it exports _only the active state_, and not _all_ preimages that have been used but are no longer part of the state.
This tool is needed for the verkle transition, in order to distribute the preimages needed for the conversion. Since only the 'active' preimages are exported, the output is shrunk from ~70GB to ~4GB.
The order of the output is the order used by the snapshot enumeration, which avoids database thrashing. However, it also means that storage-slot preimages are not deduplicated.
geth --dev can be used with an existing data directory and genesis block. Since
dev mode only works with PoS, we need to verify that the merge has happened.
Co-authored-by: Felix Lange <fjl@twurst.com>
It turns out that encoding json.RawMessage is slow because
package json basically parses the message again to ensure it is valid.
We can avoid the slowdown by encoding the entire RPC notification once,
which yields a 30% speedup.
* rpc: make subscription test faster
reduces time for TestClientSubscriptionChannelClose
from 25 sec to < 1 sec.
* trie: cache trie nodes for faster sanity check
This reduces the time spent on TestIncompleteSyncHash
from ~25s to ~16s.
* core/forkid: speed up validation test
This takes the validation test from > 5s to sub 1 sec
* core/state: improve snapshot test run
brings the time for TestSnapshotRandom from 13s down to 6s
* accounts/keystore: improve keyfile test
This removes some unnecessary waits and reduces the
runtime of TestUpdatedKeyfileContents from 5 to 3 seconds
* trie: remove resolver
* trie: only check ~5% of all trie nodes
This change adds a check to ensure that transactions added to the legacy pool are not treated as 'locals' if the global locals-management has been disabled.
This change makes the pool enforce the --txpool.pricelimit setting.
This PR verifies the accounts' storage as specified in a blockchain test's postState field
The expect-section, it does really only check that the test works. It's meant for the test-author to verify that "If the test does what it's supposed to, then the nonce of X should be 2, and the slot Y at Z should be 0x123.
This expect-section is not exhaustive (not full post-state)
It is also not auto-generated, but put there manually by the author.
We can still check it, as a test-sanity-check, in geth
* trie: use pooling of iterator states in iterator
The node iterator burns through a lot of memory while iterating a trie, and a lot of
that can be avoided by using a fairly small pool (max 40 items).
name old time/op new time/op delta
Iterator-8 6.22ms ± 3% 5.40ms ± 6% -13.18% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
Iterator-8 2.36MB ± 0% 1.67MB ± 0% -29.23% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
Iterator-8 37.0k ± 0% 29.8k ± 0% ~ (p=0.079 n=4+5)
* ethdb/memorydb: avoid one copying of key
By making the transformation from []byte to string at an earlier point,
we save an allocation which otherwise happens later on.
name old time/op new time/op delta
BatchAllocs-8 412µs ± 6% 382µs ± 2% -7.18% (p=0.016 n=5+4)
name old alloc/op new alloc/op delta
BatchAllocs-8 480kB ± 0% 490kB ± 0% +1.93% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
BatchAllocs-8 3.03k ± 0% 2.03k ± 0% -32.98% (p=0.008 n=5+5)
This PR moves our fuzzers from tests/fuzzers into whatever their respective 'native' package is.
The historical reason why they were placed in an external location, is that when they were based on go-fuzz, they could not be "hidden" via the _test.go prefix. So in order to shove them away from the go-ethereum "production code", they were put aside.
But now we've rewritten them to be based on golang testing, and thus can be brought back. I've left (in tests/) the ones that are not production (bls128381), require non-standard imports (secp requires btcec, bn256 requires gnark/google/cloudflare deps).
This PR also adds a fuzzer for precompiled contracts, because why not.
This PR utilizes a newly rewritten replacement for go-118-fuzz-build, namely gofuzz-shim, which utilises the inputs from the fuzzing engine better.
This change allows the creation of a genesis block for verkle testnets. This makes for a chunk of code that is easier to review and still touches many discussion points.
Currently, geth's will return `[]` for any `len(topics) > 4` log filter. The EVM only supports up to four logs, via LOG4 opcode, so larger criterias fail. This change makes the filter query exit early in those cases.
cockroachdb/pebble@422dce9 added Errorf to the Logger interface, this change makes it possible to compile geth with that version of pebble by adding the corresponding method to panicLogger.
* cmd/geth: more testcases for logging
This adds more edgecases around logging, particularly around handling of different types of nil-values
as concrete types and within interfaces.
Also adds tests with 'reserved' values which breaks json/logfmt formats. The json output is checked in,
but not actively used by any testcase at the moment.
* cmd/geth/testdata: remove timestamps
* core/vm: set basefee to 0 internally on eth_call
* core: nicer 0-basefee, make it work for blob fees too
* internal/ethapi: make tests a bit more complex
* core: fix blob fee checker
* core: make code a bit more readable
* core: fix some test error strings
* core/vm: Get rid of weird comment
* core: dict wrong typo
Currently, one can use the "withLogs" parameter to include logs in the
callTracer results, which allows the user to see at which trace level
was each log emitted.
This commit adds a position field to the logs which determine
the exact ordering of a call's logs and its subcalls. This would
be useful e.g. for explorers wishing to display the flow of execution.
Co-authored-by: jsvisa <delweng@gmail.com>
This change improves GenerateChain to support internal chain history access (ChainReader)
for the consensus engine and EVM.
GenerateChain takes a `parent` block and the number of blocks to create. With my changes,
the consensus engine and EVM can now access blocks from `parent` up to the block currently
being generated. This is required to make the BLOCKHASH instruction work, and also needed
to create real clique chains. Clique uses chain history to figure out if the current signer is in-turn,
for example.
I've also added some more accessors to BlockGen. These are helpful when creating transactions:
- g.Signer returns a signer instance for the current block
- g.Difficulty returns the current block difficulty
- g.Gas returns the remaining gas amount
Another fix in this commit concerns the receipts returned by GenerateChain. The receipts now
have properly derived fields (BlockHash, etc.) and should generally match what would be
returned by the RPC API.
This adds warning logs when the read does not match the expected count.
We can also remove the size limit since the function documentation explicitly states
that callers should limit the count.
accountTrieCache and storageTrieCache were introduced in this PR:
https://github.com/bnb-chain/bsc/pull/257, which is to improve performance.
Actually the performance gain is quite limited, as there is already dirty
and clean cache for trie node.
And after big merge, these 2 cache can not be used when PBSS is enabled.
So remove these code to simplify the logic.
a little copying is better than a little dependency
-- go proverb
We have this dependency on docker, a.k.a moby: a gigantic library, and we only need ~70 LOC,
so here I tried moving it inline instead.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is a bit in preparation for the slog work in #28187 .
Our current test re logging mostly test the internals, but we have no real end-to-end test of the logging output. This PR introduces a simple reexec-based log tester. This also relies upon a special mode in geth, which can be made to eject a set of predefined log messages (only available if the build-tag `integrationtests` is used
e.g. go run --tags=integrationtests ./cmd/geth --log.format terminal logtest
While working on this, I also noticed a quirk in the setup: when geth was configured to use a file output, then two separate handlers were used (one handler for the file, one handler for the console). Using two separate handlers means that two formatters are used, thus the formatting of any/all records happened twice. This PR changes the mechanism to use two separate io.Writers instead, which is both more optimal and fixes a bug which occurs due to a global statefulness in the formatter.
The String() version of BlockNumberOrHash uses decimal for all block numbers, including negative ones used to indicate labels. Switch to using BlockNumber.String() which encodes it correctly for use in the JSON-RPC API.
This PR removes panics from stacktrie (mostly), and makes the Update return errors instead. While adding tests for this, I also found that one case of possible corruption was not caught, which is now fixed.
This change closes https://github.com/ethereum/go-ethereum/issues/27730 . By using an iterator instead of a slice of transactions, we can better handle the case when an individual transaction (within an otherwise well-formed RLP-list) cannot be decoded.
* go.mod: upgrade prysm and the indrect dependency
prysm from v4.0.2 to v4.0.8, and run go mod tidy
* ci: upgrade go version from 1.19 to 1.20
* go-version: upgrade from v1.19 to v1.20
there is some dependency on go v1.20, such as go-libp2p v0.27.8
and also run go mod tidy
* dependency: upgrade docker version for security
it is not a big issue, since docker is only used for test purpose.
* rand: update the usage of math/rand after golang v1.20
2 APIs of math/rand module were deprecated since golang v1.20.
that is: rand.Seed() and rand.Read(), refer: ettps://pkg.go.dev/math/rand
"rand.Seed(seed int64)" has been replaced by: "r := rand.New(rand.NewSource(seed int64))",
need to initialize it with an instance before use
"rand.Read()" has been replaced by "crypto/rand.Read()"
* readme: need golang v1.20+ to build bsc
Fixes a bug where the ethstats omits to report full block contents. This bug was a side-effect of https://github.com/ethereum/go-ethereum/pull/26777, where `CurrentBlock` was changed to return a header instead of a block, leading to a failed type assertion.
A goroutine is used to manage the lifetime of subscriptions managed by
resubscriptions. When the subscription ends with no error, the resub
goroutine ends as well. However, the resub goroutine needs to live
long enough to read from the unsub channel. Otheriwse, an Unsubscribe
call deadlocks when writing to the unsub channel.
This is fixed by adding a buffer to the unsub channel.
* api/bind: Add CallOpts.BlockHash to allow calling contracts at a specific block hash.
* ethclient: Add BalanceAtHash, NonceAtHash and StorageAtHash functions
This change enhances the stacktrie constructor by introducing an option struct. It also simplifies the `Hash` and `Commit` operations, getting rid of the special handling round root node.
This changes fixes a bug in the fetcher, where the timeout for how long to remember underpriced transaction was erroneously compared, and the timeout never hit.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change fixes#28355, where eth_getProof failed to return the correct codehash under certain conditions. This PR changes the logic to unconditionally look up the codehash, and also adds some more tests.
This changes implements faster post-selfdestruct iteration of storage slots for deletion, by using snapshot-storage+stacktrie to recover the trienodes to be deleted. This mechanism is only implemented for path-based schema.
For hash-based schema, the entire post-selfdestruct storage iteration is skipped, with this change, since hash-based does not actually perform deletion anyway.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
During snap-sync, we request ranges of values: either a range of accounts or a range of storage values. For any large trie, e.g. the main account trie or a large storage trie, we cannot fetch everything at once.
Short version; we split it up and request in multiple stages. To do so, we use an origin field, to say "Give me all storage key/values where key > 0x20000000000000000". When the server fulfils this, the server provides the first key after origin, let's say 0x2e030000000000000 -- never providing the exact origin. However, the client-side needs to be able to verify that the 0x2e03.. indeed is the first one after 0x2000.., and therefore the attached proof concerns the origin, not the first key.
So, short-short version: the left-hand side of the proof relates to the origin, and is free-standing from the first leaf.
On the other hand, (pun intended), the right-hand side, there's no such 'gap' between "along what path does the proof walk" and the last provided leaf. The proof must prove the last element (unless there are no elements).
Therefore, we can simplify the semantics for trie.VerifyRangeProof by removing an argument. This doesn't make much difference in practice, but makes it so that we can remove some tests. The reason I am raising this is that the upcoming stacktrie-based verifier does not support such fancy features as standalone right-hand borders.
* build: upgrade to golang 1.21.2
* build: verify checksums via tool
* deps: upgrade go to 1.21.3
* build: move more build metadata into checksum file
* build: move gobootsrc to checksums
This change addresses an issue in snap sync, specifically when the entire sync process can be halted due to an encountered empty storage range.
Currently, on the snap sync client side, the response to an empty (partial) storage range is discarded as a non-delivery. However, this response can be a valid response, when the particular range requested does not contain any slots.
For instance, consider a large contract where the entire key space is divided into 16 chunks, and there are no available slots in the last chunk [0xf] -> [end]. When the node receives a request for this particular range, the response includes:
The proof with origin [0xf]
A nil storage slot set
If we simply discard this response, the finalization of the last range will be skipped, halting the entire sync process indefinitely. The test case TestSyncWithUnevenStorage can reproduce the scenario described above.
In addition, this change also defines the common variables MaxAddress and MaxHash.
* cmd, core: resolve scheme from a read-write database
* cmd, core, eth: move the scheme check in the ethereum constructor
* cmd/geth: dump should in ro mode
* cmd: reverts
This change
- Removes the owner-notion from a stacktrie; the owner is only ever needed for comitting to the database, but the commit-function, the `writeFn` is provided by the caller, so the caller can just set the owner into the `writeFn` instead of having it passed through the stacktrie.
- Removes the `encoding.BinaryMarshaler`/`encoding.BinaryUnmarshaler` interface from stacktrie. We're not using it, and it is doubtful whether anyone downstream is either.
* eth: enforce announcement metadatas and drop peers violating the protocol
* eth/fetcher: relax eth/68 validation a bit for flakey clients
* tests/fuzzers/txfetcher: pull in suggestion from Marius
* eth/fetcher: add tests for peer dropping
* eth/fetcher: linter linter linter linter linter
This is a minor refactor in preparation of changes to range verifier. This PR contains no intentional functional changes but moves (and renames) the light.NodeSet
This change refactors stacktrie to separate the stacktrie itself from the
internal representation of nodes: a stacktrie is not a recursive structure
of stacktries, rather, a framework for representing and operating upon a set of nodes.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change fixes the bug in a benchmark, where the input to the trie is reused in a way which is not correct.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change updates `evm b11r` (blockbuilder) and `evm t8n` (transition) tools to contain cancun updates (e.g. new header fields)
---------
Co-authored-by: Mario Vega <marioevz@gmail.com>
When MatcherSession encounters an error, it attempts to close the session.
Closing waits for all goroutines to finish, including the 'distributor'.
However, the distributor will not exit until all requests have returned.
This patch fixes the issue by delivering the (empty) result to the distributor
before calling Close().
So apparently in the spec the base block parameter of eth_call is optional.
I agree that "latest" is a sane default for this that most people would use.
This PR will allow a previously underpriced transaction back in after a timeout
of 5 minutes. This will block most transaction spam but allow for transactions to
be re-broadcasted on networks with less transaction flow.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Adding a space beween function opOrigin() and opcCaller() in instruciton.go.
Adding a space beween function opkeccak256() and opAddress() in instruciton.go.
This fixes an issue where the --bootnodes flag was overridden by the config file.
---------
Co-authored-by: NathanBSC <Nathan.l@nodereal.io>
Co-authored-by: Felix Lange <fjl@twurst.com>
When MatcherSession encounters an error, it attempts to close the session.
Closing waits for all goroutines to finish, including the 'distributor'. However, the
distributor will not exit until all requests have returned.
This patch fixes the issue by delivering the (empty) result to the distributor
before calling Close().
* parlia: reject header with `WithdrawalsHash` before shanghai fork
* log: output chainconfig when start up
* eth: fix TestOptionMaxPeersPerIP failure of goroutine order
This PR makes the tool use the --bootnodes list as the input to devp2p crawl.
The flag will take effect if the input/output.json file is missing or empty.
* cmd/evm: improve flags handling
This fixes some issues with flags in cmd/evm. The supported flags did not
actually show up in help output because they weren't categorized. I'm also
adding the VM-related flags to the run command here so they can be given
after the subcommand name. So it can be run like this now:
./evm run --code 6001 --debug
* cmd/evm: enable all forks by default in run command
The default genesis was just empty with no forks at all, which is annoying because
contracts will be relying on opcodes introduced in a fork. So this changes the default to
have all forks enabled.
* core/asm: fix some issues in the assembler
This fixes minor bugs in the old assembler:
- It is now possible to have comments on the same line as an instruction.
- Errors for invalid numbers in the jump instruction are reported better
- Line numbers in errors were off by one
* rlp/rlpgen: remove build tag
This tag was supposed to prevent unstable output when types reference each other. Imagine
there are two struct types A and B, where a reference to type B is in A. If I run rlpgen
on type B first, and then on type A, the generator will see the B.EncodeRLP method and
call it. However, if I run rlpgen on type A first, it will inline the encoding of B.
The solution I chose for the initial release of rlpgen was to just ignore methods
generated by rlpgen using a build tag. But there is a problem with this: if any code in
the package calls EncodeRLP explicitly, the package can't be loaded without errors anymore
in rlpgen, because the loader ignores it. Would be nice if there was a way to just make it
ignore invalid functions during type checking (they're not necessary for rlpgen), but
golang.org/x/tools/go/packages does not provide a way of ignoring them.
Luckily, the types we use rlpgen with do not reference each other right now, so we can
just remove the build tags for now.
This adds block and receipt fields for EIP-4844.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change includes a lot of things, listed below.
### Split up interfaces, write vs read
The interfaces have been split up into one write-interface and one read-interface, with `Snapshot` being the gateway from write to read. This simplifies the semantics _a lot_.
Example of splitting up an interface into one readonly 'snapshot' part, and one updatable writeonly part:
```golang
type MeterSnapshot interface {
Count() int64
Rate1() float64
Rate5() float64
Rate15() float64
RateMean() float64
}
// Meters count events to produce exponentially-weighted moving average rates
// at one-, five-, and fifteen-minutes and a mean rate.
type Meter interface {
Mark(int64)
Snapshot() MeterSnapshot
Stop()
}
```
### A note about concurrency
This PR makes the concurrency model clearer. We have actual meters and snapshot of meters. The `meter` is the thing which can be accessed from the registry, and updates can be made to it.
- For all `meters`, (`Gauge`, `Timer` etc), it is assumed that they are accessed by different threads, making updates. Therefore, all `meters` update-methods (`Inc`, `Add`, `Update`, `Clear` etc) need to be concurrency-safe.
- All `meters` have a `Snapshot()` method. This method is _usually_ called from one thread, a backend-exporter. But it's fully possible to have several exporters simultaneously: therefore this method should also be concurrency-safe.
TLDR: `meter`s are accessible via registry, all their methods must be concurrency-safe.
For all `Snapshot`s, it is assumed that an individual exporter-thread has obtained a `meter` from the registry, and called the `Snapshot` method to obtain a readonly snapshot. This snapshot is _not_ guaranteed to be concurrency-safe. There's no need for a snapshot to be concurrency-safe, since exporters should not share snapshots.
Note, though: that by happenstance a lot of the snapshots _are_ concurrency-safe, being unmutable minimal representations of a value. Only the more complex ones are _not_ threadsafe, those that lazily calculate things like `Variance()`, `Mean()`.
Example of how a background exporter typically works, obtaining the snapshot and sequentially accessing the non-threadsafe methods in it:
```golang
ms := metric.Snapshot()
...
fields := map[string]interface{}{
"count": ms.Count(),
"max": ms.Max(),
"mean": ms.Mean(),
"min": ms.Min(),
"stddev": ms.StdDev(),
"variance": ms.Variance(),
```
TLDR: `snapshots` are not guaranteed to be concurrency-safe (but often are).
### Sample changes
I also changed the `Sample` type: previously, it iterated the samples fully every time `Mean()`,`Sum()`, `Min()` or `Max()` was invoked. Since we now have readonly base data, we can just iterate it once, in the constructor, and set all four values at once.
The same thing has been done for runtimehistogram.
### ResettingTimer API
Back when ResettingTImer was implemented, as part of https://github.com/ethereum/go-ethereum/pull/15910, Anton implemented a `Percentiles` on the new type. However, the method did not conform to the other existing types which also had a `Percentiles`.
1. The existing ones, on input, took `0.5` to mean `50%`. Anton used `50` to mean `50%`.
2. The existing ones returned `float64` outputs, thus interpolating between values. A value-set of `0, 10`, at `50%` would return `5`, whereas Anton's would return either `0` or `10`.
This PR removes the 'new' version, and uses only the 'legacy' percentiles, also for the ResettingTimer type.
The resetting timer snapshot was also defined so that it would expose the internal values. This has been removed, and getters for `Max, Min, Mean` have been added instead.
### Unexport types
A lot of types were exported, but do not need to be. This PR unexports quite a lot of them.
* fix: crash of highestVerifiedHeader
* fix: panic of blobpool
* fix: genesis set up
* 1. modify NewDatabaseWithNodeDB to upstream
2. fix race use of hasher in statedb
3. fix use wrong value when updateTrie
* fix dir legacypool
* fix dir blobpool
* fix dir vote
* remove diffsync related code
* fix core/state/snapshot
* disable pipeCommit for now
* fix applyTransaction for bloom setting
* CI: fast finality in gasprice test
* CI: diffFetcher was removed
* CI: downloader, remove beaconsync test
* CI: no beaconsync in downloader, remove a failed case
TestCheckpointChallenge was removed in:
https://github.com/ethereum/go-ethereum/pull/27147
since after merge, it is useless for ethereum, but might be useful for BSC.
disable the case right now, as it is not a big issue.
* CI: bsc protocol decHandlers
* CI: receipt Bloom process
* 1. skip CheckConfigForkOrder for non-parlia engine
2. all test cases in core work well now
cd core && go test ./... -v
* fix test cases in trie dir
* CI: no beaconsync in downloader, remove a failed case(redo)
* fix dir miner
* fix dir cmd/geth
* CI: filter test, BaseFee & Finality
* fix dir graphql
* remove diffStore
* fix ethclient
* fix TestRPCGetTransactionReceipt
* fix dir internal
* ut add dir ethstats and signer
* disable pipeCommit thoroughly; fix concurrent map iteration and map write in statedb
* CI: fix snap sync
it could be changed by mistake
* fix tests/Run to generate snapshot
* prepare for merge
* remove useless
* use common hasher in getDeletedStateObject, no race here
* an critical comment for state.Prepare
* do not copy nil accessList
* add omitempty tag for unused new fields of core.Genesis
* remove totalFees
* calculate fees before FinalizeAndAssemble
* revert interface Finalize of consensus
* do not double gas limit upon london block
* use Leveldb as default
* Revert "remove diffStore"
This reverts commit df343b1374.
* Revert "remove diffsync related code"
This reverts commit 8d84b81fea.
* compile pass after revert
* remove diffsync
* fix dir eth/protocols/trust
* fix TestFastNode
* decHandlers for trust protocol
* keep persist diff in test
This updates minisign to the latest version. One new thing is that minisign (not go-minisign) has started to prehash the file, and in order to make geth pass the version-check, we need to sign the file in legacy-mode.
On ACD 163, it was agreed to bump the target and max blob values from `2/4` to `3/6` for future devnets until we could decide on final mainnet number. This change contains said update, making master pass all the hive tests. The final decision for mainnet cancun is still to be made.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
* core/forkid: skip genesis forks by time
* core/forkid: add comment about skipping non-zero fork times
* core/forkid: skip all time based forks in genesis using loop
* core/forkid: simplify logic for dropping time-based forks
This chang creates a GaugeInfo metrics type for registering informational (textual) metrics, e.g. geth version number. It also improves the testing for backend-exporters, and uses a shared subpackage in 'internal' to provide sample datasets and ordered registry.
Implements #21783
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes implements faster post-selfdestruct iteration of storage slots for deletion, by using snapshot-storage+stacktrie to recover the trienodes to be deleted. This mechanism is only implemented for path-based schema.
For hash-based schema, the entire post-selfdestruct storage iteration is skipped, with this change, since hash-based does not actually perform deletion anyway.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR makes EIP-4788 work in the engine API and miner. It also fixes some bugs related to
EIP-4844 block processing and mining. Changes in detail:
- Header.BeaconRoot has been renamed to ParentBeaconRoot.
- The engine API now implements forkchoiceUpdatedV3
- newPayloadV3 method has been updated with the parentBeaconBlockRoot parameter
- beacon root is now applied to new blocks in miner
- For EIP-4844, block creation now updates the blobGasUsed field of the header
Just some minor optimizations I figured out a while ago. By using ReadBytes instead of
Bytes on the rlp stream, we can save the allocation of a temporary buffer for the typed tx
payload.
If kind == rlp.Byte, the size reported by Stream.Kind will be zero, but we need a buffer
of size 1 for ReadBytes. Since typed txs always have to be longer than 1 byte, we can just
return an error for kind == rlp.Byte.
There is a also a small change for Log: since the first three fields of Log are the ones that
should appear in the canon encoding, we can simply ignore the remaining fields via
struct tag. Doing this removes an indirection through the rlpLog type.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
We're trying a new named pipe library, which should hopefully fix some occasional failures in CI.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
The PR #26274 broke the evm statetest command a bit, in that it stopped spitting out the stateroot following a non-successful statetest-execution.
This PR changes it back, so the stateroot is unconditionally output on stderr, and makes it so fuzzing works again.
This change implements "EIP 4788 : Beacon block root in the EVM". It implements version-2 of EPI-4788, main difference being that the contract is an actual contract rather than a precompile, as in #27289.
Currently, we trigger the logic to (un)index transactions when the node receives a new
block. However, in some cases the node may not receive new blocks (eg, when the Geth node
is configured without peer discovery, or when it acts as an RPC node for historical-only
data).
In these situations, the Geth node user may not have previously configured txlookuplimit
(i.e. the default of around one year), but later realizes they need to index all
historical blocks. However, adding txlookuplimit=0 and restarting geth has no effect. This
change makes it check for required indexing work once, on startup, to fix the issue.
Co-authored-by: Martin Holst Swende <martin@swende.se>
FastFinality puts more infor into the header.extra field to keep vote information.
For mainnet, on epoch height, it could be 1526 bytes, which was 517 bytes before.
So the hardcoded 700 bytes for header could be no longer enough, increase it by
2 times would be enough.
this bug could cause P2P sync failure for nodes that are lagging behind, since they
would request access of ancient db, and GetBlockHeaders could be failed.
Optimizations:
- Previously, if a transaction was reverting, EstimateGas would exhibit worst-case behavior and binary search up to the max gas limit (~40 state-clone + tx executions). This change allows EstimateGas to return after only a single unconstrained execution in this scenario.
- Uses the gas used from the unconstrained execution to bias the remaining binary search towards the likely solution in a simple way that doesn't impact the worst case. For a typical contract-invoking transaction, this reduces the median number of state-clone+executions from 25 to 18 (28% reduction).
Cleanup:
- added & improved function + code comments
- correct the EstimateGas documentation to clarify the gas limit determination is at latest block, not pending, if the blockNr is unspecified.
ReadSkeletonHeader can return nil if the header is missing, so we should
not access fields on it. Note that calling .Hash() on a nil header is fine, so there
is no need to actually check for nil.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes the forkID calculation to ignore time-based forks that occurred before the
genesis block. It's supposed to be done this way because the spec says:
> If a chain is configured to start with a non-Frontier ruleset already in its genesis, that is NOT considered a fork.
This raises the JSON-RPC batch request limits significantly for the engine API endpoint.
The limits are now also hard-coded, so users won't get them wrong. I have chosen these limits:
maximum batch items: 2000
maximum batch response size: 250MB
While it would also be possible to disable batch limits completely for the engine API,
I think having some limits is a good safety net against misbehaving CLs. Since this
isn't configurable, we really want to ensure this limit will never become an issue in the
CL/EL communication, so I set them quite high.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR removes the newly added txpool.Transaction wrapper type, and instead adds a way
of keeping the blob sidecar within types.Transaction. It's better this way because most
code in go-ethereum does not care about blob transactions, and probably never will. This
will start mattering especially on the client side of RPC, where all APIs are based on
types.Transaction. Users need to be able to use the same signing flows they already
have.
However, since blobs are only allowed in some places but not others, we will now need to
add checks to avoid creating invalid blocks. I'm still trying to figure out the best place
to do some of these. The way I have it currently is as follows:
- In block validation (import), txs are verified not to have a blob sidecar.
- In miner, we strip off the sidecar when committing the transaction into the block.
- In TxPool validation, txs must have a sidecar to be added into the blobpool.
- Note there is a special case here: when transactions are re-added because of a chain
reorg, we cannot use the transactions gathered from the old chain blocks as-is,
because they will be missing their blobs. This was previously handled by storing the
blobs into the 'blobpool limbo'. The code has now changed to store the full
transaction in the limbo instead, but it might be confusing for code readers why we're
not simply adding the types.Transaction we already have.
Code changes summary:
- txpool.Transaction removed and all uses replaced by types.Transaction again
- blobpool now stores types.Transaction instead of defining its own blobTx format for storage
- the blobpool limbo now stores types.Transaction instead of storing only the blobs
- checks to validate the presence/absence of the blob sidecar added in certain critical places
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
This should fix#27726. With enough load, it might happen that the SetPongHandler
callback gets invoked before the call to SetReadDeadline is made in pingLoop. When
this occurs, the socket will end up with a 30s read deadline even though it got the pong,
which will lead to a timeout.
The fix here is processing the pong on pingLoop, synchronizing with the code that
sends the ping.
Block takes a number and a hash. The spec is unclear on what should happen in this case, leaving it an implemenation detail. With this change, we return an error in case both number and hash are passed in.
This change removes a chainconfig parameter passed into rawdb.ReadLogs, which is not used nor needed.
It also modifies the filter loop slightly, avoiding a labeled break and instead using a method.
This change does not modify any behaviour.
Context: The UpdateContractCode method was introduced for the state storage commitment
schemes that include the whole code for their commitment computation. It must therefore be called
before the root hash is computed at the end of IntermediateRoot.
This should have no impact on the MPT since, in this context, the method is a no-op.
This adds support for the "yParity" field in transaction objects returned by RPC
APIs. We somehow forgot to add this field even though it has been in the spec for
a long time.
This change rearranges the accessor methods in block.go and fixes some minor issues with
the copy-on-write logic of block data. Fixed issues:
- Block.WithWithdrawals did not create a shallow copy of the block.
- Block.WithBody copied the header unnecessarily, and did not preserve withdrawals.
However, the bugs did not affect any code in go-ethereum because blocks are *always*
created using NewBlockWithHeader().WithBody().WithWithdrawals()
* all: implement path-based state scheme
* all: edits from review
* core/rawdb, trie/triedb/pathdb: review changes
* core, light, trie, eth, tests: reimplement pbss history
* core, trie/triedb/pathdb: track block number in state history
* trie/triedb/pathdb: add history documentation
* core, trie/triedb/pathdb: address comments from Peter's review
Important changes to list:
- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated
* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache
* core/rawdb: fix tests
* trie/triedb/pathdb: rename metrics, change clean cache key
* trie/triedb: manage the clean cache inside of disk layer
* trie/triedb/pathdb: move journal function
* trie/triedb/path: fix tests
* trie/triedb/pathdb: fix journal
* trie/triedb/pathdb: fix history
* trie/triedb/pathdb: try to fix tests on windows
* core, trie: address comments
* trie/triedb/pathdb: fix test issues
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Sets the `currentExcessBlobGas` from env, alternatively calculates it based on `parentExcessBlobGas` and `parentBlobGasUsed`. It then emits the `currentExcessBlobGas` and `currentBlobGasUsed` into the output, to be used as parent-values for a future iteration.
Closes#27785Closes#27783
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* tests: split up state test execution
* Revert "tests: split up state test execution"
This reverts commit 96017c248c85d24e93ad013a2bbe8b38c99327c0.
* build: bump test timeout to 20 minutes
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core/blobpool: implement txpool for blob txs
* core/txpool: track address reservations to notice any weird bugs
* core/txpool/blobpool: add support for in-memory operation for tests
* core/txpool/blobpool: fix heap updating after SetGasTip if account is evicted
* core/txpool/blobpool: fix eviction order if cheap leading txs are included
* core/txpool/blobpool: add note as to why the eviction fields are not inited in reinject
* go.mod: pull in inmem billy form upstream
* core/txpool/blobpool: fix review commens
* core/txpool/blobpool: make heap and heap test deterministic
* core/txpool/blobpool: luv u linter
* core/txpool: limit blob transactions to 16 per account
* core/txpool/blobpool: fix rebase errors
* core/txpool/blobpool: luv you linter
* go.mod: revert some strange crypto package dep updates
Hertz is equal to Berlin and London
- Plato will be enabled on BSC mainnet at height: 30720096
- Hertz will be enabled on BSC mainnet at height: 31302048
Hertz is for ecosystem compatibility, to enable some customized EIPs of Berlin&London on BSC
This is a spin-out from the EIP-4844 devnet branch, containing just the Engine API modifications
and nothing else. The newPayloadV3 endpoint won't really work in this version, but we need the
data structures for testing so I'd like to get this in early.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change removes PoW header syncing related code from LES and also deletes
duplicated packages les/catalyst, les/downloader and les/fetcher. These package copies
were created because people wanted to make changes in their eth/ counterparts, but weren't
able to adapt LES code to the API changes.
EIP-6780: SELFDESTRUCT only in same transaction
> SELFDESTRUCT will recover all funds to the caller but not delete the account, except when called in the same transaction as creation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This updates the reference tests to the latest version and also adds logic
to process EIP-4844 blob transactions into the state transition. We are now
passing most Cancun fork tests.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
It is usually best to set GOMAXPROCS to the number of available CPU cores. However, setting
it like that does not work well when the process is quota-limited to a certain number of CPUs.
The automaxprocs library configures GOMAXPROCS, taking such limits into account.
This changes the port mapping procedure such that, when the requested port is unavailable
an alternative port suggested by the router is used instead.
We now also repeatedly request the external IP from the router in order to catch any IP changes.
Co-authored-by: Felix Lange <fjl@twurst.com>
The struct logger (or opcode tracer) was missing the return data field even
if this was explicitly enabled by user via `"enableReturnData": true` in the config.
This PR fixes this issue.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change adds the ability to perform reads from freezer without size limitation. This can be useful in cases where callers are certain that out-of-memory will not happen (e.g. reading only a few elements).
The previous API was designed to behave both optimally and secure while servicing a request from a peer, whereas this change should _not_ be used when an untrusted peer can influence the query size.
This simplifies the code that initializes the discovery a bit, and
adds new flags for enabling/disabling discv4 and discv5 separately.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
Back before #27178 , we spun up a number of ethash verifiers to verify headers. So we also had tests to ensure that we were indeed able to abort verification even if we had multiple workers running.
With PR #27178, we removed the parallelism in verification, and these tests are now failing, since we now just sequentially fire away the results as fast as possible on one routine.
This change removes the (sometimes failing) tests
This removes text parsing in leveldb metrics collection code. All metrics
can now be accessed through the stats API provided by leveldb.
We also add new gauge-typed metrics that count the number of tables at each level.
---------
Co-authored-by: Exca-DK <Exca-DK@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
* core/state, light, les: make signature of ContractCode hash-independent
* push current state for feedback
* les: fix unit test
* core, les, light: fix les unittests
* core/state, trie, les, light: fix state iterator
* core, les: address comments
* les: fix lint
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: lx <92799281+brilliant-lx@users.noreply.github.com>
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Verkle trees store the code inside the trie. This PR changes the interface to pass the code, as well as the dirty flag to tell the trie package if the code is dirty and needs to be updated. This is a no-op for the MPT and the odr trie.
In all other UDPv4 methods, the deadline is checked first. It seems weird to me that ping is an exception. Deadline comparison is also less resource intensive.
Co-authored-by: Exca-DK <Exca-DK@users.noreply.github.com>
This changes the eth_getProof method implementation to re-encode the requested
storage keys, canonicalizing them in the response. For backwards-compatibility reasons,
go-ethereum accepts non-canonical hex keys. Accepting them is fine, but we should
not mirror invalid inputs into the output.
Closes#27306
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This is likely the culprit behind several data corruption issues, e.g. where data has been
written to the freezer, but the deletion from pebble does not go through due to process
crash.
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
* go.mod: update kzg libraries to use big-endian
* go.sum: ran go mod tidy
* core/testdata/precompiles: fix blob verification test
* core/testdata/precompiles: fix blob verification test
Package rpc uses cgo to find the maximum UNIX domain socket path
length. If exceeded, a warning is printed. This is the only use of cgo in this
package. It seems excessive to depend on cgo just for this warning, so
we now hard-code the usual limit for Linux instead.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ensures Reheap will be called even before the London fork activates.
Since Reheap would otherwise only be called through `SetBaseFee` after London,
the list would just keep growing if the fork was not enabled or not reached yet.
* all: move main transaction pool into a subpool
* go.mod: remove superfluous updates
* core/txpool: review fixes, handle txs rejected by all subpools
* core/txpool: typos
The logs in this function are pulled straight from disk in rawdb.ReadRawReceipts and
also modified in receipts.DeriveFields, so removing the copy should be fine.
We had to do this workaround because it wasn't possible to export typed arrays from
JS to []byte. This was added in dop251/goja@2352993, so we can use the better way now.
This adds two ways to check for subscription support. First, one can now check
whether the transport method (HTTP/WS/etc.) is capable of subscriptions using
the new Client.SupportsSubscriptions method.
Second, the error returned by Subscribe can now reliably be tested using this
pattern:
sub, err := client.Subscribe(...)
if errors.Is(err, rpc.ErrNotificationsUnsupported) {
// no subscription support
}
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds server-side limits for JSON-RPC batch requests. Before this change, batches
were limited only by processing time. The server would pick calls from the batch and
answer them until the response timeout occurred, then stop processing the remaining batch
items.
Here, we are adding two additional limits which can be configured:
- the 'item limit': batches can have at most N items
- the 'response size limit': batches can contain at most X response bytes
These limits are optional in package rpc. In Geth, we set a default limit of 1000 items
and 25MB response size.
When a batch goes over the limit, an error response is returned to the client. However,
doing this correctly isn't always possible. In JSON-RPC, only method calls with a valid
`id` can be responded to. Since batches may also contain non-call messages or
notifications, the best effort thing we can do to report an error with the batch itself is
reporting the limit violation as an error for the first method call in the batch. If a batch is
too large, but contains only notifications and responses, the error will be reported with
a null `id`.
The RPC client was also changed so it can deal with errors resulting from too large
batches. An older client connected to the server code in this PR could get stuck
until the request timeout occurred when the batch is too large. **Upgrading to a version
of the RPC client containing this change is strongly recommended to avoid timeout issues.**
For some weird reason, when writing the original client implementation, @fjl worked off of
the assumption that responses could be distributed across batches arbitrarily. So for a
batch request containing requests `[A B C]`, the server could respond with `[A B C]` but
also with `[A B] [C]` or even `[A] [B] [C]` and it wouldn't make a difference to the
client.
So in the implementation of BatchCallContext, the client waited for all requests in the
batch individually. If the server didn't respond to some of the requests in the batch, the
client would eventually just time out (if a context was used).
With the addition of batch limits into the server, we anticipate that people will hit this
kind of error way more often. To handle this properly, the client now waits for a single
response batch and expects it to contain all responses to the requests.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/txpool: abstraction prep work for secondary pools (blob pool)
* core/txpool: leave subpool concepts to a followup pr
* les: fix tests using hard coded errors
* core/txpool: use bitmaps instead of maps for tx type filtering
* cmd/evm: make evm blocktest output logs if so instructed
* Apply suggestions from code review
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This changes the journal logic to mark the state object dirty immediately when it
is reset.
We're mostly adding this change to appease the fuzzer. Marking it dirty immediately
makes no difference in practice because accounts will always be modified by EVM
right after creation.
Continuing with a series of PRs to make the Trie interface more generic, this PR moves
the RLP encoding of storage slots inside the StateTrie and light.Trie implementations,
as other types of tries don't use RLP.
* p2p/discover: remove ReadRandomNodes
Even though it's public, this method is not callable by code outside of
package p2p/discover because one can't get a valid instance of Table.
* p2p/discover: add Table.Nodes
* p2p/discover: make Table settings configurable
In unit tests and externally developed cmd/devp2p test runs, it can be
useful to tune the timer intervals used by Table.
Drop the notions of uncles, and disables activities while syncing
- Disable activities (e.g. generate pending state) while node is syncing,
- Disable empty block submission (but empty block is still kept for payload building),
- Drop uncle notion since (ethash is already deprecated)
Deserialize hex keys early to shortcut on invalid input, and re-use the account storageTrie for each proof for each proof in the account, preventing repeated deep-copying of the trie.
Closes#27308
--------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
eth: make StorageRangeAt take a block hash or number
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
RPC methods `eth_getHeaderBy*` returned a size value which was meant for internal
processes. Please instead use `size` field returned by `eth_getBlockBy*` if you're interested
in the RLP encoded storage size of the block.
Signed-off-by: jsvisa <delweng@gmail.com>
Here is the list:
a.BEP-227: Implement EIP-3198: BASEFEE opcode
b.BEP-226: Implement EIP-1559 with base fee of 0
c.BEP-228: Implement EIP-3541: Prevent deploying contracts starting with 0xEF
d.BEP-212: Implement EIP-3529: Reduction in Refunds
ethclient accepts certain negative block number values as specifiers for the "pending",
"safe" and "finalized" block. In case of "pending", the value accepted by ethclient (-1)
did not match rpc.PendingBlockNumber (-2).
This wasn't really a problem, but other values accepted by ethclient did match the
definitions in package rpc, and it's weird to have this one special case where they don't.
To fix it, we decided to change the values of the constants rather than changing ethclient.
The constant values are not otherwise significant. This is a breaking API change, but we
believe not a dangerous one.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
implements the ability to run several state-tests in one instance. By not providing a statetest path to the `evm statetest` command, the path(s) will instead be read from `stdin`.
Upgrades graphiql to v2.4.4. The interface has become much nicer, and there are extra features like tabs, history, dark mode etc.
This change also now uses golang embed to bundle the resources.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Enable following Berlin EIPs on BSC and introduce HertzBlock
** EIP-2565: ModExp Gas Cost
** EIP-2929: Gas cost increases for state access opcodes
** EIP-2718: Typed Transaction Envelope
** EIP-2930: Optional access lists
One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237)
This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests.
This changes the RPC server to ignore methods using *context.Context as parameter
and *error as return value type. Methods with such types would crash the server when
called.
This PR adds a staleness-check to AccountRLP, before checking the bloom-filter and potentially going directly into the disklayer.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* EstimateGas should use LatestBlockNumber by default
* graphql: default to use latest for gas estimation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cryto/kzg4844: pull in the C and Go libs for KZG cryptography
* go.mod: pull in the KZG libraries
* crypto/kzg4844: add basic becnhmarks for ballpark numbers
* cmd, crypto: integrate both CKZG and GoKZG all the time, add flag
* cmd/utils, crypto/kzg4844: run library init on startup
* crypto/kzg4844: make linter happy
* crypto/kzg4844: push missing file
* crypto/kzg4844: fully disable CKZG but leave in the sources
* build, crypto/kzg4844, internal: link CKZG by default and with portable mode
* crypto/kzg4844: drop verifying the trusted setup in gokzg
* internal/build: yolo until it works?
* cmd/utils: make flag description friendlier
Co-authored-by: Martin Holst Swende <martin@swende.se>
* crypto/ckzg: no need for double availability check
* build: tiny flag cleanup nitpick
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
When block import fails, the error displays the number of the first block past the import batch, not the number of the failing block. This change fixes this problem by identifying which blocks fails and reporting its number.
The first validator in snap.Recents is shifted out and allowed to seal the block
already. However, when determining if we should slash the validator in Finalize
and FinalizeAndAssemble, we mark that first validator as unable to seal block
and don't slash this validator. This commit fixes that bug and creates a
separate helper function to check if the validator recently signed a block for
consistency in all places the check happens.
rename parameter
In this case, the naming of "extapi" might create some confusion. Although it represents an External Signer Backend, its name could be mistaken for an API. In reality, it is a backend instance used for communicating with external signers. A better naming choice could be "extBackend" or "externalBackend" to more accurately describe that it is a backend instance rather than an API.
This removes the 'time' field from logs, as well as from the tracer interface. This change makes the trace output deterministic. If a tracer needs the time they can measure it themselves. No need for evm to do this.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR modifies the interface for the results of `debug_traceBlock` and `debug_traceCall` by adding the `txHash`, allowing users to identify which transaction's trace result corresponds to.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This adds logic to prepend 'M' or 'E' to Solidity identifiers when they would
otherwise violate Go identifier naming rules.
Closes#26972
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Adds an optional config parameter to eth_call which allows users to override block context fields (same functionality that was added to traceCall in #24871)
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes TALKREQ message processing to run the handler on separate goroutine,
instead of running on the main discv5 dispatcher goroutine. It's better this way because
it allows the handler to perform blocking actions.
I'm also adding a new method TalkRequestToID here. The method allows implementing
a request flow where one node A sends TALKREQ to another node B, and node B later
sends a TALKREQ back. With TalkRequestToID, node B does not need the ENR of A to
send its request.
Makes the `geth account ... ` commands usable even if a geth-process is already executing, since the account commands do not read the chaindata, it was not required for those to use the same locking mechanism.
---
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* trie: add node type common package
In trie/types package, a few node wrappers are defined, which will be used
in both trie package, trie/snap package, etc. Therefore, a standalone common
package is created to put these stuffs.
* trie: rename trie/types to trie/trienode
This is a breaking GraphQL API change. All numeric values are now encoded as
hex strings. The motivation for this change is matching JSON-RPC outputs more
closely.
Numbers in query parameters are accepted as both decimal integers and hex strings.
* all: remove notion of trusted checkpoints in the post-merge world
* light: remove unused function
* eth/ethconfig, les: remove unused config option
* les: make linter happy
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd/utils, node: switch to Pebble as the default db if none exists
* node: fall back to LevelDB on platforms not supporting Pebble
* core/rawdb, node: default to Pebble at the node level
* cmd/geth: fix some tests explicitly using leveldb
* ethdb/pebble: allow double closes, makes tests simpler
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
Follow-up to #26697, makes the crawler less verbose on route53-based scenarios.
It also changes the loglevel from debug to info on Updates, which are typically the root, and can be interesting to see.
The EIP150Hash was an idea where, after the fork, we hardcoded the forked hash as an extra defensive mechanism. It wasn't really used, since forks weren't contentious and for all the various testnets and private networks it's been a hassle to have around.
This change removes that config field.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
* core/vm: deepcopy jumptable when enabling extra eips
When the interpreter is configured to use extra-eips,
this change makes it so that all the opcodes are deep-copied,
to prevent accidental modification of the 'base' jumptable.
Original-auther: yihuang <huang@crypto.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/vm: implement EIP-3860: Limit and meter initcode
Implementation of https://eips.ethereum.org/EIPS/eip-3860, limit and meter initcode.
Most of this change takes the go-ethereum implementation as reference.
Original-author: Andrei Maiboroda <andrei@ethereum.org>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core/vm: implement EIP-3855: PUSH0 instruction
* core/vm: Implement PUSH0
* Move PUSH0 to enable3855
* Add method doc
Original-author: Alex Beregszaszi <alex@rtfs.hu>
* core/vm: enable EIP-3855 (PUSH0) in Boneh
Original-author: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
* core/vm: Fix issue of incorrect instructionSet used for jump_table
Also update the related test case for gas change.
* core/vm: fix test cases issues that not suitable for boneh
* core/vm: reuse ErrMaxInitCodeSizeExceeded as error message
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR unifies the error handling in miner.
Whenever an error occur while applying a transaction, the transaction should be regarded as invalid and all following transactions from the same sender not executable because of the nonce restriction. The only exception is the `nonceTooLow` error which is handled separately.
Prior to this change, it was possible that transactions are erroneously deemed as 'future' although they are in fact 'pending', causing them to be dropped due to 'future' not being allowed to replace 'pending'.
This change fixes that, by doing a more in-depth inspection of the queue.
Use the new atomic types in package eth/tracers
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Makes the float-gauges lock-free
name old time/op new time/op delta
CounterFloat64Parallel-8 1.45µs ±10% 0.85µs ± 6% -41.65% (p=0.008 n=5+5)
---------
Co-authored-by: Exca-DK <dev@DESKTOP-RI45P4J.localdomain>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR removes the Debug field from vmconfig, making it so that if a tracer is set, debug=true is implied.
---------
Co-authored-by: 0xTylerHolmes <tyler@ethereum.org>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
This includes a semantic change to the `callTracer` as well as `flatCallTracer`.
The value of field `gas` in the **first** call frame will change as follows:
- It previously contained gas available after initial deductions (i.e. tx costs)
- It will now contain the full tx gasLimit value
Signed-off-by: jsvisa <delweng@gmail.com>
* core/state: Remove prefetcher on stop
* core/blockchain: Wait for `triedb.Dereference` in `writeBlockWithState`
* eth/protocols/diff: Fix index mismatch in `TestGetDiffLayers`
Currently, most of transaction validation while holding the txpool mutex: one exception being an early-on signature check.
This PR changes that, so that we do all non-stateful checks before we entering the mutex area. This means they can be performed in parallel, and to enable that, certain fields have been made atomic bools and uint64.
This change enables log rotation, which can be activated using the flag --log.rotate. Additional parameters that can be given are:
- log.maxsize to set maximum size before files are rotated,
- log.maxbackups to set how many files are retailed,
- log.maxage to configure max age of rotated files,
- log.compress whether to compress rotated files
The way to configure location of the logfile(s) is left unchanged, via the `log.logfile` parameter.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Currently the t8n tool uses the same block number for the current block and its parent while calculating the base fee. This causes incorrect base fee calculation for the london fork block. This commit sets the parent block number to be one less than the current block number
Adds error handling for the case that UnpackLog or UnpackLogIntoMap is called with a log that has zero topics.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Removes the new --log.logfmt directive and hides --log.json, replacing both with log.format=(json|logfmt|terminal). The hidden log.json option is still respected if log.format is not specified for backwards compatibility.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR fixes OOM panic in the callTracer as well as panicing on
opcode validation errors (e.g. stack underflow) in callTracer and
prestateTracer.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Fixes multiple data races caused by the fact that resolving fields are done concurrently by the graphql library. It also enforces caching at the stateobject level for account fields.
This change renames StateTrie methods to remove the Try* prefix.
We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change switches to use the smaller influxdata/influxdb1-client package instead of depending on the whole infuxdb package. The new smaller client is very similar to the influxdb-v2 client, which made it possible to refactor the two reporters to reuse code a lot more.
This PR adds counter metrics for the CPU system and the Geth process.
Currently the only metrics available for these items are gauges. Gauges are
fine when the consumer scrapes metrics data at the same interval as Geth
produces new values (every 3 seconds), but it is likely that most consumers
will not scrape that often. Intervals of 10, 15, or maybe even 30 seconds
are probably more common.
So the problem is, how does the consumer estimate what the CPU was doing in
between scrapes. With a counter, it's easy ... you just subtract two
successive values and divide by the time to get a nice, accurate average.
But with a gauge, you can't do that. A gauge reading is an instantaneous
picture of what was happening at that moment, but it gives you no idea
about what was going on between scrapes. Taking an average of values is
meaningless.
This changes the Trie interface to add the plain account address as a
parameter to all storage-related methods.
After the introduction of the TryAccount* functions, TryGet, TryUpdate and
TryDelete are now only meant to read an account's storage. In their current
form, they assume that an account storage is stored in a separate trie, and
that the hashing of the slot is independent of its account's address.
The proposed structure for a stateless storage breaks these two
assumptions: the hashing of a slot key requires the address and all slots
and accounts are stored in a single trie.
This PR therefore adds an address parameter to the interface. It is ignored
in the MPT version, so this change has no functional impact, however it
will reduce the diff size when merging verkle trees.
With #25287 we made it so that preimages were not recorded by default. This had the side effect that the evm command is no longer able to dump state since it does a preimage lookup to determine the address represented by a key.
This change enables the recording of preimages when the dump command is given.
The meter for "for measuring the effective amount of data read" within the freezertable was never updated. This change remedies that.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
When interacting with geth as a library to e.g. produce state tests, it is desirable to obtain the consensus-correct jumptable definition for a given fork. This changes adds accessors so the instructionset can be obtained and characteristics about opcodes can be inspected.
This adds built-in support in package rlp for encoding, decoding and generating code dealing with uint256.Int.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Makes clear the distinction between Finalize and FinalizedAndAssemble:
- In Finalize function, a series of state operations are applied according to consensus rules. The statedb is mutated and the root hash can be checked and compared afterwards.
This function should be used in block processing(receive afrom network and apply it locally) but not block generation.
- In FinalizeAndAssemble function, after applying state mutations, the block is also to be assembled with the latest
state root computed, updating the header.
This function should be used in block generation only.
This adds two new rules to the transaction pool:
- A future transaction can not evict a pending transaction.
- A transaction can not overspend available funds of a sender.
---
Co-authored-by: dwn1998 <42262393+dwn1998@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Here, the core.Message interface turns into a plain struct and
types.Message gets removed.
This is a breaking change to packages core and core/types. While we do
not promise API stability for package core, we do for core/types. An
exception can be made for types.Message, since it doesn't have any
purpose apart from invoking the state transition in package core.
types.Message was also marked deprecated by the same commit it
got added in, 4dca5d4db7 (November 2016).
The core.Message interface was added in December 2014, in commit
db494170dc, for the purpose of 'testing' state transitions. It's the
same change that made transaction struct fields private. Before that,
the state transition used *types.Transaction directly.
Over time, multiple implementations of the interface accrued across
different packages, since constructing a Message is required whenever
one wants to invoke the state transition. These implementations all
looked very similar, a struct with private fields exposing the fields
as accessor methods.
By changing Message into a struct with public fields we can remove all
these useless interface implementations. It will also hopefully
simplify future changes to the type with less updates to apply across
all of go-ethereum when a field is added to Message.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the test to match the comment description. Using timestampedConfig in this test case is incorrect, the comment says 'local is at Gray Glacier' and isn't aware of more forks.
Accept all primitive types in Solidity for EIP-712 from intN, uintN, intN[], uintN[] for N as 0 to 256 in multiples of 8
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This ensures the "withdrawals" field will always be present in responses
to getPayloadBodiesByRangeV1 and getPayloadBodiesByHashV1.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR mitigates an issue with Ledger's on-device RLP deserialization, see
https://github.com/LedgerHQ/app-ethereum/issues/409
Ledger's RLP deserialization code does not validate the length of the RLP list received,
and it may prematurely enter the signing flow when a APDU chunk boundary falls immediately
before the EIP-155 chain_id when deserializing a transaction. Since the chain_id is
uninitialized, it is 0 during this signing flow. This may cause the user to accidentally
sign the transaction with chain_id = 0. That signature would be returned from the device 1
packet earlier than expected by the communication loop. The device blocks the
second-to-last packet waiting for the signer flow, and then errors on the successive
packet (which contains the chain_id, zeroed r, and zeroed s)
Since the signature's early arrival causes successive errors during the communication
process, geth does not parse the improper signature produced by the device, and therefore
no improperly-signed transaction can be created. User funds are not at risk.
We mitigate by selecting the highest chunk size that leaves at least 4 bytes in the
final chunk.
Checks that Transaction.MarshalJSON and newRPCTransaction JSON output can be parsed by Transaction.UnmarshalJSON
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change fixes a flaw where, in certain scenarios, the block sealer did not accurately reset the remaining gas after failing to include an invalid transaction. Fixes#26791
This PR changes metrics collection to actually measure the time interval between collections, rather
than assume 3 seconds. I did some ad hoc profiling, and on slower hardware (eg, my Raspberry Pi 4)
I routinely saw intervals between 3.3 - 3.5 seconds, with some being as high as 4.5 seconds. This
will generally cause the CPU gauge readings to be too high, and in some cases can cause impossibly
large values for the CPU load metrics (eg. greater than 400 for a 4 core CPU).
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a race in TestNewPayloadOnInvalidTerminalBlock where setting the TTD raced with
the miner. Solution: set the TTD on the blockchain config not the genesis config.
Also fixes a race in CopyHeader which resulted in race reports all over the place.
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
This fixes an issue where the withdrawal index was not calculated correctly
for multiple withdrawals in a single block.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds support for a native call tracer with the Parity format, which outputs call frames
in a flat array. This tracer accepts the following options:
- `convertParityErrors: true` will convert error messages to match those of Parity
- `includePrecompiles: true` will report all calls to precompiles. The default
matches Parity's behavior where CALL and STATICCALLs to precompiles are excluded
Incompatibilities with Parity include:
- Parity removes the result object in case of failure. This behavior is maintained
with the exception of reverts. Revert output usually contains useful information,
i.e. Solidity revert reason.
- The `gasUsed` field accounts for intrinsic gas (e.g. 21000 for simple transfers)
and refunds unlike Parity
- Block rewards are not reported
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
The change fixes unmarshaling of JSON null results into json.RawMessage.
---------
Co-authored-by: Jason Yuan <jason.yuan@curvegrid.com>
Co-authored-by: Jason Yuan <jason.yuan869@gmail.com>
This PR contains a small portion of the full pbss PR, namely
Remove the tracer from trie (and comitter), and instead using an accessList.
Related changes to the Nodeset.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The method `GetPayloadBodiesByRangeV1` now returns "-38004: Too large request" error if the requested range is too large, according to spec
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR is a (superior) alternative to https://github.com/ethereum/go-ethereum/pull/26708, it handles deprecation, primarily two specific cases.
`rand.Seed` is typically used in two ways
- `rand.Seed(time.Now().UnixNano())` -- we seed it, just to be sure to get some random, and not always get the same thing on every run. This is not needed, with global seeding, so those are just removed.
- `rand.Seed(1)` this is typically done to ensure we have a stable test. If we rely on this, we need to fix up the tests to use a deterministic prng-source. A few occurrences like this has been replaced with a proper custom source.
`rand.Read` has been replaced by `crypto/rand`.`Read` in this PR.
This PR relaxes the block body ingress handling a bit: if block body withdrawals are missing (but expected to be empty), the body withdrawals are set to 'empty list' before being passed to upper layers.
This fixes an issue where a block passed from EthereumJS to geth was deemed invalid.
Our discovery crawler spits out a huge amount of logs, most of which is pretty non-interesting. This change moves the very verbose output to Debug, and adds a 8-second status log message giving the general idea about what's going on.
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
This is a breaking change in the tracing hooks API as well as semantics of the callTracer:
- CaptureEnter hook provided a nil value argument in case of DELEGATECALL. However to stay consistent with how delegate calls behave in EVM this hook is changed to pass in the value of the parent call.
- callTracer will return parent call's value for DELEGATECALL frames.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* common, core, eth, les, trie: make prque generic
* les/vflux/server: fixed issues in priorityPool
* common, core, eth, les, trie: make priority also generic in prque
* les/flowcontrol: add test case for priority accumulator overflow
* les/flowcontrol: avoid priority value overflow
* common/prque: use int priority in some tests
No need to convert to int64 when we can just change the type used by the
queue.
* common/prque: remove comment about int64 range
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
* log: allow tabs in log messages
This fixes a regression where panic reports in RPC handlers were quoted
because they contain tab characters.
* Update format.go
According to the spec the payloadID needs to be random or dependent on all arguments, to prevent two payloads from clashing. This change adds withdrawals into the payload derivation.
---------
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR moves core/beacon to beacon/engine so that beacon-chain related code has its own top level package which also can house the the beacon lightclient-code.
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change fixes a minor flaw in the check for ipc endpoint length. The max_path_size is the max path that an ipc endpoint can have, which is 208. However, that size concerns the null-terminated pathname, so we need to account for an extra null-character too.
This change improves reusability of the EVM struct. Two methods are added:
- SetBlockContext(...)
- SetTracer(...)
Other attributes like the TransactionContext and the StateDB can already be updated.
BlockContext and Tracer are partially not updateable right now. This change fixes it and
opens the potential to reuse an EVM struct in more ways.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth: cmd: deprecate personal namespace
* eth: cmd: move deprecation to node
* node: disable toml of enablepersonal
* node: disable personal on ipc as well
* Update node/node.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* console: error -> warn
* node: less roulette
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
The execution-apis specification says that the full list of withdrawals should always be returned when requesting a block over RPC:
378c4304f7/src/schemas/block.yaml (L90-L94)
This change adopts the expected behavior.
Adds blocktest subcommand to the evm command, which is very similar to statetest, but instead of loading a StateTest static test it loads a BlockchainTest from a json file and runs it.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
This change removes the Infura rinkeby bootnode as well as two deprecated ropsten bootnodes.
Co-authored-by: Nicola Cocchiaro <ncocchiaro@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes the API so that uint64 is used for fork timestamps.
It's a good choice because types.Header also uses uint64 for time.
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a regression where the flag did not accept values without
the 0x prefix anymore. What's worse, if an invalid value was passed,
the client would just log an INFO level message and continue.
This change introduces a breaking change to miner.etherbase is configured.
Previously, users did not need to explicitly set the etherbase address via flag, since 'first' local account was used as etherbase automatically. This change removes the "default first account" feature.
In Proof-of-stake world, the fee recipient address is provided by CL, and not configured in Geth any more - meaning that miner.etherbase is mostly for legacy networks(pow, clique networks etc).
Fixes#26505 where the console crashed when a property getter
raised an exception during autocompletion. I also noticed while fixing this
issue that autocomplete wasn't working for objects/fields with numbers in
them (most importantly web3.<tab><tab>) which is also now fixed.
This PR does a few things.
It fixes a shutdown-order flaw in the chainfreezer. Previously, the chain-freezer would shutdown the freezer backend first, and then signal for the loop to exit. This can lead to a scenario where the freezer tries to fsync closed files, which is an error-conditon that could lead to exit via log.Crit.
It also makes the printout more detailed when truncating 'dangling' items, by showing the exact number instead of approximate MB.
This PR also adds calls to fsync files before closing them, and also makes the `db inspect` command slightly more robust.
* internal/flags: use filepath.Clean instead of path.Clean
* internal/flags: fix windows pipe issue
* internal/flags: modify test for windows
* internal/flags: use backticks, fix test
This PR fixes an issue which might result in data lost in freezer.
Whenever mutation happens in freezer, all data will be written into head data file
and it will be rotated with a new one in case the size of file reaches the threshold.
Theoretically, the rotated old data file should be fsync'd to prevent data loss.
In freezer.Sync function, we only fsync: (1) index file (2) meta file and (3) head
data file. So this PR forcibly fsync the head data file if mutation happens in the
boundary of data file.
In legacy (pre-merge) sync mode, headers were contiguously downloaded from the network and when no more headers were available, we checked every few seconds whether there are 64 new blocks to move the pivot.
In beacon (post-merge) sync mode, we don't need to check for new skeleton headers non stop, since those re delivered one by one by the engine API. The missing code snippet from the header fetcher was to actually look at the latest head and move the pivot if it was more than 2*64-8 away. This PR adds the missing movement logic.
This makes non-JS tracers execute all block txs on a single goroutine.
In the previous implementation, we used to prepare every tx pre-state
on one goroutine, and then run the transactions again with tracing enabled.
Native tracers are usually faster, so it is faster overall to use their output as
the pre-state for tracing the next transaction.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR removes the notion of fakeStorage from the state objects, and instead, for any state modifications that are needed, it simply makes the changes.
* p2p/discover: add more packet information in logs
This adds more fields to discv5 packet logs. These can be useful when
debugging multi-packet interactions.
The FINDNODE message also gets an additional field, OpID for debugging
purposes. This field is not encoded onto the wire.
I'm also removing topic system related message types in this change.
These will come back in the future, where support for them will be
guarded by a config flag.
* p2p/discover/v5wire: rename 'Total' to 'RespCount'
The new name captures the meaning of this field better.
Alarm is a timer utility that simplifies code where a timer needs to be rescheduled over
and over. Doing this can be tricky with time.Timer or time.AfterFunc because the channel
requires draining in some cases.
Alarm is optimized for use cases where items are tracked in a heap according to their expiry
time, and a goroutine with a for/select loop wants to be woken up whenever the next item expires.
In this application, the timer needs to be rescheduled when an item is added or removed
from the heap. Using a timer naively, these updates will always require synchronization
with the global runtime timer datastructure to update the timer using Reset. Alarm avoids
this by tracking the next expiry time and only modifies the timer if it would need to fire earlier
than already scheduled.
As an example use, I have converted p2p.dialScheduler to use Alarm instead of AfterFunc.
This changes moves the tracking of "deleted in this block" out from snap-only domain, so that it happens regardless of whether the execution is snapshot-backed or trie-backed.
This changes the StorageTrie method to return an error when the trie
is not available. It used to return an 'empty trie' in this case, but that's
not possible anymore under PBSS.
This PR builds on #26299, but also updates the tests to the most recent version, which includes tests regarding TheMerge.
This change adds checks to the beacon consensus engine, making it more strict in validating the pre- and post-headers, and not relying on the caller to have already correctly sanitized the headers/blocks.
This PR implements resettable freezer by adding a ResettableFreezer wrapper.
The resettable freezer wraps the original freezer in a way that makes it possible to ensure atomic resets. Implementation wise, it relies on the os.Rename and os.RemoveAll to atomically delete the original freezer data and re-create a new one from scratch.
This PR fixes an error in trie commit. If the trie.root is nil, it can be two possible scenarios:
- The trie was empty, and no change happens
- The trie was non-empty and all nodes are dropped
For the latter one, we should collect the deletions and apply them into database(e.g. in PBSS).
This ensures that RPC method handlers will react to a timeout or
cancelled request soon after the event occurs.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds a check that the safetxhash that we sign corresponds to the one that is expected by the input. If it differs, it tries again with the configured chainid.
A comment suggests that contract creation happens if the recipient of a call is 0x00..00 ("zero address") but in fact the sender must be nil. The zero address is a regular valid address that is commonly used as a "burn" address.
Currently calling `debug_TraceTransaction` with a transaction hash that doesn't exist returns a confusing error: `genesis is not traceable`. This PR changes the behaviour to instead return an error message saying `transaction not found`
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds a new type event.FeedOf[T], which is like event.Feed but parameterized
over the channel element type. Performance is unchanged, and it still uses reflect. But
unlike Feed, the generic version doesn't need to type-check interface{} arguments.
All panic cases are gone from the API.
While investigating another issue, I found that all callers of collectLogs have the
complete block available. rawdb.ReadReceipts loads the block from the database,
so it is better to use ReadRawReceipts here, and derive the receipt information using
the block which is already in memory.
This PR makes it possible to modify the flush interval time via RPC. On one extreme, `0s`, it would act as an archive node. If set to `1h`, means that after one hour of effective block processing time, the trie would be flushed. If one block takes 200ms, this means that a flush would occur every `5*3600=18000` blocks -- however, if the memory size of the cached states grows too large, it will flush sooner.
Essentially, this makes it possible to configure the node to be more or less "archive:ish", and without restarting the node while reconfiguring it.
--syncTarget is a feature for development purpose in post-merge world. Previously
it's added into eth.Config. But it turns out that's a stupid idea.
- syncTarget is a block object, which is hard to be put in config file(large)
- syncTarget is just a dev feature, doesn't make too much sense to add it in config file
So I remove it from the eth config object. And it also fixes the #26328
This improves readability of function 'push'.
sort.Search(N, ...) will at most return N when no match, so ix should be compared
with N. The previous version would compare ix with N+1 in case an additional item
was appended. No bug resulted from this comparison, but it's not easy to understand
why.
Co-authored-by: Felix Lange <fjl@twurst.com>
The gcproc field tracks the amount of time spent processing blocks,
and is used to trigger a state flush to disk when a certain threshold is
reached. After the merge, single block insertion by CL is the most
common source of block processing time, but this time was not added
into gcproc.
Here we add special handling for sending an error response when the write timeout of the
HTTP server is just about to expire. This is surprisingly difficult to get right, since is
must be ensured that all output is fully flushed in time, which needs support from
multiple levels of the RPC handler stack:
The timeout response can't use chunked transfer-encoding because there is no way to write
the final terminating chunk. net/http writes it when the topmost handler returns, but the
timeout will already be over by the time that happens. We decided to disable chunked
encoding by setting content-length explicitly.
Gzip compression must also be disabled for timeout responses because we don't know the
true content-length before compressing all output, i.e. compression would reintroduce
chunked transfer-encoding.
The new flag allows configuring an explicit endpoint which is to be
announced in the DHT. This feature was originally developed for the
discv5 wormhole experiment (#25798), but it's useful in other contexts
as well.
This removes the 'time' field from logs, as well as from the tracer interface. This change makes the trace output deterministic. If a tracer needs the time they can measure it themselves. No need for evm to do this.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This changes the Pop method to assign the zero value before
reducing slice size. Doing so ensures the backing array does not
reference removed item values.
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.
As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
This PR should makes it easier to sign EIP-712 typed data via the accounts.Wallet API, by using the mimetype for typed data.
Co-authored-by: nasdf <keenan.nemetz@gmail.com>
While investigating #22374, I noticed that the Sync operation of the
freezer does not take the table lock. It also doesn't call sync for all files
if there is an error with one of them. I doubt this will fix anything, but
didn't want to drop the fix on the floor either.
This removes an RPC test which takes > 90s to execute, and updates the
internal/guide tests to use lighter scrypt parameters.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a way to specify HTTP headers per request.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
rpc: fix connection tracking in Server
When upgrading to mapset/v2 with generics, the set element type used in
rpc.Server had to be changed to *ServerCodec because ServerCodec is not
'comparable'. While the distinction is technically correct, we know all
possible ServerCodec types, and all of them are comparable. So just use
a map instead.
It seems there is no fully typed library implementation of an LRU cache.
So I wrote one. Method names are the same as github.com/hashicorp/golang-lru,
and the new type can be used as a drop-in replacement.
Two reasons to do this:
- It's much easier to understand what a cache is for when the types are right there.
- Performance: the new implementation is slightly faster and performs zero memory
allocations in Add when the cache is at capacity. Overall, memory usage of the cache
is much reduced because keys are values are no longer wrapped in interface.
This PR changes the pending tx subscription to return RPCTransaction types instead of normal Transaction objects. This will fix the inconsistencies with other tx returning API methods (i.e. getTransactionByHash), and also fill in the sender value for the tx.
co-authored by @s1na
This fixes a problem in the SizeConstrainedLRU. The SCLRU uses an underlying simple lru which is not thread safe.
During the Get operation, the recentness of the accessed item is updated, so it is not a pure read-operation. Therefore, the mutex we need is a full mutex, not RLock.
This PR changes the mutex to be a regular Mutex, instead of RWMutex, so a reviewer can at a glance see that all affected locations are fixed.
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.
runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.
The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.
Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.
The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.
We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
This adds an option to direct log output to a file. This feature has been
requested a lot. It's sometimes useful to have this available when running
geth in an environment that doesn't easily allow redirecting the output.
Notably, there is no support for log rotation with this change. The --log.file option
opens the file once on startup and then keeps writing to the file handle.
This can become an issue when external log rotation tools are involved, so it's
best not to use them with this option for now.
When the interpreter is configured to use extra-eips, this change makes it so that all the opcodes are deep-copied, to prevent accidental modification of the 'base' jumptable.
Closes: #26136
Co-authored-by: Martin Holst Swende <martin@swende.se>
PR #26082 added account listing to OnSignerStartup but did not consider the case where a user has a large number of accounts which would be annoying to display.
This PR updates showAccounts() so that if there are more than 20 accounts available the user sees the first 20 displayed in the console followed by: First 20 accounts listed (N more available).
Co-authored-by: Martin Holst Swende <martin@swende.se>
Many of the other types have a function to convert the type to a big.Int,
but Address was missing this function.
It is useful to be able to turn an Address into a big.Int when doing
EVM-like computations natively in Go. Sometimes a Solidity address
type is casted to a uint256 and having a Big method on the Address
type makes this easy.
Instead of using a limit of three nodes per message, we can pack more nodes
into each message based on ENR size. In my testing, this halves the number
of sent NODES messages, because ENR size is usually < 300 bytes.
This also adds RLP helper functions that compute the encoded size of
[]byte and string.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR now also includes a fix to the problem of mult-routines building blocks on the same input. This PR works as before with regards to stopping the work, but it just will not spin up a second routine if one is already building. So if the CL does N calls to FCU+buildblock, and N calls to GetPayload, only the first of each will do something, the other calls will be mostly no-ops.
This PR also adds printout of the payload id into the logs.
* apitypes: synchronize handling of types
* signer/core/apitypes: improve array check
* apitypes: add a test for big.Int -> int32
* signer/core/apitypes: Add a test for parsing addresses from [20]byte, []byte and string
* signer/core/apitypes: add some testcases
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes geth to read the eip1559 params from the chain config instead of the globals.
This way the parameters may be changed by forking the chain config code, without creating a large diff throughout the past and future usages of the parameters.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This commit adds support for two new commands to clef, making it possible to list accounts / wallets from the command-line-interface.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Noticed that lookupDistances for FINDNODE requests didn't consider 256 a valid
distance. This is actually part of the example in the comment above the
function, surprised that wasn't tested before.
In some cases, it is desirable to capture what is triggered by each trace, when using the `callTracer`. For example: call `USDT.transfer` will trigger a `Transfer(from, to, value)` event.
This PR adds the option to capture logs to the call tracer, by specifying `{"withLog": true}` in the tracerconfig.
Any logs belonging to failed/reverted call-scopes are removed from the output, to prevent interpretation mistakes.
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Inner call reverts will now return the reason similar to the top-level call. Separately, if top-level call is of type CREATE and it fails, its `to` field will now be cleared to `0x00...00` instead of being set to the created address.
This PR adds a parameter to startup, --synctarget. The synctarget flag is a developer-flag, that can be useful in some scenarios as a replacement for a CL node. It defines a fixed block sync target:
geth --syncmode=full --synctarget=./block_15816882.hex_rlp
The --synctarget is only made available during syncmode=full
* eth/tracers: fix gasUsed in call tracer
* fix js tracers gasUsed
* fix legacy prestate tracer
* fix restGas in test
* drop intrinsicGas field from js tracers
Currently, in order to chain together sequential valid t8n transitions the caller must manually calculate the block base fee. This PR adds support for the necessary parent fee market data to calculate the base fee for the current transition.
Concretely, env is extended to accept the following:
parentBaseFee
parentGasUsed
parentGasLimit
Example usage can be found in ./cmd/evm/testdata/25.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR ports a few changes from PBSS:
- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
The prestate tracer did not report accounts that existed at a given address prior to a contract being created at that address.
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR fixes a regression causing snapshots not to be generated in "geth --import" mode. It also fixes the geth export command to be truly readonly, and adds a new test for geth export.
In some cases, inner contract creation may not be successful, and an inner contract was not created. This PR fixes a crash that could occur when doing tracing in such situations.
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR removes some optimistic tests -- a'la "do something,
wait a while, and hope it has trickled through and continue" -- and
instead uses some introspection to ensure that prerequisites are met.
This changes the node setup to ignore datadir files
static-nodes.json
trusted-nodes.json
When these files are present, it an error will be printed to the log.
This adds a
* core/vm, tests: optimized modexp + fuzzer
* common/math: modexp optimizations
* core/vm: special case base 1 in big modexp
* core/vm: disable fastexp
This changes the error message for mismatching chain ID to show
the given and expected value. Callers expecting this error must be
changed to use errors.Is.
* usbwallet support Ledger Nano S Plus
* accounts/usbwallet: add definitions + ref to ledger docs
Co-authored-by: Martin Holst Swende <martin@swende.se>
This update resolves an issue where StringSliceFlag would not be
rendered correctly in help output + mention that -H can be used multiple times
Co-authored-by: Martin Holst Swende <martin@swende.se>
Prior to this change, f.begin (and possibly end) stay negative, leading to strange results later in the code. With this change, filters using "safe" and "finalized" block produce results consistent w/ the overall behavior of this RPC method.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethclient/gethclient: improve time-sensitive flaky test
* eth/catalyst: fix (?) flaky test
* core: stop blockchains in tests after use
* core: fix dangling blockchain instances
* core: rm whitespace
* eth/gasprice, eth/tracers, consensus/clique: stop dangling blockchains in tests
* all: address review concerns
* core: goimports
* eth/catalyst: fix another time-sensitive test
* consensus/clique: add snapshot test run function
* core: rename stop() to stopWithoutSaving()
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR introduces a new mechanism in chain tracer for preventing creating too many trace states.
The workflow of chain tracer can be divided into several parts:
- state creator generates trace state in a thread
- state tracer retrieves the trace state and applies the tracing on top in another thread
- state collector gathers all result from state tracer and stream to users
It's basically a producer-consumer model here, while if we imagine that the state producer generates states too fast, then it will lead to accumulate lots of unused states in memory. Even worse, in path-based state scheme it will only keep the latest 128 states in memory, and the newly generated state will invalidate the oldest one by marking it as stale.
The solution for fixing it is to limit the speed of state generation. If there are over 128 states un-consumed in memory, then the creation will be paused until the states are be consumed properly.
Backwards compatibility warning: The result will from now on omit empty fields instead
of including a zero value (e.g. no more `balance: '0x'`).
The prestateTracer will now take an option `diffMode: bool`. In this mode
the tracer will output the pre state and post data for the modified parts of state.
Read-only accesses will be completely omitted. Creations (be it account or slot)
will be signified by omission in the `pre` list and inclusion in `post`. Whereas
deletion (be it account or slot) will be signified by inclusion in `pre` and omission
in `post` list.
Signed-off-by: Delweng <delweng@gmail.com>
This PR makes it so that the snap server responds to trie heal requests when possible, even if the snapshot does not exist. The idea being that it might prolong the lifetime of a state root, so we don't have to pivot quite as often.
This PR makes it possible to set custom headers, in particular for two scenarios:
- geth attach
- geth commands which can use --remotedb, e..g geth db inspect
The ability to use custom headers is typically useful for connecting to cloud-apis, e.g. providing an infura- or alchemy key, or for that matter access-keys for environments behind cloudflare.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR reworks tx indexer a bit. Compared to the original version, one scenario is no longer handled - upgrading from legacy geth without indexer support.
The tx indexer was introduced in 2020 and have been present through hardforks, so it can be assumed that all Geth nodes have tx indexer already. So we can simplify the tx indexer logic a bit:
- If the tail flag is not present, it means node is just initialized may or may not with an ancient store attached. In this case all blocks are regarded as unindexed
- If the tail flag is present, it means blocks below tail are unindexed, blocks above tail are indexed
This change also address some weird cornercases that could make the indexer not work after a crash.
This fixes a bug where contract code would be overridden to empty code ("0x")
when the Code field of OverrideAccount was left nil. The change also cleans up
the encoding of overrides to only send necessary fields, and improves documentation.
Fixes#25615
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
The call tracer and prestate tracer store data JSON-encoded in memory. In order to support alternative encodings (specifically RLP), it's better to keep data a native format during tracing. This PR does marshalling at the end, using gencodec.
OBS!
This PR changes the call tracer result slightly:
- Order of type and value fields are changed (should not matter).
- Output fields are completely omitted when they're empty (no more output: "0x"). Previously, this was only _sometimes_ omitted (e.g. when call ended in a non-revert error) and otherwise 0x when the output was actually empty.
Some tests define an 'expectException' error but the tests runner does not check for conditions where this test value is filled (error expected) but in which no error is returned by the test runner.
An example of this scenario is GeneralStateTests/stTransactionTest/HighGasPrice.json, which expects a 'TR_NoFunds' error, but the test runner does not return any error.
Signed-off-by: meows <b5c6@protonmail.com>
`geth dumpgenesis` currently does not respect the content of the data directory. Instead, it outputs the genesis block created by command-line flags. This PR fixes it to read the genesis from the database, if the database already exists.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: pad memory slice on oob case
* eth/tracers/js: fix testfailure due to err msg capitalization
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
The abigen exclusion pattern, previously on the form "path:type", now supports wildcards. Examples "*:type" to exclude a named type in all files, or "/path/to/foo.sol:*" all types in foo.sol.
This changes the CI build to store the git commit and date into package
internal/version instead of package main. Doing this essentially merges our
two ways of tracking the go-ethereum version into a single place, achieving
two objectives:
- Bad block reports, which use version.Info(), will now have the git commit
information even when geth is built in an environment such as
launchpad.net where git access is unavailable.
- For geth builds created by `go build ./cmd/geth` (i.e. not using `go run
build/ci.go install`), git information stored by the go tool is now used
in the p2p node name as well as in `geth version` and `geth
version-check`.
Sometimes we get stuck on db compaction, and the CL re-issues the "same" command to us multiple times. Each request get stuck on the same place, in the middle of the handler.
This changes makes it so we do not reprocess the same payload, but instead detects it early.
core/blockchain: downgrade tx indexing and unindexing logs from info to debug
If a user has a finite tx lookup limit, they will see an "unindexing" info level log each time a block is imported. This information might help a user understand that they are removing the index each block and some txs may not be retrievable by hash, but overall it is generally more of a nuisance than a benefit. This change downgrades the log to a debug log.
This shortens the chain config summary in bad block reports,
and adds go-ethereum version information as well.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes eth_getProof and eth_getStorageAt return an error when
the argument contains invalid hex in storage keys.
Co-authored-by: Felix Lange <fjl@twurst.com>
* cmd/geth: add a verkle subcommand
* fix copyright year
* remove unused command parameters
* check that the output file was successfully written to
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/geth: goimports fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
This PR makes the event-sending for deleted and new logs happen in batches, to prevent OOM situation due to large reorgs.
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the error code returned by the RPC server in certain situations:
- handler panic: code -32603
- result marshaling error: code -32603
- attempt to subscribe via HTTP: code -32001
In all of the above cases, the server previously returned the default error
code -32000.
Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The p2p msgrate tracker is a thing which tries to estimate some mean round-trip times. However, it did so in a very curious way: if a node had 200 peers, it would sort their 200 respective rtt estimates, and then it would pick item number 2 as the mean. So effectively taking third fastest and calling it mean. This probably works "ok" when the number of peers are low (there are other factors too, such as ttlScaling which takes some of the edge off this) -- however when the number of peers is high, it becomes very skewed.
This PR instead bases the 'mean' on the square root of the length of the list. Still pretty harsh, but a bit more lenient.
This PR includes minor updates to comments in trie/committer that reference insertion to the db, and adds an err != nil check for the return value of preimages.commit.
This PR simplifies the logic of chain tracer and also adds the unit tests.
The most important change has been made in this PR is the state management. Whenever a tracing state is acquired there is a corresponding release function be returned as well. It must be called once the state is used up, otherwise resource leaking can happen.
And also the logic of state management has been simplified a lot. Specifically, the state provider(eth backend, les backend) should ensure the state is available and referenced. State customers can use the state according to their own needs, or build other states based on the given state. But once the release function is called, there is no guarantee of the availability of the state.
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
The JSON-RPC spec requires the "version" field to be exactly "2.0",
so we should verify that. This change is not backwards-compatible with
sloppy client implementations, but I decided to go ahead with it anyway
because the failure will be caught via the returned error.
This adds a generic mechanism for 'dial options' in the RPC client,
and also implements a specific dial option for the JWT authentication
mechanism used by the engine API. Some real tests for the server-side
authentication handling are also added.
Co-authored-by: Joshua Gutow <jgutow@optimism.io>
Co-authored-by: Felix Lange <fjl@twurst.com>
It's a trivial PR to hide the error log when the trie node is not found in the database. The idea for this change is for all TryXXX functions, the error is already returned and we don't need to fire a log explicitly.
Recently there are a few tickets #25613#25589 reporting that the trie nodes are missing because of debug.SetHead. The root cause is after resetting, the chain rewinds to a historical point and re-imports the blocks on top.
Since the node is already synced and started to accept transactions previously, these transactions are still kept in the txpool and verified by txpool with a live state. This live state is constructed based on the live trie database, which is changed fast by node referencing and de-referencing.
Unfortunately, when we construct a live state(like the state in txpool), we don't reference the state we have. The blockchain will garbage collect the intermediate version nodes in another thread which leads the broken live state.
The best solution for this is to forcibly obtain a reference for all live states we create and call release function once it's used up. But it might end up with more junks persisted into disk. Will try to find an elegant solution later in the following PR.
Some small fixes to get the existing debug methods to conform to the spec. Mainly dropping the encoding information from the method name as it should be deduced from the debug context and allowing the method to be invoked by either block number or block hash. It also adds the method debug_getTransaction which returns the raw tx bytes by tx hash. This is pretty much equivalent to the eth_getRawTransactionByHash method.
* eth/catalyst: warn less frequently if no beacon client is available
* eth/catalyst: tweak warning frequency a bit
* eth/catalyst: some more tweaks
* Update api.go
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/fetcher: introduce some lag in tx fetching
* eth/fetcher: change conditions a bit
* eth/fetcher: use per-batch quota check
* eth/fetcher: fix some comments
* eth/fetcher: address review concerns
* eth/fetcher: fix panic + add warn log
* eth/fetcher: fix log
* eth/fetcher: fix log
* cmd/devp2p/internal/ethtest: fix ignorign tx announcements from prev. tests
* cmd/devp2p/internal/ethtest: fix TestLargeTxRequest
This increases the number of tx relay messages the test waits for. Since
go-ethereum now processes incoming txs in smaller batches, the
announcement messages it sends are also smaller.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a cache for block logs which is shared by all filters. The cache
size of is configurable using the `--cache.blocklogs` flag.
Co-authored-by: Felix Lange <fjl@twurst.com>
This avoids copying the input []byte while decoding trie nodes. In most
cases, particularly when the input slice is provided by the underlying
database, this optimization is safe to use.
For cases where the origin of the input slice is unclear, the copying version
is retained. The new code performs better even when the input must be
copied, because it is now only copied once in decodeNode.
* core, trie: flush preimages to db on database close
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* rename Close to CommitPreimages for clarity
* core, trie: nitpick fixes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This adds support for building statically-linked executables using ci.go.
Static linking is enabled by default in Docker builds, making it possible to
use the geth executable in any Docker image, regardless of the Linux
distribution the Dockerfile is based on.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR allows users to pass in a config object directly to the tracers. Previously only the struct logger was configurable.
It also adds an option to the call tracer which if enabled makes it ignore any subcall and collect only information about the top-level call. See #25419 for discussion.
The tracers will silently ignore if they are passed a config they don't care about.
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Because the goal of eth_createAccessList is providing the caller with the largest-possible
access list, it's generally not important that the gas limit used by the tracer will match the usage
of the call exactly. Avoiding the gas estimation step is a performance improvement. As long as the
call does not branch based on gas limit, the returned access list will be accurate.
This change makes http.Server.ReadHeaderTimeout configurable separately
from ReadTimeout for RPC servers. The default is set to the same as
ReadTimeout, which in order to cause no change in existing deployments.
* eth: support bubbling up bad blocks from sync to the engine API
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* Update eth/catalyst/api.go
* eth/catalyst: when forgetting bad hashes, also forget descendants
* eth/catalyst: minor bad block tweaks for resilience
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
During RPC calls such as eth_call and eth_estimateGas, st.evm.Config.NoBaseFee is set
which allows the gas price to be below the base fee. This results the tip being negative,
and balance being subtracted from the coinbase instead of added to it, which results in a
potentially negative coinbase balance interestingly. This can't happen during normal chain
processing as outside of RPC calls the gas price is required to be at least the base fee,
as NoBaseFee is false.
This change prevents this behavior by disabling fee payment when the fee is not set.
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
* Remove locking in (*BlockChain).ExportN
Since ExportN is read-only, it shouldn't need the lock. (?)
* Add hash check to detect reorgs during export.
* fix check order
* Update blockchain.go
* Update blockchain.go
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* internal/ethapi: error if tx args includes chain id that doesn't match local
* internal/ethapi: simplify code a bit
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Adds a native tracer that returns that in case of failure returns the error message or the revert reason of a transaction.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change ensures the HTTP server will always terminate within
at most 5s, even when all connections are busy and do not become
idle.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/catalyst: disallow importing blocks via newPayload during snap sync
* eth/catalyst: make tests pass by using full sync only
* eth/catalysts: make the import delay a bit cleaner
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
This changes the []byte <-> Uint8Array conversion to use an
ArrayBuffer, avoiding inefficient copying of the slice data in Goja.
Co-authored-by: Felix Lange <fjl@twurst.com>
The oss-fuzz engine crashes due to stack overflow decoding a large nested
structure into a interface{}. This PR limits the size of the input data, so
should avoid such crashes.
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
Previously on Geth startup we just logged the chain config is a semi-json-y format. Whilst that worked while we had a handful of hard-forks defined, currently it's kind of unwieldy.
This PR converts that original data dump and converts it into a user friendly - alas multiline - log output.
* accounts/abi/bind: fix duplicate field names in the generated go struct #24627
* accounts, cmd/abigen: resolve name conflicts
* ci lint, accounts/abi: remove unused function overloadedArgName
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds support for block overrides when doing debug_traceCall.
- Previously, debug_traceCall against pending erroneously used a common.Hash{} stateroot when looking up the state, meaning that a totally empty state was used -- so it always failed,
- With this change, we reject executing debug_traceCall against pending.
- And we add ability to override all evm-visible header fields.
In #24028 we flagged a warning when finding legacy receipts in the freezer. This PR nudges users a bit more strongly by preventing geth from starting in this case until receipts have been migrated.
It also adds a flag --ignore-legacy-receipts which when present allows geth to start normally.
#23773 added a JS tracer which uses Goja as its engine. In this PR I remove the previous tracer which used duktape as well as remove the dependencies.
This PR also comes with 2 fixes in the Goja tracer and one small behavioural change:
I had handled errors in the native Go functions by panicing. My oversight was that Goja only handles panics with a Goja.Value as argument. The difference is panic(goja.Value) allows JS to catch the exception whereas Interrupt(error) doesn't.
There was a race in how I handled Stop.
Because of 1. some of the methods that simply return nil on error (like memory.slice) now throw an exception.
This adds a JS tracer runtime environment based on the Goja VM. The new
runtime replaces the duktape runtime, which will be removed soon.
Goja is implemented in Go and is faster for cases where the Go <-> JS
transition overhead dominates overall performance. It is faster because
duktape is written in C, and the transition cost includes the cost of using
cgo. Another reason for using Goja is that go-duktape is not maintained
anymore.
We expect the performace of JS tracing to be at least as good or better with
this change.
* eth/catalyst: build the execution payload async
* miner: added comment, added test case
* eth/catalyst: miner: move async block production to miner
* eth/catalyst, miner: support generate seal block async
* miner: rework GetSealingBlockAsync to use a passed channel
* miner: apply rjl's diff
* eth/catalyst: nitpicks
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* ethdb/remotedb, cmd: add support for remote (readonly) databases
* ethdb/remotedb: minor changes
* ethdb/remotedb: close the conn
* cmd, ethdb: add rpc accessor for ancient data
* internal/ethapi: license
* ethdb/remotedb: linter fixes
This PR adds db tooling (geth db check-state-content) to verify the integrity of trie nodes. It iterates through the 32-byte key space in the database, which is expected to contain RLP-encoded trie nodes, addressed by hash.
* eth/protocols/snap: don't include empty snapshot slot slice
This PR fixes the snapshot storage serving handler. In snap protocol
the response is capped by the response size. Server can cutdown the
response if the accumulated byte size exceeds the local hard limit.
It means we can meet a special scenario that there is no storage slot
included for a requested account, but we attach the proof for this
account by mistake.
So in the prover side, when it meets a empty storage response but with
a valid proof proves there are some more slots left in the trie, then
requestor will reject this response and disconnect with server.
In this PR, if there is no storage slot served for the requested account,
then no proof should be attached as well.
* eth/protocols/snap: loosen restrictions for flaky tests
* eth/catalyst: fix flaky test in catalyst
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.
This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
During mining, when a new head arrives and interrupts the block building, the block being built should not be commited (but discarded). Committing the interrupted block introduces unnecessary delay, and possibly causes miner to mine on the previous head, which could result in higher uncle rate.
* core: recover the state in SetChainHead if the head state is missing
* core: disable test logging
* core: address comment from martin
* core: improve log level in case state is recovered
* core, eth, les, light: rename SetChainHead to SetCanonical
This PR improves the docker build speed for repeated builds where go.mod and go.sum do no change, by placing the downloaded dependencies in a lower layer
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
This adds the ability to run --state.fork=Merged, and have post-merge rules apply. When doing so, it also requires the input env to contain currentRandom, and enforces the currentDifficulty to be omitted or zero.
This PR fixes up the example python clef wrapper. The poc is intended to demonstrate how to wite a UI for clef, and had severely bitrotted.
With these changes, it "works" in the sense that all the built-in tests triggers the intended python callbacks (no errors about method not found). It does not "work" in the sense that the wrapper can be used as an actual UI. It will auto-reject any signing requests, for example.
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
This change removes extraneous/unnecessary checks for equality
when comparing 2 accessList values A and B. Given that we validate that
their lengths of A and B are equal, if so and if every element in A is
in B, reflexively every element in B is already in A. If that weren't
the case and an element g existed in A but not in B, that would mean
that there is an extra element and hence a mathematical contradiction.
Fixes#24658
See ethereum/go-ethereum#24554 and btcsuite/btcd#1839
This is an attempt to resolve a Go module dependency issue that arises
when both 'github.com/btcsuite/btcd/btcec/v2' and the older, non-v2
btcd module are required as dependencies.
* eth/downloader: remove stale beacon headers as backfilling progresses
* eth/downloader: remove leftover from a previous design
* eth/downloader: do partial beacon cleanups if chain is large
* eth/downloader: linter != heart
This adds a tools.go file to import all command packages used for
go:generate. Doing so makes it possible to execute go-based code
generators using 'go run', locking in the tool version using go.mod.
Co-authored-by: Felix Lange <fjl@twurst.com>
If a pending contract call errors, return that error right away rather
than ignoring it to allow an error somewhere else. This is helpful for
callers to know if perhaps a call failed because of the context deadline
being expired. This change mirrors the behavior of non-pending contract
calls.
This PR fixes a few panics in the chain marker benchmarks. The root
cause for panic is in chain marker the genesis header/block is not
accessible, while it's expected to be obtained in tests. So this PR
avoids touching genesis header at all to avoid panic.
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but t.TempDir handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* eth/tracers: refactor traceTx to separate out struct logging
review fix
Update eth/tracers/api.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Mv ExecutionResult type to logger package
review fix
impl GetResult for StructLogger
make formatLogs private
confused exit and end..
account for intrinsicGas in structlogger, fix TraceCall test
Add Stop method to logger
Simplify traceTx
Fix test
rm logger from blockchain test
account for refund in structLogger
* use tx hooks in struct logger
* minor
* avoid executionResult in struct logger
* revert blockchain test changes
* core,eth: add empty tx logger hooks
* core,eth: add initial and remaining gas to tx hooks
* store tx gasLimit in js tracer
* use gasLimit to compute intrinsic cost for js tracer
* re-use rules in transitiondb
* rm logs
* rm logs
* Mv some fields from Start to TxStart
* simplify sender lookup in prestate tracer
* mv env to TxStart
* Revert "mv env to TxStart"
This reverts commit 656939634b9aff19f55a1cd167345faf8b1ec310.
* Revert "simplify sender lookup in prestate tracer"
This reverts commit ab65bce48007cab99e68232e7aac2fe008338d50.
* Revert "Mv some fields from Start to TxStart"
This reverts commit aa50d3d9b2559addc80df966111ef5fb5d0c1b6b.
* fix intrinsic gas for prestate tracer
* add comments
* refactor
* fix test case
* simplify consumedGas calc in prestate tracer
Trie tracer is an auxiliary tool to capture all deleted nodes
which can't be captured by trie.Committer. The deleted nodes
can be removed from the disk later.
This PR adds a ExtraAllowedPath field to Solidity and exposes two APIs: CompileSource and CompileFiles, which were hidden inside CompileSolidityString and CompileSolidity before.
* eth/catalyst: only apply block if we actually have the state
* add header to payload queue
* Update cmd/geth/dbcmd.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/geth: only check for presence of legacy receipts if developer mode is not enabled
* cmd/geth: degrade log level
* cmd/geth: fix format
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd,core: add simple legacy receipt converter
core/rawdb: use forEach in migrate
core/rawdb: batch reads in forEach
core/rawdb: make forEach anonymous fn
cmd/geth: check for legacy receipts on node startup
fix err msg
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix log
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix some review comments
add warning to cmd
drop isLegacy fn from migrateTable params
add test for windows rename
test replacing in windows case
* minor fix
* sanity check for tail-deletion
* add log before moving files around
* speed-up hack for mainnet
* fix mainnet check, use networkid instead
* check mainnet genesis
* review fixes
* resume previous migration attempt
* core/rawdb: lint fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/beacon: eth/catalyst: updated engine api to new version
* core: implement exchangeTransitionConfig
* core/beacon: prevRandao instead of Random
* eth/catalyst: Fix ExchangeTransitionConfig, add test
* eth/catalyst: stop external miners on TTD reached
* node: implement --authrpc.vhosts flag
* core: allow for config override on non-mainnet networks
* eth/catalyst: fix peters comments
* eth/catalyst: make stop remote sealer more explicit
* eth/catalyst: add log output
* cmd/utils: rename authrpc.host to authrpc.addr
* eth/catalyst: disable the disabling of the miner
* eth: core: remove notion of terminal pow block
* eth: les: more of peters nitpicks
This updates the no-cgo implementations in the crypto package to use
the github.com/btcsuite/btcd/btcec/v2 module instead of the older btcec
package that was part of the main github.com/btcsuite/btcd module.
name old time/op new time/op delta
EcrecoverSignature-32 198µs ± 0% 144µs ± 0% -27.11%
VerifySignature-32 177µs ± 0% 128µs ± 0% -27.44%
DecompressPubkey-32 20.9µs ± 0% 10.1µs ± 0% -51.51%
Use (*ModNScalar).IsOverHalfOrder instead of math/big.Int when checking
for malleable signatures.
* cmd, eth: Rename whitelist argument to peer.requiredblocks
* eth/ethconfig: document PeerRequiredBlocks better
* cmd/utils: rename new flag to --eth.requiredblocks
Co-authored-by: Felix Lange <fjl@twurst.com>
* trie: fix memory leak in trie iterator
In the trie iterator, live nodes are tracked in a stack while iterating.
Popped node states should be explictly set to nil in order to get
garbage-collected.
* trie: fix empty trie iterator
* eth/downloader: implement beacon sync
* eth/downloader: fix a crash if the beacon chain is reduced in length
* eth/downloader: fix beacon sync start/stop thrashing data race
* eth/downloader: use a non-nil pivot even in degenerate sync requests
* eth/downloader: don't touch internal state on beacon Head retrieval
* eth/downloader: fix spelling mistakes
* eth/downloader: fix some typos
* eth: integrate legacy/beacon sync switchover and UX
* eth: handle UX wise being stuck on post-merge TTD
* core, eth: integrate the beacon client with the beacon sync
* eth/catalyst: make some warning messages nicer
* eth/downloader: remove Ethereum 1&2 notions in favor of merge
* core/beacon, eth: clean up engine API returns a bit
* eth/downloader: add skeleton extension tests
* eth/catalyst: keep non-kiln spec, handle mining on ttd
* eth/downloader: add beacon header retrieval tests
* eth: fixed spelling, commented failing tests out
* eth/downloader: review fixes
* eth/downloader: drop peers failing to deliver beacon headers
* core/rawdb: track beacon sync data in db inspect
* eth: fix review concerns
* internal/web3ext: nit
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR adds the `NextBaseFeePerGas` to `Block` and `EffectiveTip` to `Transaction` to make it easier for clients to compute fee history themselves via graphql queries.
* core/rawdb, cmd, ethdb, eth: implement freezer tail deletion
* core/rawdb: address comments from martin and sina
* core/rawdb: fixes cornercase in tail deletion
* core/rawdb: separate metadata into a standalone file
* core/rawdb: remove unused code
* core/rawdb: add random test
* core/rawdb: polish code
* core/rawdb: fsync meta file before manipulating the index
* core/rawdb: fix typo
* core/rawdb: address comments
This change speeds up trie hashing and all other activities that require
RLP encoding of trie nodes by approximately 20%. The speedup is achieved by
avoiding reflection overhead during node encoding.
The interface type trie.node now contains a method 'encode' that works with
rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code.
trie.hasher, which is pooled to avoid allocations, now maintains an
EncoderBuffer. This means memory resources related to trie node encoding
are tied to the hasher pool.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth, cmd: allow FdLimit to be set in config/command line (#24148)
* eth/ethconfig: format code
* cmd, eth/ethconfig: simplify fdlimit arg, disallow toml
* cnd/utils: make fdlimit setting nicer on the logs
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* rpc, node: refactor request validation and add jwt validation
* node, rpc: fix error message, ignore engine api in RegisterAPIs
* node: make authenticated port configurable
* eth/catalyst: enable unauthenticated version of engine api
* node: rework obtainjwtsecret (backport later)
* cmd/geth: added auth port flag
* node: happy lint, happy life
* node: refactor authenticated api
Modifies the authentication mechanism to use default values
* node: trim spaces and newline away from secret
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This replaces the simple selector parser in signer/fourbyte with one that
can actually handle most types. The new parser is added in accounts/abi
to also make it useable elsewhere.
* go.mod: update azure-storage-blob-go
update Azure/azure-storage-blob-go from v0.7.0 to v0.14.0.
relation #24396.
* internal/build: fix for breaking changes of azure-storage-blob-go
fix for breaking changes of update Azure/azure-storage-blob-go from v0.7.0 to v0.14.0.
relation #24396.
* internal/build: switch azure sdk from Azure/azure-storage-blob-go to Azure/azure-sdk-for-go/sdk/storage/azblob.
* internal/build refactor appending BlobItems
* internal/build: fix azure blobstore client to include container id
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.
The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.
To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.
Benchmark results:
name old time/op new time/op delta
EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8)
EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8)
name old speed new speed delta
EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8)
EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal)
This change adds a code generator tool for creating EncodeRLP method
implementations. The generated methods will behave identically to the
reflect-based encoder, but run faster because there is no reflection overhead.
Package rlp now provides the EncoderBuffer type for incremental encoding. This
is used by generated code, but the new methods can also be useful for
hand-written encoders.
There is also experimental support for generating DecodeRLP, and some new
methods have been added to the existing Stream type to support this. Creating
decoders with rlpgen is not recommended at this time because the generated
methods create very poor error reporting.
More detail about package rlp changes:
* rlp: externalize struct field processing / validation
This adds a new package, rlp/internal/rlpstruct, in preparation for the
RLP encoder generator.
I think the struct field rules are subtle enough to warrant extracting
this into their own package, even though it means that a bunch of
adapter code is needed for converting to/from rlpstruct.Type.
* rlp: add more decoder methods (for rlpgen)
This adds new methods on rlp.Stream:
- Uint64, Uint32, Uint16, Uint8, BigInt
- ReadBytes for decoding into []byte
- MoreDataInList - useful for optional list elements
* rlp: expose encoder buffer (for rlpgen)
This exposes the internal encoder buffer type for use in EncodeRLP
implementations.
The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to
encBuffer. It is implemented this way to ensure the global encBuffer pool
is handled correctly.
This PR adds an addtional API called `NewBatchWithSize` for db
batcher. It turns out that leveldb batch memory allocation is
super inefficient. The main reason is the allocation step of
leveldb Batch is too small when the batch size is large. It can
take a few second to build a leveldb batch with 100MB size.
Luckily, leveldb also offers another API called MakeBatch which can
pre-allocate the memory area. So if the approximate size of batch is
known in advance, this API can be used in this case.
It's needed in new state scheme PR which needs to commit a batch of
trie nodes in a single batch. Implement the feature in a seperate PR.
This functionality is needed in new path-based storage scheme, but
can be implemented in a seperate PR though.
When an account is deleted, then all the storage slots should be
nuked out from the disk as well. In hash-based storage scheme they
are still left in the disk but in new scheme, they will be iterated
and marked as deleted.
But why the NodeBlob API is needed in this scenario? Because when
the node is marked deleted, the previous value is also required to
be recorded to construct the reverse diff.
text: 'SA1019:"golang.org/x/crypto/openpgp"is deprecated:this package is unmaintained except for security fixes.'
- path:core/vm/contracts.go
text: 'SA1019:"golang.org/x/crypto/ripemd160"is deprecated:RIPEMD-160 is a legacy hash and should not be used for new applications.'
- path:accounts/usbwallet/trezor.go
text: 'SA1019:"github.com/golang/protobuf/proto"is deprecated:Use the "google.golang.org/protobuf/proto" package instead.'
- path:accounts/usbwallet/trezor/
text: 'SA1019:"github.com/golang/protobuf/proto"is deprecated:Use the "google.golang.org/protobuf/proto" package instead.'
exclude:
- 'SA1019: event.TypeMux is deprecated:use Feed'
- 'SA1019:strings.Title is deprecated'
- 'SA1019: strings.Title has been deprecated since Go 1.18 and an alternative has been available since Go 1.0:The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead.'
- 'SA1029:should not use built-in type string as key for value'
CVE-2024-34478 # "CWE-754: Improper Check for Unusual or Exceptional Conditions." This vulnerability is BTC only, BSC does not have the issue.
CVE-2024-6104 # "CWE-532: Information Exposure Through Log Files" This is caused by the vulnerabilities go-retryablehttp@v0.7.4, it is only used in cmd devp2p, impact is limited. will upgrade to v0.7.7 later
- sed -i '.bak' 's/repo.join/!repo.join/g' $(dirname `gem which cocoapods`)/cocoapods/sources_manager.rb
- if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then git clone --depth=1 https://github.com/CocoaPods/Specs.git ~/.cocoapods/repos/master && pod setup --verbose; fi
- xctool -version
- xcrun simctl list
# Workaround for https://github.com/golang/go/issues/23749
* [\#2021](https://github.com/bnb-chain/bsc/pull/2021) feat: support separate trie database
* [\#2224](https://github.com/bnb-chain/bsc/pull/2224) feat: support MEV
### BUGFIX
* [\#2268](https://github.com/bnb-chain/bsc/pull/2268) fix: ensure EIP-4788 not supported with Parlia Engine
### Cancun Code Merge
#### 4844 related
[internal/ethapi: add support for blobs in eth_fillTransaction (#28839)](https://github.com/bnb-chain/bsc/commit/ac5aa672d3b85a1f74667a65a15398f072aa0b2a)
[internal/ethapi: fix defaults for blob fields (#29037)](https://github.com/bnb-chain/bsc/commit/b47cf8fe1de4f97ce38417d8136a58812734a7a9)
[ethereum, ethclient: add blob transaction fields in CallMsg (#28989)](https://github.com/bnb-chain/bsc/commit/9d537f543990d9013d73433dc58fd0e985d9b2b6)
[core/txpool/blobpool: post-crash cleanup and addition/removal metrics(#28914)](https://github.com/bnb-chain/bsc/commit/62affdc9c5ea6f1a73fde42ac5ee5c9795877f88)
[core/txpool/blobpool: update the blob db with corruption handling (#29001)](https://github.com/bnb-chain/bsc/commit/3c30de219f92120248b7b7aeeb2bef82305e9627)
[core/txpool, miner: speed up blob pool pending retrievals (#29008)](https://github.com/bnb-chain/bsc/commit/6fb0d0992bd4eb91faf1e081b3c4aa46adb0ef7d)
[core/txpool, eth, miner: retrieve plain and blob txs separately (#29026)](https://github.com/bnb-chain/bsc/commit/f4852b8ddc8bef962d34210a4f7774b95767e421)
[core/txpool: reject blob txs with blob fee cap below the minimum (#29081)](https://github.com/bnb-chain/bsc/commit/32d4d6e6160432be1cb9780a43253deda7708ced)
[core/txpool/blobpool: reduce default database cap for rollout (#29090)](https://github.com/bnb-chain/bsc/commit/63aaac81007ad46b208570c17cae78b7f60931d4)
#### Clean Ups
[cmd/devp2p, eth: drop support for eth/67 (#28956)](https://github.com/bnb-chain/bsc/commit/8a76a814a2b9e5b4c1a4c6de44cd702536104507)
[all: remove the dependency from trie to triedb (#28824)](https://github.com/bnb-chain/bsc/commit/fe91d476ba3e29316b6dc99b6efd4a571481d888)
#### Others
[eth, miner: fix enforcing the minimum miner tip (#28933)](https://github.com/bnb-chain/bsc/commit/16ce7bf50fa71c907d1dc6504ed32a9161e71351)
[tests: fix goroutine leak related to state snapshot generation (#28974)](https://github.com/bnb-chain/bsc/commit/8321fe2fda0b44d6df3750bcee28b8627525173b)
[internal/ethapi:fix zero rpc gas cap in eth_createAccessList (#28846)](https://github.com/bnb-chain/bsc/commit/b87b9b45331f87fb1da379c5f17a81ebc3738c6e)
[eth/tracers: Fix callTracer logs on onlyTopCall == true (#29068)](https://github.com/bnb-chain/bsc/commit/5a0f468f8cb15b939bd85445d33c614a36942a8e)
## v1.4.1
FEATURE
NA
BUGFIX
* [\#2258](https://github.com/bnb-chain/bsc/pull/2258) core: skip checking state root existence when do snapsync by fast node
[eth, rpc: add configurable option for wsMessageSizeLimit (#27801)](https://github.com/bnb-chain/bsc/commit/705a51e566bc9215975d08f27d23ddab7baa9dd7)
[api/bind: add CallOpts.BlockHash to allow calling contracts at a specific block hash (#28084)](https://github.com/bnb-chain/bsc/commit/b85c86022e130a76eeb588a5b36e97149b342188)
[internal/ethapi: eth_call block parameter is optional (#28165)](https://github.com/bnb-chain/bsc/commit/adb9b319c9c61f092755000bf0fc4b3349f5cbbc)
[cmd/utils: fix a startup issue on deleted chaindata but dangling ancients(#27989)](https://github.com/bnb-chain/bsc/commit/00fead91c4f58bc7f56f81512280d3120860989c)
[cmd/geth, internal/flags, go.mod: colorize cli help, support env vars(#28103)](https://github.com/bnb-chain/bsc/commit/d9fbb71d631d1ad0fb1846042e4c50ab893a6fbf)
[ethclient: use 'input', not 'data' as field for transaction input (#28078)](https://github.com/bnb-chain/bsc/commit/5cf53f51ac556cfff2aee9d405efd336298a3aca)
[ethclient: fix forwarding 1559 gas fields (#28462)](https://github.com/bnb-chain/bsc/commit/e91cdb49beb4b2a3872b5f2548bf2d6559e4f561)
[ethclient/simulated: implement new sim backend (#28202)](https://github.com/bnb-chain/bsc/commit/2d08c9900996b5e798f40a3cc6b47f4e51dc487d)
[ethclient: apply accessList field in toCallArg (#28832)](https://github.com/bnb-chain/bsc/commit/1c488298c807f4daa3cbe260efb88b81902a903d)
#### Tracer
[eth/tracers: add position field for callTracer logs (#28389)](https://github.com/bnb-chain/bsc/commit/b1cec853bef98acc750298b1c9b8165f2ac6ce5a)
[eth/tracers: tx-level state in debug_traceCall (#28460)](https://github.com/bnb-chain/bsc/commit/5fb8ebc9ecb226b84181420b9871c5f61cf4f77d)
#### Txpool
[eth/fetcher: allow underpriced transactions in after timeout (#28097)](https://github.com/bnb-chain/bsc/commit/2b7bc2c36b0d0efc83e62ba8e13723b943c4fa6e)
#### Sync
[eth/downloader: prevent pivot moves after state commit (#28126)](https://github.com/bnb-chain/bsc/commit/4fa3db49a1e485b8d110c87de6a44f460b45bb9a)
[core, eth/downloader: fix genesis state missing due to state sync (#28124)](https://github.com/bnb-chain/bsc/commit/c53b0fef2ab8e2a00257b898cad5174e6b73f5fc)
[core, accounts, eth, trie: handle genesis state missing (#28171)](https://github.com/bnb-chain/bsc/commit/73f5bcb75b562fcb3c109dd9c51f21956bc1f1f4)
[eth/protocols/snap: fix snap sync failure on empty storage range (#28306)](https://github.com/bnb-chain/bsc/commit/1cb3b6aee4a16ab8e684da63f3cfcd9b961c43af)
[core, eth, trie: filter out boundary nodes and remove dangling nodes in stacktrie (#28327)](https://github.com/bnb-chain/bsc/commit/ab04aeb855605de51dd1e933f45eb84ad877e715)
[core, core/rawdb, eth/sync: no tx indexing during snap sync (#28703)](https://github.com/bnb-chain/bsc/commit/78a3c32ef4deb7755e3367e183639b66242654f7)
#### PBSS Related
[core/rawdb: skip pathdb state inspection in hashdb mode (#28108)](https://github.com/bnb-chain/bsc/commit/8d38b1fe62950e8675795abf63b7c978415ab7ba)
[eth: abort on api operations not available in pbss-mode (#28104)](https://github.com/bnb-chain/bsc/commit/b9b99a12e5159c746ef04b7c8febc4db66817b72)
[trie: remove internal nodes between shortNode and child in path mode (#28163)](https://github.com/bnb-chain/bsc/commit/4773dcbc81aac9d330df29446283361f5a7062c7)
[core/rawdb: fsync the index file after each freezer write (#28483)](https://github.com/bnb-chain/bsc/commit/326fa00759d959061035becc9514660c24897053)
[trie: remove inconsistent trie nodes during sync in path mode (#28595)](https://github.com/bnb-chain/bsc/commit/e206d3f8975bd98cc86d14055dca40f996bacc60)
[core, cmd, trie: fix the condition of pathdb initialization (#28718)](https://github.com/bnb-chain/bsc/commit/cca94792a4112687ce23e7041b95ccc7f4bf6123)
[tests/fuzzers: update fuzzers to be based on go-native fuzzing (#28352)](https://github.com/bnb-chain/bsc/commit/d10a2f6ab727f79a0acff29c8147d54c5e4689ec)
[tests/fuzzers: move fuzzers into native packages (#28467)](https://github.com/bnb-chain/bsc/commit/2391fbc676d7464bd42e248155558a2bcd6ecf64)
[core/forkid: correctly compute forkid when timestamp fork is activated in genesis (#27895)](https://github.com/bnb-chain/bsc/commit/32fde3f838d604fdeb7a3ada4f8e02d78301b83d)
[core/vm/runtime: Add Random field to config (#28001)](https://github.com/bnb-chain/bsc/commit/0ba2d3cfa4e4a0a76cd457b8dc0f49bf1a79b723)
[core/rawdb: no need to run truncateFile for readonly mode (#28145)](https://github.com/bnb-chain/bsc/commit/545f4c5547178bc8bde6af08b3ccaf68ca27f2c0)
[core/bloombits: fix deadlock when matcher session hits an error (#28184)](https://github.com/bnb-chain/bsc/commit/c2cfe35f121cb88650b4d90c958bcc4214d0ce7f)
[core/state, tests: fix memory leak via fastcache (#28387)](https://github.com/bnb-chain/bsc/commit/c1d5a012ea4b824e902db14e47bf147d727c2657)
[internal/ethapi: compact db missing key starts with 0xff (#28207)](https://github.com/bnb-chain/bsc/commit/46c850a9411d7ff15c1a0342fe29f359e6c390ae)
[internal/ethapi: fix codehash lookup in eth_getProof (#28357)](https://github.com/bnb-chain/bsc/commit/8b99ad46027a455971ccf9bd1f425b9c58ec5855)
[eth: set networkID to chainID by default (#28250)](https://github.com/bnb-chain/bsc/commit/4d9f3cd5d751efccd501b08ab6cf38a83b5e2858)
[eth: fix potential hang in waitSnapExtension (#28744)](https://github.com/bnb-chain/bsc/commit/18e154eaa24d5f7a8b3c48983ad591e6c10963ca)
[trie/triedb/hashdb: take lock around access to dirties cache (#28542)](https://github.com/bnb-chain/bsc/commit/fa0df76f3cfd186a1f06f2b80aa5dbb89555b009)
[accounts: properly close managed wallets when closing manager (#28710)](https://github.com/bnb-chain/bsc/commit/d3452a22cc871306c62de52d19295914141863c0)
[event: fix Resubscribe deadlock when unsubscribing after inner sub ends (#28359)](https://github.com/bnb-chain/bsc/commit/ffc6a0f36edda396a8421cf7a3c0feb88be20d0b)
[all: replace log15 with slog (#28187)](https://github.com/bnb-chain/bsc/commit/28e73717016cdc9ebdb5fdb3474cfbd3bd2d2524)
## v1.3.11
BUGFIX
* [\#2288](https://github.com/bnb-chain/bsc/pull/2288) fix: add FeynmanFix upgrade for a testnet issue
## v1.3.10
FEATURE
* [\#2047](https://github.com/bnb-chain/bsc/pull/2047) feat: add new fork block and precompile contract for BEP294 and BEP299
## v1.3.9
FEATURE
* [\#2186](https://github.com/bnb-chain/bsc/pull/2186) log: support maxBackups in config.toml
BUGFIX
* [\#2160](https://github.com/bnb-chain/bsc/pull/2160) cmd: fix dump cli cannot work in path mode
* [\#2183](https://github.com/bnb-chain/bsc/pull/2183) p2p: resolved deadlock on p2p server shutdown
IMPROVEMENT
* [\#2177](https://github.com/bnb-chain/bsc/pull/0000) build(deps): bump github.com/quic-go/quic-go from 0.39.3 to 0.39.4
* [\#2185](https://github.com/bnb-chain/bsc/pull/2185) consensus/parlia: set nonce before evm run
* [\#2190](https://github.com/bnb-chain/bsc/pull/2190) fix(legacypool): deprecate already known error
* [\#2195](https://github.com/bnb-chain/bsc/pull/2195) eth/fetcher: downgrade state tx log
## v1.3.8
FEATURE
* [\#2074](https://github.com/bnb-chain/bsc/pull/2074) faucet: new faucet client
* [rpc: check that "version" is "2.0" in request objects (#25570)](https://github.com/bnb-chain/bsc/pull/1840/commits/38e002f4641c2779c897ccaca575ec5ddeee9254)
* [rpc: support injecting HTTP headers through context (#26023)](https://github.com/bnb-chain/bsc/pull/1840/commits/add337e0f7bad02f3cf535c66cd31f252b0b5c99)
* [rpc: websocket should respect the "HTTP_PROXY" by default (#27264)](https://github.com/bnb-chain/bsc/pull/1840/commits/73697529994e14996b7740730481e926d5ec3e40)
* [rpc: change BlockNumber constant values to match ethclient (#27219)](https://github.com/bnb-chain/bsc/pull/1840/commits/9231770811cda0473a7fa4e2bccc95bf62aae634)
* [eth: make debug_StorageRangeAt take a block hash or number (#27328)](https://github.com/bnb-chain/bsc/pull/1840/commits/d789c68b667e13eb5cefd19d09ae84f7d016df6a)
* [eth,core: add api debug_getTrieFlushInterval (#27303)](https://github.com/bnb-chain/bsc/pull/1840/commits/0783cb7d91ad7b3cdf72ac6c6edaec8318673eb6)
* [rpc: add limit for batch request items and response size (#26681)](https://github.com/bnb-chain/bsc/pull/1840/commits/f3314bb6df4c86e650f0e47cbb5a21ca0616ac11)
* [core/types: support yParity field in JSON transactions (#27744)](https://github.com/bnb-chain/bsc/pull/1840/commits/bb148dd342ba03ce40cf04295e193c94b9dda322)
* [eth/filters: send rpctransactions in pending-subscription (#26126)](https://github.com/bnb-chain/bsc/pull/1840/commits/8c5ce1107b3110c7cb735d8dfa91c9c701393c85)
#### Flag
* [cmd/geth: rename --whitelist to --eth.requiredblocks (#24505)](https://github.com/bnb-chain/bsc/pull/1840/commits/dbfd3972624c1d82db21f5dfceab8fde7a1eee0a)
* [cmd: migrate to urfave/cli/v2 (#24751)](https://github.com/bnb-chain/bsc/pull/1840/commits/52ed3570c483693fdd6667add7e3050520ad3ba2)
* [cmd/utils: print warning when --metrics.port set without --metrics.ad…](https://github.com/bnb-chain/bsc/pull/1840/commits/8846c07d044f30dca8cd0db91c6245f71f4b24fa)
* [cmd/devp2p: add --extaddr flag (#26312)](https://github.com/bnb-chain/bsc/pull/1840/commits/b44abf56a966016cbb651648ac2d7b6705e80b11)
* [core,eth: adddebug_setTrieFlushInterval to change trie flush frequ](https://github.com/bnb-chain/bsc/pull/1840/commits/711afbc7fd76f1f206429e26f9aa5bf98bc7b43d)
* [graphql: fee history fields (#24452)](https://github.com/bnb-chain/bsc/pull/1840/commits/57cec892536270fc6dafae01ded2c528ffa370e9)
* [graphql: add rawReceipt field to transaction type (#24738)](https://github.com/bnb-chain/bsc/pull/1840/commits/d73df893a6fc528e69506397322205bd9258b6fa)
* [graphql: add raw fields to block and tx (#24816)](https://github.com/bnb-chain/bsc/pull/1840/commits/29a6b6bcac170ca7f8fceb242eba45ff15df17a1)
* [graphql: return correct logs for tx (#25612)](https://github.com/bnb-chain/bsc/pull/1840/commits/d0dc349fd36bd79f94516c866251783641ed12f1)
* [ethereum, ethclient: add FeeHistory support (#25403)](https://github.com/bnb-chain/bsc/pull/1840/commits/9ad508018e4790da0c1c00ac355f206fca12ab7c)
* [eth/filters, ethclient/gethclient: add fullTx option to pending tx fi…](https://github.com/bnb-chain/bsc/pull/1840/commits/5b1a04b9c749d804b51159fe12246c56de8515c1)
* [ethclient: include withdrawals in ethclient block responses (#26778)](https://github.com/bnb-chain/bsc/pull/1840/commits/e1b98f49a5075694c5022f5ec74425e40da415dd)
#### Tracer
* [eth/tracers/js: drop duktape engine (#24934)](https://github.com/bnb-chain/bsc/pull/1840/commits/ba47d800b13058885288c38bd174babb38560c89)
* [eth/tracers: add support for block overrides in debug_traceCall (#24871)](https://github.com/bnb-chain/bsc/pull/1840/commits/d8a2305565b1f97c451f8595e0f65358d6842714)
* [eth/tracers: add onlyTopCall option to callTracer (#25430)](https://github.com/bnb-chain/bsc/pull/1840/commits/86de2e516e5a4a2bbe1d29b46a0f460fbdde8303)
* [eth/tracers: remove revertReasonTracer, add revert reason to callTracer](https://github.com/bnb-chain/bsc/pull/1840/commits/ff1f49245d641a7268ade38cf512bdc7b26f9b7c)
* [eth/tracers: add diffMode to prestateTracer (#25422)](https://github.com/bnb-chain/bsc/pull/1840/commits/5d52a35931bba10f438ce4f41410442dd9cd396c)
* [core/vm: set tracer-observable value of a delegatecall to match parent `value`](https://github.com/bnb-chain/bsc/pull/1840/commits/b0cd8c4a5c4f0f25011ed64235a3ea1280f03c51)
* [eth/tracers/native: set created address to nil in case of failure (#2…](https://github.com/bnb-chain/bsc/pull/1840/commits/41af42e97c9d62d303a883cc3c143f560867fa34)
* [eth/tracers: report correct gasLimit in call tracers (#27029)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b76eb3708626fbd2eb9c1b58d7b4eac6a5eec15)
* [eth/tracers: addtxHashfield on txTraceResult (#27183)](https://github.com/bnb-chain/bsc/pull/1840/commits/604e215d1bb070dff98fb76aa965064c74e3633f)
* [eth/tracers: add ReturnData in the tracer's response (#27704)](https://github.com/bnb-chain/bsc/pull/1840/commits/1e069cf8026a9f71b5f7e80959465e4b273d5806)
* [core/state/snapshot: detect and clean up dangling storage snapshot in generation](https://github.com/bnb-chain/bsc/pull/1840/commits/59ac229f87831bd74b4dc07d34f54137cca78095)
* [internal/ethapi: add db operations to api (#24739)](https://github.com/bnb-chain/bsc/pull/1840/commits/16701c51697e28986feebd122c6a491e4d9ac0e7)
* [cmd/geth: adddb check-state-contentto verify integrity of trie nodes (#24840)](https://github.com/bnb-chain/bsc/pull/1840/commits/e0a9752b965f243313f2c32a91d306600dc3863c)
* [ethdb/remotedb, cmd: add support for remote (readonly) databases](https://github.com/bnb-chain/bsc/pull/1840/commits/57192bd0dc545d921306f6a4d7566c0c70c764c5)
* [cmd/abigen: accept combined-json via stdin (#24960)](https://github.com/bnb-chain/bsc/pull/1840/commits/0287e1a7c00c1eaad1a99b4ea05d70f1ed685140)
* [cmd/geth: add a verkle subcommand (#25718)](https://github.com/bnb-chain/bsc/pull/1840/commits/9d717167aaf27a48d56ad9d1a2c36f90eba1cc13)
* [cmd/geth, cmd/utils: geth attach with custom headers (#25829)](https://github.com/bnb-chain/bsc/pull/1840/commits/ea26fc8a6c44ebb48223f991048f41b2ec0a6414)
* [core/rawdb: refactor db inspector for extending multiple ancient storage](https://github.com/bnb-chain/bsc/pull/1840/commits/60e30a940bbba2c0d26de040195a5ccdb14d8c10)
* [cmd/clef: add importraw feature to clef (#26058)](https://github.com/bnb-chain/bsc/pull/1840/commits/17744639dafc5a54f21e220660bd39d765a09051)
* [cmd/devp2p: add more nodekey commands (#26129)](https://github.com/bnb-chain/bsc/pull/1840/commits/913973436bb88b652faffc10d8f97e4c19722883)
* [internal/web3ext: fix eth_call stateOverrides in console (#26265)](https://github.com/bnb-chain/bsc/pull/1840/commits/1325fef1025b9feb3342308265b6d1399614be30)
* [cmd/evm: add blocktest subcommand to evm (#26526)](https://github.com/bnb-chain/bsc/pull/1840/commits/90f15a0230be34a292c5d0574ee7910ee44267de)
#### HardFork
* [params: define cancun and prague as timestamp based forks (#26481)](https://github.com/bnb-chain/bsc/pull/1840/commits/f3a005f176372ff291dfa7c02ee1c87d18e9c788)
* [all: tie timestamp based forks to the passage of London (#27279)](https://github.com/bnb-chain/bsc/pull/1840/commits/85a4b82b3373fc5f3fa8b7c68061c55b0db0e9b7)
* [core/vm: implement EIP-3860: Limit and meter initcode (#23847)](https://github.com/bnb-chain/bsc/pull/1840/commits/793f0f9ec860f6f51e0cec943a268c10863097c7)
* [PBBS(ready to activate)](https://github.com/ethereum/go-ethereum/commits?author=rjl493456442)
#### P2P
* [eth/fetcher: throttle peers which deliver many invalid transactions (…](https://github.com/bnb-chain/bsc/pull/1840/commits/7f2890a9be1f91368582479f171248b972b45ae3)
* [all: remove concept of public/private API definitions (#25053)](https://github.com/bnb-chain/bsc/pull/1840/commits/10dc5dce0871bf8c24bac41b04e47c3b9ad2b93e)
* [cmd/geth: drop geth js command (#25000)](https://github.com/bnb-chain/bsc/pull/1840/commits/f20a56926551ae91a349498f9ce97c8ee373d6bb)
* [node: drop support for static & trusted node list files (#25610)](https://github.com/bnb-chain/bsc/pull/1840/commits/3630cafb34f7c48b9cc78cf736309275cbd70f74)
* [core: drop legacy receipt types (#26225)](https://github.com/bnb-chain/bsc/pull/1840/commits/10347c6b54d5b28a2e71d9c4993e7f44b0a359c3)
* [cmd, eth, node: deprecate personal namespace (#26390)](https://github.com/bnb-chain/bsc/pull/1840/commits/d0a4989a8def7e6bad182d1513e8d4a093c1672d)
* [accounts, build, mobile: remove Andriod and iOS support](https://github.com/bnb-chain/bsc/pull/1840/commits/d9699c8238307d5c3081c12078f78527468d7dbc)
* [params: remove EIP150Hash from chainconfig (#27087)](https://github.com/bnb-chain/bsc/pull/1840/commits/5e4d726e2a05aee80a75e5f99fd699f220dd503e)
* [all: remove notion of trusted checkpoints in the post-merge world (#2…](https://github.com/bnb-chain/bsc/pull/1840/commits/1e556d220c3a40286dd90b37a08bb5fc659ee6ee)
* [all: remove ethash pow, only retain shims needed for consensus and te](https://github.com/bnb-chain/bsc/pull/1840/commits/dde2da0efb8e9a1812f470bc43254134cd1f8cc0)
* [les: remove obsolete code related to PoW header syncing (#27737)](https://github.com/bnb-chain/bsc/pull/1840/commits/d4d88f9bce13ca9310bf28f5f26ea9f1915ba90d)
* remove diffsync
#### Others
* [accounts/usbwallet: support Ledger Nano S Plus and FTS (#25933)](https://github.com/bnb-chain/bsc/pull/1840/commits/7eafbec741d124bc53896f6bfc2408b70ab9a82a)
* [accounts/scwallet: fix keycard data signing error (#25331)](https://github.com/bnb-chain/bsc/pull/1840/commits/0c66d971e7f3557df297cbe450fe7fc7826017be)
* [core/state: replace fastcache code cache with gc-friendly structure (…](https://github.com/bnb-chain/bsc/pull/1840/commits/5fded040372784985265f83f33f15cb6a51bebdb)
* [ci: disable coverage reporting in appveyor and travis](https://github.com/bnb-chain/bsc/pull/1840/commits/a0d63bc69a659009a3884f50c563a0e58483cdd0)
* [all: change chain head markers from block to header (#26777)](https://github.com/bnb-chain/bsc/pull/1840/commits/cd31f2dee2843776e485769ce85e0524716199bc)
* [core, miner: revert block gas counter in case of invalid transaction](https://github.com/bnb-chain/bsc/pull/1840/commits/77e33e5a49be99130a02dc72d6a0e4739fdd44d6)
* [signer/core: accept all solidity primitive types for EIP-712 signing](https://github.com/bnb-chain/bsc/pull/1840/commits/02796f6bee7e014fd16ad39f0bcd3b665b51e0bb)
* [internal/ethapi: make EstimateGas use[latest] block by default (#24363)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b66d47449f61e9ebaf9e1db3ed290b59844d4c1)
* [miner: suspend miner if node is syncing (#27218)](https://github.com/bnb-chain/bsc/pull/1840/commits/d4961881d7c92603f591f9cb8c705d00d8cbdfc0)
* [all: move main transaction pool into a subpool (#27463)](https://github.com/bnb-chain/bsc/pull/1840/commits/d40a255e973775575d8d16456252f93ac75c09f0)
* [core/txpool/blobpool: 4844 blob transaction pool (#26940)](https://github.com/bnb-chain/bsc/pull/1840/commits/1662228ac68325b4024e0cb6a4ce7dde27eb4c2d)
* [eth: send big transactions by announce/retrieve only (#27618)](https://github.com/bnb-chain/bsc/pull/1840/commits/f5d3d486e459dce29130576ae88f2324ad586b50)
* [core/rawdb: support freezer batch read with no size limit (#27687)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b1f97e151e8b34a0a0d528a3472e27de1d12a9c)
@@ -5,7 +5,7 @@ The goal of BNB Smart Chain is to bring programmability and interoperability to
BNB Smart Chain starts its development based on go-ethereum fork. So you may see many toolings, binaries and also docs are based on Ethereum ones, such as the name “geth”.
@@ -36,7 +36,7 @@ To combine DPoS and PoA for consensus, BNB Smart Chain implement a novel consens
2. Validators take turns to produce blocks in a PoA manner, similar to Ethereum's Clique consensus engine.
3. Validator set are elected in and out based on a staking based governance on BNB Beacon Chain.
4. The validator set change is relayed via a cross-chain communication mechanism.
5. Parlia consensus engine will interact with a set of [system contracts](https://docs.bnbchain.org/docs/learn/system-contract) to achieve liveness slash, revenue distributing and validator set renewing func.
5. Parlia consensus engine will interact with a set of [system contracts](https://docs.bnbchain.org/bnb-smart-chain/staking/overview/#system-contracts) to achieve liveness slash, revenue distributing and validator set renewing func.
### Light Client of BNB Beacon Chain
@@ -61,7 +61,7 @@ Many of the below are the same as or similar to go-ethereum.
For prerequisites and detailed build instructions please read the [Installation Instructions](https://geth.ethereum.org/docs/getting-started/installing-geth).
Building `geth` requires both a Go (version 1.18 or later) and a C compiler. You can install
Building `geth` requires both a Go (version 1.21 or later) and a C compiler (GCC 5 or higher). You can install
them using your favourite package manager. Once the dependencies are installed, run
```shell
@@ -74,6 +74,16 @@ or, to build the full suite of utilities:
make all
```
If you get such error when running the node with self built binary:
```shell
Caught SIGILL in blst_cgo_init, consult <blst>/bindinds/go/README.md.
```
please try to add the following environment variables and build again:
```shell
exportCGO_CFLAGS="-O -D__BLST_PORTABLE__"
exportCGO_CFLAGS_ALLOW="-O -D__BLST_PORTABLE__"
```
## Executables
The bsc project comes with several wrappers/executables found in the `cmd`
@@ -87,7 +97,7 @@ directory.
| `abigen` | Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://docs.soliditylang.org/en/develop/abi-spec.html) with expanded functionality if the contract bytecode is also available. However, it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://geth.ethereum.org/docs/dapp/native-bindings) page for details. |
| `bootnode` | Stripped down version of our Ethereum client implementation that only takes part in the network node discovery protocol, but does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks. |
| `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow isolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug run`). |
| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://eth.wiki/en/fundamentals/rlp)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user-friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://ethereum.org/en/developers/docs/data-structures-and-encoding/rlp)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user-friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
## Running `geth`
@@ -100,15 +110,15 @@ on how you can run your own `geth` instance.
The hardware must meet certain requirements to run a full node on mainnet:
- VPS running recent versions of Mac OS X, Linux, or Windows.
- IMPORTANT 2 TB of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 250 MB/S throughput, read latency <1ms. (if node is started with snap sync, it will need NVMe SSD)
- IMPORTANT 3 TB(Dec 2023) of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 500 MB/S throughput, read latency <1ms. (if node is started with snap sync, it will need NVMe SSD)
- 16 cores of CPU and 64 GB of memory (RAM)
- Suggest m5zn.3xlarge instance type on AWS, c2-standard-16 on Google cloud.
- Suggest m5zn.6xlarge or r7iz.4xlarge instance type on AWS, c2-standard-16 on Google cloud.
- A broadband Internet connection with upload/download speeds of 5 MB/S
The requirement for testnet:
- VPS running recent versions of Mac OS X, Linux, or Windows.
- 500G of storage for testnet.
- 4 cores of CPU and 8 gigabytes of memory (RAM).
- 4 cores of CPU and 16 gigabytes of memory (RAM).
@@ -161,11 +181,11 @@ This tool is optional and if you leave it out you can always attach to an alread
#### 7. More
More details about [running a node](https://docs.bnbchain.org/docs/validator/fullnode) and [becoming a validator](https://docs.bnbchain.org/docs/validator/create-val)
More details about [running a node](https://docs.bnbchain.org/bnb-smart-chain/developers/node_operators/full_node/) and [becoming a validator](https://docs.bnbchain.org/bnb-smart-chain/validator/create-val/)
*Note: Although there are some internal protective measures to prevent transactions from
crossing over between the main network and test network, you should make sure to always
use separate accounts for play-money and real-money. Unless you manually move
*Note: Although some internal protective measures prevent transactions from
crossing over between the main network and test network, you should always
use separate accounts for play and realmoney. Unless you manually move
accounts, `geth` will by default correctly separate the two networks and will not make any
accounts available between them.*
@@ -178,7 +198,7 @@ configuration file via:
$ geth --config /path/to/your_config.toml
```
To get an idea how the file should look like you can use the `dumpconfig` subcommand to
To get an idea of how the file should look like you can use the `dumpconfig` subcommand to
export your existing configuration:
```shell
@@ -230,9 +250,38 @@ APIs!**
- [BSC-Deploy](https://github.com/bnb-chain/node-deploy/): deploy tool for setting up both BNB Beacon Chain, BNB Smart Chain and the cross chain infrastructure between them.
- [BSC-Docker](https://github.com/bnb-chain/bsc-docker): deploy tool for setting up local BSC cluster in container.
## Running a bootnode
Bootnodes are super-lightweight nodes that are not behind a NAT and are running just discovery protocol. When you start up a node it should log your enode, which is a public identifier that others can use to connect to your node.
First the bootnode requires a key, which can be created with the following command, which will save a key to boot.key:
```
bootnode -genkey boot.key
```
This key can then be used to generate a bootnode as follows:
Please see [Releases](https://github.com/binance-chain/bsc/releases). We recommend using the [most recently released version](https://github.com/binance-chain/bsc/releases/latest).
Please see [Releases](https://github.com/bnb-chain/bsc/releases). We recommend using the [most recently released version](https://github.com/bnb-chain/bsc/releases/latest).
## Audit reports
Audit reports are published in the `docs` folder: https://github.com/binance-chain/bsc/tree/master/docs/audits
Audit reports are published in the `docs` folder: https://github.com/bnb-chain/bsc/tree/master/docs/audits
**Please do not file a public ticket** mentioning the vulnerability.
To find out how to disclose a vulnerability in Ethereum visit [https://bugcrowd.com/binance](https://bugcrowd.com/binance) or email bounty@ethereum.org. Please read the [disclosure page](https://github.com/binance-chain/bsc/security/advisories) for more information about publicly disclosed security vulnerabilities.
To find out how to disclose a vulnerability in Ethereum visit [https://bugcrowd.com/binance](https://bugcrowd.com/binance) or email bounty@ethereum.org. Please read the [disclosure page](https://github.com/bnb-chain/bsc/security/advisories) for more information about publicly disclosed security vulnerabilities.
Use the built-in `geth version-check` feature to check whether the software is affected by any known vulnerability. This command will fetch the latest [`vulnerabilities.json`](https://geth.ethereum.org/docs/vulnerabilities/vulnerabilities.json) file which contains known security vulnerabilities concerning `geth`, and cross-check the data against its own version number.
**WARNING: FOILLOWING THESE INSTRUCTIONS WILL DESTROY THE MASTER KEY ON YOUR CARD. ONLY PROCEED IF NO FUNDS ARE ASSOCIATED WITH THESE ACCOUNTS**
**WARNING: FOLLOWING THESE INSTRUCTIONS WILL DESTROY THE MASTER KEY ON YOUR CARD. ONLY PROCEED IF NO FUNDS ARE ASSOCIATED WITH THESE ACCOUNTS**
You can use status' [keycard-cli](https://github.com/status-im/keycard-cli) and you should get _at least_ version 2.1.1 of their [smartcard application](https://github.com/status-im/status-keycard/releases/download/2.2.1/keycard_v2.2.1.cap)
// SignerCount returns the number of signers in the aggregate signature.
func(s*SyncAggregate)SignerCount()int{
varcountint
for_,v:=ranges.Signers{
count+=bits.OnesCount8(v)
}
returncount
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
