This change addresses an issue in snap sync, specifically when the entire sync process can be halted due to an encountered empty storage range.
Currently, on the snap sync client side, the response to an empty (partial) storage range is discarded as a non-delivery. However, this response can be a valid response, when the particular range requested does not contain any slots.
For instance, consider a large contract where the entire key space is divided into 16 chunks, and there are no available slots in the last chunk [0xf] -> [end]. When the node receives a request for this particular range, the response includes:
The proof with origin [0xf]
A nil storage slot set
If we simply discard this response, the finalization of the last range will be skipped, halting the entire sync process indefinitely. The test case TestSyncWithUnevenStorage can reproduce the scenario described above.
In addition, this change also defines the common variables MaxAddress and MaxHash.
* cmd, core: resolve scheme from a read-write database
* cmd, core, eth: move the scheme check in the ethereum constructor
* cmd/geth: dump should in ro mode
* cmd: reverts
This change
- Removes the owner-notion from a stacktrie; the owner is only ever needed for comitting to the database, but the commit-function, the `writeFn` is provided by the caller, so the caller can just set the owner into the `writeFn` instead of having it passed through the stacktrie.
- Removes the `encoding.BinaryMarshaler`/`encoding.BinaryUnmarshaler` interface from stacktrie. We're not using it, and it is doubtful whether anyone downstream is either.
* eth: enforce announcement metadatas and drop peers violating the protocol
* eth/fetcher: relax eth/68 validation a bit for flakey clients
* tests/fuzzers/txfetcher: pull in suggestion from Marius
* eth/fetcher: add tests for peer dropping
* eth/fetcher: linter linter linter linter linter
This is a minor refactor in preparation of changes to range verifier. This PR contains no intentional functional changes but moves (and renames) the light.NodeSet
This PR will allow a previously underpriced transaction back in after a timeout
of 5 minutes. This will block most transaction spam but allow for transactions to
be re-broadcasted on networks with less transaction flow.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
* parlia: reject header with `WithdrawalsHash` before shanghai fork
* log: output chainconfig when start up
* eth: fix TestOptionMaxPeersPerIP failure of goroutine order
* fix: crash of highestVerifiedHeader
* fix: panic of blobpool
* fix: genesis set up
* 1. modify NewDatabaseWithNodeDB to upstream
2. fix race use of hasher in statedb
3. fix use wrong value when updateTrie
* fix dir legacypool
* fix dir blobpool
* fix dir vote
* remove diffsync related code
* fix core/state/snapshot
* disable pipeCommit for now
* fix applyTransaction for bloom setting
* CI: fast finality in gasprice test
* CI: diffFetcher was removed
* CI: downloader, remove beaconsync test
* CI: no beaconsync in downloader, remove a failed case
TestCheckpointChallenge was removed in:
https://github.com/ethereum/go-ethereum/pull/27147
since after merge, it is useless for ethereum, but might be useful for BSC.
disable the case right now, as it is not a big issue.
* CI: bsc protocol decHandlers
* CI: receipt Bloom process
* 1. skip CheckConfigForkOrder for non-parlia engine
2. all test cases in core work well now
cd core && go test ./... -v
* fix test cases in trie dir
* CI: no beaconsync in downloader, remove a failed case(redo)
* fix dir miner
* fix dir cmd/geth
* CI: filter test, BaseFee & Finality
* fix dir graphql
* remove diffStore
* fix ethclient
* fix TestRPCGetTransactionReceipt
* fix dir internal
* ut add dir ethstats and signer
* disable pipeCommit thoroughly; fix concurrent map iteration and map write in statedb
* CI: fix snap sync
it could be changed by mistake
* fix tests/Run to generate snapshot
* prepare for merge
* remove useless
* use common hasher in getDeletedStateObject, no race here
* an critical comment for state.Prepare
* do not copy nil accessList
* add omitempty tag for unused new fields of core.Genesis
* remove totalFees
* calculate fees before FinalizeAndAssemble
* revert interface Finalize of consensus
* do not double gas limit upon london block
* use Leveldb as default
* Revert "remove diffStore"
This reverts commit df343b137412b0beb25298a6ba9c3c19e47f20b1.
* Revert "remove diffsync related code"
This reverts commit 8d84b81feae5d794cb5d7fcfdb7f5f7da751941b.
* compile pass after revert
* remove diffsync
* fix dir eth/protocols/trust
* fix TestFastNode
* decHandlers for trust protocol
* keep persist diff in test
This PR makes EIP-4788 work in the engine API and miner. It also fixes some bugs related to
EIP-4844 block processing and mining. Changes in detail:
- Header.BeaconRoot has been renamed to ParentBeaconRoot.
- The engine API now implements forkchoiceUpdatedV3
- newPayloadV3 method has been updated with the parentBeaconBlockRoot parameter
- beacon root is now applied to new blocks in miner
- For EIP-4844, block creation now updates the blobGasUsed field of the header
ReadSkeletonHeader can return nil if the header is missing, so we should
not access fields on it. Note that calling .Hash() on a nil header is fine, so there
is no need to actually check for nil.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes the forkID calculation to ignore time-based forks that occurred before the
genesis block. It's supposed to be done this way because the spec says:
> If a chain is configured to start with a non-Frontier ruleset already in its genesis, that is NOT considered a fork.
This PR removes the newly added txpool.Transaction wrapper type, and instead adds a way
of keeping the blob sidecar within types.Transaction. It's better this way because most
code in go-ethereum does not care about blob transactions, and probably never will. This
will start mattering especially on the client side of RPC, where all APIs are based on
types.Transaction. Users need to be able to use the same signing flows they already
have.
However, since blobs are only allowed in some places but not others, we will now need to
add checks to avoid creating invalid blocks. I'm still trying to figure out the best place
to do some of these. The way I have it currently is as follows:
- In block validation (import), txs are verified not to have a blob sidecar.
- In miner, we strip off the sidecar when committing the transaction into the block.
- In TxPool validation, txs must have a sidecar to be added into the blobpool.
- Note there is a special case here: when transactions are re-added because of a chain
reorg, we cannot use the transactions gathered from the old chain blocks as-is,
because they will be missing their blobs. This was previously handled by storing the
blobs into the 'blobpool limbo'. The code has now changed to store the full
transaction in the limbo instead, but it might be confusing for code readers why we're
not simply adding the types.Transaction we already have.
Code changes summary:
- txpool.Transaction removed and all uses replaced by types.Transaction again
- blobpool now stores types.Transaction instead of defining its own blobTx format for storage
- the blobpool limbo now stores types.Transaction instead of storing only the blobs
- checks to validate the presence/absence of the blob sidecar added in certain critical places
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
This change removes a chainconfig parameter passed into rawdb.ReadLogs, which is not used nor needed.
It also modifies the filter loop slightly, avoiding a labeled break and instead using a method.
This change does not modify any behaviour.
* core/blobpool: implement txpool for blob txs
* core/txpool: track address reservations to notice any weird bugs
* core/txpool/blobpool: add support for in-memory operation for tests
* core/txpool/blobpool: fix heap updating after SetGasTip if account is evicted
* core/txpool/blobpool: fix eviction order if cheap leading txs are included
* core/txpool/blobpool: add note as to why the eviction fields are not inited in reinject
* go.mod: pull in inmem billy form upstream
* core/txpool/blobpool: fix review commens
* core/txpool/blobpool: make heap and heap test deterministic
* core/txpool/blobpool: luv u linter
* core/txpool: limit blob transactions to 16 per account
* core/txpool/blobpool: fix rebase errors
* core/txpool/blobpool: luv you linter
* go.mod: revert some strange crypto package dep updates
This is a spin-out from the EIP-4844 devnet branch, containing just the Engine API modifications
and nothing else. The newPayloadV3 endpoint won't really work in this version, but we need the
data structures for testing so I'd like to get this in early.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change removes PoW header syncing related code from LES and also deletes
duplicated packages les/catalyst, les/downloader and les/fetcher. These package copies
were created because people wanted to make changes in their eth/ counterparts, but weren't
able to adapt LES code to the API changes.
The struct logger (or opcode tracer) was missing the return data field even
if this was explicitly enabled by user via `"enableReturnData": true` in the config.
This PR fixes this issue.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
This change adds back the 'geth --dev' mode of operation, using a cl-mocker.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: lx <92799281+brilliant-lx@users.noreply.github.com>
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
* all: move main transaction pool into a subpool
* go.mod: remove superfluous updates
* core/txpool: review fixes, handle txs rejected by all subpools
* core/txpool: typos
We had to do this workaround because it wasn't possible to export typed arrays from
JS to []byte. This was added in dop251/goja@2352993, so we can use the better way now.
* core/txpool: abstraction prep work for secondary pools (blob pool)
* core/txpool: leave subpool concepts to a followup pr
* les: fix tests using hard coded errors
* core/txpool: use bitmaps instead of maps for tx type filtering
Drop the notions of uncles, and disables activities while syncing
- Disable activities (e.g. generate pending state) while node is syncing,
- Disable empty block submission (but empty block is still kept for payload building),
- Drop uncle notion since (ethash is already deprecated)
eth: make StorageRangeAt take a block hash or number
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change implements async log retrievals via feeding logs in channels, instead of returning slices. This is a first step to implement #15063.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
ethclient accepts certain negative block number values as specifiers for the "pending",
"safe" and "finalized" block. In case of "pending", the value accepted by ethclient (-1)
did not match rpc.PendingBlockNumber (-2).
This wasn't really a problem, but other values accepted by ethclient did match the
definitions in package rpc, and it's weird to have this one special case where they don't.
To fix it, we decided to change the values of the constants rather than changing ethclient.
The constant values are not otherwise significant. This is a breaking API change, but we
believe not a dangerous one.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Enable following Berlin EIPs on BSC and introduce HertzBlock
** EIP-2565: ModExp Gas Cost
** EIP-2929: Gas cost increases for state access opcodes
** EIP-2718: Typed Transaction Envelope
** EIP-2930: Optional access lists
This removes the 'time' field from logs, as well as from the tracer interface. This change makes the trace output deterministic. If a tracer needs the time they can measure it themselves. No need for evm to do this.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR modifies the interface for the results of `debug_traceBlock` and `debug_traceCall` by adding the `txHash`, allowing users to identify which transaction's trace result corresponds to.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Adds an optional config parameter to eth_call which allows users to override block context fields (same functionality that was added to traceCall in #24871)
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* all: remove notion of trusted checkpoints in the post-merge world
* light: remove unused function
* eth/ethconfig, les: remove unused config option
* les: make linter happy
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
The EIP150Hash was an idea where, after the fork, we hardcoded the forked hash as an extra defensive mechanism. It wasn't really used, since forks weren't contentious and for all the various testnets and private networks it's been a hassle to have around.
This change removes that config field.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
This reverts commit b4f1cdffd55b7e27facec4d250009ad440d8ac74.
Revert because previous hardfork (berlin, london) need to be merged
first to avoid dependencies risk
* core/vm: deepcopy jumptable when enabling extra eips
When the interpreter is configured to use extra-eips,
this change makes it so that all the opcodes are deep-copied,
to prevent accidental modification of the 'base' jumptable.
Original-auther: yihuang <huang@crypto.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/vm: implement EIP-3860: Limit and meter initcode
Implementation of https://eips.ethereum.org/EIPS/eip-3860, limit and meter initcode.
Most of this change takes the go-ethereum implementation as reference.
Original-author: Andrei Maiboroda <andrei@ethereum.org>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core/vm: implement EIP-3855: PUSH0 instruction
* core/vm: Implement PUSH0
* Move PUSH0 to enable3855
* Add method doc
Original-author: Alex Beregszaszi <alex@rtfs.hu>
* core/vm: enable EIP-3855 (PUSH0) in Boneh
Original-author: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
* core/vm: Fix issue of incorrect instructionSet used for jump_table
Also update the related test case for gas change.
* core/vm: fix test cases issues that not suitable for boneh
* core/vm: reuse ErrMaxInitCodeSizeExceeded as error message
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Use the new atomic types in package eth/tracers
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR removes the Debug field from vmconfig, making it so that if a tracer is set, debug=true is implied.
---------
Co-authored-by: 0xTylerHolmes <tyler@ethereum.org>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
This includes a semantic change to the `callTracer` as well as `flatCallTracer`.
The value of field `gas` in the **first** call frame will change as follows:
- It previously contained gas available after initial deductions (i.e. tx costs)
- It will now contain the full tx gasLimit value
Signed-off-by: jsvisa <delweng@gmail.com>
* core/state: Remove prefetcher on stop
* core/blockchain: Wait for `triedb.Dereference` in `writeBlockWithState`
* eth/protocols/diff: Fix index mismatch in `TestGetDiffLayers`
This PR fixes OOM panic in the callTracer as well as panicing on
opcode validation errors (e.g. stack underflow) in callTracer and
prestateTracer.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change renames StateTrie methods to remove the Try* prefix.
We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Here, the core.Message interface turns into a plain struct and
types.Message gets removed.
This is a breaking change to packages core and core/types. While we do
not promise API stability for package core, we do for core/types. An
exception can be made for types.Message, since it doesn't have any
purpose apart from invoking the state transition in package core.
types.Message was also marked deprecated by the same commit it
got added in, 4dca5d4db7 (November 2016).
The core.Message interface was added in December 2014, in commit
db494170dc, for the purpose of 'testing' state transitions. It's the
same change that made transaction struct fields private. Before that,
the state transition used *types.Transaction directly.
Over time, multiple implementations of the interface accrued across
different packages, since constructing a Message is required whenever
one wants to invoke the state transition. These implementations all
looked very similar, a struct with private fields exposing the fields
as accessor methods.
By changing Message into a struct with public fields we can remove all
these useless interface implementations. It will also hopefully
simplify future changes to the type with less updates to apply across
all of go-ethereum when a field is added to Message.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This ensures the "withdrawals" field will always be present in responses
to getPayloadBodiesByRangeV1 and getPayloadBodiesByHashV1.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a race in TestNewPayloadOnInvalidTerminalBlock where setting the TTD raced with
the miner. Solution: set the TTD on the blockchain config not the genesis config.
Also fixes a race in CopyHeader which resulted in race reports all over the place.
Adds support for a native call tracer with the Parity format, which outputs call frames
in a flat array. This tracer accepts the following options:
- `convertParityErrors: true` will convert error messages to match those of Parity
- `includePrecompiles: true` will report all calls to precompiles. The default
matches Parity's behavior where CALL and STATICCALLs to precompiles are excluded
Incompatibilities with Parity include:
- Parity removes the result object in case of failure. This behavior is maintained
with the exception of reverts. Revert output usually contains useful information,
i.e. Solidity revert reason.
- The `gasUsed` field accounts for intrinsic gas (e.g. 21000 for simple transfers)
and refunds unlike Parity
- Block rewards are not reported
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
The method `GetPayloadBodiesByRangeV1` now returns "-38004: Too large request" error if the requested range is too large, according to spec
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR relaxes the block body ingress handling a bit: if block body withdrawals are missing (but expected to be empty), the body withdrawals are set to 'empty list' before being passed to upper layers.
This fixes an issue where a block passed from EthereumJS to geth was deemed invalid.
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
This is a breaking change in the tracing hooks API as well as semantics of the callTracer:
- CaptureEnter hook provided a nil value argument in case of DELEGATECALL. However to stay consistent with how delegate calls behave in EVM this hook is changed to pass in the value of the parent call.
- callTracer will return parent call's value for DELEGATECALL frames.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* fix: p2p sync with lagging peer
no need to sync with lagging peer, which could make the local chain stalling as well.
* fix: do not drop lagging peer, will retry it later.
The lagging peer is probably already the best peer with largest total difficulty.
Shoule not remove it, since p2p is a bidirectional connection, drop it could make
the peer unable to sync with this peer as well.
And the lagging peer could catch up later, so keep it.
* p2p: add lagging field in Peer
lagging peer will be connected, but won't be used to sync.
the lagging flag can be clear once the Peer updates its latest block state.
* test: fix UT compile issue
* fix: lagging peer func rename
* test: fix a UT fail of download test
errStallingPeer is replaced by errLaggingPeer in this case
* fix: lagging issue in light mode
* test: add and resolve UT of lagging peer
* common, core, eth, les, trie: make prque generic
* les/vflux/server: fixed issues in priorityPool
* common, core, eth, les, trie: make priority also generic in prque
* les/flowcontrol: add test case for priority accumulator overflow
* les/flowcontrol: avoid priority value overflow
* common/prque: use int priority in some tests
No need to convert to int64 when we can just change the type used by the
queue.
* common/prque: remove comment about int64 range
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
According to the spec the payloadID needs to be random or dependent on all arguments, to prevent two payloads from clashing. This change adds withdrawals into the payload derivation.
---------
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR moves core/beacon to beacon/engine so that beacon-chain related code has its own top level package which also can house the the beacon lightclient-code.
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* dep: upgrade secp256k1 to use btcec/v2 v2.3.2 and update insecurity pkg
* build ci: upgrade go to 1.19 and golangci-lint to 1.50.1
* docs: fix format that does not follow the goimports
* dep: redirect github.com/bnb-chain/tendermint to v0.31.13
* ci: disable GOPROXY
