* cmd/geth: standardize the action of importing bls account
* rename bLSAccountPasswordFileFlag to blsAccountPasswordFileFlag
* use password of wallet as account password when new an account or importing a raw key
accountTrieCache and storageTrieCache were introduced in this PR:
https://github.com/bnb-chain/bsc/pull/257, which is to improve performance.
Actually the performance gain is quite limited, as there is already dirty
and clean cache for trie node.
And after big merge, these 2 cache can not be used when PBSS is enabled.
So remove these code to simplify the logic.
* go.mod: upgrade prysm and the indrect dependency
prysm from v4.0.2 to v4.0.8, and run go mod tidy
* ci: upgrade go version from 1.19 to 1.20
* go-version: upgrade from v1.19 to v1.20
there is some dependency on go v1.20, such as go-libp2p v0.27.8
and also run go mod tidy
* dependency: upgrade docker version for security
it is not a big issue, since docker is only used for test purpose.
* rand: update the usage of math/rand after golang v1.20
2 APIs of math/rand module were deprecated since golang v1.20.
that is: rand.Seed() and rand.Read(), refer: ettps://pkg.go.dev/math/rand
"rand.Seed(seed int64)" has been replaced by: "r := rand.New(rand.NewSource(seed int64))",
need to initialize it with an instance before use
"rand.Read()" has been replaced by "crypto/rand.Read()"
* readme: need golang v1.20+ to build bsc
This changes implements faster post-selfdestruct iteration of storage slots for deletion, by using snapshot-storage+stacktrie to recover the trienodes to be deleted. This mechanism is only implemented for path-based schema.
For hash-based schema, the entire post-selfdestruct storage iteration is skipped, with this change, since hash-based does not actually perform deletion anyway.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
When MatcherSession encounters an error, it attempts to close the session.
Closing waits for all goroutines to finish, including the 'distributor'.
However, the distributor will not exit until all requests have returned.
This patch fixes the issue by delivering the (empty) result to the distributor
before calling Close().
* parlia: reject header with `WithdrawalsHash` before shanghai fork
* log: output chainconfig when start up
* eth: fix TestOptionMaxPeersPerIP failure of goroutine order
* fix: crash of highestVerifiedHeader
* fix: panic of blobpool
* fix: genesis set up
* 1. modify NewDatabaseWithNodeDB to upstream
2. fix race use of hasher in statedb
3. fix use wrong value when updateTrie
* fix dir legacypool
* fix dir blobpool
* fix dir vote
* remove diffsync related code
* fix core/state/snapshot
* disable pipeCommit for now
* fix applyTransaction for bloom setting
* CI: fast finality in gasprice test
* CI: diffFetcher was removed
* CI: downloader, remove beaconsync test
* CI: no beaconsync in downloader, remove a failed case
TestCheckpointChallenge was removed in:
https://github.com/ethereum/go-ethereum/pull/27147
since after merge, it is useless for ethereum, but might be useful for BSC.
disable the case right now, as it is not a big issue.
* CI: bsc protocol decHandlers
* CI: receipt Bloom process
* 1. skip CheckConfigForkOrder for non-parlia engine
2. all test cases in core work well now
cd core && go test ./... -v
* fix test cases in trie dir
* CI: no beaconsync in downloader, remove a failed case(redo)
* fix dir miner
* fix dir cmd/geth
* CI: filter test, BaseFee & Finality
* fix dir graphql
* remove diffStore
* fix ethclient
* fix TestRPCGetTransactionReceipt
* fix dir internal
* ut add dir ethstats and signer
* disable pipeCommit thoroughly; fix concurrent map iteration and map write in statedb
* CI: fix snap sync
it could be changed by mistake
* fix tests/Run to generate snapshot
* prepare for merge
* remove useless
* use common hasher in getDeletedStateObject, no race here
* an critical comment for state.Prepare
* do not copy nil accessList
* add omitempty tag for unused new fields of core.Genesis
* remove totalFees
* calculate fees before FinalizeAndAssemble
* revert interface Finalize of consensus
* do not double gas limit upon london block
* use Leveldb as default
* Revert "remove diffStore"
This reverts commit df343b1374.
* Revert "remove diffsync related code"
This reverts commit 8d84b81fea.
* compile pass after revert
* remove diffsync
* fix dir eth/protocols/trust
* fix TestFastNode
* decHandlers for trust protocol
* keep persist diff in test
FastFinality puts more infor into the header.extra field to keep vote information.
For mainnet, on epoch height, it could be 1526 bytes, which was 517 bytes before.
So the hardcoded 700 bytes for header could be no longer enough, increase it by
2 times would be enough.
this bug could cause P2P sync failure for nodes that are lagging behind, since they
would request access of ancient db, and GetBlockHeaders could be failed.
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
Context: The UpdateContractCode method was introduced for the state storage commitment
schemes that include the whole code for their commitment computation. It must therefore be called
before the root hash is computed at the end of IntermediateRoot.
This should have no impact on the MPT since, in this context, the method is a no-op.
This adds support for the "yParity" field in transaction objects returned by RPC
APIs. We somehow forgot to add this field even though it has been in the spec for
a long time.
This change rearranges the accessor methods in block.go and fixes some minor issues with
the copy-on-write logic of block data. Fixed issues:
- Block.WithWithdrawals did not create a shallow copy of the block.
- Block.WithBody copied the header unnecessarily, and did not preserve withdrawals.
However, the bugs did not affect any code in go-ethereum because blocks are *always*
created using NewBlockWithHeader().WithBody().WithWithdrawals()
* all: implement path-based state scheme
* all: edits from review
* core/rawdb, trie/triedb/pathdb: review changes
* core, light, trie, eth, tests: reimplement pbss history
* core, trie/triedb/pathdb: track block number in state history
* trie/triedb/pathdb: add history documentation
* core, trie/triedb/pathdb: address comments from Peter's review
Important changes to list:
- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated
* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache
* core/rawdb: fix tests
* trie/triedb/pathdb: rename metrics, change clean cache key
* trie/triedb: manage the clean cache inside of disk layer
* trie/triedb/pathdb: move journal function
* trie/triedb/path: fix tests
* trie/triedb/pathdb: fix journal
* trie/triedb/pathdb: fix history
* trie/triedb/pathdb: try to fix tests on windows
* core, trie: address comments
* trie/triedb/pathdb: fix test issues
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Sets the `currentExcessBlobGas` from env, alternatively calculates it based on `parentExcessBlobGas` and `parentBlobGasUsed`. It then emits the `currentExcessBlobGas` and `currentBlobGasUsed` into the output, to be used as parent-values for a future iteration.
Closes#27785Closes#27783
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* tests: split up state test execution
* Revert "tests: split up state test execution"
This reverts commit 96017c248c85d24e93ad013a2bbe8b38c99327c0.
* build: bump test timeout to 20 minutes
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core/blobpool: implement txpool for blob txs
* core/txpool: track address reservations to notice any weird bugs
* core/txpool/blobpool: add support for in-memory operation for tests
* core/txpool/blobpool: fix heap updating after SetGasTip if account is evicted
* core/txpool/blobpool: fix eviction order if cheap leading txs are included
* core/txpool/blobpool: add note as to why the eviction fields are not inited in reinject
* go.mod: pull in inmem billy form upstream
* core/txpool/blobpool: fix review commens
* core/txpool/blobpool: make heap and heap test deterministic
* core/txpool/blobpool: luv u linter
* core/txpool: limit blob transactions to 16 per account
* core/txpool/blobpool: fix rebase errors
* core/txpool/blobpool: luv you linter
* go.mod: revert some strange crypto package dep updates
Hertz is equal to Berlin and London
- Plato will be enabled on BSC mainnet at height: 30720096
- Hertz will be enabled on BSC mainnet at height: 31302048
Hertz is for ecosystem compatibility, to enable some customized EIPs of Berlin&London on BSC
This is a spin-out from the EIP-4844 devnet branch, containing just the Engine API modifications
and nothing else. The newPayloadV3 endpoint won't really work in this version, but we need the
data structures for testing so I'd like to get this in early.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change removes PoW header syncing related code from LES and also deletes
duplicated packages les/catalyst, les/downloader and les/fetcher. These package copies
were created because people wanted to make changes in their eth/ counterparts, but weren't
able to adapt LES code to the API changes.
EIP-6780: SELFDESTRUCT only in same transaction
> SELFDESTRUCT will recover all funds to the caller but not delete the account, except when called in the same transaction as creation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This updates the reference tests to the latest version and also adds logic
to process EIP-4844 blob transactions into the state transition. We are now
passing most Cancun fork tests.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
It is usually best to set GOMAXPROCS to the number of available CPU cores. However, setting
it like that does not work well when the process is quota-limited to a certain number of CPUs.
The automaxprocs library configures GOMAXPROCS, taking such limits into account.
This changes the port mapping procedure such that, when the requested port is unavailable
an alternative port suggested by the router is used instead.
We now also repeatedly request the external IP from the router in order to catch any IP changes.
Co-authored-by: Felix Lange <fjl@twurst.com>
The struct logger (or opcode tracer) was missing the return data field even
if this was explicitly enabled by user via `"enableReturnData": true` in the config.
This PR fixes this issue.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change adds the ability to perform reads from freezer without size limitation. This can be useful in cases where callers are certain that out-of-memory will not happen (e.g. reading only a few elements).
The previous API was designed to behave both optimally and secure while servicing a request from a peer, whereas this change should _not_ be used when an untrusted peer can influence the query size.
This simplifies the code that initializes the discovery a bit, and
adds new flags for enabling/disabling discv4 and discv5 separately.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
Back before #27178 , we spun up a number of ethash verifiers to verify headers. So we also had tests to ensure that we were indeed able to abort verification even if we had multiple workers running.
With PR #27178, we removed the parallelism in verification, and these tests are now failing, since we now just sequentially fire away the results as fast as possible on one routine.
This change removes the (sometimes failing) tests
This removes text parsing in leveldb metrics collection code. All metrics
can now be accessed through the stats API provided by leveldb.
We also add new gauge-typed metrics that count the number of tables at each level.
---------
Co-authored-by: Exca-DK <Exca-DK@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
* core/state, light, les: make signature of ContractCode hash-independent
* push current state for feedback
* les: fix unit test
* core, les, light: fix les unittests
* core/state, trie, les, light: fix state iterator
* core, les: address comments
* les: fix lint
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: lx <92799281+brilliant-lx@users.noreply.github.com>
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Verkle trees store the code inside the trie. This PR changes the interface to pass the code, as well as the dirty flag to tell the trie package if the code is dirty and needs to be updated. This is a no-op for the MPT and the odr trie.
In all other UDPv4 methods, the deadline is checked first. It seems weird to me that ping is an exception. Deadline comparison is also less resource intensive.
Co-authored-by: Exca-DK <Exca-DK@users.noreply.github.com>
This changes the eth_getProof method implementation to re-encode the requested
storage keys, canonicalizing them in the response. For backwards-compatibility reasons,
go-ethereum accepts non-canonical hex keys. Accepting them is fine, but we should
not mirror invalid inputs into the output.
Closes#27306
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This is likely the culprit behind several data corruption issues, e.g. where data has been
written to the freezer, but the deletion from pebble does not go through due to process
crash.
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
* go.mod: update kzg libraries to use big-endian
* go.sum: ran go mod tidy
* core/testdata/precompiles: fix blob verification test
* core/testdata/precompiles: fix blob verification test
Package rpc uses cgo to find the maximum UNIX domain socket path
length. If exceeded, a warning is printed. This is the only use of cgo in this
package. It seems excessive to depend on cgo just for this warning, so
we now hard-code the usual limit for Linux instead.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ensures Reheap will be called even before the London fork activates.
Since Reheap would otherwise only be called through `SetBaseFee` after London,
the list would just keep growing if the fork was not enabled or not reached yet.
* all: move main transaction pool into a subpool
* go.mod: remove superfluous updates
* core/txpool: review fixes, handle txs rejected by all subpools
* core/txpool: typos
The logs in this function are pulled straight from disk in rawdb.ReadRawReceipts and
also modified in receipts.DeriveFields, so removing the copy should be fine.
We had to do this workaround because it wasn't possible to export typed arrays from
JS to []byte. This was added in dop251/goja@2352993, so we can use the better way now.
This adds two ways to check for subscription support. First, one can now check
whether the transport method (HTTP/WS/etc.) is capable of subscriptions using
the new Client.SupportsSubscriptions method.
Second, the error returned by Subscribe can now reliably be tested using this
pattern:
sub, err := client.Subscribe(...)
if errors.Is(err, rpc.ErrNotificationsUnsupported) {
// no subscription support
}
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds server-side limits for JSON-RPC batch requests. Before this change, batches
were limited only by processing time. The server would pick calls from the batch and
answer them until the response timeout occurred, then stop processing the remaining batch
items.
Here, we are adding two additional limits which can be configured:
- the 'item limit': batches can have at most N items
- the 'response size limit': batches can contain at most X response bytes
These limits are optional in package rpc. In Geth, we set a default limit of 1000 items
and 25MB response size.
When a batch goes over the limit, an error response is returned to the client. However,
doing this correctly isn't always possible. In JSON-RPC, only method calls with a valid
`id` can be responded to. Since batches may also contain non-call messages or
notifications, the best effort thing we can do to report an error with the batch itself is
reporting the limit violation as an error for the first method call in the batch. If a batch is
too large, but contains only notifications and responses, the error will be reported with
a null `id`.
The RPC client was also changed so it can deal with errors resulting from too large
batches. An older client connected to the server code in this PR could get stuck
until the request timeout occurred when the batch is too large. **Upgrading to a version
of the RPC client containing this change is strongly recommended to avoid timeout issues.**
For some weird reason, when writing the original client implementation, @fjl worked off of
the assumption that responses could be distributed across batches arbitrarily. So for a
batch request containing requests `[A B C]`, the server could respond with `[A B C]` but
also with `[A B] [C]` or even `[A] [B] [C]` and it wouldn't make a difference to the
client.
So in the implementation of BatchCallContext, the client waited for all requests in the
batch individually. If the server didn't respond to some of the requests in the batch, the
client would eventually just time out (if a context was used).
With the addition of batch limits into the server, we anticipate that people will hit this
kind of error way more often. To handle this properly, the client now waits for a single
response batch and expects it to contain all responses to the requests.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/txpool: abstraction prep work for secondary pools (blob pool)
* core/txpool: leave subpool concepts to a followup pr
* les: fix tests using hard coded errors
* core/txpool: use bitmaps instead of maps for tx type filtering
* cmd/evm: make evm blocktest output logs if so instructed
* Apply suggestions from code review
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This changes the journal logic to mark the state object dirty immediately when it
is reset.
We're mostly adding this change to appease the fuzzer. Marking it dirty immediately
makes no difference in practice because accounts will always be modified by EVM
right after creation.
Continuing with a series of PRs to make the Trie interface more generic, this PR moves
the RLP encoding of storage slots inside the StateTrie and light.Trie implementations,
as other types of tries don't use RLP.
* p2p/discover: remove ReadRandomNodes
Even though it's public, this method is not callable by code outside of
package p2p/discover because one can't get a valid instance of Table.
* p2p/discover: add Table.Nodes
* p2p/discover: make Table settings configurable
In unit tests and externally developed cmd/devp2p test runs, it can be
useful to tune the timer intervals used by Table.
Drop the notions of uncles, and disables activities while syncing
- Disable activities (e.g. generate pending state) while node is syncing,
- Disable empty block submission (but empty block is still kept for payload building),
- Drop uncle notion since (ethash is already deprecated)
Deserialize hex keys early to shortcut on invalid input, and re-use the account storageTrie for each proof for each proof in the account, preventing repeated deep-copying of the trie.
Closes#27308
--------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
eth: make StorageRangeAt take a block hash or number
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
RPC methods `eth_getHeaderBy*` returned a size value which was meant for internal
processes. Please instead use `size` field returned by `eth_getBlockBy*` if you're interested
in the RLP encoded storage size of the block.
Signed-off-by: jsvisa <delweng@gmail.com>
ethclient accepts certain negative block number values as specifiers for the "pending",
"safe" and "finalized" block. In case of "pending", the value accepted by ethclient (-1)
did not match rpc.PendingBlockNumber (-2).
This wasn't really a problem, but other values accepted by ethclient did match the
definitions in package rpc, and it's weird to have this one special case where they don't.
To fix it, we decided to change the values of the constants rather than changing ethclient.
The constant values are not otherwise significant. This is a breaking API change, but we
believe not a dangerous one.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
implements the ability to run several state-tests in one instance. By not providing a statetest path to the `evm statetest` command, the path(s) will instead be read from `stdin`.
Upgrades graphiql to v2.4.4. The interface has become much nicer, and there are extra features like tabs, history, dark mode etc.
This change also now uses golang embed to bundle the resources.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237)
This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests.
This changes the RPC server to ignore methods using *context.Context as parameter
and *error as return value type. Methods with such types would crash the server when
called.
This PR adds a staleness-check to AccountRLP, before checking the bloom-filter and potentially going directly into the disklayer.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* EstimateGas should use LatestBlockNumber by default
* graphql: default to use latest for gas estimation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cryto/kzg4844: pull in the C and Go libs for KZG cryptography
* go.mod: pull in the KZG libraries
* crypto/kzg4844: add basic becnhmarks for ballpark numbers
* cmd, crypto: integrate both CKZG and GoKZG all the time, add flag
* cmd/utils, crypto/kzg4844: run library init on startup
* crypto/kzg4844: make linter happy
* crypto/kzg4844: push missing file
* crypto/kzg4844: fully disable CKZG but leave in the sources
* build, crypto/kzg4844, internal: link CKZG by default and with portable mode
* crypto/kzg4844: drop verifying the trusted setup in gokzg
* internal/build: yolo until it works?
* cmd/utils: make flag description friendlier
Co-authored-by: Martin Holst Swende <martin@swende.se>
* crypto/ckzg: no need for double availability check
* build: tiny flag cleanup nitpick
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
When block import fails, the error displays the number of the first block past the import batch, not the number of the failing block. This change fixes this problem by identifying which blocks fails and reporting its number.
rename parameter
In this case, the naming of "extapi" might create some confusion. Although it represents an External Signer Backend, its name could be mistaken for an API. In reality, it is a backend instance used for communicating with external signers. A better naming choice could be "extBackend" or "externalBackend" to more accurately describe that it is a backend instance rather than an API.
This PR modifies the interface for the results of `debug_traceBlock` and `debug_traceCall` by adding the `txHash`, allowing users to identify which transaction's trace result corresponds to.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This adds logic to prepend 'M' or 'E' to Solidity identifiers when they would
otherwise violate Go identifier naming rules.
Closes#26972
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Adds an optional config parameter to eth_call which allows users to override block context fields (same functionality that was added to traceCall in #24871)
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes TALKREQ message processing to run the handler on separate goroutine,
instead of running on the main discv5 dispatcher goroutine. It's better this way because
it allows the handler to perform blocking actions.
I'm also adding a new method TalkRequestToID here. The method allows implementing
a request flow where one node A sends TALKREQ to another node B, and node B later
sends a TALKREQ back. With TalkRequestToID, node B does not need the ENR of A to
send its request.
Makes the `geth account ... ` commands usable even if a geth-process is already executing, since the account commands do not read the chaindata, it was not required for those to use the same locking mechanism.
---
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* trie: add node type common package
In trie/types package, a few node wrappers are defined, which will be used
in both trie package, trie/snap package, etc. Therefore, a standalone common
package is created to put these stuffs.
* trie: rename trie/types to trie/trienode
This is a breaking GraphQL API change. All numeric values are now encoded as
hex strings. The motivation for this change is matching JSON-RPC outputs more
closely.
Numbers in query parameters are accepted as both decimal integers and hex strings.
* all: remove notion of trusted checkpoints in the post-merge world
* light: remove unused function
* eth/ethconfig, les: remove unused config option
* les: make linter happy
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd/utils, node: switch to Pebble as the default db if none exists
* node: fall back to LevelDB on platforms not supporting Pebble
* core/rawdb, node: default to Pebble at the node level
* cmd/geth: fix some tests explicitly using leveldb
* ethdb/pebble: allow double closes, makes tests simpler
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
Follow-up to #26697, makes the crawler less verbose on route53-based scenarios.
It also changes the loglevel from debug to info on Updates, which are typically the root, and can be interesting to see.
The EIP150Hash was an idea where, after the fork, we hardcoded the forked hash as an extra defensive mechanism. It wasn't really used, since forks weren't contentious and for all the various testnets and private networks it's been a hassle to have around.
This change removes that config field.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
This PR unifies the error handling in miner.
Whenever an error occur while applying a transaction, the transaction should be regarded as invalid and all following transactions from the same sender not executable because of the nonce restriction. The only exception is the `nonceTooLow` error which is handled separately.
Prior to this change, it was possible that transactions are erroneously deemed as 'future' although they are in fact 'pending', causing them to be dropped due to 'future' not being allowed to replace 'pending'.
This change fixes that, by doing a more in-depth inspection of the queue.
Use the new atomic types in package eth/tracers
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Makes the float-gauges lock-free
name old time/op new time/op delta
CounterFloat64Parallel-8 1.45µs ±10% 0.85µs ± 6% -41.65% (p=0.008 n=5+5)
---------
Co-authored-by: Exca-DK <dev@DESKTOP-RI45P4J.localdomain>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR removes the Debug field from vmconfig, making it so that if a tracer is set, debug=true is implied.
---------
Co-authored-by: 0xTylerHolmes <tyler@ethereum.org>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
This includes a semantic change to the `callTracer` as well as `flatCallTracer`.
The value of field `gas` in the **first** call frame will change as follows:
- It previously contained gas available after initial deductions (i.e. tx costs)
- It will now contain the full tx gasLimit value
Signed-off-by: jsvisa <delweng@gmail.com>
Currently, most of transaction validation while holding the txpool mutex: one exception being an early-on signature check.
This PR changes that, so that we do all non-stateful checks before we entering the mutex area. This means they can be performed in parallel, and to enable that, certain fields have been made atomic bools and uint64.
This change enables log rotation, which can be activated using the flag --log.rotate. Additional parameters that can be given are:
- log.maxsize to set maximum size before files are rotated,
- log.maxbackups to set how many files are retailed,
- log.maxage to configure max age of rotated files,
- log.compress whether to compress rotated files
The way to configure location of the logfile(s) is left unchanged, via the `log.logfile` parameter.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Currently the t8n tool uses the same block number for the current block and its parent while calculating the base fee. This causes incorrect base fee calculation for the london fork block. This commit sets the parent block number to be one less than the current block number
Adds error handling for the case that UnpackLog or UnpackLogIntoMap is called with a log that has zero topics.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Removes the new --log.logfmt directive and hides --log.json, replacing both with log.format=(json|logfmt|terminal). The hidden log.json option is still respected if log.format is not specified for backwards compatibility.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR fixes OOM panic in the callTracer as well as panicing on
opcode validation errors (e.g. stack underflow) in callTracer and
prestateTracer.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Fixes multiple data races caused by the fact that resolving fields are done concurrently by the graphql library. It also enforces caching at the stateobject level for account fields.
This change renames StateTrie methods to remove the Try* prefix.
We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change switches to use the smaller influxdata/influxdb1-client package instead of depending on the whole infuxdb package. The new smaller client is very similar to the influxdb-v2 client, which made it possible to refactor the two reporters to reuse code a lot more.
This PR adds counter metrics for the CPU system and the Geth process.
Currently the only metrics available for these items are gauges. Gauges are
fine when the consumer scrapes metrics data at the same interval as Geth
produces new values (every 3 seconds), but it is likely that most consumers
will not scrape that often. Intervals of 10, 15, or maybe even 30 seconds
are probably more common.
So the problem is, how does the consumer estimate what the CPU was doing in
between scrapes. With a counter, it's easy ... you just subtract two
successive values and divide by the time to get a nice, accurate average.
But with a gauge, you can't do that. A gauge reading is an instantaneous
picture of what was happening at that moment, but it gives you no idea
about what was going on between scrapes. Taking an average of values is
meaningless.
This changes the Trie interface to add the plain account address as a
parameter to all storage-related methods.
After the introduction of the TryAccount* functions, TryGet, TryUpdate and
TryDelete are now only meant to read an account's storage. In their current
form, they assume that an account storage is stored in a separate trie, and
that the hashing of the slot is independent of its account's address.
The proposed structure for a stateless storage breaks these two
assumptions: the hashing of a slot key requires the address and all slots
and accounts are stored in a single trie.
This PR therefore adds an address parameter to the interface. It is ignored
in the MPT version, so this change has no functional impact, however it
will reduce the diff size when merging verkle trees.
With #25287 we made it so that preimages were not recorded by default. This had the side effect that the evm command is no longer able to dump state since it does a preimage lookup to determine the address represented by a key.
This change enables the recording of preimages when the dump command is given.
The meter for "for measuring the effective amount of data read" within the freezertable was never updated. This change remedies that.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
When interacting with geth as a library to e.g. produce state tests, it is desirable to obtain the consensus-correct jumptable definition for a given fork. This changes adds accessors so the instructionset can be obtained and characteristics about opcodes can be inspected.
This adds built-in support in package rlp for encoding, decoding and generating code dealing with uint256.Int.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Makes clear the distinction between Finalize and FinalizedAndAssemble:
- In Finalize function, a series of state operations are applied according to consensus rules. The statedb is mutated and the root hash can be checked and compared afterwards.
This function should be used in block processing(receive afrom network and apply it locally) but not block generation.
- In FinalizeAndAssemble function, after applying state mutations, the block is also to be assembled with the latest
state root computed, updating the header.
This function should be used in block generation only.
This adds two new rules to the transaction pool:
- A future transaction can not evict a pending transaction.
- A transaction can not overspend available funds of a sender.
---
Co-authored-by: dwn1998 <42262393+dwn1998@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Here, the core.Message interface turns into a plain struct and
types.Message gets removed.
This is a breaking change to packages core and core/types. While we do
not promise API stability for package core, we do for core/types. An
exception can be made for types.Message, since it doesn't have any
purpose apart from invoking the state transition in package core.
types.Message was also marked deprecated by the same commit it
got added in, 4dca5d4db7 (November 2016).
The core.Message interface was added in December 2014, in commit
db494170dc, for the purpose of 'testing' state transitions. It's the
same change that made transaction struct fields private. Before that,
the state transition used *types.Transaction directly.
Over time, multiple implementations of the interface accrued across
different packages, since constructing a Message is required whenever
one wants to invoke the state transition. These implementations all
looked very similar, a struct with private fields exposing the fields
as accessor methods.
By changing Message into a struct with public fields we can remove all
these useless interface implementations. It will also hopefully
simplify future changes to the type with less updates to apply across
all of go-ethereum when a field is added to Message.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the test to match the comment description. Using timestampedConfig in this test case is incorrect, the comment says 'local is at Gray Glacier' and isn't aware of more forks.
Accept all primitive types in Solidity for EIP-712 from intN, uintN, intN[], uintN[] for N as 0 to 256 in multiples of 8
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This ensures the "withdrawals" field will always be present in responses
to getPayloadBodiesByRangeV1 and getPayloadBodiesByHashV1.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR mitigates an issue with Ledger's on-device RLP deserialization, see
https://github.com/LedgerHQ/app-ethereum/issues/409
Ledger's RLP deserialization code does not validate the length of the RLP list received,
and it may prematurely enter the signing flow when a APDU chunk boundary falls immediately
before the EIP-155 chain_id when deserializing a transaction. Since the chain_id is
uninitialized, it is 0 during this signing flow. This may cause the user to accidentally
sign the transaction with chain_id = 0. That signature would be returned from the device 1
packet earlier than expected by the communication loop. The device blocks the
second-to-last packet waiting for the signer flow, and then errors on the successive
packet (which contains the chain_id, zeroed r, and zeroed s)
Since the signature's early arrival causes successive errors during the communication
process, geth does not parse the improper signature produced by the device, and therefore
no improperly-signed transaction can be created. User funds are not at risk.
We mitigate by selecting the highest chunk size that leaves at least 4 bytes in the
final chunk.
Checks that Transaction.MarshalJSON and newRPCTransaction JSON output can be parsed by Transaction.UnmarshalJSON
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change fixes a flaw where, in certain scenarios, the block sealer did not accurately reset the remaining gas after failing to include an invalid transaction. Fixes#26791
This PR changes metrics collection to actually measure the time interval between collections, rather
than assume 3 seconds. I did some ad hoc profiling, and on slower hardware (eg, my Raspberry Pi 4)
I routinely saw intervals between 3.3 - 3.5 seconds, with some being as high as 4.5 seconds. This
will generally cause the CPU gauge readings to be too high, and in some cases can cause impossibly
large values for the CPU load metrics (eg. greater than 400 for a 4 core CPU).
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a race in TestNewPayloadOnInvalidTerminalBlock where setting the TTD raced with
the miner. Solution: set the TTD on the blockchain config not the genesis config.
Also fixes a race in CopyHeader which resulted in race reports all over the place.
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
This fixes an issue where the withdrawal index was not calculated correctly
for multiple withdrawals in a single block.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds support for a native call tracer with the Parity format, which outputs call frames
in a flat array. This tracer accepts the following options:
- `convertParityErrors: true` will convert error messages to match those of Parity
- `includePrecompiles: true` will report all calls to precompiles. The default
matches Parity's behavior where CALL and STATICCALLs to precompiles are excluded
Incompatibilities with Parity include:
- Parity removes the result object in case of failure. This behavior is maintained
with the exception of reverts. Revert output usually contains useful information,
i.e. Solidity revert reason.
- The `gasUsed` field accounts for intrinsic gas (e.g. 21000 for simple transfers)
and refunds unlike Parity
- Block rewards are not reported
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
The change fixes unmarshaling of JSON null results into json.RawMessage.
---------
Co-authored-by: Jason Yuan <jason.yuan@curvegrid.com>
Co-authored-by: Jason Yuan <jason.yuan869@gmail.com>
This PR contains a small portion of the full pbss PR, namely
Remove the tracer from trie (and comitter), and instead using an accessList.
Related changes to the Nodeset.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The method `GetPayloadBodiesByRangeV1` now returns "-38004: Too large request" error if the requested range is too large, according to spec
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR is a (superior) alternative to https://github.com/ethereum/go-ethereum/pull/26708, it handles deprecation, primarily two specific cases.
`rand.Seed` is typically used in two ways
- `rand.Seed(time.Now().UnixNano())` -- we seed it, just to be sure to get some random, and not always get the same thing on every run. This is not needed, with global seeding, so those are just removed.
- `rand.Seed(1)` this is typically done to ensure we have a stable test. If we rely on this, we need to fix up the tests to use a deterministic prng-source. A few occurrences like this has been replaced with a proper custom source.
`rand.Read` has been replaced by `crypto/rand`.`Read` in this PR.
This PR relaxes the block body ingress handling a bit: if block body withdrawals are missing (but expected to be empty), the body withdrawals are set to 'empty list' before being passed to upper layers.
This fixes an issue where a block passed from EthereumJS to geth was deemed invalid.
Our discovery crawler spits out a huge amount of logs, most of which is pretty non-interesting. This change moves the very verbose output to Debug, and adds a 8-second status log message giving the general idea about what's going on.
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
This is a breaking change in the tracing hooks API as well as semantics of the callTracer:
- CaptureEnter hook provided a nil value argument in case of DELEGATECALL. However to stay consistent with how delegate calls behave in EVM this hook is changed to pass in the value of the parent call.
- callTracer will return parent call's value for DELEGATECALL frames.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* common, core, eth, les, trie: make prque generic
* les/vflux/server: fixed issues in priorityPool
* common, core, eth, les, trie: make priority also generic in prque
* les/flowcontrol: add test case for priority accumulator overflow
* les/flowcontrol: avoid priority value overflow
* common/prque: use int priority in some tests
No need to convert to int64 when we can just change the type used by the
queue.
* common/prque: remove comment about int64 range
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
* log: allow tabs in log messages
This fixes a regression where panic reports in RPC handlers were quoted
because they contain tab characters.
* Update format.go
According to the spec the payloadID needs to be random or dependent on all arguments, to prevent two payloads from clashing. This change adds withdrawals into the payload derivation.
---------
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR moves core/beacon to beacon/engine so that beacon-chain related code has its own top level package which also can house the the beacon lightclient-code.
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change fixes a minor flaw in the check for ipc endpoint length. The max_path_size is the max path that an ipc endpoint can have, which is 208. However, that size concerns the null-terminated pathname, so we need to account for an extra null-character too.
This change improves reusability of the EVM struct. Two methods are added:
- SetBlockContext(...)
- SetTracer(...)
Other attributes like the TransactionContext and the StateDB can already be updated.
BlockContext and Tracer are partially not updateable right now. This change fixes it and
opens the potential to reuse an EVM struct in more ways.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth: cmd: deprecate personal namespace
* eth: cmd: move deprecation to node
* node: disable toml of enablepersonal
* node: disable personal on ipc as well
* Update node/node.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* console: error -> warn
* node: less roulette
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
The execution-apis specification says that the full list of withdrawals should always be returned when requesting a block over RPC:
378c4304f7/src/schemas/block.yaml (L90-L94)
This change adopts the expected behavior.
Adds blocktest subcommand to the evm command, which is very similar to statetest, but instead of loading a StateTest static test it loads a BlockchainTest from a json file and runs it.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
This change removes the Infura rinkeby bootnode as well as two deprecated ropsten bootnodes.
Co-authored-by: Nicola Cocchiaro <ncocchiaro@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes the API so that uint64 is used for fork timestamps.
It's a good choice because types.Header also uses uint64 for time.
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a regression where the flag did not accept values without
the 0x prefix anymore. What's worse, if an invalid value was passed,
the client would just log an INFO level message and continue.
This change introduces a breaking change to miner.etherbase is configured.
Previously, users did not need to explicitly set the etherbase address via flag, since 'first' local account was used as etherbase automatically. This change removes the "default first account" feature.
In Proof-of-stake world, the fee recipient address is provided by CL, and not configured in Geth any more - meaning that miner.etherbase is mostly for legacy networks(pow, clique networks etc).
Fixes#26505 where the console crashed when a property getter
raised an exception during autocompletion. I also noticed while fixing this
issue that autocomplete wasn't working for objects/fields with numbers in
them (most importantly web3.<tab><tab>) which is also now fixed.
This PR does a few things.
It fixes a shutdown-order flaw in the chainfreezer. Previously, the chain-freezer would shutdown the freezer backend first, and then signal for the loop to exit. This can lead to a scenario where the freezer tries to fsync closed files, which is an error-conditon that could lead to exit via log.Crit.
It also makes the printout more detailed when truncating 'dangling' items, by showing the exact number instead of approximate MB.
This PR also adds calls to fsync files before closing them, and also makes the `db inspect` command slightly more robust.
* internal/flags: use filepath.Clean instead of path.Clean
* internal/flags: fix windows pipe issue
* internal/flags: modify test for windows
* internal/flags: use backticks, fix test
This PR fixes an issue which might result in data lost in freezer.
Whenever mutation happens in freezer, all data will be written into head data file
and it will be rotated with a new one in case the size of file reaches the threshold.
Theoretically, the rotated old data file should be fsync'd to prevent data loss.
In freezer.Sync function, we only fsync: (1) index file (2) meta file and (3) head
data file. So this PR forcibly fsync the head data file if mutation happens in the
boundary of data file.
In legacy (pre-merge) sync mode, headers were contiguously downloaded from the network and when no more headers were available, we checked every few seconds whether there are 64 new blocks to move the pivot.
In beacon (post-merge) sync mode, we don't need to check for new skeleton headers non stop, since those re delivered one by one by the engine API. The missing code snippet from the header fetcher was to actually look at the latest head and move the pivot if it was more than 2*64-8 away. This PR adds the missing movement logic.
This makes non-JS tracers execute all block txs on a single goroutine.
In the previous implementation, we used to prepare every tx pre-state
on one goroutine, and then run the transactions again with tracing enabled.
Native tracers are usually faster, so it is faster overall to use their output as
the pre-state for tracing the next transaction.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR removes the notion of fakeStorage from the state objects, and instead, for any state modifications that are needed, it simply makes the changes.
* p2p/discover: add more packet information in logs
This adds more fields to discv5 packet logs. These can be useful when
debugging multi-packet interactions.
The FINDNODE message also gets an additional field, OpID for debugging
purposes. This field is not encoded onto the wire.
I'm also removing topic system related message types in this change.
These will come back in the future, where support for them will be
guarded by a config flag.
* p2p/discover/v5wire: rename 'Total' to 'RespCount'
The new name captures the meaning of this field better.
Alarm is a timer utility that simplifies code where a timer needs to be rescheduled over
and over. Doing this can be tricky with time.Timer or time.AfterFunc because the channel
requires draining in some cases.
Alarm is optimized for use cases where items are tracked in a heap according to their expiry
time, and a goroutine with a for/select loop wants to be woken up whenever the next item expires.
In this application, the timer needs to be rescheduled when an item is added or removed
from the heap. Using a timer naively, these updates will always require synchronization
with the global runtime timer datastructure to update the timer using Reset. Alarm avoids
this by tracking the next expiry time and only modifies the timer if it would need to fire earlier
than already scheduled.
As an example use, I have converted p2p.dialScheduler to use Alarm instead of AfterFunc.
This changes moves the tracking of "deleted in this block" out from snap-only domain, so that it happens regardless of whether the execution is snapshot-backed or trie-backed.
This changes the StorageTrie method to return an error when the trie
is not available. It used to return an 'empty trie' in this case, but that's
not possible anymore under PBSS.
This PR builds on #26299, but also updates the tests to the most recent version, which includes tests regarding TheMerge.
This change adds checks to the beacon consensus engine, making it more strict in validating the pre- and post-headers, and not relying on the caller to have already correctly sanitized the headers/blocks.
This PR implements resettable freezer by adding a ResettableFreezer wrapper.
The resettable freezer wraps the original freezer in a way that makes it possible to ensure atomic resets. Implementation wise, it relies on the os.Rename and os.RemoveAll to atomically delete the original freezer data and re-create a new one from scratch.
This PR fixes an error in trie commit. If the trie.root is nil, it can be two possible scenarios:
- The trie was empty, and no change happens
- The trie was non-empty and all nodes are dropped
For the latter one, we should collect the deletions and apply them into database(e.g. in PBSS).
This ensures that RPC method handlers will react to a timeout or
cancelled request soon after the event occurs.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds a check that the safetxhash that we sign corresponds to the one that is expected by the input. If it differs, it tries again with the configured chainid.
A comment suggests that contract creation happens if the recipient of a call is 0x00..00 ("zero address") but in fact the sender must be nil. The zero address is a regular valid address that is commonly used as a "burn" address.
Currently calling `debug_TraceTransaction` with a transaction hash that doesn't exist returns a confusing error: `genesis is not traceable`. This PR changes the behaviour to instead return an error message saying `transaction not found`
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds a new type event.FeedOf[T], which is like event.Feed but parameterized
over the channel element type. Performance is unchanged, and it still uses reflect. But
unlike Feed, the generic version doesn't need to type-check interface{} arguments.
All panic cases are gone from the API.
While investigating another issue, I found that all callers of collectLogs have the
complete block available. rawdb.ReadReceipts loads the block from the database,
so it is better to use ReadRawReceipts here, and derive the receipt information using
the block which is already in memory.
This PR makes it possible to modify the flush interval time via RPC. On one extreme, `0s`, it would act as an archive node. If set to `1h`, means that after one hour of effective block processing time, the trie would be flushed. If one block takes 200ms, this means that a flush would occur every `5*3600=18000` blocks -- however, if the memory size of the cached states grows too large, it will flush sooner.
Essentially, this makes it possible to configure the node to be more or less "archive:ish", and without restarting the node while reconfiguring it.
--syncTarget is a feature for development purpose in post-merge world. Previously
it's added into eth.Config. But it turns out that's a stupid idea.
- syncTarget is a block object, which is hard to be put in config file(large)
- syncTarget is just a dev feature, doesn't make too much sense to add it in config file
So I remove it from the eth config object. And it also fixes the #26328
This improves readability of function 'push'.
sort.Search(N, ...) will at most return N when no match, so ix should be compared
with N. The previous version would compare ix with N+1 in case an additional item
was appended. No bug resulted from this comparison, but it's not easy to understand
why.
Co-authored-by: Felix Lange <fjl@twurst.com>
The gcproc field tracks the amount of time spent processing blocks,
and is used to trigger a state flush to disk when a certain threshold is
reached. After the merge, single block insertion by CL is the most
common source of block processing time, but this time was not added
into gcproc.
Here we add special handling for sending an error response when the write timeout of the
HTTP server is just about to expire. This is surprisingly difficult to get right, since is
must be ensured that all output is fully flushed in time, which needs support from
multiple levels of the RPC handler stack:
The timeout response can't use chunked transfer-encoding because there is no way to write
the final terminating chunk. net/http writes it when the topmost handler returns, but the
timeout will already be over by the time that happens. We decided to disable chunked
encoding by setting content-length explicitly.
Gzip compression must also be disabled for timeout responses because we don't know the
true content-length before compressing all output, i.e. compression would reintroduce
chunked transfer-encoding.
The new flag allows configuring an explicit endpoint which is to be
announced in the DHT. This feature was originally developed for the
discv5 wormhole experiment (#25798), but it's useful in other contexts
as well.
This removes the 'time' field from logs, as well as from the tracer interface. This change makes the trace output deterministic. If a tracer needs the time they can measure it themselves. No need for evm to do this.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This changes the Pop method to assign the zero value before
reducing slice size. Doing so ensures the backing array does not
reference removed item values.
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.
As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
This PR should makes it easier to sign EIP-712 typed data via the accounts.Wallet API, by using the mimetype for typed data.
Co-authored-by: nasdf <keenan.nemetz@gmail.com>
While investigating #22374, I noticed that the Sync operation of the
freezer does not take the table lock. It also doesn't call sync for all files
if there is an error with one of them. I doubt this will fix anything, but
didn't want to drop the fix on the floor either.
This removes an RPC test which takes > 90s to execute, and updates the
internal/guide tests to use lighter scrypt parameters.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a way to specify HTTP headers per request.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
rpc: fix connection tracking in Server
When upgrading to mapset/v2 with generics, the set element type used in
rpc.Server had to be changed to *ServerCodec because ServerCodec is not
'comparable'. While the distinction is technically correct, we know all
possible ServerCodec types, and all of them are comparable. So just use
a map instead.
It seems there is no fully typed library implementation of an LRU cache.
So I wrote one. Method names are the same as github.com/hashicorp/golang-lru,
and the new type can be used as a drop-in replacement.
Two reasons to do this:
- It's much easier to understand what a cache is for when the types are right there.
- Performance: the new implementation is slightly faster and performs zero memory
allocations in Add when the cache is at capacity. Overall, memory usage of the cache
is much reduced because keys are values are no longer wrapped in interface.
This PR changes the pending tx subscription to return RPCTransaction types instead of normal Transaction objects. This will fix the inconsistencies with other tx returning API methods (i.e. getTransactionByHash), and also fill in the sender value for the tx.
co-authored by @s1na
This fixes a problem in the SizeConstrainedLRU. The SCLRU uses an underlying simple lru which is not thread safe.
During the Get operation, the recentness of the accessed item is updated, so it is not a pure read-operation. Therefore, the mutex we need is a full mutex, not RLock.
This PR changes the mutex to be a regular Mutex, instead of RWMutex, so a reviewer can at a glance see that all affected locations are fixed.
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.
runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.
The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.
Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.
The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.
We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
This adds an option to direct log output to a file. This feature has been
requested a lot. It's sometimes useful to have this available when running
geth in an environment that doesn't easily allow redirecting the output.
Notably, there is no support for log rotation with this change. The --log.file option
opens the file once on startup and then keeps writing to the file handle.
This can become an issue when external log rotation tools are involved, so it's
best not to use them with this option for now.
When the interpreter is configured to use extra-eips, this change makes it so that all the opcodes are deep-copied, to prevent accidental modification of the 'base' jumptable.
Closes: #26136
Co-authored-by: Martin Holst Swende <martin@swende.se>
PR #26082 added account listing to OnSignerStartup but did not consider the case where a user has a large number of accounts which would be annoying to display.
This PR updates showAccounts() so that if there are more than 20 accounts available the user sees the first 20 displayed in the console followed by: First 20 accounts listed (N more available).
Co-authored-by: Martin Holst Swende <martin@swende.se>
Many of the other types have a function to convert the type to a big.Int,
but Address was missing this function.
It is useful to be able to turn an Address into a big.Int when doing
EVM-like computations natively in Go. Sometimes a Solidity address
type is casted to a uint256 and having a Big method on the Address
type makes this easy.
Instead of using a limit of three nodes per message, we can pack more nodes
into each message based on ENR size. In my testing, this halves the number
of sent NODES messages, because ENR size is usually < 300 bytes.
This also adds RLP helper functions that compute the encoded size of
[]byte and string.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR now also includes a fix to the problem of mult-routines building blocks on the same input. This PR works as before with regards to stopping the work, but it just will not spin up a second routine if one is already building. So if the CL does N calls to FCU+buildblock, and N calls to GetPayload, only the first of each will do something, the other calls will be mostly no-ops.
This PR also adds printout of the payload id into the logs.
* apitypes: synchronize handling of types
* signer/core/apitypes: improve array check
* apitypes: add a test for big.Int -> int32
* signer/core/apitypes: Add a test for parsing addresses from [20]byte, []byte and string
* signer/core/apitypes: add some testcases
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes geth to read the eip1559 params from the chain config instead of the globals.
This way the parameters may be changed by forking the chain config code, without creating a large diff throughout the past and future usages of the parameters.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This commit adds support for two new commands to clef, making it possible to list accounts / wallets from the command-line-interface.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Noticed that lookupDistances for FINDNODE requests didn't consider 256 a valid
distance. This is actually part of the example in the comment above the
function, surprised that wasn't tested before.
In some cases, it is desirable to capture what is triggered by each trace, when using the `callTracer`. For example: call `USDT.transfer` will trigger a `Transfer(from, to, value)` event.
This PR adds the option to capture logs to the call tracer, by specifying `{"withLog": true}` in the tracerconfig.
Any logs belonging to failed/reverted call-scopes are removed from the output, to prevent interpretation mistakes.
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Inner call reverts will now return the reason similar to the top-level call. Separately, if top-level call is of type CREATE and it fails, its `to` field will now be cleared to `0x00...00` instead of being set to the created address.
This PR adds a parameter to startup, --synctarget. The synctarget flag is a developer-flag, that can be useful in some scenarios as a replacement for a CL node. It defines a fixed block sync target:
geth --syncmode=full --synctarget=./block_15816882.hex_rlp
The --synctarget is only made available during syncmode=full
* eth/tracers: fix gasUsed in call tracer
* fix js tracers gasUsed
* fix legacy prestate tracer
* fix restGas in test
* drop intrinsicGas field from js tracers
Currently, in order to chain together sequential valid t8n transitions the caller must manually calculate the block base fee. This PR adds support for the necessary parent fee market data to calculate the base fee for the current transition.
Concretely, env is extended to accept the following:
parentBaseFee
parentGasUsed
parentGasLimit
Example usage can be found in ./cmd/evm/testdata/25.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR ports a few changes from PBSS:
- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
The prestate tracer did not report accounts that existed at a given address prior to a contract being created at that address.
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR fixes a regression causing snapshots not to be generated in "geth --import" mode. It also fixes the geth export command to be truly readonly, and adds a new test for geth export.
In some cases, inner contract creation may not be successful, and an inner contract was not created. This PR fixes a crash that could occur when doing tracing in such situations.
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR removes some optimistic tests -- a'la "do something,
wait a while, and hope it has trickled through and continue" -- and
instead uses some introspection to ensure that prerequisites are met.
This changes the node setup to ignore datadir files
static-nodes.json
trusted-nodes.json
When these files are present, it an error will be printed to the log.
This adds a
* core/vm, tests: optimized modexp + fuzzer
* common/math: modexp optimizations
* core/vm: special case base 1 in big modexp
* core/vm: disable fastexp
This changes the error message for mismatching chain ID to show
the given and expected value. Callers expecting this error must be
changed to use errors.Is.
* usbwallet support Ledger Nano S Plus
* accounts/usbwallet: add definitions + ref to ledger docs
Co-authored-by: Martin Holst Swende <martin@swende.se>
This update resolves an issue where StringSliceFlag would not be
rendered correctly in help output + mention that -H can be used multiple times
Co-authored-by: Martin Holst Swende <martin@swende.se>
Prior to this change, f.begin (and possibly end) stay negative, leading to strange results later in the code. With this change, filters using "safe" and "finalized" block produce results consistent w/ the overall behavior of this RPC method.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethclient/gethclient: improve time-sensitive flaky test
* eth/catalyst: fix (?) flaky test
* core: stop blockchains in tests after use
* core: fix dangling blockchain instances
* core: rm whitespace
* eth/gasprice, eth/tracers, consensus/clique: stop dangling blockchains in tests
* all: address review concerns
* core: goimports
* eth/catalyst: fix another time-sensitive test
* consensus/clique: add snapshot test run function
* core: rename stop() to stopWithoutSaving()
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR introduces a new mechanism in chain tracer for preventing creating too many trace states.
The workflow of chain tracer can be divided into several parts:
- state creator generates trace state in a thread
- state tracer retrieves the trace state and applies the tracing on top in another thread
- state collector gathers all result from state tracer and stream to users
It's basically a producer-consumer model here, while if we imagine that the state producer generates states too fast, then it will lead to accumulate lots of unused states in memory. Even worse, in path-based state scheme it will only keep the latest 128 states in memory, and the newly generated state will invalidate the oldest one by marking it as stale.
The solution for fixing it is to limit the speed of state generation. If there are over 128 states un-consumed in memory, then the creation will be paused until the states are be consumed properly.
Backwards compatibility warning: The result will from now on omit empty fields instead
of including a zero value (e.g. no more `balance: '0x'`).
The prestateTracer will now take an option `diffMode: bool`. In this mode
the tracer will output the pre state and post data for the modified parts of state.
Read-only accesses will be completely omitted. Creations (be it account or slot)
will be signified by omission in the `pre` list and inclusion in `post`. Whereas
deletion (be it account or slot) will be signified by inclusion in `pre` and omission
in `post` list.
Signed-off-by: Delweng <delweng@gmail.com>
This PR makes it so that the snap server responds to trie heal requests when possible, even if the snapshot does not exist. The idea being that it might prolong the lifetime of a state root, so we don't have to pivot quite as often.
This PR makes it possible to set custom headers, in particular for two scenarios:
- geth attach
- geth commands which can use --remotedb, e..g geth db inspect
The ability to use custom headers is typically useful for connecting to cloud-apis, e.g. providing an infura- or alchemy key, or for that matter access-keys for environments behind cloudflare.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR reworks tx indexer a bit. Compared to the original version, one scenario is no longer handled - upgrading from legacy geth without indexer support.
The tx indexer was introduced in 2020 and have been present through hardforks, so it can be assumed that all Geth nodes have tx indexer already. So we can simplify the tx indexer logic a bit:
- If the tail flag is not present, it means node is just initialized may or may not with an ancient store attached. In this case all blocks are regarded as unindexed
- If the tail flag is present, it means blocks below tail are unindexed, blocks above tail are indexed
This change also address some weird cornercases that could make the indexer not work after a crash.
This fixes a bug where contract code would be overridden to empty code ("0x")
when the Code field of OverrideAccount was left nil. The change also cleans up
the encoding of overrides to only send necessary fields, and improves documentation.
Fixes#25615
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
The call tracer and prestate tracer store data JSON-encoded in memory. In order to support alternative encodings (specifically RLP), it's better to keep data a native format during tracing. This PR does marshalling at the end, using gencodec.
OBS!
This PR changes the call tracer result slightly:
- Order of type and value fields are changed (should not matter).
- Output fields are completely omitted when they're empty (no more output: "0x"). Previously, this was only _sometimes_ omitted (e.g. when call ended in a non-revert error) and otherwise 0x when the output was actually empty.
Some tests define an 'expectException' error but the tests runner does not check for conditions where this test value is filled (error expected) but in which no error is returned by the test runner.
An example of this scenario is GeneralStateTests/stTransactionTest/HighGasPrice.json, which expects a 'TR_NoFunds' error, but the test runner does not return any error.
Signed-off-by: meows <b5c6@protonmail.com>
`geth dumpgenesis` currently does not respect the content of the data directory. Instead, it outputs the genesis block created by command-line flags. This PR fixes it to read the genesis from the database, if the database already exists.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: pad memory slice on oob case
* eth/tracers/js: fix testfailure due to err msg capitalization
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
The abigen exclusion pattern, previously on the form "path:type", now supports wildcards. Examples "*:type" to exclude a named type in all files, or "/path/to/foo.sol:*" all types in foo.sol.
This changes the CI build to store the git commit and date into package
internal/version instead of package main. Doing this essentially merges our
two ways of tracking the go-ethereum version into a single place, achieving
two objectives:
- Bad block reports, which use version.Info(), will now have the git commit
information even when geth is built in an environment such as
launchpad.net where git access is unavailable.
- For geth builds created by `go build ./cmd/geth` (i.e. not using `go run
build/ci.go install`), git information stored by the go tool is now used
in the p2p node name as well as in `geth version` and `geth
version-check`.
Sometimes we get stuck on db compaction, and the CL re-issues the "same" command to us multiple times. Each request get stuck on the same place, in the middle of the handler.
This changes makes it so we do not reprocess the same payload, but instead detects it early.
core/blockchain: downgrade tx indexing and unindexing logs from info to debug
If a user has a finite tx lookup limit, they will see an "unindexing" info level log each time a block is imported. This information might help a user understand that they are removing the index each block and some txs may not be retrievable by hash, but overall it is generally more of a nuisance than a benefit. This change downgrades the log to a debug log.
This shortens the chain config summary in bad block reports,
and adds go-ethereum version information as well.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes eth_getProof and eth_getStorageAt return an error when
the argument contains invalid hex in storage keys.
Co-authored-by: Felix Lange <fjl@twurst.com>
* cmd/geth: add a verkle subcommand
* fix copyright year
* remove unused command parameters
* check that the output file was successfully written to
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/geth: goimports fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
This PR makes the event-sending for deleted and new logs happen in batches, to prevent OOM situation due to large reorgs.
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the error code returned by the RPC server in certain situations:
- handler panic: code -32603
- result marshaling error: code -32603
- attempt to subscribe via HTTP: code -32001
In all of the above cases, the server previously returned the default error
code -32000.
Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The p2p msgrate tracker is a thing which tries to estimate some mean round-trip times. However, it did so in a very curious way: if a node had 200 peers, it would sort their 200 respective rtt estimates, and then it would pick item number 2 as the mean. So effectively taking third fastest and calling it mean. This probably works "ok" when the number of peers are low (there are other factors too, such as ttlScaling which takes some of the edge off this) -- however when the number of peers is high, it becomes very skewed.
This PR instead bases the 'mean' on the square root of the length of the list. Still pretty harsh, but a bit more lenient.
This PR includes minor updates to comments in trie/committer that reference insertion to the db, and adds an err != nil check for the return value of preimages.commit.
This PR simplifies the logic of chain tracer and also adds the unit tests.
The most important change has been made in this PR is the state management. Whenever a tracing state is acquired there is a corresponding release function be returned as well. It must be called once the state is used up, otherwise resource leaking can happen.
And also the logic of state management has been simplified a lot. Specifically, the state provider(eth backend, les backend) should ensure the state is available and referenced. State customers can use the state according to their own needs, or build other states based on the given state. But once the release function is called, there is no guarantee of the availability of the state.
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
The JSON-RPC spec requires the "version" field to be exactly "2.0",
so we should verify that. This change is not backwards-compatible with
sloppy client implementations, but I decided to go ahead with it anyway
because the failure will be caught via the returned error.
This adds a generic mechanism for 'dial options' in the RPC client,
and also implements a specific dial option for the JWT authentication
mechanism used by the engine API. Some real tests for the server-side
authentication handling are also added.
Co-authored-by: Joshua Gutow <jgutow@optimism.io>
Co-authored-by: Felix Lange <fjl@twurst.com>
It's a trivial PR to hide the error log when the trie node is not found in the database. The idea for this change is for all TryXXX functions, the error is already returned and we don't need to fire a log explicitly.
Recently there are a few tickets #25613#25589 reporting that the trie nodes are missing because of debug.SetHead. The root cause is after resetting, the chain rewinds to a historical point and re-imports the blocks on top.
Since the node is already synced and started to accept transactions previously, these transactions are still kept in the txpool and verified by txpool with a live state. This live state is constructed based on the live trie database, which is changed fast by node referencing and de-referencing.
Unfortunately, when we construct a live state(like the state in txpool), we don't reference the state we have. The blockchain will garbage collect the intermediate version nodes in another thread which leads the broken live state.
The best solution for this is to forcibly obtain a reference for all live states we create and call release function once it's used up. But it might end up with more junks persisted into disk. Will try to find an elegant solution later in the following PR.
Some small fixes to get the existing debug methods to conform to the spec. Mainly dropping the encoding information from the method name as it should be deduced from the debug context and allowing the method to be invoked by either block number or block hash. It also adds the method debug_getTransaction which returns the raw tx bytes by tx hash. This is pretty much equivalent to the eth_getRawTransactionByHash method.
* eth/catalyst: warn less frequently if no beacon client is available
* eth/catalyst: tweak warning frequency a bit
* eth/catalyst: some more tweaks
* Update api.go
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/fetcher: introduce some lag in tx fetching
* eth/fetcher: change conditions a bit
* eth/fetcher: use per-batch quota check
* eth/fetcher: fix some comments
* eth/fetcher: address review concerns
* eth/fetcher: fix panic + add warn log
* eth/fetcher: fix log
* eth/fetcher: fix log
* cmd/devp2p/internal/ethtest: fix ignorign tx announcements from prev. tests
* cmd/devp2p/internal/ethtest: fix TestLargeTxRequest
This increases the number of tx relay messages the test waits for. Since
go-ethereum now processes incoming txs in smaller batches, the
announcement messages it sends are also smaller.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a cache for block logs which is shared by all filters. The cache
size of is configurable using the `--cache.blocklogs` flag.
Co-authored-by: Felix Lange <fjl@twurst.com>
This avoids copying the input []byte while decoding trie nodes. In most
cases, particularly when the input slice is provided by the underlying
database, this optimization is safe to use.
For cases where the origin of the input slice is unclear, the copying version
is retained. The new code performs better even when the input must be
copied, because it is now only copied once in decodeNode.
* core, trie: flush preimages to db on database close
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* rename Close to CommitPreimages for clarity
* core, trie: nitpick fixes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This adds support for building statically-linked executables using ci.go.
Static linking is enabled by default in Docker builds, making it possible to
use the geth executable in any Docker image, regardless of the Linux
distribution the Dockerfile is based on.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR allows users to pass in a config object directly to the tracers. Previously only the struct logger was configurable.
It also adds an option to the call tracer which if enabled makes it ignore any subcall and collect only information about the top-level call. See #25419 for discussion.
The tracers will silently ignore if they are passed a config they don't care about.
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Because the goal of eth_createAccessList is providing the caller with the largest-possible
access list, it's generally not important that the gas limit used by the tracer will match the usage
of the call exactly. Avoiding the gas estimation step is a performance improvement. As long as the
call does not branch based on gas limit, the returned access list will be accurate.
This change makes http.Server.ReadHeaderTimeout configurable separately
from ReadTimeout for RPC servers. The default is set to the same as
ReadTimeout, which in order to cause no change in existing deployments.
* eth: support bubbling up bad blocks from sync to the engine API
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* Update eth/catalyst/api.go
* eth/catalyst: when forgetting bad hashes, also forget descendants
* eth/catalyst: minor bad block tweaks for resilience
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
During RPC calls such as eth_call and eth_estimateGas, st.evm.Config.NoBaseFee is set
which allows the gas price to be below the base fee. This results the tip being negative,
and balance being subtracted from the coinbase instead of added to it, which results in a
potentially negative coinbase balance interestingly. This can't happen during normal chain
processing as outside of RPC calls the gas price is required to be at least the base fee,
as NoBaseFee is false.
This change prevents this behavior by disabling fee payment when the fee is not set.
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
* Remove locking in (*BlockChain).ExportN
Since ExportN is read-only, it shouldn't need the lock. (?)
* Add hash check to detect reorgs during export.
* fix check order
* Update blockchain.go
* Update blockchain.go
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* internal/ethapi: error if tx args includes chain id that doesn't match local
* internal/ethapi: simplify code a bit
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Adds a native tracer that returns that in case of failure returns the error message or the revert reason of a transaction.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change ensures the HTTP server will always terminate within
at most 5s, even when all connections are busy and do not become
idle.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/catalyst: disallow importing blocks via newPayload during snap sync
* eth/catalyst: make tests pass by using full sync only
* eth/catalysts: make the import delay a bit cleaner
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
This changes the []byte <-> Uint8Array conversion to use an
ArrayBuffer, avoiding inefficient copying of the slice data in Goja.
Co-authored-by: Felix Lange <fjl@twurst.com>
The oss-fuzz engine crashes due to stack overflow decoding a large nested
structure into a interface{}. This PR limits the size of the input data, so
should avoid such crashes.
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
Previously on Geth startup we just logged the chain config is a semi-json-y format. Whilst that worked while we had a handful of hard-forks defined, currently it's kind of unwieldy.
This PR converts that original data dump and converts it into a user friendly - alas multiline - log output.
* accounts/abi/bind: fix duplicate field names in the generated go struct #24627
* accounts, cmd/abigen: resolve name conflicts
* ci lint, accounts/abi: remove unused function overloadedArgName
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds support for block overrides when doing debug_traceCall.
- Previously, debug_traceCall against pending erroneously used a common.Hash{} stateroot when looking up the state, meaning that a totally empty state was used -- so it always failed,
- With this change, we reject executing debug_traceCall against pending.
- And we add ability to override all evm-visible header fields.
In #24028 we flagged a warning when finding legacy receipts in the freezer. This PR nudges users a bit more strongly by preventing geth from starting in this case until receipts have been migrated.
It also adds a flag --ignore-legacy-receipts which when present allows geth to start normally.
#23773 added a JS tracer which uses Goja as its engine. In this PR I remove the previous tracer which used duktape as well as remove the dependencies.
This PR also comes with 2 fixes in the Goja tracer and one small behavioural change:
I had handled errors in the native Go functions by panicing. My oversight was that Goja only handles panics with a Goja.Value as argument. The difference is panic(goja.Value) allows JS to catch the exception whereas Interrupt(error) doesn't.
There was a race in how I handled Stop.
Because of 1. some of the methods that simply return nil on error (like memory.slice) now throw an exception.
This adds a JS tracer runtime environment based on the Goja VM. The new
runtime replaces the duktape runtime, which will be removed soon.
Goja is implemented in Go and is faster for cases where the Go <-> JS
transition overhead dominates overall performance. It is faster because
duktape is written in C, and the transition cost includes the cost of using
cgo. Another reason for using Goja is that go-duktape is not maintained
anymore.
We expect the performace of JS tracing to be at least as good or better with
this change.
* eth/catalyst: build the execution payload async
* miner: added comment, added test case
* eth/catalyst: miner: move async block production to miner
* eth/catalyst, miner: support generate seal block async
* miner: rework GetSealingBlockAsync to use a passed channel
* miner: apply rjl's diff
* eth/catalyst: nitpicks
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* ethdb/remotedb, cmd: add support for remote (readonly) databases
* ethdb/remotedb: minor changes
* ethdb/remotedb: close the conn
* cmd, ethdb: add rpc accessor for ancient data
* internal/ethapi: license
* ethdb/remotedb: linter fixes
This PR adds db tooling (geth db check-state-content) to verify the integrity of trie nodes. It iterates through the 32-byte key space in the database, which is expected to contain RLP-encoded trie nodes, addressed by hash.
* eth/protocols/snap: don't include empty snapshot slot slice
This PR fixes the snapshot storage serving handler. In snap protocol
the response is capped by the response size. Server can cutdown the
response if the accumulated byte size exceeds the local hard limit.
It means we can meet a special scenario that there is no storage slot
included for a requested account, but we attach the proof for this
account by mistake.
So in the prover side, when it meets a empty storage response but with
a valid proof proves there are some more slots left in the trie, then
requestor will reject this response and disconnect with server.
In this PR, if there is no storage slot served for the requested account,
then no proof should be attached as well.
* eth/protocols/snap: loosen restrictions for flaky tests
* eth/catalyst: fix flaky test in catalyst
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.
This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
During mining, when a new head arrives and interrupts the block building, the block being built should not be commited (but discarded). Committing the interrupted block introduces unnecessary delay, and possibly causes miner to mine on the previous head, which could result in higher uncle rate.
* core: recover the state in SetChainHead if the head state is missing
* core: disable test logging
* core: address comment from martin
* core: improve log level in case state is recovered
* core, eth, les, light: rename SetChainHead to SetCanonical
This PR improves the docker build speed for repeated builds where go.mod and go.sum do no change, by placing the downloaded dependencies in a lower layer
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
This adds the ability to run --state.fork=Merged, and have post-merge rules apply. When doing so, it also requires the input env to contain currentRandom, and enforces the currentDifficulty to be omitted or zero.
This PR fixes up the example python clef wrapper. The poc is intended to demonstrate how to wite a UI for clef, and had severely bitrotted.
With these changes, it "works" in the sense that all the built-in tests triggers the intended python callbacks (no errors about method not found). It does not "work" in the sense that the wrapper can be used as an actual UI. It will auto-reject any signing requests, for example.
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
This change removes extraneous/unnecessary checks for equality
when comparing 2 accessList values A and B. Given that we validate that
their lengths of A and B are equal, if so and if every element in A is
in B, reflexively every element in B is already in A. If that weren't
the case and an element g existed in A but not in B, that would mean
that there is an extra element and hence a mathematical contradiction.
Fixes#24658
See ethereum/go-ethereum#24554 and btcsuite/btcd#1839
This is an attempt to resolve a Go module dependency issue that arises
when both 'github.com/btcsuite/btcd/btcec/v2' and the older, non-v2
btcd module are required as dependencies.
* eth/downloader: remove stale beacon headers as backfilling progresses
* eth/downloader: remove leftover from a previous design
* eth/downloader: do partial beacon cleanups if chain is large
* eth/downloader: linter != heart
This adds a tools.go file to import all command packages used for
go:generate. Doing so makes it possible to execute go-based code
generators using 'go run', locking in the tool version using go.mod.
Co-authored-by: Felix Lange <fjl@twurst.com>
If a pending contract call errors, return that error right away rather
than ignoring it to allow an error somewhere else. This is helpful for
callers to know if perhaps a call failed because of the context deadline
being expired. This change mirrors the behavior of non-pending contract
calls.
This PR fixes a few panics in the chain marker benchmarks. The root
cause for panic is in chain marker the genesis header/block is not
accessible, while it's expected to be obtained in tests. So this PR
avoids touching genesis header at all to avoid panic.
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but t.TempDir handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* eth/tracers: refactor traceTx to separate out struct logging
review fix
Update eth/tracers/api.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Mv ExecutionResult type to logger package
review fix
impl GetResult for StructLogger
make formatLogs private
confused exit and end..
account for intrinsicGas in structlogger, fix TraceCall test
Add Stop method to logger
Simplify traceTx
Fix test
rm logger from blockchain test
account for refund in structLogger
* use tx hooks in struct logger
* minor
* avoid executionResult in struct logger
* revert blockchain test changes
* core,eth: add empty tx logger hooks
* core,eth: add initial and remaining gas to tx hooks
* store tx gasLimit in js tracer
* use gasLimit to compute intrinsic cost for js tracer
* re-use rules in transitiondb
* rm logs
* rm logs
* Mv some fields from Start to TxStart
* simplify sender lookup in prestate tracer
* mv env to TxStart
* Revert "mv env to TxStart"
This reverts commit 656939634b9aff19f55a1cd167345faf8b1ec310.
* Revert "simplify sender lookup in prestate tracer"
This reverts commit ab65bce48007cab99e68232e7aac2fe008338d50.
* Revert "Mv some fields from Start to TxStart"
This reverts commit aa50d3d9b2559addc80df966111ef5fb5d0c1b6b.
* fix intrinsic gas for prestate tracer
* add comments
* refactor
* fix test case
* simplify consumedGas calc in prestate tracer
Trie tracer is an auxiliary tool to capture all deleted nodes
which can't be captured by trie.Committer. The deleted nodes
can be removed from the disk later.
This PR adds a ExtraAllowedPath field to Solidity and exposes two APIs: CompileSource and CompileFiles, which were hidden inside CompileSolidityString and CompileSolidity before.
* eth/catalyst: only apply block if we actually have the state
* add header to payload queue
* Update cmd/geth/dbcmd.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/geth: only check for presence of legacy receipts if developer mode is not enabled
* cmd/geth: degrade log level
* cmd/geth: fix format
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd,core: add simple legacy receipt converter
core/rawdb: use forEach in migrate
core/rawdb: batch reads in forEach
core/rawdb: make forEach anonymous fn
cmd/geth: check for legacy receipts on node startup
fix err msg
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix log
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix some review comments
add warning to cmd
drop isLegacy fn from migrateTable params
add test for windows rename
test replacing in windows case
* minor fix
* sanity check for tail-deletion
* add log before moving files around
* speed-up hack for mainnet
* fix mainnet check, use networkid instead
* check mainnet genesis
* review fixes
* resume previous migration attempt
* core/rawdb: lint fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/beacon: eth/catalyst: updated engine api to new version
* core: implement exchangeTransitionConfig
* core/beacon: prevRandao instead of Random
* eth/catalyst: Fix ExchangeTransitionConfig, add test
* eth/catalyst: stop external miners on TTD reached
* node: implement --authrpc.vhosts flag
* core: allow for config override on non-mainnet networks
* eth/catalyst: fix peters comments
* eth/catalyst: make stop remote sealer more explicit
* eth/catalyst: add log output
* cmd/utils: rename authrpc.host to authrpc.addr
* eth/catalyst: disable the disabling of the miner
* eth: core: remove notion of terminal pow block
* eth: les: more of peters nitpicks
This updates the no-cgo implementations in the crypto package to use
the github.com/btcsuite/btcd/btcec/v2 module instead of the older btcec
package that was part of the main github.com/btcsuite/btcd module.
name old time/op new time/op delta
EcrecoverSignature-32 198µs ± 0% 144µs ± 0% -27.11%
VerifySignature-32 177µs ± 0% 128µs ± 0% -27.44%
DecompressPubkey-32 20.9µs ± 0% 10.1µs ± 0% -51.51%
Use (*ModNScalar).IsOverHalfOrder instead of math/big.Int when checking
for malleable signatures.
* cmd, eth: Rename whitelist argument to peer.requiredblocks
* eth/ethconfig: document PeerRequiredBlocks better
* cmd/utils: rename new flag to --eth.requiredblocks
Co-authored-by: Felix Lange <fjl@twurst.com>
* trie: fix memory leak in trie iterator
In the trie iterator, live nodes are tracked in a stack while iterating.
Popped node states should be explictly set to nil in order to get
garbage-collected.
* trie: fix empty trie iterator
* eth/downloader: implement beacon sync
* eth/downloader: fix a crash if the beacon chain is reduced in length
* eth/downloader: fix beacon sync start/stop thrashing data race
* eth/downloader: use a non-nil pivot even in degenerate sync requests
* eth/downloader: don't touch internal state on beacon Head retrieval
* eth/downloader: fix spelling mistakes
* eth/downloader: fix some typos
* eth: integrate legacy/beacon sync switchover and UX
* eth: handle UX wise being stuck on post-merge TTD
* core, eth: integrate the beacon client with the beacon sync
* eth/catalyst: make some warning messages nicer
* eth/downloader: remove Ethereum 1&2 notions in favor of merge
* core/beacon, eth: clean up engine API returns a bit
* eth/downloader: add skeleton extension tests
* eth/catalyst: keep non-kiln spec, handle mining on ttd
* eth/downloader: add beacon header retrieval tests
* eth: fixed spelling, commented failing tests out
* eth/downloader: review fixes
* eth/downloader: drop peers failing to deliver beacon headers
* core/rawdb: track beacon sync data in db inspect
* eth: fix review concerns
* internal/web3ext: nit
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR adds the `NextBaseFeePerGas` to `Block` and `EffectiveTip` to `Transaction` to make it easier for clients to compute fee history themselves via graphql queries.
* core/rawdb, cmd, ethdb, eth: implement freezer tail deletion
* core/rawdb: address comments from martin and sina
* core/rawdb: fixes cornercase in tail deletion
* core/rawdb: separate metadata into a standalone file
* core/rawdb: remove unused code
* core/rawdb: add random test
* core/rawdb: polish code
* core/rawdb: fsync meta file before manipulating the index
* core/rawdb: fix typo
* core/rawdb: address comments
This change speeds up trie hashing and all other activities that require
RLP encoding of trie nodes by approximately 20%. The speedup is achieved by
avoiding reflection overhead during node encoding.
The interface type trie.node now contains a method 'encode' that works with
rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code.
trie.hasher, which is pooled to avoid allocations, now maintains an
EncoderBuffer. This means memory resources related to trie node encoding
are tied to the hasher pool.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth, cmd: allow FdLimit to be set in config/command line (#24148)
* eth/ethconfig: format code
* cmd, eth/ethconfig: simplify fdlimit arg, disallow toml
* cnd/utils: make fdlimit setting nicer on the logs
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* rpc, node: refactor request validation and add jwt validation
* node, rpc: fix error message, ignore engine api in RegisterAPIs
* node: make authenticated port configurable
* eth/catalyst: enable unauthenticated version of engine api
* node: rework obtainjwtsecret (backport later)
* cmd/geth: added auth port flag
* node: happy lint, happy life
* node: refactor authenticated api
Modifies the authentication mechanism to use default values
* node: trim spaces and newline away from secret
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This replaces the simple selector parser in signer/fourbyte with one that
can actually handle most types. The new parser is added in accounts/abi
to also make it useable elsewhere.
* go.mod: update azure-storage-blob-go
update Azure/azure-storage-blob-go from v0.7.0 to v0.14.0.
relation #24396.
* internal/build: fix for breaking changes of azure-storage-blob-go
fix for breaking changes of update Azure/azure-storage-blob-go from v0.7.0 to v0.14.0.
relation #24396.
* internal/build: switch azure sdk from Azure/azure-storage-blob-go to Azure/azure-sdk-for-go/sdk/storage/azblob.
* internal/build refactor appending BlobItems
* internal/build: fix azure blobstore client to include container id
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.
The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.
To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.
Benchmark results:
name old time/op new time/op delta
EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8)
EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8)
name old speed new speed delta
EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8)
EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal)
This change adds a code generator tool for creating EncodeRLP method
implementations. The generated methods will behave identically to the
reflect-based encoder, but run faster because there is no reflection overhead.
Package rlp now provides the EncoderBuffer type for incremental encoding. This
is used by generated code, but the new methods can also be useful for
hand-written encoders.
There is also experimental support for generating DecodeRLP, and some new
methods have been added to the existing Stream type to support this. Creating
decoders with rlpgen is not recommended at this time because the generated
methods create very poor error reporting.
More detail about package rlp changes:
* rlp: externalize struct field processing / validation
This adds a new package, rlp/internal/rlpstruct, in preparation for the
RLP encoder generator.
I think the struct field rules are subtle enough to warrant extracting
this into their own package, even though it means that a bunch of
adapter code is needed for converting to/from rlpstruct.Type.
* rlp: add more decoder methods (for rlpgen)
This adds new methods on rlp.Stream:
- Uint64, Uint32, Uint16, Uint8, BigInt
- ReadBytes for decoding into []byte
- MoreDataInList - useful for optional list elements
* rlp: expose encoder buffer (for rlpgen)
This exposes the internal encoder buffer type for use in EncodeRLP
implementations.
The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to
encBuffer. It is implemented this way to ensure the global encBuffer pool
is handled correctly.
This PR adds an addtional API called `NewBatchWithSize` for db
batcher. It turns out that leveldb batch memory allocation is
super inefficient. The main reason is the allocation step of
leveldb Batch is too small when the batch size is large. It can
take a few second to build a leveldb batch with 100MB size.
Luckily, leveldb also offers another API called MakeBatch which can
pre-allocate the memory area. So if the approximate size of batch is
known in advance, this API can be used in this case.
It's needed in new state scheme PR which needs to commit a batch of
trie nodes in a single batch. Implement the feature in a seperate PR.
This functionality is needed in new path-based storage scheme, but
can be implemented in a seperate PR though.
When an account is deleted, then all the storage slots should be
nuked out from the disk as well. In hash-based storage scheme they
are still left in the disk but in new scheme, they will be iterated
and marked as deleted.
But why the NodeBlob API is needed in this scenario? Because when
the node is marked deleted, the previous value is also required to
be recorded to construct the reverse diff.
text: 'SA1019:"golang.org/x/crypto/openpgp"is deprecated:this package is unmaintained except for security fixes.'
- path:core/vm/contracts.go
text: 'SA1019:"golang.org/x/crypto/ripemd160"is deprecated:RIPEMD-160 is a legacy hash and should not be used for new applications.'
- path:accounts/usbwallet/trezor.go
text: 'SA1019:"github.com/golang/protobuf/proto"is deprecated:Use the "google.golang.org/protobuf/proto" package instead.'
- path:accounts/usbwallet/trezor/
text: 'SA1019:"github.com/golang/protobuf/proto"is deprecated:Use the "google.golang.org/protobuf/proto" package instead.'
exclude:
- 'SA1019: event.TypeMux is deprecated:use Feed'
- 'SA1019:strings.Title is deprecated'
- 'SA1019: strings.Title has been deprecated since Go 1.18 and an alternative has been available since Go 1.0:The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead.'
- 'SA1029:should not use built-in type string as key for value'
- sed -i '.bak' 's/repo.join/!repo.join/g' $(dirname `gem which cocoapods`)/cocoapods/sources_manager.rb
- if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then git clone --depth=1 https://github.com/CocoaPods/Specs.git ~/.cocoapods/repos/master && pod setup --verbose; fi
- xctool -version
- xcrun simctl list
# Workaround for https://github.com/golang/go/issues/23749
* [rpc: check that "version" is "2.0" in request objects (#25570)](https://github.com/bnb-chain/bsc/pull/1840/commits/38e002f4641c2779c897ccaca575ec5ddeee9254)
* [rpc: support injecting HTTP headers through context (#26023)](https://github.com/bnb-chain/bsc/pull/1840/commits/add337e0f7bad02f3cf535c66cd31f252b0b5c99)
* [rpc: websocket should respect the "HTTP_PROXY" by default (#27264)](https://github.com/bnb-chain/bsc/pull/1840/commits/73697529994e14996b7740730481e926d5ec3e40)
* [rpc: change BlockNumber constant values to match ethclient (#27219)](https://github.com/bnb-chain/bsc/pull/1840/commits/9231770811cda0473a7fa4e2bccc95bf62aae634)
* [eth: make debug_StorageRangeAt take a block hash or number (#27328)](https://github.com/bnb-chain/bsc/pull/1840/commits/d789c68b667e13eb5cefd19d09ae84f7d016df6a)
* [eth,core: add api debug_getTrieFlushInterval (#27303)](https://github.com/bnb-chain/bsc/pull/1840/commits/0783cb7d91ad7b3cdf72ac6c6edaec8318673eb6)
* [rpc: add limit for batch request items and response size (#26681)](https://github.com/bnb-chain/bsc/pull/1840/commits/f3314bb6df4c86e650f0e47cbb5a21ca0616ac11)
* [core/types: support yParity field in JSON transactions (#27744)](https://github.com/bnb-chain/bsc/pull/1840/commits/bb148dd342ba03ce40cf04295e193c94b9dda322)
* [eth/filters: send rpctransactions in pending-subscription (#26126)](https://github.com/bnb-chain/bsc/pull/1840/commits/8c5ce1107b3110c7cb735d8dfa91c9c701393c85)
#### Flag
* [cmd/geth: rename --whitelist to --eth.requiredblocks (#24505)](https://github.com/bnb-chain/bsc/pull/1840/commits/dbfd3972624c1d82db21f5dfceab8fde7a1eee0a)
* [cmd: migrate to urfave/cli/v2 (#24751)](https://github.com/bnb-chain/bsc/pull/1840/commits/52ed3570c483693fdd6667add7e3050520ad3ba2)
* [cmd/utils: print warning when --metrics.port set without --metrics.ad…](https://github.com/bnb-chain/bsc/pull/1840/commits/8846c07d044f30dca8cd0db91c6245f71f4b24fa)
* [cmd/devp2p: add --extaddr flag (#26312)](https://github.com/bnb-chain/bsc/pull/1840/commits/b44abf56a966016cbb651648ac2d7b6705e80b11)
* [core,eth: adddebug_setTrieFlushInterval to change trie flush frequ](https://github.com/bnb-chain/bsc/pull/1840/commits/711afbc7fd76f1f206429e26f9aa5bf98bc7b43d)
* [graphql: fee history fields (#24452)](https://github.com/bnb-chain/bsc/pull/1840/commits/57cec892536270fc6dafae01ded2c528ffa370e9)
* [graphql: add rawReceipt field to transaction type (#24738)](https://github.com/bnb-chain/bsc/pull/1840/commits/d73df893a6fc528e69506397322205bd9258b6fa)
* [graphql: add raw fields to block and tx (#24816)](https://github.com/bnb-chain/bsc/pull/1840/commits/29a6b6bcac170ca7f8fceb242eba45ff15df17a1)
* [graphql: return correct logs for tx (#25612)](https://github.com/bnb-chain/bsc/pull/1840/commits/d0dc349fd36bd79f94516c866251783641ed12f1)
* [ethereum, ethclient: add FeeHistory support (#25403)](https://github.com/bnb-chain/bsc/pull/1840/commits/9ad508018e4790da0c1c00ac355f206fca12ab7c)
* [eth/filters, ethclient/gethclient: add fullTx option to pending tx fi…](https://github.com/bnb-chain/bsc/pull/1840/commits/5b1a04b9c749d804b51159fe12246c56de8515c1)
* [ethclient: include withdrawals in ethclient block responses (#26778)](https://github.com/bnb-chain/bsc/pull/1840/commits/e1b98f49a5075694c5022f5ec74425e40da415dd)
#### Tracer
* [eth/tracers/js: drop duktape engine (#24934)](https://github.com/bnb-chain/bsc/pull/1840/commits/ba47d800b13058885288c38bd174babb38560c89)
* [eth/tracers: add support for block overrides in debug_traceCall (#24871)](https://github.com/bnb-chain/bsc/pull/1840/commits/d8a2305565b1f97c451f8595e0f65358d6842714)
* [eth/tracers: add onlyTopCall option to callTracer (#25430)](https://github.com/bnb-chain/bsc/pull/1840/commits/86de2e516e5a4a2bbe1d29b46a0f460fbdde8303)
* [eth/tracers: remove revertReasonTracer, add revert reason to callTracer](https://github.com/bnb-chain/bsc/pull/1840/commits/ff1f49245d641a7268ade38cf512bdc7b26f9b7c)
* [eth/tracers: add diffMode to prestateTracer (#25422)](https://github.com/bnb-chain/bsc/pull/1840/commits/5d52a35931bba10f438ce4f41410442dd9cd396c)
* [core/vm: set tracer-observable value of a delegatecall to match parent `value`](https://github.com/bnb-chain/bsc/pull/1840/commits/b0cd8c4a5c4f0f25011ed64235a3ea1280f03c51)
* [eth/tracers/native: set created address to nil in case of failure (#2…](https://github.com/bnb-chain/bsc/pull/1840/commits/41af42e97c9d62d303a883cc3c143f560867fa34)
* [eth/tracers: report correct gasLimit in call tracers (#27029)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b76eb3708626fbd2eb9c1b58d7b4eac6a5eec15)
* [eth/tracers: addtxHashfield on txTraceResult (#27183)](https://github.com/bnb-chain/bsc/pull/1840/commits/604e215d1bb070dff98fb76aa965064c74e3633f)
* [eth/tracers: add ReturnData in the tracer's response (#27704)](https://github.com/bnb-chain/bsc/pull/1840/commits/1e069cf8026a9f71b5f7e80959465e4b273d5806)
* [core/state/snapshot: detect and clean up dangling storage snapshot in generation](https://github.com/bnb-chain/bsc/pull/1840/commits/59ac229f87831bd74b4dc07d34f54137cca78095)
* [internal/ethapi: add db operations to api (#24739)](https://github.com/bnb-chain/bsc/pull/1840/commits/16701c51697e28986feebd122c6a491e4d9ac0e7)
* [cmd/geth: adddb check-state-contentto verify integrity of trie nodes (#24840)](https://github.com/bnb-chain/bsc/pull/1840/commits/e0a9752b965f243313f2c32a91d306600dc3863c)
* [ethdb/remotedb, cmd: add support for remote (readonly) databases](https://github.com/bnb-chain/bsc/pull/1840/commits/57192bd0dc545d921306f6a4d7566c0c70c764c5)
* [cmd/abigen: accept combined-json via stdin (#24960)](https://github.com/bnb-chain/bsc/pull/1840/commits/0287e1a7c00c1eaad1a99b4ea05d70f1ed685140)
* [cmd/geth: add a verkle subcommand (#25718)](https://github.com/bnb-chain/bsc/pull/1840/commits/9d717167aaf27a48d56ad9d1a2c36f90eba1cc13)
* [cmd/geth, cmd/utils: geth attach with custom headers (#25829)](https://github.com/bnb-chain/bsc/pull/1840/commits/ea26fc8a6c44ebb48223f991048f41b2ec0a6414)
* [core/rawdb: refactor db inspector for extending multiple ancient storage](https://github.com/bnb-chain/bsc/pull/1840/commits/60e30a940bbba2c0d26de040195a5ccdb14d8c10)
* [cmd/clef: add importraw feature to clef (#26058)](https://github.com/bnb-chain/bsc/pull/1840/commits/17744639dafc5a54f21e220660bd39d765a09051)
* [cmd/devp2p: add more nodekey commands (#26129)](https://github.com/bnb-chain/bsc/pull/1840/commits/913973436bb88b652faffc10d8f97e4c19722883)
* [internal/web3ext: fix eth_call stateOverrides in console (#26265)](https://github.com/bnb-chain/bsc/pull/1840/commits/1325fef1025b9feb3342308265b6d1399614be30)
* [cmd/evm: add blocktest subcommand to evm (#26526)](https://github.com/bnb-chain/bsc/pull/1840/commits/90f15a0230be34a292c5d0574ee7910ee44267de)
#### HardFork
* [params: define cancun and prague as timestamp based forks (#26481)](https://github.com/bnb-chain/bsc/pull/1840/commits/f3a005f176372ff291dfa7c02ee1c87d18e9c788)
* [all: tie timestamp based forks to the passage of London (#27279)](https://github.com/bnb-chain/bsc/pull/1840/commits/85a4b82b3373fc5f3fa8b7c68061c55b0db0e9b7)
* [core/vm: implement EIP-3860: Limit and meter initcode (#23847)](https://github.com/bnb-chain/bsc/pull/1840/commits/793f0f9ec860f6f51e0cec943a268c10863097c7)
* [PBBS(ready to activate)](https://github.com/ethereum/go-ethereum/commits?author=rjl493456442)
#### P2P
* [eth/fetcher: throttle peers which deliver many invalid transactions (…](https://github.com/bnb-chain/bsc/pull/1840/commits/7f2890a9be1f91368582479f171248b972b45ae3)
* [all: remove concept of public/private API definitions (#25053)](https://github.com/bnb-chain/bsc/pull/1840/commits/10dc5dce0871bf8c24bac41b04e47c3b9ad2b93e)
* [cmd/geth: drop geth js command (#25000)](https://github.com/bnb-chain/bsc/pull/1840/commits/f20a56926551ae91a349498f9ce97c8ee373d6bb)
* [node: drop support for static & trusted node list files (#25610)](https://github.com/bnb-chain/bsc/pull/1840/commits/3630cafb34f7c48b9cc78cf736309275cbd70f74)
* [core: drop legacy receipt types (#26225)](https://github.com/bnb-chain/bsc/pull/1840/commits/10347c6b54d5b28a2e71d9c4993e7f44b0a359c3)
* [cmd, eth, node: deprecate personal namespace (#26390)](https://github.com/bnb-chain/bsc/pull/1840/commits/d0a4989a8def7e6bad182d1513e8d4a093c1672d)
* [accounts, build, mobile: remove Andriod and iOS support](https://github.com/bnb-chain/bsc/pull/1840/commits/d9699c8238307d5c3081c12078f78527468d7dbc)
* [params: remove EIP150Hash from chainconfig (#27087)](https://github.com/bnb-chain/bsc/pull/1840/commits/5e4d726e2a05aee80a75e5f99fd699f220dd503e)
* [all: remove notion of trusted checkpoints in the post-merge world (#2…](https://github.com/bnb-chain/bsc/pull/1840/commits/1e556d220c3a40286dd90b37a08bb5fc659ee6ee)
* [all: remove ethash pow, only retain shims needed for consensus and te](https://github.com/bnb-chain/bsc/pull/1840/commits/dde2da0efb8e9a1812f470bc43254134cd1f8cc0)
* [les: remove obsolete code related to PoW header syncing (#27737)](https://github.com/bnb-chain/bsc/pull/1840/commits/d4d88f9bce13ca9310bf28f5f26ea9f1915ba90d)
* remove diffsync
#### Others
* [accounts/usbwallet: support Ledger Nano S Plus and FTS (#25933)](https://github.com/bnb-chain/bsc/pull/1840/commits/7eafbec741d124bc53896f6bfc2408b70ab9a82a)
* [accounts/scwallet: fix keycard data signing error (#25331)](https://github.com/bnb-chain/bsc/pull/1840/commits/0c66d971e7f3557df297cbe450fe7fc7826017be)
* [core/state: replace fastcache code cache with gc-friendly structure (…](https://github.com/bnb-chain/bsc/pull/1840/commits/5fded040372784985265f83f33f15cb6a51bebdb)
* [ci: disable coverage reporting in appveyor and travis](https://github.com/bnb-chain/bsc/pull/1840/commits/a0d63bc69a659009a3884f50c563a0e58483cdd0)
* [all: change chain head markers from block to header (#26777)](https://github.com/bnb-chain/bsc/pull/1840/commits/cd31f2dee2843776e485769ce85e0524716199bc)
* [core, miner: revert block gas counter in case of invalid transaction](https://github.com/bnb-chain/bsc/pull/1840/commits/77e33e5a49be99130a02dc72d6a0e4739fdd44d6)
* [signer/core: accept all solidity primitive types for EIP-712 signing](https://github.com/bnb-chain/bsc/pull/1840/commits/02796f6bee7e014fd16ad39f0bcd3b665b51e0bb)
* [internal/ethapi: make EstimateGas use[latest] block by default (#24363)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b66d47449f61e9ebaf9e1db3ed290b59844d4c1)
* [miner: suspend miner if node is syncing (#27218)](https://github.com/bnb-chain/bsc/pull/1840/commits/d4961881d7c92603f591f9cb8c705d00d8cbdfc0)
* [all: move main transaction pool into a subpool (#27463)](https://github.com/bnb-chain/bsc/pull/1840/commits/d40a255e973775575d8d16456252f93ac75c09f0)
* [core/txpool/blobpool: 4844 blob transaction pool (#26940)](https://github.com/bnb-chain/bsc/pull/1840/commits/1662228ac68325b4024e0cb6a4ce7dde27eb4c2d)
* [eth: send big transactions by announce/retrieve only (#27618)](https://github.com/bnb-chain/bsc/pull/1840/commits/f5d3d486e459dce29130576ae88f2324ad586b50)
* [core/rawdb: support freezer batch read with no size limit (#27687)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b1f97e151e8b34a0a0d528a3472e27de1d12a9c)
@@ -61,7 +61,7 @@ Many of the below are the same as or similar to go-ethereum.
For prerequisites and detailed build instructions please read the [Installation Instructions](https://geth.ethereum.org/docs/getting-started/installing-geth).
Building `geth` requires both a Go (version 1.19 or later) and a C compiler (GCC 5 or higher). You can install
Building `geth` requires both a Go (version 1.20 or later) and a C compiler (GCC 5 or higher). You can install
them using your favourite package manager. Once the dependencies are installed, run
```shell
@@ -97,7 +97,7 @@ directory.
| `abigen` | Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://docs.soliditylang.org/en/develop/abi-spec.html) with expanded functionality if the contract bytecode is also available. However, it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://geth.ethereum.org/docs/dapp/native-bindings) page for details. |
| `bootnode` | Stripped down version of our Ethereum client implementation that only takes part in the network node discovery protocol, but does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks. |
| `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow isolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug run`). |
| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://eth.wiki/en/fundamentals/rlp)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user-friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://ethereum.org/en/developers/docs/data-structures-and-encoding/rlp)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user-friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
## Running `geth`
@@ -150,13 +150,25 @@ unzip testnet.zip
Download latest chaindata snapshot from [here](https://github.com/bnb-chain/bsc-snapshots). Follow the guide to structure your files.
Note: if you can not download the chaindata snapshot and want to sync from genesis, you have to generate the genesis block first, you have already get the genesis.json in Step 2.
So just run: `geth --datadir <datadir> init ./genesis.json`
So just run:
``` shell
## It will init genesis with Hash-Base Storage Scheme by default.
geth --datadir <datadir> init ./genesis.json
## It will init genesis with Path-Base Storage Scheme.
@@ -180,9 +192,9 @@ This tool is optional and if you leave it out you can always attach to an alread
More details about [running a node](https://docs.bnbchain.org/docs/validator/fullnode) and [becoming a validator](https://docs.bnbchain.org/docs/validator/create-val)
*Note: Although there are some internal protective measures to prevent transactions from
crossing over between the main network and test network, you should make sure to always
use separate accounts for play-money and real-money. Unless you manually move
*Note: Although some internal protective measures prevent transactions from
crossing over between the main network and test network, you should always
use separate accounts for play and realmoney. Unless you manually move
accounts, `geth` will by default correctly separate the two networks and will not make any
accounts available between them.*
@@ -195,7 +207,7 @@ configuration file via:
$ geth --config /path/to/your_config.toml
```
To get an idea how the file should look like you can use the `dumpconfig` subcommand to
To get an idea of how the file should look like you can use the `dumpconfig` subcommand to
export your existing configuration:
```shell
@@ -247,9 +259,38 @@ APIs!**
- [BSC-Deploy](https://github.com/bnb-chain/node-deploy/): deploy tool for setting up both BNB Beacon Chain, BNB Smart Chain and the cross chain infrastructure between them.
- [BSC-Docker](https://github.com/bnb-chain/bsc-docker): deploy tool for setting up local BSC cluster in container.
## Running a bootnode
Bootnodes are super-lightweight nodes that are not behind a NAT and are running just discovery protocol. When you start up a node it should log your enode, which is a public identifier that others can use to connect to your node.
First the bootnode requires a key, which can be created with the following command, which will save a key to boot.key:
```
bootnode -genkey boot.key
```
This key can then be used to generate a bootnode as follows:
Please see [Releases](https://github.com/binance-chain/bsc/releases). We recommend using the [most recently released version](https://github.com/binance-chain/bsc/releases/latest).
Please see [Releases](https://github.com/bnb-chain/bsc/releases). We recommend using the [most recently released version](https://github.com/bnb-chain/bsc/releases/latest).
## Audit reports
Audit reports are published in the `docs` folder: https://github.com/binance-chain/bsc/tree/master/docs/audits
Audit reports are published in the `docs` folder: https://github.com/bnb-chain/bsc/tree/master/docs/audits
@@ -17,6 +17,6 @@ Audit reports are published in the `docs` folder: https://github.com/binance-cha
**Please do not file a public ticket** mentioning the vulnerability.
To find out how to disclose a vulnerability in Ethereum visit [https://bugcrowd.com/binance](https://bugcrowd.com/binance) or email bounty@ethereum.org. Please read the [disclosure page](https://github.com/binance-chain/bsc/security/advisories) for more information about publicly disclosed security vulnerabilities.
To find out how to disclose a vulnerability in Ethereum visit [https://bugcrowd.com/binance](https://bugcrowd.com/binance) or email bounty@ethereum.org. Please read the [disclosure page](https://github.com/bnb-chain/bsc/security/advisories) for more information about publicly disclosed security vulnerabilities.
Use the built-in `geth version-check` feature to check whether the software is affected by any known vulnerability. This command will fetch the latest [`vulnerabilities.json`](https://geth.ethereum.org/docs/vulnerabilities/vulnerabilities.json) file which contains known security vulnerabilities concerning `geth`, and cross-check the data against its own version number.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
