* cmd, core: resolve scheme from a read-write database
* cmd, core, eth: move the scheme check in the ethereum constructor
* cmd/geth: dump should in ro mode
* cmd: reverts
This PR will allow a previously underpriced transaction back in after a timeout
of 5 minutes. This will block most transaction spam but allow for transactions to
be re-broadcasted on networks with less transaction flow.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
# eth/fetcher/tx_fetcher.go
This is a minor refactor in preparation of changes to range verifier. This PR contains no intentional functional changes but moves (and renames) the light.NodeSet
* cmd, les, tests: remove light client code
This commit removes the light client (LES) code.
Since the merge the light client has been broken and
it is hard to maintain it alongside the normal client.
We decided it would be best to remove it for now and
maybe rework and reintroduce it in the future.
* cmd, eth: remove some more mentions of light mode
* cmd: re-add flags and mark as deprecated
* cmd: warn the user about deprecated flags
* eth: better error message
* triedb/pathdb: fix async node buffer diskroot mismatches when journaling
* triedb: check whether the async flush is done
* fix: generate new eth config
---------
Co-authored-by: VM <arimas@foxmail.com>
* core: LoadChainConfig return predefined config for built-in net firstly
* cmd/geth: add a warn message for chain config in the configuration file
* consensus/parlia: change chain config log level when New parlia
* core: fix code style
* cmd, core, params: add support for the Holesky testnet
* cmd/devp2p: add support for holesky for the dns crawler
# Conflicts:
# cmd/devp2p/nodesetcmd.go
# cmd/geth/main.go
# cmd/utils/flags.go
# core/genesis.go
# params/bootnodes.go
# params/config.go
* cmd/geth: standardize the action of importing bls account
* rename bLSAccountPasswordFileFlag to blsAccountPasswordFileFlag
* use password of wallet as account password when new an account or importing a raw key
accountTrieCache and storageTrieCache were introduced in this PR:
https://github.com/bnb-chain/bsc/pull/257, which is to improve performance.
Actually the performance gain is quite limited, as there is already dirty
and clean cache for trie node.
And after big merge, these 2 cache can not be used when PBSS is enabled.
So remove these code to simplify the logic.
* go.mod: upgrade prysm and the indrect dependency
prysm from v4.0.2 to v4.0.8, and run go mod tidy
* ci: upgrade go version from 1.19 to 1.20
* go-version: upgrade from v1.19 to v1.20
there is some dependency on go v1.20, such as go-libp2p v0.27.8
and also run go mod tidy
* dependency: upgrade docker version for security
it is not a big issue, since docker is only used for test purpose.
* rand: update the usage of math/rand after golang v1.20
2 APIs of math/rand module were deprecated since golang v1.20.
that is: rand.Seed() and rand.Read(), refer: ettps://pkg.go.dev/math/rand
"rand.Seed(seed int64)" has been replaced by: "r := rand.New(rand.NewSource(seed int64))",
need to initialize it with an instance before use
"rand.Read()" has been replaced by "crypto/rand.Read()"
* readme: need golang v1.20+ to build bsc
This changes implements faster post-selfdestruct iteration of storage slots for deletion, by using snapshot-storage+stacktrie to recover the trienodes to be deleted. This mechanism is only implemented for path-based schema.
For hash-based schema, the entire post-selfdestruct storage iteration is skipped, with this change, since hash-based does not actually perform deletion anyway.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
When MatcherSession encounters an error, it attempts to close the session.
Closing waits for all goroutines to finish, including the 'distributor'.
However, the distributor will not exit until all requests have returned.
This patch fixes the issue by delivering the (empty) result to the distributor
before calling Close().
* parlia: reject header with `WithdrawalsHash` before shanghai fork
* log: output chainconfig when start up
* eth: fix TestOptionMaxPeersPerIP failure of goroutine order
* fix: crash of highestVerifiedHeader
* fix: panic of blobpool
* fix: genesis set up
* 1. modify NewDatabaseWithNodeDB to upstream
2. fix race use of hasher in statedb
3. fix use wrong value when updateTrie
* fix dir legacypool
* fix dir blobpool
* fix dir vote
* remove diffsync related code
* fix core/state/snapshot
* disable pipeCommit for now
* fix applyTransaction for bloom setting
* CI: fast finality in gasprice test
* CI: diffFetcher was removed
* CI: downloader, remove beaconsync test
* CI: no beaconsync in downloader, remove a failed case
TestCheckpointChallenge was removed in:
https://github.com/ethereum/go-ethereum/pull/27147
since after merge, it is useless for ethereum, but might be useful for BSC.
disable the case right now, as it is not a big issue.
* CI: bsc protocol decHandlers
* CI: receipt Bloom process
* 1. skip CheckConfigForkOrder for non-parlia engine
2. all test cases in core work well now
cd core && go test ./... -v
* fix test cases in trie dir
* CI: no beaconsync in downloader, remove a failed case(redo)
* fix dir miner
* fix dir cmd/geth
* CI: filter test, BaseFee & Finality
* fix dir graphql
* remove diffStore
* fix ethclient
* fix TestRPCGetTransactionReceipt
* fix dir internal
* ut add dir ethstats and signer
* disable pipeCommit thoroughly; fix concurrent map iteration and map write in statedb
* CI: fix snap sync
it could be changed by mistake
* fix tests/Run to generate snapshot
* prepare for merge
* remove useless
* use common hasher in getDeletedStateObject, no race here
* an critical comment for state.Prepare
* do not copy nil accessList
* add omitempty tag for unused new fields of core.Genesis
* remove totalFees
* calculate fees before FinalizeAndAssemble
* revert interface Finalize of consensus
* do not double gas limit upon london block
* use Leveldb as default
* Revert "remove diffStore"
This reverts commit df343b1374.
* Revert "remove diffsync related code"
This reverts commit 8d84b81fea.
* compile pass after revert
* remove diffsync
* fix dir eth/protocols/trust
* fix TestFastNode
* decHandlers for trust protocol
* keep persist diff in test
FastFinality puts more infor into the header.extra field to keep vote information.
For mainnet, on epoch height, it could be 1526 bytes, which was 517 bytes before.
So the hardcoded 700 bytes for header could be no longer enough, increase it by
2 times would be enough.
this bug could cause P2P sync failure for nodes that are lagging behind, since they
would request access of ancient db, and GetBlockHeaders could be failed.
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
Context: The UpdateContractCode method was introduced for the state storage commitment
schemes that include the whole code for their commitment computation. It must therefore be called
before the root hash is computed at the end of IntermediateRoot.
This should have no impact on the MPT since, in this context, the method is a no-op.
This adds support for the "yParity" field in transaction objects returned by RPC
APIs. We somehow forgot to add this field even though it has been in the spec for
a long time.
This change rearranges the accessor methods in block.go and fixes some minor issues with
the copy-on-write logic of block data. Fixed issues:
- Block.WithWithdrawals did not create a shallow copy of the block.
- Block.WithBody copied the header unnecessarily, and did not preserve withdrawals.
However, the bugs did not affect any code in go-ethereum because blocks are *always*
created using NewBlockWithHeader().WithBody().WithWithdrawals()
* all: implement path-based state scheme
* all: edits from review
* core/rawdb, trie/triedb/pathdb: review changes
* core, light, trie, eth, tests: reimplement pbss history
* core, trie/triedb/pathdb: track block number in state history
* trie/triedb/pathdb: add history documentation
* core, trie/triedb/pathdb: address comments from Peter's review
Important changes to list:
- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated
* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache
* core/rawdb: fix tests
* trie/triedb/pathdb: rename metrics, change clean cache key
* trie/triedb: manage the clean cache inside of disk layer
* trie/triedb/pathdb: move journal function
* trie/triedb/path: fix tests
* trie/triedb/pathdb: fix journal
* trie/triedb/pathdb: fix history
* trie/triedb/pathdb: try to fix tests on windows
* core, trie: address comments
* trie/triedb/pathdb: fix test issues
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Sets the `currentExcessBlobGas` from env, alternatively calculates it based on `parentExcessBlobGas` and `parentBlobGasUsed`. It then emits the `currentExcessBlobGas` and `currentBlobGasUsed` into the output, to be used as parent-values for a future iteration.
Closes#27785Closes#27783
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* tests: split up state test execution
* Revert "tests: split up state test execution"
This reverts commit 96017c248c85d24e93ad013a2bbe8b38c99327c0.
* build: bump test timeout to 20 minutes
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core/blobpool: implement txpool for blob txs
* core/txpool: track address reservations to notice any weird bugs
* core/txpool/blobpool: add support for in-memory operation for tests
* core/txpool/blobpool: fix heap updating after SetGasTip if account is evicted
* core/txpool/blobpool: fix eviction order if cheap leading txs are included
* core/txpool/blobpool: add note as to why the eviction fields are not inited in reinject
* go.mod: pull in inmem billy form upstream
* core/txpool/blobpool: fix review commens
* core/txpool/blobpool: make heap and heap test deterministic
* core/txpool/blobpool: luv u linter
* core/txpool: limit blob transactions to 16 per account
* core/txpool/blobpool: fix rebase errors
* core/txpool/blobpool: luv you linter
* go.mod: revert some strange crypto package dep updates
Hertz is equal to Berlin and London
- Plato will be enabled on BSC mainnet at height: 30720096
- Hertz will be enabled on BSC mainnet at height: 31302048
Hertz is for ecosystem compatibility, to enable some customized EIPs of Berlin&London on BSC
This is a spin-out from the EIP-4844 devnet branch, containing just the Engine API modifications
and nothing else. The newPayloadV3 endpoint won't really work in this version, but we need the
data structures for testing so I'd like to get this in early.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change removes PoW header syncing related code from LES and also deletes
duplicated packages les/catalyst, les/downloader and les/fetcher. These package copies
were created because people wanted to make changes in their eth/ counterparts, but weren't
able to adapt LES code to the API changes.
EIP-6780: SELFDESTRUCT only in same transaction
> SELFDESTRUCT will recover all funds to the caller but not delete the account, except when called in the same transaction as creation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This updates the reference tests to the latest version and also adds logic
to process EIP-4844 blob transactions into the state transition. We are now
passing most Cancun fork tests.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
It is usually best to set GOMAXPROCS to the number of available CPU cores. However, setting
it like that does not work well when the process is quota-limited to a certain number of CPUs.
The automaxprocs library configures GOMAXPROCS, taking such limits into account.
This changes the port mapping procedure such that, when the requested port is unavailable
an alternative port suggested by the router is used instead.
We now also repeatedly request the external IP from the router in order to catch any IP changes.
Co-authored-by: Felix Lange <fjl@twurst.com>
The struct logger (or opcode tracer) was missing the return data field even
if this was explicitly enabled by user via `"enableReturnData": true` in the config.
This PR fixes this issue.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change adds the ability to perform reads from freezer without size limitation. This can be useful in cases where callers are certain that out-of-memory will not happen (e.g. reading only a few elements).
The previous API was designed to behave both optimally and secure while servicing a request from a peer, whereas this change should _not_ be used when an untrusted peer can influence the query size.
This simplifies the code that initializes the discovery a bit, and
adds new flags for enabling/disabling discv4 and discv5 separately.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
Back before #27178 , we spun up a number of ethash verifiers to verify headers. So we also had tests to ensure that we were indeed able to abort verification even if we had multiple workers running.
With PR #27178, we removed the parallelism in verification, and these tests are now failing, since we now just sequentially fire away the results as fast as possible on one routine.
This change removes the (sometimes failing) tests
This removes text parsing in leveldb metrics collection code. All metrics
can now be accessed through the stats API provided by leveldb.
We also add new gauge-typed metrics that count the number of tables at each level.
---------
Co-authored-by: Exca-DK <Exca-DK@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
* core/state, light, les: make signature of ContractCode hash-independent
* push current state for feedback
* les: fix unit test
* core, les, light: fix les unittests
* core/state, trie, les, light: fix state iterator
* core, les: address comments
* les: fix lint
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: lx <92799281+brilliant-lx@users.noreply.github.com>
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Verkle trees store the code inside the trie. This PR changes the interface to pass the code, as well as the dirty flag to tell the trie package if the code is dirty and needs to be updated. This is a no-op for the MPT and the odr trie.
In all other UDPv4 methods, the deadline is checked first. It seems weird to me that ping is an exception. Deadline comparison is also less resource intensive.
Co-authored-by: Exca-DK <Exca-DK@users.noreply.github.com>
This changes the eth_getProof method implementation to re-encode the requested
storage keys, canonicalizing them in the response. For backwards-compatibility reasons,
go-ethereum accepts non-canonical hex keys. Accepting them is fine, but we should
not mirror invalid inputs into the output.
Closes#27306
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This is likely the culprit behind several data corruption issues, e.g. where data has been
written to the freezer, but the deletion from pebble does not go through due to process
crash.
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
* go.mod: update kzg libraries to use big-endian
* go.sum: ran go mod tidy
* core/testdata/precompiles: fix blob verification test
* core/testdata/precompiles: fix blob verification test
Package rpc uses cgo to find the maximum UNIX domain socket path
length. If exceeded, a warning is printed. This is the only use of cgo in this
package. It seems excessive to depend on cgo just for this warning, so
we now hard-code the usual limit for Linux instead.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ensures Reheap will be called even before the London fork activates.
Since Reheap would otherwise only be called through `SetBaseFee` after London,
the list would just keep growing if the fork was not enabled or not reached yet.
* all: move main transaction pool into a subpool
* go.mod: remove superfluous updates
* core/txpool: review fixes, handle txs rejected by all subpools
* core/txpool: typos
The logs in this function are pulled straight from disk in rawdb.ReadRawReceipts and
also modified in receipts.DeriveFields, so removing the copy should be fine.
We had to do this workaround because it wasn't possible to export typed arrays from
JS to []byte. This was added in dop251/goja@2352993, so we can use the better way now.
This adds two ways to check for subscription support. First, one can now check
whether the transport method (HTTP/WS/etc.) is capable of subscriptions using
the new Client.SupportsSubscriptions method.
Second, the error returned by Subscribe can now reliably be tested using this
pattern:
sub, err := client.Subscribe(...)
if errors.Is(err, rpc.ErrNotificationsUnsupported) {
// no subscription support
}
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds server-side limits for JSON-RPC batch requests. Before this change, batches
were limited only by processing time. The server would pick calls from the batch and
answer them until the response timeout occurred, then stop processing the remaining batch
items.
Here, we are adding two additional limits which can be configured:
- the 'item limit': batches can have at most N items
- the 'response size limit': batches can contain at most X response bytes
These limits are optional in package rpc. In Geth, we set a default limit of 1000 items
and 25MB response size.
When a batch goes over the limit, an error response is returned to the client. However,
doing this correctly isn't always possible. In JSON-RPC, only method calls with a valid
`id` can be responded to. Since batches may also contain non-call messages or
notifications, the best effort thing we can do to report an error with the batch itself is
reporting the limit violation as an error for the first method call in the batch. If a batch is
too large, but contains only notifications and responses, the error will be reported with
a null `id`.
The RPC client was also changed so it can deal with errors resulting from too large
batches. An older client connected to the server code in this PR could get stuck
until the request timeout occurred when the batch is too large. **Upgrading to a version
of the RPC client containing this change is strongly recommended to avoid timeout issues.**
For some weird reason, when writing the original client implementation, @fjl worked off of
the assumption that responses could be distributed across batches arbitrarily. So for a
batch request containing requests `[A B C]`, the server could respond with `[A B C]` but
also with `[A B] [C]` or even `[A] [B] [C]` and it wouldn't make a difference to the
client.
So in the implementation of BatchCallContext, the client waited for all requests in the
batch individually. If the server didn't respond to some of the requests in the batch, the
client would eventually just time out (if a context was used).
With the addition of batch limits into the server, we anticipate that people will hit this
kind of error way more often. To handle this properly, the client now waits for a single
response batch and expects it to contain all responses to the requests.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/txpool: abstraction prep work for secondary pools (blob pool)
* core/txpool: leave subpool concepts to a followup pr
* les: fix tests using hard coded errors
* core/txpool: use bitmaps instead of maps for tx type filtering
* cmd/evm: make evm blocktest output logs if so instructed
* Apply suggestions from code review
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This changes the journal logic to mark the state object dirty immediately when it
is reset.
We're mostly adding this change to appease the fuzzer. Marking it dirty immediately
makes no difference in practice because accounts will always be modified by EVM
right after creation.
Continuing with a series of PRs to make the Trie interface more generic, this PR moves
the RLP encoding of storage slots inside the StateTrie and light.Trie implementations,
as other types of tries don't use RLP.
* p2p/discover: remove ReadRandomNodes
Even though it's public, this method is not callable by code outside of
package p2p/discover because one can't get a valid instance of Table.
* p2p/discover: add Table.Nodes
* p2p/discover: make Table settings configurable
In unit tests and externally developed cmd/devp2p test runs, it can be
useful to tune the timer intervals used by Table.
Drop the notions of uncles, and disables activities while syncing
- Disable activities (e.g. generate pending state) while node is syncing,
- Disable empty block submission (but empty block is still kept for payload building),
- Drop uncle notion since (ethash is already deprecated)
Deserialize hex keys early to shortcut on invalid input, and re-use the account storageTrie for each proof for each proof in the account, preventing repeated deep-copying of the trie.
Closes#27308
--------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
eth: make StorageRangeAt take a block hash or number
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
RPC methods `eth_getHeaderBy*` returned a size value which was meant for internal
processes. Please instead use `size` field returned by `eth_getBlockBy*` if you're interested
in the RLP encoded storage size of the block.
Signed-off-by: jsvisa <delweng@gmail.com>
Here is the list:
a.BEP-227: Implement EIP-3198: BASEFEE opcode
b.BEP-226: Implement EIP-1559 with base fee of 0
c.BEP-228: Implement EIP-3541: Prevent deploying contracts starting with 0xEF
d.BEP-212: Implement EIP-3529: Reduction in Refunds
ethclient accepts certain negative block number values as specifiers for the "pending",
"safe" and "finalized" block. In case of "pending", the value accepted by ethclient (-1)
did not match rpc.PendingBlockNumber (-2).
This wasn't really a problem, but other values accepted by ethclient did match the
definitions in package rpc, and it's weird to have this one special case where they don't.
To fix it, we decided to change the values of the constants rather than changing ethclient.
The constant values are not otherwise significant. This is a breaking API change, but we
believe not a dangerous one.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
implements the ability to run several state-tests in one instance. By not providing a statetest path to the `evm statetest` command, the path(s) will instead be read from `stdin`.
Upgrades graphiql to v2.4.4. The interface has become much nicer, and there are extra features like tabs, history, dark mode etc.
This change also now uses golang embed to bundle the resources.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Enable following Berlin EIPs on BSC and introduce HertzBlock
** EIP-2565: ModExp Gas Cost
** EIP-2929: Gas cost increases for state access opcodes
** EIP-2718: Typed Transaction Envelope
** EIP-2930: Optional access lists
One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237)
This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests.
This changes the RPC server to ignore methods using *context.Context as parameter
and *error as return value type. Methods with such types would crash the server when
called.
This PR adds a staleness-check to AccountRLP, before checking the bloom-filter and potentially going directly into the disklayer.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* EstimateGas should use LatestBlockNumber by default
* graphql: default to use latest for gas estimation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cryto/kzg4844: pull in the C and Go libs for KZG cryptography
* go.mod: pull in the KZG libraries
* crypto/kzg4844: add basic becnhmarks for ballpark numbers
* cmd, crypto: integrate both CKZG and GoKZG all the time, add flag
* cmd/utils, crypto/kzg4844: run library init on startup
* crypto/kzg4844: make linter happy
* crypto/kzg4844: push missing file
* crypto/kzg4844: fully disable CKZG but leave in the sources
* build, crypto/kzg4844, internal: link CKZG by default and with portable mode
* crypto/kzg4844: drop verifying the trusted setup in gokzg
* internal/build: yolo until it works?
* cmd/utils: make flag description friendlier
Co-authored-by: Martin Holst Swende <martin@swende.se>
* crypto/ckzg: no need for double availability check
* build: tiny flag cleanup nitpick
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
When block import fails, the error displays the number of the first block past the import batch, not the number of the failing block. This change fixes this problem by identifying which blocks fails and reporting its number.
The first validator in snap.Recents is shifted out and allowed to seal the block
already. However, when determining if we should slash the validator in Finalize
and FinalizeAndAssemble, we mark that first validator as unable to seal block
and don't slash this validator. This commit fixes that bug and creates a
separate helper function to check if the validator recently signed a block for
consistency in all places the check happens.
rename parameter
In this case, the naming of "extapi" might create some confusion. Although it represents an External Signer Backend, its name could be mistaken for an API. In reality, it is a backend instance used for communicating with external signers. A better naming choice could be "extBackend" or "externalBackend" to more accurately describe that it is a backend instance rather than an API.
This removes the 'time' field from logs, as well as from the tracer interface. This change makes the trace output deterministic. If a tracer needs the time they can measure it themselves. No need for evm to do this.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR modifies the interface for the results of `debug_traceBlock` and `debug_traceCall` by adding the `txHash`, allowing users to identify which transaction's trace result corresponds to.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This adds logic to prepend 'M' or 'E' to Solidity identifiers when they would
otherwise violate Go identifier naming rules.
Closes#26972
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Adds an optional config parameter to eth_call which allows users to override block context fields (same functionality that was added to traceCall in #24871)
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes TALKREQ message processing to run the handler on separate goroutine,
instead of running on the main discv5 dispatcher goroutine. It's better this way because
it allows the handler to perform blocking actions.
I'm also adding a new method TalkRequestToID here. The method allows implementing
a request flow where one node A sends TALKREQ to another node B, and node B later
sends a TALKREQ back. With TalkRequestToID, node B does not need the ENR of A to
send its request.
Makes the `geth account ... ` commands usable even if a geth-process is already executing, since the account commands do not read the chaindata, it was not required for those to use the same locking mechanism.
---
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* trie: add node type common package
In trie/types package, a few node wrappers are defined, which will be used
in both trie package, trie/snap package, etc. Therefore, a standalone common
package is created to put these stuffs.
* trie: rename trie/types to trie/trienode
This is a breaking GraphQL API change. All numeric values are now encoded as
hex strings. The motivation for this change is matching JSON-RPC outputs more
closely.
Numbers in query parameters are accepted as both decimal integers and hex strings.
* all: remove notion of trusted checkpoints in the post-merge world
* light: remove unused function
* eth/ethconfig, les: remove unused config option
* les: make linter happy
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd/utils, node: switch to Pebble as the default db if none exists
* node: fall back to LevelDB on platforms not supporting Pebble
* core/rawdb, node: default to Pebble at the node level
* cmd/geth: fix some tests explicitly using leveldb
* ethdb/pebble: allow double closes, makes tests simpler
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
Follow-up to #26697, makes the crawler less verbose on route53-based scenarios.
It also changes the loglevel from debug to info on Updates, which are typically the root, and can be interesting to see.
The EIP150Hash was an idea where, after the fork, we hardcoded the forked hash as an extra defensive mechanism. It wasn't really used, since forks weren't contentious and for all the various testnets and private networks it's been a hassle to have around.
This change removes that config field.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
* core/vm: deepcopy jumptable when enabling extra eips
When the interpreter is configured to use extra-eips,
this change makes it so that all the opcodes are deep-copied,
to prevent accidental modification of the 'base' jumptable.
Original-auther: yihuang <huang@crypto.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/vm: implement EIP-3860: Limit and meter initcode
Implementation of https://eips.ethereum.org/EIPS/eip-3860, limit and meter initcode.
Most of this change takes the go-ethereum implementation as reference.
Original-author: Andrei Maiboroda <andrei@ethereum.org>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core/vm: implement EIP-3855: PUSH0 instruction
* core/vm: Implement PUSH0
* Move PUSH0 to enable3855
* Add method doc
Original-author: Alex Beregszaszi <alex@rtfs.hu>
* core/vm: enable EIP-3855 (PUSH0) in Boneh
Original-author: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
* core/vm: Fix issue of incorrect instructionSet used for jump_table
Also update the related test case for gas change.
* core/vm: fix test cases issues that not suitable for boneh
* core/vm: reuse ErrMaxInitCodeSizeExceeded as error message
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR unifies the error handling in miner.
Whenever an error occur while applying a transaction, the transaction should be regarded as invalid and all following transactions from the same sender not executable because of the nonce restriction. The only exception is the `nonceTooLow` error which is handled separately.
Prior to this change, it was possible that transactions are erroneously deemed as 'future' although they are in fact 'pending', causing them to be dropped due to 'future' not being allowed to replace 'pending'.
This change fixes that, by doing a more in-depth inspection of the queue.
Use the new atomic types in package eth/tracers
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Makes the float-gauges lock-free
name old time/op new time/op delta
CounterFloat64Parallel-8 1.45µs ±10% 0.85µs ± 6% -41.65% (p=0.008 n=5+5)
---------
Co-authored-by: Exca-DK <dev@DESKTOP-RI45P4J.localdomain>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR removes the Debug field from vmconfig, making it so that if a tracer is set, debug=true is implied.
---------
Co-authored-by: 0xTylerHolmes <tyler@ethereum.org>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
This includes a semantic change to the `callTracer` as well as `flatCallTracer`.
The value of field `gas` in the **first** call frame will change as follows:
- It previously contained gas available after initial deductions (i.e. tx costs)
- It will now contain the full tx gasLimit value
Signed-off-by: jsvisa <delweng@gmail.com>
* core/state: Remove prefetcher on stop
* core/blockchain: Wait for `triedb.Dereference` in `writeBlockWithState`
* eth/protocols/diff: Fix index mismatch in `TestGetDiffLayers`
Currently, most of transaction validation while holding the txpool mutex: one exception being an early-on signature check.
This PR changes that, so that we do all non-stateful checks before we entering the mutex area. This means they can be performed in parallel, and to enable that, certain fields have been made atomic bools and uint64.
This change enables log rotation, which can be activated using the flag --log.rotate. Additional parameters that can be given are:
- log.maxsize to set maximum size before files are rotated,
- log.maxbackups to set how many files are retailed,
- log.maxage to configure max age of rotated files,
- log.compress whether to compress rotated files
The way to configure location of the logfile(s) is left unchanged, via the `log.logfile` parameter.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Currently the t8n tool uses the same block number for the current block and its parent while calculating the base fee. This causes incorrect base fee calculation for the london fork block. This commit sets the parent block number to be one less than the current block number
Adds error handling for the case that UnpackLog or UnpackLogIntoMap is called with a log that has zero topics.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Removes the new --log.logfmt directive and hides --log.json, replacing both with log.format=(json|logfmt|terminal). The hidden log.json option is still respected if log.format is not specified for backwards compatibility.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR fixes OOM panic in the callTracer as well as panicing on
opcode validation errors (e.g. stack underflow) in callTracer and
prestateTracer.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Fixes multiple data races caused by the fact that resolving fields are done concurrently by the graphql library. It also enforces caching at the stateobject level for account fields.
This change renames StateTrie methods to remove the Try* prefix.
We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change switches to use the smaller influxdata/influxdb1-client package instead of depending on the whole infuxdb package. The new smaller client is very similar to the influxdb-v2 client, which made it possible to refactor the two reporters to reuse code a lot more.
This PR adds counter metrics for the CPU system and the Geth process.
Currently the only metrics available for these items are gauges. Gauges are
fine when the consumer scrapes metrics data at the same interval as Geth
produces new values (every 3 seconds), but it is likely that most consumers
will not scrape that often. Intervals of 10, 15, or maybe even 30 seconds
are probably more common.
So the problem is, how does the consumer estimate what the CPU was doing in
between scrapes. With a counter, it's easy ... you just subtract two
successive values and divide by the time to get a nice, accurate average.
But with a gauge, you can't do that. A gauge reading is an instantaneous
picture of what was happening at that moment, but it gives you no idea
about what was going on between scrapes. Taking an average of values is
meaningless.
This changes the Trie interface to add the plain account address as a
parameter to all storage-related methods.
After the introduction of the TryAccount* functions, TryGet, TryUpdate and
TryDelete are now only meant to read an account's storage. In their current
form, they assume that an account storage is stored in a separate trie, and
that the hashing of the slot is independent of its account's address.
The proposed structure for a stateless storage breaks these two
assumptions: the hashing of a slot key requires the address and all slots
and accounts are stored in a single trie.
This PR therefore adds an address parameter to the interface. It is ignored
in the MPT version, so this change has no functional impact, however it
will reduce the diff size when merging verkle trees.
With #25287 we made it so that preimages were not recorded by default. This had the side effect that the evm command is no longer able to dump state since it does a preimage lookup to determine the address represented by a key.
This change enables the recording of preimages when the dump command is given.
The meter for "for measuring the effective amount of data read" within the freezertable was never updated. This change remedies that.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
When interacting with geth as a library to e.g. produce state tests, it is desirable to obtain the consensus-correct jumptable definition for a given fork. This changes adds accessors so the instructionset can be obtained and characteristics about opcodes can be inspected.
This adds built-in support in package rlp for encoding, decoding and generating code dealing with uint256.Int.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Makes clear the distinction between Finalize and FinalizedAndAssemble:
- In Finalize function, a series of state operations are applied according to consensus rules. The statedb is mutated and the root hash can be checked and compared afterwards.
This function should be used in block processing(receive afrom network and apply it locally) but not block generation.
- In FinalizeAndAssemble function, after applying state mutations, the block is also to be assembled with the latest
state root computed, updating the header.
This function should be used in block generation only.
This adds two new rules to the transaction pool:
- A future transaction can not evict a pending transaction.
- A transaction can not overspend available funds of a sender.
---
Co-authored-by: dwn1998 <42262393+dwn1998@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Here, the core.Message interface turns into a plain struct and
types.Message gets removed.
This is a breaking change to packages core and core/types. While we do
not promise API stability for package core, we do for core/types. An
exception can be made for types.Message, since it doesn't have any
purpose apart from invoking the state transition in package core.
types.Message was also marked deprecated by the same commit it
got added in, 4dca5d4db7 (November 2016).
The core.Message interface was added in December 2014, in commit
db494170dc, for the purpose of 'testing' state transitions. It's the
same change that made transaction struct fields private. Before that,
the state transition used *types.Transaction directly.
Over time, multiple implementations of the interface accrued across
different packages, since constructing a Message is required whenever
one wants to invoke the state transition. These implementations all
looked very similar, a struct with private fields exposing the fields
as accessor methods.
By changing Message into a struct with public fields we can remove all
these useless interface implementations. It will also hopefully
simplify future changes to the type with less updates to apply across
all of go-ethereum when a field is added to Message.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the test to match the comment description. Using timestampedConfig in this test case is incorrect, the comment says 'local is at Gray Glacier' and isn't aware of more forks.
Accept all primitive types in Solidity for EIP-712 from intN, uintN, intN[], uintN[] for N as 0 to 256 in multiples of 8
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This ensures the "withdrawals" field will always be present in responses
to getPayloadBodiesByRangeV1 and getPayloadBodiesByHashV1.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR mitigates an issue with Ledger's on-device RLP deserialization, see
https://github.com/LedgerHQ/app-ethereum/issues/409
Ledger's RLP deserialization code does not validate the length of the RLP list received,
and it may prematurely enter the signing flow when a APDU chunk boundary falls immediately
before the EIP-155 chain_id when deserializing a transaction. Since the chain_id is
uninitialized, it is 0 during this signing flow. This may cause the user to accidentally
sign the transaction with chain_id = 0. That signature would be returned from the device 1
packet earlier than expected by the communication loop. The device blocks the
second-to-last packet waiting for the signer flow, and then errors on the successive
packet (which contains the chain_id, zeroed r, and zeroed s)
Since the signature's early arrival causes successive errors during the communication
process, geth does not parse the improper signature produced by the device, and therefore
no improperly-signed transaction can be created. User funds are not at risk.
We mitigate by selecting the highest chunk size that leaves at least 4 bytes in the
final chunk.
Checks that Transaction.MarshalJSON and newRPCTransaction JSON output can be parsed by Transaction.UnmarshalJSON
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change fixes a flaw where, in certain scenarios, the block sealer did not accurately reset the remaining gas after failing to include an invalid transaction. Fixes#26791
This PR changes metrics collection to actually measure the time interval between collections, rather
than assume 3 seconds. I did some ad hoc profiling, and on slower hardware (eg, my Raspberry Pi 4)
I routinely saw intervals between 3.3 - 3.5 seconds, with some being as high as 4.5 seconds. This
will generally cause the CPU gauge readings to be too high, and in some cases can cause impossibly
large values for the CPU load metrics (eg. greater than 400 for a 4 core CPU).
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a race in TestNewPayloadOnInvalidTerminalBlock where setting the TTD raced with
the miner. Solution: set the TTD on the blockchain config not the genesis config.
Also fixes a race in CopyHeader which resulted in race reports all over the place.
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
This fixes an issue where the withdrawal index was not calculated correctly
for multiple withdrawals in a single block.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds support for a native call tracer with the Parity format, which outputs call frames
in a flat array. This tracer accepts the following options:
- `convertParityErrors: true` will convert error messages to match those of Parity
- `includePrecompiles: true` will report all calls to precompiles. The default
matches Parity's behavior where CALL and STATICCALLs to precompiles are excluded
Incompatibilities with Parity include:
- Parity removes the result object in case of failure. This behavior is maintained
with the exception of reverts. Revert output usually contains useful information,
i.e. Solidity revert reason.
- The `gasUsed` field accounts for intrinsic gas (e.g. 21000 for simple transfers)
and refunds unlike Parity
- Block rewards are not reported
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
The change fixes unmarshaling of JSON null results into json.RawMessage.
---------
Co-authored-by: Jason Yuan <jason.yuan@curvegrid.com>
Co-authored-by: Jason Yuan <jason.yuan869@gmail.com>
This PR contains a small portion of the full pbss PR, namely
Remove the tracer from trie (and comitter), and instead using an accessList.
Related changes to the Nodeset.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The method `GetPayloadBodiesByRangeV1` now returns "-38004: Too large request" error if the requested range is too large, according to spec
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR is a (superior) alternative to https://github.com/ethereum/go-ethereum/pull/26708, it handles deprecation, primarily two specific cases.
`rand.Seed` is typically used in two ways
- `rand.Seed(time.Now().UnixNano())` -- we seed it, just to be sure to get some random, and not always get the same thing on every run. This is not needed, with global seeding, so those are just removed.
- `rand.Seed(1)` this is typically done to ensure we have a stable test. If we rely on this, we need to fix up the tests to use a deterministic prng-source. A few occurrences like this has been replaced with a proper custom source.
`rand.Read` has been replaced by `crypto/rand`.`Read` in this PR.
This PR relaxes the block body ingress handling a bit: if block body withdrawals are missing (but expected to be empty), the body withdrawals are set to 'empty list' before being passed to upper layers.
This fixes an issue where a block passed from EthereumJS to geth was deemed invalid.
Our discovery crawler spits out a huge amount of logs, most of which is pretty non-interesting. This change moves the very verbose output to Debug, and adds a 8-second status log message giving the general idea about what's going on.
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
This is a breaking change in the tracing hooks API as well as semantics of the callTracer:
- CaptureEnter hook provided a nil value argument in case of DELEGATECALL. However to stay consistent with how delegate calls behave in EVM this hook is changed to pass in the value of the parent call.
- callTracer will return parent call's value for DELEGATECALL frames.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* fix: p2p sync with lagging peer
no need to sync with lagging peer, which could make the local chain stalling as well.
* fix: do not drop lagging peer, will retry it later.
The lagging peer is probably already the best peer with largest total difficulty.
Shoule not remove it, since p2p is a bidirectional connection, drop it could make
the peer unable to sync with this peer as well.
And the lagging peer could catch up later, so keep it.
* p2p: add lagging field in Peer
lagging peer will be connected, but won't be used to sync.
the lagging flag can be clear once the Peer updates its latest block state.
* test: fix UT compile issue
* fix: lagging peer func rename
* test: fix a UT fail of download test
errStallingPeer is replaced by errLaggingPeer in this case
* fix: lagging issue in light mode
* test: add and resolve UT of lagging peer
* common, core, eth, les, trie: make prque generic
* les/vflux/server: fixed issues in priorityPool
* common, core, eth, les, trie: make priority also generic in prque
* les/flowcontrol: add test case for priority accumulator overflow
* les/flowcontrol: avoid priority value overflow
* common/prque: use int priority in some tests
No need to convert to int64 when we can just change the type used by the
queue.
* common/prque: remove comment about int64 range
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
* log: allow tabs in log messages
This fixes a regression where panic reports in RPC handlers were quoted
because they contain tab characters.
* Update format.go
According to the spec the payloadID needs to be random or dependent on all arguments, to prevent two payloads from clashing. This change adds withdrawals into the payload derivation.
---------
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR moves core/beacon to beacon/engine so that beacon-chain related code has its own top level package which also can house the the beacon lightclient-code.
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change fixes a minor flaw in the check for ipc endpoint length. The max_path_size is the max path that an ipc endpoint can have, which is 208. However, that size concerns the null-terminated pathname, so we need to account for an extra null-character too.
This change improves reusability of the EVM struct. Two methods are added:
- SetBlockContext(...)
- SetTracer(...)
Other attributes like the TransactionContext and the StateDB can already be updated.
BlockContext and Tracer are partially not updateable right now. This change fixes it and
opens the potential to reuse an EVM struct in more ways.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth: cmd: deprecate personal namespace
* eth: cmd: move deprecation to node
* node: disable toml of enablepersonal
* node: disable personal on ipc as well
* Update node/node.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* console: error -> warn
* node: less roulette
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* dep: upgrade secp256k1 to use btcec/v2 v2.3.2 and update insecurity pkg
* build ci: upgrade go to 1.19 and golangci-lint to 1.50.1
* docs: fix format that does not follow the goimports
* dep: redirect github.com/bnb-chain/tendermint to v0.31.13
* ci: disable GOPROXY
The execution-apis specification says that the full list of withdrawals should always be returned when requesting a block over RPC:
378c4304f7/src/schemas/block.yaml (L90-L94)
This change adopts the expected behavior.
Adds blocktest subcommand to the evm command, which is very similar to statetest, but instead of loading a StateTest static test it loads a BlockchainTest from a json file and runs it.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
This change removes the Infura rinkeby bootnode as well as two deprecated ropsten bootnodes.
Co-authored-by: Nicola Cocchiaro <ncocchiaro@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes the API so that uint64 is used for fork timestamps.
It's a good choice because types.Header also uses uint64 for time.
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a regression where the flag did not accept values without
the 0x prefix anymore. What's worse, if an invalid value was passed,
the client would just log an INFO level message and continue.
This change introduces a breaking change to miner.etherbase is configured.
Previously, users did not need to explicitly set the etherbase address via flag, since 'first' local account was used as etherbase automatically. This change removes the "default first account" feature.
In Proof-of-stake world, the fee recipient address is provided by CL, and not configured in Geth any more - meaning that miner.etherbase is mostly for legacy networks(pow, clique networks etc).
Fixes#26505 where the console crashed when a property getter
raised an exception during autocompletion. I also noticed while fixing this
issue that autocomplete wasn't working for objects/fields with numbers in
them (most importantly web3.<tab><tab>) which is also now fixed.
This PR does a few things.
It fixes a shutdown-order flaw in the chainfreezer. Previously, the chain-freezer would shutdown the freezer backend first, and then signal for the loop to exit. This can lead to a scenario where the freezer tries to fsync closed files, which is an error-conditon that could lead to exit via log.Crit.
It also makes the printout more detailed when truncating 'dangling' items, by showing the exact number instead of approximate MB.
This PR also adds calls to fsync files before closing them, and also makes the `db inspect` command slightly more robust.
* internal/flags: use filepath.Clean instead of path.Clean
* internal/flags: fix windows pipe issue
* internal/flags: modify test for windows
* internal/flags: use backticks, fix test
This PR fixes an issue which might result in data lost in freezer.
Whenever mutation happens in freezer, all data will be written into head data file
and it will be rotated with a new one in case the size of file reaches the threshold.
Theoretically, the rotated old data file should be fsync'd to prevent data loss.
In freezer.Sync function, we only fsync: (1) index file (2) meta file and (3) head
data file. So this PR forcibly fsync the head data file if mutation happens in the
boundary of data file.
In legacy (pre-merge) sync mode, headers were contiguously downloaded from the network and when no more headers were available, we checked every few seconds whether there are 64 new blocks to move the pivot.
In beacon (post-merge) sync mode, we don't need to check for new skeleton headers non stop, since those re delivered one by one by the engine API. The missing code snippet from the header fetcher was to actually look at the latest head and move the pivot if it was more than 2*64-8 away. This PR adds the missing movement logic.
This makes non-JS tracers execute all block txs on a single goroutine.
In the previous implementation, we used to prepare every tx pre-state
on one goroutine, and then run the transactions again with tracing enabled.
Native tracers are usually faster, so it is faster overall to use their output as
the pre-state for tracing the next transaction.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR removes the notion of fakeStorage from the state objects, and instead, for any state modifications that are needed, it simply makes the changes.
* p2p/discover: add more packet information in logs
This adds more fields to discv5 packet logs. These can be useful when
debugging multi-packet interactions.
The FINDNODE message also gets an additional field, OpID for debugging
purposes. This field is not encoded onto the wire.
I'm also removing topic system related message types in this change.
These will come back in the future, where support for them will be
guarded by a config flag.
* p2p/discover/v5wire: rename 'Total' to 'RespCount'
The new name captures the meaning of this field better.
Alarm is a timer utility that simplifies code where a timer needs to be rescheduled over
and over. Doing this can be tricky with time.Timer or time.AfterFunc because the channel
requires draining in some cases.
Alarm is optimized for use cases where items are tracked in a heap according to their expiry
time, and a goroutine with a for/select loop wants to be woken up whenever the next item expires.
In this application, the timer needs to be rescheduled when an item is added or removed
from the heap. Using a timer naively, these updates will always require synchronization
with the global runtime timer datastructure to update the timer using Reset. Alarm avoids
this by tracking the next expiry time and only modifies the timer if it would need to fire earlier
than already scheduled.
As an example use, I have converted p2p.dialScheduler to use Alarm instead of AfterFunc.
This changes moves the tracking of "deleted in this block" out from snap-only domain, so that it happens regardless of whether the execution is snapshot-backed or trie-backed.
This changes the StorageTrie method to return an error when the trie
is not available. It used to return an 'empty trie' in this case, but that's
not possible anymore under PBSS.
This PR builds on #26299, but also updates the tests to the most recent version, which includes tests regarding TheMerge.
This change adds checks to the beacon consensus engine, making it more strict in validating the pre- and post-headers, and not relying on the caller to have already correctly sanitized the headers/blocks.
This PR implements resettable freezer by adding a ResettableFreezer wrapper.
The resettable freezer wraps the original freezer in a way that makes it possible to ensure atomic resets. Implementation wise, it relies on the os.Rename and os.RemoveAll to atomically delete the original freezer data and re-create a new one from scratch.
This PR fixes an error in trie commit. If the trie.root is nil, it can be two possible scenarios:
- The trie was empty, and no change happens
- The trie was non-empty and all nodes are dropped
For the latter one, we should collect the deletions and apply them into database(e.g. in PBSS).
This PR adds an addtional API called `NewBatchWithSize` for db
batcher. It turns out that leveldb batch memory allocation is
super inefficient. The main reason is the allocation step of
leveldb Batch is too small when the batch size is large. It can
take a few second to build a leveldb batch with 100MB size.
Luckily, leveldb also offers another API called MakeBatch which can
pre-allocate the memory area. So if the approximate size of batch is
known in advance, this API can be used in this case.
It's needed in new state scheme PR which needs to commit a batch of
trie nodes in a single batch. Implement the feature in a seperate PR.
* metrics: add unlock address to metrics when miner module is enabled
* metrics: add miner config into metrics server
* metrics: add device-info into metrics server
* metrics: fix the format of device info
* metrics: remove device-info
This ensures that RPC method handlers will react to a timeout or
cancelled request soon after the event occurs.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds a check that the safetxhash that we sign corresponds to the one that is expected by the input. If it differs, it tries again with the configured chainid.
* eth: fix a rare datarace on CHT challenge reply / shutdown
* trie: check childrens' existence concurrently for snap heal
* eth/protocols/snap: fix problems due to idle-but-busy peers
* eth/filters: change filter block to be by-ref (#26054)
This PR changes the block field in the filter to be a pointer, to disambiguate between empty hash and no hash
* rpc: handle wrong HTTP batch response length (#26064)
* eth/protocols/snap: throttle trie heal requests when peers DoS us (#25666)
* eth/protocols/snap: throttle trie heal requests when peers DoS us
* eth/protocols/snap: lower heal throttle log to debug
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/protocols/snap: fix comment
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Jordan Krage <jmank88@gmail.com>
A comment suggests that contract creation happens if the recipient of a call is 0x00..00 ("zero address") but in fact the sender must be nil. The zero address is a regular valid address that is commonly used as a "burn" address.
Currently calling `debug_TraceTransaction` with a transaction hash that doesn't exist returns a confusing error: `genesis is not traceable`. This PR changes the behaviour to instead return an error message saying `transaction not found`
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds a new type event.FeedOf[T], which is like event.Feed but parameterized
over the channel element type. Performance is unchanged, and it still uses reflect. But
unlike Feed, the generic version doesn't need to type-check interface{} arguments.
All panic cases are gone from the API.
While investigating another issue, I found that all callers of collectLogs have the
complete block available. rawdb.ReadReceipts loads the block from the database,
so it is better to use ReadRawReceipts here, and derive the receipt information using
the block which is already in memory.
This PR makes it possible to modify the flush interval time via RPC. On one extreme, `0s`, it would act as an archive node. If set to `1h`, means that after one hour of effective block processing time, the trie would be flushed. If one block takes 200ms, this means that a flush would occur every `5*3600=18000` blocks -- however, if the memory size of the cached states grows too large, it will flush sooner.
Essentially, this makes it possible to configure the node to be more or less "archive:ish", and without restarting the node while reconfiguring it.
--syncTarget is a feature for development purpose in post-merge world. Previously
it's added into eth.Config. But it turns out that's a stupid idea.
- syncTarget is a block object, which is hard to be put in config file(large)
- syncTarget is just a dev feature, doesn't make too much sense to add it in config file
So I remove it from the eth config object. And it also fixes the #26328
This improves readability of function 'push'.
sort.Search(N, ...) will at most return N when no match, so ix should be compared
with N. The previous version would compare ix with N+1 in case an additional item
was appended. No bug resulted from this comparison, but it's not easy to understand
why.
Co-authored-by: Felix Lange <fjl@twurst.com>
The gcproc field tracks the amount of time spent processing blocks,
and is used to trigger a state flush to disk when a certain threshold is
reached. After the merge, single block insertion by CL is the most
common source of block processing time, but this time was not added
into gcproc.
Here we add special handling for sending an error response when the write timeout of the
HTTP server is just about to expire. This is surprisingly difficult to get right, since is
must be ensured that all output is fully flushed in time, which needs support from
multiple levels of the RPC handler stack:
The timeout response can't use chunked transfer-encoding because there is no way to write
the final terminating chunk. net/http writes it when the topmost handler returns, but the
timeout will already be over by the time that happens. We decided to disable chunked
encoding by setting content-length explicitly.
Gzip compression must also be disabled for timeout responses because we don't know the
true content-length before compressing all output, i.e. compression would reintroduce
chunked transfer-encoding.
The new flag allows configuring an explicit endpoint which is to be
announced in the DHT. This feature was originally developed for the
discv5 wormhole experiment (#25798), but it's useful in other contexts
as well.
This removes the 'time' field from logs, as well as from the tracer interface. This change makes the trace output deterministic. If a tracer needs the time they can measure it themselves. No need for evm to do this.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This changes the Pop method to assign the zero value before
reducing slice size. Doing so ensures the backing array does not
reference removed item values.
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.
As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
1.remove the unnecessary NewTxsEvent subscriber, which was used for PoW resubmit check.
2.unsubscribe ASAP before another fillTransactions, to avoid block others.
This PR should makes it easier to sign EIP-712 typed data via the accounts.Wallet API, by using the mimetype for typed data.
Co-authored-by: nasdf <keenan.nemetz@gmail.com>
While investigating #22374, I noticed that the Sync operation of the
freezer does not take the table lock. It also doesn't call sync for all files
if there is an error with one of them. I doubt this will fix anything, but
didn't want to drop the fix on the floor either.
* worker: add double sign check for safety.
And for corner cases, such as reorg after reorg...
use slice to record all broadcast blocks's parents to avoid overwritten.
When new block is imported, there is no need to commit the current
work, even the new imported block is offturn and itself is inturn.
That is because when offturn block is received, the inturn block is
already later to broadcast block, deliver the later block will cause
many reorg, which is not reasonable.
And also make sure all useless work can be discarded, to avoid goroutine leak.
It may not efficient if schedule fillTransactions when new transactions arrive.
It could make the CPU keep running.
To make is more efficient:
1.schedule fillTransactions when a certain amount of transaction are arrived.
2.or there is not much time left.
Currently, validator only try once to get transactions from TxPool to produce the block.
However, new transactions could arrive while the validator is committing transaction.
Validator should be allowed to add these new arrived transactions as long as
Header.Timestamp is not reached
This commit will:
** commitTransactions return with error code
** drop current mining block on new block imported
** try fillTransactions several times for the best
not use append mode to follow the GasPrice rule.
** check if there is enough time for another fillTransactions.
This avoids copying the input []byte while decoding trie nodes. In most
cases, particularly when the input slice is provided by the underlying
database, this optimization is safe to use.
For cases where the origin of the input slice is unclear, the copying version
is retained. The new code performs better even when the input must be
copied, because it is now only copied once in decodeNode.
This removes an RPC test which takes > 90s to execute, and updates the
internal/guide tests to use lighter scrypt parameters.
Co-authored-by: Felix Lange <fjl@twurst.com>
`fillTransactions` will call `commitTransactions` twice, if the delay
timer is expired during the first call, it will make the delay timer
never be triggered in the second commitTransactions call.
Pseudo code:
x := time.NewTimer(time.Second)
<-x.C
fmt.Println("read delay 1")
<-x.C
fmt.Println("read delay 2") // will never hit
This adds a way to specify HTTP headers per request.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
pre-seal empty block is for PoW to deliver an empty block ASAP to
gain the block mine reward.
It is useless for PoS consensus and it does not work for BSC either.
Delete the code to make worker simpler.
It could be a very old PoW logic, which try to add more transaction
into the pending block when mining is stopped.
Mining can be stopped when:
1.download started.
2.manually stopped by RPC.
It is unnecessary to add more transaction into the pending block if a validator is stopped.
And updateSnapshot() is not needed as well, it is to get the pending mining snapshot.
Right now, DelayLeftOver is used to reserve time for block finalize, not block
broadcast. And the code does not work as expected.
The general block generation could be described as:
|- fillTransactions -|- finalize a block -|- wait until the period(3s) reached -|- broadcast -|
rpc: fix connection tracking in Server
When upgrading to mapset/v2 with generics, the set element type used in
rpc.Server had to be changed to *ServerCodec because ServerCodec is not
'comparable'. While the distinction is technically correct, we know all
possible ServerCodec types, and all of them are comparable. So just use
a map instead.
resubmit intervalAdjust is for PoW only, to remove it to make worker simpler.
With PoW, there will be a periodic timer to check if it is the time to stop
packing transaction and start calculating the desired hash value, since other miner
could succeed in hash compute if it spends too much time packing transactions.
It will commit the current fruit to calculate root at a reasonable time.
And it will schedule a new work to get a big block if new transaction was received.
When there are too many transactions in the TxPool, the interval of the resubmit timer would be
increased and vice versa.
But it is not needed with PoS related consensus, since the block interval is determined in PoS,
and there is already a timer to stop too long packing.
This change ensures the HTTP server will always terminate within
at most 5s, even when all connections are busy and do not become
idle.
Co-authored-by: Felix Lange <fjl@twurst.com>
* internal/ethapi: error if tx args includes chain id that doesn't match local
* internal/ethapi: simplify code a bit
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* Remove locking in (*BlockChain).ExportN
Since ExportN is read-only, it shouldn't need the lock. (?)
* Add hash check to detect reorgs during export.
* fix check order
* Update blockchain.go
* Update blockchain.go
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
It seems there is no fully typed library implementation of an LRU cache.
So I wrote one. Method names are the same as github.com/hashicorp/golang-lru,
and the new type can be used as a drop-in replacement.
Two reasons to do this:
- It's much easier to understand what a cache is for when the types are right there.
- Performance: the new implementation is slightly faster and performs zero memory
allocations in Add when the cache is at capacity. Overall, memory usage of the cache
is much reduced because keys are values are no longer wrapped in interface.
This PR changes the pending tx subscription to return RPCTransaction types instead of normal Transaction objects. This will fix the inconsistencies with other tx returning API methods (i.e. getTransactionByHash), and also fill in the sender value for the tx.
co-authored by @s1na
This fixes a problem in the SizeConstrainedLRU. The SCLRU uses an underlying simple lru which is not thread safe.
During the Get operation, the recentness of the accessed item is updated, so it is not a pure read-operation. Therefore, the mutex we need is a full mutex, not RLock.
This PR changes the mutex to be a regular Mutex, instead of RWMutex, so a reviewer can at a glance see that all affected locations are fixed.
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.
runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.
The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.
Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.
The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.
We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
This adds an option to direct log output to a file. This feature has been
requested a lot. It's sometimes useful to have this available when running
geth in an environment that doesn't easily allow redirecting the output.
Notably, there is no support for log rotation with this change. The --log.file option
opens the file once on startup and then keeps writing to the file handle.
This can become an issue when external log rotation tools are involved, so it's
best not to use them with this option for now.
When the interpreter is configured to use extra-eips, this change makes it so that all the opcodes are deep-copied, to prevent accidental modification of the 'base' jumptable.
Closes: #26136
Co-authored-by: Martin Holst Swende <martin@swende.se>
PR #26082 added account listing to OnSignerStartup but did not consider the case where a user has a large number of accounts which would be annoying to display.
This PR updates showAccounts() so that if there are more than 20 accounts available the user sees the first 20 displayed in the console followed by: First 20 accounts listed (N more available).
Co-authored-by: Martin Holst Swende <martin@swende.se>
Many of the other types have a function to convert the type to a big.Int,
but Address was missing this function.
It is useful to be able to turn an Address into a big.Int when doing
EVM-like computations natively in Go. Sometimes a Solidity address
type is casted to a uint256 and having a Big method on the Address
type makes this easy.
Instead of using a limit of three nodes per message, we can pack more nodes
into each message based on ENR size. In my testing, this halves the number
of sent NODES messages, because ENR size is usually < 300 bytes.
This also adds RLP helper functions that compute the encoded size of
[]byte and string.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR now also includes a fix to the problem of mult-routines building blocks on the same input. This PR works as before with regards to stopping the work, but it just will not spin up a second routine if one is already building. So if the CL does N calls to FCU+buildblock, and N calls to GetPayload, only the first of each will do something, the other calls will be mostly no-ops.
This PR also adds printout of the payload id into the logs.
* apitypes: synchronize handling of types
* signer/core/apitypes: improve array check
* apitypes: add a test for big.Int -> int32
* signer/core/apitypes: Add a test for parsing addresses from [20]byte, []byte and string
* signer/core/apitypes: add some testcases
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes geth to read the eip1559 params from the chain config instead of the globals.
This way the parameters may be changed by forking the chain config code, without creating a large diff throughout the past and future usages of the parameters.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This commit adds support for two new commands to clef, making it possible to list accounts / wallets from the command-line-interface.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Noticed that lookupDistances for FINDNODE requests didn't consider 256 a valid
distance. This is actually part of the example in the comment above the
function, surprised that wasn't tested before.
In some cases, it is desirable to capture what is triggered by each trace, when using the `callTracer`. For example: call `USDT.transfer` will trigger a `Transfer(from, to, value)` event.
This PR adds the option to capture logs to the call tracer, by specifying `{"withLog": true}` in the tracerconfig.
Any logs belonging to failed/reverted call-scopes are removed from the output, to prevent interpretation mistakes.
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Inner call reverts will now return the reason similar to the top-level call. Separately, if top-level call is of type CREATE and it fails, its `to` field will now be cleared to `0x00...00` instead of being set to the created address.
This PR adds a parameter to startup, --synctarget. The synctarget flag is a developer-flag, that can be useful in some scenarios as a replacement for a CL node. It defines a fixed block sync target:
geth --syncmode=full --synctarget=./block_15816882.hex_rlp
The --synctarget is only made available during syncmode=full
* eth/tracers: fix gasUsed in call tracer
* fix js tracers gasUsed
* fix legacy prestate tracer
* fix restGas in test
* drop intrinsicGas field from js tracers
Currently, in order to chain together sequential valid t8n transitions the caller must manually calculate the block base fee. This PR adds support for the necessary parent fee market data to calculate the base fee for the current transition.
Concretely, env is extended to accept the following:
parentBaseFee
parentGasUsed
parentGasLimit
Example usage can be found in ./cmd/evm/testdata/25.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR ports a few changes from PBSS:
- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
* accounts/abi/bind: fix duplicate field names in the generated go struct #24627
* accounts, cmd/abigen: resolve name conflicts
* ci lint, accounts/abi: remove unused function overloadedArgName
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The prestate tracer did not report accounts that existed at a given address prior to a contract being created at that address.
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This changes the []byte <-> Uint8Array conversion to use an
ArrayBuffer, avoiding inefficient copying of the slice data in Goja.
Co-authored-by: Felix Lange <fjl@twurst.com>
The oss-fuzz engine crashes due to stack overflow decoding a large nested
structure into a interface{}. This PR limits the size of the input data, so
should avoid such crashes.
This PR fixes a regression causing snapshots not to be generated in "geth --import" mode. It also fixes the geth export command to be truly readonly, and adds a new test for geth export.
In some cases, inner contract creation may not be successful, and an inner contract was not created. This PR fixes a crash that could occur when doing tracing in such situations.
This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.
In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR removes some optimistic tests -- a'la "do something,
wait a while, and hope it has trickled through and continue" -- and
instead uses some introspection to ensure that prerequisites are met.
This changes the node setup to ignore datadir files
static-nodes.json
trusted-nodes.json
When these files are present, it an error will be printed to the log.
This adds a
* core/vm, tests: optimized modexp + fuzzer
* common/math: modexp optimizations
* core/vm: special case base 1 in big modexp
* core/vm: disable fastexp
This changes the error message for mismatching chain ID to show
the given and expected value. Callers expecting this error must be
changed to use errors.Is.
* usbwallet support Ledger Nano S Plus
* accounts/usbwallet: add definitions + ref to ledger docs
Co-authored-by: Martin Holst Swende <martin@swende.se>
This update resolves an issue where StringSliceFlag would not be
rendered correctly in help output + mention that -H can be used multiple times
Co-authored-by: Martin Holst Swende <martin@swende.se>
Prior to this change, f.begin (and possibly end) stay negative, leading to strange results later in the code. With this change, filters using "safe" and "finalized" block produce results consistent w/ the overall behavior of this RPC method.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethclient/gethclient: improve time-sensitive flaky test
* eth/catalyst: fix (?) flaky test
* core: stop blockchains in tests after use
* core: fix dangling blockchain instances
* core: rm whitespace
* eth/gasprice, eth/tracers, consensus/clique: stop dangling blockchains in tests
* all: address review concerns
* core: goimports
* eth/catalyst: fix another time-sensitive test
* consensus/clique: add snapshot test run function
* core: rename stop() to stopWithoutSaving()
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR introduces a new mechanism in chain tracer for preventing creating too many trace states.
The workflow of chain tracer can be divided into several parts:
- state creator generates trace state in a thread
- state tracer retrieves the trace state and applies the tracing on top in another thread
- state collector gathers all result from state tracer and stream to users
It's basically a producer-consumer model here, while if we imagine that the state producer generates states too fast, then it will lead to accumulate lots of unused states in memory. Even worse, in path-based state scheme it will only keep the latest 128 states in memory, and the newly generated state will invalidate the oldest one by marking it as stale.
The solution for fixing it is to limit the speed of state generation. If there are over 128 states un-consumed in memory, then the creation will be paused until the states are be consumed properly.
Backwards compatibility warning: The result will from now on omit empty fields instead
of including a zero value (e.g. no more `balance: '0x'`).
The prestateTracer will now take an option `diffMode: bool`. In this mode
the tracer will output the pre state and post data for the modified parts of state.
Read-only accesses will be completely omitted. Creations (be it account or slot)
will be signified by omission in the `pre` list and inclusion in `post`. Whereas
deletion (be it account or slot) will be signified by inclusion in `pre` and omission
in `post` list.
Signed-off-by: Delweng <delweng@gmail.com>
This PR makes it so that the snap server responds to trie heal requests when possible, even if the snapshot does not exist. The idea being that it might prolong the lifetime of a state root, so we don't have to pivot quite as often.
This PR makes it possible to set custom headers, in particular for two scenarios:
- geth attach
- geth commands which can use --remotedb, e..g geth db inspect
The ability to use custom headers is typically useful for connecting to cloud-apis, e.g. providing an infura- or alchemy key, or for that matter access-keys for environments behind cloudflare.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR reworks tx indexer a bit. Compared to the original version, one scenario is no longer handled - upgrading from legacy geth without indexer support.
The tx indexer was introduced in 2020 and have been present through hardforks, so it can be assumed that all Geth nodes have tx indexer already. So we can simplify the tx indexer logic a bit:
- If the tail flag is not present, it means node is just initialized may or may not with an ancient store attached. In this case all blocks are regarded as unindexed
- If the tail flag is present, it means blocks below tail are unindexed, blocks above tail are indexed
This change also address some weird cornercases that could make the indexer not work after a crash.
This fixes a bug where contract code would be overridden to empty code ("0x")
when the Code field of OverrideAccount was left nil. The change also cleans up
the encoding of overrides to only send necessary fields, and improves documentation.
Fixes#25615
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
The call tracer and prestate tracer store data JSON-encoded in memory. In order to support alternative encodings (specifically RLP), it's better to keep data a native format during tracing. This PR does marshalling at the end, using gencodec.
OBS!
This PR changes the call tracer result slightly:
- Order of type and value fields are changed (should not matter).
- Output fields are completely omitted when they're empty (no more output: "0x"). Previously, this was only _sometimes_ omitted (e.g. when call ended in a non-revert error) and otherwise 0x when the output was actually empty.
Some tests define an 'expectException' error but the tests runner does not check for conditions where this test value is filled (error expected) but in which no error is returned by the test runner.
An example of this scenario is GeneralStateTests/stTransactionTest/HighGasPrice.json, which expects a 'TR_NoFunds' error, but the test runner does not return any error.
Signed-off-by: meows <b5c6@protonmail.com>
`geth dumpgenesis` currently does not respect the content of the data directory. Instead, it outputs the genesis block created by command-line flags. This PR fixes it to read the genesis from the database, if the database already exists.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: pad memory slice on oob case
* eth/tracers/js: fix testfailure due to err msg capitalization
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
The abigen exclusion pattern, previously on the form "path:type", now supports wildcards. Examples "*:type" to exclude a named type in all files, or "/path/to/foo.sol:*" all types in foo.sol.
This changes the CI build to store the git commit and date into package
internal/version instead of package main. Doing this essentially merges our
two ways of tracking the go-ethereum version into a single place, achieving
two objectives:
- Bad block reports, which use version.Info(), will now have the git commit
information even when geth is built in an environment such as
launchpad.net where git access is unavailable.
- For geth builds created by `go build ./cmd/geth` (i.e. not using `go run
build/ci.go install`), git information stored by the go tool is now used
in the p2p node name as well as in `geth version` and `geth
version-check`.
Sometimes we get stuck on db compaction, and the CL re-issues the "same" command to us multiple times. Each request get stuck on the same place, in the middle of the handler.
This changes makes it so we do not reprocess the same payload, but instead detects it early.
This PR improves the docker build speed for repeated builds where go.mod and go.sum do no change, by placing the downloaded dependencies in a lower layer
This change removes extraneous/unnecessary checks for equality
when comparing 2 accessList values A and B. Given that we validate that
their lengths of A and B are equal, if so and if every element in A is
in B, reflexively every element in B is already in A. If that weren't
the case and an element g existed in A but not in B, that would mean
that there is an extra element and hence a mathematical contradiction.
Fixes#24658
This PR fixes a few panics in the chain marker benchmarks. The root
cause for panic is in chain marker the genesis header/block is not
accessible, while it's expected to be obtained in tests. So this PR
avoids touching genesis header at all to avoid panic.
core/blockchain: downgrade tx indexing and unindexing logs from info to debug
If a user has a finite tx lookup limit, they will see an "unindexing" info level log each time a block is imported. This information might help a user understand that they are removing the index each block and some txs may not be retrievable by hash, but overall it is generally more of a nuisance than a benefit. This change downgrades the log to a debug log.
This shortens the chain config summary in bad block reports,
and adds go-ethereum version information as well.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes eth_getProof and eth_getStorageAt return an error when
the argument contains invalid hex in storage keys.
Co-authored-by: Felix Lange <fjl@twurst.com>
* cmd/geth: add a verkle subcommand
* fix copyright year
* remove unused command parameters
* check that the output file was successfully written to
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/geth: goimports fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
This PR makes the event-sending for deleted and new logs happen in batches, to prevent OOM situation due to large reorgs.
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the error code returned by the RPC server in certain situations:
- handler panic: code -32603
- result marshaling error: code -32603
- attempt to subscribe via HTTP: code -32001
In all of the above cases, the server previously returned the default error
code -32000.
Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The p2p msgrate tracker is a thing which tries to estimate some mean round-trip times. However, it did so in a very curious way: if a node had 200 peers, it would sort their 200 respective rtt estimates, and then it would pick item number 2 as the mean. So effectively taking third fastest and calling it mean. This probably works "ok" when the number of peers are low (there are other factors too, such as ttlScaling which takes some of the edge off this) -- however when the number of peers is high, it becomes very skewed.
This PR instead bases the 'mean' on the square root of the length of the list. Still pretty harsh, but a bit more lenient.
This PR includes minor updates to comments in trie/committer that reference insertion to the db, and adds an err != nil check for the return value of preimages.commit.
This PR simplifies the logic of chain tracer and also adds the unit tests.
The most important change has been made in this PR is the state management. Whenever a tracing state is acquired there is a corresponding release function be returned as well. It must be called once the state is used up, otherwise resource leaking can happen.
And also the logic of state management has been simplified a lot. Specifically, the state provider(eth backend, les backend) should ensure the state is available and referenced. State customers can use the state according to their own needs, or build other states based on the given state. But once the release function is called, there is no guarantee of the availability of the state.
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
The JSON-RPC spec requires the "version" field to be exactly "2.0",
so we should verify that. This change is not backwards-compatible with
sloppy client implementations, but I decided to go ahead with it anyway
because the failure will be caught via the returned error.
This adds a generic mechanism for 'dial options' in the RPC client,
and also implements a specific dial option for the JWT authentication
mechanism used by the engine API. Some real tests for the server-side
authentication handling are also added.
Co-authored-by: Joshua Gutow <jgutow@optimism.io>
Co-authored-by: Felix Lange <fjl@twurst.com>
It's a trivial PR to hide the error log when the trie node is not found in the database. The idea for this change is for all TryXXX functions, the error is already returned and we don't need to fire a log explicitly.
Recently there are a few tickets #25613#25589 reporting that the trie nodes are missing because of debug.SetHead. The root cause is after resetting, the chain rewinds to a historical point and re-imports the blocks on top.
Since the node is already synced and started to accept transactions previously, these transactions are still kept in the txpool and verified by txpool with a live state. This live state is constructed based on the live trie database, which is changed fast by node referencing and de-referencing.
Unfortunately, when we construct a live state(like the state in txpool), we don't reference the state we have. The blockchain will garbage collect the intermediate version nodes in another thread which leads the broken live state.
The best solution for this is to forcibly obtain a reference for all live states we create and call release function once it's used up. But it might end up with more junks persisted into disk. Will try to find an elegant solution later in the following PR.
This change speeds up trie hashing and all other activities that require
RLP encoding of trie nodes by approximately 20%. The speedup is achieved by
avoiding reflection overhead during node encoding.
The interface type trie.node now contains a method 'encode' that works with
rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code.
trie.hasher, which is pooled to avoid allocations, now maintains an
EncoderBuffer. This means memory resources related to trie node encoding
are tied to the hasher pool.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.
The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.
To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.
Benchmark results:
name old time/op new time/op delta
EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8)
EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8)
name old speed new speed delta
EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8)
EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal)
This change adds a code generator tool for creating EncodeRLP method
implementations. The generated methods will behave identically to the
reflect-based encoder, but run faster because there is no reflection overhead.
Package rlp now provides the EncoderBuffer type for incremental encoding. This
is used by generated code, but the new methods can also be useful for
hand-written encoders.
There is also experimental support for generating DecodeRLP, and some new
methods have been added to the existing Stream type to support this. Creating
decoders with rlpgen is not recommended at this time because the generated
methods create very poor error reporting.
More detail about package rlp changes:
* rlp: externalize struct field processing / validation
This adds a new package, rlp/internal/rlpstruct, in preparation for the
RLP encoder generator.
I think the struct field rules are subtle enough to warrant extracting
this into their own package, even though it means that a bunch of
adapter code is needed for converting to/from rlpstruct.Type.
* rlp: add more decoder methods (for rlpgen)
This adds new methods on rlp.Stream:
- Uint64, Uint32, Uint16, Uint8, BigInt
- ReadBytes for decoding into []byte
- MoreDataInList - useful for optional list elements
* rlp: expose encoder buffer (for rlpgen)
This exposes the internal encoder buffer type for use in EncodeRLP
implementations.
The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to
encBuffer. It is implemented this way to ensure the global encBuffer pool
is handled correctly.
Some small fixes to get the existing debug methods to conform to the spec. Mainly dropping the encoding information from the method name as it should be deduced from the debug context and allowing the method to be invoked by either block number or block hash. It also adds the method debug_getTransaction which returns the raw tx bytes by tx hash. This is pretty much equivalent to the eth_getRawTransactionByHash method.
* eth/catalyst: warn less frequently if no beacon client is available
* eth/catalyst: tweak warning frequency a bit
* eth/catalyst: some more tweaks
* Update api.go
Co-authored-by: Felix Lange <fjl@twurst.com>
* trie: fix memory leak in trie iterator
In the trie iterator, live nodes are tracked in a stack while iterating.
Popped node states should be explictly set to nil in order to get
garbage-collected.
* trie: fix empty trie iterator
* eth/fetcher: introduce some lag in tx fetching
* eth/fetcher: change conditions a bit
* eth/fetcher: use per-batch quota check
* eth/fetcher: fix some comments
* eth/fetcher: address review concerns
* eth/fetcher: fix panic + add warn log
* eth/fetcher: fix log
* eth/fetcher: fix log
* cmd/devp2p/internal/ethtest: fix ignorign tx announcements from prev. tests
* cmd/devp2p/internal/ethtest: fix TestLargeTxRequest
This increases the number of tx relay messages the test waits for. Since
go-ethereum now processes incoming txs in smaller batches, the
announcement messages it sends are also smaller.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a cache for block logs which is shared by all filters. The cache
size of is configurable using the `--cache.blocklogs` flag.
Co-authored-by: Felix Lange <fjl@twurst.com>
This avoids copying the input []byte while decoding trie nodes. In most
cases, particularly when the input slice is provided by the underlying
database, this optimization is safe to use.
For cases where the origin of the input slice is unclear, the copying version
is retained. The new code performs better even when the input must be
copied, because it is now only copied once in decodeNode.
* core, trie: flush preimages to db on database close
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* rename Close to CommitPreimages for clarity
* core, trie: nitpick fixes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This adds support for building statically-linked executables using ci.go.
Static linking is enabled by default in Docker builds, making it possible to
use the geth executable in any Docker image, regardless of the Linux
distribution the Dockerfile is based on.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR allows users to pass in a config object directly to the tracers. Previously only the struct logger was configurable.
It also adds an option to the call tracer which if enabled makes it ignore any subcall and collect only information about the top-level call. See #25419 for discussion.
The tracers will silently ignore if they are passed a config they don't care about.
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Because the goal of eth_createAccessList is providing the caller with the largest-possible
access list, it's generally not important that the gas limit used by the tracer will match the usage
of the call exactly. Avoiding the gas estimation step is a performance improvement. As long as the
call does not branch based on gas limit, the returned access list will be accurate.
This change makes http.Server.ReadHeaderTimeout configurable separately
from ReadTimeout for RPC servers. The default is set to the same as
ReadTimeout, which in order to cause no change in existing deployments.
* feature: do trie prefetch on state prefetch
Currently, state prefetch just pre execute the transactions and discard the results.
It is helpful to increase the snapshot cache hit rate.
It would be more helpful, if it can do trie prefetch at the same time, since the it will
preload the trie node and build the trie tree in advance.
This patch is to implement it, by reusing the main trie prefetch and doing finalize after
transaction is executed.
* some code improvements for trie prefetch
** increase pendingSize before dispatch tasks
** use throwaway StateDB for TriePrefetchInAdvance and remove the prefetcherLock
** remove the necessary drain operation in trie prefetch mainloop,
trie prefetcher won't be used after close.
* trie prefetcher for From/To address in advance
We found that trie prefetch could be not fast enough, especially trie prefetch of
the outer big state trie tree.
Instead of do trie prefetch until a transaction is finalized, we could do trie prefetch
in advance. Try to prefetch the trie node of the From/To accounts, since their root hash
are most likely to be changed.
* Parallel TriePrefetch for large trie update.
Currently, we create a subfetch for each account address to do trie prefetch. If the address
has very large state change, trie prefetch could be not fast enough, e.g. a contract modified
lots of KV pair or a large number of account's root hash is changed in a block.
With this commit, there will be children subfetcher created to do trie prefetch in parallell if
the parent subfetch's workload exceed the threshold.
* some improvemnts of parallel trie prefetch implementation
1.childrenLock is removed, since it is not necessary
APIs of triePrefetcher is not thread safe, they should be used sequentially.
A prefetch will be interrupted by trie() or clos(), so we only need mark it as
interrupted and check before call scheduleParallel to avoid the concurrent access to paraChildren
2.rename subfetcher.children to subfetcher.paraChildren
3.use subfetcher.pendingSize to replace totalSize & processedIndex
4.randomly select the start child to avoid always feed the first one
5.increase threshold and capacity to avoid create too many child routine
* fix review comments
** nil check refine
** create a separate routine for From/To prefetch, avoid blocking the cirtical path
* remove the interrupt member
* not create a signer for each transaction
* some changes to triePrefetcher
** remove the abortLoop, move the subfetcher abort operation into mainLoop
since we want to make subfetcher's create & schedule & abort within a loop to
avoid concurrent access locks.
** no wait subfetcher's term signal in abort()
it could speed up the close by closing subfetcher concurrently.
we send stop signnal to all subfetchers in burst and wait their term signal later.
* some coding improve for subfetcher.scheduleParallel
* fix a UT crash of s.prefetcher == nil
* update parallel trie prefetcher configuration
tested with different combination of parallelTriePrefetchThreshold & parallelTriePrefetchCapacity,
found the most efficient configure could be:
parallelTriePrefetchThreshold = 10
parallelTriePrefetchCapacity = 20
* fix review comments: code refine
* eth: support bubbling up bad blocks from sync to the engine API
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* Update eth/catalyst/api.go
* eth/catalyst: when forgetting bad hashes, also forget descendants
* eth/catalyst: minor bad block tweaks for resilience
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
During RPC calls such as eth_call and eth_estimateGas, st.evm.Config.NoBaseFee is set
which allows the gas price to be below the base fee. This results the tip being negative,
and balance being subtracted from the coinbase instead of added to it, which results in a
potentially negative coinbase balance interestingly. This can't happen during normal chain
processing as outside of RPC calls the gas price is required to be at least the base fee,
as NoBaseFee is false.
This change prevents this behavior by disabling fee payment when the fee is not set.
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
* Remove locking in (*BlockChain).ExportN
Since ExportN is read-only, it shouldn't need the lock. (?)
* Add hash check to detect reorgs during export.
* fix check order
* Update blockchain.go
* Update blockchain.go
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* internal/ethapi: error if tx args includes chain id that doesn't match local
* internal/ethapi: simplify code a bit
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Adds a native tracer that returns that in case of failure returns the error message or the revert reason of a transaction.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change ensures the HTTP server will always terminate within
at most 5s, even when all connections are busy and do not become
idle.
Co-authored-by: Felix Lange <fjl@twurst.com>
* Redesign triePrefetcher to make it thread safe
There are 2 types of triePrefetcher instances:
1.New created triePrefetcher: it is key to do trie prefetch to speed up validation phase.
2.Copied triePrefetcher: it only copy the prefetched trie information, actually it won't do
prefetch at all, the copied tries are all kept in p.fetches.
Here we try to improve the new created one, to make it concurrent safe, while the copied one's
behavior stay unchanged(its logic is very simple).
As commented in triePrefetcher struct, its APIs are not thread safe. So callers should make sure
the created triePrefetcher should be used within a single routine.
As we are trying to improve triePrefetcher, we would use it concurrently, so it is necessary to
redesign it for concurrent access.
The design is simple:
** start a mainLoop to do all the work, APIs just send channel message.
Others:
** remove the metrics copy, since it is useless for copied triePrefetcher
** for trie(), only get subfetcher through channel to reduce the workload of mainloop
* some code enhancement for triePrefetcher redesign
* some fixup: rename, temporary trie chan for concurrent safe.
* fix review comments
* add some protection in case the trie prefetcher is already stopped
* fix review comments
** make close concurrent safe
** fix potential deadlock
* replace channel by RWMutex for a few triePrefetcher APIs
For APIs like: trie(), copy(), used(), it is simpler and more efficient to
use a RWMutex instead of channel communicaton.
Since the mainLoop would be busy handling trie request, while these trie request
can be processed in parallism.
We would only keep prefetch and close within the mainLoop, since they could update
the fetchers
* add lock for subfecter.used access to make it concurrent safe
* no need to create channel for copied triePrefetcher
* fix trie_prefetcher_test.go
trie prefetcher’s behavior has changed, prefetch() won't create subfetcher immediately.
it is reasonable, but break the UT, to fix the failed UT
* eth/catalyst: disallow importing blocks via newPayload during snap sync
* eth/catalyst: make tests pass by using full sync only
* eth/catalysts: make the import delay a bit cleaner
* eth/catalyst: fix typo
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* add prune ancient feature
* change notes and log for pr suggest
* change StableStateBloc to SafePointBlock and enhanced 'pruneancient' flag hints
* change pruneancient usage
* fix note misspelling
* fix set SafePointBlockNumber by mpt height replace current block number
Co-authored-by: user <joeycli0919@qq.com>
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
This changes the []byte <-> Uint8Array conversion to use an
ArrayBuffer, avoiding inefficient copying of the slice data in Goja.
Co-authored-by: Felix Lange <fjl@twurst.com>
** replace atomic read by channel close to interrupt state prefetch
** try to do state prefetch when the prefetch thread is idle, no need to divide into 3 sub-arrays
it will make the prefetchers workload balance
The oss-fuzz engine crashes due to stack overflow decoding a large nested
structure into a interface{}. This PR limits the size of the input data, so
should avoid such crashes.
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
Previously on Geth startup we just logged the chain config is a semi-json-y format. Whilst that worked while we had a handful of hard-forks defined, currently it's kind of unwieldy.
This PR converts that original data dump and converts it into a user friendly - alas multiline - log output.
* accounts/abi/bind: fix duplicate field names in the generated go struct #24627
* accounts, cmd/abigen: resolve name conflicts
* ci lint, accounts/abi: remove unused function overloadedArgName
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds support for block overrides when doing debug_traceCall.
- Previously, debug_traceCall against pending erroneously used a common.Hash{} stateroot when looking up the state, meaning that a totally empty state was used -- so it always failed,
- With this change, we reject executing debug_traceCall against pending.
- And we add ability to override all evm-visible header fields.
In #24028 we flagged a warning when finding legacy receipts in the freezer. This PR nudges users a bit more strongly by preventing geth from starting in this case until receipts have been migrated.
It also adds a flag --ignore-legacy-receipts which when present allows geth to start normally.
#23773 added a JS tracer which uses Goja as its engine. In this PR I remove the previous tracer which used duktape as well as remove the dependencies.
This PR also comes with 2 fixes in the Goja tracer and one small behavioural change:
I had handled errors in the native Go functions by panicing. My oversight was that Goja only handles panics with a Goja.Value as argument. The difference is panic(goja.Value) allows JS to catch the exception whereas Interrupt(error) doesn't.
There was a race in how I handled Stop.
Because of 1. some of the methods that simply return nil on error (like memory.slice) now throw an exception.
This adds a JS tracer runtime environment based on the Goja VM. The new
runtime replaces the duktape runtime, which will be removed soon.
Goja is implemented in Go and is faster for cases where the Go <-> JS
transition overhead dominates overall performance. It is faster because
duktape is written in C, and the transition cost includes the cost of using
cgo. Another reason for using Goja is that go-duktape is not maintained
anymore.
We expect the performace of JS tracing to be at least as good or better with
this change.
* eth/catalyst: build the execution payload async
* miner: added comment, added test case
* eth/catalyst: miner: move async block production to miner
* eth/catalyst, miner: support generate seal block async
* miner: rework GetSealingBlockAsync to use a passed channel
* miner: apply rjl's diff
* eth/catalyst: nitpicks
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* ethdb/remotedb, cmd: add support for remote (readonly) databases
* ethdb/remotedb: minor changes
* ethdb/remotedb: close the conn
* cmd, ethdb: add rpc accessor for ancient data
* internal/ethapi: license
* ethdb/remotedb: linter fixes
This PR adds db tooling (geth db check-state-content) to verify the integrity of trie nodes. It iterates through the 32-byte key space in the database, which is expected to contain RLP-encoded trie nodes, addressed by hash.
* eth/protocols/snap: don't include empty snapshot slot slice
This PR fixes the snapshot storage serving handler. In snap protocol
the response is capped by the response size. Server can cutdown the
response if the accumulated byte size exceeds the local hard limit.
It means we can meet a special scenario that there is no storage slot
included for a requested account, but we attach the proof for this
account by mistake.
So in the prover side, when it meets a empty storage response but with
a valid proof proves there are some more slots left in the trie, then
requestor will reject this response and disconnect with server.
In this PR, if there is no storage slot served for the requested account,
then no proof should be attached as well.
* eth/protocols/snap: loosen restrictions for flaky tests
* eth/catalyst: fix flaky test in catalyst
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.
This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
During mining, when a new head arrives and interrupts the block building, the block being built should not be commited (but discarded). Committing the interrupted block introduces unnecessary delay, and possibly causes miner to mine on the previous head, which could result in higher uncle rate.
* core: recover the state in SetChainHead if the head state is missing
* core: disable test logging
* core: address comment from martin
* core: improve log level in case state is recovered
* core, eth, les, light: rename SetChainHead to SetCanonical
This PR improves the docker build speed for repeated builds where go.mod and go.sum do no change, by placing the downloaded dependencies in a lower layer
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
This adds the ability to run --state.fork=Merged, and have post-merge rules apply. When doing so, it also requires the input env to contain currentRandom, and enforces the currentDifficulty to be omitted or zero.
This PR fixes up the example python clef wrapper. The poc is intended to demonstrate how to wite a UI for clef, and had severely bitrotted.
With these changes, it "works" in the sense that all the built-in tests triggers the intended python callbacks (no errors about method not found). It does not "work" in the sense that the wrapper can be used as an actual UI. It will auto-reject any signing requests, for example.
* feat: add BEP-127 and BEP-131 hardfork bytecode to upgrade
* feat: force check that Euler height cannot be a multiple of 200, fix getCurrentValidators, raise SystemTxsGas after Euler fork
Co-authored-by: goth <goth>
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
* fix logic issue: handlers.removePeer() is called twice.
There is a logic issue which cause "Ethereum peer removal failed, err=peer not registered" occur quite often.
handler.runEthPeer set up a defer removePeer(). This is always called after a peer is disconnected.
However removePeer is also called by mulitple functions like downloader/fetcher. After those kind of functions removePeer(), peer handler executes defer removePeer(). This makes removePeer() happened twice, and this is the reason we often see "Ethereum peer removal failed, err=peer not registered".
To solve this, removePeer only needs to hard Disconnect peer from networking layer. Then defer unregisterPeer() will do the cleanup task after then.
* fix: modify test function for close testing.
reference from go-thereum.
Co-authored-by: zjubfd <296179868@qq.com>
This change removes extraneous/unnecessary checks for equality
when comparing 2 accessList values A and B. Given that we validate that
their lengths of A and B are equal, if so and if every element in A is
in B, reflexively every element in B is already in A. If that weren't
the case and an element g existed in A but not in B, that would mean
that there is an extra element and hence a mathematical contradiction.
Fixes#24658
See ethereum/go-ethereum#24554 and btcsuite/btcd#1839
This is an attempt to resolve a Go module dependency issue that arises
when both 'github.com/btcsuite/btcd/btcec/v2' and the older, non-v2
btcd module are required as dependencies.
* eth/downloader: remove stale beacon headers as backfilling progresses
* eth/downloader: remove leftover from a previous design
* eth/downloader: do partial beacon cleanups if chain is large
* eth/downloader: linter != heart
This adds a tools.go file to import all command packages used for
go:generate. Doing so makes it possible to execute go-based code
generators using 'go run', locking in the tool version using go.mod.
Co-authored-by: Felix Lange <fjl@twurst.com>
If a pending contract call errors, return that error right away rather
than ignoring it to allow an error somewhere else. This is helpful for
callers to know if perhaps a call failed because of the context deadline
being expired. This change mirrors the behavior of non-pending contract
calls.
This PR fixes a few panics in the chain marker benchmarks. The root
cause for panic is in chain marker the genesis header/block is not
accessible, while it's expected to be obtained in tests. So this PR
avoids touching genesis header at all to avoid panic.
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but t.TempDir handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* eth/tracers: refactor traceTx to separate out struct logging
review fix
Update eth/tracers/api.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Mv ExecutionResult type to logger package
review fix
impl GetResult for StructLogger
make formatLogs private
confused exit and end..
account for intrinsicGas in structlogger, fix TraceCall test
Add Stop method to logger
Simplify traceTx
Fix test
rm logger from blockchain test
account for refund in structLogger
* use tx hooks in struct logger
* minor
* avoid executionResult in struct logger
* revert blockchain test changes
* core,eth: add empty tx logger hooks
* core,eth: add initial and remaining gas to tx hooks
* store tx gasLimit in js tracer
* use gasLimit to compute intrinsic cost for js tracer
* re-use rules in transitiondb
* rm logs
* rm logs
* Mv some fields from Start to TxStart
* simplify sender lookup in prestate tracer
* mv env to TxStart
* Revert "mv env to TxStart"
This reverts commit 656939634b9aff19f55a1cd167345faf8b1ec310.
* Revert "simplify sender lookup in prestate tracer"
This reverts commit ab65bce48007cab99e68232e7aac2fe008338d50.
* Revert "Mv some fields from Start to TxStart"
This reverts commit aa50d3d9b2559addc80df966111ef5fb5d0c1b6b.
* fix intrinsic gas for prestate tracer
* add comments
* refactor
* fix test case
* simplify consumedGas calc in prestate tracer
Trie tracer is an auxiliary tool to capture all deleted nodes
which can't be captured by trie.Committer. The deleted nodes
can be removed from the disk later.
This PR adds a ExtraAllowedPath field to Solidity and exposes two APIs: CompileSource and CompileFiles, which were hidden inside CompileSolidityString and CompileSolidity before.
* eth/catalyst: only apply block if we actually have the state
* add header to payload queue
* Update cmd/geth/dbcmd.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* pipeline state verification
* update codes and add logs for debug
* refactor
* update and add logs
* refactor
* refactor
* remove unneeded logs
* fix a blocking issue
* fix sync issue when force kill
* remove logs
* refactor based on comments
* refactor based on comments
* refactor based on comments
* refactor based on comments
* refactor based on comments
* fix a deadlock issue
* fix merkle root mismatch issue during sync
* refactor based on review comments
* remove unnecessary code
* remove unnecessary code
* refactor based on review comments
* change based on comments
* refactor
* uew dummyRoot to replace emptyRoot
* add nil check
* add comments
* remove unneeded codes
* format comments
Co-authored-by: forcodedancing <liguo.fudan@gmail.com>
* add sharedStorage for prefetching to L1
* remote originStorage in stateObjects
* fix core
* fix bug of sync map
* remove read lock when get & set keys
* statedb copy use CopyWithSharedStorage
* reduce lock access
* fix comment
* avoid sharedPool effects on other modules
* remove tryPreload
* fix comment
* fix var name
* fix lint
* fix L1 miss data && data condition
* fix comment
* cmd/geth: only check for presence of legacy receipts if developer mode is not enabled
* cmd/geth: degrade log level
* cmd/geth: fix format
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd,core: add simple legacy receipt converter
core/rawdb: use forEach in migrate
core/rawdb: batch reads in forEach
core/rawdb: make forEach anonymous fn
cmd/geth: check for legacy receipts on node startup
fix err msg
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix log
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix some review comments
add warning to cmd
drop isLegacy fn from migrateTable params
add test for windows rename
test replacing in windows case
* minor fix
* sanity check for tail-deletion
* add log before moving files around
* speed-up hack for mainnet
* fix mainnet check, use networkid instead
* check mainnet genesis
* review fixes
* resume previous migration attempt
* core/rawdb: lint fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/beacon: eth/catalyst: updated engine api to new version
* core: implement exchangeTransitionConfig
* core/beacon: prevRandao instead of Random
* eth/catalyst: Fix ExchangeTransitionConfig, add test
* eth/catalyst: stop external miners on TTD reached
* node: implement --authrpc.vhosts flag
* core: allow for config override on non-mainnet networks
* eth/catalyst: fix peters comments
* eth/catalyst: make stop remote sealer more explicit
* eth/catalyst: add log output
* cmd/utils: rename authrpc.host to authrpc.addr
* eth/catalyst: disable the disabling of the miner
* eth: core: remove notion of terminal pow block
* eth: les: more of peters nitpicks
This updates the no-cgo implementations in the crypto package to use
the github.com/btcsuite/btcd/btcec/v2 module instead of the older btcec
package that was part of the main github.com/btcsuite/btcd module.
name old time/op new time/op delta
EcrecoverSignature-32 198µs ± 0% 144µs ± 0% -27.11%
VerifySignature-32 177µs ± 0% 128µs ± 0% -27.44%
DecompressPubkey-32 20.9µs ± 0% 10.1µs ± 0% -51.51%
Use (*ModNScalar).IsOverHalfOrder instead of math/big.Int when checking
for malleable signatures.
* cmd, eth: Rename whitelist argument to peer.requiredblocks
* eth/ethconfig: document PeerRequiredBlocks better
* cmd/utils: rename new flag to --eth.requiredblocks
Co-authored-by: Felix Lange <fjl@twurst.com>
* trie: fix memory leak in trie iterator
In the trie iterator, live nodes are tracked in a stack while iterating.
Popped node states should be explictly set to nil in order to get
garbage-collected.
* trie: fix empty trie iterator
* eth/downloader: implement beacon sync
* eth/downloader: fix a crash if the beacon chain is reduced in length
* eth/downloader: fix beacon sync start/stop thrashing data race
* eth/downloader: use a non-nil pivot even in degenerate sync requests
* eth/downloader: don't touch internal state on beacon Head retrieval
* eth/downloader: fix spelling mistakes
* eth/downloader: fix some typos
* eth: integrate legacy/beacon sync switchover and UX
* eth: handle UX wise being stuck on post-merge TTD
* core, eth: integrate the beacon client with the beacon sync
* eth/catalyst: make some warning messages nicer
* eth/downloader: remove Ethereum 1&2 notions in favor of merge
* core/beacon, eth: clean up engine API returns a bit
* eth/downloader: add skeleton extension tests
* eth/catalyst: keep non-kiln spec, handle mining on ttd
* eth/downloader: add beacon header retrieval tests
* eth: fixed spelling, commented failing tests out
* eth/downloader: review fixes
* eth/downloader: drop peers failing to deliver beacon headers
* core/rawdb: track beacon sync data in db inspect
* eth: fix review concerns
* internal/web3ext: nit
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR adds the `NextBaseFeePerGas` to `Block` and `EffectiveTip` to `Transaction` to make it easier for clients to compute fee history themselves via graphql queries.
* core/rawdb, cmd, ethdb, eth: implement freezer tail deletion
* core/rawdb: address comments from martin and sina
* core/rawdb: fixes cornercase in tail deletion
* core/rawdb: separate metadata into a standalone file
* core/rawdb: remove unused code
* core/rawdb: add random test
* core/rawdb: polish code
* core/rawdb: fsync meta file before manipulating the index
* core/rawdb: fix typo
* core/rawdb: address comments
This change speeds up trie hashing and all other activities that require
RLP encoding of trie nodes by approximately 20%. The speedup is achieved by
avoiding reflection overhead during node encoding.
The interface type trie.node now contains a method 'encode' that works with
rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code.
trie.hasher, which is pooled to avoid allocations, now maintains an
EncoderBuffer. This means memory resources related to trie node encoding
are tied to the hasher pool.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth, cmd: allow FdLimit to be set in config/command line (#24148)
* eth/ethconfig: format code
* cmd, eth/ethconfig: simplify fdlimit arg, disallow toml
* cnd/utils: make fdlimit setting nicer on the logs
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* rpc, node: refactor request validation and add jwt validation
* node, rpc: fix error message, ignore engine api in RegisterAPIs
* node: make authenticated port configurable
* eth/catalyst: enable unauthenticated version of engine api
* node: rework obtainjwtsecret (backport later)
* cmd/geth: added auth port flag
* node: happy lint, happy life
* node: refactor authenticated api
Modifies the authentication mechanism to use default values
* node: trim spaces and newline away from secret
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This replaces the simple selector parser in signer/fourbyte with one that
can actually handle most types. The new parser is added in accounts/abi
to also make it useable elsewhere.
* go.mod: update azure-storage-blob-go
update Azure/azure-storage-blob-go from v0.7.0 to v0.14.0.
relation #24396.
* internal/build: fix for breaking changes of azure-storage-blob-go
fix for breaking changes of update Azure/azure-storage-blob-go from v0.7.0 to v0.14.0.
relation #24396.
* internal/build: switch azure sdk from Azure/azure-storage-blob-go to Azure/azure-sdk-for-go/sdk/storage/azblob.
* internal/build refactor appending BlobItems
* internal/build: fix azure blobstore client to include container id
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.
The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.
To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.
Benchmark results:
name old time/op new time/op delta
EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8)
EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8)
name old speed new speed delta
EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8)
EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal)
This change adds a code generator tool for creating EncodeRLP method
implementations. The generated methods will behave identically to the
reflect-based encoder, but run faster because there is no reflection overhead.
Package rlp now provides the EncoderBuffer type for incremental encoding. This
is used by generated code, but the new methods can also be useful for
hand-written encoders.
There is also experimental support for generating DecodeRLP, and some new
methods have been added to the existing Stream type to support this. Creating
decoders with rlpgen is not recommended at this time because the generated
methods create very poor error reporting.
More detail about package rlp changes:
* rlp: externalize struct field processing / validation
This adds a new package, rlp/internal/rlpstruct, in preparation for the
RLP encoder generator.
I think the struct field rules are subtle enough to warrant extracting
this into their own package, even though it means that a bunch of
adapter code is needed for converting to/from rlpstruct.Type.
* rlp: add more decoder methods (for rlpgen)
This adds new methods on rlp.Stream:
- Uint64, Uint32, Uint16, Uint8, BigInt
- ReadBytes for decoding into []byte
- MoreDataInList - useful for optional list elements
* rlp: expose encoder buffer (for rlpgen)
This exposes the internal encoder buffer type for use in EncodeRLP
implementations.
The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to
encBuffer. It is implemented this way to ensure the global encBuffer pool
is handled correctly.
This PR adds an addtional API called `NewBatchWithSize` for db
batcher. It turns out that leveldb batch memory allocation is
super inefficient. The main reason is the allocation step of
leveldb Batch is too small when the batch size is large. It can
take a few second to build a leveldb batch with 100MB size.
Luckily, leveldb also offers another API called MakeBatch which can
pre-allocate the memory area. So if the approximate size of batch is
known in advance, this API can be used in this case.
It's needed in new state scheme PR which needs to commit a batch of
trie nodes in a single batch. Implement the feature in a seperate PR.
This functionality is needed in new path-based storage scheme, but
can be implemented in a seperate PR though.
When an account is deleted, then all the storage slots should be
nuked out from the disk as well. In hash-based storage scheme they
are still left in the disk but in new scheme, they will be iterated
and marked as deleted.
But why the NodeBlob API is needed in this scenario? Because when
the node is marked deleted, the previous value is also required to
be recorded to construct the reverse diff.
This change makes it so WaitMined no longer logs an error when the receipt
is unavailable. It also changes the simulated backend to return NotFound for
unavailable receipts, just like ethclient does.
I believe the sentence is attempting to explain that the URL is "[used] by upper layers to define a sorting order over all wallets from multiple backends."
* eth/tracers: add initial native prestate tracer
* fix balance hex
* handle prestate for tx from and to
* drop created contract from prestate
* fix sender balance
* use switch instead
Co-authored-by: Martin Holst Swende <martin@swende.se>
* minor fix
* lookup create2 account
* mv code around a bit
* check stackLen for create2
* fix transfer tx for js prestate tracer
* fix create2 addr
* track extcodehash in js prestate tracer
Co-authored-by: Martin Holst Swende <martin@swende.se>
When talking to an HTTP2 server, there are situations where it needs to
"rewind" the Request.Body. To allow this, we have to set up the Request.GetBody
function to return a brand new instance of the body.
If not set, we can end up with the following error:
http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
See this commit for more information: cffdcf672a
* eth, miner: remove duplicated code
* eth/catalyst: remove unneeded code
* miner: keep update pending state even the Merge is happened
* eth, miner: rebase
* miner: fix tests
* eth, miner: address comments from marius
* miner: use empty zero randomness for pending blocks after the merge
* eth/catalyst: gofmt
* miner: add warning log for state recovery
* miner: ignore uncles for post-merge blocks
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* trie/proof: edge case for VerifyRangeProof
* more consistency with other tests in the file
* trie: fix test todo
Co-authored-by: Martin Holst Swende <martin@swende.se>
This replaces the sketchy and undocumented string context keys for HTTP requests
with a defined interface. Using string keys with context is discouraged because
they may clash with keys created by other packages.
We added these keys to make connection metadata available in the signer, so this
change also updates signer/core to use the new PeerInfo API.
* eth/catalyst: evict old payloads, type PayloadID
* eth/catalyst: added tracing info to engine api
* eth/catalyst: add test for create payload timestamps
* catalyst: better logs
* eth/catalyst: computePayloadId return style
* catalyst: add queue for payloads
* eth/catalyst: nitpicks
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* offline block prune
* update
* update
* update and add unit test
* addressed comments from walt
* Addressed comments from walt and Igor
* ensure MPT and snapshot matched
* add one more parameter to indicate blockprune
* update the logic of creating freezerDb
* update flag command description
* expose the function for db inspect the offset/startBlockNumber
* add flags to inspect prune info
* rename flag of reserved-recent-blocks to block-amount-reserved
* addressed comments from walt
* handle the case of command interruption
* refined goimports
* addressed comments from walt
* change the logic as restarting prune after interruption
* addressed comments
* reclaimed freezer logic
* introduce flag to enable/disable check between MPT and snapshot
* update the logic of frozen field in freezerDB
* update the code in all places related to freezer change
* addressed comments from dylan
* update the logic for backup block difficulty
* addressed comments from dylan
* freezer: add readonly flag to table
* freezer: enforce readonly in table repair
* freezer: enforce readonly in newFreezer
* minor fix
* minor
* core/rawdb: test that writing during readonly fails
* rm unused log
* check readonly on batch append
* minor
* Revert "check readonly on batch append"
This reverts commit 2ddb5ec4ba7534bf6edbdfec158ea99a2eed5036.
* review fixes
* minor test refactor
* attempt at fixing windows issue
* add comment re windows sync issue
* k->kind
* open readonly db for genesis check
Co-authored-by: Martin Holst Swende <martin@swende.se>
Using curve P384 for encryption causes the error "ecies: shared key params
are too big". Also, readme.md says curve P384 should use AES192 not AES256.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core: implement eip-4399 random opcode
* core: make vmconfig threadsafe
* core: miner: pass vmConfig by value not reference
* all: enable 4399 by Rules
* core: remove diff (f)
* tests: set proper difficulty (f)
* smaller diff (f)
* eth/catalyst: nit
* core: make RANDOM a pointer which is only set post-merge
* cmd/evm/internal/t8ntool: fix t8n tracing of 4399
* tests: set difficulty
* cmd/evm/internal/t8ntool: check that baserules are london before applying the merge chainrules
SyncProgress was modified in PR #23576 to add the fields reported for
snap sync. The PR also changed ethclient to use the SyncProgress struct
directly instead of wrapping it for hex-decoding. This broke the
SyncProgress method.
Fix it by putting back the custom wrapper. While here, also put back the
fast sync related fields because SyncProgress is stable API and thus
removing fields is not allowed.
Fixes#24180Fixes#24176
Fixes#24167
New behaviour is that the endpoint returns results only for available
blocks without returning an error when it doesn't find a block. Note we
skip any block after a non-existent block.
This adds a header fetch for every block in range (even if header
is not needed). Alternatively, we could do the check in every field's
resolver method to avoid this overhead.
* eth/tracers: implement debug.intermediateRoots (#23594)
This PR implements a new debug method, which I've talked briefly about to some other client developers. It allows the caller to obtain the intermediate state roots for a block (which might be either a canon block or a 'bad' block).
Signed-off-by: wenbiao <delweng@gmail.com>
* core, rpc: disable memory output by default in traces (#23558)
* core: cmd: invert disableMemory
* core: fix missed inversion
* cmd/evm: preserve Flags but change default value
* Apply suggestions from code review
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: abort evm execution when trace is aborted (#23580)
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: avoid unsyncronized mutations on trie database (#23632)
This PR fixes an issue in traceChain, where the statedb Commit operation was performed asynchronously with dereference-operations agains the underlying trie.Database instance. Due to how the reference counting works within the trie database (where parent count is recursively updated when new parents are added), doing dereferencing in the middle of Commit can cause the refcount to become wrong, leading to an inconsistent state.
This was fixed by doing Commit/Deref from the same routine.
Signed-off-by: wenbiao <delweng@gmail.com>
* core,eth: call frame tracing (#23087)
This change introduces 2 new optional methods; `enter()` and `exit()` for js tracers, and makes `step()` optiona. The two new methods are invoked when entering and exiting a call frame (but not invoked for the outermost scope, which has it's own methods). Currently these are the data fields passed to each of them:
enter: type (opcode), from, to, input, gas, value
exit: output, gasUsed, error
The PR also comes with a re-write of the callTracer. As a backup we keep the previous tracing script under the name `callTracerLegacy`. Behaviour of both tracers are equivalent for the most part, although there are some small differences (improvements), where the new tracer is more correct / has more information.
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: re-write of 4byte tracer using enter/exit (#23622)
* eth/tracers: add re-write of 4byte tracer using enter/exit
* eth/tracers: fix 4byte indent
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: tx.BaseFee not implemented
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: do the JSON serialization via .js to capture C faults
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: fix callTracer fault handling (#23667)
* eth/tracers: fix calltracer fault handling
* eth/tracers: fix calltracer indentation
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: invoke enter/exit on 0-value calls to inex accounts (#23828)
Signed-off-by: wenbiao <delweng@gmail.com>
* eth: make traceChain avoid OOM on long-running tracing (#23736)
This PR changes long-running chain tracing, so that it at some points releases the memory trie db, and switch over to a fresh disk-backed trie.
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: expose contextual infos (block hash, tx hash, tx index)
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: redefine Context
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: support for golang tracers + add golang callTracer (#23708)
* eth/tracers: add basic native loader
* eth/tracers: add GetResult to tracer interface
* eth/tracers: add native call tracer
* eth/tracers: fix call tracer json result
* eth/tracers: minor fix
* eth/tracers: fix
* eth/tracers: fix benchTracer
* eth/tracers: test native call tracer
* eth/tracers: fix
* eth/tracers: rm extra make
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: rm extra make
* eth/tracers: make callFrame private
* eth/tracers: clean-up and comments
* eth/tracers: add license
* eth/tracers: rework the model a bit
* eth/tracers: move tracecall tests to subpackage
* cmd/geth: load native tracers
* eth/tracers: minor fix
* eth/tracers: impl stop
* eth/tracers: add native noop tracer
* renamings
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: more renamings
* eth/tracers: make jstracer non-exported, avoid cast
* eth/tracers, core/vm: rename vm.Tracer to vm.EVMLogger for clarity
* eth/tracers: minor comment fix
* eth/tracers/testing: lint nitpicks
* core,eth: cancel evm on nativecalltracer stop
* Revert "core,eth: cancel evm on nativecalltracer stop"
This reverts commit 01bb908790a369c1bb9d3937df9325c6857bf855.
* eth/tracers: linter nits
* eth/tracers: fix output on err
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: make native calltracer default (#23867)
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: package restructuring (#23857)
* eth/tracers: restructure tracer package
* core/vm/runtime: load js tracers
* eth/tracers: mv bigint js code to own file
* eth/tracers: add method docs for native tracers
* eth/tracers: minor doc fix
* core,eth: cancel evm on nativecalltracer stop
* core/vm: fix failing test
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: ethapi.TransactionArgs was not merged
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: fix the api_test with ErrInsufficientFunds to ErrInsufficientFundsForTransfer
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: check posa before statedb.Prepare in IntermiateRoots api
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: make js calltracer default, compatible with old version
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: fix the default callTrace name of callTracerJs
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* Revert "eth/tracers: fix the default callTrace name of callTracerJs"
This reverts commit 62a3bc215d.
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* Revert "eth/tracers: make js calltracer default, compatible with old version"
This reverts commit 85ef42c0ea.
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: fix the variable race condition
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* core/vm: reverse bit order in bytes of code bitmap
This bit order is more natural for bit manipulation operations and we
can eliminate some small number of CPU instructions.
* core/vm: drop lookup table
* consensus: use the maxGasLimit constant to check the header.GasLimit to avoid creating new variables repeatedly
* consensus: check the header.GasLimit by the public constant MaxGasLimit
* consensus: check the header.GasLimit by the constant MaxGasLimit
Previously, Ctrl-C (SIGINT) was ignored during JS execution, so it was not
possible to get out of infinite loops in the console. With this change,
Ctrl-C now interrupts JS.
Fixes#23344
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR reduces the amount of work we do when answering header queries, e.g. when a peer
is syncing from us.
For some items, e.g block bodies, when we read the rlp-data from database, we plug it
directly into the response package. We didn't do that for headers, but instead read
headers-rlp, decode to types.Header, and re-encode to rlp. This PR changes that to keep it
in RLP-form as much as possible. When a node is syncing from us, it typically requests 192
contiguous headers. On master it has the following effect:
- For headers not in ancient: 2 db lookups. One for translating hash->number (even though
the request is by number), and another for reading by hash (this latter one is sometimes
cached).
- For headers in ancient: 1 file lookup/syscall for translating hash->number (even though
the request is by number), and another for reading the header itself. After this, it
also performes a hashing of the header, to ensure that the hash is what it expected. In
this PR, I instead move the logic for "give me a sequence of blocks" into the lower
layers, where the database can determine how and what to read from leveldb and/or
ancients.
There are basically four types of requests; three of them are improved this way. The
fourth, by hash going backwards, is more tricky to optimize. However, since we know that
the gap is 0, we can look up by the parentHash, and stlil shave off all the number->hash
lookups.
The gapped collection can be optimized similarly, as a follow-up, at least in three out of
four cases.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR fixes a special corner case in transaction indexing.
When the chain is rewound by SetHead to a historical point which is even lower than the transaction indexes tail, then system will report Failed to decode block body error all the time, because the relevant blocks are already deleted.
In order to avoid this "non-critical-but-annoying" issue, we can recap the indexing target to head+1(to is excluded, so it means indexing transactions from 0 to head).
Previous link leads to incorrect (more recent) version of web3.js docs.
go-ethereum uses v0.20.1. The docs for 0.2x.x have been archived at this Github link.
* core/vm: Remove interpreter loop interruption check
* core/vm: Unit test for interpreter loop interruption
* core/vm: Check for interpreter loop abort on every jump
The `structs` map is populated by iterating over all methods except the constructor, which results in a nil-pointer dereference.
I've first reproduced the problem with a new test and then implemented the fix.
Co-authored-by: Arran Schlosberg <me@arranschlosberg.com>
* eth/tracers: Add support for REVERT in evmdis_tracer
* evm/tracers: Fix evmdis_tracer to use SELFDESTRUCT instead of SUICIDE
* eth/tracers: Regenerate tracer library
* core/vm: Move interpreter.ReadOnly check into the opcode implementations
Also remove the same check from the interpreter inner loop.
* core/vm: Remove obsolete operation.writes flag
* core/vm: Capture fault states in logger
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/vm: Remove panic added for testing
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/vm: break loop on any error
* core/vm: move ErrExecutionReverted to opRevert()
* core/vm: use "stop token" to stop the loop
* core/vm: unconditionally pc++ in the loop
* core/vm: set return data in instruction impls
Trim the search key from head as it's being pushed deeper into the trie. Previously the search key was never modified but each node kept information how to slice and compare it in keyOffset. Now the keyOffset is not needed as this information is included in the slice of the search key. This way the keyOffset can be removed and key manipulation
simplified.
* reannouce local pending transactions
* add tests for tx_pool reannouce local pending transactions
* add tests for handler reannounce local pending transactions
* all: work for eth1/2 transtition
* consensus/beacon, eth: change beacon difficulty to 0
* eth: updates
* all: add terminalBlockDifficulty config, fix rebasing issues
* eth: implemented merge interop spec
* internal/ethapi: update to v1.0.0.alpha.2
This commit updates the code to the new spec, moving payloadId into
it's own object. It also fixes an issue with finalizing an empty blockhash.
It also properly sets the basefee
* all: sync polishes, other fixes + refactors
* core, eth: correct semantics for LeavePoW, EnterPoS
* core: fixed rebasing artifacts
* core: light: performance improvements
* core: use keyed field (f)
* core: eth: fix compilation issues + tests
* eth/catalyst: dbetter error codes
* all: move Merger to consensus/, remove reliance on it in bc
* all: renamed EnterPoS and LeavePoW to ReachTDD and FinalizePoS
* core: make mergelogs a function
* core: use InsertChain instead of InsertBlock
* les: drop merger from lightchain object
* consensus: add merger
* core: recoverAncestors in catalyst mode
* core: fix nitpick
* all: removed merger from beacon, use TTD, nitpicks
* consensus: eth: add docstring, removed unnecessary code duplication
* consensus/beacon: better comment
* all: easy to fix nitpicks by karalabe
* consensus/beacon: verify known headers to be sure
* core: comments
* core: eth: don't drop peers who advertise blocks, nitpicks
* core: never add beacon blocks to the future queue
* core: fixed nitpicks
* consensus/beacon: simplify IsTTDReached check
* consensus/beacon: correct IsTTDReached check
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* all: mv loggers to eth/tracers
* core/vm: minor
* eth/tracers: tmp comment out testStoreCapture
* eth/tracers: uncomment and fix logger test
* eth/tracers: simplify test
* core/vm: re-add license
* core/vm: minor
* rename LogConfig to Config
* reannouce local pending transactions
* add tests for tx_pool reannouce local pending transactions
* add tests for handler reannounce local pending transactions
This PR fixes two problems in devp2p tests (and through them, hive).
- Make the output more detailed about what is returned (always print packet kind).
- Allow Ping response to unsolicited findnode.
Without this PR, nethermind fails a hive protocol test, and I misinterpreted the result (NethermindEth/nethermind#3617). Ergo, the output was not fool-proof.
* cmd/evm: rename t8n args to improve clarity when tracing
* cmd/evm: add back removed tracing flags and note that they are deprecated
* cmd/evm: add warning when using deprecated flag
This fixes a bug in TransactionSender where it would return the
zero address for transactions where the sender address wasn't
cached already.
Co-authored-by: Felix Lange <fjl@twurst.com>
* cmd, core: add flag --dev.gaslimit to allow configuring initial block gas limit in dev mode
* core: use provided gaslimit
Co-authored-by: Martin Holst Swende <martin@swende.se>
This was apparently recently changed by Cloudflare, and
began returning an error: 'TTL must be between 60 and 86400
seconds, or 1 for Automatic'
Date: 2021-11-10 15:25:20-08:00
Signed-off-by: meows <b5c6@protonmail.com>
Debugging recent geth failures in hive, it took a while to realize that it's because
geth doesn't support eth/65 any longer. This PR makes such failures a bit more
easy to figure out.
The price limit is supposed to exclude transactions with too low fee
amount. Before EIP-1559, it was sufficient to check the limit against
the gas price of the transaction. After 1559, it is more complicated
because the concept of 'transaction gas price' does not really exist.
When mining, the price limit is used to exclude transactions below a
certain effective fee amount. This change makes it apply the same check
earlier, in tx validation. Transactions below the specified fee amount
cannot enter the pool.
Fixes#23837
This PR fixes a problem which arises on clique networks when there is a network stall. Previously, the worker packages were tracked, even if the sealing engine decided not to seal the block (due to clique rules about recent signing). These tracked-but-not-sealed blocks kept building up in memory.
This PR changes the situation so the sealing engine instead returns an error, and the worker can thus un-track the package.
This avoids quadratic time complexity in the lookup of the batch element
corresponding to an RPC response. Unfortunately, the new approach
requires additional memory for the mapping from ID to index.
Fixes#22805
When we map a file for generating the DAG, we do a simple truncate to e.g. 1Gb. This is fine, even if we have nowhere near 1Gb disk available, as the actual file doesn't take up the full 1Gb, merely a few bytes. When we start generating into it, however, it eventually crashes with a unexpected fault address .
This change fixes it (on linux systems) by using the Fallocate syscall, which preallocates suffcient space on disk to avoid that situation.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR offers two more database sub commands for exporting and importing data.
Two exporters are implemented: preimage and snapshot data respectively.
The import command is generic, it can take any data export and import into leveldb.
The data format has a 'magic' for disambiguation, and a version field for future compatibility.
It is because write known block only checks block and state without snapshot, which could lead to gap between newest snapshot and newest block state. However, new blocks which would cause snapshot to become fixed were ignored, since state was already known.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/state/snapshot: fix BAD BLOCK error when snapshot is generating
* core/state/snapshot: alternative fix for the snapshot generator
* add comments and minor update
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/state/snapshot: fix BAD BLOCK error when snapshot is generating
* core/state/snapshot: alternative fix for the snapshot generator
* add comments and minor update
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Ziyuan Zhong <zzy.albert@163.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Some benchmarks in eth/filters were not good: they weren't reproducible, relying on geth chaindata to be present.
Another one was rejected because the receipt was lacking a backing transcation.
The p2p simulation benchmark had a lot of the warnings below, due to the framework calling both
Stop() and Close(). Apparently, the simulated adapter is the only implementation which has a Close(),
and there is no need to call both Stop and Close on it.
* core: write test showing that TD is not stored properly at genesis
The ToBlock method applies a default value for an empty
difficulty value. This default is not carried over through the Commit
method because the TotalDifficulty database write writes the
original difficulty value (nil) instead of the defaulty value
present on the genesis Block.
Date: 2021-10-22 08:25:32-07:00
Signed-off-by: meows <b5c6@protonmail.com>
* core: write TD value from Block, not original genesis value
This an issue where a default TD value was not written to
the database, resulting in a 0 value TD at genesis.
A test for this issue was provided at 90e3ffd393
Date: 2021-10-22 08:28:00-07:00
Signed-off-by: meows <b5c6@protonmail.com>
* core: fix tests by adding GenesisDifficulty to expected result
See prior two commits.
Date: 2021-10-22 09:16:01-07:00
Signed-off-by: meows <b5c6@protonmail.com>
* les: fix test with genesis change
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.
* core/state/snapshot: fix BAD BLOCK error when snapshot is generating
* core/state/snapshot: alternative fix for the snapshot generator
* add comments and minor update
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core: fix warning flagging the use of DeepEqual on error
* apply the same change everywhere possible
* revert change that was committed by mistake
* fix build error
* Update config.go
* revert changes to ConfigCompatError
* review feedback
Co-authored-by: Felix Lange <fjl@twurst.com>
* cmd/puppeth: use geth's prompt to read input
* remove wizard.in
* cmd/puppeth: fix compilation errors
* reset prompt (don't exit) on receiving ctrl-c
* make promptInput spin until the user enters a value or interrupts (ctrl-d)
* make promptInput use parameter
Co-authored-by: Martin Holst Swende <martin@swende.se>
Adds suppor for passing regular strings to db `put`/`get`/`delete`, to avoid having to hex-encode when operating on fixed-key items like `SnapshotSyncStatus`, `SnapshotRecovery` etc.
Signed-off-by: Maxim Zhiburt <zhiburt@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This is the initial step for support of Solidity errors in contract bindings.
As of this change, errors can be decoded, but are not supported in
bindings yet.
Closes#23157
This fixes a bug where gas-related fields of the TransactOpts passed
to transaction methods would be modified, skipping gas estimation for
subsequent transactions.
Co-authored-by: Yondon Fu <yondon.fu@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This doesn't fix all go-critic warnings, just the most serious ones.
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth,rpc: allow for flag configured timeouts for eth_call
* lint: account for package-local import order
* cr: rename `rpc.calltimeout` to `rpc.evmtimeout`
This removes some code:
- The clique engine calculated the snapshot twice when verifying headers/blocks.
- The method GetBlockHashesFromHash in Header/Block/Lightchain was only used by tests. It
is now removed from the API.
- The method GetTdByHash internally looked up the number before calling GetTd(hash, num).
In many cases, callers already had the number, and used this method just because it has a
shorter name. I have removed the method to make the API surface smaller.
* build: increase thread stack size when running alpine linux
* review feedback: force a stack size of 8M on all linux distribs
* fix missing extldflags
node.Node no longer registers any account manager backends by default,
they need to be registered explicitly.
For ethash-based tests, we actually don't need any accounts in the miner
keystore. Just set the etherbase instead to make mining work. For
clique, the signer account must be in the keystore.
The change also adds interrupt handling in stress tests.
This fixes a data race on worker.current by moving the call to StopPrefetcher
into the main loop.
The commit also contains fixes for two other races in unit tests of unrelated packages.
Go 1.17.2 fixes some miscompilation issues on amd64 and a runtime issue with timers.
While the upgrade is not strictly necessary for go-ethereum right now, it is still good
to be up-to-date.
Fixes#23681
After the fix I get the address 0x6d6d02e83c4ced98204e20126acf27e9d87b8af2 for the
tx mentioned in the ticket, which agrees with etherscan.
This change removes misuses of sync.WaitGroup in BlockChain. Before this change,
block insertion modified the WaitGroup counter in order to ensure that Stop would wait
for pending operations to complete. This was racy and could even lead to crashes
if Stop was called at an unfortunate time. The issue is resolved by adding a specialized
'closable' mutex, which prevents chain modifications after stopping while also
synchronizing writers with each other.
Co-authored-by: Felix Lange <fjl@twurst.com>
This resolves a long-standing TODO. The point of copying the address is
to ensure that all data referenced by types.Transaction is independent of the
data passed into the constructor.
Co-authored-by: Felix Lange <fjl@twurst.com>
The test did not synchronize with per-case goroutines, and thus didn't notice
that some tests were just hanging. This change adds missing synchronization
and fixes the broken tests.
This PR fixes an issue in traceChain, where the statedb Commit operation was performed asynchronously with dereference-operations agains the underlying trie.Database instance. Due to how the reference counting works within the trie database (where parent count is recursively updated when new parents are added), doing dereferencing in the middle of Commit can cause the refcount to become wrong, leading to an inconsistent state.
This was fixed by doing Commit/Deref from the same routine.
* core/types: rm extranous check in test
* core/rawdb: add lightweight types for block logs
* core/rawdb,eth: use lightweight accessor for log filtering
* core/rawdb: add bench for decoding into rlpLogs
This PR implements a new debug method, which I've talked briefly about to some other client developers. It allows the caller to obtain the intermediate state roots for a block (which might be either a canon block or a 'bad' block).
* internal: support optional filter expression for debug.stacks
* internal/debug: fix string regexp
* internal/debug: support searching for line numbers too
* ethclient/gethclient: fix flaky test (due to map key ordering)
* accounts/keystore: fix test failing due to rand collision due to low time resolution on windows
This change introduces 2 new optional methods; `enter()` and `exit()` for js tracers, and makes `step()` optiona. The two new methods are invoked when entering and exiting a call frame (but not invoked for the outermost scope, which has it's own methods). Currently these are the data fields passed to each of them:
enter: type (opcode), from, to, input, gas, value
exit: output, gasUsed, error
The PR also comes with a re-write of the callTracer. As a backup we keep the previous tracing script under the name `callTracerLegacy`. Behaviour of both tracers are equivalent for the most part, although there are some small differences (improvements), where the new tracer is more correct / has more information.
* cmd/devp2p/internal/ethtest: only use eth66 if eth66 is negotiated
* cmd/devp2p/internal/ethtest: switch on concrete type not pointer
* cmd/devp2p/internal/ethtest: switch on concrete type not pointer
This PR ensures that wiping all data associated with a node (apart from its nodekey)
will not generate already used sequence number for the ENRs, since all remote nodes
would reject them until they out-number the previously published largest one.
The big complication with this scheme is that every local update to the ENR can
potentially bump the sequence number by one. In order to ensure that local updates
do not outrun the clock, the sequence number is a millisecond-precision timestamp,
and updates are throttled to occur at most once per millisecond.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change is a rewrite of the freezer code.
When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.
In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.
To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.
Co-authored-by: Felix Lange <fjl@twurst.com>
* remove rpc flags
* remove legacy rpc flags
* remove legacy rpc flags
* remove legacy rpc commands
* (hopefully) fix most of the build errors
* fix build errors
https://app.travis-ci.com/github/ethereum/go-ethereum/jobs/530318686
* cmd/utils: fix syntax error
* empty commit to unbreak travis ci
* fix syntax error
* syntax fixes
* syntax fixes
* fix
fixes "cmd/geth/usage.go:234:7: expected '(', found init (typecheck)"
* fix
* various fixes in usage.go
* various fixes in flags.go
* adds extra space
reverts the spacing to how it was before I resolved the merge conflict
* more fixes in usage.go
* fix
fix for cmd/geth/usage.go:243:17: expected operand, found ':=' (typecheck) in travis
* Update cmd/utils/flags.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* fix error
fixes these errors:
cmd/utils/flags_legacy.go:21:2: "strings" imported but not used (typecheck)
"strings"
^
cmd/utils/flags_legacy.go:24:2: "github.com/ethereum/go-ethereum/node" imported but not used (typecheck)
"github.com/ethereum/go-ethereum/node"
^
* goimports
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/debug: remove deprecated flags
The removed flags are removed in the main portion of geth, this removes it internally too.
* internal/debug: remove legacy --debug and legacy --backtrace flag
* Update flags.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Currently, setDefaults overwrites the transaction input value if only input is provided. This causes personal_sendTransaction to estimate the gas based on a transaction with empty data. eth_estimateGas never calls setDefaults so it was unaffected by this.
Currently rpc.BlockNumber is marshalled to JSON as a numeric value, which is
wrong because BlockNumber.UnmarshalJSON() wants it to either be hex-encoded
or string "earliest"/"latest"/"pending". As a result, the call chain
rpc.BlockNumberOrHashWithNumber(123) -> json.Marshal() -> json.Unmarshal()
fails with error "cannot unmarshal object into Go value of type string".
As per benchmark results below, these changes speed up encoding/decoding of
consensus objects a bit.
name old time/op new time/op delta
EncodeRLP/legacy-header-8 384ns ± 1% 331ns ± 3% -13.83% (p=0.000 n=7+8)
EncodeRLP/london-header-8 411ns ± 1% 359ns ± 2% -12.53% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 251ns ± 0% 239ns ± 0% -4.97% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 319ns ± 0% 300ns ± 0% -5.89% (p=0.000 n=8+7)
EncodeRLP/legacy-transaction-8 389ns ± 1% 387ns ± 1% ~ (p=0.099 n=8+8)
EncodeRLP/access-transaction-8 607ns ± 0% 581ns ± 0% -4.26% (p=0.000 n=8+8)
EncodeRLP/1559-transaction-8 627ns ± 0% 606ns ± 1% -3.44% (p=0.000 n=8+8)
DecodeRLP/legacy-header-8 831ns ± 1% 813ns ± 1% -2.20% (p=0.000 n=8+8)
DecodeRLP/london-header-8 824ns ± 0% 804ns ± 1% -2.44% (p=0.000 n=8+7)
* rlp: pass length to byteArrayBytes
This makes it possible to inline byteArrayBytes. For arrays, the length is known
at encoder construction time, so the call to v.Len() can be avoided.
* rlp: avoid IsNil for pointer encoding
It's actually cheaper to use Elem first, because it performs less checks
on the value. If the pointer was nil, the result of Elem is 'invalid'.
* rlp: minor optimizations for slice/array encoding
For empty slices/arrays, we can avoid storing a list header entry in the
encoder buffer. Also avoid doing the tail check at encoding time because
it is already known at encoder construction time.
This PR adds functionality to the evm t8n to calculate ethash difficulty. If the caller does not provide a currentDifficulty, but instead provides the parentTimestamp (well, semi-optional, will default to 0 if not given), and parentDifficulty, we can calculate it for him.
The caller can also provide a parentUncleHash. In most, but not all cases, the parent uncle hash also affects the formula. If no such hash is provided (or, if the empty all-zero hash is provided), it's assumed that there were no uncles.
The new linter version is built with go 1.17 and thus includes the go vet
check for mismatched +build and go:build lines.
Fortunately, no new warnings are reported with this update.
In p2p/dial.go, conn.flags was accessed without using sync/atomic.
This race is fixed by removing the access.
In p2p/enode/iter_test.go, a similar race is resolved by writing the field atomically.
Co-authored-by: Felix Lange <fjl@twurst.com>
* fixed wrong state creation which resulted in fail of tracing transaction
* sorted package imports
* added the check to `standardTraceBlockToFile` that resolves the #314
Co-authored-by: alexqrid <>
This PR adds flag to enable InfluxDB v2 (--metrics.influxdbv2), flags for v2-specific features (--metrics.influxdb.token, --metrics.influxdb.bucket), also carries over addition of support for specifying organization (--metrics.influxdb.organization), but still retains backwards compatibility with InfluxDB v1.
* core/rawdb: implement sequential reads in freezer_table
* core/rawdb, ethdb: add sequential reader to db interface
* core/rawdb: lint nitpicks
* core/rawdb: fix some nitpicks
* core/rawdb: fix flaw with deferred reads not being performed
* core/rawdb: better documentation
* internal/ethapi/api: cap highest gas limit by account balance for 1559 fee parameters
* accounts/abi/bind: port gas limit cap for 1559 parameters to simulated backend
* accounts/abi/bind: add test for 1559 gas estimates for the simulated backend
* internal/ethapi/api: fix comment
* accounts/abi/bind/backends, internal/ethapi: unify naming style
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
In many cases, it's desireable to use already-signed transactions as input to the state transition, instead of having the evm sign them internally (for example to use malformed or not-yet-valid transactions). This PR adds support + docs for that feature.
This adds a check to verify that a sender-account does not have code, which means that the codehash is either `emptyCodeHash` _OR_ not present. The latter occurs IFF the sender did not previously exist, a situation which can only occur with zero cost gasprices.
This PR modifies the post-PING-send expectations to both be laxer and stricter: it doesn't care what order the packets arrive, but also verifies that exactly one PING and one PONG is returned.
Ticket #23273 found a flaw where we were unable to sign legacy-transactions
using the external signer, even if we're still on non-london network. That's
fixed in this PR.
Additionally, I found that even when supplying all parameters, it was impossible
to sign a london-transaction on an unsynched node. It's a pretty common usecase
that someone wants to sign a transaction using an unsynced 'vanilla' node,
providing all necessary data. Our setDefaults, however, insisted on checking the
current block against the config. This PR therefore adds a case, so that if both
MaxPriorityFeePerGas and MaxFeePerGas are provided, we accept them as given.
OBS This PR fixes a regression -- on current master, we are unable to sign a
london-transaction unless the node is synched, which may break scenarios where
geth (or clef) is used as a cold wallet.
Fixes#23273
* focus on performance improvement in many aspects.
1. Do BlockBody verification concurrently;
2. Do calculation of intermediate root concurrently;
3. Preload accounts before processing blocks;
4. Make the snapshot layers configurable.
5. Reuse some object to reduce GC.
add
* rlp: improve decoder stream implementation (#22858)
This commit makes various cleanup changes to rlp.Stream.
* rlp: shrink Stream struct
This removes a lot of unused padding space in Stream by reordering the
fields. The size of Stream changes from 120 bytes to 88 bytes. Stream
instances are internally cached and reused using sync.Pool, so this does
not improve performance.
* rlp: simplify list stack
The list stack kept track of the size of the current list context as
well as the current offset into it. The size had to be stored in the
stack in order to subtract it from the remaining bytes of any enclosing
list in ListEnd. It seems that this can be implemented in a simpler
way: just subtract the size from the enclosing list context in List instead.
* rlp: use atomic.Value for type cache (#22902)
All encoding/decoding operations read the type cache to find the
writer/decoder function responsible for a type. When analyzing CPU
profiles of geth during sync, I found that the use of sync.RWMutex in
cache lookups appears in the profiles. It seems we are running into
CPU cache contention problems when package rlp is heavily used
on all CPU cores during sync.
This change makes it use atomic.Value + a writer lock instead of
sync.RWMutex. In the common case where the typeinfo entry is present in
the cache, we simply fetch the map and lookup the type.
* rlp: optimize byte array handling (#22924)
This change improves the performance of encoding/decoding [N]byte.
name old time/op new time/op delta
DecodeByteArrayStruct-8 336ns ± 0% 246ns ± 0% -26.98% (p=0.000 n=9+10)
EncodeByteArrayStruct-8 225ns ± 1% 148ns ± 1% -34.12% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
DecodeByteArrayStruct-8 120B ± 0% 48B ± 0% -60.00% (p=0.000 n=10+10)
EncodeByteArrayStruct-8 0.00B 0.00B ~ (all equal)
* rlp: optimize big.Int decoding for size <= 32 bytes (#22927)
This change grows the static integer buffer in Stream to 32 bytes,
making it possible to decode 256bit integers without allocating a
temporary buffer.
In the recent commit 088da24, Stream struct size decreased from 120
bytes down to 88 bytes. This commit grows the struct to 112 bytes again,
but the size change will not degrade performance because Stream
instances are internally cached in sync.Pool.
name old time/op new time/op delta
DecodeBigInts-8 12.2µs ± 0% 8.6µs ± 4% -29.58% (p=0.000 n=9+10)
name old speed new speed delta
DecodeBigInts-8 230MB/s ± 0% 326MB/s ± 4% +42.04% (p=0.000 n=9+10)
* eth/protocols/eth, les: avoid Raw() when decoding HashOrNumber (#22841)
Getting the raw value is not necessary to decode this type, and
decoding it directly from the stream is faster.
* fix testcase
* debug no lazy
* fix can not repair
* address comments
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/tracers: improve tracing performance (#23016)
Improves the performance of debug.traceTransaction
* Update the import order
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: John.H <john.h@binance.com>
* internal/ethapi: revert + fix properly in al tracer
* internal/ethapi: use toMessage instead of creating new message
* internal/ethapi: remove ineffassign
* core: fix invalid unmarshalling, fix test
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/ethapi/api: use hexutil.uint for blockCount parameter instead of int for feeHistory
* return hex value for oldestBlock instead of number
* return uint64 from oracle.resolveBlockRange
* eth/gasprice: fixed test
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
When processing a transaction with London fork rules, EIP-1559 mandates
checking that the sender must have sufficient balance to cover gas * gasFeeCap.
In the EIP's pseudocode, this check happens after the value transferred by the
transaction has already been deducted. However, in go-ethereum, the balance
has not yet been updated when the check happens, and therefore needs to be
added explicitly.
Co-authored-by: Martin Holst Swende <martin@swende.se>
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like #17106 changed the
storage schema of logs to be the same as the consensus
encoding.
Support for the legacy format was dropped in #22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.
Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.
Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR fixes a false positive PONG 'to' endpoint mismatch seen in hive tests:
got {IP:172.17.0.7 UDP:44025 TCP:44025}, want {IP:172.17.0.7 UDP:44025 TCP:0}
Co-authored-by: Felix Lange <fjl@twurst.com>
Some tests take quite some time during exit, which I think causes
some appveyor fails like this:
https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3
One of the things that seem to take time during exit is waiting
(up to 100ms) for the syncbloom to close. This PR changes it to use
a channel, instead of looping with a 100ms wait.
This also includes some unrelated changes improving the reliability of
eth/fetcher tests, which fail a lot because they are time-dependent.
This commit adds the package gethclient which is similar to the ethclient
and implements some geth specific functionality.
Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes transaction sending in the case where an app using go-ethereum v1.10.4
is talking to a pre-EIP-1559 RPC node. In this case, the eth_maxPriorityFeePerGas
endpoint is not available and we can only rely on eth_gasPrice.
* core,eth/tracers: make isPrecompiled dependent on HF
* eth/tracers: use keys when constructing chain config struct
* eth/tracers: dont initialize activePrecompiles with random value
This change increases the cache size from 64 to 256 Mb for block bodies.
Benchmarks have shown this to be one bottleneck when trying to achieve
higher download speeds.
The commit also includes a minor optimization for header inserts in package
core: previously, the presence of headers in the database was checked for
every header before writing it. With the change, if one header fails the
presence check, all subsequent headers are also assumed to be missing.
This is an improvement because in practice, the headers are almost always
missing during sync.
Previously, the test waited a second and then failed if geth had not
started. This caused the test to fail intermittently. This change checks
whether the IPC is open 10 times over a 5 second period and then fails
if geth is still not available.
The `README.md` links the Gitter channel for discussions, but the
official docs and even the Gitter channel itself recommend using the
official Discord Server for such discussions.
This PR simply changes the Gitter link and provides Discord invite link.
When deleting in fullNode, and the new child node nn is not nil, there is no need
to check the number of non-nil entries in the node. This is because the fullNode
must've contained at least two children before deletion, so there must be another
child node other than nn.
Co-authored-by: Felix Lange <fjl@twurst.com>
Copy the CliqueConfig instead of reusing the pointer.
This makes DeveloperGenesisBlock thread safe and prevents it from
changing params.AllCliqueProtocolChanges.Clique.Epoch.
* accounts/abi/bind: fix bounded contracts and sim backend for 1559
* accounts/abi/bind, ethclient: don't rely on chain config for gas prices
* all: enable London for all internal tests
* les: get receipt type info in les tests
* les: fix weird test
Co-authored-by: Martin Holst Swende <martin@swende.se>
This function is not used in the code base, so probably safe to do rename, or remove in its entirety, but I'm assuming the logic from the original creator still applies so rename probably better.
* internal/ethapi: add baseFee to RPCMarshalHeader
* internal/ethapi: add FeeCap, Tip and correct GasPrice to EIP-1559 RPCTransaction results
* core,eth,les,internal: add support for tip estimation in gas price oracle
* internal/ethapi,eth/gasprice: don't suggest tip larger than fee cap
* core/types,internal: use correct eip1559 terminology for json marshalling
* eth, internal/ethapi: fix rebase problems
* internal/ethapi: fix rpc name of basefee
* internal/ethapi: address review concerns
* core, eth, internal, les: simplify gasprice oracle (#25)
* core, eth, internal, les: simplify gasprice oracle
* eth/gasprice: fix typo
* internal/ethapi: minor tweak in tx args
* internal/ethapi: calculate basefee for pending block
* internal/ethapi: fix panic
* internal/ethapi, eth/tracers: simplify txargs ToMessage
* internal/ethapi: remove unused param
* core, eth, internal: fix regressions wrt effective gas price in the evm
* eth/gasprice: drop weird debug println
* internal/jsre/deps: hack in 1559 gas conversions into embedded web3
* internal/jsre/deps: hack basFee to decimal conversion
* internal/ethapi: init feecap and tipcap for legacy txs too
* eth, graphql, internal, les: fix gas price suggestion on all combos
* internal/jsre/deps: handle decimal tipcap and feecap
* eth, internal: minor review fixes
* graphql, internal: export max fee cap RPC endpoint
* internal/ethapi: fix crash in transaction_args
* internal/ethapi: minor refactor to make the code safer
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: gary rong <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This commit changes the behavior of BitCurve.Add to be more inline
with btcd. It fixes two different bugs:
1) When adding a point at infinity to another point, the other point
should be returned. While this is undefined behavior, it is better
to be more inline with the go standard library.
Thus (0,0) + (a, b) = (a,b)
2) Adding the same point to itself produced the point at infinity.
This is incorrect, now doubleJacobian is used to correctly calculate it.
Thus (a,b) + (a,b) == 2* (a,b) and not (0,0) anymore.
The change also adds a differential fuzzer for Add, testing it against btcd.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change significantly improves the performance of RLPx message reads
and writes. In the previous implementation, reading and writing of
message frames performed multiple reads and writes on the underlying
network connection, and allocated a new []byte buffer for every read.
In the new implementation, reads and writes re-use buffers, and perform
much fewer system calls on the underlying connection. This doubles the
theoretically achievable throughput on a single connection, as shown by
the benchmark result:
name old speed new speed delta
Throughput-8 70.3MB/s ± 0% 155.4MB/s ± 0% +121.11% (p=0.000 n=9+8)
The change also removes support for the legacy, pre-EIP-8 handshake encoding.
As of May 2021, no actively maintained client sends this format.
Fixes the case (example below) where the value passed
to --ethstats flag would be parsed wrongly because the
node name and/or password value contained the special
characters '@' or ':'
--ethstats "ETC Labs Metrics @meowsbits":mypass@ws://mordor.dash.fault.dev:3000
This PR refactors the eth test suite to make it more readable and
easier to use. Some notable differences:
- A new file helpers.go stores all of the methods used between
both eth66 and eth65 and below tests, as well as methods shared
among many test functions.
- suite.go now contains all of the test functions for both eth65
tests and eth66 tests.
- The utesting.T object doesn't get passed through to other helper methods,
but is instead only used within the scope of the test function,
whereas helper methods return errors, so only the test function
itself can fatal out in the case of an error.
- The full test suite now only takes 13.5 seconds to run.
There are two transaction parameter structures defined in
the codebase, although for different purposes. But most of
the parameters are shared. So it's nice to reduce the code
duplication by merging them together.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This removes the error log message that says
Ethereum peer removal failed ... err="peer not registered"
The error happened because removePeer was called multiple
times: once to disconnect the peer, and another time when the
handler exited. With this change, removePeer now has the sole
purpose of disconnecting the peer. Unregistering happens exactly
once, when the handler exits.
This change grows the static integer buffer in Stream to 32 bytes,
making it possible to decode 256bit integers without allocating a
temporary buffer.
In the recent commit 088da24, Stream struct size decreased from 120
bytes down to 88 bytes. This commit grows the struct to 112 bytes again,
but the size change will not degrade performance because Stream
instances are internally cached in sync.Pool.
name old time/op new time/op delta
DecodeBigInts-8 12.2µs ± 0% 8.6µs ± 4% -29.58% (p=0.000 n=9+10)
name old speed new speed delta
DecodeBigInts-8 230MB/s ± 0% 326MB/s ± 4% +42.04% (p=0.000 n=9+10)
All encoding/decoding operations read the type cache to find the
writer/decoder function responsible for a type. When analyzing CPU
profiles of geth during sync, I found that the use of sync.RWMutex in
cache lookups appears in the profiles. It seems we are running into
CPU cache contention problems when package rlp is heavily used
on all CPU cores during sync.
This change makes it use atomic.Value + a writer lock instead of
sync.RWMutex. In the common case where the typeinfo entry is present in
the cache, we simply fetch the map and lookup the type.
* core/types, miner: create TxWithMinerFee wrapper, add EIP-1559 support to TransactionsByMinerFeeAndNonce
miner: set base fee when creating a new header, handle gas limit, log miner fees
* all: rename to NewTransactionsByPriceAndNonce
* core/types, miner: rename to NewTransactionsByPriceAndNonce + EffectiveTip
miner: activate 1559 for testGenerateBlockAndImport tests
* core,miner: revert naming to TransactionsByPriceAndTime
* core/types/transaction: update effective tip calculation logic
* miner: update aleut to london
* core/types/transaction_test: use correct signer for 1559 txs + add back sender check
* miner/worker: calculate gas target from gas limit
* core, miner: fix block gas limits for 1559
Co-authored-by: Ansgar Dietrichs <adietrichs@gmail.com>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
This change extracts the peer QoS tracking logic from eth/downloader, moving
it into the new package p2p/msgrate. The job of msgrate.Tracker is determining
suitable timeout values and request sizes per peer.
The snap sync scheduler now uses msgrate.Tracker instead of the hard-coded 15s
timeout. This should make the sync work better on network links with high latency.
This commit makes various cleanup changes to rlp.Stream.
* rlp: shrink Stream struct
This removes a lot of unused padding space in Stream by reordering the
fields. The size of Stream changes from 120 bytes to 88 bytes. Stream
instances are internally cached and reused using sync.Pool, so this does
not improve performance.
* rlp: simplify list stack
The list stack kept track of the size of the current list context as
well as the current offset into it. The size had to be stored in the
stack in order to subtract it from the remaining bytes of any enclosing
list in ListEnd. It seems that this can be implemented in a simpler
way: just subtract the size from the enclosing list context in List instead.
This changes the definitions of Ping and Pong, adding an optional field
for the sequence number. This field was previously encoded/decoded using
the "tail" struct tag, but using "optional" is much nicer.
This adds support for a new struct tag "optional". Using this tag, structs used
for RLP encoding/decoding can be extended in a backwards-compatible way,
by adding new fields at the end.
This changes the SimultaneousRequests test to send the requests from the same
connection, as it doesn't really make sense to test whether a node can respond
to two requests with different request IDs from separate connections.
With the update to a newer AppVeyor build image, creating the Windows
installer no longer worked because of a string quoting error in EnvVarUpdate.nsh.
This applies the fix recommended in https://stackoverflow.com/questions/62081765.
This PR cleans up the CI build system and fixes a couple of issues.
- The go tool launcher code has been moved to internal/build. With the new
toolchain functions, the environment of the host Go (i.e. the one that built
ci.go) and the target Go (i.e. the toolchain downloaded by -dlgo) are isolated
more strictly. This is important to make cross compilation and -dlgo work
correctly in more cases.
- The -dlgo option now skips the download and uses the host Go if the running Go
version matches dlgoVersion exactly.
- The 'test' command now supports -dlgo, -cc and -arch. Running unit tests with
foreign GOARCH is occasionally useful. For example, it can be used to run
32-bit tests on Windows. It can also be used to run darwin/amd64 tests on
darwin/arm64 using Rosetta 2.
- The 'aar', 'xcode' and 'xgo' commands now use a slightly different method to
install external tools. They previously used `go get`, but this comes with the
annoying side effect of modifying go.mod. They now use `go install` instead,
which is the recommended way of installing tools without modifying the local
module.
- The old build warning about outdated Go version has been removed because we're
much better at keeping backwards compatibility now.
This removes auto-configuration of the snap.*.ethdisco.net DNS discovery tree.
Since measurements have shown that > 75% of nodes in all.*.ethdisco.net support
snap, we have decided to retire the dedicated index for snap and just use the eth
tree instead.
The dial iterators of eth and snap now use the same DNS tree in the default configuration,
so both iterators should use the same DNS discovery client instance. This ensures that
the record cache and rate limit are shared. Records will not be requested multiple times.
While testing the change, I noticed that duplicate DNS requests do happen even
when the client instance is shared. This is because the two iterators request the tree
root, link tree root, and first levels of the tree in lockstep. To avoid this problem, the
change also adds a singleflight.Group instance in the client. When one iterator
attempts to resolve an entry which is already being resolved, the singleflight object
waits for the existing resolve call to finish and returns the entry to both places.
* params: remove dependency on crypto
Package params should not depend on package crypto because building
crypto requires cgo.
Since build/ci.go needs package params to get the go-ethereum version
number, C code must be compiled in order to run the build tool, which is
annoying for certain cross-compilation setups.
* params: add SectionHead
* eth/protocols/snap: generate storage trie from full dirty snap data
* eth/protocols/snap: get rid of some more dead code
* eth/protocols/snap: less frequent logs, also log during trie generation
* eth/protocols/snap: implement dirty account range stack-hashing
* eth/protocols/snap: don't loop on account trie generation
* eth/protocols/snap: fix account format in trie
* core, eth, ethdb: glue snap packets together, but not chunks
* eth/protocols/snap: print completion log for snap phase
* eth/protocols/snap: extended tests
* eth/protocols/snap: make testcase pass
* eth/protocols/snap: fix account stacktrie commit without defer
* ethdb: fix key counts on reset
* eth/protocols: fix typos
* eth/protocols/snap: make better use of delivered data (#44)
* eth/protocols/snap: make better use of delivered data
* squashme
* eth/protocols/snap: reduce chunking
* squashme
* eth/protocols/snap: reduce chunking further
* eth/protocols/snap: break out hash range calculations
* eth/protocols/snap: use sort.Search instead of looping
* eth/protocols/snap: prevent crash on storage response with no keys
* eth/protocols/snap: nitpicks all around
* eth/protocols/snap: clear heal need on 1-chunk storage completion
* eth/protocols/snap: fix range chunker, add tests
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* trie: fix test API error
* eth/protocols/snap: fix some further liter issues
* eth/protocols/snap: fix accidental batch reuse
Co-authored-by: Martin Holst Swende <martin@swende.se>
Previously, the makeCallVariantGasCallEIP2929 charged the cold account access cost directly, leading to an incorrect gas cost passed to the tracer from the main execution loop.
This change still temporarily charges the cost (to allow for an accurate calculation of the available gas for the call), but then afterwards refunds it and instead returns the correct total gas cost to be then properly charged in the main loop.
The Append / truncate operations were racy. When a datafile reaches 2Gb, a new file is needed. For this operation, we require a writelock, which is not needed in the 99.99% of all cases where the data does fit in the current head-file.
This transition from readlock to writelock was incorrect, and as the readlock was released, a truncate operation could slip in between, and truncate the data. This would have been fine, however, the Append operation continued writing as if no truncation had occurred, e.g writing item 5 where item 0 should reside.
This PR changes the behaviour, so that if when we run into the situation that a new file is needed, it aborts, and retries, this time with a writelock.
The outcome of the situation described above, running on this PR, would instead be that the Append operation exits with a failure.
When receiving PING from an IPv4 address over IPv6, the implementation sent
back a IPv4-in-IPv6 address. This change makes it reflect the IPv4 address.
* core/state/snapshot: reuse memory data instead of hitting disk when generating
* trie: minor nitpicks wrt the resolver optimization
* core/state/snapshot, trie: use key/value store for resolver
* trie: fix linter
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* eth/protocols, prp/tracker: add support for req/rep rtt tracking
* p2p/tracker: sanity cap the number of pending requests
* pap/tracker: linter <3
* p2p/tracker: disable entire tracker if no metrics are enabled
The new error type is returned by client operations contains details of
the response error code and response body.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change improves the efficiency of the nodeIterator seek
operation. Previously, seek essentially ran the iterator forward
until it found the matching node. With this change, it skips
over fullnode children and avoids resolving them from the database.
The new -limit option makes the filter operate on top N nodes by score.
This also adds ENR attribute stats in the nodeset info command.
Node set commands are now documented in README.
This change adds the --catalyst flag, enabling an RPC API for eth2 integration.
In this initial version, catalyst mode also disables all peer-to-peer networking.
Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The stacktrie is a bit un-untuitive, API-wise: since it mutates input values.
Such behaviour is dangerous, and easy to get wrong if the calling code 'forgets' this quirk. The behaviour is fixed by this PR, so that the input values are not modified by the stacktrie.
Note: just as with the Trie, the stacktrie still references the live input objects, so it's still _not_ safe to mutate the values form the callsite.
* log: fix formatting of big.Int
The implementation of formatLogfmtBigInt had two issues: it crashed when
the number was actually large enough to hit the big integer case, and
modified the big.Int while formatting it.
* log: don't call FormatLogfmtInt64 for int16
* log: separate from decimals back, not front
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* all: add thousandths separators for big numbers on log messages
* p2p/sentry: drop accidental file
* common, log: add fast number formatter
* common, eth/protocols/snap: simplifty fancy num types
* log: handle nil big ints
* replaces `an chance` with `a chance`
* replaces `SignHashWithPassphrase` with `SignTextWithPassphrase` as there was no SignHashWithPasspharse function in the file
* core/vm: implement AccessListTracer
* eth: implement debug.createAccessList
* core/vm: fixed nil panics in accessListTracer
* eth: better error messages for createAccessList
* eth: some fixes on CreateAccessList
* eth: allow for provided accesslists
* eth: pass accesslist by value
* eth: remove created acocunt from accesslist
* core/vm: simplify access list tracer
* core/vm: unexport accessListTracer
* eth: return best guess if al iteration times out
* eth: return best guess if al iteration times out
* core: docstring, unexport methods
* eth: typo
* internal/ethapi: move createAccessList to eth package
* internal/ethapi: remove reexec from createAccessList
* internal/ethapi: break if al is equal to last run, not if gas is equal
* internal/web3ext: fixed arguments
* core/types: fixed equality check for accesslist
* core/types: no hardcoded vals
* core, internal: simplify access list generation, make it precise
* core/vm: fix typo
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change adds support for logging JSON records when the --log.json flag is
given. The --debug and --backtrace flags are deprecated and replaced by
--log.debug and --log.backtrace.
While changing this, it was noticed that the --memprofilerate and
--blockprofilerate were ineffective (they were always overridden even if
--pprof.memprofilerate was not set). This is also fixed.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds support for EIP-2718 access list transactions in the GraphQL API.
Co-authored-by: Amit Shah <amitshah0t7@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This removes the duplicated definition of eth_chainID
in package eth and updates the definition in internal/ethapi
to treat chain ID as a bigint.
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a rare issue where the client subscription forwarding loop
would attempt send on the subscription's channel after Unsubscribe has
returned, leading to a panic if the subscription channel was already
closed by the user. Example:
sub, _ := client.Subscribe(..., channel, ...)
sub.Unsubscribe()
close(channel)
The race occurred because Unsubscribe called quitWithServer to tell the
forwarding loop to stop sending on sub.channel, but did not wait for the
loop to actually come down. This is fixed by adding an additional channel
to track the shutdown, on which Unsubscribe now waits.
Fixes#22322
* core/state/snapshot, ethdb: track deletions more accurately
* core/state/snapshot: don't reset the iterator, leveldb's screwy
* ethdb: don't mess with the insert batches for now
This fixes an issue where the ethstats service could crash if geth was
started and then immediately stopped due to an internal error. The
cause of the crash was a nil subscription being returned by the backend,
because the background goroutine creating them was scheduled after
the backend had already shut down.
Moving the creation of subscriptions into the Start method, which runs
synchronously during startup of the node, means the returned subscriptions
can never be 'nil'.
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
The main idea behind it is: the range compaction is very expensive
which can take a few hours to finish. During this long procedure,
a lot of exceptions can occur, e.g.
- Geth is killed manually
- Geth is killed because of machine crash
- etc
In order to minimize the effect of the exceptions, the compaction
is moved out of the pruning. So that even the compaction is not
finished, the pruning is regarded as done.
This upgrades the cloudflare client dependency to v0.14.0. The new
version changes the API because all methods now require a context
parameter. This change also reduces the log level of the 'Skipping...'
message to debug, following a similar change in the AWS deployer.
The PR implements the --miner.notify.full flag that enables full pending block
notifications. When this flag is used, the block notifications sent to mining
endpoints contain the complete block header JSON instead of a work package
array.
Co-authored-by: AlexSSD7 <alexandersadovskyi7@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/devp2p: fix comparison of TXT record value
The AWS API returns quoted DNS strings, so we must encode the new value
before comparing it against the existing record content.
* cmd/devp2p: add test
* cmd/devp2p: fix typo and rename val -> newValue
Fixes the CaptureStart api to include the EVM, thus being able to set the statedb early on. This pr also exposes the struct we used internally in the interpreter to encapsulate the contract, mem, stack, rstack, so we pass it as a single struct to the tracer, and removes the error returns on the capture methods.
In Geth v1.10, we changed the structure of the "les" ENR entry. As a result, the DHT crawler that creates the DNS lists
no longer recognizes the les nodes, which is fixed in this commit.
* cmd/devp2p: skip ENR field tails properly in nodeset filter
* cmd/devp2p: fix tail decoder for snap as well
* les: fix tail decoding in "eth" ENR entry
This PR fixes a regression introduced in #22360, when we updated to the v2 of the AWS sdk, which causes current crawler to just get the same first 100 results over and over, and get stuck in a loop.
This fixes the calculation of the tree branch factor. With the new
formula, we now creat at most 13 children instead of 30, ensuring
the TXT record size will be below 370 bytes.
This updates the DNS deployer to use AWS SDK v2. Migration is relatively
seamless, although there were two locations that required a slightly
different approach to achieve the same results. In particular, waiting for
DNS change propagation is very different with SDK v2.
This change also optimizes DNS updates by publishing all changes before
waiting for propagation.
The wantErr field was disused, and the error returned by HeaderByNumber
was not properly tested.
This simplifies the error checking using errors.Is and asserts that getting
an expected missing header returns ethereum.NotFound. Also adds a nil
check condition for header.Number before using big.Int's Sign method.
This PR fixes an issue with the eth66 test suite where, during a readAndServe when
the test is manually responding to GetBlockHeader requests, it now responds
with a BlockHeaders eth66 packet that includes the inbound request ID.
This PR fixes multiple issues with the UDP connection pre-negotiation feature:
- the enable condition was wrong (it checked the existence of the DiscV5 struct where it wasn't initialized yet, disabling the feature even if discv5 was enabled)
- the server pool queried already connected nodes when the discovery iterators returned them again
- servers responded positively before they were synced and really willing to accept connections
Metrics are also added on the server side that count the positive and negative replies to served connection queries.
This PR fixes multiple issues with the UDP connection pre-negotiation feature:
- the enable condition was wrong (it checked the existence of the DiscV5 struct where it wasn't initialized yet, disabling the feature even if discv5 was enabled)
- the server pool queried already connected nodes when the discovery iterators returned them again
- servers responded positively before they were synced and really willing to accept connections
Metrics are also added on the server side that count the positive and negative replies to served connection queries.
The oss-fuzz fuzzer has been reporting some failing testcases for les. They're all spurious, and cannot reliably be reproduced. However, running them showed that there was a goroutine leak: the tests created a lot of new clients, which started an exec queue that was never torn down.
This PR fixes the goroutine leak, and also a log message which was erroneously formatted.
This updates the consensus tests to commit 31d6630 and
adds support for access list transactions in the test runner.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR implements the first one of the "lespay" UDP queries which
is already useful in itself: the capacity query. The server pool is making
use of this query by doing a cheap UDP query to determine whether it is
worth starting the more expensive TCP connection process.
This replaces the github.com/pborman/uuid dependency with
github.com/google/uuid because the former is only a wrapper for
the latter (since v1.0.0).
Co-authored-by: Felix Lange <fjl@twurst.com>
* les: move serverPool to les/vflux/client
* les: add metrics
* les: moved ValueTracker inside ServerPool
* les: protect against node registration before server pool is started
* les/vflux/client: fixed tests
* les: make peer registration safe
This adds support for EIP-2718 typed transactions as well as EIP-2930
access list transactions (tx type 1). These EIPs are scheduled for the
Berlin fork.
There very few changes to existing APIs in core/types, and several new APIs
to deal with access list transactions. In particular, there are two new
constructor functions for transactions: types.NewTx and types.SignNewTx.
Since the canonical encoding of typed transactions is not RLP-compatible,
Transaction now has new methods for encoding and decoding: MarshalBinary
and UnmarshalBinary.
The existing EIP-155 signer does not support the new transaction types.
All code dealing with transaction signatures should be updated to use the
newer EIP-2930 signer. To make this easier for future updates, we have
added new constructor functions for types.Signer: types.LatestSigner and
types.LatestSignerForChainID.
This change also adds support for the YoloV3 testnet.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
Transaction unindexing will be enabled by default as of 1.10, which causes tx status retrieval will be broken without this PR.
This PR introduces a retry mechanism in TxStatus retrieval.
This PR adds a more CLI flag, so that the les-server can serve light clients even the local node is not synced yet.
This functionality is needed in some testing environments(e.g. hive). After launching the les server, no more blocks will be imported so the node is always marked as "non-synced".
This PR prevents users from submitting transactions without EIP-155 enabled. This behaviour can be overridden by specifying the flag --rpc.allow-unprotected-txs=true.
This PR introduces:
- db.put to put a value into the database
- db.get to read a value from the database
- db.delete to delete a value from the database
- db.stats to check compaction info from the database
- db.compact to trigger a db compaction
It also moves inspectdb to db.inspect.
* les: refactored server handler
* tests/fuzzers/les: add fuzzer for les server handler
* tests, les: update les fuzzer
tests: update les fuzzer
tests/fuzzer/les: release resources
tests/fuzzer/les: pre-initialize all resources
* les: refactored server handler and fuzzer
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
In the random sync algorithm used by the DNS node iterator, we first pick a random
tree and then perform one sync action on that tree. This happens in a loop until any
node is found. If no trees contain any nodes, the iterator will enter a hot loop spinning
at 100% CPU.
The fix is complicated. The iterator now checks if a meaningful sync action can
be performed on any tree. If there is nothing to do, it waits for the next root record
recheck time to arrive and then tries again.
Fixes#22306
io.Reader may return n > 0 and io.EOF at the end of the input stream.
readFull did not handle this correctly, looking only at the error. This fixes
it to check for n == len(buf) as well.
This PR optimizes the broadcast loop. Instead of iterating twice through a given set of transactions to weed out which peers have and which do not have a tx, to send/announce transactions, we do it only once.
There was a dormant error with structured inputs that failed unpacking.
This commit fixes the error by switching casting to the better abi.ConvertType function.
It also adds a test for calling a view function that returns a struct
Prevents a situation where we (not running snap) connects with a peer running snap, and get stalled waiting for snap registration to succeed (which will never happen), which cause a waitgroup wait to halt shutdown
This moves the eth config definition into a separate package, eth/ethconfig.
Packages eth and les can now import this common package instead of
importing eth from les, reducing dependencies.
Co-authored-by: Felix Lange <fjl@twurst.com>
fixes an issue where local transactions that were included in the chain before a SetHead were rejected if resubmitted, since the txpool had not reset the state to the current (older) state.
The PR makes use of the stacktrie, which is is more lenient on resource consumption, than the regular trie, in cases where we only need it for DeriveSha
* remove uneeded convertion type
* remove redundant type in composite literal
* omit explicit type where implicit
* remove unused redundant parenthesis
* remove redundant import alias duktape
This change allows users to set a custom path prefix on which to mount the http-rpc
or ws-rpc handlers via the new flags --http.rpcprefix and --ws.rpcprefix.
Fixes#21826
Co-authored-by: Felix Lange <fjl@twurst.com>
Removes the yolov2 definition, adds yolov3, including EIP-2565. This PR also disables some of the erroneously generated blockchain and statetests, and adds the new genesis hash + alloc for yolov3.
This PR disables the CLI switches for yolo, since it's not complete until we merge support for 2930.
This PR enables running the new discv5 protocol in both LES client
and server mode. In client mode it mixes discv5 and dnsdisc iterators
(if both are enabled) and filters incoming ENRs for "les" tag and fork ID.
The old p2p/discv5 package and all references to it are removed.
Co-authored-by: Felix Lange <fjl@twurst.com>
* accounts/scwallet: use go-ethereum crypto instead of go-ecdh
github.com/wsddn/go-ecdh is a wrapper package for ECDH functionality
with any elliptic curve.
Since 'generic' ECDH is not required in accounts/scwallet (the curve is
always secp256k1), we can just use the standard library functionality
and our own crypto libraries to perform ECDH and save a dependency.
* Update accounts/scwallet/securechannel.go
Co-authored-by: Guillaume Ballet <gballet@gmail.com>
* Use the correct key
Co-authored-by: Guillaume Ballet <gballet@gmail.com>
This PR implements the following modifications
- Don't shortcut check if block is present, thus avoid disk lookup
- Don't check hash ancestry in early-check (it's still done in parallel checker)
- Don't check time.Now for every single header
Charts and background info can be found here: https://github.com/holiman/headerimport/blob/main/README.md
With these changes, writing 1M headers goes down to from 80s to 62s.
This moves the tracing RPC API implementation to package eth/tracers.
By doing so, package eth no longer depends on tracing and the duktape JS engine.
The change also enables tracing using the light client. All tracing methods work with the
light client, but it's a lot slower compared to using a full node.
This PR fixes the receipt status field to be decimal instead of a hex string,
as called for by the spec.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Squashed from the following commits:
core/state: lazily init snapshot storage map
core/state: fix flawed meter on storage reads
core/state: make statedb/stateobjects reuse a hasher
core/blockchain, core/state: implement new trie prefetcher
core: make trie prefetcher deliver tries to statedb
core/state: refactor trie_prefetcher, export storage tries
blockchain: re-enable the next-block-prefetcher
state: remove panics in trie prefetcher
core/state/trie_prefetcher: address some review concerns
sq
This PR introduces a new config field SyncFromCheckpoint for light client.
In some special scenarios, it's required to start synchronization from some
arbitrary checkpoint or even from the scratch. So this PR offers this
flexibility to users so that the synchronization start point can be configured.
There are two relevant configs: SyncFromCheckpoint and Checkpoint.
- If the SyncFromCheckpoint is true, the light client will try to sync from the
specified checkpoint.
- If the Checkpoint is not configured, then the light client will sync from the
scratch(from the latest header if the database is not empty)
Additional notes: these two configs are not visible in the CLI flags but only
accessable in the config file.
Example Usage:
[Eth]
SyncFromCheckpoint = true
[Eth.Checkpoint]
SectionIndex = 100
SectionHead = "0xabc"
CHTRoot = "0xabc"
BloomRoot = "0xabc"
PS. Historical checkpoint can be retrieved from the synced full node or light
client via les_getCheckpoint API.
Adding warnings of free disk space left and graceful shutdown when there is not enough space left.
This also adds a flag datadir.minfreedisk which can be used to set the trigger for low disk space, and setting it to zero disables the check.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR has two changes in the les protocol:
- the auxRoot is not supported. See ethereum/devp2p#171 for more information
- the empty response will be returned in GetHelperTrieProofsMsg request if the merkle
proving is failed. note, for backward compatibility, the empty merkle proof as well as
the request auxiliary data will still be returned in les2/3 protocol no matter the proving
is successful or not. the proving failure can happen e.g. request the proving for a
non-included entry in helper trie (unstable header).
The tests sometimes failed with certain go versions because
the behavior of http.Server.Shutdown changed over time. A bug
that was fixed in Go 1.15 could cause active connections on unrelated
servers to close unexpectedly. This is fixed by avoiding use of the
same port number in all tests.
USB enumeration still occured. Make sure it will only occur if --usb is set.
This also deprecates the 'NoUSB' config file option in favor of a new option 'USB'.
Resolves https://github.com/etclabscore/core-geth/issues/273
jsre.JSRE already handles establishing preload
file paths relative to the 'assets' path (aka docroot),
where it joins the assets dir and the file path if relative,
or uses the file path only if absolute.
The duplication of this logic by MakeConsolePreloads
caused preloaded files to have paths which contained
duplicate references to the assets dir path.
Date: 2020-12-30 08:25:01-06:00
Signed-off-by: meows <b5c6@protonmail.com>
This changes the chainID RPC method to return an error when EIP-155 is not yet
active at the current block height. It used to simply return zero in this case, but
that's confusing.
This commit extends the ethclient test suite and increases code coverage of the ethclient
package from ~15% to >55%. These tests act as early smoke tests to signal issues in the
RPC-interface. E.g. if a functionality like eth_chainId or eth_call breaks, the test
will break.
This changes the JSON encoding of blocks returned by the API
to have decimal instead of hexadecimal numbers. The spec wants
it this way.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR removes a logic in the miner, which was originally intended to help temporary testnets based on ethash from "running off into the future". If the difficulty was low, and a few computers started mining several blocks per second, the ethash rules (which demand 1s delay between blocks) would push the blocktimes further and further away.
The solution was to make the miner sleep while this happened.
Nowadays, this problem is solved instead by PoA chains, and it's recommended to let testnets and devnets be based on clique instead. The existing logic is problematic, since it can cause stalls within the miner making it difficult for remote workers to submit work if the channel is blocked on a sleep.
Credits to Saar Tochner for reporting this via the bug bounty
The legacy dot-org URL was displaying a message about the repository
having migrated to the dot-com service, which now covers open-source
projects as well.
During the snap and eth refactor, the net_version rpc call was falsely deprecated.
This restores the net_version RPC handler as most eth2 nodes and other software
depend on it.
* les: allow tx unindexing in les/4 light server mode
* les: minor fixes
* les: more small fixes
* les: add meaningful constants for recentTxIndex handshake field
This PR implements unclean shutdown marker. Every time geth boots, it adds a timestamp to a list of timestamps in the database. This list is capped at 10. At a clean shutdown, the timestamp is removed again.
Thus, when geth exits unclean, the marker remains, and at boot up we show the most recent unclean shutdowns to the user, which makes it easier to diagnose root-causes to certain problems.
Co-authored-by: Nagy Salem <me@muhnagy.com>
This PR adds re-written difficulty calculators, which are based on uint256. It also adds a fuzzer + oss-fuzz integration for the new fuzzer. It does differential fuzzing between the new and old calculators.
Note: this PR does not actually enable the new calculators.
This PR adds support for using Twitter API to query the tweet and author details. There are two reasons behind this change:
- Twitter will be deprecating the legacy website on 15th December. The current method is expected to stop working then.
- More importantly, the current system uses Twitter handle for spam protection but the Twitter handle can be changed via automated calls. This allows bots to use the same tweet to withdraw funds infinite times as long as they keep changing their handle between every request. The Rinkeby as well as the Goerli faucet are being actively drained via this method. This PR changes the spam protection to be based on Twitter IDs instead of usernames. A user can not change their Twitter ID.
In miner/worker.go, there are two goroutine using channel w.newWorkCh: newWorkerLoop() sends to this channel, and mainLoop() receives from this channel. Only the receive operation is in a select.
However, w.exitCh may be closed by another goroutine. This is fine for the receive since receive is in select, but if the send operation is blocking, then it will block forever. This commit puts the send in a select, so it won't block even if w.exitCh is closed.
Similarly, there are two goroutines using channel errc: the parent that runs the test receives from it, and the child created at line 573 sends to it. If the parent goroutine exits too early by calling t.Fatalf() at line 614, then the child goroutine will be blocked at line 574 forever. This commit adds 1 buffer to errc. Now send will not block, and receive is not influenced because receive still needs to wait for the send.
The previous fix#21960 converted the float to an intermediate signed int, before attempting the uint conversion. Although this works, this doesn't guarantee that other architectures will work the same.
This commit fixes a flaw in two testcases, and brings down the exec-time from ~40s to ~8s for trie/TestIncompleteSync.
The checkConsistency was performed over and over again on the complete set of nodes, not just the recently added, turning it into a quadratic runtime.
The database panicked for invalid IPs. This is usually no problem
because all code paths leading to node DB access verify the IP, but it's
dangerous because improper validation can turn this panic into a DoS
vulnerability. The quick fix here is to just turn database accesses
using invalid IP into a noop. This isn't great, but I'm planning to
remove the node DB for discv5 long-term, so it should be fine to have
this quick fix for half a year.
Fixes#21849
This fixes some issues in crypto/signify and makes release signing work.
The archive signing step in ci.go used getenvBase64, which decodes the key data.
This is incorrect here because crypto/signify already base64-decodes the key.
* core: add test for headerchain inserts
* core, light: write headerchains in batches
* core: change to one callback per batch of inserted headers + review concerns
* core: error-check on batch write
* core: unexport writeHeaders
* core: remove callback parameter in InsertHeaderChain
The semantics of InsertHeaderChain are now much simpler: it is now an
all-or-nothing operation. The new WriteStatus return value allows
callers to check for the canonicality of the insertion. This change
simplifies use of HeaderChain in package les, where the callback was
previously used to post chain events.
* core: skip some hashing when writing headers
* core: less hashing in header validation
* core: fix headerchain flaw regarding blacklisted hashes
Co-authored-by: Felix Lange <fjl@twurst.com>
Both Hash and Address have a String method, which returns the value as
hex with 0x prefix. They also had a Format method which tried to print
the value using printf of []byte. The way Format worked was at odds with
String though, leading to a situation where fmt.Sprintf("%v", hash)
returned the decimal notation and hash.String() returned a hex string.
This commit makes it consistent again. Both types now support the %v,
%s, %q format verbs for 0x-prefixed hex output. %x, %X creates
unprefixed hex output. %d is also supported and returns the decimal
notation "[1 2 3...]".
For Address, the case of hex characters in %v, %s, %q output is
determined using the EIP-55 checksum. Using %x, %X with Address
disables checksumming.
Co-authored-by: Felix Lange <fjl@twurst.com>
This commit enables users to specify which signer they want to use while creating their transactOpts.
Previously all contract interactions used the homestead signer. Now a user can specify whether they
want to sign with homestead or EIP155 and specify the chainID which adds another layer of security.
Closes#16484
* cmd/geth: implement vulnerability check
* cmd/geth: use minisign to verify vulnerability feed
* cmd/geth: add the test too
* cmd/geth: more minisig/signify testing
* cmd/geth: support multiple pubfiles for signing
* cmd/geth: add @holiman minisig pubkey
* cmd/geth: polishes on vulnerability check
* cmd/geth: fix ineffassign linter nit
* cmd/geth: add CVE to version check struct
* cmd/geth/testdata: add missing testfile
* cmd/geth: add more keys to versionchecker
* cmd/geth: support file:// URLs in version check
* cmd/geth: improve key ID printing when signature check fails
Co-authored-by: Felix Lange <fjl@twurst.com>
A lot of times when we hit 'core' errors, example: invalid tx, the information provided is
insufficient. We miss several pieces of information: what account has nonce too high,
and what transaction in that block was offending?
This PR adds that information, using the new type of wrapped errors.
It also adds a testcase which (partly) verifies the output from the errors.
The first commit changes all usage of direct equality-checks on core errors, into
using errors.Is. The second commit adds contextual information. This wraps most
of the core errors with more information, and also wraps it one more time in
stateprocessor, to further provide tx index and tx hash, if such a tx is encoutered in
a block. The third commit uses the chainmaker to try to generate chains with such
errors in them, thus triggering the errors and checking that the generated string meets
expectations.
This PR fixes a deadlock reported here: #21925
The cause is that many operations may be pending, but if the close happens, only one of them gets awoken and exits, the others remain waiting for a signal that never comes.
* signer/core/api: fix derivation of ledger live accounts
For ledger hardware wallets, change account iteration as follows:
- ledger legacy: m/44'/60'/0'/X; for 0<=X<5
- ledger live: m/44'/60'/0'/0/X; for 0<=X<5
- ledger legacy: m/44'/60'/0'/X; for 0<=X<10
- ledger live: m/44'/60'/X'/0/0; for 0<=X<10
Non-ledger derivation is unchanged and remains as:
- non-ledger: m/44'/60'/0'/0/X; for 0<=X<10
* signer/core/api: derive ten default paths for all hardware wallets, plus ten legacy and ten live paths for ledger wallets
* signer/core/api: as .../0'/0/0 already included by default paths, do not include it again with ledger live paths
* accounts, signer: implement path iterators for hd wallets
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/build: implement signify's signing func
* Add signify to the ci utility
* fix output file format
* Add unit test for signify
* holiman's + travis' feedback
* internal/build: verify signify's output
* crypto: move signify to common dir
* use go-minisign to verify binaries
* more holiman feedback
* crypto, ci: support minisign output
* only accept one-line trusted comments
* configurable untrusted comments
* code cleanup in tests
* revert to use ed25519 from the stdlib
* bug: fix for empty untrusted comments
* write timestamp as comment if trusted comment isn't present
* rename line checker to commentHasManyLines
* crypto: added signify fuzzer (#6)
* crypto: added signify fuzzer
* stuff
* crypto: updated signify fuzzer to fuzz comments
* crypto: repro signify crashes
* rebased fuzzer on build-signify branch
* hide fuzzer behind gofuzz build flag
* extract key data inside a single function
* don't treat \r as a newline
* travis: fix signing command line
* do not use an external binary in tests
* crypto: move signify to crypto/signify
* travis: fix formatting issue
* ci: fix linter build after package move
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This fixes a deadlock that could occur when a response packet arrived
after a call had already received enough responses and was about to
signal completion to the dispatch loop.
Co-authored-by: Felix Lange <fjl@twurst.com>
* Make sure to return 400 when errors are present in the response
* graphql: use less memory in chainconfig for tests
Co-authored-by: Martin Holst Swende <martin@swende.se>
* added bls fuzzer
* crypto/bls12381: revert bls-changes, fixup fuzzer tests
* fuzzers: split bls fuzzing into 8 different units
* fuzzers/bls: remove (now stale) corpus
* crypto/bls12381: added blsfuzz corpus
* fuzzers/bls12381: fix the bls corpus
* fuzzers: fix oss-fuzz script
* tests/fuzzers: fixups on bls corpus
* test/fuzzers: remove leftover corpus
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* crypto/bn256: fix bn256Mul fuzzer to not hang on large input
* Update crypto/bn256/bn256_fuzz.go
Co-authored-by: ligi <ligi@ligi.de>
Co-authored-by: ligi <ligi@ligi.de>
* trie: fix tests to work on 32-bit systems
* les: make test work on 32-bit platform
* cmd/geth: fix windows-issues on tests
* trie: improve balance
* cmd/geth: make account tests less verbose + less mem intense
* rpc: make debug-level log output less verbose
* cmd/geth: lint
* Only compare hostnames in ws.origins
Also using a helper function for ToLower consolidates all preparation steps in one function for more maintainable consistency.
Spaces => tabs
Remove a semicolon
Add space at start of comment
Remove parens around conditional
Handle case wehre parsed hostname is empty
When passing a single word like "localhost" the parsed hostname is an empty string. Handle this and the error-parsing case together as default, and the nonempty hostname case in the conditional.
Refactor with new originIsAllowed functions
Adds originIsAllowed() & ruleAllowsOrigin(); removes prepOriginForComparison
Remove blank line
Added tests for simple allowed-orign rule
which does not specify a protocol or port, just a hostname
Fix copy-paste: `:=` => `=`
Remove parens around conditional
Remove autoadded whitespace on blank lines
Compare scheme, hostname, and port with rule
if the rule specifies those portions.
Remove one autoadded trailing whitespace
Better handle case where only origin host is given
e.g. "localhost"
Remove parens around conditional
Refactor: attemptWebsocketConnectionFromOrigin DRY
Include return type on helper function
Provide srv obj in helper fn
Provide srv to helper fn
Remove stray underscore
Remove blank line
parent 93e666b4c1e7e49b8406dc83ed93f4a02ea49ac1
author wbt <wbt@users.noreply.github.com> 1598559718 -0400
committer Martin Holst Swende <martin@swende.se> 1605602257 +0100
gpgsig -----BEGIN PGP SIGNATURE-----
iQFFBAABCAAvFiEEypmrtbNuJK1doP1AaDtDjAWl3fAFAl+zi9ARHG1hcnRpbkBz
d2VuZGUuc2UACgkQaDtDjAWl3fDRiwgAoMtzU8dwRV7Q9xkCwWEx9Wz2f3n6jUr2
VWBycDKGKwRkPPOER3oc9kzjGU/P1tFlK07PjfnAKZ9KWzxpDcJZwYM3xCBurG7A
16y4YsQnzgPNONv3xIkdi3RZtDBIiPFFEmdZFFvZ/jKexfI6JIYPngCAoqdTIFb9
On/aPvvVWQn1ExfmarsvvJ7kUDUG77tZipuacEH5FfFsfelBWOEYPe+I9ToUHskv
+qO6rOkV1Ojk8eBc6o0R1PnApwCAlEhJs7aM/SEOg4B4ZJJneiFuEXBIG9+0yS2I
NOicuDPLGucOB5nBsfIKI3USPeE+3jxdT8go2lN5Nrhm6MimoILDsQ==
=sgUp
-----END PGP SIGNATURE-----
Refactor: drop err var for more concise test lines
Add several tests for new WebSocket origin checks
Remove autoadded whitespace on blank lines
Restore TestWebsocketOrigins originally-named test
and rename the others to be helpers rather than full tests
Remove autoadded whitespace on blank line
Temporarily comment out new test sets
Uncomment test around origin rule with scheme
Remove tests without scheme on browser origin
per https://github.com/ethereum/go-ethereum/pull/21481/files#r479371498
Uncomment tests with port; remove some blank lines
Handle when browser does not specify scheme/port
Uncomment test for including scheme & port in rule
Add IP tests
* node: more tests + table-driven, ws origin changes
Co-authored-by: Martin Holst Swende <martin@swende.se>
The z == 0 check is hit whenever we Add two points with the same x1/x2
coordinate. crypto/elliptic uses the same check in their affineFromJacobian
function. This change does not affect block processing or tx signature verification
in any way, because it does not use the Add or Double methods.
* consensus/ethash: only use *reflect.SliceHeader, not reflect.SliceHeader. See comment here: https://github.com/golang/go/issues/40397\#issuecomment-663748689
* consensus/ethash: pr feedback from @mdempsky, makes a copy of dest such that is not mutated
* consensus/ethash: remove noop assign
* consensus/ethash: apply same fix to another location
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* all: core: split vm.Config into BlockConfig and TxConfig
* core: core/vm: reset EVM between tx in block instead of creating new
* core/vm: added docs
This fixes cross-build and mobile framework failures.
It also disables the mac test builder because it was failing
all the time in hard to understand ways and we can't afford
it anymore under Travis CI's new pricing.
This new flag downloads a known version of Go and builds with it. This
is meant for environments where we can't easily upgrade the installed Go
version.
* .travis.yml: remove install step for PR test builders
We added this step originally to avoid re-building everything
for every test. go test has become much smarter in recent go
releases, so we no longer need to install anything here.
- Remove the ws:// prefix from the status endpoint since
the ws:// is already included in the stack.WSEndpoint().
- Don't register the services again in the node start.
Registration is already done in the initialization stage.
- Expose admin namespace via websocket.
This namespace is necessary for connecting the peers via websocket.
- Offer logging relevant options for exec adapter.
It's really painful to mix all log output in the single console. So
this PR offers two additional options for exec adapter in this case
testers can config the log output(e.g. file output) and log level
for each p2p node.
* trie: fix error in stacktrie not committing small roots
* fuzzers: make trie-fuzzer use correct returnvalues
* trie: improved tests
* tests/fuzzers: fuzzer for stacktrie vs regular trie
* test/fuzzers: make stacktrie fuzzer use 32-byte keys
* trie: fix error in stacktrie with small nodes
* trie: add (skipped) testcase for stacktrie
* tests/fuzzers: address review comments for stacktrie fuzzer
* trie: fix docs in stacktrie
This PR contains a minor optimization in derivesha, by exposing the RLP
int-encoding and making use of it to write integers directly to a
buffer (an RLP integer is known to never require more than 9 bytes
total). rlp.AppendUint64 might be useful in other places too.
The code assumes, just as before, that the hasher (a trie) will copy the
key internally, which it does when doing keybytesToHex(key).
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR replaces the old test genesis.json and chain.rlp files in the testdata
directory for the eth protocol test suite, and also adds documentation for
running the eth test suite locally.
It also improves the test output text and adds more timeouts.
Co-authored-by: Felix Lange <fjl@twurst.com>
TAP is a text format for test results. Parsers for it are available in many languages,
making it easy to consume. I want TAP output from our protocol tests because the
Hive wrapper around them needs to know about the test names and their individual
results and logs. It would also be possible to just write this info as JSON, but I don't
want to invent a new format.
This also improves the normal console output for tests (when running without --tap).
It now prints -- RUN lines before any output from the test, and indents the log output
by one space.
* core/state/snapshot: print warning if failed to resolve journal
* core/state/snapshot: fix snapshot recovery
When we meet the snapshot journal consisted with:
- disk layer generator with new-format
- diff layer journal with old-format
The base layer should be returned without error.
The broken diff layer can be reconstructed later
but we definitely don't want to reconstruct the
huge diff layer.
* core: add tests
ToHex was deprecated a couple years ago. The last remaining use
was in ToHexArray, which itself only had a single call site.
This just moves ToHexArray near its only remaining call site and
implements it using hexutil.Encode. This changes the default behaviour
of ToHexArray and with it the behaviour of eth_getProof. Previously we
encoded an empty slice as 0, now the empty slice is encoded as 0x.
This adds a few tiny fixes for les and the p2p simulation framework:
LES Parts
- Keep the LES-SERVER connection even it's non-synced
We had this idea to reject the connections in LES protocol if the les-server itself is
not synced. However, in LES protocol we will also receive the connection from another
les-server. In this case even the local node is not synced yet, we should keep the tcp
connection for other protocols(e.g. eth protocol).
- Don't count "invalid message" for non-existing GetBlockHeadersMsg request
In the eth syncing mechanism (full sync, fast sync, light sync), it will try to fetch
some non-existent blocks or headers(to ensure we indeed download all the missing chain).
In this case, it's possible that the les-server will receive the request for
non-existent headers. So don't count it as the "invalid message" for scheduling
dropping.
- Copy the announce object in the closure
Before the les-server pushes the latest headers to all connected clients, it will create
a closure and queue it in the underlying request scheduler. In some scenarios it's
problematic. E.g, in private networks, the block can be mined very fast. So before the
first closure is executed, we may already update the latest_announce object. So actually
the "announce" object we want to send is replaced.
The downsize is the client will receive two announces with the same td and then drop the
server.
P2P Simulation Framework
- Don't double register the protocol services in p2p-simulation "Start".
The protocols upon the devp2p are registered in the "New node stage". So don't reigster
them again when starting a node in the p2p simulation framework
- Add one more new config field "ExternalSigner", in order to use clef service in the
framework.
* core/state/snapshot: introduce snapshot journal version
* core: update the disk layer in an atomic way
* core: persist the disk layer generator periodically
* core/state/snapshot: improve logging
* core/state/snapshot: forcibly ensure the legacy snapshot is matched
* core/state/snapshot: add debug logs
* core, tests: fix tests and special recovery case
* core: polish
* core: add more blockchain tests for snapshot recovery
* core/state: fix comment
* core: add recovery flag for snapshot
* core: add restart after start-after-crash tests
* core/rawdb: fix imports
* core: fix tests
* core: remove log
* core/state/snapshot: fix snapshot
* core: avoid callbacks in SetHead
* core: fix setHead cornercase where the threshold root has state
* core: small docs for the test cases
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core/state/snapshot: add diskRoot function
* core/state/snapshot: disable iteration if the snapshot is generating
* core/state/snapshot: simplify the function
* core/state: panic for undefined layer
The old one was wrong in two ways: the first block in chain.rlp was the
genesis block, and the genesis difficulty was below minimum difficulty.
This also contains some other fixes to the test.
* accounts/keystore: add timeout to test to prevent failure on travis
The TestWalletNotifications test sporadically fails on travis.
This is because we shutdown the event collection before all events are received.
Adding a small timeout (10 milliseconds) allows the collector to be scheduled
and to consume all pending events before we shut it down.
* accounts/keystore: added newlines back in
* accounts/keystore: properly fix the walletNotifications test
* miner: exit loop when downloader Done or Failed
Following the logic of the comment at the method,
this fixes a regression introduced at 7cf56d6f06
, which would allow external parties to DoS with
blocks, preventing mining progress.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: remove ineff assign (lint)
Signed-off-by: meows <b5c6@protonmail.com>
* miner: update test re downloader events
Signed-off-by: meows <b5c6@protonmail.com>
* Revert "miner: remove ineff assign (lint)"
This reverts commit eaefcd34ab4862ebc936fb8a07578aa2744bc058.
* Revert "miner: exit loop when downloader Done or Failed"
This reverts commit 23abd34265aa246c38fc390bb72572ad6ae9fe3b.
* miner: add test showing imprecise TestMiner
Signed-off-by: meows <b5c6@protonmail.com>
* miner: fix waitForMiningState precision
This helper function would return an affirmation
on the first positive match on a desired bool.
This was imprecise; it return false positives
by not waiting initially for an 'updated' value.
This fix causes TestMiner_2 to fail, which is
expected.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: remove TestMiner_2 demonstrating broken test
This test demonstrated the imprecision of the test
helper function waitForMiningState. This function
has been fixed with 6d365c2851, and this test test
may now be removed.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: fix test regarding downloader event/mining expectations
See comment for logic.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: add test describing expectations for downloader/mining events
We expect that once the downloader emits a DoneEvent,
signaling a successful sync, that subsequent StartEvents
are not longer permitted to stop the miner.
This prevents a security vulnerability where forced syncs via
fake high blocks would stall mining operation.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: use 'canStop' state to fix downloader event handling
- Break downloader event handling into event
separating Done and Failed events. We need to
treat these cases differently since a DoneEvent
should prevent the miner from being stopped on
subsequent downloader Start events.
- Use canStop state to handle the one-off
case when a downloader first succeeds.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: improve comment wording
Signed-off-by: meows <b5c6@protonmail.com>
* miner: start mining on downloader events iff not already mining
Signed-off-by: meows <b5c6@protonmail.com>
* miner: refactor miner update logic w/r/t downloader events
This makes mining pause/start logic regarding downloader
events more explicit. Instead of eternally handling downloader
events after the first done event, the subscription is closed
when downloader events are no longer actionable.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: fix handling downloader events on subcription closed
Signed-off-by: meows <b5c6@protonmail.com>
* miner: (lint:gosimple) use range over chan instead of for/select
Signed-off-by: meows <b5c6@protonmail.com>
* miner: refactor update loop to remove race condition
The go routine handling the downloader events handling
vars in parallel with the parent routine, causing a
race condition.
This change, though ugly, remove the condition while
still allowing the downloader event subscription to be
closed when the miner has no further use for it (ie DoneEvent).
* miner: alternate fix for miner-flaw
Co-authored-by: meows <b5c6@protonmail.com>
* peer: return localAddr instead of name to prevent spam
We currently use the name (which can be freely set by the peer) in several log messages.
This enables malicious actors to write spam into your geth log.
This commit returns the localAddr instead of the freely settable name.
* p2p: reduce usage of peer.Name in warn messages
* eth, p2p: use truncated names
* Update peer.go
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
For some reason, using the shared hash causes a cryptographic incompatibility
when using Go 1.15. I noticed this during the development of Discovery v5.1
when I added test vector verification.
The go library commit that broke this is golang/go@97240d5, but the
way we used HKDF is slightly dodgy anyway and it's not a regression.
* core/types: tests for bloom
* core/types: refactored bloom filter for receipts, added tests
core/types: replaced old bloom implementation
core/types: change interface of bloom add+test
* core/types: refactor bloom
* core/types: minor tweak on LogsBloom
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core/state/snapshot: exit Geth if generator hits missing trie nodes
* core/state/snapshot: error instead of hard die on generator fault
* core/state/snapshot: don't enable logging on the tests
* trie: update tests to check commit integrity
* trie: polish committer
* trie: fix typo
* trie: remove hasvalue notion
According to the benchmarks, type assertion between the pointer and
interface is extremely fast.
BenchmarkIntmethod-12 1000000000 1.91 ns/op
BenchmarkInterface-12 1000000000 2.13 ns/op
BenchmarkTypeSwitch-12 1000000000 1.81 ns/op
BenchmarkTypeAssertion-12 2000000000 1.78 ns/op
So the overhead for asserting whether the shortnode has "valuenode"
child is super tiny. No necessary to have another field.
* trie: linter nitpicks
Co-authored-by: Martin Holst Swende <martin@swende.se>
* accounts, signer: implement gnosis safe support
* common/math: add type for marshalling big to dec
* accounts, signer: properly sign gnosis requests
* signer, clef: implement account_signGnosisTx
* signer: fix auditlog print, change rpc-name (signGnosisTx to signGnosisSafeTx)
* signer: pass validation-messages/warnings to the UI for gnonsis-safe txs
* signer/core: minor change to validationmessages of typed data
core/types: use stacktrie for derivesha
trie: add stacktrie file
trie: fix linter
core/types: use stacktrie for derivesha
rebased: adapt stacktrie to the newer version of DeriveSha
Co-authored-by: Martin Holst Swende <martin@swende.se>
More linter fixes
review feedback: no key offset for nodes converted to hashes
trie: use EncodeRLP for full nodes
core/types: insert txs in order in derivesha
trie: tests for derivesha with stacktrie
trie: make stacktrie use pooled hashers
trie: make stacktrie reuse tmp slice space
trie: minor polishes on stacktrie
trie/stacktrie: less rlp dancing
core/types: explain the contorsions in DeriveSha
ci: fix goimport errors
trie: clear mem on subtrie hashing
squashme: linter fix
stracktrie: use pooling, less allocs (#3)
trie: in-place hex prefix, reduce allocs and add rawNode.EncodeRLP
Reintroduce the `[]node` method, add the missing `EncodeRLP` implementation for `rawNode` and calculate the hex prefix in place.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This allows users to estimate gas on top of arbitrary blocks as well as pending and latest.
Tracing on pending is useful for most users as it takes into account the current txpool while
tracing on latest might be useful for users that have little to know knowledge of the current
transactions in the network.
If blockNrOrHash is not specified, estimateGas defaults to pending
This change moves the RLPx protocol implementation into a separate package,
p2p/rlpx. The new package can be used to establish RLPx connections for
protocol testing purposes.
Co-authored-by: Felix Lange <fjl@twurst.com>
* database: added counters
* Improved stats for ancient db
* Small improvement
* Better message and added percentage while counting receipts
* Fast counting for receipts
* added info message
* Show both receips itemscount from ancient db and counted receipts
* Fixed default case
* Removed counter for receipts in ancient store
* Removed counting of receipts present in leveldb
When the license was added to the repository, its text was changed (some
sections at the end removed) and, worse, the authors of go-ethereum
tried to claim copyright on the license text.
The correct way to apply GPL to a project is to copy it verbatim.
This change reverts the text of the GPL to the original.
This PR adds an extra guarantee to NodeStateMachine: it ensures that all
immediate effects of a certain change are processed before any subsequent
effects of any of the immediate effects on the same node. In the original
version, if a cascaded change caused a subscription callback to be called
multiple times for the same node then these calls might have happened in a
wrong chronological order.
For example:
- a subscription to flag0 changes flag1 and flag2
- a subscription to flag1 changes flag3
- a subscription to flag1, flag2 and flag3 was called in the following order:
[flag1] -> [flag1, flag3]
[] -> [flag1]
[flag1, flag3] -> [flag1, flag2, flag3]
This happened because the tree of changes was traversed in a "depth-first
order". Now it is traversed in a "breadth-first order"; each node has a
FIFO queue for pending callbacks and each triggered subscription callback
is added to the end of the list. The already existing guarantees are
retained; no SetState or SetField returns until the callback queue of the
node is empty again. Just like before, it is the responsibility of the
state machine design to ensure that infinite state loops are not possible.
Multiple changes affecting the same node can still happen simultaneously;
in this case the changes can be interleaved in the FIFO of the node but the
correct order is still guaranteed.
A new unit test is also added to verify callback order in the above scenario.
This PR changes several different things:
- Adds test cases for the miner loop
- Stops the worker if it wasn't already stopped in worker.Close()
- Uses channels instead of atomics in the miner.update() loop
Co-authored-by: Felix Lange <fjl@twurst.com>
* core/vm/testdata: add gascost expectations to testcases
* core/vm: verify expected gas in tests for precompiles
* core/vm: fix overflow flaw in gas/s calculation
* whisper: remove whisper
* Update cmd/geth/config.go
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* cmd/geth: warn on enabling whisper + remove more whisper deps
* mobile: remove all whisper references
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This changes how the downloader works, a little bit. Previously, when block sync started,
we immediately started filling up to 8192 blocks. Usually this is fine, blocks are small
in the early numbers. The threshold then is lowered as we measure the size of the blocks
that are filled.
However, if the node is shut down and restarts syncing while we're in a heavy segment,
that might be bad. This PR introduces a more conservative initial threshold of 2K blocks
instead.
* "Downloader queue stats" is now a DEBUG information
I think this info is more a DEBUG related information then an INFO. If it must remains an INFO, maybe it can be slow down to one time every 5 minutes or so.
* Update queue.go
"Downloader queue stats" information is now provided once every minute instead of once every 10 seconds.
* accounts/abi: fix a bug in getTypeSize method
e.g. for "Tuple[2]" type, the element of the array is a tuple type and the size of the tuple may not be 32.
* accounts/abi: add unit test of getTypeSize method
This change improves discovery behavior in small networks. Very small
networks would often fail to bootstrap because all member nodes were
dropping table content due to findnode failure. The check is now changed
to avoid dropping nodes on findnode failure when their bucket is almost
empty. It also relaxes the liveness check requirement for FINDNODE/v4
response nodes, returning unverified nodes as results when there aren't
any verified nodes yet.
The "findnode failed" log now reports whether the node was dropped
instead of the number of results. The value of the "results" was
always zero by definition.
Co-authored-by: Felix Lange <fjl@twurst.com>
* metrics: zero temp variable in updateMeter
Previously the temp variable was not updated properly after summing it to count.
This meant we had astronomically high metrics, now we zero out the temp whenever we
sum it onto the snapshot count
* metrics: move temp variable to be aligned, unit tests
Moves the temp variable in MeterSnapshot to be 64-bit aligned because of the atomic bug.
Adds a unit test, that catches the previous bug.
* chaincmd should make config nodes instead of full nodes
* add documentation for using makeConfigNode instead of makeFullNode;
* add documentation to functions
* code style
This adds a lock around requests because some routers can't handle
concurrent requests. Requests are also rate-limited.
The Map function request a new mapping exactly when the map timeout
occurs instead of 5 minutes earlier. This should prevent duplicate mappings.
* core: avoid modification of accountSet cache in tx_pool
when runReorg, we may copy the dirtyAccounts' accountSet cache to promoteAddrs
in which accounts will be promoted, however, if we have reset request at the
same time, we may reuse promoteAddrs and modify the cache content which is
against the original intention of accountSet cache. So, we need to make a new
slice here to avoid modify accountSet cache.
* core: fix flatten condition + comment
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR significantly changes the APIs for instantiating Ethereum nodes in
a Go program. The new APIs are not backwards-compatible, but we feel that
this is made up for by the much simpler way of registering services on
node.Node. You can find more information and rationale in the design
document: https://gist.github.com/renaynay/5bec2de19fde66f4d04c535fd24f0775.
There is also a new feature in Node's Go API: it is now possible to
register arbitrary handlers on the user-facing HTTP server. In geth, this
facility is used to enable GraphQL.
There is a single minor change relevant for geth users in this PR: The
GraphQL API is no longer available separately from the JSON-RPC HTTP
server. If you want GraphQL, you need to enable it using the
./geth --http --graphql flag combination.
The --graphql.port and --graphql.addr flags are no longer available.
This replaces the two-stage shutdown scheme with the one we
use almost everywhere else: a single quit channel signalling
termination.
Co-authored-by: Felix Lange <fjl@twurst.com>
* Disable symbol table and DWARF generation by default.
Trimpath if compiling with Go >= 1.13
* Set Go to minimum version 1.13. Revert debug symbol changes.
* init
notes
removed some mentions of eth62, bumped protocol err too old to >=63
* remove sanity checks and bump supported protocol version up to 63
* remove 62 tests, still need to add 65
* remove 65 tests
Solves issue#20582. Non-executable transactions should not be evicted on each tick if there are no promote transactions or if a pending/reset empties the pending list. Tests and logging expanded to handle these cases in the future.
core/tx_pool: use a ts for each tx in the queue, but only update the heartbeat on promotion or pending replaced
queuedTs proper naming
* eth/downloader: refactor downloader + queue
downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache
downloader: more accurate deliverytime calculation, less mem overhead in state requests
downloader/queue: increase underlying buffer of results, new throttle mechanism
eth/downloader: updates to tests
eth/downloader: fix up some review concerns
eth/downloader/queue: minor fixes
eth/downloader: minor fixes after review call
eth/downloader: testcases for queue.go
eth/downloader: minor change, don't set progress unless progress...
eth/downloader: fix flaw which prevented useless peers from being dropped
eth/downloader: try to fix tests
eth/downloader: verify non-deliveries against advertised remote head
eth/downloader: fix flaw with checking closed-status causing hang
eth/downloader: hashing avoidance
eth/downloader: review concerns + simplify resultcache and queue
eth/downloader: add back some locks, address review concerns
downloader/queue: fix remaining lock flaw
* eth/downloader: nitpick fixes
* eth/downloader: remove the *2*3/4 throttling threshold dance
* eth/downloader: print correct throttle threshold in stats
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This fixes the iOS framework build by naming the second parameter of the
Signer interface method. The name is important because it becomes part
of the objc method signature.
Fixes#21340
* geth,utils: implement configurable developer account options
Prior to this change --dev (developer) mode
generated one account with an empty password,
irrespective of existing --password and --miner.etherbase
options.
This change makes --dev mode compatible with these
existing flags.
--dev mode may now be used in conjunction with
--password and --miner.etherbase flags to configure
the developer faucet using an existing keystore or
in creating a new account.
Signed-off-by: meows <b5c6@protonmail.com>
* main: remove key/pass flags from usage developer section
These flags are included already in other sections,
and it is not desired to duplicate them.
They were originally included in this section
along with added support for these flags in the
developer mode.
Signed-off-by: meows <b5c6@protonmail.com>
* core: added local tx pool test case
* core, crypto: various allocation savings regarding tx handling
* core/txlist, txpool: save a reheap operation, avoid some bigint allocs
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
This fixes two issues with state sync restarts:
When sync restarts with a new root, some peers can have in-flight requests.
Since all peers with active requests were marked idle when exiting sync,
the new sync would schedule more requests for those peers. When the
response for the earlier request arrived, the new sync would reject it and
mark the peer idle again, rendering the peer useless until it disconnected.
The other issue was that peers would not be marked idle when they had
delivered a response, but the response hadn't been processed before
restarting the state sync. This also made the peer useless because it
would be permanently marked busy.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a test suite for discovery v4. The test suite is a port of the Hive suite for
discovery, and will replace the current suite on Hive soon-ish. The tests can be
run locally with this command:
devp2p discv4 test -remote enode//...
Co-authored-by: Felix Lange <fjl@twurst.com>
This change further improves the performance of RLP encoding by removing
allocations for big.Int and [...]byte types. I have added a new benchmark
that measures RLP encoding of types.Block to verify that performance is
improved.
Exposing /debug/metrics and /debug/metrics/prometheus was dependent
on --pprof, which also exposes other HTTP APIs. This change makes it possible
to run the metrics server on an independent endpoint without enabling pprof.
This PR reduces the bandwidth used by the light client to compute the
recommended gas price. The current mechanism for suggesting the price is:
- retrieve recent 20 blocks
- get the lowest gas price of these blocks
- sort the price array and return the middle(60%) one
This works for full nodes, which have all blocks available locally.
However, this is very expensive for the light client because the light
client needs to retrieve block bodies from the network.
The PR changes the default options for light client. With the new config,
the light client only retrieves the two latest blocks, but in order to
collect more sample transactions, the 3 lowest prices are collected from
each block.
This PR also changes the behavior for empty blocks. If the block is empty,
the lastest price is reused for sampling.
* core: use uint64 for total tx costs instead of big.Int
* core: added local tx pool test case
* core, crypto: various allocation savings regarding tx handling
* Update core/tx_list.go
* core: added tx.GasPriceIntCmp for comparison without allocation
adds a method to remove unneeded allocation in comparison to tx.gasPrice
* core: handle pools full of locals better
* core/tests: benchmark for tx_list
* core/txlist, txpool: save a reheap operation, avoid some bigint allocs
Co-authored-by: Martin Holst Swende <martin@swende.se>
List headers made up 11% of all allocations during sync. This change
removes most of those allocations by keeping the list header values
cached in the encoder buffer instead. Since encoder buffers are pooled,
list headers are no longer allocated in the common case where an
encoder buffer is available for reuse.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/downloaded: fixed datarace between synchronize and Progress
There was a race condition between `downloader.synchronize()` and `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders`
This PR changes the behavior of the downloader a bit.
Previously the functions `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders` read the syncMode anew within their loops. Now they read the syncMode at the start of their function and don't change it during their runtime.
* eth/downloaded: comment
* eth/downloader: added comment
* core, crypto: various allocation savings regarding tx handling
* core: reduce allocs for gas price comparison
This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees
* core: added tx.GasPriceIntCmp for comparison without allocation
adds a method to remove unneeded allocation in comparison to tx.gasPrice
* core/types: pool legacykeccak256 objects in rlpHash
rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.
* reverted some changes
* reverted some changes
* trie: use crypto.KeccakState instead of replicating code
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR implements the EVM state transition tool, which is intended
to be the replacement for our retesteth client implementation.
Documentation is present in the cmd/evm/README.md file.
Co-authored-by: Felix Lange <fjl@twurst.com>
Without use of global, these flags didn't actually modify
miner configuration, since we weren't grabbing from the
proper context scope, which should be global (vs. subcommand).
Signed-off-by: meows <b5c6@protonmail.com>
The import command should not return a 0 status
code if the import finishes prematurely becaues
of an import error.
Returning the error causes the program to exit with 1
if the err is non nil.
Signed-off-by: meows <b5c6@protonmail.com>
* core/vm: fix incorrect computation of discount
During testing on Yolov1 we found that the way geth calculates the discount
is not in line with the specification. Basically what we did is calculate
128 * Bls12381GXMulGas * discount / 1000 whenever we received more than 128 pairs
of values. Correct would be to calculate k * Bls12381... for k > 128.
* core/vm: better logic for discount calculation
* core/vm: better calculation logic, added worstcase benchmarks
* core/vm: better benchmarking logic
The ancients variable in the freezer is a list of hashes, which
identifies all of the hashes to be frozen. The slice is being allocated
with a capacity of `limit`, which is the number of the last block
this batch will attempt to add to the freezer. That means we are
allocating memory for all of the blocks in the freezer, not just
the ones to be added.
If instead we allocate `limit - f.frozen`, we will only allocate
enough space for the blocks we're about to add to the freezer. On
mainnet this reduces usage by about 320 MB.
* eth/downloader tests: fix spurious failing test due to race between receipts/headers
* miner tests: fix travis failure on arm64
* eth/downloader: tests - store td in ancients too
* core/vm: use fixed uint256 library instead of big
* core/vm: remove intpools
* core/vm: upgrade uint256, fixes uint256.NewFromBig
* core/vm: use uint256.Int by value in Stack
* core/vm: upgrade uint256 to v1.0.0
* core/vm: don't preallocate space for 1024 stack items (only 16)
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/ethapi: return revert reason for eth_call
* internal/ethapi: moved revert reason logic to doCall
* accounts/abi/bind/backends: added revert reason logic to simulated backend
* internal/ethapi: fixed linting error
* internal/ethapi: check if require reason can be unpacked
* internal/ethapi: better error logic
* internal/ethapi: simplify logic
* internal/ethapi: return vmError()
* internal/ethapi: move handling of revert out of docall
* graphql: removed revert logic until spec change
* rpc: internal/ethapi: added custom error types
* graphql: use returndata instead of return
Return() checks if there is an error. If an error is found, we return nil.
For most use cases it can be beneficial to return the output even if there
was an error. This code should be changed anyway once the spec supports
error reasons in graphql responses
* accounts/abi/bind/backends: added tests for revert reason
* internal/ethapi: add errorCode to revert error
* internal/ethapi: add errorCode of 3 to revertError
* internal/ethapi: unified estimateGasErrors, simplified logic
* internal/ethapi: unified handling of errors in DoEstimateGas
* rpc: print error data field
* accounts/abi/bind/backends: unify simulatedBackend and RPC
* internal/ethapi: added binary data to revertError data
* internal/ethapi: refactored unpacking logic into newRevertError
* accounts/abi/bind/backends: fix EstimateGas
* accounts, console, internal, rpc: minor error interface cleanups
* Revert "accounts, console, internal, rpc: minor error interface cleanups"
This reverts commit 2d3ef53c5304e429a04983210a417c1f4e0dafb7.
* re-apply the good parts of 2d3ef53c53
* rpc: add test for returning server error data from client
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Once we detect an invalid transaction during recovering signatures, we should
directly exclude this transaction to avoid validating the signatures hereafter.
This should optimize the validations times of transactions with invalid signatures
to only one time.
* replace gosigar with gopsutil
* removed check for whether GOOS is openbsd
* removed accidental import of runtime
* potential fix for difference in units between gosig and gopsutil
* fixed lint error
* remove multiplication factor
* uses cpu.ClocksPerSec as the multiplication factor
* changed dependency from shirou to renaynay (#20)
* updated dep
* switching back from using renaynay fork to using upstream as PRs were merged on upstream
* removed empty line
* optimized imports
* tidied go mod
This PR makes use of go 1.13 error handling, wrapping errors and using
errors.Is to check a wrapped root-cause. It also removes the travis
builders for go 1.11 and go 1.12.
This adds a new API method on core.BlockChain to allow interrupting
running data inserts, and calls the method before shutting down the
downloader.
The BlockChain interrupt checks are now done through a method instead
of inlining the atomic load everywhere. There is no loss of efficiency from
this and it makes the interrupt protocol a lot clearer because the check is
defined next to the method that sets the flag.
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.
Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.
This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4
* abi/bind/backends: testcase for double-lock
* accounts: add blockByNumberNoLock to avoid double-lock
* backend/simulated: use stateroot, not blockhash for retrieveing state
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/clef, signer/core: use better terminal input for passwords, make it possible to avoid boot-up warning
* all: move commonly used prompter to isolated (small) package
* cmd/clef: Add new --acceptWarn to clef README
* cmd/clef: rename flag 'acceptWarn' to 'suppress-bootwarn'
Co-authored-by: ligi <ligi@ligi.de>
* accounts/abi: simplified reflection logic
* accounts/abi: simplified reflection logic
* accounts/abi: removed unpack
* accounts/abi: removed comments
* accounts/abi: removed uneccessary complications
* accounts/abi: minor changes in error messages
* accounts/abi: removed unnused code
* accounts/abi: fixed indexed argument unpacking
* accounts/abi: removed superfluous test cases
This commit removes two test cases. The first one is trivially invalid as we have the same
test cases as passing in packing_test.go L375. The second one passes now,
because we don't need the mapArgNamesToStructFields in unpack_atomic anymore.
Checking for purely underscored arg names generally should not be something we do
as the abi/contract is generally out of the control of the user.
* accounts/abi: removed comments, debug println
* accounts/abi: added commented out code
* accounts/abi: addressed comments
* accounts/abi: remove unnecessary dst.CanSet check
* accounts/abi: added dst.CanSet checks
* accounts/abi: allow overloaded argument names
In solidity it is possible to create the following contract:
```
contract Overloader {
struct F { uint _f; uint __f; uint f; }
function f(F memory f) public {}
}
```
This however resulted in a panic in the abi package.
* accounts/abi fixed error handling
* accounts/abi/bind: added test cases for waitDeployed
* accounts/abi/bind: added test case for boundContract
* accounts/abi/bind: removed unnecessary resolve methods
* accounts/abi: moved topics from /bind to /abi
* accounts/abi/bind: cleaned up format... functions
* accounts/abi: improved log message
* accounts/abi: added type tests
* accounts/abi/bind: remove superfluous template methods
* cmd, miner: add noempty-precommit flag
* cmd, miner: get rid of external flag
* miner: change bool to atomic int
* miner: fix tiny typo
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* cmd, core, eth: init tx lookup in background
* core/rawdb: tiny log fixes to make it clearer what's happening
* core, eth: fix rebase errors
* core/rawdb: make reindexing less generic, but more optimal
* rlp: implement rlp list iterator
* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data
* core/rawdb, cmd/utils: fix review concerns
* cmd/utils: fix merge issue
* core/rawdb: add some log formatting polishes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* p2p: add low port check in dialer
We already have a check like this for UDP ports, add a similar one in
the dialer. This prevents dials to port zero and it's also an extra
layer of protection against spamming HTTP servers.
* p2p/discover: use errLowPort in v4 code
* p2p: change port check
* p2p: add comment
* p2p/simulations/adapters: ensure assigned port is in all node records
* core/rawdb: Stop freezer process as part of freezer.Close()
When you call db.Close(), it was closing the leveldb database first,
then closing the freezer, but never stopping the freezer process.
This could cause the freezer to attempt to write to leveldb after
leveldb had been closed, leading to a crash with a non-zero exit code.
This change adds a quit channel to the freezer, and freezer.Close()
will not return until the freezer process has stopped.
Additionally, when you call freezerdb.Close(), it will close the
AncientStore before closing leveldb, to ensure that the freezer goroutine
will be stopped before leveldb is closed.
* core/rawdb: Fix formatting for golint
* core/rawdb: Use backoff flag to avoid repeating select
* core/rawdb: Include accidentally omitted backoff
* accounts/abi/bind/backend, internal/ethapi: recap gas limit with balance
* accounts, internal: address comment and fix lint
* accounts, internal: extend log message
* tiny nits to format hexutil.Big and nil properly
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core/state: more verbose stateb errors
* core/state: fix flaw
* core/state: fixed lint
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This finally adds the error check that the documentation of StateDB.dbErr
promises to do. dbErr was added in 9e5f03b6c (June 2017), and the check was
already missing in that commit. We somehow survived without it for three years.
* accounts/abi/bind: add void if no return args specified
Currently the java generator generates invalid input on pure/view functions
that have no return type. e.g. `function f(uint u) view public {}`
This is not a problem in practice as people rarely ever write functions like this.
* accounts/abi/bind: use elseif instead of nested if
* rpc flags related to starting http server renamed to http
* old rpc flags aliased and still functional
* pprof flags fixed
* renames gpo related flags
* linted
* renamed rpc flags for consistency and clarity
* added warn logs
* added more warn logs for all deprecated flags for consistency
* moves legacy flags to separate file, hides older flags under show-deprecated-flags command
* legacy prefix and moved some more legacy flags to legacy file
* fixed circular import
* added docs
* fixed imports lint error
* added notes about when flags were deprecated
* cmd/utils: group flags by deprecation date + reorder by date,
* modified deprecated comments for consistency, added warn log for --rpc
* making sure deprecated flags are still functional
* show-deprecated-flags command cleaned up
* fixed lint errors
* corrected merge conflict
* IsSet --> GlobalIsSet
* uncategorized flags, if not deprecated, displayed under misc
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/state/snapshot: implement storage iterator
* core/state/snapshot, tests: implement helper function
* core/state/snapshot: fix storage issue
If an account is deleted in the tx_1 but recreated in the tx_2,
the it can happen that in this diff layer, both destructedSet
and storageData records this account. In this case, the storage
iterator should be able to iterate the slots belong to new account
but disable further iteration in deeper layers(belong to old account)
* core/state/snapshot: address peter and martin's comment
* core/state: address comments
* core/state/snapshot: fix test
golang-lru is now a go module, and the upgrade corrects a couple
of minor issues. In particular, the library could crash if you inserted
nil into an LRU cache.
This is a resubmit of #20668 which rewrites the problematic test
without any additional goroutines. It also documents the test better.
The purpose of this test is checking whether log events are sent
correctly when importing blocks. The test was written at a time when
blockchain events were delivered asynchronously, making the check hard
to pull off. Now that core.BlockChain delivers events synchronously
during the call to InsertChain, the test can be simplified.
Co-authored-by: BurtonQin <bobbqqin@gmail.com>
This corrects the call to eth_getBlockByNumber, which previously
returned this error:
can't get latest block: missing value for required argument 1
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a theoretical double lock condition which could occur in
indexer.AddChildIndexer(indexer)
Nobody would ever do that though.
Co-authored-by: Felix Lange <fjl@twurst.com>
* accounts/abi: added documentation
* accounts/abi: reduced usage of arguments.LengthNonIndexed
* accounts/abi: simplified reflection logic
* accounts/abi: moved testjson data into global declaration
* accounts/abi: removed duplicate test cases
* accounts/abi: reworked abi tests
* accounts/abi: added more tests for abi packing
* accounts/abi/bind: refactored base tests
* accounts/abi: run pack tests as subtests
* accounts/abi: removed duplicate tests
* accounts/abi: removed unnused arguments.LengthNonIndexed
Due to refactors to the code, we do not need the arguments.LengthNonIndexed function anymore.
You can still get the length by calling len(arguments.NonIndexed())
* accounts/abi: added type test
* accounts/abi: modified unpack test to pack test
* accounts/abi: length check on arrayTy
* accounts/abi: test invalid abi
* accounts/abi: fixed rebase error
* accounts/abi: fixed rebase errors
* accounts/abi: removed unused definition
* accounts/abi: merged packing/unpacking tests
* accounts/abi: fixed [][][32]bytes encoding
* accounts/abi: added tuple test cases
* accounts/abi: renamed getMockLog -> newMockLog
* accounts/abi: removed duplicate test
* accounts/abi: bools -> booleans
Rather than just closing the underlying network listener to stop our
HTTP servers, use the graceful shutdown procedure, waiting for any
in-process requests to finish.
It is possible to specify enode URLs using domain name since
commit b90cdbaa79, but the code comment still said that only
IP addresses are allowed.
Co-authored-by: admin@komgo.io <KomgoRocks2018!>
* trie: initial implementation for range proof
* trie: add benchmark
* trie: fix lint
* trie: fix minor issue
* trie: unset the edge valuenode as well
* trie: unset the edge valuenode as nilValuenode
* core: fix the condition of reorg
* core: fix nitpick to only retrieve head once
* core: don't reorg if received chain is longer at same diff
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* all: seperate consensus error and evm internal error
There are actually two types of error will be returned when
a tranaction/message call is executed: (a) consensus error
(b) evm internal error. The former should be converted to
a consensus issue, e.g. The sender doesn't enough asset to
purchase the gas it specifies. The latter is allowed since
evm itself is a blackbox and internal error is allowed to happen.
This PR emphasizes the difference by introducing a executionResult
structure. The evm error is embedded inside. So if any error
returned, it indicates consensus issue happens.
And also this PR improve the `EstimateGas` API to return the concrete
revert reason if the transaction always fails
* all: polish
* accounts/abi/bind/backends: add tests
* accounts/abi/bind/backends, internal: cleanup error message
* all: address comments
* core: fix lint
* accounts, core, eth, internal: address comments
* accounts, internal: resolve revert reason if possible
* accounts, internal: address comments
* accounts/abi: prevent recalculation of ID, Sig and String
* accounts/abi: fixed unpacking of no values
* accounts/abi: multiple fixes to arguments
* accounts/abi: refactored methodName and eventName
This commit moves the complicated logic of how we assign method names
and event names if they already exist into their own functions for
better readability.
* accounts/abi: prevent recalculation of internal
In this commit, I changed the way we calculate the string
representations, sig representations and the id's of methods. Before
that these fields would be recalculated everytime someone called .Sig()
.String() or .ID() on a method or an event.
Additionally this commit fixes issue #20856 as we assign names to inputs
with no name (input with name "" becomes "arg0")
* accounts/abi: added unnamed event params test
* accounts/abi: fixed rebasing errors in method sig
* accounts/abi: fixed rebasing errors in method sig
* accounts/abi: addressed comments
* accounts/abi: added FunctionType enumeration
* accounts/abi/bind: added test for unnamed arguments
* accounts/abi: improved readability in NewMethod, nitpicks
* accounts/abi: method/eventName -> overloadedMethodName
This adds two new methods to UDPv5, AllNodes and LocalNode.
AllNodes returns all the nodes stored in the local table; this is
useful for the purposes of metrics collection and also debugging any
potential issues with other discovery v5 implementations.
LocalNode returns the local node object. The reason for exposing this
is so that users can modify and set/delete new key-value entries in
the local record.
This should only be used in very rare cases e.g. if you are not 100% sure if something is a bug or asking a question that leads to improving the documentation. For general questions please use [discord](https://discord.gg/nthXNEv) or the Ethereum stack exchange at https://ethereum.stackexchange.com.
text: 'SA1019:"golang.org/x/crypto/openpgp"is deprecated:this package is unmaintained except for security fixes.'
- path:core/vm/contracts.go
text: 'SA1019:"golang.org/x/crypto/ripemd160"is deprecated:RIPEMD-160 is a legacy hash and should not be used for new applications.'
- path:accounts/usbwallet/trezor.go
text: 'SA1019:"github.com/golang/protobuf/proto"is deprecated:Use the "google.golang.org/protobuf/proto" package instead.'
- path:accounts/usbwallet/trezor/
text: 'SA1019:"github.com/golang/protobuf/proto"is deprecated:Use the "google.golang.org/protobuf/proto" package instead.'
exclude:
- 'SA1019: event.TypeMux is deprecated:use Feed'
- 'SA1019:strings.Title is deprecated'
- 'SA1019: strings.Title has been deprecated since Go 1.18 and an alternative has been available since Go 1.0:The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead.'
- 'SA1029:should not use built-in type string as key for value'
- sed -i '.bak' 's/repo.join/!repo.join/g' $(dirname `gem which cocoapods`)/cocoapods/sources_manager.rb
- if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then git clone --depth=1 https://github.com/CocoaPods/Specs.git ~/.cocoapods/repos/master && pod setup --verbose; fi
- xctool -version
- xcrun simctl list
# Workaround for https://github.com/golang/go/issues/23749
* [rpc: check that "version" is "2.0" in request objects (#25570)](https://github.com/bnb-chain/bsc/pull/1840/commits/38e002f4641c2779c897ccaca575ec5ddeee9254)
* [rpc: support injecting HTTP headers through context (#26023)](https://github.com/bnb-chain/bsc/pull/1840/commits/add337e0f7bad02f3cf535c66cd31f252b0b5c99)
* [rpc: websocket should respect the "HTTP_PROXY" by default (#27264)](https://github.com/bnb-chain/bsc/pull/1840/commits/73697529994e14996b7740730481e926d5ec3e40)
* [rpc: change BlockNumber constant values to match ethclient (#27219)](https://github.com/bnb-chain/bsc/pull/1840/commits/9231770811cda0473a7fa4e2bccc95bf62aae634)
* [eth: make debug_StorageRangeAt take a block hash or number (#27328)](https://github.com/bnb-chain/bsc/pull/1840/commits/d789c68b667e13eb5cefd19d09ae84f7d016df6a)
* [eth,core: add api debug_getTrieFlushInterval (#27303)](https://github.com/bnb-chain/bsc/pull/1840/commits/0783cb7d91ad7b3cdf72ac6c6edaec8318673eb6)
* [rpc: add limit for batch request items and response size (#26681)](https://github.com/bnb-chain/bsc/pull/1840/commits/f3314bb6df4c86e650f0e47cbb5a21ca0616ac11)
* [core/types: support yParity field in JSON transactions (#27744)](https://github.com/bnb-chain/bsc/pull/1840/commits/bb148dd342ba03ce40cf04295e193c94b9dda322)
* [eth/filters: send rpctransactions in pending-subscription (#26126)](https://github.com/bnb-chain/bsc/pull/1840/commits/8c5ce1107b3110c7cb735d8dfa91c9c701393c85)
#### Flag
* [cmd/geth: rename --whitelist to --eth.requiredblocks (#24505)](https://github.com/bnb-chain/bsc/pull/1840/commits/dbfd3972624c1d82db21f5dfceab8fde7a1eee0a)
* [cmd: migrate to urfave/cli/v2 (#24751)](https://github.com/bnb-chain/bsc/pull/1840/commits/52ed3570c483693fdd6667add7e3050520ad3ba2)
* [cmd/utils: print warning when --metrics.port set without --metrics.ad…](https://github.com/bnb-chain/bsc/pull/1840/commits/8846c07d044f30dca8cd0db91c6245f71f4b24fa)
* [cmd/devp2p: add --extaddr flag (#26312)](https://github.com/bnb-chain/bsc/pull/1840/commits/b44abf56a966016cbb651648ac2d7b6705e80b11)
* [core,eth: adddebug_setTrieFlushInterval to change trie flush frequ](https://github.com/bnb-chain/bsc/pull/1840/commits/711afbc7fd76f1f206429e26f9aa5bf98bc7b43d)
* [graphql: fee history fields (#24452)](https://github.com/bnb-chain/bsc/pull/1840/commits/57cec892536270fc6dafae01ded2c528ffa370e9)
* [graphql: add rawReceipt field to transaction type (#24738)](https://github.com/bnb-chain/bsc/pull/1840/commits/d73df893a6fc528e69506397322205bd9258b6fa)
* [graphql: add raw fields to block and tx (#24816)](https://github.com/bnb-chain/bsc/pull/1840/commits/29a6b6bcac170ca7f8fceb242eba45ff15df17a1)
* [graphql: return correct logs for tx (#25612)](https://github.com/bnb-chain/bsc/pull/1840/commits/d0dc349fd36bd79f94516c866251783641ed12f1)
* [ethereum, ethclient: add FeeHistory support (#25403)](https://github.com/bnb-chain/bsc/pull/1840/commits/9ad508018e4790da0c1c00ac355f206fca12ab7c)
* [eth/filters, ethclient/gethclient: add fullTx option to pending tx fi…](https://github.com/bnb-chain/bsc/pull/1840/commits/5b1a04b9c749d804b51159fe12246c56de8515c1)
* [ethclient: include withdrawals in ethclient block responses (#26778)](https://github.com/bnb-chain/bsc/pull/1840/commits/e1b98f49a5075694c5022f5ec74425e40da415dd)
#### Tracer
* [eth/tracers/js: drop duktape engine (#24934)](https://github.com/bnb-chain/bsc/pull/1840/commits/ba47d800b13058885288c38bd174babb38560c89)
* [eth/tracers: add support for block overrides in debug_traceCall (#24871)](https://github.com/bnb-chain/bsc/pull/1840/commits/d8a2305565b1f97c451f8595e0f65358d6842714)
* [eth/tracers: add onlyTopCall option to callTracer (#25430)](https://github.com/bnb-chain/bsc/pull/1840/commits/86de2e516e5a4a2bbe1d29b46a0f460fbdde8303)
* [eth/tracers: remove revertReasonTracer, add revert reason to callTracer](https://github.com/bnb-chain/bsc/pull/1840/commits/ff1f49245d641a7268ade38cf512bdc7b26f9b7c)
* [eth/tracers: add diffMode to prestateTracer (#25422)](https://github.com/bnb-chain/bsc/pull/1840/commits/5d52a35931bba10f438ce4f41410442dd9cd396c)
* [core/vm: set tracer-observable value of a delegatecall to match parent `value`](https://github.com/bnb-chain/bsc/pull/1840/commits/b0cd8c4a5c4f0f25011ed64235a3ea1280f03c51)
* [eth/tracers/native: set created address to nil in case of failure (#2…](https://github.com/bnb-chain/bsc/pull/1840/commits/41af42e97c9d62d303a883cc3c143f560867fa34)
* [eth/tracers: report correct gasLimit in call tracers (#27029)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b76eb3708626fbd2eb9c1b58d7b4eac6a5eec15)
* [eth/tracers: addtxHashfield on txTraceResult (#27183)](https://github.com/bnb-chain/bsc/pull/1840/commits/604e215d1bb070dff98fb76aa965064c74e3633f)
* [eth/tracers: add ReturnData in the tracer's response (#27704)](https://github.com/bnb-chain/bsc/pull/1840/commits/1e069cf8026a9f71b5f7e80959465e4b273d5806)
* [core/state/snapshot: detect and clean up dangling storage snapshot in generation](https://github.com/bnb-chain/bsc/pull/1840/commits/59ac229f87831bd74b4dc07d34f54137cca78095)
* [internal/ethapi: add db operations to api (#24739)](https://github.com/bnb-chain/bsc/pull/1840/commits/16701c51697e28986feebd122c6a491e4d9ac0e7)
* [cmd/geth: adddb check-state-contentto verify integrity of trie nodes (#24840)](https://github.com/bnb-chain/bsc/pull/1840/commits/e0a9752b965f243313f2c32a91d306600dc3863c)
* [ethdb/remotedb, cmd: add support for remote (readonly) databases](https://github.com/bnb-chain/bsc/pull/1840/commits/57192bd0dc545d921306f6a4d7566c0c70c764c5)
* [cmd/abigen: accept combined-json via stdin (#24960)](https://github.com/bnb-chain/bsc/pull/1840/commits/0287e1a7c00c1eaad1a99b4ea05d70f1ed685140)
* [cmd/geth: add a verkle subcommand (#25718)](https://github.com/bnb-chain/bsc/pull/1840/commits/9d717167aaf27a48d56ad9d1a2c36f90eba1cc13)
* [cmd/geth, cmd/utils: geth attach with custom headers (#25829)](https://github.com/bnb-chain/bsc/pull/1840/commits/ea26fc8a6c44ebb48223f991048f41b2ec0a6414)
* [core/rawdb: refactor db inspector for extending multiple ancient storage](https://github.com/bnb-chain/bsc/pull/1840/commits/60e30a940bbba2c0d26de040195a5ccdb14d8c10)
* [cmd/clef: add importraw feature to clef (#26058)](https://github.com/bnb-chain/bsc/pull/1840/commits/17744639dafc5a54f21e220660bd39d765a09051)
* [cmd/devp2p: add more nodekey commands (#26129)](https://github.com/bnb-chain/bsc/pull/1840/commits/913973436bb88b652faffc10d8f97e4c19722883)
* [internal/web3ext: fix eth_call stateOverrides in console (#26265)](https://github.com/bnb-chain/bsc/pull/1840/commits/1325fef1025b9feb3342308265b6d1399614be30)
* [cmd/evm: add blocktest subcommand to evm (#26526)](https://github.com/bnb-chain/bsc/pull/1840/commits/90f15a0230be34a292c5d0574ee7910ee44267de)
#### HardFork
* [params: define cancun and prague as timestamp based forks (#26481)](https://github.com/bnb-chain/bsc/pull/1840/commits/f3a005f176372ff291dfa7c02ee1c87d18e9c788)
* [all: tie timestamp based forks to the passage of London (#27279)](https://github.com/bnb-chain/bsc/pull/1840/commits/85a4b82b3373fc5f3fa8b7c68061c55b0db0e9b7)
* [core/vm: implement EIP-3860: Limit and meter initcode (#23847)](https://github.com/bnb-chain/bsc/pull/1840/commits/793f0f9ec860f6f51e0cec943a268c10863097c7)
* [PBBS(ready to activate)](https://github.com/ethereum/go-ethereum/commits?author=rjl493456442)
#### P2P
* [eth/fetcher: throttle peers which deliver many invalid transactions (…](https://github.com/bnb-chain/bsc/pull/1840/commits/7f2890a9be1f91368582479f171248b972b45ae3)
* [all: remove concept of public/private API definitions (#25053)](https://github.com/bnb-chain/bsc/pull/1840/commits/10dc5dce0871bf8c24bac41b04e47c3b9ad2b93e)
* [cmd/geth: drop geth js command (#25000)](https://github.com/bnb-chain/bsc/pull/1840/commits/f20a56926551ae91a349498f9ce97c8ee373d6bb)
* [node: drop support for static & trusted node list files (#25610)](https://github.com/bnb-chain/bsc/pull/1840/commits/3630cafb34f7c48b9cc78cf736309275cbd70f74)
* [core: drop legacy receipt types (#26225)](https://github.com/bnb-chain/bsc/pull/1840/commits/10347c6b54d5b28a2e71d9c4993e7f44b0a359c3)
* [cmd, eth, node: deprecate personal namespace (#26390)](https://github.com/bnb-chain/bsc/pull/1840/commits/d0a4989a8def7e6bad182d1513e8d4a093c1672d)
* [accounts, build, mobile: remove Andriod and iOS support](https://github.com/bnb-chain/bsc/pull/1840/commits/d9699c8238307d5c3081c12078f78527468d7dbc)
* [params: remove EIP150Hash from chainconfig (#27087)](https://github.com/bnb-chain/bsc/pull/1840/commits/5e4d726e2a05aee80a75e5f99fd699f220dd503e)
* [all: remove notion of trusted checkpoints in the post-merge world (#2…](https://github.com/bnb-chain/bsc/pull/1840/commits/1e556d220c3a40286dd90b37a08bb5fc659ee6ee)
* [all: remove ethash pow, only retain shims needed for consensus and te](https://github.com/bnb-chain/bsc/pull/1840/commits/dde2da0efb8e9a1812f470bc43254134cd1f8cc0)
* [les: remove obsolete code related to PoW header syncing (#27737)](https://github.com/bnb-chain/bsc/pull/1840/commits/d4d88f9bce13ca9310bf28f5f26ea9f1915ba90d)
* remove diffsync
#### Others
* [accounts/usbwallet: support Ledger Nano S Plus and FTS (#25933)](https://github.com/bnb-chain/bsc/pull/1840/commits/7eafbec741d124bc53896f6bfc2408b70ab9a82a)
* [accounts/scwallet: fix keycard data signing error (#25331)](https://github.com/bnb-chain/bsc/pull/1840/commits/0c66d971e7f3557df297cbe450fe7fc7826017be)
* [core/state: replace fastcache code cache with gc-friendly structure (…](https://github.com/bnb-chain/bsc/pull/1840/commits/5fded040372784985265f83f33f15cb6a51bebdb)
* [ci: disable coverage reporting in appveyor and travis](https://github.com/bnb-chain/bsc/pull/1840/commits/a0d63bc69a659009a3884f50c563a0e58483cdd0)
* [all: change chain head markers from block to header (#26777)](https://github.com/bnb-chain/bsc/pull/1840/commits/cd31f2dee2843776e485769ce85e0524716199bc)
* [core, miner: revert block gas counter in case of invalid transaction](https://github.com/bnb-chain/bsc/pull/1840/commits/77e33e5a49be99130a02dc72d6a0e4739fdd44d6)
* [signer/core: accept all solidity primitive types for EIP-712 signing](https://github.com/bnb-chain/bsc/pull/1840/commits/02796f6bee7e014fd16ad39f0bcd3b665b51e0bb)
* [internal/ethapi: make EstimateGas use[latest] block by default (#24363)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b66d47449f61e9ebaf9e1db3ed290b59844d4c1)
* [miner: suspend miner if node is syncing (#27218)](https://github.com/bnb-chain/bsc/pull/1840/commits/d4961881d7c92603f591f9cb8c705d00d8cbdfc0)
* [all: move main transaction pool into a subpool (#27463)](https://github.com/bnb-chain/bsc/pull/1840/commits/d40a255e973775575d8d16456252f93ac75c09f0)
* [core/txpool/blobpool: 4844 blob transaction pool (#26940)](https://github.com/bnb-chain/bsc/pull/1840/commits/1662228ac68325b4024e0cb6a4ce7dde27eb4c2d)
* [eth: send big transactions by announce/retrieve only (#27618)](https://github.com/bnb-chain/bsc/pull/1840/commits/f5d3d486e459dce29130576ae88f2324ad586b50)
* [core/rawdb: support freezer batch read with no size limit (#27687)](https://github.com/bnb-chain/bsc/pull/1840/commits/0b1f97e151e8b34a0a0d528a3472e27de1d12a9c)
The goal of Binance Smart Chain is to bring programmability and interoperability to Binance Chain. In order to embrace the existing popular community and advanced technology, it will bring huge benefits by staying compatible with all the existing smart contracts on Ethereum and Ethereum tooling. And to achieve that, the easiest solution is to develop based on go-ethereum fork, as we respect the great work of Ethereum very much.
The goal of BNB Smart Chain is to bring programmability and interoperability to BNB Beacon Chain. In order to embrace the existing popular community and advanced technology, it will bring huge benefits by staying compatible with all the existing smart contracts on Ethereum and Ethereum tooling. And to achieve that, the easiest solution is to develop based on go-ethereum fork, as we respect the great work of Ethereum very much.
Binance Smart Chain starts its development based on go-ethereum fork. So you may see many toolings, binaries and also docs are based on Ethereum ones, such as the name “geth”.
BNB Smart Chain starts its development based on go-ethereum fork. So you may see many toolings, binaries and also docs are based on Ethereum ones, such as the name “geth”.
But from that baseline of EVM compatible, Binance Smart Chain introduces a system of 21 validators with Proof of Staked Authority (PoSA) consensus that can support short block time and lower fees. The most bonded validator candidates of staking will become validators and produce blocks. The double-sign detection and other slashing logic guarantee security, stability, and chain finality.
Cross-chain transfer and other communication are possible due to native support of interoperability. Relayers and on-chain contracts are developed to support that. Binance DEX remains a liquid venue of the exchange of assets on both chains. This dual-chain architecture will be ideal for users to take advantage of the fast trading on one side and build their decentralized apps on the other side. **The Binance Smart Chain** will be:
But from that baseline of EVM compatible, BNB Smart Chain introduces a system of 21 validators with Proof of Staked Authority (PoSA) consensus that can support short block time and lower fees. The most bonded validator candidates of staking will become validators and produce blocks. The double-sign detection and other slashing logic guarantee security, stability, and chain finality.
Cross-chain transfer and other communication are possible due to native support of interoperability. Relayers and on-chain contracts are developed to support that. BNB Beacon Chain DEX remains a liquid venue of the exchange of assets on both chains. This dual-chain architecture will be ideal for users to take advantage of the fast trading on one side and build their decentralized apps on the other side. **The BNB Smart Chain** will be:
- **A self-sovereign blockchain**: Provides security and safety with elected validators.
- **EVM-compatible**: Supports all the existing Ethereum tooling along with faster finality and cheaper transaction fees.
- **Interoperable**: Comes with efficient native dual chain communication; Optimized for scaling high-performance dApps that require fast and smooth user experience.
- **Distributed with on-chain governance**: Proof of Staked Authority brings in decentralization and community participants. As the native token, BNB will serve as both the gas of smart contract execution and tokens for staking.
More details in [White Paper](http://binance.org/en#smartChain).
More details in [White Paper](https://www.bnbchain.org/en#smartChain).
## Key features
@@ -25,38 +30,38 @@ Meanwhile, the PoA protocol is most criticized for being not as decentralized as
Other blockchains, such as EOS and Cosmos both, introduce different types of Deputy Proof of Stake (DPoS) to allow the token holders to vote and elect the validator set. It increases the decentralization and favors community governance.
To combine DPoS and PoA for consensus, Binance Smart Chain implement a novel consensus engine called Parlia that:
To combine DPoS and PoA for consensus, BNB Smart Chain implement a novel consensus engine called Parlia that:
1. Blocks are produced by a limited set of validators.
2. Validators take turns to produce blocks in a PoA manner, similar to Ethereum's Clique consensus engine.
3. Validator set are elected in and out based on a staking based governance on Binance Chain.
3. Validator set are elected in and out based on a staking based governance on BNB Beacon Chain.
4. The validator set change is relayed via a cross-chain communication mechanism.
5. Parlia consensus engine will interact with a set of [system contracts](https://github.com/binance-chain/docs-site/blob/add-bsc/docs/smart-chain/guides/concepts/system-contract.md) to achieve liveness slash, revenue distributing and validator set renewing func.
5. Parlia consensus engine will interact with a set of [system contracts](https://docs.bnbchain.org/docs/learn/system-contract) to achieve liveness slash, revenue distributing and validator set renewing func.
### Light Client of Binance Chain
### Light Client of BNB Beacon Chain
To achieve the cross-chain communication from Binance Chain to Binance Smart Chain, need introduce a on-chain light client verification algorithm.
To achieve the cross-chain communication from BNB Beacon Chain to BNB Smart Chain, need introduce a on-chain light client verification algorithm.
It contains two parts:
1. [Stateless Precompiled contracts](https://github.com/binance-chain/bsc/blob/master/core/vm/contracts_lightclient.go) to do tendermint header verification and Merkle Proof verification.
2. [Stateful solidity contracts](https://github.com/binance-chain/bsc-genesis-contract/blob/master/contracts/TendermintLightClient.sol) to store validator set and trusted appHash.
1. [Stateless Precompiled contracts](https://github.com/bnb-chain/bsc/blob/master/core/vm/contracts_lightclient.go) to do tendermint header verification and Merkle Proof verification.
2. [Stateful solidity contracts](https://github.com/bnb-chain/bsc-genesis-contract/blob/master/contracts/TendermintLightClient.sol) to store validator set and trusted appHash.
## Native Token
BNB will run on Binance Smart Chain in the same way as ETH runs on Ethereum so that it remains as `native token` for BSC. This means,
BNB will run on BNB Smart Chain in the same way as ETH runs on Ethereum so that it remains as `native token` for BSC. This means,
BNB will be used to:
1. pay `gas` to deploy or invoke Smart Contract on BSC
2. perform cross-chain operations, such as transfer token assets across Binance Smart Chain and Binance Chain.
2. perform cross-chain operations, such as transfer token assets across BNB Smart Chain and BNB Beacon Chain.
## Building the source
Many of the below are the same as or similar to go-ethereum.
For prerequisites and detailed build instructions please read the [Installation Instructions](https://github.com/ethereum/go-ethereum/wiki/Building-Ethereum) on the wiki.
For prerequisites and detailed build instructions please read the [Installation Instructions](https://geth.ethereum.org/docs/getting-started/installing-geth).
Building `geth` requires both a Go (version 1.13 or later) and a C compiler. You can install
Building `geth` requires both a Go (version 1.21 or later) and a C compiler (GCC 5 or higher). You can install
them using your favourite package manager. Once the dependencies are installed, run
```shell
@@ -69,55 +74,145 @@ or, to build the full suite of utilities:
make all
```
If you get such error when running the node with self built binary:
```shell
Caught SIGILL in blst_cgo_init, consult <blst>/bindinds/go/README.md.
```
please try to add the following environment variables and build again:
```shell
exportCGO_CFLAGS="-O -D__BLST_PORTABLE__"
exportCGO_CFLAGS_ALLOW="-O -D__BLST_PORTABLE__"
```
## Executables
The bsc project comes with several wrappers/executables found in the `cmd`
|**`geth`** | Main Binance Smart Chain client binary. It is the entry point into the BSC network (main-, test- or private net), capable of running as a full node (default), archive node (retaining all historical state) or a light node (retrieving data live). It has the same and more RPC and other interface as go-ethereum and can be used by other processes as a gateway into the BSC network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. `geth --help` and the [CLI Wiki page](https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options) for command line options. |
| `abigen`,`bootnode`,`evm`, `gethrpctest`,`rlpdump`,`puppeth` | **These binaries are exactly the same as the binaries built in go-ethereum repo.**|
| **`geth`** | Main BNB Smart Chain client binary. It is the entry point into the BSC network (main-, test- or private net), capable of running as a full node (default), archive node (retaining all historical state) or a light node (retrieving data live). It has the same and more RPC and other interface as go-ethereum and can be used by other processes as a gateway into the BSC network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. `geth --help` and the [CLI page](https://geth.ethereum.org/docs/interface/command-line-options) for command line options. |
| `clef` | Stand-alone signing tool, which can be used as a backend signer for `geth`. |
| `devp2p` | Utilities to interact with nodes on the networking layer, without running a full blockchain. |
| `abigen` | Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://docs.soliditylang.org/en/develop/abi-spec.html) with expanded functionality if the contract bytecode is also available. However, it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://geth.ethereum.org/docs/dapp/native-bindings) page for details. |
| `bootnode` | Stripped down version of our Ethereum client implementation that only takes part in the network node discovery protocol, but does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks. |
| `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow isolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug run`). |
| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://ethereum.org/en/developers/docs/data-structures-and-encoding/rlp)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user-friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
## Running `geth`
Going through all the possible command line flags is out of scope here (please consult our
[CLI Wiki page](https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options)).
[CLI Wiki page](https://geth.ethereum.org/docs/fundamentals/command-line-options)),
but we've enumerated a few common parameter combos to get you up to speed quickly
on how you can run your own `geth` instance.
### Hardware Requirements
The hardware must meet certain requirements to run a full node.
- VPS running recent versions of Mac OS X or Linux.
-500 GB of free disk space
-8 cores of CPU and 16 gigabytes of memory (RAM) for mainnet.
-4 cores of CPU and 8 gigabytes of memory (RAM) for testnet.
- A broadband Internet connection with upload/download speeds of at least 1 megabyte per second
The hardware must meet certain requirements to run a full node on mainnet:
- VPS running recent versions of Mac OS X, Linux, or Windows.
-IMPORTANT 3 TB(Dec 2023) of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 500 MB/S throughput, read latency <1ms. (if node is started with snap sync, it will need NVMe SSD)
-16 cores of CPU and 64 GB of memory (RAM)
-Suggest m5zn.6xlarge or r7iz.4xlarge instance type on AWS, c2-standard-16 on Google cloud.
- A broadband Internet connection with upload/download speeds of 5 MB/S
### A Full node on the Rialto testnetwork
The requirement for testnet:
- VPS running recent versions of Mac OS X, Linux, or Windows.
- 500G of storage for testnet.
- 4 cores of CPU and 16 gigabytes of memory (RAM).
Steps:
### Steps to Run a Fullnode
1. Download the binary, config and genesis files from [release](https://github.com/binance-chain/bsc/releases/download/v1.0.0-alpha.0/binary.zip), or compile the binary by `make geth`.
3. Start your fullnode: `./geth --config ./config.toml --datadir ./node`.
4. Or start a validator node: `./geth --config ./config.toml --datadir ./node -unlock ${validatorAddr} --mine --allow-insecure-unlock`. The ${validatorAddr} is the wallet account address of your running validator node.
More details about [running a node](https://docs.binance.org/smart-chain/developer/fullnode.html) and [becoming a validator](https://docs.binance.org/smart-chain/validator/candidate.html).
Download latest chaindata snapshot from [here](https://github.com/bnb-chain/bsc-snapshots). Follow the guide to structure your files.
Note: If you encounter difficulties downloading the chaindata snapshot and prefer to synchronize from the genesis block on the Chapel testnet, remember to include the additional flag `--chapel` when initially launching Geth.
Start up `geth`'s built-in interactive [JavaScript console](https://geth.ethereum.org/docs/interface/javascript-console),
(via the trailing `console` subcommand) through which you can interact using [`web3` methods](https://web3js.readthedocs.io/en/)
(note: the `web3` version bundled within `geth` is very old, and not up to date with official docs),
as well as `geth`'s own [management APIs](https://geth.ethereum.org/docs/rpc/server).
This tool is optional and if you leave it out you can always attach to an already running
`geth` instance with `geth attach`.
#### 7. More
More details about [running a node](https://docs.bnbchain.org/docs/validator/fullnode) and [becoming a validator](https://docs.bnbchain.org/docs/validator/create-val)
*Note: Although some internal protective measures prevent transactions from
crossing over between the main network and test network, you should always
use separate accounts for play and real money. Unless you manually move
accounts, `geth` will by default correctly separate the two networks and will not make any
accounts available between them.*
### Configuration
As an alternative to passing the numerous flags to the `geth` binary, you can also pass a
configuration file via:
```shell
$ geth --config /path/to/your_config.toml
```
To get an idea of how the file should look like you can use the `dumpconfig` subcommand to
export your existing configuration:
```shell
$ geth --your-favourite-flags dumpconfig
```
### Programmatically interfacing `geth` nodes
As a developer, sooner rather than later you'll want to start interacting with `geth` and the
Binance Smart Chain network via your own programs and not manually through the console. To aid
this, `geth` has built-in support for a JSON-RPC based APIs ([standard APIs](https://github.com/ethereum/wiki/wiki/JSON-RPC)
and [`geth` specific APIs](https://github.com/ethereum/go-ethereum/wiki/Management-APIs)).
BSC network via your own programs and not manually through the console. To aid
this, `geth` has built-in support for a JSON-RPC based APIs ([standard APIs](https://ethereum.github.io/execution-apis/api-documentation/)
and [`geth` specific APIs](https://geth.ethereum.org/docs/interacting-with-geth/rpc)).
These can be exposed via HTTP, WebSockets and IPC (UNIX sockets on UNIX based
platforms, and named pipes on Windows).
@@ -128,19 +223,19 @@ you'd expect.
HTTP based JSON-RPC API options:
*`--rpc` Enable the HTTP-RPC server
*`--rpcaddr` HTTP-RPC server listening interface (default: `localhost`)
*`--rpcport` HTTP-RPC server listening port (default: `8545`)
*`--rpcapi` API's offered over the HTTP-RPC interface (default: `eth,net,web3`)
*`--rpccorsdomain` Comma separated list of domains from which to accept cross origin requests (browser enforced)
*`--ws` Enable the WS-RPC server
*`--wsaddr` WS-RPC server listening interface (default: `localhost`)
*`--wsport` WS-RPC server listening port (default: `8546`)
*`--wsapi` API's offered over the WS-RPC interface (default: `eth,net,web3`)
*`--wsorigins` Origins from which to accept websockets requests
*`--ipcdisable` Disable the IPC-RPC server
*`--ipcapi` API's offered over the IPC-RPC interface (default: `admin,debug,eth,miner,net,personal,shh,txpool,web3`)
*`--ipcpath` Filename for IPC socket/pipe within the datadir (explicit paths escape it)
*`--http` Enable the HTTP-RPC server
*`--http.addr` HTTP-RPC server listening interface (default: `localhost`)
*`--http.port` HTTP-RPC server listening port (default: `8545`)
*`--http.api` API's offered over the HTTP-RPC interface (default: `eth,net,web3`)
*`--http.corsdomain` Comma separated list of domains from which to accept cross origin requests (browser enforced)
*`--ws` Enable the WS-RPC server
*`--ws.addr` WS-RPC server listening interface (default: `localhost`)
*`--ws.port` WS-RPC server listening port (default: `8546`)
*`--ws.api` API's offered over the WS-RPC interface (default: `eth,net,web3`)
*`--ws.origins` Origins from which to accept WebSocket requests
*`--ipcdisable` Disable the IPC-RPC server
*`--ipcapi` API's offered over the IPC-RPC interface (default: `admin,debug,eth,miner,net,personal,txpool,web3`)
*`--ipcpath` Filename for IPC socket/pipe within the datadir (explicit paths escape it)
You'll need to use your own programming environments' capabilities (libraries, tools, etc) to
connect via HTTP, WS or IPC to a `geth` node configured with the above flags and you'll
@@ -153,6 +248,65 @@ BSC nodes with exposed APIs! Further, all browser tabs can access locally
running web servers, so malicious web pages could try to subvert locally available
APIs!**
### Operating a private network
- [BSC-Deploy](https://github.com/bnb-chain/node-deploy/): deploy tool for setting up both BNB Beacon Chain, BNB Smart Chain and the cross chain infrastructure between them.
- [BSC-Docker](https://github.com/bnb-chain/bsc-docker): deploy tool for setting up local BSC cluster in container.
## Running a bootnode
Bootnodes are super-lightweight nodes that are not behind a NAT and are running just discovery protocol. When you start up a node it should log your enode, which is a public identifier that others can use to connect to your node.
First the bootnode requires a key, which can be created with the following command, which will save a key to boot.key:
```
bootnode -genkey boot.key
```
This key can then be used to generate a bootnode as follows:
Please see Releases. We recommend to use the most recent released version.
Please see [Releases](https://github.com/bnb-chain/bsc/releases). We recommend using the [most recently released version](https://github.com/bnb-chain/bsc/releases/latest).
## Audit reports
Audit reports are published in the `docs` folder: https://github.com/ethereum/go-ethereum/tree/master/docs/audits
Audit reports are published in the `docs` folder: https://github.com/bnb-chain/bsc/tree/master/docs/audits
**Please do not file a public ticket** mentioning the vulnerability.
To find out how to disclose a vulnerability in Ethereum visit [https://bounty.ethereum.org](https://bounty.ethereum.org) or email bounty@ethereum.org.
To find out how to disclose a vulnerability in Ethereum visit [https://bugcrowd.com/binance](https://bugcrowd.com/binance) or email bounty@ethereum.org. Please read the [disclosure page](https://github.com/bnb-chain/bsc/security/advisories) for more information about publicly disclosed security vulnerabilities.
The following key may be used to communicate sensitive information to developers.
Use the built-in `geth version-check` feature to check whether the software is affected by any known vulnerability. This command will fetch the latest [`vulnerabilities.json`](https://geth.ethereum.org/docs/vulnerabilities/vulnerabilities.json) file which contains known security vulnerabilities concerning `geth`, and cross-check the data against its own version number.
//lint:ignore ST1005 brand name displayed on the console
returncommon.Address{},nil,fmt.Errorf("Ledger v%d.%d.%d doesn't support signing this transaction, please update to v1.0.3 at least",w.version[0],w.version[1],w.version[2])
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.