* offline block prune
* update
* update
* update and add unit test
* addressed comments from walt
* Addressed comments from walt and Igor
* ensure MPT and snapshot matched
* add one more parameter to indicate blockprune
* update the logic of creating freezerDb
* update flag command description
* expose the function for db inspect the offset/startBlockNumber
* add flags to inspect prune info
* rename flag of reserved-recent-blocks to block-amount-reserved
* addressed comments from walt
* handle the case of command interruption
* refined goimports
* addressed comments from walt
* change the logic as restarting prune after interruption
* addressed comments
* reclaimed freezer logic
* introduce flag to enable/disable check between MPT and snapshot
* update the logic of frozen field in freezerDB
* update the code in all places related to freezer change
* addressed comments from dylan
* update the logic for backup block difficulty
* addressed comments from dylan
* eth/tracers: implement debug.intermediateRoots (#23594)
This PR implements a new debug method, which I've talked briefly about to some other client developers. It allows the caller to obtain the intermediate state roots for a block (which might be either a canon block or a 'bad' block).
Signed-off-by: wenbiao <delweng@gmail.com>
* core, rpc: disable memory output by default in traces (#23558)
* core: cmd: invert disableMemory
* core: fix missed inversion
* cmd/evm: preserve Flags but change default value
* Apply suggestions from code review
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: abort evm execution when trace is aborted (#23580)
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: avoid unsyncronized mutations on trie database (#23632)
This PR fixes an issue in traceChain, where the statedb Commit operation was performed asynchronously with dereference-operations agains the underlying trie.Database instance. Due to how the reference counting works within the trie database (where parent count is recursively updated when new parents are added), doing dereferencing in the middle of Commit can cause the refcount to become wrong, leading to an inconsistent state.
This was fixed by doing Commit/Deref from the same routine.
Signed-off-by: wenbiao <delweng@gmail.com>
* core,eth: call frame tracing (#23087)
This change introduces 2 new optional methods; `enter()` and `exit()` for js tracers, and makes `step()` optiona. The two new methods are invoked when entering and exiting a call frame (but not invoked for the outermost scope, which has it's own methods). Currently these are the data fields passed to each of them:
enter: type (opcode), from, to, input, gas, value
exit: output, gasUsed, error
The PR also comes with a re-write of the callTracer. As a backup we keep the previous tracing script under the name `callTracerLegacy`. Behaviour of both tracers are equivalent for the most part, although there are some small differences (improvements), where the new tracer is more correct / has more information.
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: re-write of 4byte tracer using enter/exit (#23622)
* eth/tracers: add re-write of 4byte tracer using enter/exit
* eth/tracers: fix 4byte indent
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: tx.BaseFee not implemented
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: do the JSON serialization via .js to capture C faults
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: fix callTracer fault handling (#23667)
* eth/tracers: fix calltracer fault handling
* eth/tracers: fix calltracer indentation
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: invoke enter/exit on 0-value calls to inex accounts (#23828)
Signed-off-by: wenbiao <delweng@gmail.com>
* eth: make traceChain avoid OOM on long-running tracing (#23736)
This PR changes long-running chain tracing, so that it at some points releases the memory trie db, and switch over to a fresh disk-backed trie.
Signed-off-by: wenbiao <delweng@gmail.com>
* eth/tracers: expose contextual infos (block hash, tx hash, tx index)
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: redefine Context
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: support for golang tracers + add golang callTracer (#23708)
* eth/tracers: add basic native loader
* eth/tracers: add GetResult to tracer interface
* eth/tracers: add native call tracer
* eth/tracers: fix call tracer json result
* eth/tracers: minor fix
* eth/tracers: fix
* eth/tracers: fix benchTracer
* eth/tracers: test native call tracer
* eth/tracers: fix
* eth/tracers: rm extra make
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: rm extra make
* eth/tracers: make callFrame private
* eth/tracers: clean-up and comments
* eth/tracers: add license
* eth/tracers: rework the model a bit
* eth/tracers: move tracecall tests to subpackage
* cmd/geth: load native tracers
* eth/tracers: minor fix
* eth/tracers: impl stop
* eth/tracers: add native noop tracer
* renamings
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/tracers: more renamings
* eth/tracers: make jstracer non-exported, avoid cast
* eth/tracers, core/vm: rename vm.Tracer to vm.EVMLogger for clarity
* eth/tracers: minor comment fix
* eth/tracers/testing: lint nitpicks
* core,eth: cancel evm on nativecalltracer stop
* Revert "core,eth: cancel evm on nativecalltracer stop"
This reverts commit 01bb908790a369c1bb9d3937df9325c6857bf855.
* eth/tracers: linter nits
* eth/tracers: fix output on err
Co-authored-by: Martin Holst Swende <martin@swende.se>
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: make native calltracer default (#23867)
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: package restructuring (#23857)
* eth/tracers: restructure tracer package
* core/vm/runtime: load js tracers
* eth/tracers: mv bigint js code to own file
* eth/tracers: add method docs for native tracers
* eth/tracers: minor doc fix
* core,eth: cancel evm on nativecalltracer stop
* core/vm: fix failing test
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: ethapi.TransactionArgs was not merged
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: fix the api_test with ErrInsufficientFunds to ErrInsufficientFundsForTransfer
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: check posa before statedb.Prepare in IntermiateRoots api
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: make js calltracer default, compatible with old version
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: fix the default callTrace name of callTracerJs
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* Revert "eth/tracers: fix the default callTrace name of callTracerJs"
This reverts commit 62a3bc215d9f07e422a4c659289bb3ba4f9ed2fa.
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* Revert "eth/tracers: make js calltracer default, compatible with old version"
This reverts commit 85ef42c0ea651f0b228d4209b1b2598b24e12f1f.
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
* eth/tracers: fix the variable race condition
Signed-off-by: wenbiao <wenbiao.zheng@ambergroup.io>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* reannouce local pending transactions
* add tests for tx_pool reannouce local pending transactions
* add tests for handler reannounce local pending transactions
* focus on performance improvement in many aspects.
1. Do BlockBody verification concurrently;
2. Do calculation of intermediate root concurrently;
3. Preload accounts before processing blocks;
4. Make the snapshot layers configurable.
5. Reuse some object to reduce GC.
add
* rlp: improve decoder stream implementation (#22858)
This commit makes various cleanup changes to rlp.Stream.
* rlp: shrink Stream struct
This removes a lot of unused padding space in Stream by reordering the
fields. The size of Stream changes from 120 bytes to 88 bytes. Stream
instances are internally cached and reused using sync.Pool, so this does
not improve performance.
* rlp: simplify list stack
The list stack kept track of the size of the current list context as
well as the current offset into it. The size had to be stored in the
stack in order to subtract it from the remaining bytes of any enclosing
list in ListEnd. It seems that this can be implemented in a simpler
way: just subtract the size from the enclosing list context in List instead.
* rlp: use atomic.Value for type cache (#22902)
All encoding/decoding operations read the type cache to find the
writer/decoder function responsible for a type. When analyzing CPU
profiles of geth during sync, I found that the use of sync.RWMutex in
cache lookups appears in the profiles. It seems we are running into
CPU cache contention problems when package rlp is heavily used
on all CPU cores during sync.
This change makes it use atomic.Value + a writer lock instead of
sync.RWMutex. In the common case where the typeinfo entry is present in
the cache, we simply fetch the map and lookup the type.
* rlp: optimize byte array handling (#22924)
This change improves the performance of encoding/decoding [N]byte.
name old time/op new time/op delta
DecodeByteArrayStruct-8 336ns ± 0% 246ns ± 0% -26.98% (p=0.000 n=9+10)
EncodeByteArrayStruct-8 225ns ± 1% 148ns ± 1% -34.12% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
DecodeByteArrayStruct-8 120B ± 0% 48B ± 0% -60.00% (p=0.000 n=10+10)
EncodeByteArrayStruct-8 0.00B 0.00B ~ (all equal)
* rlp: optimize big.Int decoding for size <= 32 bytes (#22927)
This change grows the static integer buffer in Stream to 32 bytes,
making it possible to decode 256bit integers without allocating a
temporary buffer.
In the recent commit 088da24, Stream struct size decreased from 120
bytes down to 88 bytes. This commit grows the struct to 112 bytes again,
but the size change will not degrade performance because Stream
instances are internally cached in sync.Pool.
name old time/op new time/op delta
DecodeBigInts-8 12.2µs ± 0% 8.6µs ± 4% -29.58% (p=0.000 n=9+10)
name old speed new speed delta
DecodeBigInts-8 230MB/s ± 0% 326MB/s ± 4% +42.04% (p=0.000 n=9+10)
* eth/protocols/eth, les: avoid Raw() when decoding HashOrNumber (#22841)
Getting the raw value is not necessary to decode this type, and
decoding it directly from the stream is faster.
* fix testcase
* debug no lazy
* fix can not repair
* address comments
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the SimultaneousRequests test to send the requests from the same
connection, as it doesn't really make sense to test whether a node can respond
to two requests with different request IDs from separate connections.
This removes auto-configuration of the snap.*.ethdisco.net DNS discovery tree.
Since measurements have shown that > 75% of nodes in all.*.ethdisco.net support
snap, we have decided to retire the dedicated index for snap and just use the eth
tree instead.
The dial iterators of eth and snap now use the same DNS tree in the default configuration,
so both iterators should use the same DNS discovery client instance. This ensures that
the record cache and rate limit are shared. Records will not be requested multiple times.
While testing the change, I noticed that duplicate DNS requests do happen even
when the client instance is shared. This is because the two iterators request the tree
root, link tree root, and first levels of the tree in lockstep. To avoid this problem, the
change also adds a singleflight.Group instance in the client. When one iterator
attempts to resolve an entry which is already being resolved, the singleflight object
waits for the existing resolve call to finish and returns the entry to both places.
The new -limit option makes the filter operate on top N nodes by score.
This also adds ENR attribute stats in the nodeset info command.
Node set commands are now documented in README.
This change adds the --catalyst flag, enabling an RPC API for eth2 integration.
In this initial version, catalyst mode also disables all peer-to-peer networking.
Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
* all: add thousandths separators for big numbers on log messages
* p2p/sentry: drop accidental file
* common, log: add fast number formatter
* common, eth/protocols/snap: simplifty fancy num types
* log: handle nil big ints
This upgrades the cloudflare client dependency to v0.14.0. The new
version changes the API because all methods now require a context
parameter. This change also reduces the log level of the 'Skipping...'
message to debug, following a similar change in the AWS deployer.
The PR implements the --miner.notify.full flag that enables full pending block
notifications. When this flag is used, the block notifications sent to mining
endpoints contain the complete block header JSON instead of a work package
array.
Co-authored-by: AlexSSD7 <alexandersadovskyi7@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/devp2p: fix comparison of TXT record value
The AWS API returns quoted DNS strings, so we must encode the new value
before comparing it against the existing record content.
* cmd/devp2p: add test
* cmd/devp2p: fix typo and rename val -> newValue
In Geth v1.10, we changed the structure of the "les" ENR entry. As a result, the DHT crawler that creates the DNS lists
no longer recognizes the les nodes, which is fixed in this commit.
* cmd/devp2p: skip ENR field tails properly in nodeset filter
* cmd/devp2p: fix tail decoder for snap as well
* les: fix tail decoding in "eth" ENR entry
This PR fixes a regression introduced in #22360, when we updated to the v2 of the AWS sdk, which causes current crawler to just get the same first 100 results over and over, and get stuck in a loop.
This updates the DNS deployer to use AWS SDK v2. Migration is relatively
seamless, although there were two locations that required a slightly
different approach to achieve the same results. In particular, waiting for
DNS change propagation is very different with SDK v2.
This change also optimizes DNS updates by publishing all changes before
waiting for propagation.
This PR fixes an issue with the eth66 test suite where, during a readAndServe when
the test is manually responding to GetBlockHeader requests, it now responds
with a BlockHeaders eth66 packet that includes the inbound request ID.
