This PR fixes a few panics in the chain marker benchmarks. The root
cause for panic is in chain marker the genesis header/block is not
accessible, while it's expected to be obtained in tests. So this PR
avoids touching genesis header at all to avoid panic.
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.
The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.
To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.
Benchmark results:
name old time/op new time/op delta
EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8)
EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8)
name old speed new speed delta
EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8)
EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal)
* feature: do trie prefetch on state prefetch
Currently, state prefetch just pre execute the transactions and discard the results.
It is helpful to increase the snapshot cache hit rate.
It would be more helpful, if it can do trie prefetch at the same time, since the it will
preload the trie node and build the trie tree in advance.
This patch is to implement it, by reusing the main trie prefetch and doing finalize after
transaction is executed.
* some code improvements for trie prefetch
** increase pendingSize before dispatch tasks
** use throwaway StateDB for TriePrefetchInAdvance and remove the prefetcherLock
** remove the necessary drain operation in trie prefetch mainloop,
trie prefetcher won't be used after close.
* trie prefetcher for From/To address in advance
We found that trie prefetch could be not fast enough, especially trie prefetch of
the outer big state trie tree.
Instead of do trie prefetch until a transaction is finalized, we could do trie prefetch
in advance. Try to prefetch the trie node of the From/To accounts, since their root hash
are most likely to be changed.
* Parallel TriePrefetch for large trie update.
Currently, we create a subfetch for each account address to do trie prefetch. If the address
has very large state change, trie prefetch could be not fast enough, e.g. a contract modified
lots of KV pair or a large number of account's root hash is changed in a block.
With this commit, there will be children subfetcher created to do trie prefetch in parallell if
the parent subfetch's workload exceed the threshold.
* some improvemnts of parallel trie prefetch implementation
1.childrenLock is removed, since it is not necessary
APIs of triePrefetcher is not thread safe, they should be used sequentially.
A prefetch will be interrupted by trie() or clos(), so we only need mark it as
interrupted and check before call scheduleParallel to avoid the concurrent access to paraChildren
2.rename subfetcher.children to subfetcher.paraChildren
3.use subfetcher.pendingSize to replace totalSize & processedIndex
4.randomly select the start child to avoid always feed the first one
5.increase threshold and capacity to avoid create too many child routine
* fix review comments
** nil check refine
** create a separate routine for From/To prefetch, avoid blocking the cirtical path
* remove the interrupt member
* not create a signer for each transaction
* some changes to triePrefetcher
** remove the abortLoop, move the subfetcher abort operation into mainLoop
since we want to make subfetcher's create & schedule & abort within a loop to
avoid concurrent access locks.
** no wait subfetcher's term signal in abort()
it could speed up the close by closing subfetcher concurrently.
we send stop signnal to all subfetchers in burst and wait their term signal later.
* some coding improve for subfetcher.scheduleParallel
* fix a UT crash of s.prefetcher == nil
* update parallel trie prefetcher configuration
tested with different combination of parallelTriePrefetchThreshold & parallelTriePrefetchCapacity,
found the most efficient configure could be:
parallelTriePrefetchThreshold = 10
parallelTriePrefetchCapacity = 20
* fix review comments: code refine
* Redesign triePrefetcher to make it thread safe
There are 2 types of triePrefetcher instances:
1.New created triePrefetcher: it is key to do trie prefetch to speed up validation phase.
2.Copied triePrefetcher: it only copy the prefetched trie information, actually it won't do
prefetch at all, the copied tries are all kept in p.fetches.
Here we try to improve the new created one, to make it concurrent safe, while the copied one's
behavior stay unchanged(its logic is very simple).
As commented in triePrefetcher struct, its APIs are not thread safe. So callers should make sure
the created triePrefetcher should be used within a single routine.
As we are trying to improve triePrefetcher, we would use it concurrently, so it is necessary to
redesign it for concurrent access.
The design is simple:
** start a mainLoop to do all the work, APIs just send channel message.
Others:
** remove the metrics copy, since it is useless for copied triePrefetcher
** for trie(), only get subfetcher through channel to reduce the workload of mainloop
* some code enhancement for triePrefetcher redesign
* some fixup: rename, temporary trie chan for concurrent safe.
* fix review comments
* add some protection in case the trie prefetcher is already stopped
* fix review comments
** make close concurrent safe
** fix potential deadlock
* replace channel by RWMutex for a few triePrefetcher APIs
For APIs like: trie(), copy(), used(), it is simpler and more efficient to
use a RWMutex instead of channel communicaton.
Since the mainLoop would be busy handling trie request, while these trie request
can be processed in parallism.
We would only keep prefetch and close within the mainLoop, since they could update
the fetchers
* add lock for subfecter.used access to make it concurrent safe
* no need to create channel for copied triePrefetcher
* fix trie_prefetcher_test.go
trie prefetcher’s behavior has changed, prefetch() won't create subfetcher immediately.
it is reasonable, but break the UT, to fix the failed UT
* cmd/geth: add db cmd to show metadata
* cmd/geth: better output generator status
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
* cmd: minor
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* freezer: add readonly flag to table
* freezer: enforce readonly in table repair
* freezer: enforce readonly in newFreezer
* minor fix
* minor
* core/rawdb: test that writing during readonly fails
* rm unused log
* check readonly on batch append
* minor
* Revert "check readonly on batch append"
This reverts commit 2ddb5ec4ba7534bf6edbdfec158ea99a2eed5036.
* review fixes
* minor test refactor
* attempt at fixing windows issue
* add comment re windows sync issue
* k->kind
* open readonly db for genesis check
Co-authored-by: Martin Holst Swende <martin@swende.se>
