* focus on performance improvement in many aspects.
1. Do BlockBody verification concurrently;
2. Do calculation of intermediate root concurrently;
3. Preload accounts before processing blocks;
4. Make the snapshot layers configurable.
5. Reuse some object to reduce GC.
add
* rlp: improve decoder stream implementation (#22858)
This commit makes various cleanup changes to rlp.Stream.
* rlp: shrink Stream struct
This removes a lot of unused padding space in Stream by reordering the
fields. The size of Stream changes from 120 bytes to 88 bytes. Stream
instances are internally cached and reused using sync.Pool, so this does
not improve performance.
* rlp: simplify list stack
The list stack kept track of the size of the current list context as
well as the current offset into it. The size had to be stored in the
stack in order to subtract it from the remaining bytes of any enclosing
list in ListEnd. It seems that this can be implemented in a simpler
way: just subtract the size from the enclosing list context in List instead.
* rlp: use atomic.Value for type cache (#22902)
All encoding/decoding operations read the type cache to find the
writer/decoder function responsible for a type. When analyzing CPU
profiles of geth during sync, I found that the use of sync.RWMutex in
cache lookups appears in the profiles. It seems we are running into
CPU cache contention problems when package rlp is heavily used
on all CPU cores during sync.
This change makes it use atomic.Value + a writer lock instead of
sync.RWMutex. In the common case where the typeinfo entry is present in
the cache, we simply fetch the map and lookup the type.
* rlp: optimize byte array handling (#22924)
This change improves the performance of encoding/decoding [N]byte.
name old time/op new time/op delta
DecodeByteArrayStruct-8 336ns ± 0% 246ns ± 0% -26.98% (p=0.000 n=9+10)
EncodeByteArrayStruct-8 225ns ± 1% 148ns ± 1% -34.12% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
DecodeByteArrayStruct-8 120B ± 0% 48B ± 0% -60.00% (p=0.000 n=10+10)
EncodeByteArrayStruct-8 0.00B 0.00B ~ (all equal)
* rlp: optimize big.Int decoding for size <= 32 bytes (#22927)
This change grows the static integer buffer in Stream to 32 bytes,
making it possible to decode 256bit integers without allocating a
temporary buffer.
In the recent commit 088da24, Stream struct size decreased from 120
bytes down to 88 bytes. This commit grows the struct to 112 bytes again,
but the size change will not degrade performance because Stream
instances are internally cached in sync.Pool.
name old time/op new time/op delta
DecodeBigInts-8 12.2µs ± 0% 8.6µs ± 4% -29.58% (p=0.000 n=9+10)
name old speed new speed delta
DecodeBigInts-8 230MB/s ± 0% 326MB/s ± 4% +42.04% (p=0.000 n=9+10)
* eth/protocols/eth, les: avoid Raw() when decoding HashOrNumber (#22841)
Getting the raw value is not necessary to decode this type, and
decoding it directly from the stream is faster.
* fix testcase
* debug no lazy
* fix can not repair
* address comments
Co-authored-by: Felix Lange <fjl@twurst.com>
* remove uneeded convertion type
* remove redundant type in composite literal
* omit explicit type where implicit
* remove unused redundant parenthesis
* remove redundant import alias duktape
Squashed from the following commits:
core/state: lazily init snapshot storage map
core/state: fix flawed meter on storage reads
core/state: make statedb/stateobjects reuse a hasher
core/blockchain, core/state: implement new trie prefetcher
core: make trie prefetcher deliver tries to statedb
core/state: refactor trie_prefetcher, export storage tries
blockchain: re-enable the next-block-prefetcher
state: remove panics in trie prefetcher
core/state/trie_prefetcher: address some review concerns
sq
This fixes some issues in crypto/signify and makes release signing work.
The archive signing step in ci.go used getenvBase64, which decodes the key data.
This is incorrect here because crypto/signify already base64-decodes the key.
* internal/build: implement signify's signing func
* Add signify to the ci utility
* fix output file format
* Add unit test for signify
* holiman's + travis' feedback
* internal/build: verify signify's output
* crypto: move signify to common dir
* use go-minisign to verify binaries
* more holiman feedback
* crypto, ci: support minisign output
* only accept one-line trusted comments
* configurable untrusted comments
* code cleanup in tests
* revert to use ed25519 from the stdlib
* bug: fix for empty untrusted comments
* write timestamp as comment if trusted comment isn't present
* rename line checker to commentHasManyLines
* crypto: added signify fuzzer (#6)
* crypto: added signify fuzzer
* stuff
* crypto: updated signify fuzzer to fuzz comments
* crypto: repro signify crashes
* rebased fuzzer on build-signify branch
* hide fuzzer behind gofuzz build flag
* extract key data inside a single function
* don't treat \r as a newline
* travis: fix signing command line
* do not use an external binary in tests
* crypto: move signify to crypto/signify
* travis: fix formatting issue
* ci: fix linter build after package move
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* added bls fuzzer
* crypto/bls12381: revert bls-changes, fixup fuzzer tests
* fuzzers: split bls fuzzing into 8 different units
* fuzzers/bls: remove (now stale) corpus
* crypto/bls12381: added blsfuzz corpus
* fuzzers/bls12381: fix the bls corpus
* fuzzers: fix oss-fuzz script
* tests/fuzzers: fixups on bls corpus
* test/fuzzers: remove leftover corpus
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* crypto/bn256: fix bn256Mul fuzzer to not hang on large input
* Update crypto/bn256/bn256_fuzz.go
Co-authored-by: ligi <ligi@ligi.de>
Co-authored-by: ligi <ligi@ligi.de>
The z == 0 check is hit whenever we Add two points with the same x1/x2
coordinate. crypto/elliptic uses the same check in their affineFromJacobian
function. This change does not affect block processing or tx signature verification
in any way, because it does not use the Add or Double methods.
* core, crypto: various allocation savings regarding tx handling
* core: reduce allocs for gas price comparison
This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees
* core: added tx.GasPriceIntCmp for comparison without allocation
adds a method to remove unneeded allocation in comparison to tx.gasPrice
* core/types: pool legacykeccak256 objects in rlpHash
rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.
* reverted some changes
* reverted some changes
* trie: use crypto.KeccakState instead of replicating code
Co-authored-by: Martin Holst Swende <martin@swende.se>
This improves error messages when the file is too short or too long.
Also rewrite the test for SaveECDSA because LoadECDSA has its own
test now.
Co-authored-by: Felix Lange <fjl@twurst.com>
This removes a bunch of weird code around the counter overflow check in
concatKDF and makes it actually work for different hash output sizes.
The overflow check worked as follows: concatKDF applies the hash function N
times, where N is roundup(kdLen, hashsize) / hashsize. N should not
overflow 32 bits because that would lead to a repetition in the KDF output.
A couple issues with the overflow check:
- It used the hash.BlockSize, which is wrong because the
block size is about the input of the hash function. Luckily, all standard
hash functions have a block size that's greater than the output size, so
concatKDF didn't crash, it just generated too much key material.
- The check used big.Int to compare against 2^32-1.
- The calculation could still overflow before reaching the check.
The new code in concatKDF doesn't check for overflow. Instead, there is a
new check on ECIESParams which ensures that params.KeyLen is < 512. This
removes any possibility of overflow.
There are a couple of miscellaneous improvements bundled in with this
change:
- The key buffer is pre-allocated instead of appending the hash output
to an initially empty slice.
- The code that uses concatKDF to derive keys is now shared between Encrypt
and Decrypt.
- There was a redundant invocation of IsOnCurve in Decrypt. This is now removed
because elliptic.Unmarshal already checks whether the input is a valid curve
point since Go 1.5.
Co-authored-by: Felix Lange <fjl@twurst.com>
* build: use golangci-lint
This changes build/ci.go to download and run golangci-lint instead
of gometalinter.
* core/state: fix unnecessary conversion
* p2p/simulations: fix lock copying (found by go vet)
* signer/core: fix unnecessary conversions
* crypto/ecies: remove unused function cmpPublic
* core/rawdb: remove unused function print
* core/state: remove unused function xTestFuzzCutter
* core/vm: disable TestWriteExpectedValues in a different way
* core/forkid: remove unused function checksum
* les: remove unused type proofsData
* cmd/utils: remove unused functions prefixedNames, prefixFor
* crypto/bn256: run goimports
* p2p/nat: fix goimports lint issue
* cmd/clef: avoid using unkeyed struct fields
* les: cancel context in testRequest
* rlp: delete unreachable code
* core: gofmt
* internal/build: simplify DownloadFile for Go 1.11 compatibility
* build: remove go test --short flag
* .travis.yml: disable build cache
* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement
* .golangci.yml: enable goconst and ineffassign linters
* build: print message when there are no lint issues
* internal/build: refactor download a bit
Most of these changes are related to the Go 1.13 changes to test binary
flag handling.
* cmd/geth: make attach tests more reliable
This makes the test wait for the endpoint to come up by polling
it instead of waiting for two seconds.
* tests: fix test binary flags for Go 1.13
Calling flag.Parse during package initialization is prohibited
as of Go 1.13 and causes test failures. Call it in TestMain instead.
* crypto/ecies: remove useless -dump flag in tests
* p2p/simulations: fix test binary flags for Go 1.13
Calling flag.Parse during package initialization is prohibited
as of Go 1.13 and causes test failures. Call it in TestMain instead.
* build: remove workaround for ./... vendor matching
This workaround was necessary for Go 1.8. The Go 1.9 release changed
the expansion rules to exclude vendored packages.
* Makefile: use relative path for GOBIN
This makes the "Run ./build/bin/..." line look nicer.
* les: fix test binary flags for Go 1.13
Calling flag.Parse during package initialization is prohibited
as of Go 1.13 and causes test failures. Call it in TestMain instead.
The precompile at 0x09 wraps the BLAKE2b F compression function:
https://tools.ietf.org/html/rfc7693#section-3.2
The precompile requires 6 inputs tightly encoded, taking exactly 213
bytes, as explained below.
- `rounds` - the number of rounds - 32-bit unsigned big-endian word
- `h` - the state vector - 8 unsigned 64-bit little-endian words
- `m` - the message block vector - 16 unsigned 64-bit little-endian words
- `t_0, t_1` - offset counters - 2 unsigned 64-bit little-endian words
- `f` - the final block indicator flag - 8-bit word
[4 bytes for rounds][64 bytes for h][128 bytes for m][8 bytes for t_0]
[8 bytes for t_1][1 byte for f]
The boolean `f` parameter is considered as `true` if set to `1`.
The boolean `f` parameter is considered as `false` if set to `0`.
All other values yield an invalid encoding of `f` error.
The precompile should compute the F function as specified in the RFC
(https://tools.ietf.org/html/rfc7693#section-3.2) and return the updated
state vector `h` with unchanged encoding (little-endian).
See EIP-152 for details.
* Removed comment section referring to Cloudflare's bn curve parameters
* Added comment to clarify the nature of the parameters
* Changed value of xi to i+9
Package crypto works with or without cgo, which is great. However, to make it
work without cgo required setting the build tag `nocgo`. It's common to disable
cgo by instead just setting the environment variable `CGO_ENABLED=0`. Setting
this environment variable does _not_ implicitly set the build tag `nocgo`. So
projects that try to build the crypto package with `CGO_ENABLED=0` will fail. I
have done this myself several times. Until today, I had just assumed that this
meant that this package requires cgo.
But a small build tag change will make this case work. Instead of using `nocgo`
and `!nocgo`, we can use `!cgo` and `cgo`, respectively. The `cgo` build tag is
automatically set if cgo is enabled, and unset if it is disabled.
