Commit Graph

1956 Commits

Author SHA1 Message Date
aaronbuchwald
56f533d00c
docs: fix docstring on read head block (#22776) 2021-04-29 19:23:07 +02:00
Péter Szilágyi
fae165a5de
core, eth, ethdb, trie: simplify range proofs 2021-04-29 10:59:08 +03:00
Martin Holst Swende
a81cf0d2b3
trie: remove redundant returns + use stacktrie where applicable (#22760)
* trie: add benchmark for proofless range

* trie: remove unused returns + use stacktrie
2021-04-28 22:47:48 +03:00
Péter Szilágyi
caea6c4661
eth/protocols/snap: generate storage trie from full dirty snap data (#22668)
* eth/protocols/snap: generate storage trie from full dirty snap data

* eth/protocols/snap: get rid of some more dead code

* eth/protocols/snap: less frequent logs, also log during trie generation

* eth/protocols/snap: implement dirty account range stack-hashing

* eth/protocols/snap: don't loop on account trie generation

* eth/protocols/snap: fix account format in trie

* core, eth, ethdb: glue snap packets together, but not chunks

* eth/protocols/snap: print completion log for snap phase

* eth/protocols/snap: extended tests

* eth/protocols/snap: make testcase pass

* eth/protocols/snap: fix account stacktrie commit without defer

* ethdb: fix key counts on reset

* eth/protocols: fix typos

* eth/protocols/snap: make better use of delivered data (#44)

* eth/protocols/snap: make better use of delivered data

* squashme

* eth/protocols/snap: reduce chunking

* squashme

* eth/protocols/snap: reduce chunking further

* eth/protocols/snap: break out hash range calculations

* eth/protocols/snap: use sort.Search instead of looping

* eth/protocols/snap: prevent crash on storage response with no keys

* eth/protocols/snap: nitpicks all around

* eth/protocols/snap: clear heal need on 1-chunk storage completion

* eth/protocols/snap: fix range chunker, add tests

Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* trie: fix test API error

* eth/protocols/snap: fix some further liter issues

* eth/protocols/snap: fix accidental batch reuse

Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-04-27 17:19:59 +03:00
Martin Holst Swende
65a1c2d829
core/vm: make gas cost reporting to tracers correct (#22702)
Previously, the makeCallVariantGasCallEIP2929 charged the cold account access cost directly, leading to an incorrect gas cost passed to the tracer from the main execution loop.
This change still temporarily charges the cost (to allow for an accurate calculation of the available gas for the call), but then afterwards refunds it and instead returns the correct total gas cost to be then properly charged in the main loop.
2021-04-27 13:21:41 +02:00
Martin Holst Swende
9b99e3dfe0
core/rawdb: fix datarace in freezer (#22728)
The Append / truncate operations were racy. When a datafile reaches 2Gb, a new file is needed. For this operation, we require a writelock, which is not needed in the 99.99% of all cases where the data does fit in the current head-file.

This transition from readlock to writelock was incorrect, and as the readlock was released, a truncate operation could slip in between, and truncate the data. This would have been fine, however, the Append operation continued writing as if no truncation had occurred, e.g writing item 5 where item 0 should reside.

This PR changes the behaviour, so that if when we run into the situation that a new file is needed, it aborts, and retries, this time with a writelock.

The outcome of the situation described above, running on this PR, would instead be that the Append operation exits with a failure.
2021-04-26 18:19:07 +02:00
Martin Holst Swende
83375b0873
core: remove old conversion to shuffle leveldb blocks into ancients 2021-04-26 14:27:56 +02:00
Martin Holst Swende
49281ab84f
core/state/snapshot, true: reuse dirty data instead of hitting disk when generating (#22667)
* core/state/snapshot: reuse memory data instead of hitting disk when generating

* trie: minor nitpicks wrt the resolver optimization

* core/state/snapshot, trie: use key/value store for resolver

* trie: fix linter

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-23 14:39:18 +03:00
gary rong
d6ffa14035
core: nuke legacy snapshot supporting (#22663) 2021-04-20 08:27:46 +03:00
Péter Szilágyi
e43ac53264
Merge pull request #22686 from holiman/minor_fixes
core/state/snapshot: avoid copybytes for stacktrie
2021-04-18 17:53:01 +03:00
Guillaume Ballet
f79cce5de9
eth/catalyst: add catalyst API prototype (#22641)
This change adds the --catalyst flag, enabling an RPC API for eth2 integration.
In this initial version, catalyst mode also disables all peer-to-peer networking.

Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-04-16 21:29:22 +02:00
Martin Holst Swende
09d44e9925
core/state/snapshot: avoid copybytes for stacktrie 2021-04-16 14:58:23 +02:00
meowsbits
3cfd0fe7a8
core: add TestGenesisHashes and fix YoloV3 (#22559)
This adds simple unit test checking if the hard-coded genesis
hash values in package params match the actual genesis block hashes.
2021-04-16 00:32:16 +02:00
Péter Szilágyi
1e207342b5
all: make logs a bit easier on the eye to digest (#22665)
* all: add thousandths separators for big numbers on log messages

* p2p/sentry: drop accidental file

* common, log: add fast number formatter

* common, eth/protocols/snap: simplifty fancy num types

* log: handle nil big ints
2021-04-15 20:35:00 +03:00
Péter Szilágyi
d5e57948d1
core/types: drop some relice data types 2021-04-14 23:39:42 +03:00
gary rong
7088f1e814
core, eth: faster snapshot generation (#22504)
* eth/protocols: persist received state segments

* core: initial implementation

* core/state/snapshot: add tests

* core, eth: updates

* eth/protocols/snapshot: count flat state size

* core/state: add metrics

* core/state/snapshot: skip unnecessary deletion

* core/state/snapshot: rename

* core/state/snapshot: use the global batch

* core/state/snapshot: add logs and fix wiping

* core/state/snapshot: fix

* core/state/snapshot: save generation progress even if the batch is empty

* core/state/snapshot: fixes

* core/state/snapshot: fix initial account range length

* core/state/snapshot: fix initial account range

* eth/protocols/snap: store flat states during the healing

* eth/protocols/snap: print logs

* core/state/snapshot: refactor (#4)

* core/state/snapshot: refactor

* core/state/snapshot: tiny fix and polish

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* core, eth: fixes

* core, eth: fix healing writer

* core, trie, eth: fix paths

* eth/protocols/snap: fix encoding

* eth, core: add debug log

* core/state/generate: release iterator asap (#5)

core/state/snapshot: less copy

core/state/snapshot: revert split loop

core/state/snapshot: handle storage becoming empty, improve test robustness

core/state: test modified codehash

core/state/snapshot: polish

* core/state/snapshot: optimize stats counter

* core, eth: add metric

* core/state/snapshot: update comments

* core/state/snapshot: improve tests

* core/state/snapshot: replace secure trie with standard trie

* core/state/snapshot: wrap return as the struct

* core/state/snapshot: skip wiping correct states

* core/state/snapshot: updates

* core/state/snapshot: fixes

* core/state/snapshot: fix panic due to reference flaw in closure

* core/state/snapshot: fix errors in state generation logic + fix log output

* core/state/snapshot: remove an error case

* core/state/snapshot: fix condition-check for exhausted snap state

* core/state/snapshot: use stackTrie for small tries

* core/state/snapshot: don't resolve small storage tries in vain

* core/state/snapshot: properly clean up storage of deleted accounts

* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks

* core/state/snapshot: fix error (+testcase)

* core/state/snapshot: clean up tests a bit

* core/state/snapshot: work in progress on better tests

* core/state/snapshot: polish code

* core/state/snapshot: fix trie iteration abortion trigger

* core/state/snapshot: fixes flaws

* core/state/snapshot: remove panic

* core/state/snapshot: fix abort

* core/state/snapshot: more tests (plus failing testcase)

* core/state/snapshot: more testcases + fix for failing test

* core/state/snapshot: testcase for malformed data

* core/state/snapshot: some test nitpicks

* core/state/snapshot: improvements to logging

* core/state/snapshot: testcase to demo error in abortion

* core/state/snapshot: fix abortion

* cmd/geth: make verify-state report the root

* trie: fix failing test

* core/state/snapshot: add timer metrics

* core/state/snapshot: fix metrics

* core/state/snapshot: udpate tests

* eth/protocols/snap: write snapshot account even if code or state is needed

* core/state/snapshot: fix diskmore check

* core/state/snapshot: review fixes

* core/state/snapshot: improve error message

* cmd/geth: rename 'error' to 'err' in logs

* core/state/snapshot: fix some review concerns

* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync

* core: add error log

* core/state/snapshot: use proper timers for metrics collection

* core/state/snapshot: address some review concerns

* eth/protocols/snap: improved log message

* eth/protocols/snap: fix heal logs to condense infos

* core/state/snapshot: wait for generator termination before restarting

* core/state/snapshot: revert timers to counters to track total time

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
Martin Holst Swende
271e5b7fc9
cmd/geth: add db-command to inspect freezer index (#22633)
This PR makes it easier to inspect the freezer index, which could be useful to investigate things like #22111
2021-04-13 15:45:30 +02:00
Marius van der Wijden
9d10856e84
core, eth, internal/ethapi: create access list RPC API (#22550)
* core/vm: implement AccessListTracer

* eth: implement debug.createAccessList

* core/vm: fixed nil panics in accessListTracer

* eth: better error messages for createAccessList

* eth: some fixes on CreateAccessList

* eth: allow for provided accesslists

* eth: pass accesslist by value

* eth: remove created acocunt from accesslist

* core/vm: simplify access list tracer

* core/vm: unexport accessListTracer

* eth: return best guess if al iteration times out

* eth: return best guess if al iteration times out

* core: docstring, unexport methods

* eth: typo

* internal/ethapi: move createAccessList to eth package

* internal/ethapi: remove reexec from createAccessList

* internal/ethapi: break if al is equal to last run, not if gas is equal

* internal/web3ext: fixed arguments

* core/types: fixed equality check for accesslist

* core/types: no hardcoded vals

* core, internal: simplify access list generation, make it precise

* core/vm: fix typo

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-07 17:54:31 +03:00
Edgar Aroutiounian
c79fc209cd
core/state/snapshot: fix data race in diff layer (#22540) 2021-04-06 11:57:00 +03:00
Péter Szilágyi
61ff3e86b2
core/state/snapshot, ethdb: track deletions more accurately (#22582)
* core/state/snapshot, ethdb: track deletions more accurately

* core/state/snapshot: don't reset the iterator, leveldb's screwy

* ethdb: don't mess with the insert batches for now
2021-03-30 19:04:22 +03:00
nebojsa94
44fe466999
core/vm: fix Byzantium address list (#22603) 2021-03-30 16:38:53 +03:00
rene
b6912c1047
core: add BlockGen.GetBalance method (#22589) 2021-03-29 20:54:49 +02:00
gary rong
76700ac892
core/state/pruner: move the compaction out of the pruning procedure (#22579)
The main idea behind it is: the range compaction is very expensive
which can take a few hours to finish. During this long procedure,
a lot of exceptions can occur, e.g.
- Geth is killed manually
- Geth is killed because of machine crash
- etc

In order to minimize the effect of the exceptions, the compaction
is moved out of the pruning. So that even the compaction is not
finished, the pruning is regarded as done.
2021-03-29 11:09:29 +02:00
Martin Holst Swende
497448bf90
core: fix condition on header verification 2021-03-25 11:50:14 +01:00
Martin Holst Swende
0fda25e471
eth/tracers, core: use scopecontext in tracers, provide statedb in capturestart (#22333)
Fixes the CaptureStart api to include the EVM, thus being able to set the statedb early on. This pr also exposes the struct we used internally in the interpreter to encapsulate the contract, mem, stack, rstack, so we pass it as a single struct to the tracer, and removes the error returns on the capture methods.
2021-03-25 10:13:14 +01:00
Péter Szilágyi
a31f6d54df
core/state/snapshot: fix panic on missing parent 2021-03-22 22:41:28 +02:00
gary rong
0c70b83e00
all: add read-only option to database (#22407)
* all: add read-only option to database

* all: fixes tests

* cmd/geth: migrate flags

* cmd/geth: fix the compact

* cmd/geth: fix the format

* cmd/geth: fix log

* cmd: add chain-readonly

* core: add readonly notion to freezer

* core/rawdb: add log

* core/rawdb: fix freezer close

* cmd: fix

* cmd, core: construct db

* core: update tests
2021-03-22 19:06:30 +01:00
Derek Chiang
eaccdba4ab
core: fix method comment for txpool.requestReset (#22543) 2021-03-22 09:10:51 +02:00
Sina Mahmoodi
c454717fa6
core: fix potential race in chainIndexerTest (#22346) 2021-03-19 13:32:57 +01:00
Martin Holst Swende
94ab4ea341
core/rawdb: fix transaction indexing/unindexing hashing error (#22457)
* core/rawdb: more verbose error logs + better hashing

* core/rawdb: add failing testcase

* core/rawdb: properly hash transactions while indexing/unindexing

* core/rawdb: exit on error + better log msg
2021-03-16 12:15:14 +02:00
lightclient
99830720f6
core/types: improve comments in new EIP-2718 code (#22402)
Responding to these comments:

    https://github.com/ethereum/go-ethereum/pull/21502/files#r579010962
    https://github.com/ethereum/go-ethereum/pull/21502/files#r579021565
    https://github.com/ethereum/go-ethereum/pull/21502/files#r579023510
    https://github.com/ethereum/go-ethereum/pull/21502/files#r578983734
2021-03-16 09:48:54 +01:00
Marius van der Wijden
be87f769f6
core/types: reduce allocations in GasPriceCmp (#22456) 2021-03-08 15:23:28 +02:00
Péter Szilágyi
44c0bb2b44
Merge pull request #22426 from ethereum/revert-22413-unship-berlin
Revert "core/forkid, params: unset Berlin fork number"
2021-03-07 19:39:11 +02:00
Péter Szilágyi
658cb9fc4f
Merge pull request #22414 from karalabe/unship-2315
core, eth: unship EIP 2315
2021-03-07 19:38:49 +02:00
Péter Szilágyi
9230ca4924 Revert "core/forkid, params: unset Berlin fork number (#22413)"
This reverts commit ba999105ef89473cfe39e5e53354f7099e67a290.
2021-03-04 10:44:37 +02:00
Péter Szilágyi
ba999105ef
core/forkid, params: unset Berlin fork number (#22413) 2021-03-03 12:05:27 +02:00
Péter Szilágyi
430f69e01e
core/vm/runtime: more unshipping 2021-03-02 23:51:03 +02:00
Péter Szilágyi
7834e4a278
core, eth: unship EIP 2315 2021-03-02 23:40:57 +02:00
gary rong
19d7a37abb
core/rawdb: fix the transaction indexer (#22395) 2021-03-01 11:26:10 +02:00
Martin Holst Swende
498458b410
core/state: fix eta calculation on pruning (#22386) 2021-02-26 16:33:37 +01:00
Péter Szilágyi
0928562670
all: define Berlin hard fork spec 2021-02-26 14:24:07 +02:00
lightclient
bbfb1e4008
all: add support for EIP-2718, EIP-2930 transactions (#21502)
This adds support for EIP-2718 typed transactions as well as EIP-2930
access list transactions (tx type 1). These EIPs are scheduled for the
Berlin fork.

There very few changes to existing APIs in core/types, and several new APIs
to deal with access list transactions. In particular, there are two new
constructor functions for transactions: types.NewTx and types.SignNewTx.
Since the canonical encoding of typed transactions is not RLP-compatible,
Transaction now has new methods for encoding and decoding: MarshalBinary
and UnmarshalBinary.

The existing EIP-155 signer does not support the new transaction types.
All code dealing with transaction signatures should be updated to use the
newer EIP-2930 signer. To make this easier for future updates, we have
added new constructor functions for types.Signer: types.LatestSigner and
types.LatestSignerForChainID. 

This change also adds support for the YoloV3 testnet.

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
2021-02-25 15:26:57 +01:00
Martin Holst Swende
c4a2b682ff
cmd/geth: add db commands stats, compact, put, get, delete (#22014)
This PR introduces:

- db.put to put a value into the database
- db.get to read a value from the database
- db.delete to delete a value from the database
- db.stats to check compaction info from the database
- db.compact to trigger a db compaction

It also moves inspectdb to db.inspect.
2021-02-23 11:27:32 +01:00
Martin Holst Swende
52e5c38aa5
core/state: copy the snap when copying the state (#22340)
* core/state: copy the snap when copying the state

* core/state: deep-copy snap stuff during state Copy
2021-02-18 10:05:47 +02:00
Péter Szilágyi
9ec3329899
core/state/snapshot: ensure Cap retains a min number of layers 2021-02-16 15:25:20 +02:00
Péter Szilágyi
ef227c5f42
core: fix temp memory blowup caused by defers holding on to state 2021-02-12 12:45:34 +02:00
Martin Holst Swende
74dbc20260
core/state/pruner: fix compaction range error 2021-02-08 20:31:52 +01:00
Péter Szilágyi
2728672c28
core/state/pruner: fix compaction after pruning 2021-02-08 19:18:40 +02:00
gary rong
f566dd305e
all: bloom-filter based pruning mechanism (#21724)
* cmd, core, tests: initial state pruner

core: fix db inspector

cmd/geth: add verify-state

cmd/geth: add verification tool

core/rawdb: implement flatdb

cmd, core: fix rebase

core/state: use new contract code layout

core/state/pruner: avoid deleting genesis state

cmd/geth: add helper function

core, cmd: fix extract genesis

core: minor fixes

contracts: remove useless

core/state/snapshot: plugin stacktrie

core: polish

core/state/snapshot: iterate storage concurrently

core/state/snapshot: fix iteration

core: add comments

core/state/snapshot: polish code

core/state: polish

core/state/snapshot: rebase

core/rawdb: add comments

core/rawdb: fix tests

core/rawdb: improve tests

core/state/snapshot: fix concurrent iteration

core/state: run pruning during the recovery

core, trie: implement martin's idea

core, eth: delete flatdb and polish pruner

trie: fix import

core/state/pruner: add log

core/state/pruner: fix issues

core/state/pruner: don't read back

core/state/pruner: fix contract code write

core/state/pruner: check root node presence

cmd, core: polish log

core/state: use HEAD-127 as the target

core/state/snapshot: improve tests

cmd/geth: fix verification tool

cmd/geth: use HEAD as the verification default target

all: replace the bloomfilter with martin's fork

cmd, core: polish code

core, cmd: forcibly delete state root

core/state/pruner: add hash64

core/state/pruner: fix blacklist

core/state: remove blacklist

cmd, core: delete trie clean cache before pruning

cmd, core: fix lint

cmd, core: fix rebase

core/state: fix the special case for clique networks

core/state/snapshot: remove useless code

core/state/pruner: capping the snapshot after pruning

cmd, core, eth: fixes

core/rawdb: update db inspector

cmd/geth: polish code

core/state/pruner: fsync bloom filter

cmd, core: print warning log

core/state/pruner: adjust the parameters for bloom filter

cmd, core: create the bloom filter by size

core: polish

core/state/pruner: sanitize invalid bloomfilter size

cmd: address comments

cmd/geth: address comments

cmd/geth: address comment

core/state/pruner: address comments

core/state/pruner: rename homedir to datadir

cmd, core: address comments

core/state/pruner: address comment

core/state: address comments

core, cmd, tests: address comments

core: address comments

core/state/pruner: release the iterator after each commit

core/state/pruner: improve pruner

cmd, core: adjust bloom paramters

core/state/pruner: fix lint

core/state/pruner: fix tests

core: fix rebase

core/state/pruner: remove atomic rename

core/state/pruner: address comments

all: run go mod tidy

core/state/pruner: avoid false-positive for the middle state roots

core/state/pruner: add checks for middle roots

cmd/geth: replace crit with error

* core/state/pruner: fix lint

* core: drop legacy bloom filter

* core/state/snapshot: improve pruner

* core/state/snapshot: polish concurrent logs to report ETA vs. hashes

* core/state/pruner: add progress report for pruning and compaction too

* core: fix snapshot test API

* core/state: fix some pruning logs

* core/state/pruner: support recovering from bloom flush fail

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 13:16:30 +02:00
gary rong
098a2b6e26
eth: move eth.Config to a common package (#22205)
This moves the eth config definition into a separate package, eth/ethconfig. 
Packages eth and les can now import this common package instead of
importing eth from les, reducing dependencies.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-02-05 13:51:15 +01:00
Marius van der Wijden
3512b41c5c
core: reset txpool state on sethead (#22247)
fixes an issue where local transactions that were included in the chain before a SetHead were rejected if resubmitted, since the txpool had not reset the state to the current (older) state.
2021-02-03 11:02:35 +01:00
ucwong
83e4c49e2b
trie : use trie.NewStackTrie instead of new(trie.Trie) (#22246)
The PR makes use of the stacktrie, which is is more lenient on resource consumption, than the regular trie, in cases where we only need it for DeriveSha
2021-02-02 13:09:23 +01:00
Alex Prut
ef84da8481
all: remove unneeded parentheses (#21921)
* remove uneeded convertion type

* remove redundant type in composite literal

* omit explicit type where implicit

* remove unused redundant parenthesis

* remove redundant import alias duktape
2021-02-02 11:32:44 +02:00
Sina Mahmoodi
eb21c652c0
cmd,core,eth,params,tests: define yolov3 + enable EIP-2565 (#22213)
Removes the yolov2 definition, adds yolov3, including EIP-2565. This PR also disables some of the erroneously generated blockchain and statetests, and adds the new genesis hash + alloc for yolov3. 
This PR disables the CLI switches for yolo, since it's not complete until we merge support for 2930.
2021-01-28 21:19:07 +01:00
Martin Holst Swende
681618275c
core: speed up header import (#21967)
This PR implements the following modifications

- Don't shortcut check if block is present, thus avoid disk lookup
- Don't check hash ancestry in early-check (it's still done in parallel checker)
- Don't check time.Now for every single header

Charts and background info can be found here: https://github.com/holiman/headerimport/blob/main/README.md
With these changes, writing 1M headers goes down to from 80s to 62s.
2021-01-26 12:17:11 +01:00
Martin Holst Swende
14d495491d
core/state: fix panic in state dumping (#22225) 2021-01-26 12:15:31 +01:00
gary rong
49cdcf5c70
core: reset to genesis when middle block is missing (#22135)
When a sethead/rewind finds that the targeted block is missing, it resets to genesis instead of crashing. Closes #22129
2021-01-25 14:29:45 +01:00
Melvin Junhee Woo
04a72260c5
snapshot: merge loops for better performance (#22160) 2021-01-25 14:25:55 +01:00
Péter Szilágyi
ddadc3d273
Merge pull request #21047 from holiman/improve_updates_2
core: improve trie updates (part 2)
2021-01-21 01:48:08 +02:00
Péter Szilágyi
42f9f1f073
core/state: convert prefetcher to concurrent per-trie loader 2021-01-21 01:47:14 +02:00
Martin Holst Swende
1e1865b73f
core: implement background trie prefetcher
Squashed from the following commits:

core/state: lazily init snapshot storage map
core/state: fix flawed meter on storage reads
core/state: make statedb/stateobjects reuse a hasher
core/blockchain, core/state: implement new trie prefetcher
core: make trie prefetcher deliver tries to statedb
core/state: refactor trie_prefetcher, export storage tries
blockchain: re-enable the next-block-prefetcher
state: remove panics in trie prefetcher
core/state/trie_prefetcher: address some review concerns

sq
2021-01-21 01:46:38 +02:00
gary rong
5e9f5ca5d3
core/state/snapshot: write snapshot generator in batch (#22163)
* core/state/snapshot: write snapshot generator in batch

* core: refactor the tests

* core: update tests

* core: update tests
2021-01-18 14:39:43 +01:00
Péter Szilágyi
c4deebbf1e
core/state/snapshot: add generation logs to storage too 2021-01-15 12:26:46 +02:00
Martin Holst Swende
93a89b2681
go.mod: use github.com/holiman/bloomfilter/v2 (#22044)
* deps: use improved bloom filter implementation

* eth/handler, trie: use 4 keys for syncbloom + minor fixes

* eth/protocols, trie: revert change on syncbloom method signature
2021-01-12 17:39:31 +01:00
gary rong
5a1b384352
core: persist bad blocks (#21827)
* core: persist bad blocks

* core, eth, internal: address comments

* core/rawdb: add badblocks to inspector

* core, eth: update

* internal: revert

* core, eth: only save 10 bad blocks

* core/rawdb: address comments

* core/rawdb: fix

* core: address comments
2021-01-10 12:54:15 +01:00
Melvin Junhee Woo
d2e1b17f18
snapshot, trie: fixed typos, mostly in snapshot pkg (#22133) 2021-01-07 08:36:21 +02:00
Melvin Junhee Woo
167ff563d1
core/state/snapshot: gethring -> gathering typo (#22104) 2021-01-04 10:07:43 +02:00
ucwong
a425a47ddc
core/rawdb, eth/protocols : Method name typo fix (#22026) 2020-12-27 22:38:16 +01:00
Martin Holst Swende
b9012a039b
common,crypto: move fuzzers out of core (#22029)
* common,crypto: move fuzzers out of core

* fuzzers: move vm fuzzer out from core

* fuzzing: rework cover package logic

* fuzzers: lint
2020-12-23 17:44:45 +01:00
Martin Holst Swende
8cde2966af
eth, core: speed up some tests (#22000) 2020-12-15 18:52:51 +01:00
Péter Szilágyi
017831dd5b
core, eth: split eth package, implement snap protocol (#21482)
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md) , but does not enable it by default. 

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Martin Holst Swende
4d48980e74
core, eth, les: implement unclean-shutdown marker (#21893)
This PR implements unclean shutdown marker. Every time geth boots, it adds a timestamp to a list of timestamps in the database. This list is capped at 10. At a clean shutdown, the timestamp is removed again. 
Thus, when geth exits unclean, the marker remains, and at boot up we show the most recent unclean shutdowns to the user, which makes it easier to diagnose root-causes to certain problems. 

Co-authored-by: Nagy Salem <me@muhnagy.com>
2020-12-11 15:56:00 +01:00
gary rong
88c696240d
core/txpool: remove "local" notion from the txpool price heap (#21478)
* core: separate the local notion from the pricedHeap

* core: add benchmarks

* core: improve tests

* core: address comments

* core: degrade the panic to error message

* core: fix typo

* core: address comments

* core: address comment

* core: use PEAK instead of POP

* core: address comments
2020-12-11 10:44:57 +01:00
Martin Holst Swende
b44f24e3e6
core, trie: speed up some tests with quadratic processing flaw (#21987)
This commit fixes a flaw in two testcases, and brings down the exec-time from ~40s to ~8s for trie/TestIncompleteSync.

The checkConsistency was performed over and over again on the complete set of nodes, not just the recently added, turning it into a quadratic runtime.
2020-12-10 14:48:32 +01:00
Martin Holst Swende
40b6ccf383
core,les: headerchain import in batches (#21471)
* core: add test for headerchain inserts

* core, light: write headerchains in batches

* core: change to one callback per batch of inserted headers + review concerns

* core: error-check on batch write

* core: unexport writeHeaders

* core: remove callback parameter in InsertHeaderChain

The semantics of InsertHeaderChain are now much simpler: it is now an
all-or-nothing operation. The new WriteStatus return value allows
callers to check for the canonicality of the insertion. This change
simplifies use of HeaderChain in package les, where the callback was
previously used to post chain events.

* core: skip some hashing when writing headers

* core: less hashing in header validation

* core: fix headerchain flaw regarding blacklisted hashes

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-12-09 11:13:02 +01:00
Martin Holst Swende
7770e41cb5
core: improve contextual information on core errors (#21869)
A lot of times when we hit 'core' errors, example: invalid tx, the information provided is
insufficient. We miss several pieces of information: what account has nonce too high,
and what transaction in that block was offending?

This PR adds that information, using the new type of wrapped errors.
It also adds a testcase which (partly) verifies the output from the errors.

The first commit changes all usage of direct equality-checks on core errors, into
using errors.Is. The second commit adds contextual information. This wraps most
of the core errors with more information, and also wraps it one more time in
stateprocessor, to further provide tx index and tx hash, if such a tx is encoutered in
a block. The third commit uses the chainmaker to try to generate chains with such
errors in them, thus triggering the errors and checking that the generated string meets
expectations.
2020-12-04 12:22:19 +01:00
Chris Ziogas
62cedb3aab
core/vm/runtime: remove duplicated line (#21956)
This line is duplicated, though it doesn't cause any issues.
2020-12-04 08:54:07 +01:00
Alex Prut
810f9e057d
all: remove redundant conversions and import names (#21903) 2020-11-25 21:00:23 +01:00
Alex Prut
c92faee66e
all: simplify nested complexity and if blocks ending with a return statement (#21854)
Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
2020-11-25 09:24:50 +01:00
Marius van der Wijden
29efe1fc7e
core/types: fixed typo (#21897) 2020-11-25 08:53:20 +01:00
gary rong
23524f8900
all: disable recording preimage of trie keys (#21402)
* cmd, core, eth, light, trie: disable recording preimage by default

* core, eth: fix unit tests

* core: fix import

* all: change to nopreimage

* cmd, core, eth, trie: use cache.preimages flag

* cmd: enable preimages for archive node

* cmd/utils, trie: simplify preimage tracking a bit

* core: fix linter

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-11-18 11:51:33 +02:00
Marius van der Wijden
2045a2bba3
core, all: split vm.Context into BlockContext and TxContext (#21672)
* all: core: split vm.Config into BlockConfig and TxConfig

* core: core/vm: reset EVM between tx in block instead of creating new

* core/vm: added docs
2020-11-13 13:42:19 +01:00
Martin Holst Swende
6f4cccf8d2
core/vm, protocol_params: implement eip-2565 modexp repricing (#21607)
* core/vm, protocol_params: implement eip-2565 modexp repricing

* core/vm: fix review concerns
2020-11-13 13:39:59 +01:00
Péter Szilágyi
7b7b327ff2
core/state/snapshot: update generator marker in sync with flushes 2020-11-09 16:03:58 +02:00
Martin Holst Swende
175506e7fd
core/types, rlp: optimize derivesha (#21728)
This PR contains a minor optimization in derivesha, by exposing the RLP
int-encoding and making use of it to write integers directly to a
buffer (an RLP integer is known to never require more than 9 bytes
total). rlp.AppendUint64 might be useful in other places too.

The code assumes, just as before, that the hasher (a trie) will copy the
key internally, which it does when doing keybytesToHex(key).

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-11-04 19:29:24 +01:00
gary rong
e6402677c2
core/state/snapshot: fix journal recovery from generating old journal (#21775)
* core/state/snapshot: print warning if failed to resolve journal

* core/state/snapshot: fix snapshot recovery

When we meet the snapshot journal consisted with:
- disk layer generator with new-format
- diff layer journal with old-format

The base layer should be returned without error.
The broken diff layer can be reconstructed later
but we definitely don't want to reconstruct the
huge diff layer.

* core: add tests
2020-11-04 13:41:46 +02:00
gary rong
b63e3c37a6
core: improve snapshot journal recovery (#21594)
* core/state/snapshot: introduce snapshot journal version

* core: update the disk layer in an atomic way

* core: persist the disk layer generator periodically

* core/state/snapshot: improve logging

* core/state/snapshot: forcibly ensure the legacy snapshot is matched

* core/state/snapshot: add debug logs

* core, tests: fix tests and special recovery case

* core: polish

* core: add more blockchain tests for snapshot recovery

* core/state: fix comment

* core: add recovery flag for snapshot

* core: add restart after start-after-crash tests

* core/rawdb: fix imports

* core: fix tests

* core: remove log

* core/state/snapshot: fix snapshot

* core: avoid callbacks in SetHead

* core: fix setHead cornercase where the threshold root has state

* core: small docs for the test cases

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-10-29 21:01:58 +02:00
gary rong
43c278cdf9
core/state: disable snapshot iteration if it's not fully constructed (#21682)
* core/state/snapshot: add diskRoot function

* core/state/snapshot: disable iteration if the snapshot is generating

* core/state/snapshot: simplify the function

* core/state: panic for undefined layer
2020-10-28 14:27:37 +02:00
gary rong
18145adf08
core/state: maintain one more diff layer (#21730)
* core/state: maintain one more diff layer

* core/state: address comment
2020-10-28 14:00:22 +02:00
Martin Holst Swende
6487c002f6
all: implement EIP-2929 (gas cost increases for state access opcodes) + yolo-v2 (#21509)
* core/vm, core/state: implement EIP-2929 + YOLOv2

* core/state, core/vm: fix some review concerns

* core/state, core/vm: address review concerns

* core/vm: address review concerns

* core/vm: better documentation

* core/vm: unify sload cost as fully dynamic

* core/vm: fix typo

* core/vm/runtime: fix compilation flaw

* core/vm/runtime: fix renaming-err leftovers

* core/vm: renaming

* params/config: use correct yolov2 chainid for config

* core, params: use a proper new genesis for yolov2

* core/state/tests: golinter nitpicks
2020-10-23 08:26:57 +02:00
hwanjo
91c4607979
core: fix blockchain insert report time interval calculation (#21723) 2020-10-21 16:53:30 +02:00
gary rong
9d25f34263
core: track and improve tx indexing/unindexing (#21331)
* core: add background indexer to waitgroup

* core: make indexer stopable

* core/rawdb: add unit tests

* core/rawdb: fix lint

* core/rawdb: fix tests

* core/rawdb: fix linter
2020-10-20 16:34:50 +02:00
Marius van der Wijden
b305591e14
core/vm: marshall returnData as hexstring in trace logs (#21715)
* core/vm: marshall returnData as hexstring in trace logs

* core/vm: marshall returnData as hexstring in trace logs
2020-10-16 11:28:03 +02:00
mr_franklin
c37e68e7c1
all: replace RWMutex with Mutex in places where RLock is not used (#21622) 2020-10-13 10:58:41 +02:00
mr_franklin
66c3eb2f1a
accouts, consensus, core: fix some comments (#21617) 2020-10-12 15:02:38 +02:00
Martin Holst Swende
706f5e3b98
core: fix txpool off-by-one error (#21683) 2020-10-09 12:23:46 +03:00
Guillaume Ballet
43cd31ea9f
core/vm: dedup config check in markdown logger (#21655)
* core/vm: dedup config check

* review feedback: reuse buffer
2020-10-08 14:03:24 +02:00
Martin Holst Swende
e43d827a19
core/types: optimize bloom filters (#21624)
* core/types: tests for bloom

* core/types: refactored bloom filter for receipts, added tests

core/types: replaced old bloom implementation

core/types: change interface of bloom add+test

* core/types: refactor bloom

* core/types: minor tweak on LogsBloom

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2020-10-06 16:57:00 +03:00
Martin Holst Swende
eb87121300
core/bloombits: faster generator (#21625)
* core/bloombits: add benchmark

* core/bloombits: optimize inserts
2020-10-06 16:34:29 +03:00
Péter Szilágyi
a15d71a255
core/state/snapshot: stop generator if it hits missing trie nodes (#21649)
* core/state/snapshot: exit Geth if generator hits missing trie nodes

* core/state/snapshot: error instead of hard die on generator fault

* core/state/snapshot: don't enable logging on the tests
2020-10-05 11:52:36 +03:00
Guillaume Ballet
6c8310ebb4
trie: use stacktrie for Derivesha operation (#21407)
core/types: use stacktrie for derivesha

trie: add stacktrie file

trie: fix linter

core/types: use stacktrie for derivesha

rebased: adapt stacktrie to the newer version of DeriveSha

Co-authored-by: Martin Holst Swende <martin@swende.se>

More linter fixes

review feedback: no key offset for nodes converted to hashes

trie: use EncodeRLP for full nodes

core/types: insert txs in order in derivesha

trie: tests for derivesha with stacktrie

trie: make stacktrie use pooled hashers

trie: make stacktrie reuse tmp slice space

trie: minor polishes on stacktrie

trie/stacktrie: less rlp dancing

core/types: explain the contorsions in DeriveSha

ci: fix goimport errors

trie: clear mem on subtrie hashing

squashme: linter fix

stracktrie: use pooling, less allocs (#3)

trie: in-place hex prefix, reduce allocs and add rawNode.EncodeRLP

Reintroduce the `[]node` method, add the missing `EncodeRLP` implementation for `rawNode` and calculate the hex prefix in place.

Co-authored-by: Martin Holst Swende <martin@swende.se>

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-09-29 17:38:13 +02:00
aaronbuchwald
eebfb13053
core: free pointer from slice after popping element from price heap (#21572)
* Fix potential memory leak in price heap

* core: nil free pointer slice (alternative version)

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-09-28 14:24:01 +02:00
Martin Holst Swende
0ddd4612b7
core/vm, params: make 2200 in line with spec (#21605) 2020-09-28 14:14:45 +02:00
rene
a25899f3dc
cmd/devp2p: add eth protocol test suite (#21598)
This change adds a test framework for the "eth" protocol and some basic
tests. The tests can be run using the './devp2p rlpx eth-test' command.
2020-09-23 15:18:17 +02:00
Guillaume Ballet
f354c622ca
core: fix a typo in comment (#21439) 2020-09-18 14:26:19 +02:00
Giuseppe Bertone
07751c3d26
cmd/geth: added counters to the geth inspect report (#21495)
* database: added counters

* Improved stats for ancient db

* Small improvement

* Better message and added percentage while counting receipts

* Fast counting for receipts

* added info message

* Show both receips itemscount  from ancient db and counted receipts

* Fixed default case

* Removed counter for receipts in ancient store

* Removed counting of receipts present in leveldb
2020-09-17 10:23:56 +02:00
Giuseppe Bertone
0185ee0993
core/rawdb: single point of maintenance for writing and deleting tx lookup indexes (#21480) 2020-09-15 10:37:01 +02:00
Martin Holst Swende
a3cd8a040a
core/vm: fix benchmark overflow + prep for precompile repricings (#21530)
* core/vm/testdata: add gascost expectations to testcases

* core/vm: verify expected gas in tests for precompiles

* core/vm: fix overflow flaw in gas/s calculation
2020-09-10 09:19:30 +02:00
Péter Szilágyi
eeaf191633
core, eth, trie: prepare trie sync for path based operation 2020-09-02 13:21:32 +03:00
Péter Szilágyi
d8da0b3d81
core/state, eth, trie: stabilize memory use, fix memory leak 2020-08-26 13:05:06 +03:00
Péter Szilágyi
5655dce3b8
core/rawdb: only complain loudly if truncating many items 2020-08-25 09:03:14 +03:00
Péter Szilágyi
3ee91b9f2e
core/state/snapshot: reduce disk layer depth during generation 2020-08-24 13:22:36 +03:00
Martin Holst Swende
1b5a867eec
core: do less lookups when writing fast-sync block bodies (#21468) 2020-08-22 18:12:04 +02:00
gary rong
87c0ba9213
core, eth, les, trie: add a prefix to contract code (#21080) 2020-08-21 15:10:40 +03:00
Péter Szilágyi
8cbdc8638f
core: define and test chain rewind corner cases (#21409)
* core: define and test chain reparation cornercases

* core: write up a variety of set-head tests

* core, eth: unify chain rollbacks, handle all the cases

* core: make linter smile

* core: remove commented out legacy code

* core, eth/downloader: fix review comments

* core: revert a removed recovery mechanism
2020-08-20 13:01:24 +03:00
Marius van der Wijden
0bdd295cc0
core: more detailed metering for reorgs (#21420) 2020-08-20 09:49:35 +02:00
Giuseppe Bertone
2ff464b29d
core/state: fixed some comments (#21450) 2020-08-19 09:54:21 +03:00
Péter Szilágyi
4fde0cabc1
Merge pull request #21411 from holiman/fix_codelookup
core/vm: avoid map lookups for accessing jumpdest analysis
2020-08-06 08:09:15 +03:00
Robert Zaremba
5a88a7cf5b
core: use errors.Is for consensus errors check (#21095) 2020-08-05 09:52:54 +02:00
Martin Holst Swende
8ead45c20b
core/vm: avoid map lookups for accessing jumpdest analysis 2020-08-04 15:45:35 +02:00
Hao Duan
b35e4fce99
core: avoid modification of accountSet cache in tx_pool (#21159)
* core: avoid modification of accountSet cache in tx_pool

when runReorg, we may copy the dirtyAccounts' accountSet cache to promoteAddrs
in which accounts will be promoted, however, if we have reset request at the
same time, we may reuse promoteAddrs and modify the cache content which is
against the original intention of accountSet cache. So, we need to make a new
slice here to avoid modify accountSet cache.

* core: fix flatten condition + comment

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-08-04 11:51:53 +02:00
rene
c0c01612e9
node: refactor package node (#21105)
This PR significantly changes the APIs for instantiating Ethereum nodes in
a Go program. The new APIs are not backwards-compatible, but we feel that
this is made up for by the much simpler way of registering services on
node.Node. You can find more information and rationale in the design
document: https://gist.github.com/renaynay/5bec2de19fde66f4d04c535fd24f0775.

There is also a new feature in Node's Go API: it is now possible to
register arbitrary handlers on the user-facing HTTP server. In geth, this
facility is used to enable GraphQL.

There is a single minor change relevant for geth users in this PR: The
GraphQL API is no longer available separately from the JSON-RPC HTTP
server. If you want GraphQL, you need to enable it using the
./geth --http --graphql flag combination.

The --graphql.port and --graphql.addr flags are no longer available.
2020-08-03 19:40:46 +02:00
Péter Szilágyi
a00dc5095b
Merge pull request #21358 from hendrikhofstadt/fix/tx-sort-time
core: sort txs at the same gas price by received time
2020-07-30 10:23:36 +03:00
meowsbits
ff90894636
core/rawdb: convert some comments to godoc convention (#21384) 2020-07-29 21:53:59 +02:00
ucwong
9e04c5ec83
core/bloombits: use single channel for shutdown (#20878)
This replaces the two-stage shutdown scheme with the one we
use almost everywhere else: a single quit channel signalling
termination.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-07-29 13:49:12 +02:00
Péter Szilágyi
298a19bbc6
core: API-less transaction time sorting 2020-07-28 17:13:40 +03:00
Hendrik Hofstadt
c47052a580
core: sort txs at the same gas price by received time 2020-07-28 17:13:39 +03:00
gary rong
93da0cf8a1
cmd, core, eth, light, trie: dump clean cache periodically (#20391)
* cmd, core, eth, light, trie: dump clean cache periodically

* eth: update config

* trie: minor fix

* core, trie: address comments

* eth: remove useless

* trie: print clean cache dump start too

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-07-28 16:30:31 +03:00
Péter Szilágyi
6793ffa12b
Merge pull request #21300 from rjl493456442/txpool-fix-queued-evictions
core: fix queued transaction eviction
2020-07-24 11:14:42 +03:00
rjl493456442
5413df1dfa
core: fix heartbeat in txpool
core: address comment
2020-07-24 11:12:59 +03:00
villanuevawill
c374447401
core: fix queued transaction eviction
Solves issue#20582. Non-executable transactions should not be evicted on each tick if there are no promote transactions or if a pending/reset empties the pending list. Tests and logging expanded to handle these cases in the future.

core/tx_pool: use a ts for each tx in the queue, but only update the heartbeat on promotion or pending replaced

queuedTs proper naming
2020-07-24 11:11:57 +03:00
Martin Holst Swende
105922180f
eth/downloader: refactor downloader + queue (#21263)
* eth/downloader: refactor downloader + queue

downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache

downloader: more accurate deliverytime calculation, less mem overhead in state requests

downloader/queue: increase underlying buffer of results, new throttle mechanism

eth/downloader: updates to tests

eth/downloader: fix up some review concerns

eth/downloader/queue: minor fixes

eth/downloader: minor fixes after review call

eth/downloader: testcases for queue.go

eth/downloader: minor change, don't set progress unless progress...

eth/downloader: fix flaw which prevented useless peers from being dropped

eth/downloader: try to fix tests

eth/downloader: verify non-deliveries against advertised remote head

eth/downloader: fix flaw with checking closed-status causing hang

eth/downloader: hashing avoidance

eth/downloader: review concerns + simplify resultcache and queue

eth/downloader: add back some locks, address review concerns

downloader/queue: fix remaining lock flaw

* eth/downloader: nitpick fixes

* eth/downloader: remove the *2*3/4 throttling threshold dance

* eth/downloader: print correct throttle threshold in stats

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-07-24 10:46:26 +03:00
Sammy Libre
7163a6664e
ethclient: serialize negative block number as "pending" (#21177)
Fixes #21175

Co-authored-by: sammy007 <sammy007@users.noreply.github.com>
Co-authored-by: Adam Schmideg <adamschmideg@users.noreply.github.com>
2020-07-21 10:51:15 +02:00
Péter Szilágyi
508891e64b
core/vm: use pointers to operations vs. copy by value 2020-07-16 15:32:01 +03:00
Nikola Madjarevic
9e88224eb8
core: raise gas limit in --dev mode, seed blake precompile (#21323)
* Set gasLimit in --dev mode to be 9m.

* core: Set gasLimit to 11.5 milion and add 1 wei allocation for BLAKE2b
2020-07-16 15:08:38 +03:00
Martin Holst Swende
295693759e
core/vm: less allocations for various call variants (#21222)
* core/vm/runtime/tests: add more benchmarks

* core/vm: initial work on improving alloc count for calls to precompiles

name                                  old time/op    new time/op    delta
SimpleLoop/identity-precompile-10M-6     117ms ±75%      43ms ± 1%  -63.09%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   79.6ms ± 4%    70.5ms ± 1%  -11.42%  (p=0.008 n=5+5)

name                                  old alloc/op   new alloc/op   delta
SimpleLoop/identity-precompile-10M-6    24.4MB ± 0%     4.9MB ± 0%  -79.94%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   13.2kB ± 0%    13.2kB ± 0%     ~     (p=0.357 n=5+5)

name                                  old allocs/op  new allocs/op  delta
SimpleLoop/identity-precompile-10M-6      382k ± 0%      153k ± 0%  -59.99%  (p=0.000 n=5+4)
SimpleLoop/loop-10M-6                     40.0 ± 0%      40.0 ± 0%     ~     (all equal)

* core/vm: don't allocate big.int for touch

name                                  old time/op    new time/op    delta
SimpleLoop/identity-precompile-10M-6    43.3ms ± 1%    42.4ms ± 7%     ~     (p=0.151 n=5+5)
SimpleLoop/loop-10M-6                   70.5ms ± 1%    76.7ms ± 1%   +8.67%  (p=0.008 n=5+5)

name                                  old alloc/op   new alloc/op   delta
SimpleLoop/identity-precompile-10M-6    4.90MB ± 0%    2.46MB ± 0%  -49.83%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   13.2kB ± 0%    13.2kB ± 1%     ~     (p=0.571 n=5+5)

name                                  old allocs/op  new allocs/op  delta
SimpleLoop/identity-precompile-10M-6      153k ± 0%       76k ± 0%  -49.98%  (p=0.029 n=4+4)
SimpleLoop/loop-10M-6                     40.0 ± 0%      40.0 ± 0%     ~     (all equal)

* core/vm: reduce allocs in staticcall

name                                  old time/op    new time/op    delta
SimpleLoop/identity-precompile-10M-6    42.4ms ± 7%    37.5ms ± 6%  -11.68%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   76.7ms ± 1%    69.1ms ± 1%   -9.82%  (p=0.008 n=5+5)

name                                  old alloc/op   new alloc/op   delta
SimpleLoop/identity-precompile-10M-6    2.46MB ± 0%    0.02MB ± 0%  -99.35%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   13.2kB ± 1%    13.2kB ± 0%     ~     (p=0.143 n=5+5)

name                                  old allocs/op  new allocs/op  delta
SimpleLoop/identity-precompile-10M-6     76.4k ± 0%      0.1k ± 0%     ~     (p=0.079 n=4+5)
SimpleLoop/loop-10M-6                     40.0 ± 0%      40.0 ± 0%     ~     (all equal)

* trie: better use of hasher keccakState

* core/state/statedb: reduce allocations in getDeletedStateObject

* core/vm: reduce allocations in all call derivates

* core/vm: reduce allocations in call variants

- Make returnstack `uint32`
- Use a `sync.Pool` of `stack`s

* core/vm: fix tests

* core/vm: goimports

* core/vm: tracer fix + staticcall gas fix

* core/vm: add back snapshot to staticcall

* core/vm: review concerns + make returnstack pooled + enable returndata in traces

* core/vm: fix some test tracer method signatures

* core/vm: run gencodec, minor comment polish

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-07-16 15:06:19 +03:00
Martin Holst Swende
6c9f040ebe
core: transaction pool optimizations (#21328)
* core: added local tx pool test case

* core, crypto: various allocation savings regarding tx handling

* core/txlist, txpool: save a reheap operation, avoid some bigint allocs

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2020-07-14 21:42:32 +02:00
gary rong
79addac698
core/rawdb: better log messages for ancient failure (#21327) 2020-07-13 20:43:30 +02:00
gary rong
6eef141aef
les: historical data garbage collection (#19570)
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
2020-07-13 11:02:54 +02:00
Felix Lange
bcb3087450 Revert "core, txpool: less allocations when handling transactions (#21232)"
Reverting because this change started handling account balances as
uint64 in the transaction pool, which is incorrect.

This reverts commit af5c97aebe1d37486635521ef553cb8bd4bada13.
2020-07-09 14:02:03 +02:00
Felix Lange
6315b6fcc0
rlp: reduce allocations for big.Int and byte array encoding (#21291)
This change further improves the performance of RLP encoding by removing
allocations for big.Int and [...]byte types. I have added a new benchmark
that measures RLP encoding of types.Block to verify that performance is
improved.
2020-07-06 11:17:09 +02:00
Martin Holst Swende
12867d152c
rpc, internal/ethapi: default rpc gascap at 25M + better error message (#21229)
* rpc, internal/ethapi: default rpc gascap at 50M + better error message

* eth,internal: make globalgascap uint64

* core/tests: fix compilation failure

* eth/config: gascap at 25M + minor review concerns
2020-07-01 19:54:21 +02:00
Marius van der Wijden
af5c97aebe
core, txpool: less allocations when handling transactions (#21232)
* core: use uint64 for total tx costs instead of big.Int

* core: added local tx pool test case

* core, crypto: various allocation savings regarding tx handling

* Update core/tx_list.go

* core: added tx.GasPriceIntCmp for comparison without allocation

adds a method to remove unneeded allocation in comparison to tx.gasPrice

* core: handle pools full of locals better

* core/tests: benchmark for tx_list

* core/txlist, txpool: save a reheap operation, avoid some bigint allocs

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-07-01 19:35:26 +02:00
Marius van der Wijden
ddeea1e0c6
core: types: less allocations when hashing and tx handling (#21265)
* core, crypto: various allocation savings regarding tx handling

* core: reduce allocs for gas price comparison

This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees

* core: added tx.GasPriceIntCmp for comparison without allocation

adds a method to remove unneeded allocation in comparison to tx.gasPrice

* core/types: pool legacykeccak256 objects in rlpHash

rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.

* reverted some changes

* reverted some changes

* trie: use crypto.KeccakState instead of replicating code

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-06-30 11:59:06 +02:00
Martin Holst Swende
e376d2fb31
cmd/evm: add state transition tool for testing (#20958)
This PR implements the EVM state transition tool, which is intended
to be the replacement for our retesteth client implementation.
Documentation is present in the cmd/evm/README.md file.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-06-30 10:12:51 +02:00
Marius van der Wijden
0c82928981
core/vm: fix incorrect computation of BLS discount (#21253)
* core/vm: fix incorrect computation of discount

During testing on Yolov1 we found that the way geth calculates the discount
is not in line with the specification. Basically what we did is calculate
128 * Bls12381GXMulGas * discount / 1000 whenever we received more than 128 pairs
of values. Correct would be to calculate k * Bls12381... for k > 128.

* core/vm: better logic for discount calculation

* core/vm: better calculation logic, added worstcase benchmarks

* core/vm: better benchmarking logic
2020-06-24 21:58:28 +02:00
AusIV
3ebfeb09fe
core/rawdb: fix high memory usage in freezer (#21243)
The ancients variable in the freezer is a list of hashes, which
identifies all of the hashes to be frozen. The slice is being allocated
with a capacity of `limit`, which is the number of the last block
this batch will attempt to add to the freezer. That means we are
allocating memory for all of the blocks in the freezer, not just
the ones to be added.

If instead we allocate `limit - f.frozen`, we will only allocate
enough space for the blocks we're about to add to the freezer. On
mainnet this reduces usage by about 320 MB.
2020-06-19 10:51:37 +03:00
zhangsoledad
bcf19bc4be
core/rawdb: swap tailId and itemOffset for deleted items in freezer (#21220)
* fix(freezer): tailId filenum offset were misplaced

* core/rawdb: assume first item in freezer always start from zero
2020-06-17 09:41:07 +02:00
Péter Szilágyi
eb9d7d15ec
Merge pull request #21170 from duanhao0814/fix-dup-ecrecover
core: filter out txs with invalid signatures as soon as possible
2020-06-16 11:35:16 +03:00
gary rong
4a19c0e7b8
core, eth, internal: include read storage entries in structlog output (#21204)
* core, eth, internal: extend structLog tracer

* core/vm, internal: add storage view

* core, internal: add slots to storage directly

* core: remove useless

* core: address martin's comment

* core/vm: fix tests
2020-06-10 11:46:13 +02:00
Péter Szilágyi
80e887d7bf
core/state: avoid escape analysis fault when accessing cached state 2020-06-08 16:11:37 +03:00
Paweł Bylica
cf6674539c
core/vm: use uint256 in EVM implementation (#20787)
* core/vm: use fixed uint256 library instead of big

* core/vm: remove intpools

* core/vm: upgrade uint256, fixes uint256.NewFromBig

* core/vm: use uint256.Int by value in Stack

* core/vm: upgrade uint256 to v1.0.0

* core/vm: don't preallocate space for 1024 stack items (only 16)

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-06-08 15:24:40 +03:00
Ev
88125d8bd0
core: fix typo in comments (#21181) 2020-06-08 10:53:56 +03:00
Marius van der Wijden
55f30db0ae
core/vm, crypt/bls12381: fixed comments in bls (#21182)
* core/vm: crypto/bls12381: minor code comments

* crypto/bls12381: fix comment
2020-06-08 10:53:19 +03:00
Hao Duan
8523ad450d core: filter out txs with invalid signatures as soon as possible
Once we detect an invalid transaction during recovering signatures, we should
directly exclude this transaction to avoid validating the signatures hereafter.

This should optimize the validations times of transactions with invalid signatures
to only one time.
2020-06-04 10:37:21 +08:00
Martin Holst Swende
890757f03a
cmd, core, params: inital support for yolo-v1 testnet (#21154)
* core,params,puppeth: inital support for yolo-v1 testnet

* cmd/geth, core: add yolov1 console flag

* cmd, core, params: YoloV1 bakein fixups

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-06-03 12:05:15 +03:00
kilic
4fc678542d
core/vm, crypto/bls12381, params: add bls12-381 elliptic curve precompiles (#21018)
* crypto: add bls12-381 elliptic curve wrapper

* params: add bls12-381 precompile gas parameters

* core/vm: add bls12-381 precompiles

* core/vm: add bls12-381 precompile tests

* go.mod, go.sum: use latest bls12381 lib

* core/vm: move point encode/decode functions to base library

* crypto/bls12381: introduce bls12-381 library init function

* crypto/bls12381: import bls12381 elliptic curve implementation

* go.mod, go.sum: remove bls12-381 library

* remove unsued frobenious coeffs

supress warning for inp that used in asm

* add mappings tests for zero inputs

fix swu g2 minus z inverse constant

* crypto/bls12381: fix typo

* crypto/bls12381: better comments for bls12381 constants

* crypto/bls12381: swu, use single conditional for e2

* crypto/bls12381: utils, delete empty line

* crypto/bls12381: utils, use FromHex for string to big

* crypto/bls12381: g1, g2, strict length check for FromBytes

* crypto/bls12381: field_element, comparision changes

* crypto/bls12381: change swu, isogeny constants with hex values

* core/vm: fix point multiplication comments

* core/vm: fix multiexp gas calculation and lookup for g1 and g2

* core/vm: simpler imput length check for multiexp and pairing precompiles

* core/vm: rm empty multiexp result declarations

* crypto/bls12381: remove modulus type definition

* crypto/bls12381: use proper init function

* crypto/bls12381: get rid of new lines at fatal desciprtions

* crypto/bls12-381: fix no-adx assembly multiplication

* crypto/bls12-381: remove old config function

* crypto/bls12381: update multiplication backend

this commit changes mul backend to 6limb eip1962 backend

mul assign operations are dropped

* core/vm/contracts_tests: externalize test vectors for precompiles

* core/vm/contracts_test: externalize failure-cases for precompiles

* core/vm: linting

* go.mod: tiny up sum file

* core/vm: fix goimports linter issues

* crypto/bls12381: build tags for plain ASM or ADX implementation

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-06-03 09:44:32 +03:00
chenglin
3f649d4852
core: collect NewTxsEvent items without holding reorg lock (#21145) 2020-06-02 18:52:20 +02:00
Greg Colvin
cd57d5cd38
core/vm: EIP-2315, JUMPSUB for the EVM (#20619)
* core/vm: implement EIP 2315, subroutines for the EVM

* core/vm: eip 2315 - lintfix + check jump dest validity + check ret stack size constraints

  logger: markdown-friendly traces, validate jumpdest, more testcase, correct opcodes

* core/vm: update subroutines acc to eip: disallow walk-into

* core/vm/eips: gas cost changes for subroutines

* core/vm: update opcodes for EIP-2315

* core/vm: define RETURNSUB as a 'jumping' operation + review concerns

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-06-02 13:30:16 +03:00
Felix Lange
9219e0fba4
eth: interrupt chain insertion on shutdown (#21114)
This adds a new API method on core.BlockChain to allow interrupting
running data inserts, and calls the method before shutting down the
downloader.

The BlockChain interrupt checks are now done through a method instead
of inlining the atomic load everywhere. There is no loss of efficiency from
this and it makes the interrupt protocol a lot clearer because the check is
defined next to the method that sets the flag.
2020-05-26 21:37:37 +02:00
Martin Holst Swende
e868adde30
core/vm: improve jumpdest lookup (#21123) 2020-05-25 16:12:48 +02:00
Martin Michlmayr
4f2784b38f
all: fix typos in comments (#21118) 2020-05-25 10:21:28 +02:00
Felföldi Zsolt
b4a2681120
les, les/lespay: implement new server pool (#20758)
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.

Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4
2020-05-22 13:46:34 +02:00
gary rong
d02301f758
core: fix missing receipt on Clique crashes (#21045)
* core: fix missing receipt

* core: address comment
2020-05-13 11:33:48 +03:00
ucwong
7540c53e72
core/rawdb: remove unused math (#21065) 2020-05-12 12:19:15 +03:00
ucwong
aaede53738
core/rawdb : log format fix for Unindexing transaction (#21064)
* core/rawdb : log format fix for Unindexing transaction

* core/rawdb: tiny fixup

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-12 11:46:35 +03:00
Martin Holst Swende
7ace5a3a8b
core: fixup blockchain tests (#21062)
core: fixup blockchain tests
2020-05-12 08:46:10 +03:00
Martin Holst Swende
4535230059
cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-11 18:58:43 +03:00
AusIV
069a7e1f8a
core/rawdb: stop freezer process as part of freezer.Close() (#21010)
* core/rawdb: Stop freezer process as part of freezer.Close()

When you call db.Close(), it was closing the leveldb database first,
then closing the freezer, but never stopping the freezer process.
This could cause the freezer to attempt to write to leveldb after
leveldb had been closed, leading to a crash with a non-zero exit code.

This change adds a quit channel to the freezer, and freezer.Close()
will not return until the freezer process has stopped.

Additionally, when you call freezerdb.Close(), it will close the
AncientStore before closing leveldb, to ensure that the freezer goroutine
will be stopped before leveldb is closed.

* core/rawdb: Fix formatting for golint

* core/rawdb: Use backoff flag to avoid repeating select

* core/rawdb: Include accidentally omitted backoff
2020-05-11 15:11:17 +03:00
Péter Szilágyi
0b2edf05bb
core/state: make GetCodeSize mirror GetCode implementation wise 2020-05-11 10:28:56 +03:00
Martin Holst Swende
82f9ed49fa
core/state: avoid statedb.dbErr due to emptyCode (#21051)
* core/state: more verbose stateb errors

* core/state: fix flaw

* core/state: fixed lint

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2020-05-08 21:52:20 +03:00
Martin Holst Swende
b0b65d017f
core/state: abort commit if read errors have occurred (#21039)
This finally adds the error check that the documentation of StateDB.dbErr
promises to do. dbErr was added in 9e5f03b6c (June 2017), and the check was
already missing in that commit. We somehow survived without it for three years.
2020-05-07 15:13:34 +02:00
Martin Holst Swende
1152f45849
core/state: include zero-address in state dump if present (#21038)
* Include 0x0000 address into the dump if it is present

* core/state: go fmt

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-05-07 16:06:31 +03:00
gary rong
85944c2561
core/state/snapshot: fix typo (#21037) 2020-05-07 10:07:59 +03:00
Péter Szilágyi
90af6dae6e
core/state/snapshot: don't create storage list for non-existing accounts 2020-05-06 17:22:38 +03:00
Boqin Qin
39c64d85a2
core: avoid double-lock in tx_pool_test (#20984) 2020-05-06 15:47:59 +02:00
Péter Szilágyi
da59147014
core/state/snapshot: release iterator after verification 2020-05-04 15:14:08 +03:00
gary rong
2337aa64eb
core/state/snapshot: fix trie generator reporter (#21004) 2020-04-30 10:43:50 +03:00
Péter Szilágyi
4b6f6ffe23
core/state/snapshot: fix journal nil deserialziation 2020-04-29 18:00:29 +03:00
gary rong
26d271dfbb
core/state/snapshot: implement storage iterator (#20971)
* core/state/snapshot: implement storage iterator

* core/state/snapshot, tests: implement helper function

* core/state/snapshot: fix storage issue

If an account is deleted in the tx_1 but recreated in the tx_2,
the it can happen that in this diff layer, both destructedSet
and storageData records this account. In this case, the storage
iterator should be able to iterate the slots belong to new account
but disable further iteration in deeper layers(belong to old account)

* core/state/snapshot: address peter and martin's comment

* core/state: address comments

* core/state/snapshot: fix test
2020-04-29 12:53:08 +03:00
Felix Lange
92a7538ed3
core: improve TestLogRebirth (#20961)
This is a resubmit of #20668 which rewrites the problematic test
without any additional goroutines. It also documents the test better.

The purpose of this test is checking whether log events are sent
correctly when importing blocks. The test was written at a time when
blockchain events were delivered asynchronously, making the check hard
to pull off. Now that core.BlockChain delivers events synchronously
during the call to InsertChain, the test can be simplified.

Co-authored-by: BurtonQin <bobbqqin@gmail.com>
2020-04-28 10:06:49 +02:00
Boqin Qin
ba068d40dd
core: add check in AddChildIndexer to avoid double lock (#20982)
This fixes a theoretical double lock condition which could occur in

    indexer.AddChildIndexer(indexer)

Nobody would ever do that though.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-04-27 15:16:30 +02:00
gary rong
8a2e8faadd
core/state/snapshot: fix binary iterator (#20970) 2020-04-24 14:43:49 +03:00
icodezjb
2ec7232191
core: mirror full node reorg logic in light client too (#20931)
* core: fix the condition of reorg

* core: fix nitpick to only retrieve head once

* core: don't reorg if received chain is longer at same diff

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-04-22 11:27:47 +03:00
gary rong
b9df7ecdc3
all: seperate consensus error and evm internal error (#20830)
* all: seperate consensus error and evm internal error

There are actually two types of error will be returned when
a tranaction/message call is executed: (a) consensus error
(b) evm internal error. The former should be converted to
a consensus issue, e.g. The sender doesn't enough asset to
purchase the gas it specifies. The latter is allowed since
evm itself is a blackbox and internal error is allowed to happen.

This PR emphasizes the difference by introducing a executionResult
structure. The evm error is embedded inside. So if any error
returned, it indicates consensus issue happens.

And also this PR improve the `EstimateGas` API to return the concrete
revert reason if the transaction always fails

* all: polish

* accounts/abi/bind/backends: add tests

* accounts/abi/bind/backends, internal: cleanup error message

* all: address comments

* core: fix lint

* accounts, core, eth, internal: address comments

* accounts, internal: resolve revert reason if possible

* accounts, internal: address comments
2020-04-22 11:25:36 +03:00
Martin Holst Swende
c036fe35a8
core/state/snapshot: make difflayer account iterator seek operation inclusive 2020-04-21 16:26:02 +02:00
Boqin Qin
7599999dcd
snapshot: add Unlock before return (#20948)
* Forget Unlock in snapshot

* Remove Unlock before panic
2020-04-21 11:11:38 +03:00
rene
648b0cb714
cmd, core: remove override muir glacier and override istanbul (#20942) 2020-04-20 12:46:38 +03:00
Martin Holst Swende
6402c42b67
all: simplify and fix database iteration with prefix/start (#20808)
* core/state/snapshot: start fixing disk iterator seek

* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start

* les, core/state/snapshot: iterator fixes

* all: remove two iterator methods

* all: rename Iteratee.NewIteratorWith -> NewIterator

* ethdb: fix review concerns
2020-04-15 14:08:53 +03:00
Marius van der Wijden
2a836bb259
core/rawdb: fix data race between Retrieve and Close (#20919)
* core/rawdb: fixed data race between retrieve and close

closes https://github.com/ethereum/go-ethereum/issues/20420

* core/rawdb: use non-atomic load while holding mutex
2020-04-14 18:13:47 +03:00
Raw Pong Ghmoa
15540ae992
cmd: deprecate --testnet, use named networks instead (#20852)
* cmd/utils: make goerli the default testnet

* cmd/geth: explicitly rename testnet to ropsten

* core: explicitly rename testnet to ropsten

* params: explicitly rename testnet to ropsten

* cmd: explicitly rename testnet to ropsten

* miner: explicitly rename testnet to ropsten

* mobile: allow for returning the goerli spec

* tests: explicitly rename testnet to ropsten

* docs: update readme to reflect changes to the default testnet

* mobile: allow for configuring goerli and rinkeby nodes

* cmd/geth: revert --testnet back to ropsten and mark as legacy

* cmd/util: mark --testnet flag as deprecated

* docs: update readme to properly reflect the 3 testnets

* cmd/utils: add an explicit deprecation warning on startup

* cmd/utils: swap goerli and ropsten in usage

* cmd/geth: swap goerli and ropsten in usage

* cmd/geth: if running a known preset, log it for convenience

* docs: improve readme on usage of ropsten's testnet datadir

* cmd/utils: check if legacy `testnet` datadir exists for ropsten

* cmd/geth: check for legacy testnet path in console command

* cmd/geth: use switch statement for complex conditions in main

* cmd/geth: move known preset log statement to the very top

* cmd/utils: create new ropsten configurations in the ropsten datadir

* cmd/utils: makedatadir should check for existing testnet dir

* cmd/geth: add legacy testnet flag to the copy db command

* cmd/geth: add legacy testnet flag to the inspect command
2020-04-09 12:09:58 +03:00
Felix Lange
c8e9a91672
build: upgrade to golangci-lint 1.24.0 (#20901)
* accounts/scwallet: remove unnecessary uses of fmt.Sprintf

* cmd/puppeth: remove unnecessary uses of fmt.Sprintf

* p2p/discv5: remove unnecessary use of fmt.Sprintf

* whisper/mailserver: remove unnecessary uses of fmt.Sprintf

* core: goimports -w tx_pool_test.go

* eth/downloader: goimports -w downloader_test.go

* build: upgrade to golangci-lint 1.24.0
2020-04-08 11:07:29 +03:00
Martin Holst Swende
8dc8941551
core/vm: use a callcontext struct (#20761)
* core/vm: use a callcontext struct

* core/vm: fix tests

* core/vm/runtime: benchmark

* core/vm: make intpool push inlineable, unexpose callcontext
2020-04-07 12:45:21 +03:00
ucwong
f98cabad7c
core: add missing Timer.Stop call in TestLogReorgs (#20870) 2020-04-02 16:04:45 +02:00
Wenbiao Zheng
03fe9de2cb
eth: add debug_accountRange API (#19645)
This new API allows reading accounts and their content by address range.

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2020-03-31 12:08:44 +02:00
Martin Holst Swende
76eed9e50d
snapshotter/tests: verify snapdb post-state against trie (#20812)
* core/state/snapshot: basic trie-to-hash implementation

* tests: validate snapshot after test

* core/state/snapshot: fix review concerns
2020-03-31 10:25:41 +02:00
Martin Holst Swende
55a73f556a
core: bump txpool tx max size to 128KB 2020-03-30 10:47:09 +02:00
Felix Lange
d6c5f2417c
eth: improve shutdown synchronization (#20695)
* eth: improve shutdown synchronization

Most goroutines started by eth.Ethereum didn't have any shutdown sync at
all, which lead to weird error messages when quitting the client.

This change improves the clean shutdown path by stopping all internal
components in dependency order and waiting for them to actually be
stopped before shutdown is considered done. In particular, we now stop
everything related to peers before stopping 'resident' parts such as
core.BlockChain.

* eth: rewrite sync controller

* eth: remove sync start debug message

* eth: notify chainSyncer about new peers after handshake

* eth: move downloader.Cancel call into chainSyncer

* eth: make post-sync block broadcast synchronous

* eth: add comments

* core: change blockchain stop message

* eth: change closeBloomHandler channel type
2020-03-27 15:03:20 +02:00
meowsbits
a75c0610b7
core/blockchain: simplify atomic store after writeBlockWithState (#20798)
Signed-off-by: meows <b5c6@protonmail.com>
2020-03-23 15:05:15 +01:00
Péter Szilágyi
613af7ceea
Merge pull request #20152 from karalabe/snapshot-5
Dynamic state snapshots
2020-03-23 12:57:31 +02:00
Martin Holst Swende
074efe6c8d
core: fix two snapshot iterator flaws, decollide snap storage prefix
* core/state/snapshot/iterator: fix two disk iterator flaws

* core/rawdb: change SnapshotStoragePrefix to avoid prefix collision with preimagePrefix
2020-03-23 12:34:27 +02:00
Péter Szilágyi
36e93d2dd8
Merge pull request #20779 from meowsbits/patch-3
core/rawdb: fix freezer table test error check
2020-03-18 16:30:58 +02:00
gary rong
6283391c99
core/rawdb: improve table database (#20703)
This PR fixes issues in TableDatabase.

TableDatabase is a wrapper of underlying ethdb.Database with an additional prefix.
The prefix is applied to all entries it maintains. However when we try to retrieve entries
from it we don't handle the key properly. In theory the prefix should be truncated and
only user key is returned. But we don't do it in some cases, e.g. the iterator and batch
replayer created from it. So this PR is the fix to these issues.
2020-03-18 13:15:49 +01:00
meowsbits
20a092fb9f
core/rawdb: fix freezer table test error check
Fixes: Condition is always 'false' because 'err' is always 'nil'
2020-03-18 06:55:30 -05:00
Péter Szilágyi
fab0ee3bfa
core/state/snapshot: fix various iteration issues due to destruct set 2020-03-04 15:06:04 +02:00
Martin Holst Swende
bc5d742c66
core: more blockchain tests 2020-03-04 14:39:27 +02:00
Martin Holst Swende
eff7cfbb03
core/state/snapshot: handle deleted accounts in fast iterator 2020-03-04 14:38:55 +02:00
Péter Szilágyi
328de180a7
core/state: fix resurrection state clearing and access 2020-03-04 10:22:48 +02:00
Péter Szilágyi
dcb22a9f99
core/state: fix account root hash update point 2020-03-03 16:55:06 +02:00
Péter Szilágyi
a4cf279494
core/state: extend snapshotter to handle account resurrections 2020-03-03 15:52:00 +02:00
Péter Szilágyi
6e05ccd845
core/state/snapshot, tests: sync snap gen + snaps in consensus tests 2020-03-03 09:17:13 +02:00
Ali Atiia
556888c4a9
core/vm: fix method doc (#20730)
typo in func name in the comment
2020-03-02 19:20:27 +02:00
Martin Holst Swende
fe8347ea8a
squashme 2020-03-02 14:06:44 +01:00
Martin Holst Swende
361a6f08ac
core/tests: test for destroy+recreate contract with storage 2020-03-02 13:46:56 +01:00
Péter Szilágyi
92ec07d63b
core/state: fix an account resurrection issue 2020-02-27 15:03:10 +02:00
Péter Szilágyi
06d4470b41
core: fix broken tests due to API changes + linter 2020-02-25 12:51:16 +02:00
Martin Holst Swende
19099421dc
core/state/snapshot: faster account iteration, CLI integration 2020-02-25 12:51:15 +02:00
Péter Szilágyi
6ddb92a089
core/state/snapshot: full featured account iteration 2020-02-25 12:51:14 +02:00
Martin Holst Swende
e570835356
core/state/snapshot: implement iterator priority for fast direct data lookup 2020-02-25 12:51:14 +02:00
Péter Szilágyi
e567675473
core/state/snapshot: move iterator out into its own files 2020-02-25 12:51:13 +02:00
Martin Holst Swende
7e38996301
core/state/snapshot: implement snapshot layer iteration 2020-02-25 12:51:12 +02:00
Péter Szilágyi
22c494d399
core/state/snapshot: bloom, metrics and prefetcher fixes 2020-02-25 12:51:11 +02:00
Martin Holst Swende
3ad4335acc
core/state/snapshot: node behavioural difference on bloom content 2020-02-25 12:51:11 +02:00
Péter Szilágyi
fd39f722a3
core: journal the snapshot inside leveldb, not a flat file 2020-02-25 12:51:10 +02:00
Martin Holst Swende
d5d7c0c24b
core/state/snapshot: fix difflayer origin-initalization after flatten 2020-02-25 12:51:09 +02:00
Péter Szilágyi
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation 2020-02-25 12:51:08 +02:00
Martin Holst Swende
f300c0df01
core/state/snapshot: replace bigcache with fastcache 2020-02-25 12:51:08 +02:00
Péter Szilágyi
d754091a87
core/state/snapshot: unlink snapshots from blocks, quad->linear cleanup 2020-02-25 12:51:07 +02:00
Martin Holst Swende
cdf3f016df
snapshot: iteration and buffering optimizations 2020-02-25 12:51:06 +02:00
Péter Szilágyi
d7d81d7c12
core/state/snapshot: extract and split cap method, cover corners 2020-02-25 12:51:05 +02:00
Martin Holst Swende
e146fbe4e7
core/state: lazy sorting, snapshot invalidation 2020-02-25 12:51:05 +02:00
Péter Szilágyi
542df8898e
core: initial version of state snapshots 2020-02-25 12:51:04 +02:00
Gregory Markou
4be8840120
core/vm: use dedicated SLOAD gas constant for EIP-2200 (#20646) 2020-02-18 15:07:41 +01:00
Martin Holst Swende
855690523a
core: ensure state exists for prefetcher (#20627) 2020-02-14 10:54:02 +02:00
Péter Szilágyi
9938d954c8
eth: rework tx fetcher to use O(1) ops + manage network requests 2020-02-13 15:27:15 +02:00
rjl493456442
049e17116e
core, eth: implement eth/65 transaction fetcher 2020-02-11 13:56:36 +02:00
Martin Holst Swende
058a4ac5f1
core/evm: less iteration in blockhash (#20589)
* core/vm/runtime: add test for blockhash

* core/evm: less iteration in blockhash

* core/vm/runtime: nitpickfix

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-02-04 12:32:31 +02:00
Martin Holst Swende
5a9c96454e
trie: separate hashes and committer, collapse on commit
* trie:  make db insert use size instead of full data

* core/state: minor optimization in state onleaf allocation

* trie: implement dedicated committer and hasher

* trie: use dedicated committer/hasher

* trie: linter nitpicks

* core/state, trie: avoid unnecessary storage trie load+commit

* trie: review feedback, mainly docs + minor changes

* trie: start deprecating old hasher

* trie: fix misspell+lint

* trie: deprecate hasher.go, make proof framework use new hasher

* trie: rename pure_committer/hasher to committer/hasher

* trie, core/state: fix review concerns

* trie: more review concerns

* trie: make commit collapse into hashnode, don't touch dirtyness

* trie: goimports fixes

* trie: remove panics
2020-02-03 17:28:30 +02:00
Guillaume Ballet
24cab2d535
core/vm/runtime: fix typos in comment (#20608) 2020-01-30 11:21:10 +01:00
gary rong
b88b4632c2 core: fix chain indexer unit test (#20506) 2020-01-20 10:38:08 +02:00
gary rong
770316dc20 core, light: write chain data in atomic way (#20287)
* core: write chain data in atomic way

* core, light: address comments

* core, light: fix linter

* core, light: address comments
2020-01-17 12:49:32 +02:00
Martin Holst Swende
b2de0bd87b
core: set max tx size down to 2 slots (64KB) 2020-01-14 11:49:36 +01:00
MichaelRiabzev-StarkWare
8bd37a1d91 core: count tx size in slots, bump max size ot 4x32KB (#20352)
* tests for tx size

* alow multiple slots transactions

* tests for tx size limit (32 KB)

* change tx size tests to use addRemoteSync instead of validateTx (requested in pool request).

* core: minor tx slotting polishes, add slot tracking metric

Co-authored-by: Michael Riabzev <RiabzevMichael@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-01-10 11:40:03 +02:00
Martin Holst Swende
fc392395fb
core/state: add more verbosity to panic 2020-01-10 10:12:32 +01:00
Felix Lange
d90d1db609
eth/filters: remove use of event.TypeMux for pending logs (#20312) 2019-12-10 12:39:14 +01:00
Marius van der Wijden
f383eaa102
core: removed old invalid comment 2019-12-10 11:50:16 +01:00
Martin Holst Swende
bc01593afb consensus/ethash, params: eip-2384: bump difficulty bomb (#20347)
* consensus/ethash, params: implement eip-2384: bump difficulty bomb

* params: EIP 2384 compat checks

* consensus, params: add Muir Glacier block number (mainnet,ropsten) + official name

* core/forkid: forkid tests for muir glacier

* params/config: address review concerns

* params, core/forkid: review nitpicks

* cmd/geth,eth,les: add override option for muir glacier

* params: nit fix
2019-12-06 11:36:40 +02:00
gary rong
fc7e0fe6c7 core, miner: remove PostChainEvents (#19396)
This change:

- removes the PostChainEvents method on core.BlockChain.
- sorts 'removed log' events by block number.
- fire the NewChainHead event if we inject a canonical block into the chain
  even if the entire insertion is not successful.
- guarantees correct event ordering in all cases.
2019-11-29 14:22:08 +01:00
Péter Szilágyi
58e33d9e5a
Merge pull request #20403 from karalabe/fix-freezer-reinit
core/rawdb: fix reinit regression caused by the hash check PR
2019-11-27 15:05:58 +02:00
Péter Szilágyi
717f8a4e8f
core/rawdb: fix reinit regression caused by the hash check PR 2019-11-27 14:41:47 +02:00