This PR fixes a problem which arises on clique networks when there is a network stall. Previously, the worker packages were tracked, even if the sealing engine decided not to seal the block (due to clique rules about recent signing). These tracked-but-not-sealed blocks kept building up in memory.
This PR changes the situation so the sealing engine instead returns an error, and the worker can thus un-track the package.
node.Node no longer registers any account manager backends by default,
they need to be registered explicitly.
For ethash-based tests, we actually don't need any accounts in the miner
keystore. Just set the etherbase instead to make mining work. For
clique, the signer account must be in the keystore.
The change also adds interrupt handling in stress tests.
This fixes a data race on worker.current by moving the call to StopPrefetcher
into the main loop.
The commit also contains fixes for two other races in unit tests of unrelated packages.
* accounts/abi/bind: fix bounded contracts and sim backend for 1559
* accounts/abi/bind, ethclient: don't rely on chain config for gas prices
* all: enable London for all internal tests
* les: get receipt type info in les tests
* les: fix weird test
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/types, miner: create TxWithMinerFee wrapper, add EIP-1559 support to TransactionsByMinerFeeAndNonce
miner: set base fee when creating a new header, handle gas limit, log miner fees
* all: rename to NewTransactionsByPriceAndNonce
* core/types, miner: rename to NewTransactionsByPriceAndNonce + EffectiveTip
miner: activate 1559 for testGenerateBlockAndImport tests
* core,miner: revert naming to TransactionsByPriceAndTime
* core/types/transaction: update effective tip calculation logic
* miner: update aleut to london
* core/types/transaction_test: use correct signer for 1559 txs + add back sender check
* miner/worker: calculate gas target from gas limit
* core, miner: fix block gas limits for 1559
Co-authored-by: Ansgar Dietrichs <adietrichs@gmail.com>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
The PR implements the --miner.notify.full flag that enables full pending block
notifications. When this flag is used, the block notifications sent to mining
endpoints contain the complete block header JSON instead of a work package
array.
Co-authored-by: AlexSSD7 <alexandersadovskyi7@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This adds support for EIP-2718 typed transactions as well as EIP-2930
access list transactions (tx type 1). These EIPs are scheduled for the
Berlin fork.
There very few changes to existing APIs in core/types, and several new APIs
to deal with access list transactions. In particular, there are two new
constructor functions for transactions: types.NewTx and types.SignNewTx.
Since the canonical encoding of typed transactions is not RLP-compatible,
Transaction now has new methods for encoding and decoding: MarshalBinary
and UnmarshalBinary.
The existing EIP-155 signer does not support the new transaction types.
All code dealing with transaction signatures should be updated to use the
newer EIP-2930 signer. To make this easier for future updates, we have
added new constructor functions for types.Signer: types.LatestSigner and
types.LatestSignerForChainID.
This change also adds support for the YoloV3 testnet.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
This moves the eth config definition into a separate package, eth/ethconfig.
Packages eth and les can now import this common package instead of
importing eth from les, reducing dependencies.
Co-authored-by: Felix Lange <fjl@twurst.com>
The PR makes use of the stacktrie, which is is more lenient on resource consumption, than the regular trie, in cases where we only need it for DeriveSha
USB enumeration still occured. Make sure it will only occur if --usb is set.
This also deprecates the 'NoUSB' config file option in favor of a new option 'USB'.
This PR removes a logic in the miner, which was originally intended to help temporary testnets based on ethash from "running off into the future". If the difficulty was low, and a few computers started mining several blocks per second, the ethash rules (which demand 1s delay between blocks) would push the blocktimes further and further away.
The solution was to make the miner sleep while this happened.
Nowadays, this problem is solved instead by PoA chains, and it's recommended to let testnets and devnets be based on clique instead. The existing logic is problematic, since it can cause stalls within the miner making it difficult for remote workers to submit work if the channel is blocked on a sleep.
Credits to Saar Tochner for reporting this via the bug bounty
In miner/worker.go, there are two goroutine using channel w.newWorkCh: newWorkerLoop() sends to this channel, and mainLoop() receives from this channel. Only the receive operation is in a select.
However, w.exitCh may be closed by another goroutine. This is fine for the receive since receive is in select, but if the send operation is blocking, then it will block forever. This commit puts the send in a select, so it won't block even if w.exitCh is closed.
Similarly, there are two goroutines using channel errc: the parent that runs the test receives from it, and the child created at line 573 sends to it. If the parent goroutine exits too early by calling t.Fatalf() at line 614, then the child goroutine will be blocked at line 574 forever. This commit adds 1 buffer to errc. Now send will not block, and receive is not influenced because receive still needs to wait for the send.
A lot of times when we hit 'core' errors, example: invalid tx, the information provided is
insufficient. We miss several pieces of information: what account has nonce too high,
and what transaction in that block was offending?
This PR adds that information, using the new type of wrapped errors.
It also adds a testcase which (partly) verifies the output from the errors.
The first commit changes all usage of direct equality-checks on core errors, into
using errors.Is. The second commit adds contextual information. This wraps most
of the core errors with more information, and also wraps it one more time in
stateprocessor, to further provide tx index and tx hash, if such a tx is encoutered in
a block. The third commit uses the chainmaker to try to generate chains with such
errors in them, thus triggering the errors and checking that the generated string meets
expectations.
* miner: exit loop when downloader Done or Failed
Following the logic of the comment at the method,
this fixes a regression introduced at 7cf56d6f064869cb62b1673f9ee437020c595391
, which would allow external parties to DoS with
blocks, preventing mining progress.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: remove ineff assign (lint)
Signed-off-by: meows <b5c6@protonmail.com>
* miner: update test re downloader events
Signed-off-by: meows <b5c6@protonmail.com>
* Revert "miner: remove ineff assign (lint)"
This reverts commit eaefcd34ab4862ebc936fb8a07578aa2744bc058.
* Revert "miner: exit loop when downloader Done or Failed"
This reverts commit 23abd34265aa246c38fc390bb72572ad6ae9fe3b.
* miner: add test showing imprecise TestMiner
Signed-off-by: meows <b5c6@protonmail.com>
* miner: fix waitForMiningState precision
This helper function would return an affirmation
on the first positive match on a desired bool.
This was imprecise; it return false positives
by not waiting initially for an 'updated' value.
This fix causes TestMiner_2 to fail, which is
expected.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: remove TestMiner_2 demonstrating broken test
This test demonstrated the imprecision of the test
helper function waitForMiningState. This function
has been fixed with 6d365c2851, and this test test
may now be removed.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: fix test regarding downloader event/mining expectations
See comment for logic.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: add test describing expectations for downloader/mining events
We expect that once the downloader emits a DoneEvent,
signaling a successful sync, that subsequent StartEvents
are not longer permitted to stop the miner.
This prevents a security vulnerability where forced syncs via
fake high blocks would stall mining operation.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: use 'canStop' state to fix downloader event handling
- Break downloader event handling into event
separating Done and Failed events. We need to
treat these cases differently since a DoneEvent
should prevent the miner from being stopped on
subsequent downloader Start events.
- Use canStop state to handle the one-off
case when a downloader first succeeds.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: improve comment wording
Signed-off-by: meows <b5c6@protonmail.com>
* miner: start mining on downloader events iff not already mining
Signed-off-by: meows <b5c6@protonmail.com>
* miner: refactor miner update logic w/r/t downloader events
This makes mining pause/start logic regarding downloader
events more explicit. Instead of eternally handling downloader
events after the first done event, the subscription is closed
when downloader events are no longer actionable.
Signed-off-by: meows <b5c6@protonmail.com>
* miner: fix handling downloader events on subcription closed
Signed-off-by: meows <b5c6@protonmail.com>
* miner: (lint:gosimple) use range over chan instead of for/select
Signed-off-by: meows <b5c6@protonmail.com>
* miner: refactor update loop to remove race condition
The go routine handling the downloader events handling
vars in parallel with the parent routine, causing a
race condition.
This change, though ugly, remove the condition while
still allowing the downloader event subscription to be
closed when the miner has no further use for it (ie DoneEvent).
* miner: alternate fix for miner-flaw
Co-authored-by: meows <b5c6@protonmail.com>
This PR changes several different things:
- Adds test cases for the miner loop
- Stops the worker if it wasn't already stopped in worker.Close()
- Uses channels instead of atomics in the miner.update() loop
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR significantly changes the APIs for instantiating Ethereum nodes in
a Go program. The new APIs are not backwards-compatible, but we feel that
this is made up for by the much simpler way of registering services on
node.Node. You can find more information and rationale in the design
document: https://gist.github.com/renaynay/5bec2de19fde66f4d04c535fd24f0775.
There is also a new feature in Node's Go API: it is now possible to
register arbitrary handlers on the user-facing HTTP server. In geth, this
facility is used to enable GraphQL.
There is a single minor change relevant for geth users in this PR: The
GraphQL API is no longer available separately from the JSON-RPC HTTP
server. If you want GraphQL, you need to enable it using the
./geth --http --graphql flag combination.
The --graphql.port and --graphql.addr flags are no longer available.
* eth/downloader tests: fix spurious failing test due to race between receipts/headers
* miner tests: fix travis failure on arm64
* eth/downloader: tests - store td in ancients too
* cmd, miner: add noempty-precommit flag
* cmd, miner: get rid of external flag
* miner: change bool to atomic int
* miner: fix tiny typo
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* cmd, core, eth: init tx lookup in background
* core/rawdb: tiny log fixes to make it clearer what's happening
* core, eth: fix rebase errors
* core/rawdb: make reindexing less generic, but more optimal
* rlp: implement rlp list iterator
* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data
* core/rawdb, cmd/utils: fix review concerns
* cmd/utils: fix merge issue
* core/rawdb: add some log formatting polishes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* cmd/utils: make goerli the default testnet
* cmd/geth: explicitly rename testnet to ropsten
* core: explicitly rename testnet to ropsten
* params: explicitly rename testnet to ropsten
* cmd: explicitly rename testnet to ropsten
* miner: explicitly rename testnet to ropsten
* mobile: allow for returning the goerli spec
* tests: explicitly rename testnet to ropsten
* docs: update readme to reflect changes to the default testnet
* mobile: allow for configuring goerli and rinkeby nodes
* cmd/geth: revert --testnet back to ropsten and mark as legacy
* cmd/util: mark --testnet flag as deprecated
* docs: update readme to properly reflect the 3 testnets
* cmd/utils: add an explicit deprecation warning on startup
* cmd/utils: swap goerli and ropsten in usage
* cmd/geth: swap goerli and ropsten in usage
* cmd/geth: if running a known preset, log it for convenience
* docs: improve readme on usage of ropsten's testnet datadir
* cmd/utils: check if legacy `testnet` datadir exists for ropsten
* cmd/geth: check for legacy testnet path in console command
* cmd/geth: use switch statement for complex conditions in main
* cmd/geth: move known preset log statement to the very top
* cmd/utils: create new ropsten configurations in the ropsten datadir
* cmd/utils: makedatadir should check for existing testnet dir
* cmd/geth: add legacy testnet flag to the copy db command
* cmd/geth: add legacy testnet flag to the inspect command
The leaks were mostly in unit tests, and could all be resolved by
adding suitably-sized channel buffers or by restructuring the test
to not send on a channel after an error has occurred.
There is an unavoidable goroutine leak in Console.Interactive: when
we receive a signal, the line reader cannot be unblocked and will get
stuck. This leak is now documented and I've tried to make it slightly
less bad by adding a one-element buffer to the output channels of
the line-reading loop. Should the reader eventually awake from its
blocked state (i.e. when stdin is closed), at least it won't get stuck
trying to send to the interpreter loop which has quit long ago.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change:
- removes the PostChainEvents method on core.BlockChain.
- sorts 'removed log' events by block number.
- fire the NewChainHead event if we inject a canonical block into the chain
even if the entire insertion is not successful.
- guarantees correct event ordering in all cases.
* travis: Enable ARM support
* Include fixes from 20039
* Add a trace to debug the invalid lookup issue
* Try increasing the timeout to see if the arm test passes
* Investigate the resolver issue
* Increase arm64 timeout for clique test
* increase timeout in tests for arm64
* Only test the failing tests
* Review feedback: don't export epsilon
* Remove investigation tricks+include fjl's feeback
* Revert the retry ahead of using the mock resolver
* Fix rebase errors
This PR adds a new unit test in miner package which will create some blocks from miner and then import into another chain. In this way, we can ensure all blocks generated by Geth miner obey consensus rules.
* cmd, eth, miner: disable advance sealing if user require
* cmd, console, miner, les, eth: wrap the miner config
* eth: remove todo
* cmd, miner: revert noadvance flag
The reason for this is: if the transaction execution is even longer
than block time, then this kind of transactions is DoS attack.
* node: close AccountsManager in new Close method
* p2p/simulations, p2p/simulations/adapters: handle node close on shutdown
* node: move node ephemeralKeystore cleanup to stop method
* node: call Stop in Node.Close method
* cmd/geth: close node.Node created with makeFullNode in cli commands
* node: close Node instances in tests
* cmd/geth, node: minor code style fixes
* cmd, console, miner, mobile: proper node Close() termination
Until this commit, when sending an RPC request that called `NewEVM`, a blank `vm.Config`
would be taken so as to set some options, based on the default configuration. If some extra
configuration switches were passed to the blockchain, those would be ignored.
This PR adds a function to get the config from the blockchain, and this is what is now used
for RPC calls.
Some subsequent changes need to be made, see https://github.com/ethereum/go-ethereum/pull/17955#pullrequestreview-182237244
for the details of the discussion.
* miner: commit state which is relative with sealing result
* consensus, core, miner, mobile: introduce sealHash interface
* miner: evict pending task with threshold
* miner: go fmt
* consensus/ethash: start remote ggoroutine to handle remote mining
* consensus/ethash: expose remote miner api
* consensus/ethash: expose submitHashrate api
* miner, ethash: push empty block to sealer without waiting execution
* consensus, internal: add getHashrate API for ethash
* consensus: add three method for consensus interface
* miner: expose consensus engine running status to miner
* eth, miner: specify etherbase when miner created
* miner: commit new work when consensus engine is started
* consensus, miner: fix some logics
* all: delete useless interfaces
* consensus: polish a bit
* cmd, consensus, core, miner: instatx clique for --dev
* cmd, consensus, clique: support configurable --dev block times
* cmd, core: allow --dev to use persistent storage too
* ethdb: add Putter interface and Has method
* ethdb: improve docs and add IdealBatchSize
* ethdb: remove memory batch lock
Batches are not safe for concurrent use.
* core: use ethdb.Putter for Write* functions
This covers the easy cases.
* core/state: simplify StateSync
* trie: optimize local node check
* ethdb: add ValueSize to Batch
* core: optimize HasHeader check
This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.
* core: write fast sync block data in batches
Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.
* eth/downloader: commit larger state batches
Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.
* core: optimize HasBlock check
This avoids a random database read to get the number.
* core: use numberCache in HasHeader
numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.
* core: write imported block data using a batch
Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.
This commit also removes posting of logs when a forked block is imported.
* core: fix DB write error handling
* ethdb: use RLock for Has
* core: fix HasBlock comment
* core: remove redundant storage of transactions and receipts
* core, eth, internal: new transaction schema usage polishes
* eth: implement upgrade mechanism for db deduplication
* core, eth: drop old sequential key db upgrader
* eth: close last iterator on successful db upgrage
* core: prefix the lookup entries to make their purpose clearer
With this commit, core/state's access to the underlying key/value database is
mediated through an interface. Database errors are tracked in StateDB and
returned by CommitTo or the new Error method.
Motivation for this change: We can remove the light client's duplicated copy of
core/state. The light client now supports node iteration, so tracing and storage
enumeration can work with the light client (not implemented in this commit).
This commit is a preparation for the upcoming metropolis hardfork. It
prepares the state, core and vm packages such that integration with
metropolis becomes less of a hassle.
* Difficulty calculation requires header instead of individual
parameters
* statedb.StartRecord renamed to statedb.Prepare and added Finalise
method required by metropolis, which removes unwanted accounts from
the state (i.e. selfdestruct)
* State keeps record of destructed objects (in addition to dirty
objects)
* core/vm pre-compiles may now return errors
* core/vm pre-compiles gas check now take the full byte slice as argument
instead of just the size
* core/vm now keeps several hard-fork instruction tables instead of a
single instruction table and removes the need for hard-fork checks in
the instructions
* core/vm contains a empty restruction function which is added in
preparation of metropolis write-only mode operations
* Adds the bn256 curve
* Adds and sets the metropolis chain config block parameters (2^64-1)
This commit adds pluggable consensus engines to go-ethereum. In short, it
introduces a generic consensus interface, and refactors the entire codebase to
use this interface.
* common: remove CurrencyToString
Move denomination values to params instead.
* common: delete dead code
* common: move big integer operations to common/math
This commit consolidates all big integer operations into common/math and
adds tests and documentation.
There should be no change in semantics for BigPow, BigMin, BigMax, S256,
U256, Exp and their behaviour is now locked in by tests.
The BigD, BytesToBig and Bytes2Big functions don't provide additional
value, all uses are replaced by new(big.Int).SetBytes().
BigToBytes is now called PaddedBigBytes, its minimum output size
parameter is now specified as the number of bytes instead of bits. The
single use of this function is in the EVM's MSTORE instruction.
Big and String2Big are replaced by ParseBig, which is slightly stricter.
It previously accepted leading zeros for hexadecimal inputs but treated
decimal inputs as octal if a leading zero digit was present.
ParseUint64 is used in places where String2Big was used to decode a
uint64.
The new functions MustParseBig and MustParseUint64 are now used in many
places where parsing errors were previously ignored.
* common: delete unused big integer variables
* accounts/abi: replace uses of BytesToBig with use of encoding/binary
* common: remove BytesToBig
* common: remove Bytes2Big
* common: remove BigTrue
* cmd/utils: add BigFlag and use it for error-checked integer flags
While here, remove environment variable processing for DirectoryFlag
because we don't use it.
* core: add missing error checks in genesis block parser
* common: remove String2Big
* cmd/evm: use utils.BigFlag
* common/math: check for 256 bit overflow in ParseBig
This is supposed to prevent silent overflow/truncation of values in the
genesis block JSON. Without this check, a genesis block that set a
balance larger than 256 bits would lead to weird behaviour in the VM.
* cmd/utils: fixup import
Reworked the EVM gas instructions to use 64bit integers rather than
arbitrary size big ints. All gas operations, be it additions,
multiplications or divisions, are checked and guarded against 64 bit
integer overflows.
In additon, most of the protocol paramaters in the params package have
been converted to uint64 and are now constants rather than variables.
* common/math: added overflow check ops
* core: vmenv, env renamed to evm
* eth, internal/ethapi, les: unmetered eth_call and cancel methods
* core/vm: implemented big.Int pool for evm instructions
* core/vm: unexported intPool methods & verification methods
* core/vm: added memoryGasCost overflow check and test
Reworked the EVM gas instructions to use 64bit integers rather than
arbitrary size big ints. All gas operations, be it additions,
multiplications or divisions, are checked and guarded against 64 bit
integer overflows.
In additon, most of the protocol paramaters in the params package have
been converted to uint64 and are now constants rather than variables.
* common/math: added overflow check ops
* core: vmenv, env renamed to evm
* eth, internal/ethapi, les: unmetered eth_call and cancel methods
* core/vm: implemented big.Int pool for evm instructions
* core/vm: unexported intPool methods & verification methods
* core/vm: added memoryGasCost overflow check and test
The Subscription type is gone, all uses are replaced by
*TypeMuxSubscription. This change is prep-work for the
introduction of the new Subscription type in a later commit.
gorename -from '"github.com/ethereum/go-ethereum/event"::Event' -to TypeMuxEvent
gorename -from '"github.com/ethereum/go-ethereum/event"::muxsub' -to TypeMuxSubscription
gofmt -w -r 'Subscription -> *TypeMuxSubscription' ./event/*.go
find . -name '*.go' -and -not -regex '\./vendor/.*' \| xargs gofmt -w -r 'event.Subscription -> *event.TypeMuxSubscription'
This significantly reduces the dependency closure of ethclient, which no
longer depends on core/vm as of this change.
All uses of vm.Logs are replaced by []*types.Log. NewLog is gone too,
the constructor simply returned a literal.
The run loop, which previously contained custom opcode executes have been
removed and has been simplified to a few checks.
Each operation consists of 4 elements: execution function, gas cost function,
stack validation function and memory size function. The execution function
implements the operation's runtime behaviour, the gas cost function implements
the operation gas costs function and greatly depends on the memory and stack,
the stack validation function validates the stack and makes sure that enough
items can be popped off and pushed on and the memory size function calculates
the memory required for the operation and returns it.
This commit also allows the EVM to go unmetered. This is helpful for offline
operations such as contract calls.
The transaction pool keeps track of the current nonce in its local pendingState. When a
new block comes in the pendingState is reset. During the reset it fetches multiple times
the current state through the use of the currentState callback. When a second block comes
in during the reset its possible that the state changes during the reset. If that block
holds transactions that are currently in the pool the local pendingState that is used to
determine nonces can get out of sync.
