* core: use finalized block as the chain freeze indicator (#28683)
* core/rawdb: use max(finality, head-90k) as chain freezing threshold
* core: impl multi database for block data
* core: fix db inspect total size bug
* core: add tips for user who use multi-database
* core: adapt some cmd for multi-database
* core: adapter blockBlobSidecars
* core: fix freezer readHeader bug
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* ci: temp enable blobtx branch ci run;
* Switch ON blobpool & ensure Cancun hardfork can occur (#2223)
* feat: support blob storage & miscs; (#2229)
* chainconfig: use cancun fork for BSC;
* feat: fill WithdrawalsHash when BSC enable cancun fork;
* rawdb: support to CRUD blobs;
* freezer: support to freeze block blobs;
* blockchain: add blob cache & blob query helper;
* freezer: refactor addition table logic, add uts;
* blobexpiry: add more extra expiry time, and logs;
* parlia: implement IsDataAvailable function;
* blob: refactor blob transfer logic;
* blob: support config blob extra reserve;
* blockchian: support to import block with blob & blobGasFee; (#2260)
* blob: implement min&max gas price logic;
* blockchian: support import side chain;
* blobpool: reject the banned address;
* blockchain: add chasing head for DA check;
* params: update blob related config;
* blockchain: opt data available checking performance;
* params: modify blob related params;
* gasprice: support BEP-336 blob gas price calculate;
* blobTx: mining + brodcasting (#2253)
* blobtx mining pass (#2282)
* Sidecar fetching changes for 4844 (#2283)
* ci: temp enable blobtx branch ci run;
* Switch ON blobpool & ensure Cancun hardfork can occur (#2223)
* feat: support blob storage & miscs; (#2229)
* chainconfig: use cancun fork for BSC;
feat: fill WithdrawalsHash when BSC enable cancun fork;
* rawdb: support to CRUD blobs;
* freezer: support to freeze block blobs;
* blockchain: add blob cache & blob query helper;
* freezer: refactor addition table logic, add uts;
* blobexpiry: add more extra expiry time, and logs;
* parlia: implement IsDataAvailable function;
* blob: refactor blob transfer logic;
* blob: support config blob extra reserve;
* blockchian: support to import block with blob & blobGasFee; (#2260)
* blob: implement min&max gas price logic;
* blockchian: support import side chain;
* blobpool: reject the banned address;
* blockchain: add chasing head for DA check;
* params: update blob related config;
* blockchain: opt data available checking performance;
* params: modify blob related params;
* gasprice: support BEP-336 blob gas price calculate;
* fix failed check for WithdrawalsHash (#2276)
* eth: include sidecars in fitering of body
* core: refactor sidecars name
* eth: sidecars type refactor
* core: remove extra from bad merge
* eth: fix handlenewblock test after merge
* Implement eth_getBlobSidecars && eth_getBlobSidecarByTxHash (#2286)
* execution: add blob gas fee reward to system;
* syncing: support blob syncing & DA checking;
* naming: rename blobs to sidecars;
* fix the semantics of WithXXX (#2293)
* config: reduce sidecar cache to 1024 and rename (#2297)
* fix: Withdrawals turn into empty from nil when BlockBody has Sidecars (#2301)
* internal/api_test: add test case for eth_getBlobSidecars && eth_getBlobSidecarByTxHash (#2300)
* consensus/misc: rollback CalcBlobFee (#2306)
* flags: add new flags to override blobs' params;
* freezer: fix blob ancient save error;
* blobsidecar: add new sidecar struct with metadata; (#2315)
* core/rawdb: optimize write block with sidecars (#2318)
* core: more check for validity of sidecars
* mev: add TxIndex for mev bid (#2325)
* remove useless Config() (#2326)
* fix WithSidecars (#2327)
* fix: fix mined block sidecar issue; (#2328)
* fix WithSidecars (#2329)
---------
Co-authored-by: GalaIO <GalaIO@users.noreply.github.com>
Co-authored-by: buddho <galaxystroller@gmail.com>
Co-authored-by: Satyajit Das <emailtovamos@gmail.com>
Co-authored-by: Eric <45141191+zlacfzy@users.noreply.github.com>
Co-authored-by: zzzckck <152148891+zzzckck@users.noreply.github.com>
This change simplifies the logic for indexing transactions and enhances the UX when transaction is not found by returning more information to users.
Transaction indexing is now considered as a part of the initial sync, and `eth.syncing` will thus be `true` if transaction indexing is not yet finished. API consumers can use the syncing status to determine if the node is ready to serve users.
This adds warning logs when the read does not match the expected count.
We can also remove the size limit since the function documentation explicitly states
that callers should limit the count.
FastFinality puts more infor into the header.extra field to keep vote information.
For mainnet, on epoch height, it could be 1526 bytes, which was 517 bytes before.
So the hardcoded 700 bytes for header could be no longer enough, increase it by
2 times would be enough.
this bug could cause P2P sync failure for nodes that are lagging behind, since they
would request access of ancient db, and GetBlockHeaders could be failed.
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
This change removes a chainconfig parameter passed into rawdb.ReadLogs, which is not used nor needed.
It also modifies the filter loop slightly, avoiding a labeled break and instead using a method.
This change does not modify any behaviour.
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
This change implements withdrawals as specified in EIP-4895.
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: marioevz <marioevz@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.
As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
This PR reworks tx indexer a bit. Compared to the original version, one scenario is no longer handled - upgrading from legacy geth without indexer support.
The tx indexer was introduced in 2020 and have been present through hardforks, so it can be assumed that all Geth nodes have tx indexer already. So we can simplify the tx indexer logic a bit:
- If the tail flag is not present, it means node is just initialized may or may not with an ancient store attached. In this case all blocks are regarded as unindexed
- If the tail flag is present, it means blocks below tail are unindexed, blocks above tail are indexed
This change also address some weird cornercases that could make the indexer not work after a crash.
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.
This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
* core/rawdb, cmd, ethdb, eth: implement freezer tail deletion
* core/rawdb: address comments from martin and sina
* core/rawdb: fixes cornercase in tail deletion
* core/rawdb: separate metadata into a standalone file
* core/rawdb: remove unused code
* core/rawdb: add random test
* core/rawdb: polish code
* core/rawdb: fsync meta file before manipulating the index
* core/rawdb: fix typo
* core/rawdb: address comments
This PR reduces the amount of work we do when answering header queries, e.g. when a peer
is syncing from us.
For some items, e.g block bodies, when we read the rlp-data from database, we plug it
directly into the response package. We didn't do that for headers, but instead read
headers-rlp, decode to types.Header, and re-encode to rlp. This PR changes that to keep it
in RLP-form as much as possible. When a node is syncing from us, it typically requests 192
contiguous headers. On master it has the following effect:
- For headers not in ancient: 2 db lookups. One for translating hash->number (even though
the request is by number), and another for reading by hash (this latter one is sometimes
cached).
- For headers in ancient: 1 file lookup/syscall for translating hash->number (even though
the request is by number), and another for reading the header itself. After this, it
also performes a hashing of the header, to ensure that the hash is what it expected. In
this PR, I instead move the logic for "give me a sequence of blocks" into the lower
layers, where the database can determine how and what to read from leveldb and/or
ancients.
There are basically four types of requests; three of them are improved this way. The
fourth, by hash going backwards, is more tricky to optimize. However, since we know that
the gap is 0, we can look up by the parentHash, and stlil shave off all the number->hash
lookups.
The gapped collection can be optimized similarly, as a follow-up, at least in three out of
four cases.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.
* core/types: rm extranous check in test
* core/rawdb: add lightweight types for block logs
* core/rawdb,eth: use lightweight accessor for log filtering
* core/rawdb: add bench for decoding into rlpLogs
This change is a rewrite of the freezer code.
When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.
In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.
To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
* cmd, core, eth: init tx lookup in background
* core/rawdb: tiny log fixes to make it clearer what's happening
* core, eth: fix rebase errors
* core/rawdb: make reindexing less generic, but more optimal
* rlp: implement rlp list iterator
* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data
* core/rawdb, cmd/utils: fix review concerns
* cmd/utils: fix merge issue
* core/rawdb: add some log formatting polishes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core, eth: some fixes for freezer
* vendor, core/rawdb, cmd/geth: add db inspector
* core, cmd/utils: check ancient store path forceily
* cmd/geth, common, core/rawdb: a few fixes
* cmd/geth: support windows file rename and fix rename error
* core: support ancient plugin
* core, cmd: streaming file copy
* cmd, consensus, core, tests: keep genesis in leveldb
* core: write txlookup during ancient init
* core: bump database version
* all: freezer style syncing
core, eth, les, light: clean up freezer relative APIs
core, eth, les, trie, ethdb, light: clean a bit
core, eth, les, light: add unit tests
core, light: rewrite setHead function
core, eth: fix downloader unit tests
core: add receipt chain insertion test
core: use constant instead of hardcoding table name
core: fix rollback
core: fix setHead
core/rawdb: remove canonical block first and then iterate side chain
core/rawdb, ethdb: add hasAncient interface
eth/downloader: calculate ancient limit via cht first
core, eth, ethdb: lots of fixes
* eth/downloader: print ancient disable log only for fast sync
