Compare commits

..

150 Commits

Author SHA1 Message Date
joeylichang
ffa6c04478 chore: reopen prefetcher 2024-09-30 15:22:22 +08:00
joeylichang
7bf93179b9 chore: delete write batch and change versa config 2024-09-30 15:11:05 +08:00
joeylichang
37b942eb2c fix: delete commit and commit num log 2024-09-29 14:54:43 +08:00
joeylichang
2fbd77d64c chore: change default config 2024-09-29 11:13:55 +08:00
joeylichang
46f6c75cc0 chore: change default config 2024-09-29 11:04:15 +08:00
joeylichang
7a0d393c0d chore: change storage writebatch to single update 2024-09-28 15:44:13 +08:00
joeylichang
ed573ee1a8 chore: change versa config' 2024-09-28 07:39:25 +08:00
joeylichang
6252589246 chore: delete prefetcher 2024-09-28 07:33:39 +08:00
joeylichang
5271bf9bdb feat: support versa write batch 2024-09-27 17:01:12 +08:00
joeylichang
2a6bb8720d fix: commit and calc num 2024-09-26 17:07:43 +08:00
joeylichang
53b585e7ab chore: add check for genesis 2024-09-26 16:55:15 +08:00
joeylichang
5018895927 chore: add commit and calc root num log 2024-09-26 16:25:32 +08:00
joeylichang
3dc1030979 chore: add state object commit panic check 2024-09-26 15:39:30 +08:00
will-2012
b8f5b6a135 Merge pull request #2709 from will-2012/update-versadb-config0
chore: update trivals
2024-09-14 16:43:17 +08:00
will@2012
4b17959733 chore: update trivals 2024-09-14 16:42:46 +08:00
will-2012
5bbfa69f35 Merge pull request #2708 from will-2012/update-versadb-config
chore: update versadb default config
2024-09-14 16:37:30 +08:00
will@2012
99678970e1 chore: update versadb default config 2024-09-14 16:34:57 +08:00
joeycli
6d082af0f3 feat: add prefetcher 2024-09-09 14:22:11 +08:00
joeycli
089845c5f3 chore: add statedb get metrices 2024-09-06 14:06:47 +08:00
joeycli
d7e873c28c fix: delete redundancy state close 2024-09-06 09:14:55 +08:00
joeycli
f646d4f1c0 feat: support version and root as key to state
fix: genesis version to -1

fix: copy with version

fix: change version type from uint64 to int64

fix: set genesis for rewind
2024-09-05 14:55:49 +08:00
joeycli
eaddc87def fix: check version and root when rewind
fix: rewind for genesis
2024-09-04 15:49:06 +08:00
joeycli
2c59e15b7a chore: add validate metrics 2024-09-04 10:21:33 +08:00
joeycli
a5e12fe178 feat: support rewind for versa db
fix: rewind ancient store
2024-09-02 17:35:32 +08:00
joeycli
4ed6b8e36f chore: add execute, verify and commit total metrics 2024-09-02 09:12:54 +08:00
joeycli
b76062c354 fix: rewind ancient store 2024-09-01 12:09:00 +08:00
joeycli
b298f4b6e6 chore: add mgasps metrics 2024-08-29 16:40:46 +08:00
joeycli
06366d743d chore: add pprof service 2024-08-29 12:15:39 +08:00
joeycli
3e8e74d5c2 feat: delete statedb debug system 2024-08-27 09:29:24 +08:00
joeycli
d20bbb4799 feat: integrate versa db
feat: add version commit

chore: forbid versiondb rewind

feat: add mode for new caching db

feat: add version scheme for startup

feat: init genesis for versa db

feat: add caching db and trie copy

feat: support HasState on versa db and must fullsync under versa db

fix: append open strorage trie error to statedb

fix: storage tree value encode

fix: add state object trie expire interface

fix: blockchain stateat use rw state

chore: forbid prefetcher

chore: delete storage pool

feat: hold version in state objet for repeat search account tree

fix: version mismatch that add contract balance without update storage tree

fix: 373559 blocks issue add breakpoint

feat: add version state debug system

feat: add hash state db debug system

feat: add version and hash state diff system

fix: the timing release debug state instance
2024-08-27 09:01:25 +08:00
zzzckck
83a9b13771 Merge pull request #2607 from NathanBSC/for_release_v1.4.12
Revert "miner/worker: broadcast block immediately once sealed (bnb-chain#2576)"
2024-07-24 12:07:13 +08:00
NathanBSC
c6cb43b7ca Revert "miner/worker: broadcast block immediately once sealed (#2576)"
This reverts commit 6d5b4ad64d.
2024-07-22 18:23:10 +08:00
NathanBSC
222e10810e core: cache block after wroten into db 2024-07-22 18:22:56 +08:00
zzzckck
26b236fb5f Merge pull request #2586 from bnb-chain/master_2_develop
Draft release v1.4.12
2024-07-17 17:45:39 +08:00
zzzckck
900cf26c65 Merge branch 'master' into master_2_develop 2024-07-17 17:29:11 +08:00
zzzckck
21e6dcfc79 release: prepare for release v1.4.12 (#2585) 2024-07-17 17:07:57 +08:00
Chris Li
a262acfb00 fix: remove delete and dangling side chains in prunefreezer (#2582) 2024-07-17 11:01:40 +08:00
Chris Li
87e622e51f fix: the bug of blobsidecars and downloader with multi-database (#2564) 2024-07-16 22:37:03 +08:00
Chris Li
13d454796f fix: pruneancient freeze from the previous position when the first time (#2542) 2024-07-16 22:36:18 +08:00
galaio
c6af48100d freezer: Opt freezer env checking (#2580) 2024-07-16 21:44:39 +08:00
buddho
6d5b4ad64d miner/worker: broadcast block immediately once sealed (#2576) 2024-07-16 21:24:37 +08:00
stellrust
d35b57ae36 chore: fix some comments (#2581)
Signed-off-by: stellrust <gohunter@foxmail.com>
2024-07-16 17:10:45 +08:00
Eric
21fc2d3ac4 cmd/jsutill: add log about validator name (#2583) 2024-07-16 17:10:16 +08:00
Nathan
c96fab04a3 core/vote: not vote if too late for next in turn validator (#2568) 2024-07-11 22:39:07 +08:00
buddho
27d86948fa core: adapt highestVerifiedHeader to FastFinality (#2574) 2024-07-11 15:01:01 +08:00
Mars
a04e287cb6 feat: enhance bid comparison and reply bidding results && detail logs (#2538) 2024-07-11 11:31:24 +08:00
Nathan
c3d6155fff cmd/utils: support use NetworkId to distinguish chapel when do syncing (#2573) 2024-07-10 16:10:07 +08:00
will-2012
a00ffa762c fix: fix statedb copy (#2567) 2024-07-10 15:39:20 +08:00
buddho
863fdea026 core: avoid to cache block before wroten into db (#2566) 2024-07-10 14:46:11 +08:00
Nathan
90e67970ae core: clearup testflag for Cancun and Haber (#2572) 2024-07-09 14:30:19 +08:00
Eric
e8456c2d08 cmd/jsutils: add a tool to get slash count (#2569) 2024-07-08 22:37:05 +08:00
Chris Li
bc970e5893 fix: delete unexpected block (#2562) 2024-07-08 19:21:56 +08:00
wayen
a5810fefc9 fix: fix state inspect error after pruned state (#2557) 2024-07-04 22:24:22 +08:00
Matus Kysel
971c0fa380 tests: fix unstable test (#2561) 2024-07-04 07:56:35 +02:00
KeefeL
88225c1a4b chore: update greenfield cometbft version (#2556) 2024-07-03 15:14:19 +08:00
zzzckck
51e27f9e3b nancy: ignore go-retryablehttp@v0.7.4 in .nancy-ignore (#2559) 2024-07-02 15:00:14 +08:00
zzzckck
f1a85ec306 go.mod: update missing dependency (#2546) 2024-06-28 14:14:22 +08:00
dependabot[bot]
75d162983a build(deps): bump github.com/hashicorp/go-retryablehttp (#2537)
Bumps [github.com/hashicorp/go-retryablehttp](https://github.com/hashicorp/go-retryablehttp) from 0.7.4 to 0.7.7.
- [Changelog](https://github.com/hashicorp/go-retryablehttp/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/go-retryablehttp/compare/v0.7.4...v0.7.7)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-retryablehttp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-28 13:51:53 +08:00
Nathan
727c07116d cmd/jsutils: add a tool to get performance between a range of blocks (#2513) 2024-06-28 13:48:17 +08:00
zoro
719412551a Merge pull request #2549 from bnb-chain/develop
release: prepare for v1.4.11
2024-06-27 17:37:26 +08:00
zoro
c2226a0c9f release: bump version to 1.4.11 (#2550) 2024-06-27 17:35:46 +08:00
zoro
d52628aa82 doc: add changelog for v1.4.11 (#2548) 2024-06-27 17:24:49 +08:00
Roshan
f7de51f74e upgrade: add HaberFix hardfork (#2535) 2024-06-26 14:51:52 +08:00
irrun
55cbf31f18 fix: nil pointer when clear simulating bid (#2534) 2024-06-24 16:42:17 +08:00
zzzckck
f0c7795542 Merge pull request #2527 from bnb-chain/develop
Draft release v1.4.10
2024-06-21 16:13:48 +08:00
zzzckck
1548452def release: prepare for release v1.4.10 (#2526) 2024-06-21 15:31:43 +08:00
Nathan
f2e7f1dd24 fix: ensure empty withdrawals after cancun before broadcast (#2525) 2024-06-21 15:00:56 +08:00
Eric
27a3ec5d72 fix getBlobSidecars by ethclient (#2515) 2024-06-20 11:29:33 +08:00
zzzckck
6094d7157e UT: random failure of TestSnapSyncWithBlobs (#2519)
this case failed randomly in github CI, the ratio of around 20%
but can not be reproduced locally by:
```
  go test -p 1 -v -run TestSnapSyncWithBlobs -count 100
```
there could be a potential risk between `runEthPeer` and `runSnapExtension`,as
they shared the same handler.peers.
Add extra wait time for `runEthPeer` to resolve it.

And same for another UT: testBroadcastBlock
2024-06-20 11:14:52 +08:00
zzzckck
be0fbfb79e fix: remove zero gasprice check for BSC (#2518)
note: blobGasFeePrice can not be zero right now.
2024-06-19 19:51:30 +08:00
Nathan
00f094c37e core/forkchoice: improve stability when inturn block not generated (#2451) 2024-06-18 15:53:05 +08:00
Chris Li
7b08a70a23 perf: optimize chain commit performance for multi-database (#2509) 2024-06-18 15:20:23 +08:00
will-2012
99d31aeb28 perf: speedup pbss trienode read (#2508) 2024-06-18 11:47:22 +08:00
irrun
f467c6018b feat: add mev helper params and func (#2512) 2024-06-13 11:13:39 +08:00
zzzckck
4566ac7659 Merge pull request #2511 from bnb-chain/develop
Draft release v1.4.9
2024-06-11 11:51:22 +08:00
zzzckck
aab4b8812a release: prepare for release v1.4.9 (#2510) 2024-06-11 10:47:44 +08:00
irrun
af7e9b95bd fix: waiting for the last simulation before pick best bid (#2507) 2024-06-07 16:41:20 +08:00
zoupingshi
1047f0e59a chore: fix function name (#2501)
Signed-off-by: zoupingshi <hellocatty@tom.com>
2024-06-05 15:12:48 +08:00
VM
9bb4fed1bf fix: add an empty freeze db (#2495) 2024-06-04 15:47:16 +08:00
zzzckck
35e71a769b doc: update url linker (#2499) 2024-05-30 14:08:48 +08:00
zzzckck
6b02ac7ac5 doc: update url linker (#2498) 2024-05-30 12:19:07 +08:00
Nathan
8b9558bb4d params/config: add Bohr hardfork (#2497) 2024-05-30 11:51:34 +08:00
Chris Li
63e7eac394 fix: keep 9W blocks in ancient db when prune block (#2481) 2024-05-29 15:11:52 +08:00
Chris Li
05543e558d fix: fix inspect database error (#2484) 2024-05-27 15:10:59 +08:00
Satyajit Das
b0146261c7 jsutils: faucet successful requests within blocks (#2470) 2024-05-23 15:08:36 +08:00
lx
d7b9866d3b Merge pull request #2487 from bnb-chain/master
merge PRs from master to develop branch
2024-05-22 13:56:19 +08:00
zzzckck
f5ba30ed47 release: prepare for release v1.4.8 (#2486) 2024-05-21 18:30:28 +08:00
Nathan
f190c49252 core/vm: add secp256r1 into PrecompiledContractsHaber (#2483) 2024-05-21 17:46:42 +08:00
Mars
08769ead2b dev: ensure consistency in BPS bundle result (#2479)
* dev: ensure consistency in BPS bundle result

* fix: remove env operation once the sim is discarded & rename
2024-05-21 12:24:41 +08:00
Matus Kysel
4d0f1e7117 RIP-7212: Precompile for secp256r1 Curve Support (#2400) 2024-05-21 11:44:21 +08:00
irrun
c77bb1110d fix: limit the gas price of the mev bid (#2473) 2024-05-20 14:33:47 +08:00
Mars
c856d21719 fix: move mev op to MinerAPI & add command to console (#2475) 2024-05-20 14:00:28 +08:00
Nathan
f45305b1ad cmd/utils: add a flag to change breathe block interval for testing (#2472) 2024-05-17 16:18:29 +08:00
Eric
d16532d678 internal/ethapi: add optional parameter for blobSidecars (#2468) 2024-05-16 19:07:14 +08:00
Eric
5edd032cdb internal/ethapi: add optional parameter for blobSidecars (#2467) 2024-05-16 19:06:49 +08:00
galaio
6b8cbbe172 sync: fix some sync issues caused by prune-block. (#2466) 2024-05-16 12:07:13 +08:00
setunapo
5ea2ada0ee utils: add check_blobtx.js (#2463) 2024-05-15 18:17:57 +08:00
Fynn
b230a02006 cmd: fix memory leak when big dataset (#2455) 2024-05-15 15:28:57 +08:00
Nathan
86e3a02490 cmd/utils: add a flag to change breathe block interval for testing (#2462) 2024-05-15 15:27:05 +08:00
Nathan
0c0958ff87 eth/handler: check lists in body before broadcast blocks (#2461) 2024-05-15 14:54:25 +08:00
buddho
c577ce3720 Merge pull request #2460 from bnb-chain/develop
merge some PRs for v1.4.7(2nd)
2024-05-14 19:59:57 +08:00
Eric
d436f9e2e8 eth/fetcher/block_fetcher.go: add metrics for import failed block (#2459) 2024-05-14 19:29:43 +08:00
Eric
97c3b9b267 internal/api.go: add choice about not show full blob (#2458) 2024-05-14 17:17:58 +08:00
Nathan
0560685460 core: clearup feynman testflag and rialto code (#2457) 2024-05-14 16:37:17 +08:00
rjl493456442
bf16a39876 ethdb/pebble: print warning log if pebble performance degrades (#29478) 2024-05-14 15:26:21 +08:00
Devon Bear
2c8720016d go.mod: bump pebble db to official release (#29038)
bump pebble
2024-05-14 15:26:21 +08:00
Nathan
f2ec3cc6a5 eth/handler: check blobs before broadcast blocks (#2450) 2024-05-13 17:21:45 +08:00
Martin HS
0a2e1282d2 core/rawdb: add sanity-limit to header accessor (#29534) 2024-05-13 17:21:45 +08:00
Nathan
adb5e8fe86 eth/filters: enforce topic-limit early on filter criterias (#29535) (#2448)
This PR adds a limit of 1000 to the "inner" topics in a filter-criteria

Co-authored-by: Martin HS <martin@swende.se>
2024-05-13 17:21:45 +08:00
Nathan
23f6194fad eth/handler: check blobs before broadcast blocks (#2450) 2024-05-13 16:20:50 +08:00
Martin HS
691d195526 core/rawdb: add sanity-limit to header accessor (#29534) 2024-05-11 10:24:39 +08:00
Nathan
b57c779759 eth/filters: enforce topic-limit early on filter criterias (#29535) (#2448)
This PR adds a limit of 1000 to the "inner" topics in a filter-criteria

Co-authored-by: Martin HS <martin@swende.se>
2024-05-11 10:24:19 +08:00
zzzckck
4ab1c865b2 Merge pull request #2441 from bnb-chain/develop
Draft release v1.4.7
2024-05-10 13:10:56 +08:00
zzzckck
a7d5b02919 release: prepare for release v1.4.7 (#2442)
* release: prepare for release v1.4.7

* code: avoid golang-lint error for TxLookupLimit
2024-05-09 19:47:52 +08:00
zzzckck
1ce9bb044d config: setup Mainnet Tycho(Cancun) hardfork date (#2439)
expected hard fork date:
Mainnet: 2024-06-20 06:05:00 AM UTC
2024-05-09 16:24:04 +08:00
dependabot[bot]
7948950f7a build(deps): bump golang.org/x/net from 0.19.0 to 0.23.0 (#2411) 2024-05-09 16:06:28 +08:00
knowmost
0c101e618a chore: fix some typos (#2433) 2024-05-09 16:05:46 +08:00
zzzckck
571ea2c4b9 nancy: add files .nancy-ignore (#2440) 2024-05-09 15:58:06 +08:00
galaio
7bc5a3353d txpool: limit max gas when mining is enabled; (#2435) 2024-05-09 15:54:31 +08:00
Chris Li
901ea2e0d2 fix: performance issue when load journal (#2438) 2024-05-08 15:36:01 +08:00
buddho
1d81f3316f metrics: add blockInsertMgaspsGauge to trace mgasps (#2396) 2024-04-30 17:58:04 +08:00
zzzckck
43b2ffa63b Merge pull request #2427 from bnb-chain/develop
Draft release v1.4.6
2024-04-29 14:07:29 +08:00
zzzckck
0567715760 release: prepare for release v1.4.6 2024-04-29 10:47:43 +08:00
zzzckck
e32fcf5b93 Revert "github: add branch protect rule (#2343)"
This reverts commit e7c5ce2e94.
2024-04-29 10:47:43 +08:00
zzzckck
e55028d788 metrics: refine the double sign detect code 2024-04-29 10:47:43 +08:00
Satyajit Das
9d8df917b8 metrics: add doublesign counter (#2419) 2024-04-29 09:42:43 +08:00
irrun
9e170972f4 fix: oom caused by non-discarded mev simulation env (#2430) 2024-04-28 19:45:42 +08:00
irrun
ba6726325a feat: recommit bid when newBidCh is empty to maximize mev reward (#2424) 2024-04-28 11:05:09 +08:00
Chris Li
6573254a62 fix: adapt journal for cmd (#2425) 2024-04-28 11:02:14 +08:00
galaio
31d92c50ad chore: add metric & log for blobTx; (#2428) 2024-04-27 06:37:42 +08:00
Eric
7cab9c622c eth/gasprice: add query limit to defend DDOS attack (#2423) 2024-04-25 14:05:12 +08:00
irrun
2a0e399c38 Revert "fix: wrong way to get blob tx sidecar in `BidRuntime.commitTransactio…" (#2418)
This reverts commit 14023fae6d.
2024-04-23 11:41:56 +08:00
forcedebug
182c841374 fix: fix function names (#2416) 2024-04-23 10:06:50 +08:00
Roshan
14023fae6d fix: wrong way to get blob tx sidecar in BidRuntime.commitTransaction (#2417) 2024-04-22 18:46:44 +08:00
Chris Li
d653cda82e feat: adaptive for loading journal file or journal kv during loadJournal (#2406)
* core: check journalType before load journal
* fix: when delete trieJournal delete from kv & file
2024-04-19 16:23:28 +08:00
careworry
4b54601d5c chore: fix some typos in comments (#2408) 2024-04-19 13:17:58 +08:00
Ng Wei Han
3b7f0e4279 cmd, p2p: filter peers by regex on name (#2404) 2024-04-18 16:11:32 +08:00
TechVest
fe1fff8c77 chore: fix some typos in comments (#2399) 2024-04-18 15:43:57 +08:00
Chris Li
c0afdc9a98 core: separated databases for block data (#2227)
* core: use finalized block as the chain freeze indicator (#28683)
* core/rawdb: use max(finality, head-90k) as chain freezing threshold
* core: impl multi database for block data
* core: fix db inspect total size bug
* core: add tips for user who use multi-database
* core: adapt some cmd for multi-database
* core: adapter blockBlobSidecars
* core: fix freezer readHeader bug

---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2024-04-18 15:12:05 +08:00
buddho
fb435eb5f1 fix: allow fast node to rewind after abnormal shutdown (#2401) 2024-04-18 13:45:02 +08:00
irrun
5cc253a2cd fix: NPE (#2403) 2024-04-18 13:25:38 +08:00
buddho
cbcd26c9a9 fix: no import blocks before or equal to the finalized height (#2398) 2024-04-17 14:19:50 +08:00
Chris Li
90eb5b33e8 fix: trieJournal format compatible old db format (#2395) 2024-04-16 11:49:04 +08:00
Ng Wei Han
837de88057 cmd/geth: fix importBlock (#2244) 2024-04-15 16:40:03 +08:00
buddho
b4fb2f6ffc fix: print value instead of pointer in ConfigCompatError (#2391) 2024-04-15 14:46:14 +08:00
dylanhuang
11503edeb2 chore: render system bytecode by go:embed (#2201) 2024-04-15 11:23:23 +08:00
Chris Li
3a6e3c67f2 core/trie: persist TrieJournal to journal file instead of kv database (#2341) 2024-04-15 10:47:54 +08:00
hugehope
335be39905 chore: fix function names in comment (#2390) 2024-04-11 19:01:43 +08:00
irrun
b7972bcd77 feat: greedy merge tx in bid (#2363) 2024-04-11 15:15:46 +08:00
buddho
4bb1bd1a77 deps: update prsym to solve warning about quic-go version (#2389) 2024-04-11 14:11:15 +08:00
269 changed files with 12566 additions and 2427 deletions

20
.github/CODEOWNERS vendored
View File

@@ -1,3 +1,21 @@
# Lines starting with '#' are comments.
# Each line is a file pattern followed by one or more owners.
* @zzzckck @zjubfd
accounts/usbwallet @karalabe
accounts/scwallet @gballet
accounts/abi @gballet @MariusVanDerWijden
cmd/clef @holiman
consensus @karalabe
core/ @karalabe @holiman @rjl493456442
eth/ @karalabe @holiman @rjl493456442
eth/catalyst/ @gballet
eth/tracers/ @s1na
graphql/ @s1na
les/ @zsfelfoldi @rjl493456442
light/ @zsfelfoldi @rjl493456442
node/ @fjl
p2p/ @fjl @zsfelfoldi
rpc/ @fjl @holiman
p2p/simulations @fjl
p2p/protocols @fjl
p2p/testing @fjl
signer/ @holiman

2
.nancy-ignore Normal file
View File

@@ -0,0 +1,2 @@
CVE-2024-34478 # "CWE-754: Improper Check for Unusual or Exceptional Conditions." This vulnerability is BTC only, BSC does not have the issue.
CVE-2024-6104 # "CWE-532: Information Exposure Through Log Files" This is caused by the vulnerabilities go-retryablehttp@v0.7.4, it is only used in cmd devp2p, impact is limited. will upgrade to v0.7.7 later

View File

@@ -1,4 +1,134 @@
# Changelog
## v1.4.12
### BUGFIX
* [\#2557](https://github.com/bnb-chain/bsc/pull/2557) fix: fix state inspect error after pruned state
* [\#2562](https://github.com/bnb-chain/bsc/pull/2562) fix: delete unexpected block
* [\#2566](https://github.com/bnb-chain/bsc/pull/2566) core: avoid to cache block before wroten into db
* [\#2567](https://github.com/bnb-chain/bsc/pull/2567) fix: fix statedb copy
* [\#2574](https://github.com/bnb-chain/bsc/pull/2574) core: adapt highestVerifiedHeader to FastFinality
* [\#2542](https://github.com/bnb-chain/bsc/pull/2542) fix: pruneancient freeze from the previous position when the first time
* [\#2564](https://github.com/bnb-chain/bsc/pull/2564) fix: the bug of blobsidecars and downloader with multi-database
* [\#2582](https://github.com/bnb-chain/bsc/pull/2582) fix: remove delete and dangling side chains in prunefreezer
### FEATURE
* [\#2513](https://github.com/bnb-chain/bsc/pull/2513) cmd/jsutils: add a tool to get performance between a range of blocks
* [\#2569](https://github.com/bnb-chain/bsc/pull/2569) cmd/jsutils: add a tool to get slash count
* [\#2583](https://github.com/bnb-chain/bsc/pull/2583) cmd/jsutill: add log about validator name
### IMPROVEMENT
* [\#2546](https://github.com/bnb-chain/bsc/pull/2546) go.mod: update missing dependency
* [\#2559](https://github.com/bnb-chain/bsc/pull/2559) nancy: ignore go-retryablehttp@v0.7.4 in .nancy-ignore
* [\#2556](https://github.com/bnb-chain/bsc/pull/2556) chore: update greenfield cometbft version
* [\#2561](https://github.com/bnb-chain/bsc/pull/2561) tests: fix unstable test
* [\#2572](https://github.com/bnb-chain/bsc/pull/2572) core: clearup testflag for Cancun and Haber
* [\#2573](https://github.com/bnb-chain/bsc/pull/2573) cmd/utils: support use NetworkId to distinguish chapel when do syncing
* [\#2538](https://github.com/bnb-chain/bsc/pull/2538) feat: enhance bid comparison and reply bidding results && detail logs
* [\#2568](https://github.com/bnb-chain/bsc/pull/2568) core/vote: not vote if too late for next in turn validator
* [\#2576](https://github.com/bnb-chain/bsc/pull/2576) miner/worker: broadcast block immediately once sealed
* [\#2580](https://github.com/bnb-chain/bsc/pull/2580) freezer: Opt freezer env checking
## v1.4.11
### BUGFIX
* [\#2534](https://github.com/bnb-chain/bsc/pull/2534) fix: nil pointer when clear simulating bid
* [\#2535](https://github.com/bnb-chain/bsc/pull/2535) upgrade: add HaberFix hardfork
## v1.4.10
### FEATURE
NA
### IMPROVEMENT
* [\#2512](https://github.com/bnb-chain/bsc/pull/2512) feat: add mev helper params and func
* [\#2508](https://github.com/bnb-chain/bsc/pull/2508) perf: speedup pbss trienode read
* [\#2509](https://github.com/bnb-chain/bsc/pull/2509) perf: optimize chain commit performance for multi-database
* [\#2451](https://github.com/bnb-chain/bsc/pull/2451) core/forkchoice: improve stability when inturn block not generate
### BUGFIX
* [\#2518](https://github.com/bnb-chain/bsc/pull/2518) fix: remove zero gasprice check for BSC
* [\#2519](https://github.com/bnb-chain/bsc/pull/2519) UT: random failure of TestSnapSyncWithBlobs
* [\#2515](https://github.com/bnb-chain/bsc/pull/2515) fix getBlobSidecars by ethclient
* [\#2525](https://github.com/bnb-chain/bsc/pull/2525) fix: ensure empty withdrawals after cancun before broadcast
## v1.4.9
### FEATURE
* [\#2463](https://github.com/bnb-chain/bsc/pull/2463) utils: add check_blobtx.js
* [\#2470](https://github.com/bnb-chain/bsc/pull/2470) jsutils: faucet successful requests within blocks
* [\#2467](https://github.com/bnb-chain/bsc/pull/2467) internal/ethapi: add optional parameter for blobSidecars
### IMPROVEMENT
* [\#2462](https://github.com/bnb-chain/bsc/pull/2462) cmd/utils: add a flag to change breathe block interval for testing
* [\#2497](https://github.com/bnb-chain/bsc/pull/2497) params/config: add Bohr hardfork
* [\#2479](https://github.com/bnb-chain/bsc/pull/2479) dev: ensure consistency in BPS bundle result
### BUGFIX
* [\#2461](https://github.com/bnb-chain/bsc/pull/2461) eth/handler: check lists in body before broadcast blocks
* [\#2455](https://github.com/bnb-chain/bsc/pull/2455) cmd: fix memory leak when big dataset
* [\#2466](https://github.com/bnb-chain/bsc/pull/2466) sync: fix some sync issues caused by prune-block.
* [\#2475](https://github.com/bnb-chain/bsc/pull/2475) fix: move mev op to MinerAPI & add command to console
* [\#2473](https://github.com/bnb-chain/bsc/pull/2473) fix: limit the gas price of the mev bid
* [\#2484](https://github.com/bnb-chain/bsc/pull/2484) fix: fix inspect database error
* [\#2481](https://github.com/bnb-chain/bsc/pull/2481) fix: keep 9W blocks in ancient db when prune block
* [\#2495](https://github.com/bnb-chain/bsc/pull/2495) fix: add an empty freeze db
* [\#2507](https://github.com/bnb-chain/bsc/pull/2507) fix: waiting for the last simulation before pick best bid
## v1.4.8
### FEATURE
* [\#2483](https://github.com/bnb-chain/bsc/pull/2483) core/vm: add secp256r1 into PrecompiledContractsHaber
* [\#2400](https://github.com/bnb-chain/bsc/pull/2400) RIP-7212: Precompile for secp256r1 Curve Support
### IMPROVEMENT
NA
### BUGFIX
NA
## v1.4.7
### FEATURE
* [\#2439](https://github.com/bnb-chain/bsc/pull/2439) config: setup Mainnet Tycho(Cancun) hardfork date
### IMPROVEMENT
* [\#2396](https://github.com/bnb-chain/bsc/pull/2396) metrics: add blockInsertMgaspsGauge to trace mgasps
* [\#2411](https://github.com/bnb-chain/bsc/pull/2411) build(deps): bump golang.org/x/net from 0.19.0 to 0.23.0
* [\#2435](https://github.com/bnb-chain/bsc/pull/2435) txpool: limit max gas when mining is enabled
* [\#2438](https://github.com/bnb-chain/bsc/pull/2438) fix: performance issue when load journal
* [\#2440](https://github.com/bnb-chain/bsc/pull/2440) nancy: add files .nancy-ignore
### BUGFIX
NA
## v1.4.6
### FEATURE
* [\#2227](https://github.com/bnb-chain/bsc/pull/2227) core: separated databases for block data
* [\#2404](https://github.com/bnb-chain/bsc/pull/2404) cmd, p2p: filter peers by regex on name
### IMPROVEMENT
* [\#2201](https://github.com/bnb-chain/bsc/pull/2201) chore: render system bytecode by go:embed
* [\#2363](https://github.com/bnb-chain/bsc/pull/2363) feat: greedy merge tx in bid
* [\#2389](https://github.com/bnb-chain/bsc/pull/2389) deps: update prsym to solve warning about quic-go version
* [\#2341](https://github.com/bnb-chain/bsc/pull/2341) core/trie: persist TrieJournal to journal file instead of kv database
* [\#2395](https://github.com/bnb-chain/bsc/pull/2395) fix: trieJournal format compatible old db format
* [\#2406](https://github.com/bnb-chain/bsc/pull/2406) feat: adaptive for loading journal file or journal kv during loadJournal
* [\#2390](https://github.com/bnb-chain/bsc/pull/2390) chore: fix function names in comment
* [\#2399](https://github.com/bnb-chain/bsc/pull/2399) chore: fix some typos in comments
* [\#2408](https://github.com/bnb-chain/bsc/pull/2408) chore: fix some typos in comments
* [\#2416](https://github.com/bnb-chain/bsc/pull/2416) fix: fix function names
* [\#2424](https://github.com/bnb-chain/bsc/pull/2424) feat: recommit bid when newBidCh is empty to maximize mev reward
* [\#2430](https://github.com/bnb-chain/bsc/pull/2430) fix: oom caused by non-discarded mev simulation env
* [\#2428](https://github.com/bnb-chain/bsc/pull/2428) chore: add metric & log for blobTx
* [\#2419](https://github.com/bnb-chain/bsc/pull/2419) metrics: add doublesign counter
### BUGFIX
* [\#2244](https://github.com/bnb-chain/bsc/pull/2244) cmd/geth: fix importBlock
* [\#2391](https://github.com/bnb-chain/bsc/pull/2391) fix: print value instead of pointer in ConfigCompatError
* [\#2398](https://github.com/bnb-chain/bsc/pull/2398) fix: no import blocks before or equal to the finalized height
* [\#2401](https://github.com/bnb-chain/bsc/pull/2401) fix: allow fast node to rewind after abnormal shutdown
* [\#2403](https://github.com/bnb-chain/bsc/pull/2403) fix: NPE
* [\#2423](https://github.com/bnb-chain/bsc/pull/2423) eth/gasprice: add query limit to defend DDOS attack
* [\#2425](https://github.com/bnb-chain/bsc/pull/2425) fix: adapt journal for cmd
## v1.4.5
### FEATURE
* [\#2378](https://github.com/bnb-chain/bsc/pull/2378) config: setup Testnet Tycho(Cancun) hardfork date

View File

@@ -26,7 +26,7 @@ COPY --from=builder /go-ethereum/build/bin/* /usr/local/bin/
EXPOSE 8545 8546 30303 30303/udp
# Add some metadata labels to help programatic image consumption
# Add some metadata labels to help programmatic image consumption
ARG COMMIT=""
ARG VERSION=""
ARG BUILDNUM=""

View File

@@ -36,7 +36,7 @@ To combine DPoS and PoA for consensus, BNB Smart Chain implement a novel consens
2. Validators take turns to produce blocks in a PoA manner, similar to Ethereum's Clique consensus engine.
3. Validator set are elected in and out based on a staking based governance on BNB Beacon Chain.
4. The validator set change is relayed via a cross-chain communication mechanism.
5. Parlia consensus engine will interact with a set of [system contracts](https://docs.bnbchain.org/docs/learn/system-contract) to achieve liveness slash, revenue distributing and validator set renewing func.
5. Parlia consensus engine will interact with a set of [system contracts](https://docs.bnbchain.org/bnb-smart-chain/staking/overview/#system-contracts) to achieve liveness slash, revenue distributing and validator set renewing func.
### Light Client of BNB Beacon Chain
@@ -149,13 +149,11 @@ unzip testnet.zip
#### 3. Download snapshot
Download latest chaindata snapshot from [here](https://github.com/bnb-chain/bsc-snapshots). Follow the guide to structure your files.
Note: If you encounter difficulties downloading the chaindata snapshot and prefer to synchronize from the genesis block on the Chapel testnet, remember to include the additional flag `--chapel` when initially launching Geth.
#### 4. Start a full node
```shell
./geth --config ./config.toml --datadir ./node --cache 8000 --rpc.allow-unprotected-txs --history.transactions 0
## It is recommand to run fullnode with `--tries-verify-mode none` if you want high performance and care little about state consistency
## It is recommend to run fullnode with `--tries-verify-mode none` if you want high performance and care little about state consistency
## It will run with Hash-Base Storage Scheme by default
./geth --config ./config.toml --datadir ./node --cache 8000 --rpc.allow-unprotected-txs --history.transactions 0 --tries-verify-mode none
@@ -183,7 +181,7 @@ This tool is optional and if you leave it out you can always attach to an alread
#### 7. More
More details about [running a node](https://docs.bnbchain.org/docs/validator/fullnode) and [becoming a validator](https://docs.bnbchain.org/docs/validator/create-val)
More details about [running a node](https://docs.bnbchain.org/bnb-smart-chain/developers/node_operators/full_node/) and [becoming a validator](https://docs.bnbchain.org/bnb-smart-chain/validator/create-val/)
*Note: Although some internal protective measures prevent transactions from
crossing over between the main network and test network, you should always

View File

@@ -51,7 +51,7 @@ var (
}
)
// waitWatcherStarts waits up to 1s for the keystore watcher to start.
// waitWatcherStart waits up to 1s for the keystore watcher to start.
func waitWatcherStart(ks *KeyStore) bool {
// On systems where file watch is not supported, just return "ok".
if !ks.cache.watcher.enabled() {

View File

@@ -343,7 +343,7 @@ func TestWalletNotifications(t *testing.T) {
checkEvents(t, wantEvents, events)
}
// TestImportExport tests the import functionality of a keystore.
// TestImportECDSA tests the import functionality of a keystore.
func TestImportECDSA(t *testing.T) {
t.Parallel()
_, ks := tmpKeyStore(t, true)
@@ -362,7 +362,7 @@ func TestImportECDSA(t *testing.T) {
}
}
// TestImportECDSA tests the import and export functionality of a keystore.
// TestImportExport tests the import and export functionality of a keystore.
func TestImportExport(t *testing.T) {
t.Parallel()
_, ks := tmpKeyStore(t, true)

View File

@@ -525,7 +525,7 @@ func (f *faucet) apiHandler(w http.ResponseWriter, r *http.Request) {
}
f.lock.Unlock()
// Send an error if too frequent funding, othewise a success
// Send an error if too frequent funding, otherwise a success
if !fund {
if err = sendError(wsconn, fmt.Errorf("%s left until next allowance", common.PrettyDuration(time.Until(timeout)))); err != nil { // nolint: gosimple
log.Warn("Failed to send funding error to client", "err", err)
@@ -642,7 +642,7 @@ func (f *faucet) loop() {
f.lock.RUnlock()
}
}()
// Wait for various events and assing to the appropriate background threads
// Wait for various events and assign to the appropriate background threads
for {
select {
case head := <-heads:

View File

@@ -48,7 +48,7 @@ func TestAttachWithHeaders(t *testing.T) {
// This is fixed in a follow-up PR.
}
// TestAttachWithHeaders tests that 'geth db --remotedb' with custom headers works, i.e
// TestRemoteDbWithHeaders tests that 'geth db --remotedb' with custom headers works, i.e
// that custom headers are forwarded to the target.
func TestRemoteDbWithHeaders(t *testing.T) {
t.Parallel()

View File

@@ -12,16 +12,16 @@ import (
"github.com/google/uuid"
"github.com/logrusorgru/aurora"
"github.com/prysmaticlabs/prysm/v4/crypto/bls"
"github.com/prysmaticlabs/prysm/v4/encoding/bytesutil"
"github.com/prysmaticlabs/prysm/v4/io/prompt"
validatorpb "github.com/prysmaticlabs/prysm/v4/proto/prysm/v1alpha1/validator-client"
"github.com/prysmaticlabs/prysm/v4/validator/accounts"
"github.com/prysmaticlabs/prysm/v4/validator/accounts/iface"
"github.com/prysmaticlabs/prysm/v4/validator/accounts/petnames"
"github.com/prysmaticlabs/prysm/v4/validator/accounts/wallet"
"github.com/prysmaticlabs/prysm/v4/validator/keymanager"
"github.com/prysmaticlabs/prysm/v4/validator/keymanager/local"
"github.com/prysmaticlabs/prysm/v5/crypto/bls"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
"github.com/prysmaticlabs/prysm/v5/io/prompt"
validatorpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1/validator-client"
"github.com/prysmaticlabs/prysm/v5/validator/accounts"
"github.com/prysmaticlabs/prysm/v5/validator/accounts/iface"
"github.com/prysmaticlabs/prysm/v5/validator/accounts/petnames"
"github.com/prysmaticlabs/prysm/v5/validator/accounts/wallet"
"github.com/prysmaticlabs/prysm/v5/validator/keymanager"
"github.com/prysmaticlabs/prysm/v5/validator/keymanager/local"
"github.com/urfave/cli/v2"
keystorev4 "github.com/wealdtech/go-eth2-wallet-encryptor-keystorev4"

View File

@@ -41,6 +41,7 @@ import (
"github.com/ethereum/go-ethereum/core/state"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/eth"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/internal/era"
"github.com/ethereum/go-ethereum/internal/flags"
@@ -61,8 +62,9 @@ var (
ArgsUsage: "<genesisPath>",
Flags: flags.Merge([]cli.Flag{
utils.CachePreimagesFlag,
utils.OverrideCancun,
utils.OverrideBohr,
utils.OverrideVerkle,
utils.MultiDataBaseFlag,
}, utils.DatabaseFlags),
Description: `
The init command initializes a new genesis block and definition for the network.
@@ -251,9 +253,9 @@ func initGenesis(ctx *cli.Context) error {
defer stack.Close()
var overrides core.ChainOverrides
if ctx.IsSet(utils.OverrideCancun.Name) {
v := ctx.Uint64(utils.OverrideCancun.Name)
overrides.OverrideCancun = &v
if ctx.IsSet(utils.OverrideBohr.Name) {
v := ctx.Uint64(utils.OverrideBohr.Name)
overrides.OverrideBohr = &v
}
if ctx.IsSet(utils.OverrideVerkle.Name) {
v := ctx.Uint64(utils.OverrideVerkle.Name)
@@ -267,15 +269,21 @@ func initGenesis(ctx *cli.Context) error {
defer chaindb.Close()
// if the trie data dir has been set, new trie db with a new state database
if ctx.IsSet(utils.SeparateDBFlag.Name) {
if ctx.IsSet(utils.MultiDataBaseFlag.Name) {
statediskdb, dbErr := stack.OpenDatabaseWithFreezer(name+"/state", 0, 0, "", "", false, false, false, false)
if dbErr != nil {
utils.Fatalf("Failed to open separate trie database: %v", dbErr)
}
chaindb.SetStateStore(statediskdb)
blockdb, err := stack.OpenDatabaseWithFreezer(name+"/block", 0, 0, "", "", false, false, false, false)
if err != nil {
utils.Fatalf("Failed to open separate block database: %v", err)
}
chaindb.SetBlockStore(blockdb)
log.Warn("Multi-database is an experimental feature")
}
triedb := utils.MakeTrieDatabase(ctx, chaindb, ctx.Bool(utils.CachePreimagesFlag.Name), false, genesis.IsVerkle())
triedb := utils.MakeTrieDatabase(ctx, stack, chaindb, ctx.Bool(utils.CachePreimagesFlag.Name), false, genesis.IsVerkle())
defer triedb.Close()
_, hash, err := core.SetupGenesisBlockWithOverride(chaindb, triedb, genesis, &overrides)
@@ -473,6 +481,13 @@ func dumpGenesis(ctx *cli.Context) error {
}
continue
}
// set the separate state & block database
if stack.CheckIfMultiDataBase() && err == nil {
stateDiskDb := utils.MakeStateDataBase(ctx, stack, true, false)
db.SetStateStore(stateDiskDb)
blockDb := utils.MakeBlockDatabase(ctx, stack, true, false)
db.SetBlockStore(blockDb)
}
genesis, err := core.ReadGenesis(db)
if err != nil {
utils.Fatalf("failed to read genesis: %s", err)
@@ -500,10 +515,16 @@ func importChain(ctx *cli.Context) error {
// Start system runtime metrics collection
go metrics.CollectProcessMetrics(3 * time.Second)
stack, _ := makeConfigNode(ctx)
stack, cfg := makeConfigNode(ctx)
defer stack.Close()
chain, db := utils.MakeChain(ctx, stack, false)
backend, err := eth.New(stack, &cfg.Eth)
if err != nil {
return err
}
chain := backend.BlockChain()
db := backend.ChainDb()
defer db.Close()
// Start periodically gathering memory profiles
@@ -758,7 +779,7 @@ func parseDumpConfig(ctx *cli.Context, stack *node.Node) (*state.DumpConfig, eth
} else {
// Use latest
if scheme == rawdb.PathScheme {
triedb := triedb.NewDatabase(db, &triedb.Config{PathDB: pathdb.ReadOnly})
triedb := triedb.NewDatabase(db, &triedb.Config{PathDB: utils.PathDBConfigAddJournalFilePath(stack, pathdb.ReadOnly)})
defer triedb.Close()
if stateRoot := triedb.Head(); stateRoot != (common.Hash{}) {
header.Root = stateRoot
@@ -808,10 +829,11 @@ func dump(ctx *cli.Context) error {
if err != nil {
return err
}
triedb := utils.MakeTrieDatabase(ctx, db, true, true, false) // always enable preimage lookup
defer db.Close()
triedb := utils.MakeTrieDatabase(ctx, stack, db, true, true, false) // always enable preimage lookup
defer triedb.Close()
state, err := state.New(root, state.NewDatabaseWithNodeDB(db, triedb), nil)
state, err := state.New(root, state.NewDatabaseWithNodeDB(db, triedb, false), nil)
if err != nil {
return err
}
@@ -828,7 +850,7 @@ func dumpAllRootHashInPath(ctx *cli.Context) error {
defer stack.Close()
db := utils.MakeChainDatabase(ctx, stack, true, false)
defer db.Close()
triedb := triedb.NewDatabase(db, &triedb.Config{PathDB: pathdb.ReadOnly})
triedb := triedb.NewDatabase(db, &triedb.Config{PathDB: utils.PathDBConfigAddJournalFilePath(stack, pathdb.ReadOnly)})
defer triedb.Close()
scheme, err := rawdb.ParseStateScheme(ctx.String(utils.StateSchemeFlag.Name), db)

View File

@@ -185,22 +185,14 @@ func makeFullNode(ctx *cli.Context) (*node.Node, ethapi.Backend) {
params.RialtoGenesisHash = common.HexToHash(v)
}
if ctx.IsSet(utils.OverrideCancun.Name) {
v := ctx.Uint64(utils.OverrideCancun.Name)
cfg.Eth.OverrideCancun = &v
if ctx.IsSet(utils.OverrideBohr.Name) {
v := ctx.Uint64(utils.OverrideBohr.Name)
cfg.Eth.OverrideBohr = &v
}
if ctx.IsSet(utils.OverrideVerkle.Name) {
v := ctx.Uint64(utils.OverrideVerkle.Name)
cfg.Eth.OverrideVerkle = &v
}
if ctx.IsSet(utils.OverrideFeynman.Name) {
v := ctx.Uint64(utils.OverrideFeynman.Name)
cfg.Eth.OverrideFeynman = &v
}
if ctx.IsSet(utils.OverrideFeynmanFix.Name) {
v := ctx.Uint64(utils.OverrideFeynmanFix.Name)
cfg.Eth.OverrideFeynmanFix = &v
}
if ctx.IsSet(utils.OverrideFullImmutabilityThreshold.Name) {
params.FullImmutabilityThreshold = ctx.Uint64(utils.OverrideFullImmutabilityThreshold.Name)
downloader.FullMaxForkAncestry = ctx.Uint64(utils.OverrideFullImmutabilityThreshold.Name)
@@ -211,9 +203,10 @@ func makeFullNode(ctx *cli.Context) (*node.Node, ethapi.Backend) {
if ctx.IsSet(utils.OverrideDefaultExtraReserveForBlobRequests.Name) {
params.DefaultExtraReserveForBlobRequests = ctx.Uint64(utils.OverrideDefaultExtraReserveForBlobRequests.Name)
}
if ctx.IsSet(utils.SeparateDBFlag.Name) && !stack.IsSeparatedDB() {
utils.Fatalf("Failed to locate separate database subdirectory when separatedb parameter has been set")
if ctx.IsSet(utils.OverrideBreatheBlockInterval.Name) {
params.BreatheBlockInterval = ctx.Uint64(utils.OverrideBreatheBlockInterval.Name)
}
backend, eth := utils.RegisterEthService(stack, &cfg.Eth)
// Create gauge with geth system and build information

View File

@@ -18,6 +18,7 @@ package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"math"
@@ -34,6 +35,7 @@ import (
"github.com/ethereum/go-ethereum/common/hexutil"
"github.com/ethereum/go-ethereum/console/prompt"
"github.com/ethereum/go-ethereum/core/rawdb"
"github.com/ethereum/go-ethereum/core/state"
"github.com/ethereum/go-ethereum/core/state/snapshot"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/ethdb"
@@ -91,6 +93,9 @@ Remove blockchain and state databases`,
dbHbss2PbssCmd,
dbTrieGetCmd,
dbTrieDeleteCmd,
getVersionDBState,
getHashDBState,
diffDebugStateDB,
},
}
dbInspectCmd = &cli.Command{
@@ -106,12 +111,12 @@ Remove blockchain and state databases`,
dbInspectTrieCmd = &cli.Command{
Action: inspectTrie,
Name: "inspect-trie",
ArgsUsage: "<blocknum> <jobnum>",
ArgsUsage: "<blocknum> <jobnum> <topn>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.SyncModeFlag,
},
Usage: "Inspect the MPT tree of the account and contract.",
Usage: "Inspect the MPT tree of the account and contract. 'blocknum' can be latest/snapshot/number. 'topn' means output the top N storage tries info ranked by the total number of TrieNodes",
Description: `This commands iterates the entrie WorldState.`,
}
dbCheckStateContentCmd = &cli.Command{
@@ -286,8 +291,141 @@ WARNING: This is a low-level operation which may cause database corruption!`,
Description: `This commands will read current offset from kvdb, which is the current offset and starting BlockNumber
of ancientStore, will also displays the reserved number of blocks in ancientStore `,
}
getVersionDBState = &cli.Command{
Action: getDebugVersionState,
Name: "get-debug-version-state",
Flags: []cli.Flag{
utils.VersionStateDirFlag,
utils.BlockNumber,
},
}
getHashDBState = &cli.Command{
Action: getDebugHashState,
Name: "get-debug-hash-state",
Flags: []cli.Flag{
utils.HashStateDirFlag,
utils.BlockNumber,
},
}
diffDebugStateDB = &cli.Command{
Action: diffDebugState,
Name: "diff-debug-state",
Flags: []cli.Flag{
utils.VersionStateDirFlag,
utils.HashStateDirFlag,
utils.BlockNumber,
},
}
)
func diffDebugState(ctx *cli.Context) error {
if !ctx.IsSet(utils.VersionStateDirFlag.Name) {
return fmt.Errorf("please set `--versionstatedir` flag")
}
if !ctx.IsSet(utils.BlockNumber.Name) {
return fmt.Errorf("please set `--block` flag")
}
if !ctx.IsSet(utils.HashStateDirFlag.Name) {
return fmt.Errorf("please set `--hashstatedir` flag")
}
verDir := ctx.String(utils.VersionStateDirFlag.Name)
hasDir := ctx.String(utils.HashStateDirFlag.Name)
block := ctx.Int64(utils.BlockNumber.Name)
vdb, err := rawdb.Open(rawdb.OpenOptions{
ReadOnly: true,
Type: "pebble",
Directory: verDir,
})
if err != nil {
return err
}
verData, err := vdb.Get(state.DebugVersionStateKey(block))
if err != nil {
return err
}
verDebugState := &state.DebugVersionState{}
err = json.Unmarshal(verData, verDebugState)
if err != nil {
return nil
}
hdb, err := rawdb.Open(rawdb.OpenOptions{
ReadOnly: true,
Type: "pebble",
Directory: hasDir,
})
if err != nil {
return err
}
hashData, err := hdb.Get(state.DebugHashStateKey(block))
if err != nil {
return err
}
hasDebugState := &state.DebugHashState{}
err = json.Unmarshal(hashData, hasDebugState)
if err != nil {
return err
}
res := state.GenerateDebugStateDiff(verDebugState, hasDebugState)
fmt.Println(res)
return nil
}
func getDebugVersionState(ctx *cli.Context) error {
if !ctx.IsSet(utils.VersionStateDirFlag.Name) {
return fmt.Errorf("please set `--versionstatedir` flag")
}
if !ctx.IsSet(utils.BlockNumber.Name) {
return fmt.Errorf("please set `--block` flag")
}
dir := ctx.String(utils.VersionStateDirFlag.Name)
block := ctx.Int64(utils.BlockNumber.Name)
db, err := rawdb.Open(rawdb.OpenOptions{
ReadOnly: true,
Type: "pebble",
Directory: dir,
})
if err != nil {
return err
}
data, err := db.Get(state.DebugVersionStateKey(block))
if err != nil {
return err
}
fmt.Println(string(data))
return nil
}
func getDebugHashState(ctx *cli.Context) error {
if !ctx.IsSet(utils.HashStateDirFlag.Name) {
return fmt.Errorf("please set `--hashstatedir` flag")
}
if !ctx.IsSet(utils.BlockNumber.Name) {
return fmt.Errorf("please set `--block` flag")
}
dir := ctx.String(utils.HashStateDirFlag.Name)
block := ctx.Int64(utils.BlockNumber.Name)
db, err := rawdb.Open(rawdb.OpenOptions{
ReadOnly: true,
Type: "pebble",
Directory: dir,
})
if err != nil {
return err
}
data, err := db.Get(state.DebugHashStateKey(block))
if err != nil {
return err
}
fmt.Println(string(data))
return nil
}
func removeDB(ctx *cli.Context) error {
stack, config := makeConfigNode(ctx)
@@ -386,6 +524,7 @@ func inspectTrie(ctx *cli.Context) error {
blockNumber uint64
trieRootHash common.Hash
jobnum uint64
topN uint64
)
stack, _ := makeConfigNode(ctx)
@@ -405,24 +544,37 @@ func inspectTrie(ctx *cli.Context) error {
var err error
blockNumber, err = strconv.ParseUint(ctx.Args().Get(0), 10, 64)
if err != nil {
return fmt.Errorf("failed to Parse blocknum, Args[0]: %v, err: %v", ctx.Args().Get(0), err)
return fmt.Errorf("failed to parse blocknum, Args[0]: %v, err: %v", ctx.Args().Get(0), err)
}
}
if ctx.NArg() == 1 {
jobnum = 1000
topN = 10
} else if ctx.NArg() == 2 {
var err error
jobnum, err = strconv.ParseUint(ctx.Args().Get(1), 10, 64)
if err != nil {
return fmt.Errorf("failed to parse jobnum, Args[1]: %v, err: %v", ctx.Args().Get(1), err)
}
topN = 10
} else {
var err error
jobnum, err = strconv.ParseUint(ctx.Args().Get(1), 10, 64)
if err != nil {
return fmt.Errorf("failed to Parse jobnum, Args[1]: %v, err: %v", ctx.Args().Get(1), err)
return fmt.Errorf("failed to parse jobnum, Args[1]: %v, err: %v", ctx.Args().Get(1), err)
}
topN, err = strconv.ParseUint(ctx.Args().Get(2), 10, 64)
if err != nil {
return fmt.Errorf("failed to parse topn, Args[1]: %v, err: %v", ctx.Args().Get(1), err)
}
}
if blockNumber != math.MaxUint64 {
headerBlockHash = rawdb.ReadCanonicalHash(db, blockNumber)
if headerBlockHash == (common.Hash{}) {
return errors.New("ReadHeadBlockHash empry hash")
return errors.New("ReadHeadBlockHash empty hash")
}
blockHeader := rawdb.ReadHeader(db, headerBlockHash, blockNumber)
trieRootHash = blockHeader.Root
@@ -436,7 +588,8 @@ func inspectTrie(ctx *cli.Context) error {
var config *triedb.Config
if dbScheme == rawdb.PathScheme {
config = &triedb.Config{
PathDB: pathdb.ReadOnly,
PathDB: utils.PathDBConfigAddJournalFilePath(stack, pathdb.ReadOnly),
Cache: 0,
}
} else if dbScheme == rawdb.HashScheme {
config = triedb.HashDefaults
@@ -448,7 +601,7 @@ func inspectTrie(ctx *cli.Context) error {
fmt.Printf("fail to new trie tree, err: %v, rootHash: %v\n", err, trieRootHash.String())
return err
}
theInspect, err := trie.NewInspector(theTrie, triedb, trieRootHash, blockNumber, jobnum)
theInspect, err := trie.NewInspector(theTrie, triedb, trieRootHash, blockNumber, jobnum, int(topN))
if err != nil {
return err
}
@@ -493,7 +646,7 @@ func ancientInspect(ctx *cli.Context) error {
stack, _ := makeConfigNode(ctx)
defer stack.Close()
db := utils.MakeChainDatabase(ctx, stack, true, true)
db := utils.MakeChainDatabase(ctx, stack, true, false)
defer db.Close()
return rawdb.AncientInspect(db)
}
@@ -519,7 +672,7 @@ func checkStateContent(ctx *cli.Context) error {
db := utils.MakeChainDatabase(ctx, stack, true, false)
defer db.Close()
var (
it = rawdb.NewKeyLengthIterator(db.NewIterator(prefix, start), 32)
it ethdb.Iterator
hasher = crypto.NewKeccakState()
got = make([]byte, 32)
errs int
@@ -527,6 +680,11 @@ func checkStateContent(ctx *cli.Context) error {
startTime = time.Now()
lastLog = time.Now()
)
if stack.CheckIfMultiDataBase() {
it = rawdb.NewKeyLengthIterator(db.StateStore().NewIterator(prefix, start), 32)
} else {
it = rawdb.NewKeyLengthIterator(db.NewIterator(prefix, start), 32)
}
for it.Next() {
count++
k := it.Key()
@@ -573,9 +731,11 @@ func dbStats(ctx *cli.Context) error {
defer db.Close()
showLeveldbStats(db)
if db.StateStore() != nil {
if stack.CheckIfMultiDataBase() {
fmt.Println("show stats of state store")
showLeveldbStats(db.StateStore())
fmt.Println("show stats of block store")
showLeveldbStats(db.BlockStore())
}
return nil
@@ -591,10 +751,11 @@ func dbCompact(ctx *cli.Context) error {
log.Info("Stats before compaction")
showLeveldbStats(db)
statediskdb := db.StateStore()
if statediskdb != nil {
if stack.CheckIfMultiDataBase() {
fmt.Println("show stats of state store")
showLeveldbStats(statediskdb)
showLeveldbStats(db.StateStore())
fmt.Println("show stats of block store")
showLeveldbStats(db.BlockStore())
}
log.Info("Triggering compaction")
@@ -603,8 +764,12 @@ func dbCompact(ctx *cli.Context) error {
return err
}
if statediskdb != nil {
if err := statediskdb.Compact(nil, nil); err != nil {
if stack.CheckIfMultiDataBase() {
if err := db.StateStore().Compact(nil, nil); err != nil {
log.Error("Compact err", "error", err)
return err
}
if err := db.BlockStore().Compact(nil, nil); err != nil {
log.Error("Compact err", "error", err)
return err
}
@@ -612,9 +777,11 @@ func dbCompact(ctx *cli.Context) error {
log.Info("Stats after compaction")
showLeveldbStats(db)
if statediskdb != nil {
if stack.CheckIfMultiDataBase() {
fmt.Println("show stats of state store after compaction")
showLeveldbStats(statediskdb)
showLeveldbStats(db.StateStore())
fmt.Println("show stats of block store after compaction")
showLeveldbStats(db.BlockStore())
}
return nil
}
@@ -635,18 +802,18 @@ func dbGet(ctx *cli.Context) error {
log.Info("Could not decode the key", "error", err)
return err
}
statediskdb := db.StateStore()
data, err := db.Get(key)
if err != nil {
// if separate trie db exist, try to get it from separate db
if statediskdb != nil {
statedata, dberr := statediskdb.Get(key)
if dberr == nil {
fmt.Printf("key %#x: %#x\n", key, statedata)
return nil
}
opDb := db
if stack.CheckIfMultiDataBase() {
keyType := rawdb.DataTypeByKey(key)
if keyType == rawdb.StateDataType {
opDb = db.StateStore()
} else if keyType == rawdb.BlockDataType {
opDb = db.BlockStore()
}
}
data, err := opDb.Get(key)
if err != nil {
log.Info("Get operation failed", "key", fmt.Sprintf("%#x", key), "error", err)
return err
}
@@ -809,11 +976,21 @@ func dbDelete(ctx *cli.Context) error {
log.Info("Could not decode the key", "error", err)
return err
}
data, err := db.Get(key)
opDb := db
if stack.CheckIfMultiDataBase() {
keyType := rawdb.DataTypeByKey(key)
if keyType == rawdb.StateDataType {
opDb = db.StateStore()
} else if keyType == rawdb.BlockDataType {
opDb = db.BlockStore()
}
}
data, err := opDb.Get(key)
if err == nil {
fmt.Printf("Previous value: %#x\n", data)
}
if err = db.Delete(key); err != nil {
if err = opDb.Delete(key); err != nil {
log.Info("Delete operation returned an error", "key", fmt.Sprintf("%#x", key), "error", err)
return err
}
@@ -923,11 +1100,22 @@ func dbPut(ctx *cli.Context) error {
log.Info("Could not decode the value", "error", err)
return err
}
data, err = db.Get(key)
opDb := db
if stack.CheckIfMultiDataBase() {
keyType := rawdb.DataTypeByKey(key)
if keyType == rawdb.StateDataType {
opDb = db.StateStore()
} else if keyType == rawdb.BlockDataType {
opDb = db.BlockStore()
}
}
data, err = opDb.Get(key)
if err == nil {
fmt.Printf("Previous value: %#x\n", data)
}
return db.Put(key, value)
return opDb.Put(key, value)
}
// dbDumpTrie shows the key-value slots of a given storage trie
@@ -940,8 +1128,7 @@ func dbDumpTrie(ctx *cli.Context) error {
db := utils.MakeChainDatabase(ctx, stack, true, false)
defer db.Close()
triedb := utils.MakeTrieDatabase(ctx, db, false, true, false)
triedb := utils.MakeTrieDatabase(ctx, stack, db, false, true, false)
defer triedb.Close()
var (
@@ -1019,7 +1206,7 @@ func freezerInspect(ctx *cli.Context) error {
stack, _ := makeConfigNode(ctx)
ancient := stack.ResolveAncient("chaindata", ctx.String(utils.AncientFlag.Name))
stack.Close()
return rawdb.InspectFreezerTable(ancient, freezer, table, start, end)
return rawdb.InspectFreezerTable(ancient, freezer, table, start, end, stack.CheckIfMultiDataBase())
}
func importLDBdata(ctx *cli.Context) error {
@@ -1159,7 +1346,7 @@ func showMetaData(ctx *cli.Context) error {
defer stack.Close()
db := utils.MakeChainDatabase(ctx, stack, true, false)
defer db.Close()
ancients, err := db.Ancients()
ancients, err := db.BlockStore().Ancients()
if err != nil {
fmt.Fprintf(os.Stderr, "Error accessing ancients: %v", err)
}
@@ -1206,7 +1393,7 @@ func hbss2pbss(ctx *cli.Context) error {
defer stack.Close()
db := utils.MakeChainDatabase(ctx, stack, false, false)
db.Sync()
db.BlockStore().Sync()
stateDiskDb := db.StateStore()
defer db.Close()

View File

@@ -25,6 +25,8 @@ import (
"strings"
"time"
"github.com/ethereum/go-ethereum/params"
"github.com/ethereum/go-ethereum/accounts"
"github.com/ethereum/go-ethereum/accounts/keystore"
"github.com/ethereum/go-ethereum/cmd/utils"
@@ -70,13 +72,12 @@ var (
utils.USBFlag,
utils.SmartCardDaemonPathFlag,
utils.RialtoHash,
utils.OverrideCancun,
utils.OverrideBohr,
utils.OverrideVerkle,
utils.OverrideFeynman,
utils.OverrideFeynmanFix,
utils.OverrideFullImmutabilityThreshold,
utils.OverrideMinBlocksForBlobRequests,
utils.OverrideDefaultExtraReserveForBlobRequests,
utils.OverrideBreatheBlockInterval,
utils.EnablePersonal,
utils.TxPoolLocalsFlag,
utils.TxPoolNoLocalsFlag,
@@ -103,6 +104,7 @@ var (
utils.TransactionHistoryFlag,
utils.StateHistoryFlag,
utils.PathDBSyncFlag,
utils.JournalFileFlag,
utils.LightServeFlag, // deprecated
utils.LightIngressFlag, // deprecated
utils.LightEgressFlag, // deprecated
@@ -123,6 +125,7 @@ var (
utils.CacheSnapshotFlag,
// utils.CacheNoPrefetchFlag,
utils.CachePreimagesFlag,
utils.MultiDataBaseFlag,
utils.PersistDiffFlag,
utils.DiffBlockFlag,
utils.PruneAncientDataFlag,
@@ -144,6 +147,7 @@ var (
// utils.MinerNewPayloadTimeout,
utils.NATFlag,
utils.NoDiscoverFlag,
utils.PeerFilterPatternsFlag,
utils.DiscoveryV4Flag,
utils.DiscoveryV5Flag,
utils.InstanceFlag,
@@ -331,9 +335,6 @@ func prepare(ctx *cli.Context) {
5. Networking is disabled; there is no listen-address, the maximum number of peers is set
to 0, and discovery is disabled.
`)
case !ctx.IsSet(utils.NetworkIdFlag.Name):
log.Info("Starting Geth on BSC mainnet...")
}
// If we're a full node on mainnet without --cache specified, bump default cache allowance
if !ctx.IsSet(utils.CacheFlag.Name) && !ctx.IsSet(utils.NetworkIdFlag.Name) {
@@ -360,6 +361,8 @@ func geth(ctx *cli.Context) error {
defer stack.Close()
startNode(ctx, stack, backend, false)
// TODO:: debug code , will be deleted in the future
debug.StartPProf("127.0.0.1:7060", !ctx.IsSet("metrics.addr"))
stack.Wait()
return nil
}
@@ -454,6 +457,10 @@ func startNode(ctx *cli.Context, stack *node.Node, backend ethapi.Backend, isCon
// Set the gas price to the limits from the CLI and start mining
gasprice := flags.GlobalBig(ctx, utils.MinerGasPriceFlag.Name)
ethBackend.TxPool().SetGasTip(gasprice)
gasCeil := ethBackend.Miner().GasCeil()
if gasCeil > params.SystemTxsGas {
ethBackend.TxPool().SetMaxGas(gasCeil - params.SystemTxsGas)
}
if err := ethBackend.StartMining(); err != nil {
utils.Fatalf("Failed to start mining: %v", err)
}

View File

@@ -75,7 +75,7 @@ func NewLevelDBDatabaseWithFreezer(file string, cache int, handles int, ancient
if err != nil {
return nil, err
}
frdb, err := rawdb.NewDatabaseWithFreezer(kvdb, ancient, namespace, readonly, disableFreeze, isLastOffset, pruneAncientData)
frdb, err := rawdb.NewDatabaseWithFreezer(kvdb, ancient, namespace, readonly, disableFreeze, isLastOffset, pruneAncientData, false)
if err != nil {
kvdb.Close()
return nil, err
@@ -155,6 +155,12 @@ func BlockchainCreator(t *testing.T, chaindbPath, AncientPath string, blockRemai
triedb := triedb.NewDatabase(db, nil)
defer triedb.Close()
if err = db.SetupFreezerEnv(&ethdb.FreezerEnv{
ChainCfg: gspec.Config,
BlobExtraReserve: params.DefaultExtraReserveForBlobRequests,
}); err != nil {
t.Fatalf("Failed to create chain: %v", err)
}
genesis := gspec.MustCommit(db, triedb)
// Initialize a fresh chain with only a genesis block
blockchain, err := core.NewBlockChain(db, config, gspec, nil, engine, vm.Config{}, nil, nil)

View File

@@ -43,9 +43,11 @@ import (
"github.com/ethereum/go-ethereum/log"
"github.com/ethereum/go-ethereum/metrics"
"github.com/ethereum/go-ethereum/node"
"github.com/ethereum/go-ethereum/params"
"github.com/ethereum/go-ethereum/rlp"
"github.com/ethereum/go-ethereum/trie"
"github.com/ethereum/go-ethereum/triedb"
"github.com/ethereum/go-ethereum/triedb/pathdb"
cli "github.com/urfave/cli/v2"
)
@@ -245,7 +247,16 @@ func accessDb(ctx *cli.Context, stack *node.Node) (ethdb.Database, error) {
NoBuild: true,
AsyncBuild: false,
}
snaptree, err := snapshot.New(snapconfig, chaindb, triedb.NewDatabase(chaindb, nil), headBlock.Root(), TriesInMemory, false)
dbScheme := rawdb.ReadStateScheme(chaindb)
var config *triedb.Config
if dbScheme == rawdb.PathScheme {
config = &triedb.Config{
PathDB: utils.PathDBConfigAddJournalFilePath(stack, pathdb.ReadOnly),
}
} else if dbScheme == rawdb.HashScheme {
config = triedb.HashDefaults
}
snaptree, err := snapshot.New(snapconfig, chaindb, triedb.NewDatabase(chaindb, config), headBlock.Root(), TriesInMemory, false)
if err != nil {
log.Error("snaptree error", "err", err)
return nil, err // The relevant snapshot(s) might not exist
@@ -333,6 +344,9 @@ func pruneBlock(ctx *cli.Context) error {
stack, config = makeConfigNode(ctx)
defer stack.Close()
blockAmountReserved = ctx.Uint64(utils.BlockAmountReserved.Name)
if blockAmountReserved < params.FullImmutabilityThreshold {
return fmt.Errorf("block-amount-reserved must be greater than or equal to %d", params.FullImmutabilityThreshold)
}
chaindb, err = accessDb(ctx, stack)
if err != nil {
return err
@@ -355,7 +369,15 @@ func pruneBlock(ctx *cli.Context) error {
if !ctx.IsSet(utils.AncientFlag.Name) {
return errors.New("datadir.ancient must be set")
} else {
oldAncientPath = ctx.String(utils.AncientFlag.Name)
if stack.CheckIfMultiDataBase() {
ancientPath := ctx.String(utils.AncientFlag.Name)
index := strings.LastIndex(ancientPath, "/ancient/chain")
if index != -1 {
oldAncientPath = ancientPath[:index] + "/block/ancient/chain"
}
} else {
oldAncientPath = ctx.String(utils.AncientFlag.Name)
}
if !filepath.IsAbs(oldAncientPath) {
// force absolute paths, which often fail due to the splicing of relative paths
return errors.New("datadir.ancient not abs path")
@@ -401,7 +423,7 @@ func pruneBlock(ctx *cli.Context) error {
}
if _, err := os.Stat(newAncientPath); err == nil {
// No file lock found for old ancientDB but new ancientDB exsisted, indicating the geth was interrupted
// No file lock found for old ancientDB but new ancientDB existed, indicating the geth was interrupted
// after old ancientDB removal, this happened after backup successfully, so just rename the new ancientDB
if err := blockpruner.AncientDbReplacer(); err != nil {
log.Error("Failed to rename new ancient directory")
@@ -524,7 +546,7 @@ func verifyState(ctx *cli.Context) error {
log.Error("Failed to load head block")
return errors.New("no head block")
}
triedb := utils.MakeTrieDatabase(ctx, chaindb, false, true, false)
triedb := utils.MakeTrieDatabase(ctx, stack, chaindb, false, true, false)
defer triedb.Close()
snapConfig := snapshot.Config{
@@ -579,7 +601,7 @@ func traverseState(ctx *cli.Context) error {
chaindb := utils.MakeChainDatabase(ctx, stack, true, false)
defer chaindb.Close()
triedb := utils.MakeTrieDatabase(ctx, chaindb, false, true, false)
triedb := utils.MakeTrieDatabase(ctx, stack, chaindb, false, true, false)
defer triedb.Close()
headBlock := rawdb.ReadHeadBlock(chaindb)
@@ -688,7 +710,7 @@ func traverseRawState(ctx *cli.Context) error {
chaindb := utils.MakeChainDatabase(ctx, stack, true, false)
defer chaindb.Close()
triedb := utils.MakeTrieDatabase(ctx, chaindb, false, true, false)
triedb := utils.MakeTrieDatabase(ctx, stack, chaindb, false, true, false)
defer triedb.Close()
headBlock := rawdb.ReadHeadBlock(chaindb)
@@ -852,7 +874,8 @@ func dumpState(ctx *cli.Context) error {
if err != nil {
return err
}
triedb := utils.MakeTrieDatabase(ctx, db, false, true, false)
defer db.Close()
triedb := utils.MakeTrieDatabase(ctx, stack, db, false, true, false)
defer triedb.Close()
snapConfig := snapshot.Config{
@@ -935,7 +958,7 @@ func snapshotExportPreimages(ctx *cli.Context) error {
chaindb := utils.MakeChainDatabase(ctx, stack, true, false)
defer chaindb.Close()
triedb := utils.MakeTrieDatabase(ctx, chaindb, false, true, false)
triedb := utils.MakeTrieDatabase(ctx, stack, chaindb, false, true, false)
defer triedb.Close()
var root common.Hash

View File

@@ -24,4 +24,24 @@ testnet validators version
### 2.Get Transaction Count
```bash
node gettxcount.js --rpc ${url} --startNum ${start} --endNum ${end} --miner ${miner} (optional)
```
```
### 3. Get Performance
```bash
node get_perf.js --rpc ${url} --startNum ${start} --endNum ${end}
```
output as following
```bash
Get the performance between [ 19470 , 19670 )
txCountPerBlock = 3142.81 txCountTotal = 628562 BlockCount = 200 avgBlockTime = 3.005 inturnBlocksRatio = 0.975
txCountPerSecond = 1045.8602329450914 avgGasUsedPerBlock = 250.02062627 avgGasUsedPerSecond = 83.20153952412646
```
### 4. Get validators slash count
```bash
use the latest block
node getslashcount.js --Rpc ${ArchiveRpc}
use a block number
node getslashcount.js --Rpc ${ArchiveRpc} --Num ${blockNum}
```

View File

@@ -0,0 +1,51 @@
import { ethers } from "ethers";
import program from "commander";
// depends on ethjs v6.11.0+ for 4844, https://github.com/ethers-io/ethers.js/releases/tag/v6.11.0
// BSC testnet enabled 4844 on block: 39539137
// Usage:
// nvm use 20
// node check_blobtx.js --rpc https://data-seed-prebsc-1-s1.binance.org:8545 --startNum 39539137
// node check_blobtx.js --rpc https://data-seed-prebsc-1-s1.binance.org:8545 --startNum 39539137 --endNum 40345994
program.option("--rpc <Rpc>", "Rpc Server URL");
program.option("--startNum <Num>", "start block", 0);
program.option("--endNum <Num>", "end block", 0);
program.parse(process.argv);
const provider = new ethers.JsonRpcProvider(program.rpc);
const main = async () => {
var startBlock = parseInt(program.startNum)
var endBlock = parseInt(program.endNum)
if (isNaN(endBlock) || isNaN(startBlock) || startBlock == 0) {
console.error("invalid input, --startNum", program.startNum, "--end", program.endNum)
return
}
// if --endNum is not specified, set it to the latest block number.
if (endBlock == 0) {
endBlock = await provider.getBlockNumber();
}
if (startBlock > endBlock) {
console.error("invalid input, startBlock:",startBlock, " endBlock:", endBlock);
return
}
for (let i = startBlock; i <= endBlock; i++) {
let blockData = await provider.getBlock(i);
console.log("startBlock:",startBlock, "endBlock:", endBlock, "curBlock", i, "blobGasUsed", blockData.blobGasUsed);
if (blockData.blobGasUsed == 0) {
continue
}
for (let txIndex = 0; txIndex<= blockData.transactions.length - 1; txIndex++) {
let txHash = blockData.transactions[txIndex]
let txData = await provider.getTransaction(txHash);
if (txData.type == 3) {
console.log("BlobTx in block:",i, " txIndex:", txIndex, " txHash:", txHash);
}
}
}
};
main().then(() => process.exit(0))
.catch((error) => {
console.error(error);
process.exit(1);
});

View File

@@ -0,0 +1,49 @@
import { ethers } from "ethers";
import program from "commander";
// Usage:
// node faucet_request.js --rpc localhost:8545 --startNum 39539137
// node faucet_request.js --rpc localhost:8545 --startNum 39539137 --endNum 40345994
// node faucet_request.js --rpc https://data-seed-prebsc-1-s1.bnbchain.org:8545 --startNum 39539137 --endNum 40345994
program.option("--rpc <Rpc>", "Rpc Server URL");
program.option("--startNum <Num>", "start block", 0);
program.option("--endNum <Num>", "end block", 0);
program.parse(process.argv);
const provider = new ethers.JsonRpcProvider(program.rpc);
const main = async () => {
var startBlock = parseInt(program.startNum)
var endBlock = parseInt(program.endNum)
if (isNaN(endBlock) || isNaN(startBlock) || startBlock == 0) {
console.error("invalid input, --startNum", program.startNum, "--end", program.endNum)
return
}
// if --endNum is not specified, set it to the latest block number.
if (endBlock == 0) {
endBlock = await provider.getBlockNumber();
}
if (startBlock > endBlock) {
console.error("invalid input, startBlock:",startBlock, " endBlock:", endBlock);
return
}
let startBalance = await provider.getBalance("0xaa25Aa7a19f9c426E07dee59b12f944f4d9f1DD3", startBlock)
let endBalance = await provider.getBalance("0xaa25Aa7a19f9c426E07dee59b12f944f4d9f1DD3", endBlock)
const faucetAmount = BigInt(0.3 * 10**18); // Convert 0.3 ether to wei as a BigInt
const numFaucetRequest = (startBalance - endBalance) / faucetAmount;
// Convert BigInt to ether
const startBalanceEth = Number(startBalance) / 10**18;
const endBalanceEth = Number(endBalance) / 10**18;
console.log(`Start Balance: ${startBalanceEth} ETH`);
console.log(`End Balance: ${endBalanceEth} ETH`);
console.log("successful faucet request: ",numFaucetRequest);
};
main().then(() => process.exit(0))
.catch((error) => {
console.error(error);
process.exit(1);
});

58
cmd/jsutils/get_perf.js Normal file
View File

@@ -0,0 +1,58 @@
import { ethers } from "ethers";
import program from "commander";
program.option("--rpc <rpc>", "Rpc");
program.option("--startNum <startNum>", "start num")
program.option("--endNum <endNum>", "end num")
program.parse(process.argv);
const provider = new ethers.JsonRpcProvider(program.rpc)
const main = async () => {
let txCountTotal = 0;
let gasUsedTotal = 0;
let inturnBlocks = 0;
for (let i = program.startNum; i < program.endNum; i++) {
let txCount = await provider.send("eth_getBlockTransactionCountByNumber", [
ethers.toQuantity(i)]);
txCountTotal += ethers.toNumber(txCount)
let header = await provider.send("eth_getHeaderByNumber", [
ethers.toQuantity(i)]);
let gasUsed = eval(eval(header.gasUsed).toString(10))
gasUsedTotal += gasUsed
let difficulty = eval(eval(header.difficulty).toString(10))
if (difficulty == 2) {
inturnBlocks += 1
}
let timestamp = eval(eval(header.timestamp).toString(10))
console.log("BlockNumber =", i, "mod =", i%4, "miner =", header.miner , "difficulty =", difficulty, "txCount =", ethers.toNumber(txCount), "gasUsed", gasUsed, "timestamp", timestamp)
}
let blockCount = program.endNum - program.startNum
let txCountPerBlock = txCountTotal/blockCount
let startHeader = await provider.send("eth_getHeaderByNumber", [
ethers.toQuantity(program.startNum)]);
let startTime = eval(eval(startHeader.timestamp).toString(10))
let endHeader = await provider.send("eth_getHeaderByNumber", [
ethers.toQuantity(program.endNum)]);
let endTime = eval(eval(endHeader.timestamp).toString(10))
let timeCost = endTime - startTime
let avgBlockTime = timeCost/blockCount
let inturnBlocksRatio = inturnBlocks/blockCount
let tps = txCountTotal/timeCost
let M = 1000000
let avgGasUsedPerBlock = gasUsedTotal/blockCount/M
let avgGasUsedPerSecond = gasUsedTotal/timeCost/M
console.log("Get the performance between [", program.startNum, ",", program.endNum, ")");
console.log("txCountPerBlock =", txCountPerBlock, "txCountTotal =", txCountTotal, "BlockCount =", blockCount, "avgBlockTime =", avgBlockTime, "inturnBlocksRatio =", inturnBlocksRatio);
console.log("txCountPerSecond =", tps, "avgGasUsedPerBlock =", avgGasUsedPerBlock, "avgGasUsedPerSecond =", avgGasUsedPerSecond);
};
main().then(() => process.exit(0))
.catch((error) => {
console.error(error);
process.exit(1);
});

View File

@@ -0,0 +1,119 @@
import { ethers } from "ethers";
import program from "commander";
program.option("--Rpc <Rpc>", "Rpc");
program.option("--Num <Num>", "num", 0)
program.parse(process.argv);
const provider = new ethers.JsonRpcProvider(program.Rpc);
const slashAbi = [
"function getSlashIndicator(address validatorAddr) external view returns (uint256, uint256)"
]
const validatorSetAbi = [
"function getLivingValidators() external view returns (address[], bytes[])"
]
const stakeHubAbi = [
"function getValidatorDescription(address validatorAddr) external view returns (tuple(string, string, string, string))",
"function consensusToOperator(address consensusAddr) public view returns (address)"
]
const addrValidatorSet = '0x0000000000000000000000000000000000001000';
const validatorSet = new ethers.Contract(addrValidatorSet, validatorSetAbi, provider);
const addrSlash = '0x0000000000000000000000000000000000001001';
const slashIndicator = new ethers.Contract(addrSlash, slashAbi, provider)
const addrStakeHub = '0x0000000000000000000000000000000000002002';
const stakeHub = new ethers.Contract(addrStakeHub, stakeHubAbi, provider)
const validatorMap = new Map([
//BSC
["0x37e9627A91DD13e453246856D58797Ad6583D762", "LegendII"],
["0xB4647b856CB9C3856d559C885Bed8B43e0846a47", "CertiK"],
["0x75B851a27D7101438F45fce31816501193239A83", "Figment"],
["0x502aECFE253E6AA0e8D2A06E12438FFeD0Fe16a0", "BscScan"],
["0xCa503a7eD99eca485da2E875aedf7758472c378C", "InfStones"],
["0x5009317FD4F6F8FeEa9dAe41E5F0a4737BB7A7D5", "NodeReal"],
["0x1cFDBd2dFf70C6e2e30df5012726F87731F38164", "Tranchess"],
["0xF8de5e61322302b2c6e0a525cC842F10332811bf", "Namelix"],
["0xCcB42A9b8d6C46468900527Bc741938E78AB4577", "Turing"],
["0x9f1b7FAE54BE07F4FEE34Eb1aaCb39A1F7B6FC92", "TWStaking"],
["0x7E1FdF03Eb3aC35BF0256694D7fBe6B6d7b3E0c8","LegendIII"],
["0x7b501c7944185130DD4aD73293e8Aa84eFfDcee7","MathW"],
["0x58567F7A51a58708C8B40ec592A38bA64C0697De","Legend"],
["0x460A252B4fEEFA821d3351731220627D7B7d1F3d","Defibit"],
["0x8A239732871AdC8829EA2f47e94087C5FBad47b6","The48Club"],
["0xD3b0d838cCCEAe7ebF1781D11D1bB741DB7Fe1A7","BNBEve"],
["0xF8B99643fAfC79d9404DE68E48C4D49a3936f787","Avengers"],
["0x4e5acf9684652BEa56F2f01b7101a225Ee33d23f","HashKey"],
["0x9bb56C2B4DBE5a06d79911C9899B6f817696ACFc","Feynman"],
["0xbdcc079BBb23C1D9a6F36AA31309676C258aBAC7","Fuji"],
["0x38944092685a336CB6B9ea58836436709a2adC89","Shannon"],
["0xfC1004C0f296Ec3Df4F6762E9EabfcF20EB304a2","Aoraki"],
["0xa0884bb00E5F23fE2427f0E5eC9E51F812848563","Coda"],
["0xe7776De78740f28a96412eE5cbbB8f90896b11A5","Ankr"],
["0xA2D969E82524001Cb6a2357dBF5922B04aD2FCD8","Pexmons"],
["0x5cf810AB8C718ac065b45f892A5BAdAB2B2946B9","Zen"],
["0x4d15D9BCd0c2f33E7510c0de8b42697CA558234a","LegendVII"],
["0x1579ca96EBd49A0B173f86C372436ab1AD393380","LegendV"],
["0xd1F72d433f362922f6565FC77c25e095B29141c8","LegendVI"],
["0xf9814D93b4d904AaA855cBD4266D6Eb0Ec1Aa478","Legend8"],
["0x025a4e09Ea947b8d695f53ddFDD48ddB8F9B06b7","Ciscox"],
["0xE9436F6F30b4B01b57F2780B2898f3820EbD7B98","LegendIV"],
["0xC2d534F079444E6E7Ff9DabB3FD8a26c607932c8","Axion"],
["0x9F7110Ba7EdFda83Fc71BeA6BA3c0591117b440D","LegendIX"],
["0xB997Bf1E3b96919fBA592c1F61CE507E165Ec030","Seoraksan"],
["0x286C1b674d48cFF67b4096b6c1dc22e769581E91","Sigm8"],
["0x73A26778ef9509a6E94b55310eE7233795a9EB25","Coinlix"],
["0x18c44f4FBEde9826C7f257d500A65a3D5A8edebc","Nozti"],
["0xA100FCd08cE722Dc68Ddc3b54237070Cb186f118","Tiollo"],
["0x0F28847cfdbf7508B13Ebb9cEb94B2f1B32E9503","Raptas"],
["0xfD85346c8C991baC16b9c9157e6bdfDACE1cD7d7","Glorin"],
["0x978F05CED39A4EaFa6E8FD045Fe2dd6Da836c7DF","NovaX"],
["0xd849d1dF66bFF1c2739B4399425755C2E0fAbbAb","Nexa"],
["0xA015d9e9206859c13201BB3D6B324d6634276534","Star"],
["0x5ADde0151BfAB27f329e5112c1AeDeed7f0D3692","Veri"],
//Chapel
["0x08265dA01E1A65d62b903c7B34c08cB389bF3D99","Ararat"],
["0x7f5f2cF1aec83bF0c74DF566a41aa7ed65EA84Ea","Kita"],
["0x53387F3321FD69d1E030BB921230dFb188826AFF","Fuji"],
["0x76D76ee8823dE52A1A431884c2ca930C5e72bff3","Seoraksan"],
["0xd447b49CD040D20BC21e49ffEa6487F5638e4346","Everest"],
["0x1a3d9D7A717D64e6088aC937d5aAcDD3E20ca963","Elbrus"],
["0x40D3256EB0BaBE89f0ea54EDAa398513136612f5","Bloxroute"],
["0xF9a1Db0d6f22Bd78ffAECCbc8F47c83Df9FBdbCf","Test"]
]);
const main = async () => {
let blockNum = ethers.getNumber(program.Num)
if (blockNum === 0) {
blockNum = await provider.getBlockNumber()
}
let block = await provider.getBlock(blockNum)
console.log("At block", blockNum, "time", block.date)
const data = await validatorSet.getLivingValidators({blockTag:blockNum})
let totalSlash = 0
for (let i = 0; i < data[0].length; i++) {
let addr = data[0][i];
var val
if (!validatorMap.has(addr)) {
let opAddr = await stakeHub.consensusToOperator(addr, {blockTag:blockNum})
let value = await stakeHub.getValidatorDescription(opAddr, {blockTag:blockNum})
val = value[0]
console.log(addr, val)
} else {
val = validatorMap.get(addr)
}
let info = await slashIndicator.getSlashIndicator(addr, {blockTag:blockNum})
let count = ethers.toNumber(info[1])
totalSlash += count
console.log("Slash:", count, addr, val)
}
console.log("Total slash count", totalSlash)
};
main().then(() => process.exit(0))
.catch((error) => {
console.error(error);
process.exit(1);
});

View File

@@ -343,7 +343,7 @@ func missingBlocks(chain *core.BlockChain, blocks []*types.Block) []*types.Block
continue
}
// If we're above the chain head, state availability is a must
if !chain.HasBlockAndState(block.Hash(), block.NumberU64()) {
if !chain.HasBlockAndState(block.Hash(), block.Number().Int64()) {
return blocks[i:]
}
}

View File

@@ -93,10 +93,10 @@ var (
Value: flags.DirectoryString(node.DefaultDataDir()),
Category: flags.EthCategory,
}
SeparateDBFlag = &cli.BoolFlag{
Name: "separatedb",
Usage: "Enable a separated trie database, it will be created within a subdirectory called state, " +
"Users can copy this state directory to another directory or disk, and then create a symbolic link to the state directory under the chaindata",
MultiDataBaseFlag = &cli.BoolFlag{
Name: "multidatabase",
Usage: "Enable a separated state and block database, it will be created within two subdirectory called state and block, " +
"Users can copy this state or block directory to another directory or disk, and then create a symbolic link to the state directory under the chaindata",
Category: flags.EthCategory,
}
DirectBroadcastFlag = &cli.BoolFlag{
@@ -224,7 +224,7 @@ var (
// hbss2pbss command options
ForceFlag = &cli.BoolFlag{
Name: "force",
Usage: "Force convert hbss trie node to pbss trie node. Ingore any metadata",
Usage: "Force convert hbss trie node to pbss trie node. Ignore any metadata",
Value: false,
}
// Dump command options.
@@ -305,9 +305,9 @@ var (
Usage: "Manually specify the Rialto Genesis Hash, to trigger builtin network logic",
Category: flags.EthCategory,
}
OverrideCancun = &cli.Uint64Flag{
Name: "override.cancun",
Usage: "Manually specify the Cancun fork timestamp, overriding the bundled setting",
OverrideBohr = &cli.Uint64Flag{
Name: "override.bohr",
Usage: "Manually specify the Bohr fork timestamp, overriding the bundled setting",
Category: flags.EthCategory,
}
OverrideVerkle = &cli.Uint64Flag{
@@ -315,15 +315,6 @@ var (
Usage: "Manually specify the Verkle fork timestamp, overriding the bundled setting",
Category: flags.EthCategory,
}
OverrideFeynman = &cli.Uint64Flag{
Name: "override.feynman",
Usage: "Manually specify the Feynman fork timestamp, overriding the bundled setting",
Category: flags.EthCategory,
}
OverrideFeynmanFix = &cli.Uint64Flag{
Name: "override.feynmanfix",
Usage: "Manually specify the FeynmanFix fork timestamp, overriding the bundled setting",
}
OverrideFullImmutabilityThreshold = &cli.Uint64Flag{
Name: "override.immutabilitythreshold",
Usage: "It is the number of blocks after which a chain segment is considered immutable, only for testing purpose",
@@ -342,6 +333,12 @@ var (
Value: params.DefaultExtraReserveForBlobRequests,
Category: flags.EthCategory,
}
OverrideBreatheBlockInterval = &cli.Uint64Flag{
Name: "override.breatheblockinterval",
Usage: "It changes the interval between breathe blocks, only for testing purpose",
Value: params.BreatheBlockInterval,
Category: flags.EthCategory,
}
SyncModeFlag = &flags.TextMarshalerFlag{
Name: "syncmode",
Usage: `Blockchain sync mode ("snap" or "full")`,
@@ -356,7 +353,7 @@ var (
}
StateSchemeFlag = &cli.StringFlag{
Name: "state.scheme",
Usage: "Scheme to use for storing ethereum state ('hash' or 'path')",
Usage: "Scheme to use for storing ethereum state ('hash', 'path', 'version')",
Category: flags.StateCategory,
}
PathDBSyncFlag = &cli.BoolFlag{
@@ -365,6 +362,12 @@ var (
Value: false,
Category: flags.StateCategory,
}
JournalFileFlag = &cli.BoolFlag{
Name: "journalfile",
Usage: "Enable using journal file to store the TrieJournal instead of KVDB in pbss (default = false)",
Value: false,
Category: flags.StateCategory,
}
StateHistoryFlag = &cli.Uint64Flag{
Name: "history.state",
Usage: "Number of recent blocks to retain state history for (default = 90,000 blocks, 0 = entire chain)",
@@ -876,6 +879,11 @@ var (
Usage: "Disables the peer discovery mechanism (manual peer addition)",
Category: flags.NetworkingCategory,
}
PeerFilterPatternsFlag = &cli.StringSliceFlag{
Name: "peerfilter",
Usage: "Disallow peers connection if peer name matches the given regular expressions",
Category: flags.NetworkingCategory,
}
DiscoveryV4Flag = &cli.BoolFlag{
Name: "discovery.v4",
Aliases: []string{"discv4"},
@@ -1069,6 +1077,7 @@ Please note that --` + MetricsHTTPFlag.Name + ` must be set to start the server.
Name: "block-amount-reserved",
Usage: "Sets the expected remained amount of blocks for offline block prune",
Category: flags.BlockHistoryCategory,
Value: params.FullImmutabilityThreshold,
}
CheckSnapshotWithMPT = &cli.BoolFlag{
@@ -1126,6 +1135,25 @@ Please note that --` + MetricsHTTPFlag.Name + ` must be set to start the server.
Value: params.DefaultExtraReserveForBlobRequests,
Category: flags.MiscCategory,
}
BlockNumber = &cli.Int64Flag{
Name: "block",
Value: int64(0),
}
VersionStateDirFlag = &flags.DirectoryFlag{
Name: "versionstatedir",
Usage: "Data directory for the version databases and keystore",
Value: flags.DirectoryString(node.DefaultDataDir()),
Category: flags.EthCategory,
}
HashStateDirFlag = &flags.DirectoryFlag{
Name: "hashstatedir",
Usage: "Data directory for the version databases and keystore",
Value: flags.DirectoryString(node.DefaultDataDir()),
Category: flags.EthCategory,
}
)
var (
@@ -1144,7 +1172,6 @@ var (
DBEngineFlag,
StateSchemeFlag,
HttpHeaderFlag,
SeparateDBFlag,
}
)
@@ -1542,6 +1569,9 @@ func SetP2PConfig(ctx *cli.Context, cfg *p2p.Config) {
if ctx.IsSet(NoDiscoverFlag.Name) {
cfg.NoDiscovery = true
}
if ctx.IsSet(PeerFilterPatternsFlag.Name) {
cfg.PeerFilterPatterns = ctx.StringSlice(PeerFilterPatternsFlag.Name)
}
CheckExclusive(ctx, DiscoveryV4Flag, NoDiscoverFlag)
CheckExclusive(ctx, DiscoveryV5Flag, NoDiscoverFlag)
@@ -1942,11 +1972,17 @@ func SetEthConfig(ctx *cli.Context, stack *node.Node, cfg *ethconfig.Config) {
if ctx.IsSet(StateHistoryFlag.Name) {
cfg.StateHistory = ctx.Uint64(StateHistoryFlag.Name)
}
scheme, err := ParseCLIAndConfigStateScheme(ctx.String(StateSchemeFlag.Name), cfg.StateScheme)
if err != nil {
Fatalf("%v", err)
if ctx.String(StateSchemeFlag.Name) != rawdb.VersionScheme {
scheme, err := ParseCLIAndConfigStateScheme(ctx.String(StateSchemeFlag.Name), cfg.StateScheme)
if err != nil {
Fatalf("%v", err)
}
cfg.StateScheme = scheme
} else {
// TODO:: compatible with cli line and configuration file, currently only supports cli.
cfg.StateScheme = rawdb.VersionScheme
}
cfg.StateScheme = scheme
// Parse transaction history flag, if user is still using legacy config
// file with 'TxLookupLimit' configured, copy the value to 'TransactionHistory'.
if cfg.TransactionHistory == ethconfig.Defaults.TransactionHistory && cfg.TxLookupLimit != ethconfig.Defaults.TxLookupLimit {
@@ -1962,6 +1998,10 @@ func SetEthConfig(ctx *cli.Context, stack *node.Node, cfg *ethconfig.Config) {
if ctx.IsSet(PathDBSyncFlag.Name) {
cfg.PathSyncFlush = true
}
if ctx.IsSet(JournalFileFlag.Name) {
cfg.JournalFileEnabled = true
}
if ctx.String(GCModeFlag.Name) == "archive" && cfg.TransactionHistory != 0 {
cfg.TransactionHistory = 0
log.Warn("Disabled transaction unindexing for archive node")
@@ -2057,7 +2097,7 @@ func SetEthConfig(ctx *cli.Context, stack *node.Node, cfg *ethconfig.Config) {
}
cfg.Genesis = core.DefaultBSCGenesisBlock()
SetDNSDiscoveryDefaults(cfg, params.BSCGenesisHash)
case ctx.Bool(ChapelFlag.Name):
case ctx.Bool(ChapelFlag.Name) || cfg.NetworkId == 97:
if !ctx.IsSet(NetworkIdFlag.Name) {
cfg.NetworkId = 97
}
@@ -2377,9 +2417,11 @@ func MakeChainDatabase(ctx *cli.Context, stack *node.Node, readonly, disableFree
default:
chainDb, err = stack.OpenDatabaseWithFreezer("chaindata", cache, handles, ctx.String(AncientFlag.Name), "", readonly, disableFreeze, false, false)
// set the separate state database
if stack.IsSeparatedDB() && err == nil {
if stack.CheckIfMultiDataBase() && err == nil {
stateDiskDb := MakeStateDataBase(ctx, stack, readonly, false)
chainDb.SetStateStore(stateDiskDb)
blockDb := MakeBlockDatabase(ctx, stack, readonly, false)
chainDb.SetBlockStore(blockDb)
}
}
if err != nil {
@@ -2391,7 +2433,7 @@ func MakeChainDatabase(ctx *cli.Context, stack *node.Node, readonly, disableFree
// MakeStateDataBase open a separate state database using the flags passed to the client and will hard crash if it fails.
func MakeStateDataBase(ctx *cli.Context, stack *node.Node, readonly, disableFreeze bool) ethdb.Database {
cache := ctx.Int(CacheFlag.Name) * ctx.Int(CacheDatabaseFlag.Name) / 100
handles := MakeDatabaseHandles(ctx.Int(FDLimitFlag.Name)) / 2
handles := MakeDatabaseHandles(ctx.Int(FDLimitFlag.Name)) * 90 / 100
statediskdb, err := stack.OpenDatabaseWithFreezer("chaindata/state", cache, handles, "", "", readonly, disableFreeze, false, false)
if err != nil {
Fatalf("Failed to open separate trie database: %v", err)
@@ -2399,6 +2441,23 @@ func MakeStateDataBase(ctx *cli.Context, stack *node.Node, readonly, disableFree
return statediskdb
}
// MakeBlockDatabase open a separate block database using the flags passed to the client and will hard crash if it fails.
func MakeBlockDatabase(ctx *cli.Context, stack *node.Node, readonly, disableFreeze bool) ethdb.Database {
cache := ctx.Int(CacheFlag.Name) * ctx.Int(CacheDatabaseFlag.Name) / 100
handles := MakeDatabaseHandles(ctx.Int(FDLimitFlag.Name)) / 10
blockDb, err := stack.OpenDatabaseWithFreezer("chaindata/block", cache, handles, "", "", readonly, disableFreeze, false, false)
if err != nil {
Fatalf("Failed to open separate block database: %v", err)
}
return blockDb
}
func PathDBConfigAddJournalFilePath(stack *node.Node, config *pathdb.Config) *pathdb.Config {
path := fmt.Sprintf("%s/%s", stack.ResolvePath("chaindata"), eth.JournalFileName)
config.JournalFilePath = path
return config
}
// tryMakeReadOnlyDatabase try to open the chain database in read-only mode,
// or fallback to write mode if the database is not initialized.
//
@@ -2541,7 +2600,7 @@ func MakeConsolePreloads(ctx *cli.Context) []string {
}
// MakeTrieDatabase constructs a trie database based on the configured scheme.
func MakeTrieDatabase(ctx *cli.Context, disk ethdb.Database, preimage bool, readOnly bool, isVerkle bool) *triedb.Database {
func MakeTrieDatabase(ctx *cli.Context, stack *node.Node, disk ethdb.Database, preimage bool, readOnly bool, isVerkle bool) *triedb.Database {
config := &triedb.Config{
Preimages: preimage,
IsVerkle: isVerkle,
@@ -2562,6 +2621,7 @@ func MakeTrieDatabase(ctx *cli.Context, disk ethdb.Database, preimage bool, read
} else {
config.PathDB = pathdb.Defaults
}
config.PathDB.JournalFilePath = fmt.Sprintf("%s/%s", stack.ResolvePath("chaindata"), eth.JournalFileName)
return triedb.NewDatabase(disk, config)
}

View File

@@ -83,7 +83,7 @@ func TestHistoryImportAndExport(t *testing.T) {
t.Fatalf("unable to initialize chain: %v", err)
}
if _, err := chain.InsertChain(blocks); err != nil {
t.Fatalf("error insterting chain: %v", err)
t.Fatalf("error inserting chain: %v", err)
}
// Make temp directory for era files.
@@ -163,7 +163,7 @@ func TestHistoryImportAndExport(t *testing.T) {
// Now import Era.
freezer := t.TempDir()
db2, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), freezer, "", false, false, false, false)
db2, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), freezer, "", false, false, false, false, false)
if err != nil {
panic(err)
}

View File

@@ -15,14 +15,13 @@ import (
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/internal/ethapi"
"github.com/ethereum/go-ethereum/log"
"github.com/ethereum/go-ethereum/params"
"github.com/ethereum/go-ethereum/rpc"
)
const SecondsPerDay uint64 = 86400
// the params should be two blocks' time(timestamp)
func sameDayInUTC(first, second uint64) bool {
return first/SecondsPerDay == second/SecondsPerDay
return first/params.BreatheBlockInterval == second/params.BreatheBlockInterval
}
func isBreatheBlock(lastBlockTime, blockTime uint64) bool {

View File

@@ -16,7 +16,7 @@ import (
lru "github.com/hashicorp/golang-lru"
"github.com/holiman/uint256"
"github.com/prysmaticlabs/prysm/v4/crypto/bls"
"github.com/prysmaticlabs/prysm/v5/crypto/bls"
"github.com/willf/bitset"
"golang.org/x/crypto/sha3"
@@ -48,8 +48,9 @@ import (
)
const (
inMemorySnapshots = 256 // Number of recent snapshots to keep in memory
inMemorySignatures = 4096 // Number of recent block signatures to keep in memory
inMemorySnapshots = 256 // Number of recent snapshots to keep in memory
inMemorySignatures = 4096 // Number of recent block signatures to keep in memory
inMemoryHeaders = 86400 // Number of recent headers to keep in memory for double sign detection,
checkpointInterval = 1024 // Number of blocks after which to save the snapshot to the database
defaultEpochLength = uint64(100) // Default number of blocks of checkpoint to update validatorSet from contract
@@ -80,6 +81,7 @@ var (
verifyVoteAttestationErrorCounter = metrics.NewRegisteredCounter("parlia/verifyVoteAttestation/error", nil)
updateAttestationErrorCounter = metrics.NewRegisteredCounter("parlia/updateAttestation/error", nil)
validVotesfromSelfCounter = metrics.NewRegisteredCounter("parlia/VerifyVote/self", nil)
doubleSignCounter = metrics.NewRegisteredCounter("parlia/doublesign", nil)
systemContracts = map[common.Address]bool{
common.HexToAddress(systemcontracts.ValidatorContract): true,
@@ -216,8 +218,11 @@ type Parlia struct {
genesisHash common.Hash
db ethdb.Database // Database to store and retrieve snapshot checkpoints
recentSnaps *lru.ARCCache // Snapshots for recent block to speed up
signatures *lru.ARCCache // Signatures of recent blocks to speed up mining
recentSnaps *lru.ARCCache // Snapshots for recent block to speed up
signatures *lru.ARCCache // Signatures of recent blocks to speed up mining
recentHeaders *lru.ARCCache //
// Recent headers to check for double signing: key includes block number and miner. value is the block header
// If same key's value already exists for different block header roots then double sign is detected
signer types.Signer
@@ -263,6 +268,10 @@ func New(
if err != nil {
panic(err)
}
recentHeaders, err := lru.NewARC(inMemoryHeaders)
if err != nil {
panic(err)
}
vABIBeforeLuban, err := abi.JSON(strings.NewReader(validatorSetABIBeforeLuban))
if err != nil {
panic(err)
@@ -286,6 +295,7 @@ func New(
db: db,
ethAPI: ethAPI,
recentSnaps: recentSnaps,
recentHeaders: recentHeaders,
signatures: signatures,
validatorSetABIBeforeLuban: vABIBeforeLuban,
validatorSetABI: vABI,
@@ -297,6 +307,10 @@ func New(
return c
}
func (p *Parlia) Period() uint64 {
return p.config.Period
}
func (p *Parlia) IsSystemTransaction(tx *types.Transaction, header *types.Header) (bool, error) {
// deploy a contract
if tx.To() == nil {
@@ -809,6 +823,17 @@ func (p *Parlia) verifySeal(chain consensus.ChainHeaderReader, header *types.Hea
return errCoinBaseMisMatch
}
// check for double sign & add to cache
key := proposalKey(*header)
preHash, ok := p.recentHeaders.Get(key)
if ok && preHash != header.Hash() {
doubleSignCounter.Inc(1)
log.Warn("DoubleSign detected", " block", header.Number, " miner", header.Coinbase,
"hash1", preHash.(common.Hash), "hash2", header.Hash())
} else {
p.recentHeaders.Add(key, header.Hash())
}
if _, ok := snap.Validators[signer]; !ok {
return errUnauthorizedValidator(signer.String())
}
@@ -2011,3 +2036,8 @@ func applyMessage(
}
return msg.Gas() - returnGas, err
}
// proposalKey build a key which is a combination of the block number and the proposer address.
func proposalKey(header types.Header) string {
return header.ParentHash.String() + header.Coinbase.String()
}

View File

@@ -25,6 +25,7 @@ import (
"github.com/ethereum/go-ethereum/consensus"
"github.com/ethereum/go-ethereum/core/state"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/metrics"
"github.com/ethereum/go-ethereum/params"
"github.com/ethereum/go-ethereum/trie"
)
@@ -40,6 +41,12 @@ func EnableRemoteVerifyManager(remoteValidator *remoteVerifyManager) BlockValida
}
}
var (
validateBloomTimer = metrics.NewRegisteredTimer("validate/bloom/time", nil)
validateReceiptTimer = metrics.NewRegisteredTimer("validate/receipt/time", nil)
validateRootTimer = metrics.NewRegisteredTimer("validate/root/time", nil)
)
// BlockValidator is responsible for validating block headers, uncles and
// processed state.
//
@@ -66,12 +73,37 @@ func NewBlockValidator(config *params.ChainConfig, blockchain *BlockChain, engin
return validator
}
// ValidateListsInBody validates that UncleHash, WithdrawalsHash, and WithdrawalsHash correspond to the lists in the block body, respectively.
func ValidateListsInBody(block *types.Block) error {
header := block.Header()
if hash := types.CalcUncleHash(block.Uncles()); hash != header.UncleHash {
return fmt.Errorf("uncle root hash mismatch (header value %x, calculated %x)", header.UncleHash, hash)
}
if hash := types.DeriveSha(block.Transactions(), trie.NewStackTrie(nil)); hash != header.TxHash {
return fmt.Errorf("transaction root hash mismatch: have %x, want %x", hash, header.TxHash)
}
// Withdrawals are present after the Shanghai fork.
if header.WithdrawalsHash != nil {
// Withdrawals list must be present in body after Shanghai.
if block.Withdrawals() == nil {
return errors.New("missing withdrawals in block body")
}
if hash := types.DeriveSha(block.Withdrawals(), trie.NewStackTrie(nil)); hash != *header.WithdrawalsHash {
return fmt.Errorf("withdrawals root hash mismatch (header value %x, calculated %x)", *header.WithdrawalsHash, hash)
}
} else if block.Withdrawals() != nil { // Withdrawals turn into empty from nil when BlockBody has Sidecars
// Withdrawals are not allowed prior to shanghai fork
return errors.New("withdrawals present in block body")
}
return nil
}
// ValidateBody validates the given block's uncles and verifies the block
// header's transaction and uncle roots. The headers are assumed to be already
// validated at this point.
func (v *BlockValidator) ValidateBody(block *types.Block) error {
// Check whether the block is already imported.
if v.bc.HasBlockAndState(block.Hash(), block.NumberU64()) {
if v.bc.HasBlockAndState(block.Hash(), block.Number().Int64()) {
return ErrKnownBlock
}
if v.bc.isCachedBadBlock(block) {
@@ -83,31 +115,12 @@ func (v *BlockValidator) ValidateBody(block *types.Block) error {
if err := v.engine.VerifyUncles(v.bc, block); err != nil {
return err
}
if hash := types.CalcUncleHash(block.Uncles()); hash != header.UncleHash {
return fmt.Errorf("uncle root hash mismatch (header value %x, calculated %x)", header.UncleHash, hash)
}
validateFuns := []func() error{
func() error {
if hash := types.DeriveSha(block.Transactions(), trie.NewStackTrie(nil)); hash != header.TxHash {
return fmt.Errorf("transaction root hash mismatch: have %x, want %x", hash, header.TxHash)
}
return nil
return ValidateListsInBody(block)
},
func() error {
// Withdrawals are present after the Shanghai fork.
if header.WithdrawalsHash != nil {
// Withdrawals list must be present in body after Shanghai.
if block.Withdrawals() == nil {
return errors.New("missing withdrawals in block body")
}
if hash := types.DeriveSha(block.Withdrawals(), trie.NewStackTrie(nil)); hash != *header.WithdrawalsHash {
return fmt.Errorf("withdrawals root hash mismatch (header value %x, calculated %x)", *header.WithdrawalsHash, hash)
}
} else if block.Withdrawals() != nil { // Withdrawals turn into empty from nil when BlockBody has Sidecars
// Withdrawals are not allowed prior to shanghai fork
return errors.New("withdrawals present in block body")
}
// Blob transactions may be present after the Cancun fork.
var blobs int
for i, tx := range block.Transactions() {
@@ -136,7 +149,7 @@ func (v *BlockValidator) ValidateBody(block *types.Block) error {
return nil
},
func() error {
if !v.bc.HasBlockAndState(block.ParentHash(), block.NumberU64()-1) {
if !v.bc.HasBlockAndState(block.ParentHash(), block.Number().Int64()-1) {
if !v.bc.HasBlock(block.ParentHash(), block.NumberU64()-1) {
return consensus.ErrUnknownAncestor
}
@@ -178,6 +191,10 @@ func (v *BlockValidator) ValidateState(block *types.Block, statedb *state.StateD
// For valid blocks this should always validate to true.
validateFuns := []func() error{
func() error {
defer func(start time.Time) {
validateBloomTimer.UpdateSince(start)
}(time.Now())
rbloom := types.CreateBloom(receipts)
if rbloom != header.Bloom {
return fmt.Errorf("invalid bloom (remote: %x local: %x)", header.Bloom, rbloom)
@@ -185,6 +202,9 @@ func (v *BlockValidator) ValidateState(block *types.Block, statedb *state.StateD
return nil
},
func() error {
defer func(start time.Time) {
validateReceiptTimer.UpdateSince(start)
}(time.Now())
receiptSha := types.DeriveSha(receipts, trie.NewStackTrie(nil))
if receiptSha != header.ReceiptHash {
return fmt.Errorf("invalid receipt root hash (remote: %x local: %x)", header.ReceiptHash, receiptSha)
@@ -203,6 +223,9 @@ func (v *BlockValidator) ValidateState(block *types.Block, statedb *state.StateD
})
} else {
validateFuns = append(validateFuns, func() error {
defer func(start time.Time) {
validateRootTimer.UpdateSince(start)
}(time.Now())
if root := statedb.IntermediateRoot(v.config.IsEIP158(header.Number)); header.Root != root {
return fmt.Errorf("invalid merkle root (remote: %x local: %x) dberr: %w", header.Root, root, statedb.Error())
}

File diff suppressed because it is too large Load Diff

View File

@@ -48,18 +48,24 @@ func (st *insertStats) report(chain []*types.Block, index int, snapDiffItems, sn
// If we're at the last block of the batch or report period reached, log
if index == len(chain)-1 || elapsed >= statsReportLimit {
// Count the number of transactions in this segment
var txs int
var txs, blobs int
for _, block := range chain[st.lastIndex : index+1] {
txs += len(block.Transactions())
for _, sidecar := range block.Sidecars() {
blobs += len(sidecar.Blobs)
}
}
end := chain[index]
// Assemble the log context and send it to the logger
mgasps := float64(st.usedGas) * 1000 / float64(elapsed)
context := []interface{}{
"number", end.Number(), "hash", end.Hash(), "miner", end.Coinbase(),
"blocks", st.processed, "txs", txs, "mgas", float64(st.usedGas) / 1000000,
"elapsed", common.PrettyDuration(elapsed), "mgasps", float64(st.usedGas) * 1000 / float64(elapsed),
"blocks", st.processed, "txs", txs, "blobs", blobs, "mgas", float64(st.usedGas) / 1000000,
"elapsed", common.PrettyDuration(elapsed), "mgasps", mgasps,
}
mGasPsGauge.Update(int64(mgasps))
blockInsertMgaspsGauge.Update(int64(mgasps))
if timestamp := time.Unix(int64(end.Time()), 0); time.Since(timestamp) > time.Minute {
context = append(context, []interface{}{"age", common.PrettyAge(timestamp)}...)
}

View File

@@ -18,6 +18,7 @@ package core
import (
"errors"
"fmt"
"math/big"
"github.com/ethereum/go-ethereum/common"
@@ -338,7 +339,7 @@ func (bc *BlockChain) GetTd(hash common.Hash, number uint64) *big.Int {
}
// HasState checks if state trie is fully present in the database or not.
func (bc *BlockChain) HasState(hash common.Hash) bool {
func (bc *BlockChain) HasState(number int64, hash common.Hash) bool {
if bc.NoTries() {
return bc.snaps != nil && bc.snaps.Snapshot(hash) != nil
}
@@ -348,19 +349,24 @@ func (bc *BlockChain) HasState(hash common.Hash) bool {
return true
}
}
_, err := bc.stateCache.OpenTrie(hash)
return err == nil
return bc.stateCache.HasState(number, hash)
}
// HasBlockAndState checks if a block and associated state trie is fully present
// in the database or not, caching it if present.
func (bc *BlockChain) HasBlockAndState(hash common.Hash, number uint64) bool {
func (bc *BlockChain) HasBlockAndState(hash common.Hash, number int64) bool {
// Check first that the block itself is known
block := bc.GetBlock(hash, number)
if block == nil {
return false
var root common.Hash
if number < 0 {
root = types.EmptyRootHash
} else {
block := bc.GetBlock(hash, uint64(number))
if block == nil {
return false
}
root = block.Root()
}
return bc.HasState(block.Root())
return bc.HasState(number, root)
}
// stateRecoverable checks if the specified state is recoverable.
@@ -391,12 +397,19 @@ func (bc *BlockChain) ContractCodeWithPrefix(hash common.Hash) ([]byte, error) {
// State returns a new mutable state based on the current HEAD block.
func (bc *BlockChain) State() (*state.StateDB, error) {
return bc.StateAt(bc.CurrentBlock().Root)
return bc.StateAt(bc.CurrentBlock().Number.Int64(), bc.CurrentBlock().Root)
}
// StateAt returns a new mutable state based on a particular point in time.
func (bc *BlockChain) StateAt(root common.Hash) (*state.StateDB, error) {
stateDb, err := state.New(root, bc.stateCache, bc.snaps)
func (bc *BlockChain) StateAt(number int64, root common.Hash) (*state.StateDB, error) {
// new state db with no need commit mode
has := bc.HasState(number, root)
if !has {
return nil, fmt.Errorf(fmt.Sprintf("do not has state, verison: %d, root: %s", number, root.String()))
}
sdb := state.NewDatabaseWithNodeDB(bc.db, bc.triedb, false)
sdb.SetVersion(number)
stateDb, err := state.New(root, sdb, bc.snaps)
if err != nil {
return nil, err
}
@@ -511,3 +524,12 @@ func (bc *BlockChain) SubscribeBlockProcessingEvent(ch chan<- bool) event.Subscr
func (bc *BlockChain) SubscribeFinalizedHeaderEvent(ch chan<- FinalizedHeaderEvent) event.Subscription {
return bc.scope.Track(bc.finalizedHeaderFeed.Subscribe(ch))
}
// AncientTail retrieves the tail the ancients blocks
func (bc *BlockChain) AncientTail() (uint64, error) {
tail, err := bc.db.BlockStore().Tail()
if err != nil {
return 0, err
}
return tail, nil
}

View File

@@ -26,6 +26,8 @@ import (
"testing"
"time"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/consensus/ethash"
"github.com/ethereum/go-ethereum/core/rawdb"
@@ -1795,6 +1797,13 @@ func testRepairWithScheme(t *testing.T, tt *rewindTest, snapshots bool, scheme s
config.SnapshotWait = true
}
config.TriesInMemory = 128
if err = db.SetupFreezerEnv(&ethdb.FreezerEnv{
ChainCfg: gspec.Config,
BlobExtraReserve: params.DefaultExtraReserveForBlobRequests,
}); err != nil {
t.Fatalf("Failed to create chain: %v", err)
}
chain, err := NewBlockChain(db, config, gspec, nil, engine, vm.Config{}, nil, nil)
if err != nil {
t.Fatalf("Failed to create chain: %v", err)

View File

@@ -27,6 +27,8 @@ import (
"testing"
"time"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/consensus/ethash"
"github.com/ethereum/go-ethereum/core/rawdb"
@@ -1998,6 +2000,13 @@ func testSetHeadWithScheme(t *testing.T, tt *rewindTest, snapshots bool, scheme
config.SnapshotWait = true
}
config.TriesInMemory = 128
if err = db.SetupFreezerEnv(&ethdb.FreezerEnv{
ChainCfg: gspec.Config,
BlobExtraReserve: params.DefaultExtraReserveForBlobRequests,
}); err != nil {
t.Fatalf("Failed to create chain: %v", err)
}
chain, err := NewBlockChain(db, config, gspec, nil, engine, vm.Config{}, nil, nil)
if err != nil {
t.Fatalf("Failed to create chain: %v", err)

View File

@@ -974,7 +974,7 @@ func testFastVsFullChains(t *testing.T, scheme string) {
t.Fatalf("failed to insert receipt %d: %v", n, err)
}
// Freezer style fast import the chain.
ancientDb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
ancientDb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create temp freezer db: %v", err)
}
@@ -1069,7 +1069,7 @@ func testLightVsFastVsFullChainHeads(t *testing.T, scheme string) {
// makeDb creates a db instance for testing.
makeDb := func() ethdb.Database {
db, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
db, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create temp freezer db: %v", err)
}
@@ -1957,7 +1957,7 @@ func testLargeReorgTrieGC(t *testing.T, scheme string) {
competitor, _ := GenerateChain(genesis.Config, shared[len(shared)-1], engine, genDb, 2*TriesInMemory+1, func(i int, b *BlockGen) { b.SetCoinbase(common.Address{3}) })
// Import the shared chain and the original canonical one
db, _ := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
db, _ := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
defer db.Close()
chain, err := NewBlockChain(db, DefaultCacheConfigWithScheme(scheme), genesis, nil, engine, vm.Config{}, nil, nil)
@@ -2026,7 +2026,7 @@ func testBlockchainRecovery(t *testing.T, scheme string) {
_, blocks, receipts := GenerateChainWithGenesis(gspec, ethash.NewFaker(), int(height), nil)
// Import the chain as a ancient-first node and ensure all pointers are updated
ancientDb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
ancientDb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create temp freezer db: %v", err)
}
@@ -2097,7 +2097,7 @@ func testInsertReceiptChainRollback(t *testing.T, scheme string) {
}
// Set up a BlockChain that uses the ancient store.
ancientDb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
ancientDb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create temp freezer db: %v", err)
}
@@ -2167,7 +2167,7 @@ func testLowDiffLongChain(t *testing.T, scheme string) {
})
// Import the canonical chain
diskdb, _ := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
diskdb, _ := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
defer diskdb.Close()
chain, err := NewBlockChain(diskdb, DefaultCacheConfigWithScheme(scheme), genesis, nil, engine, vm.Config{}, nil, nil)
@@ -2384,7 +2384,7 @@ func testInsertKnownChainData(t *testing.T, typ string, scheme string) {
b.OffsetTime(-9) // A higher difficulty
})
// Import the shared chain and the original canonical one
chaindb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
chaindb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create temp freezer db: %v", err)
}
@@ -2555,7 +2555,7 @@ func testInsertKnownChainDataWithMerging(t *testing.T, typ string, mergeHeight i
}
})
// Import the shared chain and the original canonical one
chaindb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
chaindb, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create temp freezer db: %v", err)
}
@@ -3858,7 +3858,7 @@ func testSetCanonical(t *testing.T, scheme string) {
}
gen.AddTx(tx)
})
diskdb, _ := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false)
diskdb, _ := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), t.TempDir(), "", false, false, false, false, false)
defer diskdb.Close()
chain, err := NewBlockChain(diskdb, DefaultCacheConfigWithScheme(scheme), gspec, nil, engine, vm.Config{}, nil, nil)
@@ -4483,7 +4483,7 @@ func (c *mockParlia) CalcDifficulty(chain consensus.ChainHeaderReader, time uint
func TestParliaBlobFeeReward(t *testing.T) {
// Have N headers in the freezer
frdir := t.TempDir()
db, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), frdir, "", false, false, false, false)
db, err := rawdb.NewDatabaseWithFreezer(rawdb.NewMemoryDatabase(), frdir, "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create database with ancient backend")
}

View File

@@ -401,7 +401,7 @@ func GenerateChain(config *params.ChainConfig, parent *types.Block, engine conse
defer triedb.Close()
for i := 0; i < n; i++ {
statedb, err := state.New(parent.Root(), state.NewDatabaseWithNodeDB(db, triedb), nil)
statedb, err := state.New(parent.Root(), state.NewDatabaseWithNodeDB(db, triedb, true), nil)
if err != nil {
panic(err)
}

View File

@@ -5,6 +5,9 @@ import (
"errors"
"fmt"
"sync"
"time"
"github.com/ethereum/go-ethereum/metrics"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/gopool"
@@ -14,6 +17,10 @@ import (
"github.com/ethereum/go-ethereum/params"
)
var (
daCheckTimer = metrics.NewRegisteredTimer("chain/dacheck", nil)
)
// validateBlobSidecar it is same as validateBlobSidecar in core/txpool/validation.go
func validateBlobSidecar(hashes []common.Hash, sidecar *types.BlobSidecar) error {
if len(sidecar.Blobs) != len(hashes) {
@@ -46,6 +53,10 @@ func validateBlobSidecar(hashes []common.Hash, sidecar *types.BlobSidecar) error
// IsDataAvailable it checks that the blobTx block has available blob data
func IsDataAvailable(chain consensus.ChainHeaderReader, block *types.Block) (err error) {
defer func(start time.Time) {
daCheckTimer.Update(time.Since(start))
}(time.Now())
// refer logic in ValidateBody
if !chain.Config().IsCancun(block.Number(), block.Time()) {
if block.Sidecars() != nil {

View File

@@ -86,9 +86,16 @@ func (f *ForkChoice) ReorgNeeded(current *types.Header, extern *types.Header) (b
localTD = f.chain.GetTd(current.Hash(), current.Number.Uint64())
externTd = f.chain.GetTd(extern.Hash(), extern.Number.Uint64())
)
if localTD == nil || externTd == nil {
if localTD == nil {
return false, errors.New("missing td")
}
if externTd == nil {
ptd := f.chain.GetTd(extern.ParentHash, extern.Number.Uint64()-1)
if ptd == nil {
return false, consensus.ErrUnknownAncestor
}
externTd = new(big.Int).Add(ptd, extern.Difficulty)
}
// Accept the new header as the chain head if the transition
// is already triggered. We assume all the headers after the
// transition come from the trusted consensus layer.
@@ -114,7 +121,9 @@ func (f *ForkChoice) ReorgNeeded(current *types.Header, extern *types.Header) (b
if f.preserve != nil {
currentPreserve, externPreserve = f.preserve(current), f.preserve(extern)
}
reorg = !currentPreserve && (externPreserve || f.rand.Float64() < 0.5)
reorg = !currentPreserve && (externPreserve ||
extern.Time < current.Time ||
extern.Time == current.Time && f.rand.Float64() < 0.5)
}
return reorg, nil
}

View File

@@ -126,7 +126,10 @@ func hashAlloc(ga *types.GenesisAlloc, isVerkle bool) (common.Hash, error) {
}
// Create an ephemeral in-memory database for computing hash,
// all the derived states will be discarded to not pollute disk.
db := state.NewDatabaseWithConfig(rawdb.NewMemoryDatabase(), config)
db := state.NewDatabaseWithConfig(rawdb.NewMemoryDatabase(), config, true)
log.Info("genesis calc root hash use hash mode triedb")
db.SetVersion(0)
defer db.Release()
statedb, err := state.New(types.EmptyRootHash, db, nil)
if err != nil {
return common.Hash{}, err
@@ -154,7 +157,10 @@ func flushAlloc(ga *types.GenesisAlloc, db ethdb.Database, triedb *triedb.Databa
if triedbConfig != nil {
triedbConfig.NoTries = false
}
statedb, err := state.New(types.EmptyRootHash, state.NewDatabaseWithNodeDB(db, triedb), nil)
cachingdb := state.NewDatabaseWithNodeDB(db, triedb, true)
cachingdb.SetVersion(-1)
defer cachingdb.Release()
statedb, err := state.New(types.EmptyRootHash, cachingdb, nil)
if err != nil {
return err
}
@@ -174,7 +180,7 @@ func flushAlloc(ga *types.GenesisAlloc, db ethdb.Database, triedb *triedb.Databa
return err
}
// Commit newly generated states into disk if it's not empty.
if root != types.EmptyRootHash {
if root != types.EmptyRootHash && triedb.Scheme() != rawdb.VersionScheme {
if err := triedb.Commit(root, true); err != nil {
return err
}
@@ -216,10 +222,8 @@ func (e *GenesisMismatchError) Error() string {
// ChainOverrides contains the changes to chain config
// Typically, these modifications involve hardforks that are not enabled on the BSC mainnet, intended for testing purposes.
type ChainOverrides struct {
OverrideCancun *uint64
OverrideVerkle *uint64
OverrideFeynman *uint64
OverrideFeynmanFix *uint64
OverrideBohr *uint64
OverrideVerkle *uint64
}
// SetupGenesisBlock writes or updates the genesis block in db.
@@ -245,18 +249,12 @@ func SetupGenesisBlockWithOverride(db ethdb.Database, triedb *triedb.Database, g
}
applyOverrides := func(config *params.ChainConfig) {
if config != nil {
if overrides != nil && overrides.OverrideCancun != nil {
config.CancunTime = overrides.OverrideCancun
if overrides != nil && overrides.OverrideBohr != nil {
config.BohrTime = overrides.OverrideBohr
}
if overrides != nil && overrides.OverrideVerkle != nil {
config.VerkleTime = overrides.OverrideVerkle
}
if overrides != nil && overrides.OverrideFeynman != nil {
config.FeynmanTime = overrides.OverrideFeynman
}
if overrides != nil && overrides.OverrideFeynmanFix != nil {
config.FeynmanFixTime = overrides.OverrideFeynmanFix
}
}
}
// Just commit the new block if there is no stored genesis block.
@@ -276,6 +274,8 @@ func SetupGenesisBlockWithOverride(db ethdb.Database, triedb *triedb.Database, g
log.Info("genesis block hash", "hash", block.Hash())
return genesis.Config, block.Hash(), nil
}
log.Info("init genesis", "stored root", stored.String())
// The genesis block is present(perhaps in ancient database) while the
// state database is not initialized yet. It can happen that the node
// is initialized with an external ancient store. Commit genesis state
@@ -493,13 +493,13 @@ func (g *Genesis) Commit(db ethdb.Database, triedb *triedb.Database) (*types.Blo
if err := flushAlloc(&g.Alloc, db, triedb, block.Hash()); err != nil {
return nil, err
}
rawdb.WriteTd(db, block.Hash(), block.NumberU64(), block.Difficulty())
rawdb.WriteBlock(db, block)
rawdb.WriteReceipts(db, block.Hash(), block.NumberU64(), nil)
rawdb.WriteCanonicalHash(db, block.Hash(), block.NumberU64())
rawdb.WriteHeadBlockHash(db, block.Hash())
rawdb.WriteHeadFastBlockHash(db, block.Hash())
rawdb.WriteHeadHeaderHash(db, block.Hash())
rawdb.WriteTd(db.BlockStore(), block.Hash(), block.NumberU64(), block.Difficulty())
rawdb.WriteBlock(db.BlockStore(), block)
rawdb.WriteReceipts(db.BlockStore(), block.Hash(), block.NumberU64(), nil)
rawdb.WriteCanonicalHash(db.BlockStore(), block.Hash(), block.NumberU64())
rawdb.WriteHeadBlockHash(db.BlockStore(), block.Hash())
rawdb.WriteHeadFastBlockHash(db.BlockStore(), block.Hash())
rawdb.WriteHeadHeaderHash(db.BlockStore(), block.Hash())
rawdb.WriteChainConfig(db, block.Hash(), config)
return block, nil
}

View File

@@ -172,9 +172,9 @@ func (hc *HeaderChain) Reorg(headers []*types.Header) error {
// pile them onto the existing chain. Otherwise, do the necessary
// reorgs.
var (
first = headers[0]
last = headers[len(headers)-1]
batch = hc.chainDb.NewBatch()
first = headers[0]
last = headers[len(headers)-1]
blockBatch = hc.chainDb.BlockStore().NewBatch()
)
if first.ParentHash != hc.currentHeaderHash {
// Delete any canonical number assignments above the new head
@@ -183,7 +183,7 @@ func (hc *HeaderChain) Reorg(headers []*types.Header) error {
if hash == (common.Hash{}) {
break
}
rawdb.DeleteCanonicalHash(batch, i)
rawdb.DeleteCanonicalHash(blockBatch, i)
}
// Overwrite any stale canonical number assignments, going
// backwards from the first header in this import until the
@@ -194,7 +194,7 @@ func (hc *HeaderChain) Reorg(headers []*types.Header) error {
headHash = header.Hash()
)
for rawdb.ReadCanonicalHash(hc.chainDb, headNumber) != headHash {
rawdb.WriteCanonicalHash(batch, headHash, headNumber)
rawdb.WriteCanonicalHash(blockBatch, headHash, headNumber)
if headNumber == 0 {
break // It shouldn't be reached
}
@@ -209,16 +209,16 @@ func (hc *HeaderChain) Reorg(headers []*types.Header) error {
for i := 0; i < len(headers)-1; i++ {
hash := headers[i+1].ParentHash // Save some extra hashing
num := headers[i].Number.Uint64()
rawdb.WriteCanonicalHash(batch, hash, num)
rawdb.WriteHeadHeaderHash(batch, hash)
rawdb.WriteCanonicalHash(blockBatch, hash, num)
rawdb.WriteHeadHeaderHash(blockBatch, hash)
}
// Write the last header
hash := headers[len(headers)-1].Hash()
num := headers[len(headers)-1].Number.Uint64()
rawdb.WriteCanonicalHash(batch, hash, num)
rawdb.WriteHeadHeaderHash(batch, hash)
rawdb.WriteCanonicalHash(blockBatch, hash, num)
rawdb.WriteHeadHeaderHash(blockBatch, hash)
if err := batch.Write(); err != nil {
if err := blockBatch.Write(); err != nil {
return err
}
// Last step update all in-memory head header markers
@@ -244,7 +244,7 @@ func (hc *HeaderChain) WriteHeaders(headers []*types.Header) (int, error) {
newTD = new(big.Int).Set(ptd) // Total difficulty of inserted chain
inserted []rawdb.NumberHash // Ephemeral lookup of number/hash for the chain
parentKnown = true // Set to true to force hc.HasHeader check the first iteration
batch = hc.chainDb.NewBatch()
blockBatch = hc.chainDb.BlockStore().NewBatch()
)
for i, header := range headers {
var hash common.Hash
@@ -264,10 +264,10 @@ func (hc *HeaderChain) WriteHeaders(headers []*types.Header) (int, error) {
alreadyKnown := parentKnown && hc.HasHeader(hash, number)
if !alreadyKnown {
// Irrelevant of the canonical status, write the TD and header to the database.
rawdb.WriteTd(batch, hash, number, newTD)
rawdb.WriteTd(blockBatch, hash, number, newTD)
hc.tdCache.Add(hash, new(big.Int).Set(newTD))
rawdb.WriteHeader(batch, header)
rawdb.WriteHeader(blockBatch, header)
inserted = append(inserted, rawdb.NumberHash{Number: number, Hash: hash})
hc.headerCache.Add(hash, header)
hc.numberCache.Add(hash, number)
@@ -280,7 +280,7 @@ func (hc *HeaderChain) WriteHeaders(headers []*types.Header) (int, error) {
return 0, errors.New("aborted")
}
// Commit to disk!
if err := batch.Write(); err != nil {
if err := blockBatch.Write(); err != nil {
log.Crit("Failed to write headers", "error", err)
}
return len(inserted), nil
@@ -642,7 +642,7 @@ func (hc *HeaderChain) setHead(headBlock uint64, headTime uint64, updateFn Updat
}
var (
parentHash common.Hash
batch = hc.chainDb.NewBatch()
blockBatch = hc.chainDb.BlockStore().NewBatch()
origin = true
)
done := func(header *types.Header) bool {
@@ -668,7 +668,7 @@ func (hc *HeaderChain) setHead(headBlock uint64, headTime uint64, updateFn Updat
// first then remove the relative data from the database.
//
// Update head first(head fast block, head full block) before deleting the data.
markerBatch := hc.chainDb.NewBatch()
markerBatch := hc.chainDb.BlockStore().NewBatch()
if updateFn != nil {
newHead, force := updateFn(markerBatch, parent)
if force && ((headTime > 0 && newHead.Time < headTime) || (headTime == 0 && newHead.Number.Uint64() < headBlock)) {
@@ -691,7 +691,7 @@ func (hc *HeaderChain) setHead(headBlock uint64, headTime uint64, updateFn Updat
// we don't end up with dangling daps in the database
var nums []uint64
if origin {
for n := num + 1; len(rawdb.ReadAllHashes(hc.chainDb, n)) > 0; n++ {
for n := num + 1; len(rawdb.ReadAllHashes(hc.chainDb.BlockStore(), n)) > 0; n++ {
nums = append([]uint64{n}, nums...) // suboptimal, but we don't really expect this path
}
origin = false
@@ -701,23 +701,23 @@ func (hc *HeaderChain) setHead(headBlock uint64, headTime uint64, updateFn Updat
// Remove the related data from the database on all sidechains
for _, num := range nums {
// Gather all the side fork hashes
hashes := rawdb.ReadAllHashes(hc.chainDb, num)
hashes := rawdb.ReadAllHashes(hc.chainDb.BlockStore(), num)
if len(hashes) == 0 {
// No hashes in the database whatsoever, probably frozen already
hashes = append(hashes, hdr.Hash())
}
for _, hash := range hashes {
if delFn != nil {
delFn(batch, hash, num)
delFn(blockBatch, hash, num)
}
rawdb.DeleteHeader(batch, hash, num)
rawdb.DeleteTd(batch, hash, num)
rawdb.DeleteHeader(blockBatch, hash, num)
rawdb.DeleteTd(blockBatch, hash, num)
}
rawdb.DeleteCanonicalHash(batch, num)
rawdb.DeleteCanonicalHash(blockBatch, num)
}
}
// Flush all accumulated deletions.
if err := batch.Write(); err != nil {
if err := blockBatch.Write(); err != nil {
log.Crit("Failed to rewind block", "error", err)
}
// Clear out any stale content from the caches

View File

@@ -34,14 +34,23 @@ import (
"golang.org/x/exp/slices"
)
// Support Multi-Database Based on Data Pattern, the Chaindata will be divided into three stores: BlockStore, StateStore, and ChainStore,
// according to data schema and read/write behavior. When using the following data interfaces, you should take note of the following:
//
// 1) Block-Related Data: For CanonicalHash, Header, Body, Td, Receipts, and BlobSidecars, the Write, Delete, and Iterator
// operations should carefully ensure that the database being used is BlockStore.
// 2) Meta-Related Data: For HeaderNumber, HeadHeaderHash, HeadBlockHash, HeadFastBlockHash, and FinalizedBlockHash, the
// Write and Delete operations should carefully ensure that the database being used is BlockStore.
// 3) Ancient Data: When using a multi-database, Ancient data will use the BlockStore.
// ReadCanonicalHash retrieves the hash assigned to a canonical block number.
func ReadCanonicalHash(db ethdb.Reader, number uint64) common.Hash {
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
data, _ = reader.Ancient(ChainFreezerHashTable, number)
if len(data) == 0 {
// Get it by hash from leveldb
data, _ = db.Get(headerHashKey(number))
data, _ = db.BlockStoreReader().Get(headerHashKey(number))
}
return nil
})
@@ -144,8 +153,8 @@ func ReadAllCanonicalHashes(db ethdb.Iteratee, from uint64, to uint64, limit int
}
// ReadHeaderNumber returns the header number assigned to a hash.
func ReadHeaderNumber(db ethdb.KeyValueReader, hash common.Hash) *uint64 {
data, _ := db.Get(headerNumberKey(hash))
func ReadHeaderNumber(db ethdb.MultiDatabaseReader, hash common.Hash) *uint64 {
data, _ := db.BlockStoreReader().Get(headerNumberKey(hash))
if len(data) != 8 {
return nil
}
@@ -170,8 +179,8 @@ func DeleteHeaderNumber(db ethdb.KeyValueWriter, hash common.Hash) {
}
// ReadHeadHeaderHash retrieves the hash of the current canonical head header.
func ReadHeadHeaderHash(db ethdb.KeyValueReader) common.Hash {
data, _ := db.Get(headHeaderKey)
func ReadHeadHeaderHash(db ethdb.MultiDatabaseReader) common.Hash {
data, _ := db.BlockStoreReader().Get(headHeaderKey)
if len(data) == 0 {
return common.Hash{}
}
@@ -186,8 +195,8 @@ func WriteHeadHeaderHash(db ethdb.KeyValueWriter, hash common.Hash) {
}
// ReadHeadBlockHash retrieves the hash of the current canonical head block.
func ReadHeadBlockHash(db ethdb.KeyValueReader) common.Hash {
data, _ := db.Get(headBlockKey)
func ReadHeadBlockHash(db ethdb.MultiDatabaseReader) common.Hash {
data, _ := db.BlockStoreReader().Get(headBlockKey)
if len(data) == 0 {
return common.Hash{}
}
@@ -202,8 +211,8 @@ func WriteHeadBlockHash(db ethdb.KeyValueWriter, hash common.Hash) {
}
// ReadHeadFastBlockHash retrieves the hash of the current fast-sync head block.
func ReadHeadFastBlockHash(db ethdb.KeyValueReader) common.Hash {
data, _ := db.Get(headFastBlockKey)
func ReadHeadFastBlockHash(db ethdb.MultiDatabaseReader) common.Hash {
data, _ := db.BlockStoreReader().Get(headFastBlockKey)
if len(data) == 0 {
return common.Hash{}
}
@@ -217,6 +226,22 @@ func WriteHeadFastBlockHash(db ethdb.KeyValueWriter, hash common.Hash) {
}
}
// ReadFinalizedBlockHash retrieves the hash of the finalized block.
func ReadFinalizedBlockHash(db ethdb.MultiDatabaseReader) common.Hash {
data, _ := db.BlockStoreReader().Get(headFinalizedBlockKey)
if len(data) == 0 {
return common.Hash{}
}
return common.BytesToHash(data)
}
// WriteFinalizedBlockHash stores the hash of the finalized block.
func WriteFinalizedBlockHash(db ethdb.KeyValueWriter, hash common.Hash) {
if err := db.Put(headFinalizedBlockKey, hash.Bytes()); err != nil {
log.Crit("Failed to store last finalized block's hash", "err", err)
}
}
// ReadLastPivotNumber retrieves the number of the last pivot block. If the node
// full synced, the last pivot will always be nil.
func ReadLastPivotNumber(db ethdb.KeyValueReader) *uint64 {
@@ -281,13 +306,13 @@ func ReadHeaderRange(db ethdb.Reader, number uint64, count uint64) []rlp.RawValu
// It's ok to request block 0, 1 item
count = number + 1
}
limit, _ := db.Ancients()
limit, _ := db.BlockStoreReader().Ancients()
// First read live blocks
if i >= limit {
// If we need to read live blocks, we need to figure out the hash first
hash := ReadCanonicalHash(db, number)
for ; i >= limit && count > 0; i-- {
if data, _ := db.Get(headerKey(i, hash)); len(data) > 0 {
if data, _ := db.BlockStoreReader().Get(headerKey(i, hash)); len(data) > 0 {
rlpHeaders = append(rlpHeaders, data)
// Get the parent hash for next query
hash = types.HeaderParentHashFromRLP(data)
@@ -300,8 +325,8 @@ func ReadHeaderRange(db ethdb.Reader, number uint64, count uint64) []rlp.RawValu
if count == 0 {
return rlpHeaders
}
// read remaining from ancients
data, err := db.AncientRange(ChainFreezerHeaderTable, i+1-count, count, 0)
// read remaining from ancients, cap at 2M
data, err := db.BlockStoreReader().AncientRange(ChainFreezerHeaderTable, i+1-count, count, 2*1024*1024)
if err != nil {
log.Error("Failed to read headers from freezer", "err", err)
return rlpHeaders
@@ -320,7 +345,7 @@ func ReadHeaderRange(db ethdb.Reader, number uint64, count uint64) []rlp.RawValu
// ReadHeaderRLP retrieves a block header in its raw RLP database encoding.
func ReadHeaderRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValue {
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
// First try to look up the data in ancient database. Extra hash
// comparison is necessary since ancient database only maintains
// the canonical data.
@@ -329,7 +354,7 @@ func ReadHeaderRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValu
return nil
}
// If not, try reading from leveldb
data, _ = db.Get(headerKey(number, hash))
data, _ = db.BlockStoreReader().Get(headerKey(number, hash))
return nil
})
return data
@@ -337,10 +362,10 @@ func ReadHeaderRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValu
// HasHeader verifies the existence of a block header corresponding to the hash.
func HasHeader(db ethdb.Reader, hash common.Hash, number uint64) bool {
if isCanon(db, number, hash) {
if isCanon(db.BlockStoreReader(), number, hash) {
return true
}
if has, err := db.Has(headerKey(number, hash)); !has || err != nil {
if has, err := db.BlockStoreReader().Has(headerKey(number, hash)); !has || err != nil {
return false
}
return true
@@ -427,14 +452,14 @@ func ReadBodyRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValue
// comparison is necessary since ancient database only maintains
// the canonical data.
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
// Check if the data is in ancients
if isCanon(reader, number, hash) {
data, _ = reader.Ancient(ChainFreezerBodiesTable, number)
return nil
}
// If not, try reading from leveldb
data, _ = db.Get(blockBodyKey(number, hash))
data, _ = db.BlockStoreReader().Get(blockBodyKey(number, hash))
return nil
})
return data
@@ -444,7 +469,7 @@ func ReadBodyRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValue
// block at number, in RLP encoding.
func ReadCanonicalBodyRLP(db ethdb.Reader, number uint64) rlp.RawValue {
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
data, _ = reader.Ancient(ChainFreezerBodiesTable, number)
if len(data) > 0 {
return nil
@@ -452,8 +477,8 @@ func ReadCanonicalBodyRLP(db ethdb.Reader, number uint64) rlp.RawValue {
// Block is not in ancients, read from leveldb by hash and number.
// Note: ReadCanonicalHash cannot be used here because it also
// calls ReadAncients internally.
hash, _ := db.Get(headerHashKey(number))
data, _ = db.Get(blockBodyKey(number, common.BytesToHash(hash)))
hash, _ := db.BlockStoreReader().Get(headerHashKey(number))
data, _ = db.BlockStoreReader().Get(blockBodyKey(number, common.BytesToHash(hash)))
return nil
})
return data
@@ -468,10 +493,10 @@ func WriteBodyRLP(db ethdb.KeyValueWriter, hash common.Hash, number uint64, rlp
// HasBody verifies the existence of a block body corresponding to the hash.
func HasBody(db ethdb.Reader, hash common.Hash, number uint64) bool {
if isCanon(db, number, hash) {
if isCanon(db.BlockStoreReader(), number, hash) {
return true
}
if has, err := db.Has(blockBodyKey(number, hash)); !has || err != nil {
if has, err := db.BlockStoreReader().Has(blockBodyKey(number, hash)); !has || err != nil {
return false
}
return true
@@ -500,6 +525,13 @@ func WriteBody(db ethdb.KeyValueWriter, hash common.Hash, number uint64, body *t
WriteBodyRLP(db, hash, number, data)
}
// DeleteBody removes all block body data associated with a hash.
func DeleteBody(db ethdb.KeyValueWriter, hash common.Hash, number uint64) {
if err := db.Delete(blockBodyKey(number, hash)); err != nil {
log.Crit("Failed to delete block body", "err", err)
}
}
func WriteDiffLayer(db ethdb.KeyValueWriter, hash common.Hash, layer *types.DiffLayer) {
data, err := rlp.EncodeToBytes(layer)
if err != nil {
@@ -538,24 +570,17 @@ func DeleteDiffLayer(db ethdb.KeyValueWriter, blockHash common.Hash) {
}
}
// DeleteBody removes all block body data associated with a hash.
func DeleteBody(db ethdb.KeyValueWriter, hash common.Hash, number uint64) {
if err := db.Delete(blockBodyKey(number, hash)); err != nil {
log.Crit("Failed to delete block body", "err", err)
}
}
// ReadTdRLP retrieves a block's total difficulty corresponding to the hash in RLP encoding.
func ReadTdRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValue {
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
// Check if the data is in ancients
if isCanon(reader, number, hash) {
data, _ = reader.Ancient(ChainFreezerDifficultyTable, number)
return nil
}
// If not, try reading from leveldb
data, _ = db.Get(headerTDKey(number, hash))
data, _ = db.BlockStoreReader().Get(headerTDKey(number, hash))
return nil
})
return data
@@ -596,10 +621,10 @@ func DeleteTd(db ethdb.KeyValueWriter, hash common.Hash, number uint64) {
// HasReceipts verifies the existence of all the transaction receipts belonging
// to a block.
func HasReceipts(db ethdb.Reader, hash common.Hash, number uint64) bool {
if isCanon(db, number, hash) {
if isCanon(db.BlockStoreReader(), number, hash) {
return true
}
if has, err := db.Has(blockReceiptsKey(number, hash)); !has || err != nil {
if has, err := db.BlockStoreReader().Has(blockReceiptsKey(number, hash)); !has || err != nil {
return false
}
return true
@@ -608,14 +633,14 @@ func HasReceipts(db ethdb.Reader, hash common.Hash, number uint64) bool {
// ReadReceiptsRLP retrieves all the transaction receipts belonging to a block in RLP encoding.
func ReadReceiptsRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValue {
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
// Check if the data is in ancients
if isCanon(reader, number, hash) {
data, _ = reader.Ancient(ChainFreezerReceiptTable, number)
return nil
}
// If not, try reading from leveldb
data, _ = db.Get(blockReceiptsKey(number, hash))
data, _ = db.BlockStoreReader().Get(blockReceiptsKey(number, hash))
return nil
})
return data
@@ -868,14 +893,14 @@ func WriteAncientBlocks(db ethdb.AncientWriter, blocks []*types.Block, receipts
// ReadBlobSidecarsRLP retrieves all the transaction blobs belonging to a block in RLP encoding.
func ReadBlobSidecarsRLP(db ethdb.Reader, hash common.Hash, number uint64) rlp.RawValue {
var data []byte
db.ReadAncients(func(reader ethdb.AncientReaderOp) error {
db.BlockStoreReader().ReadAncients(func(reader ethdb.AncientReaderOp) error {
// Check if the data is in ancients
if isCanon(reader, number, hash) {
data, _ = reader.Ancient(ChainFreezerBlobSidecarTable, number)
return nil
}
// If not, try reading from leveldb
data, _ = db.Get(blockBlobSidecarsKey(number, hash))
data, _ = db.BlockStoreReader().Get(blockBlobSidecarsKey(number, hash))
return nil
})
return data

View File

@@ -518,7 +518,7 @@ func checkBlobSidecarsRLP(have, want types.BlobSidecars) error {
func TestAncientStorage(t *testing.T) {
// Freezer style fast import the chain.
frdir := t.TempDir()
db, err := NewDatabaseWithFreezer(NewMemoryDatabase(), frdir, "", false, false, false, false)
db, err := NewDatabaseWithFreezer(NewMemoryDatabase(), frdir, "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create database with ancient backend")
}
@@ -657,7 +657,7 @@ func TestHashesInRange(t *testing.T) {
func BenchmarkWriteAncientBlocks(b *testing.B) {
// Open freezer database.
frdir := b.TempDir()
db, err := NewDatabaseWithFreezer(NewMemoryDatabase(), frdir, "", false, false, false, false)
db, err := NewDatabaseWithFreezer(NewMemoryDatabase(), frdir, "", false, false, false, false, false)
if err != nil {
b.Fatalf("failed to create database with ancient backend")
}
@@ -1001,7 +1001,7 @@ func TestHeadersRLPStorage(t *testing.T) {
// Have N headers in the freezer
frdir := t.TempDir()
db, err := NewDatabaseWithFreezer(NewMemoryDatabase(), frdir, "", false, false, false, false)
db, err := NewDatabaseWithFreezer(NewMemoryDatabase(), frdir, "", false, false, false, false, false)
if err != nil {
t.Fatalf("failed to create database with ancient backend")
}

View File

@@ -46,6 +46,8 @@ const HashScheme = "hash"
// on extra state diffs to survive deep reorg.
const PathScheme = "path"
const VersionScheme = "version"
// hasher is used to compute the sha256 hash of the provided data.
type hasher struct{ sha crypto.KeccakState }
@@ -314,7 +316,7 @@ func ReadStateScheme(db ethdb.Reader) string {
// ValidateStateScheme used to check state scheme whether is valid.
// Valid state scheme: hash and path.
func ValidateStateScheme(stateScheme string) bool {
if stateScheme == HashScheme || stateScheme == PathScheme {
if stateScheme == HashScheme || stateScheme == PathScheme || stateScheme == VersionScheme {
return true
}
return false

View File

@@ -18,6 +18,8 @@ package rawdb
import (
"fmt"
"io"
"os"
"path/filepath"
"github.com/ethereum/go-ethereum/common"
@@ -98,6 +100,18 @@ func inspectFreezers(db ethdb.Database) ([]freezerInfo, error) {
if err != nil {
return nil, err
}
file, err := os.Open(filepath.Join(datadir, StateFreezerName))
if err != nil {
return nil, err
}
defer file.Close()
// if state freezer folder has been pruned, there is no need for inspection
_, err = file.Readdirnames(1)
if err == io.EOF {
continue
}
f, err := NewStateFreezer(datadir, true, 0)
if err != nil {
return nil, err
@@ -121,16 +135,25 @@ func inspectFreezers(db ethdb.Database) ([]freezerInfo, error) {
// ancient indicates the path of root ancient directory where the chain freezer can
// be opened. Start and end specify the range for dumping out indexes.
// Note this function can only be used for debugging purposes.
func InspectFreezerTable(ancient string, freezerName string, tableName string, start, end int64) error {
func InspectFreezerTable(ancient string, freezerName string, tableName string, start, end int64, multiDatabase bool) error {
var (
path string
tables map[string]bool
)
switch freezerName {
case ChainFreezerName:
path, tables = resolveChainFreezerDir(ancient), chainFreezerNoSnappy
if multiDatabase {
path, tables = resolveChainFreezerDir(filepath.Dir(ancient)+"/block/ancient"), chainFreezerNoSnappy
} else {
path, tables = resolveChainFreezerDir(ancient), chainFreezerNoSnappy
}
case StateFreezerName:
path, tables = filepath.Join(ancient, freezerName), stateFreezerNoSnappy
if multiDatabase {
path, tables = filepath.Join(filepath.Dir(ancient)+"/state/ancient", freezerName), stateFreezerNoSnappy
} else {
path, tables = filepath.Join(ancient, freezerName), stateFreezerNoSnappy
}
default:
return fmt.Errorf("unknown freezer, supported ones: %v", freezers)
}

View File

@@ -24,12 +24,12 @@ import (
"sync/atomic"
"time"
"github.com/ethereum/go-ethereum/rlp"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/log"
"github.com/ethereum/go-ethereum/params"
"github.com/ethereum/go-ethereum/rlp"
)
const (
@@ -58,11 +58,14 @@ type chainFreezer struct {
wg sync.WaitGroup
trigger chan chan struct{} // Manual blocking freeze trigger, test determinism
freezeEnv atomic.Value
freezeEnv atomic.Value
waitEnvTimes int
multiDatabase bool
}
// newChainFreezer initializes the freezer for ancient chain data.
func newChainFreezer(datadir string, namespace string, readonly bool, offset uint64) (*chainFreezer, error) {
func newChainFreezer(datadir string, namespace string, readonly bool, offset uint64, multiDatabase bool) (*chainFreezer, error) {
freezer, err := NewChainFreezer(datadir, namespace, readonly, offset)
if err != nil {
return nil, err
@@ -87,6 +90,57 @@ func (f *chainFreezer) Close() error {
return f.Freezer.Close()
}
// readHeadNumber returns the number of chain head block. 0 is returned if the
// block is unknown or not available yet.
func (f *chainFreezer) readHeadNumber(db ethdb.Reader) uint64 {
hash := ReadHeadBlockHash(db)
if hash == (common.Hash{}) {
log.Error("Head block is not reachable")
return 0
}
number := ReadHeaderNumber(db, hash)
if number == nil {
log.Error("Number of head block is missing")
return 0
}
return *number
}
// readFinalizedNumber returns the number of finalized block. 0 is returned
// if the block is unknown or not available yet.
func (f *chainFreezer) readFinalizedNumber(db ethdb.Reader) uint64 {
hash := ReadFinalizedBlockHash(db)
if hash == (common.Hash{}) {
return 0
}
number := ReadHeaderNumber(db, hash)
if number == nil {
log.Error("Number of finalized block is missing")
return 0
}
return *number
}
// freezeThreshold returns the threshold for chain freezing. It's determined
// by formula: max(finality, HEAD-params.FullImmutabilityThreshold).
func (f *chainFreezer) freezeThreshold(db ethdb.Reader) (uint64, error) {
var (
head = f.readHeadNumber(db)
final = f.readFinalizedNumber(db)
headLimit uint64
)
if head > params.FullImmutabilityThreshold {
headLimit = head - params.FullImmutabilityThreshold
}
if final == 0 && headLimit == 0 {
return 0, errors.New("freezing threshold is not available")
}
if final > headLimit {
return final, nil
}
return headLimit, nil
}
// freeze is a background thread that periodically checks the blockchain for any
// import progress and moves ancient data from the fast database into the freezer.
//
@@ -125,74 +179,125 @@ func (f *chainFreezer) freeze(db ethdb.KeyValueStore) {
}
}
// check freezer env first, it must wait a while when the env is necessary
err := f.checkFreezerEnv()
if err == missFreezerEnvErr {
log.Warn("Freezer need related env, may wait for a while", "err", err)
backoff = true
continue
}
if err != nil {
log.Error("Freezer check FreezerEnv err", "err", err)
backoff = true
continue
var (
frozen uint64
threshold uint64
first uint64 // the first block to freeze
last uint64 // the last block to freeze
hash common.Hash
number *uint64
head *types.Header
err error
)
// use finalized block as the chain freeze indicator was used for multiDatabase feature, if multiDatabase is false, keep 9W blocks in db
if f.multiDatabase {
threshold, err = f.freezeThreshold(nfdb)
if err != nil {
backoff = true
log.Debug("Current full block not old enough to freeze", "err", err)
continue
}
frozen = f.frozen.Load()
// Short circuit if the blocks below threshold are already frozen.
if frozen != 0 && frozen-1 >= threshold {
backoff = true
log.Debug("Ancient blocks frozen already", "threshold", threshold, "frozen", frozen)
continue
}
hash = ReadHeadBlockHash(nfdb)
if hash == (common.Hash{}) {
log.Debug("Current full block hash unavailable") // new chain, empty database
backoff = true
continue
}
number = ReadHeaderNumber(nfdb, hash)
if number == nil {
log.Error("Current full block number unavailable", "hash", hash)
backoff = true
continue
}
head = ReadHeader(nfdb, hash, *number)
if head == nil {
log.Error("Current full block unavailable", "number", *number, "hash", hash)
backoff = true
continue
}
first = frozen
last = threshold
if last-first+1 > freezerBatchLimit {
last = freezerBatchLimit + first - 1
}
} else {
// Retrieve the freezing threshold.
hash = ReadHeadBlockHash(nfdb)
if hash == (common.Hash{}) {
log.Debug("Current full block hash unavailable") // new chain, empty database
backoff = true
continue
}
number = ReadHeaderNumber(nfdb, hash)
threshold = f.threshold.Load()
frozen = f.frozen.Load()
switch {
case number == nil:
log.Error("Current full block number unavailable", "hash", hash)
backoff = true
continue
case *number < threshold:
log.Debug("Current full block not old enough to freeze", "number", *number, "hash", hash, "delay", threshold)
backoff = true
continue
case *number-threshold <= frozen:
log.Debug("Ancient blocks frozen already", "number", *number, "hash", hash, "frozen", frozen)
backoff = true
continue
}
head = ReadHeader(nfdb, hash, *number)
if head == nil {
log.Error("Current full block unavailable", "number", *number, "hash", hash)
backoff = true
continue
}
first, _ = f.Ancients()
last = *number - threshold
if last-first > freezerBatchLimit {
last = first + freezerBatchLimit
}
}
// Retrieve the freezing threshold.
hash := ReadHeadBlockHash(nfdb)
if hash == (common.Hash{}) {
log.Debug("Current full block hash unavailable") // new chain, empty database
backoff = true
continue
}
number := ReadHeaderNumber(nfdb, hash)
threshold := f.threshold.Load()
frozen := f.frozen.Load()
switch {
case number == nil:
log.Error("Current full block number unavailable", "hash", hash)
backoff = true
continue
case *number < threshold:
log.Debug("Current full block not old enough to freeze", "number", *number, "hash", hash, "delay", threshold)
backoff = true
continue
case *number-threshold <= frozen:
log.Debug("Ancient blocks frozen already", "number", *number, "hash", hash, "frozen", frozen)
backoff = true
continue
}
head := ReadHeader(nfdb, hash, *number)
if head == nil {
log.Error("Current full block unavailable", "number", *number, "hash", hash)
// check env first before chain freeze, it must wait when the env is necessary
if err := f.checkFreezerEnv(); err != nil {
f.waitEnvTimes++
if f.waitEnvTimes%30 == 0 {
log.Warn("Freezer need related env, may wait for a while, and it's not a issue when non-import block", "err", err)
return
}
backoff = true
continue
}
// Seems we have data ready to be frozen, process in usable batches
var (
start = time.Now()
first, _ = f.Ancients()
limit = *number - threshold
start = time.Now()
)
if limit-first > freezerBatchLimit {
limit = first + freezerBatchLimit
}
ancients, err := f.freezeRangeWithBlobs(nfdb, first, limit)
ancients, err := f.freezeRangeWithBlobs(nfdb, first, last)
if err != nil {
log.Error("Error in block freeze operation", "err", err)
backoff = true
continue
}
// Batch of blocks have been frozen, flush them before wiping from leveldb
if err := f.Sync(); err != nil {
log.Crit("Failed to flush frozen tables", "err", err)
}
// Wipe out all data from the active database
batch := db.NewBatch()
for i := 0; i < len(ancients); i++ {
@@ -358,13 +463,16 @@ func (f *chainFreezer) freezeRangeWithBlobs(nfdb *nofreezedb, number, limit uint
return hashes, err
}
// freezeRange moves a batch of chain segments from the fast database to the freezer.
// The parameters (number, limit) specify the relevant block range, both of which
// are included.
func (f *chainFreezer) freezeRange(nfdb *nofreezedb, number, limit uint64) (hashes []common.Hash, err error) {
if number > limit {
return nil, nil
}
env, _ := f.freezeEnv.Load().(*ethdb.FreezerEnv)
hashes = make([]common.Hash, 0, limit-number)
hashes = make([]common.Hash, 0, limit-number+1)
_, err = f.ModifyAncients(func(op ethdb.AncientWriteOp) error {
for ; number <= limit; number++ {
// Retrieve all the components of the canonical block.
@@ -437,14 +545,7 @@ func (f *chainFreezer) checkFreezerEnv() error {
if exist {
return nil
}
blobFrozen, err := f.TableAncients(ChainFreezerBlobSidecarTable)
if err != nil {
return err
}
if blobFrozen > 0 {
return missFreezerEnvErr
}
return nil
return missFreezerEnvErr
}
func isCancun(env *ethdb.FreezerEnv, num *big.Int, time uint64) bool {

View File

@@ -35,16 +35,16 @@ import (
// injects into the database the block hash->number mappings.
func InitDatabaseFromFreezer(db ethdb.Database) {
// If we can't access the freezer or it's empty, abort
frozen, err := db.ItemAmountInAncient()
frozen, err := db.BlockStore().ItemAmountInAncient()
if err != nil || frozen == 0 {
return
}
var (
batch = db.NewBatch()
batch = db.BlockStore().NewBatch()
start = time.Now()
logged = start.Add(-7 * time.Second) // Unindex during import is fast, don't double log
hash common.Hash
offset = db.AncientOffSet()
offset = db.BlockStore().AncientOffSet()
)
for i := uint64(0) + offset; i < frozen+offset; i++ {
// We read 100K hashes at a time, for a total of 3.2M
@@ -52,7 +52,7 @@ func InitDatabaseFromFreezer(db ethdb.Database) {
if i+count > frozen+offset {
count = frozen + offset - i
}
data, err := db.AncientRange(ChainFreezerHashTable, i, count, 32*count)
data, err := db.BlockStore().AncientRange(ChainFreezerHashTable, i, count, 32*count)
if err != nil {
log.Crit("Failed to init database from freezer", "err", err)
}
@@ -80,8 +80,8 @@ func InitDatabaseFromFreezer(db ethdb.Database) {
}
batch.Reset()
WriteHeadHeaderHash(db, hash)
WriteHeadFastBlockHash(db, hash)
WriteHeadHeaderHash(db.BlockStore(), hash)
WriteHeadFastBlockHash(db.BlockStore(), hash)
log.Info("Initialized database from freezer", "blocks", frozen, "elapsed", common.PrettyDuration(time.Since(start)))
}
@@ -100,7 +100,7 @@ func iterateTransactions(db ethdb.Database, from uint64, to uint64, reverse bool
number uint64
rlp rlp.RawValue
}
if offset := db.AncientOffSet(); offset > from {
if offset := db.BlockStore().AncientOffSet(); offset > from {
from = offset
}
if to <= from {
@@ -187,7 +187,7 @@ func iterateTransactions(db ethdb.Database, from uint64, to uint64, reverse bool
// signal received.
func indexTransactions(db ethdb.Database, from uint64, to uint64, interrupt chan struct{}, hook func(uint64) bool, report bool) {
// short circuit for invalid range
if offset := db.AncientOffSet(); offset > from {
if offset := db.BlockStore().AncientOffSet(); offset > from {
from = offset
}
if from >= to {
@@ -286,7 +286,7 @@ func indexTransactionsForTesting(db ethdb.Database, from uint64, to uint64, inte
// signal received.
func unindexTransactions(db ethdb.Database, from uint64, to uint64, interrupt chan struct{}, hook func(uint64) bool, report bool) {
// short circuit for invalid range
if offset := db.AncientOffSet(); offset > from {
if offset := db.BlockStore().AncientOffSet(); offset > from {
from = offset
}
if from >= to {

View File

@@ -26,14 +26,13 @@ import (
"strings"
"time"
"github.com/olekukonko/tablewriter"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/ethdb/leveldb"
"github.com/ethereum/go-ethereum/ethdb/memorydb"
"github.com/ethereum/go-ethereum/ethdb/pebble"
"github.com/ethereum/go-ethereum/log"
"github.com/olekukonko/tablewriter"
)
// freezerdb is a database wrapper that enables freezer data retrievals.
@@ -44,6 +43,7 @@ type freezerdb struct {
ethdb.AncientFreezer
diffStore ethdb.KeyValueStore
stateStore ethdb.Database
blockStore ethdb.Database
}
func (frdb *freezerdb) StateStoreReader() ethdb.Reader {
@@ -53,6 +53,20 @@ func (frdb *freezerdb) StateStoreReader() ethdb.Reader {
return frdb.stateStore
}
func (frdb *freezerdb) BlockStoreReader() ethdb.Reader {
if frdb.blockStore == nil {
return frdb
}
return frdb.blockStore
}
func (frdb *freezerdb) BlockStoreWriter() ethdb.Writer {
if frdb.blockStore == nil {
return frdb
}
return frdb.blockStore
}
// AncientDatadir returns the path of root ancient directory.
func (frdb *freezerdb) AncientDatadir() (string, error) {
return frdb.ancientRoot, nil
@@ -78,6 +92,11 @@ func (frdb *freezerdb) Close() error {
errs = append(errs, err)
}
}
if frdb.blockStore != nil {
if err := frdb.blockStore.Close(); err != nil {
errs = append(errs, err)
}
}
if len(errs) != 0 {
return fmt.Errorf("%v", errs)
}
@@ -99,6 +118,13 @@ func (frdb *freezerdb) StateStore() ethdb.Database {
return frdb.stateStore
}
func (frdb *freezerdb) GetStateStore() ethdb.Database {
if frdb.stateStore != nil {
return frdb.stateStore
}
return frdb
}
func (frdb *freezerdb) SetStateStore(state ethdb.Database) {
if frdb.stateStore != nil {
frdb.stateStore.Close()
@@ -106,6 +132,25 @@ func (frdb *freezerdb) SetStateStore(state ethdb.Database) {
frdb.stateStore = state
}
func (frdb *freezerdb) BlockStore() ethdb.Database {
if frdb.blockStore != nil {
return frdb.blockStore
} else {
return frdb
}
}
func (frdb *freezerdb) SetBlockStore(block ethdb.Database) {
if frdb.blockStore != nil {
frdb.blockStore.Close()
}
frdb.blockStore = block
}
func (frdb *freezerdb) HasSeparateBlockStore() bool {
return frdb.blockStore != nil
}
// Freeze is a helper method used for external testing to trigger and block until
// a freeze cycle completes, without having to sleep for a minute to trigger the
// automatic background run.
@@ -118,7 +163,6 @@ func (frdb *freezerdb) Freeze(threshold uint64) error {
frdb.AncientStore.(*chainFreezer).threshold.Store(old)
}(frdb.AncientStore.(*chainFreezer).threshold.Load())
frdb.AncientStore.(*chainFreezer).threshold.Store(threshold)
// Trigger a freeze cycle and block until it's done
trigger := make(chan struct{}, 1)
frdb.AncientStore.(*chainFreezer).trigger <- trigger
@@ -135,6 +179,7 @@ type nofreezedb struct {
ethdb.KeyValueStore
diffStore ethdb.KeyValueStore
stateStore ethdb.Database
blockStore ethdb.Database
}
// HasAncient returns an error as we don't have a backing chain freezer.
@@ -157,7 +202,7 @@ func (db *nofreezedb) Ancients() (uint64, error) {
return 0, errNotSupported
}
// Ancients returns an error as we don't have a backing chain freezer.
// ItemAmountInAncient returns an error as we don't have a backing chain freezer.
func (db *nofreezedb) ItemAmountInAncient() (uint64, error) {
return 0, errNotSupported
}
@@ -218,6 +263,13 @@ func (db *nofreezedb) SetStateStore(state ethdb.Database) {
db.stateStore = state
}
func (db *nofreezedb) GetStateStore() ethdb.Database {
if db.stateStore != nil {
return db.stateStore
}
return db
}
func (db *nofreezedb) StateStoreReader() ethdb.Reader {
if db.stateStore != nil {
return db.stateStore
@@ -225,6 +277,35 @@ func (db *nofreezedb) StateStoreReader() ethdb.Reader {
return db
}
func (db *nofreezedb) BlockStore() ethdb.Database {
if db.blockStore != nil {
return db.blockStore
}
return db
}
func (db *nofreezedb) SetBlockStore(block ethdb.Database) {
db.blockStore = block
}
func (db *nofreezedb) HasSeparateBlockStore() bool {
return db.blockStore != nil
}
func (db *nofreezedb) BlockStoreReader() ethdb.Reader {
if db.blockStore != nil {
return db.blockStore
}
return db
}
func (db *nofreezedb) BlockStoreWriter() ethdb.Writer {
if db.blockStore != nil {
return db.blockStore
}
return db
}
func (db *nofreezedb) ReadAncients(fn func(reader ethdb.AncientReaderOp) error) (err error) {
// Unlike other ancient-related methods, this method does not return
// errNotSupported when invoked.
@@ -266,6 +347,111 @@ func NewDatabase(db ethdb.KeyValueStore) ethdb.Database {
return &nofreezedb{KeyValueStore: db}
}
type emptyfreezedb struct {
ethdb.KeyValueStore
}
// HasAncient returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) HasAncient(kind string, number uint64) (bool, error) {
return false, nil
}
// Ancient returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) Ancient(kind string, number uint64) ([]byte, error) {
return nil, nil
}
// AncientRange returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) AncientRange(kind string, start, max, maxByteSize uint64) ([][]byte, error) {
return nil, nil
}
// Ancients returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) Ancients() (uint64, error) {
return 0, nil
}
// ItemAmountInAncient returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) ItemAmountInAncient() (uint64, error) {
return 0, nil
}
// Tail returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) Tail() (uint64, error) {
return 0, nil
}
// AncientSize returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) AncientSize(kind string) (uint64, error) {
return 0, nil
}
// ModifyAncients returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) ModifyAncients(func(ethdb.AncientWriteOp) error) (int64, error) {
return 0, nil
}
// TruncateHead returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) TruncateHead(items uint64) (uint64, error) {
return 0, nil
}
// TruncateTail returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) TruncateTail(items uint64) (uint64, error) {
return 0, nil
}
// TruncateTableTail returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) TruncateTableTail(kind string, tail uint64) (uint64, error) {
return 0, nil
}
// ResetTable returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) ResetTable(kind string, startAt uint64, onlyEmpty bool) error {
return nil
}
// Sync returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) Sync() error {
return nil
}
func (db *emptyfreezedb) DiffStore() ethdb.KeyValueStore { return db }
func (db *emptyfreezedb) SetDiffStore(diff ethdb.KeyValueStore) {}
func (db *emptyfreezedb) StateStore() ethdb.Database { return db }
func (db *emptyfreezedb) GetStateStore() ethdb.Database { return db }
func (db *emptyfreezedb) SetStateStore(state ethdb.Database) {}
func (db *emptyfreezedb) StateStoreReader() ethdb.Reader { return db }
func (db *emptyfreezedb) BlockStore() ethdb.Database { return db }
func (db *emptyfreezedb) SetBlockStore(block ethdb.Database) {}
func (db *emptyfreezedb) HasSeparateBlockStore() bool { return false }
func (db *emptyfreezedb) BlockStoreReader() ethdb.Reader { return db }
func (db *emptyfreezedb) BlockStoreWriter() ethdb.Writer { return db }
func (db *emptyfreezedb) ReadAncients(fn func(reader ethdb.AncientReaderOp) error) (err error) {
return nil
}
func (db *emptyfreezedb) AncientOffSet() uint64 { return 0 }
// MigrateTable returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) MigrateTable(kind string, convert convertLegacyFn) error {
return nil
}
// AncientDatadir returns nil for pruned db that we don't have a backing chain freezer.
func (db *emptyfreezedb) AncientDatadir() (string, error) {
return "", nil
}
func (db *emptyfreezedb) SetupFreezerEnv(env *ethdb.FreezerEnv) error {
return nil
}
// NewEmptyFreezeDB is used for CLI such as `geth db inspect` in pruned db that we don't
// have a backing chain freezer.
// WARNING: it must be only used in the above case.
func NewEmptyFreezeDB(db ethdb.KeyValueStore) ethdb.Database {
return &emptyfreezedb{KeyValueStore: db}
}
// NewFreezerDb only create a freezer without statedb.
func NewFreezerDb(db ethdb.KeyValueStore, frz, namespace string, readonly bool, newOffSet uint64) (*Freezer, error) {
// Create the idle freezer instance, this operation should be atomic to avoid mismatch between offset and acientDB.
@@ -306,7 +492,7 @@ func resolveChainFreezerDir(ancient string) string {
// value data store with a freezer moving immutable chain segments into cold
// storage. The passed ancient indicates the path of root ancient directory
// where the chain freezer can be opened.
func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace string, readonly, disableFreeze, isLastOffset, pruneAncientData bool) (ethdb.Database, error) {
func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace string, readonly, disableFreeze, isLastOffset, pruneAncientData, multiDatabase bool) (ethdb.Database, error) {
var offset uint64
// The offset of ancientDB should be handled differently in different scenarios.
if isLastOffset {
@@ -315,6 +501,12 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
offset = ReadOffSetOfCurrentAncientFreezer(db)
}
// This case is used for someone who wants to execute geth db inspect CLI in a pruned db
if !disableFreeze && readonly && ReadAncientType(db) == PruneFreezerType {
log.Warn("Disk db is pruned, using an empty freezer db for CLI")
return NewEmptyFreezeDB(db), nil
}
if pruneAncientData && !disableFreeze && !readonly {
frdb, err := newPrunedFreezer(resolveChainFreezerDir(ancient), db, offset)
if err != nil {
@@ -342,9 +534,18 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
}
// Create the idle freezer instance
frdb, err := newChainFreezer(resolveChainFreezerDir(ancient), namespace, readonly, offset)
frdb, err := newChainFreezer(resolveChainFreezerDir(ancient), namespace, readonly, offset, multiDatabase)
// We are creating the freezerdb here because the validation logic for db and freezer below requires certain interfaces
// that need a database type. Therefore, we are pre-creating it for subsequent use.
freezerDb := &freezerdb{
ancientRoot: ancient,
KeyValueStore: db,
AncientStore: frdb,
AncientFreezer: frdb,
}
if err != nil {
printChainMetadata(db)
printChainMetadata(freezerDb)
return nil, err
}
@@ -380,10 +581,10 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
// the freezer and the key-value store.
frgenesis, err := frdb.Ancient(ChainFreezerHashTable, 0)
if err != nil {
printChainMetadata(db)
printChainMetadata(freezerDb)
return nil, fmt.Errorf("failed to retrieve genesis from ancient %v", err)
} else if !bytes.Equal(kvgenesis, frgenesis) {
printChainMetadata(db)
printChainMetadata(freezerDb)
return nil, fmt.Errorf("genesis mismatch: %#x (leveldb) != %#x (ancients)", kvgenesis, frgenesis)
}
// Key-value store and freezer belong to the same network. Ensure that they
@@ -391,7 +592,7 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
if kvhash, _ := db.Get(headerHashKey(frozen)); len(kvhash) == 0 {
// Subsequent header after the freezer limit is missing from the database.
// Reject startup if the database has a more recent head.
if head := *ReadHeaderNumber(db, ReadHeadHeaderHash(db)); head > frozen-1 {
if head := *ReadHeaderNumber(freezerDb, ReadHeadHeaderHash(freezerDb)); head > frozen-1 {
// Find the smallest block stored in the key-value store
// in range of [frozen, head]
var number uint64
@@ -401,7 +602,7 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
}
}
// We are about to exit on error. Print database metadata before exiting
printChainMetadata(db)
printChainMetadata(freezerDb)
return nil, fmt.Errorf("gap in the chain between ancients [0 - #%d] and leveldb [#%d - #%d] ",
frozen-1, number, head)
}
@@ -416,11 +617,11 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
// store, otherwise we'll end up missing data. We check block #1 to decide
// if we froze anything previously or not, but do take care of databases with
// only the genesis block.
if ReadHeadHeaderHash(db) != common.BytesToHash(kvgenesis) {
if ReadHeadHeaderHash(freezerDb) != common.BytesToHash(kvgenesis) {
// Key-value store contains more data than the genesis block, make sure we
// didn't freeze anything yet.
if kvblob, _ := db.Get(headerHashKey(1)); len(kvblob) == 0 {
printChainMetadata(db)
printChainMetadata(freezerDb)
return nil, errors.New("ancient chain segments already extracted, please set --datadir.ancient to the correct path")
}
// Block #1 is still in the database, we're allowed to init a new freezer
@@ -429,6 +630,7 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
// freezer.
}
}
// no prune ancient start success
if !readonly {
WriteAncientType(db, EntireFreezerType)
@@ -441,12 +643,7 @@ func NewDatabaseWithFreezer(db ethdb.KeyValueStore, ancient string, namespace st
frdb.wg.Done()
}()
}
return &freezerdb{
ancientRoot: ancient,
KeyValueStore: db,
AncientStore: frdb,
AncientFreezer: frdb,
}, nil
return freezerDb, nil
}
// NewMemoryDatabase creates an ephemeral in-memory key-value database without a
@@ -518,9 +715,12 @@ type OpenOptions struct {
DisableFreeze bool
IsLastOffset bool
PruneAncientData bool
// Ephemeral means that filesystem sync operations should be avoided: data integrity in the face of
// a crash is not important. This option should typically be used in tests.
Ephemeral bool
MultiDataBase bool
}
// openKeyValueDatabase opens a disk-based key-value database, e.g. leveldb or pebble.
@@ -565,13 +765,13 @@ func Open(o OpenOptions) (ethdb.Database, error) {
}
if ReadAncientType(kvdb) == PruneFreezerType {
if !o.PruneAncientData {
log.Warn("Disk db is pruned")
log.Warn("NOTICE: You're opening a pruned disk db!")
}
}
if len(o.AncientsDirectory) == 0 {
return kvdb, nil
}
frdb, err := NewDatabaseWithFreezer(kvdb, o.AncientsDirectory, o.Namespace, o.ReadOnly, o.DisableFreeze, o.IsLastOffset, o.PruneAncientData)
frdb, err := NewDatabaseWithFreezer(kvdb, o.AncientsDirectory, o.Namespace, o.ReadOnly, o.DisableFreeze, o.IsLastOffset, o.PruneAncientData, o.MultiDataBase)
if err != nil {
kvdb.Close()
return nil, err
@@ -613,7 +813,7 @@ func AncientInspect(db ethdb.Database) error {
offset := counter(ReadOffSetOfCurrentAncientFreezer(db))
// Get number of ancient rows inside the freezer.
ancients := counter(0)
if count, err := db.ItemAmountInAncient(); err != nil {
if count, err := db.BlockStore().ItemAmountInAncient(); err != nil {
log.Error("failed to get the items amount in ancientDB", "err", err)
return err
} else {
@@ -661,6 +861,48 @@ func PruneHashTrieNodeInDataBase(db ethdb.Database) error {
return nil
}
type DataType int
const (
StateDataType DataType = iota
BlockDataType
ChainDataType
Unknown
)
func DataTypeByKey(key []byte) DataType {
switch {
// state
case IsLegacyTrieNode(key, key),
bytes.HasPrefix(key, stateIDPrefix) && len(key) == len(stateIDPrefix)+common.HashLength,
IsAccountTrieNode(key),
IsStorageTrieNode(key):
return StateDataType
// block
case bytes.HasPrefix(key, headerPrefix) && len(key) == (len(headerPrefix)+8+common.HashLength),
bytes.HasPrefix(key, blockBodyPrefix) && len(key) == (len(blockBodyPrefix)+8+common.HashLength),
bytes.HasPrefix(key, blockReceiptsPrefix) && len(key) == (len(blockReceiptsPrefix)+8+common.HashLength),
bytes.HasPrefix(key, headerPrefix) && bytes.HasSuffix(key, headerTDSuffix),
bytes.HasPrefix(key, headerPrefix) && bytes.HasSuffix(key, headerHashSuffix),
bytes.HasPrefix(key, headerNumberPrefix) && len(key) == (len(headerNumberPrefix)+common.HashLength):
return BlockDataType
default:
for _, meta := range [][]byte{
fastTrieProgressKey, persistentStateIDKey, trieJournalKey, snapSyncStatusFlagKey} {
if bytes.Equal(key, meta) {
return StateDataType
}
}
for _, meta := range [][]byte{headHeaderKey, headFinalizedBlockKey, headBlockKey, headFastBlockKey} {
if bytes.Equal(key, meta) {
return BlockDataType
}
}
return ChainDataType
}
}
// InspectDatabase traverses the entire database and checks the size
// of all different categories of data.
func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error {
@@ -668,10 +910,15 @@ func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error {
defer it.Release()
var trieIter ethdb.Iterator
var blockIter ethdb.Iterator
if db.StateStore() != nil {
trieIter = db.StateStore().NewIterator(keyPrefix, nil)
defer trieIter.Release()
}
if db.HasSeparateBlockStore() {
blockIter = db.BlockStore().NewIterator(keyPrefix, nil)
defer blockIter.Release()
}
var (
count int64
start = time.Now()
@@ -801,6 +1048,7 @@ func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error {
value = trieIter.Value()
size = common.StorageSize(len(key) + len(value))
)
total += size
switch {
case IsLegacyTrieNode(key, value):
@@ -814,9 +1062,10 @@ func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error {
default:
var accounted bool
for _, meta := range [][]byte{
fastTrieProgressKey, persistentStateIDKey, trieJournalKey} {
fastTrieProgressKey, persistentStateIDKey, trieJournalKey, snapSyncStatusFlagKey} {
if bytes.Equal(key, meta) {
metadata.Add(size)
accounted = true
break
}
}
@@ -830,6 +1079,54 @@ func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error {
logged = time.Now()
}
}
log.Info("Inspecting separate state database", "count", count, "elapsed", common.PrettyDuration(time.Since(start)))
}
// inspect separate block db
if blockIter != nil {
count = 0
logged = time.Now()
for blockIter.Next() {
var (
key = blockIter.Key()
value = blockIter.Value()
size = common.StorageSize(len(key) + len(value))
)
total += size
switch {
case bytes.HasPrefix(key, headerPrefix) && len(key) == (len(headerPrefix)+8+common.HashLength):
headers.Add(size)
case bytes.HasPrefix(key, blockBodyPrefix) && len(key) == (len(blockBodyPrefix)+8+common.HashLength):
bodies.Add(size)
case bytes.HasPrefix(key, blockReceiptsPrefix) && len(key) == (len(blockReceiptsPrefix)+8+common.HashLength):
receipts.Add(size)
case bytes.HasPrefix(key, headerPrefix) && bytes.HasSuffix(key, headerTDSuffix):
tds.Add(size)
case bytes.HasPrefix(key, headerPrefix) && bytes.HasSuffix(key, headerHashSuffix):
numHashPairings.Add(size)
case bytes.HasPrefix(key, headerNumberPrefix) && len(key) == (len(headerNumberPrefix)+common.HashLength):
hashNumPairings.Add(size)
default:
var accounted bool
for _, meta := range [][]byte{headHeaderKey, headFinalizedBlockKey, headBlockKey, headFastBlockKey} {
if bytes.Equal(key, meta) {
metadata.Add(size)
accounted = true
break
}
}
if !accounted {
unaccounted.Add(size)
}
}
count++
if count%1000 == 0 && time.Since(logged) > 8*time.Second {
log.Info("Inspecting separate block database", "count", count, "elapsed", common.PrettyDuration(time.Since(start)))
logged = time.Now()
}
}
log.Info("Inspecting separate block database", "count", count, "elapsed", common.PrettyDuration(time.Since(start)))
}
// Display the database statistic of key-value store.
stats := [][]string{
@@ -856,7 +1153,7 @@ func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error {
{"Light client", "Bloom trie nodes", bloomTrieNodes.Size(), bloomTrieNodes.Count()},
}
// Inspect all registered append-only file store then.
ancients, err := inspectFreezers(db)
ancients, err := inspectFreezers(db.BlockStore())
if err != nil {
return err
}
@@ -962,7 +1259,7 @@ func DeleteTrieState(db ethdb.Database) error {
}
// printChainMetadata prints out chain metadata to stderr.
func printChainMetadata(db ethdb.KeyValueStore) {
func printChainMetadata(db ethdb.Reader) {
fmt.Fprintf(os.Stderr, "Chain metadata\n")
for _, v := range ReadChainMetadata(db) {
fmt.Fprintf(os.Stderr, " %s\n", strings.Join(v, ": "))
@@ -973,7 +1270,7 @@ func printChainMetadata(db ethdb.KeyValueStore) {
// ReadChainMetadata returns a set of key/value pairs that contains information
// about the database chain status. This can be used for diagnostic purposes
// when investigating the state of the node.
func ReadChainMetadata(db ethdb.KeyValueStore) [][]string {
func ReadChainMetadata(db ethdb.Reader) [][]string {
pp := func(val *uint64) string {
if val == nil {
return "<nil>"

View File

@@ -239,7 +239,7 @@ func (f *Freezer) Ancient(kind string, number uint64) ([]byte, error) {
// - if maxBytes is not specified, 'count' items will be returned if they are present.
func (f *Freezer) AncientRange(kind string, start, count, maxBytes uint64) ([][]byte, error) {
if table := f.tables[kind]; table != nil {
return table.RetrieveItems(start, count, maxBytes)
return table.RetrieveItems(start-f.offset, count, maxBytes)
}
return nil, errUnknownTable
}
@@ -252,7 +252,7 @@ func (f *Freezer) Ancients() (uint64, error) {
func (f *Freezer) TableAncients(kind string) (uint64, error) {
f.writeLock.RLock()
defer f.writeLock.RUnlock()
return f.tables[kind].items.Load(), nil
return f.tables[kind].items.Load() + f.offset, nil
}
// ItemAmountInAncient returns the actual length of current ancientDB.
@@ -541,41 +541,6 @@ func gcKvStore(db ethdb.KeyValueStore, ancients []common.Hash, first uint64, fro
}
batch.Reset()
// Step into the future and delete and dangling side chains
if frozen > 0 {
tip := frozen
nfdb := &nofreezedb{KeyValueStore: db}
for len(dangling) > 0 {
drop := make(map[common.Hash]struct{})
for _, hash := range dangling {
log.Debug("Dangling parent from freezer", "number", tip-1, "hash", hash)
drop[hash] = struct{}{}
}
children := ReadAllHashes(db, tip)
for i := 0; i < len(children); i++ {
// Dig up the child and ensure it's dangling
child := ReadHeader(nfdb, children[i], tip)
if child == nil {
log.Error("Missing dangling header", "number", tip, "hash", children[i])
continue
}
if _, ok := drop[child.ParentHash]; !ok {
children = append(children[:i], children[i+1:]...)
i--
continue
}
// Delete all block data associated with the child
log.Debug("Deleting dangling block", "number", tip, "hash", children[i], "parent", child.ParentHash)
DeleteBlock(batch, children[i], tip)
}
dangling = children
tip++
}
if err := batch.Write(); err != nil {
log.Crit("Failed to delete dangling side blocks", "err", err)
}
}
// Log something friendly for the user
context := []interface{}{
"blocks", frozen - first, "elapsed", common.PrettyDuration(time.Since(start)), "number", frozen - 1,

View File

@@ -127,6 +127,11 @@ func newFreezerTable(path, name string, disableSnappy, readonly bool) (*freezerT
return newTable(path, name, metrics.NilMeter{}, metrics.NilMeter{}, metrics.NilGauge{}, freezerTableSize, disableSnappy, readonly)
}
// newAdditionTable opens the given path as a addition table.
func newAdditionTable(path, name string, disableSnappy, readonly bool) (*freezerTable, error) {
return openAdditionTable(path, name, metrics.NilMeter{}, metrics.NilMeter{}, metrics.NilGauge{}, freezerTableSize, disableSnappy, readonly)
}
// newTable opens a freezer table, creating the data and index files if they are
// non-existent. Both files are truncated to the shortest common length to ensure
// they don't go out of sync.

View File

@@ -8,6 +8,8 @@ import (
"sync/atomic"
"time"
"golang.org/x/exp/slices"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/log"
@@ -66,28 +68,49 @@ func newPrunedFreezer(datadir string, db ethdb.KeyValueStore, offset uint64) (*p
// repair init frozen , compatible disk-ancientdb and pruner-block-tool.
func (f *prunedfreezer) repair(datadir string) error {
offset := atomic.LoadUint64(&f.frozen)
// compatible freezer
min := uint64(math.MaxUint64)
minItems := uint64(math.MaxUint64)
for name, disableSnappy := range chainFreezerNoSnappy {
table, err := newFreezerTable(datadir, name, disableSnappy, false)
var (
table *freezerTable
err error
)
if slices.Contains(additionTables, name) {
table, err = newAdditionTable(datadir, name, disableSnappy, false)
} else {
table, err = newFreezerTable(datadir, name, disableSnappy, false)
}
if err != nil {
return err
}
// addition tables only align head
if slices.Contains(additionTables, name) {
if EmptyTable(table) {
continue
}
}
items := table.items.Load()
if min > items {
min = items
if minItems > items {
minItems = items
}
table.Close()
}
log.Info("Read ancientdb item counts", "items", min)
offset += min
if frozen := ReadFrozenOfAncientFreezer(f.db); frozen > offset {
offset = frozen
// If minItems is non-zero, it indicates that the chain freezer was previously enabled, and we should use minItems as the current frozen value.
// If minItems is zero, it indicates that the pruneAncient was previously enabled, and we should continue using frozen
// (retrieved from CurrentAncientFreezer) as the current frozen value.
offset := minItems
if offset == 0 {
// no item in ancientDB, init `offset` to the `f.frozen`
offset = atomic.LoadUint64(&f.frozen)
}
log.Info("Read ancientdb item counts", "items", minItems, "offset", offset)
atomic.StoreUint64(&f.frozen, offset)
// FrozenOfAncientFreezer is the progress of the last prune-freezer freeze.
frozenInDB := ReadFrozenOfAncientFreezer(f.db)
maxOffset := max(offset, frozenInDB)
atomic.StoreUint64(&f.frozen, maxOffset)
if err := f.Sync(); err != nil {
return nil
}
@@ -299,9 +322,8 @@ func (f *prunedfreezer) freeze() {
log.Error("Append ancient err", "number", f.frozen, "hash", hash, "err", err)
break
}
if hash != (common.Hash{}) {
ancients = append(ancients, hash)
}
// may include common.Hash{}, will be delete in gcKvStore
ancients = append(ancients, hash)
}
// Batch of blocks have been frozen, flush them before wiping from leveldb
if err := f.Sync(); err != nil {

View File

@@ -40,6 +40,9 @@ var (
// headFastBlockKey tracks the latest known incomplete block's hash during fast sync.
headFastBlockKey = []byte("LastFast")
// headFinalizedBlockKey tracks the latest known finalized block hash.
headFinalizedBlockKey = []byte("LastFinalized")
// persistentStateIDKey tracks the id of latest stored state(for path-based only).
persistentStateIDKey = []byte("LastStateID")

View File

@@ -27,6 +27,26 @@ type table struct {
prefix string
}
func (t *table) BlockStoreReader() ethdb.Reader {
return t
}
func (t *table) BlockStoreWriter() ethdb.Writer {
return t
}
func (t *table) BlockStore() ethdb.Database {
return t
}
func (t *table) SetBlockStore(block ethdb.Database) {
panic("not implement")
}
func (t *table) HasSeparateBlockStore() bool {
panic("not implement")
}
// NewTable returns a database object that prefixes all keys with a given string.
func NewTable(db ethdb.Database, prefix string) ethdb.Database {
return &table{
@@ -231,6 +251,10 @@ func (t *table) SetStateStore(state ethdb.Database) {
panic("not implement")
}
func (t *table) GetStateStore() ethdb.Database {
return nil
}
func (t *table) StateStoreReader() ethdb.Reader {
return nil
}

View File

@@ -0,0 +1,526 @@
package state
import (
"errors"
"fmt"
"sync/atomic"
versa "github.com/bnb-chain/versioned-state-database"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/lru"
"github.com/ethereum/go-ethereum/common/math"
"github.com/ethereum/go-ethereum/core/rawdb"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/log"
"github.com/ethereum/go-ethereum/rlp"
"github.com/ethereum/go-ethereum/trie"
"github.com/ethereum/go-ethereum/trie/trienode"
"github.com/ethereum/go-ethereum/triedb"
)
const InvalidSateObjectVersion int64 = math.MinInt64
type cachingVersaDB struct {
version int64
triedb *triedb.Database
versionDB versa.Database
codeDB ethdb.KeyValueStore
codeSizeCache *lru.Cache[common.Hash, int]
codeCache *lru.SizeConstrainedCache[common.Hash, []byte]
accTree *VersaTree
state versa.StateHandler
root common.Hash
mode versa.StateMode
hasState atomic.Bool
//debug *DebugVersionState
}
// NewVersaDatabase should be call by NewDatabaseWithNodeDB
// TODO:: NewDatabaseWithNodeDB should add mode param.
func NewVersaDatabase(db ethdb.Database, triedb *triedb.Database, mode versa.StateMode) Database {
return &cachingVersaDB{
triedb: triedb,
versionDB: triedb.VersaDB(),
codeDB: db,
codeSizeCache: lru.NewCache[common.Hash, int](codeSizeCacheSize),
codeCache: lru.NewSizeConstrainedCache[common.Hash, []byte](codeCacheSize),
mode: mode,
state: versa.ErrStateHandler,
}
}
func (cv *cachingVersaDB) Copy() Database {
cp := &cachingVersaDB{}
cp.codeCache = cv.codeCache
cp.codeSizeCache = cv.codeSizeCache
cp.triedb = cv.triedb
cp.versionDB = cv.versionDB
cp.codeDB = cv.codeDB
cp.version = cv.version
cp.mode = versa.S_RW // it is important
// TODO:: maybe add lock for cv.root
if cv.hasState.Load() {
_, err := cp.OpenTrie(cv.root)
if err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to open trie in copy caching versa db, error: %s", err.Error()))
//}
return cp
}
}
return cp
}
// CopyTrie is used with Copy()
func (cv *cachingVersaDB) CopyTrie(tr Trie) Trie {
vtr, ok := tr.(*VersaTree)
if !ok {
panic("caching versa db copy non versa tree")
}
if vtr.accountTree {
if cv.accTree != nil {
if cv.accTree.root.Cmp(vtr.root) != 0 {
panic("copy acc trie mismatch")
}
return cv.accTree
}
tree, err := cv.OpenTrie(vtr.root)
if err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to open trie in copy versa trie, error: %s", err.Error()))
//}
return nil
}
return tree
} else {
tree, err := cv.OpenStorageTrie(vtr.stateRoot, vtr.address, vtr.root, nil)
if err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to open storage trie in copy versa trie, error: %s", err.Error()))
//}
return nil
}
return tree
}
return nil
}
func (cv *cachingVersaDB) HasState(version int64, root common.Hash) bool {
return cv.versionDB.HasState(version, root)
}
func (cv *cachingVersaDB) OpenTrie(root common.Hash) (Trie, error) {
if cv.hasState.Load() {
//TODO:: will change to log.Error after stabilization
panic("account tree has open")
}
// TODO:: if root tree, versa db should ignore check version, temp use -1
state, err := cv.versionDB.OpenState(cv.version, root, cv.mode)
if err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to open state, root:%s, error: %s", root.String(), err.Error()))
//}
return nil, err
}
//if cv.debug != nil {
// cv.debug.OnOpenState(state)
//}
handler, err := cv.versionDB.OpenTree(state, cv.version, common.Hash{}, root)
if err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to open account trie, root:%s, error: %s", root.String(), err.Error()))
//}
return nil, err
}
tree := &VersaTree{
db: cv.versionDB,
handler: handler,
accountTree: true,
root: root,
mode: cv.mode,
//debug: cv.debug,
}
cv.state = state
cv.hasState.Store(true) // if set, can't change
cv.accTree = tree
cv.root = root
//if cv.debug != nil {
// cv.debug.OnOpenTree(handler, common.Hash{}, common.Address{})
//}
return tree, nil
}
func (cv *cachingVersaDB) OpenStorageTrie(stateRoot common.Hash, address common.Address, root common.Hash, _ Trie) (Trie, error) {
version, _, err := cv.accTree.getAccountWithVersion(address)
if err != nil {
return nil, err
}
return cv.openStorageTreeWithVersion(version, stateRoot, address, root)
}
func (cv *cachingVersaDB) openStorageTreeWithVersion(version int64, stateRoot common.Hash, address common.Address, root common.Hash) (Trie, error) {
if !cv.hasState.Load() {
//TODO:: will change to log.Error after stabilization
panic("open account tree, before open storage tree")
}
if cv.root.Cmp(stateRoot) != 0 {
panic(fmt.Sprintf("account root mismatch, on open storage tree, actual: %s, expect: %s", root.String(), cv.root.String()))
}
owner := crypto.Keccak256Hash(address.Bytes())
handler, err := cv.versionDB.OpenTree(cv.state, version, owner, root)
if err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to open storage trie, version: %d,stateRoot:%s, address:%s, root: %s, error: %s",
// version, stateRoot.String(), address.String(), root.String(), err.Error()))
//}
return nil, err
}
//if cv.debug != nil {
// cv.debug.OnOpenTree(handler, owner, address)
//}
tree := &VersaTree{
db: cv.versionDB,
handler: handler,
version: version,
root: stateRoot,
stateRoot: root,
address: address,
mode: cv.mode,
//debug: cv.debug,
}
return tree, nil
}
// Flush unique to versa
func (cv *cachingVersaDB) Flush() error {
err := cv.versionDB.Flush(cv.state)
//if err != nil && cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to flush state, version: %d, root:%s, mode:%d, error: %s",
// cv.accTree.version, cv.accTree.root.String(), cv.accTree.mode, err.Error()))
//}
return err
}
func (cv *cachingVersaDB) SetVersion(version int64) {
cv.version = version
//cv.debug = NewDebugVersionState(cv.codeDB, cv.versionDB)
//cv.debug.Version = version
}
func (cv *cachingVersaDB) GetVersion() int64 {
return cv.version
}
// Release unique to versa
func (cv *cachingVersaDB) Release() error {
//log.Info("close state", "state info", cv.versionDB.ParseStateHandler(cv.state))
if cv.state != versa.ErrStateHandler {
//if cv.debug != nil {
// cv.debug.OnCloseState(cv.state)
//}
if err := cv.versionDB.CloseState(cv.state); err != nil {
//if cv.debug != nil {
// cv.debug.OnError(fmt.Errorf("failed to close state in release, version: %d, root:%s, mode:%d, error: %s",
// cv.accTree.version, cv.accTree.root.String(), cv.accTree.mode, err.Error()))
//}
return err
}
cv.hasState.Store(false)
cv.accTree = nil
cv.state = versa.ErrStateHandler
cv.root = common.Hash{}
//cv.debug = nil
}
return nil
}
func (cv *cachingVersaDB) Reset() {
if cv.state != versa.ErrStateHandler {
_ = cv.versionDB.CloseState(cv.state)
panic("close state in reset")
}
cv.hasState.Store(false)
cv.accTree = nil
cv.state = versa.ErrStateHandler
cv.root = common.Hash{}
}
func (cv *cachingVersaDB) HasTreeExpired(tr Trie) bool {
vtr, ok := tr.(*VersaTree)
if !ok {
panic("trie type mismatch")
}
return cv.versionDB.HasTreeExpired(vtr.handler)
}
func (cv *cachingVersaDB) Scheme() string {
return cv.triedb.Scheme()
}
func (cv *cachingVersaDB) ContractCode(addr common.Address, codeHash common.Hash) ([]byte, error) {
//if cv.debug != nil {
// cv.debug.OnGetCode(addr, codeHash)
//}
code, _ := cv.codeCache.Get(codeHash)
if len(code) > 0 {
return code, nil
}
code = rawdb.ReadCode(cv.codeDB, codeHash)
if len(code) > 0 {
cv.codeCache.Add(codeHash, code)
cv.codeSizeCache.Add(codeHash, len(code))
return code, nil
}
return nil, errors.New("not found")
}
func (cv *cachingVersaDB) ContractCodeSize(addr common.Address, codeHash common.Hash) (int, error) {
if cached, ok := cv.codeSizeCache.Get(codeHash); ok {
return cached, nil
}
code, err := cv.ContractCode(addr, codeHash)
return len(code), err
}
func (cv *cachingVersaDB) ContractCodeWithPrefix(address common.Address, codeHash common.Hash) ([]byte, error) {
code, _ := cv.codeCache.Get(codeHash)
if len(code) > 0 {
return code, nil
}
code = rawdb.ReadCodeWithPrefix(cv.codeDB, codeHash)
if len(code) > 0 {
cv.codeCache.Add(codeHash, code)
cv.codeSizeCache.Add(codeHash, len(code))
return code, nil
}
return nil, errors.New("not found")
}
func (cv *cachingVersaDB) DiskDB() ethdb.KeyValueStore {
return cv.codeDB
}
func (cv *cachingVersaDB) TrieDB() *triedb.Database {
return cv.triedb
}
func (cv *cachingVersaDB) NoTries() bool {
// TODO:: not support fastnode
return false
}
type VersaTree struct {
db versa.Database
handler versa.TreeHandler
version int64
accountTree bool
//debug *DebugVersionState
// TODO:: debugging, used for logging
stateRoot common.Hash
root common.Hash
address common.Address
mode versa.StateMode
}
func (vt *VersaTree) GetKey(key []byte) []byte {
_, val, err := vt.db.Get(vt.handler, key)
if err != nil {
log.Warn("failed to get key from version db")
}
return val
}
func (vt *VersaTree) GetAccount(address common.Address) (*types.StateAccount, error) {
_, res, err := vt.getAccountWithVersion(address)
return res, err
}
func (vt *VersaTree) getAccountWithVersion(address common.Address) (int64, *types.StateAccount, error) {
vt.CheckAccountTree()
ver, res, err := vt.db.Get(vt.handler, address.Bytes())
if res == nil || err != nil {
//if err != nil && vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to get account, root: %s, address: %s, error: %s",
// vt.root.String(), address.String(), err.Error()))
//}
return ver, nil, err
}
ret := new(types.StateAccount)
err = rlp.DecodeBytes(res, ret)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to rlp decode account, root: %s, address: %s, error: %s",
// vt.root.String(), address.String(), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnGetAccount(address, ret)
//}
return ver, ret, err
}
func (vt *VersaTree) GetStorage(address common.Address, key []byte) ([]byte, error) {
if vt.address.Cmp(address) != 0 {
panic(fmt.Sprintf("address mismatch in get storage, expect: %s, actul: %s", vt.address.String(), address.String()))
}
vt.CheckStorageTree()
_, enc, err := vt.db.Get(vt.handler, key)
if err != nil || len(enc) == 0 {
//if err != nil && vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to get storage, root: %s, stateRoot: %s, address:%s, key: %s, error: %s",
// vt.root.String(), vt.stateRoot.String(), address.String(), common.Bytes2Hex(key), err.Error()))
//}
return nil, err
}
_, content, _, err := rlp.Split(enc)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to rlp decode storage, root: %s, stateRoot: %s, address: %s, key: %s,error: %s",
// vt.root.String(), vt.stateRoot.String(), address.String(), common.Bytes2Hex(key), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnGetStorage(vt.handler, address, key, content)
//}
return content, err
}
func (vt *VersaTree) UpdateAccount(address common.Address, account *types.StateAccount) error {
vt.CheckAccountTree()
data, err := rlp.EncodeToBytes(account)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to update account, root: %s, address: %s, account: %s, error: %s",
// vt.root.String(), address.String(), account.String(), err.Error()))
//}
return err
}
//if vt.debug != nil {
// vt.debug.OnUpdateAccount(address, account)
//}
return vt.db.Put(vt.handler, address.Bytes(), data)
}
func (vt *VersaTree) WriteBatch(values map[string][]byte) error {
return vt.db.WriteBatch(vt.handler, values)
}
func (vt *VersaTree) UpdateStorage(address common.Address, key, value []byte) error {
if vt.address.Cmp(address) != 0 {
panic(fmt.Sprintf("address mismatch in get storage, expect: %s, actul: %s", vt.address.String(), address.String()))
}
vt.CheckStorageTree()
v, _ := rlp.EncodeToBytes(value)
err := vt.db.Put(vt.handler, key, v)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to update storage, root: %s, stateRoot: %s, address: %s, key: %s, val: %s, error: %s",
// vt.root.String(), vt.stateRoot.String(), address.String(), common.Bytes2Hex(key), common.Bytes2Hex(value), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnUpdateStorage(vt.handler, address, key, value)
//}
return err
}
func (vt *VersaTree) DeleteAccount(address common.Address) error {
vt.CheckAccountTree()
err := vt.db.Delete(vt.handler, address.Bytes())
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to delete account, root: %s, address: %s, error: %s",
// vt.root.String(), address.String(), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnDeleteAccount(address)
//}
return err
}
func (vt *VersaTree) DeleteStorage(address common.Address, key []byte) error {
vt.CheckStorageTree()
err := vt.db.Delete(vt.handler, key)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to delete storage, root: %s, stateRoot: %s, address: %s, key: %s, error: %s",
// vt.root.String(), vt.stateRoot.String(), address.String(), common.Bytes2Hex(key), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnDeleteStorage(vt.handler, address, key)
//}
return err
}
func (vt *VersaTree) UpdateContractCode(address common.Address, codeHash common.Hash, code []byte) error {
return nil
}
func (vt *VersaTree) Hash() common.Hash {
hash, err := vt.db.CalcRootHash(vt.handler)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to calc root, root: %s, stateRoot%s, error:%s",
// vt.root.String(), vt.stateRoot.String(), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnCalcHash(vt.address, hash)
//}
return hash
}
func (vt *VersaTree) Commit(_ bool) (common.Hash, *trienode.NodeSet, error) {
hash, err := vt.db.Commit(vt.handler)
if err != nil {
//if vt.debug != nil {
// vt.debug.OnError(fmt.Errorf("failed to commit versa tree, root: %s, stateRoot: %s, error: %s",
// vt.root.String(), vt.stateRoot.String(), err.Error()))
//}
}
//if vt.debug != nil {
// vt.debug.OnCalcHash(vt.address, hash)
// vt.debug.OnCommitTree(vt.address, vt.handler)
//}
return hash, nil, err
}
func (vt *VersaTree) NodeIterator(startKey []byte) (trie.NodeIterator, error) {
panic("versa tree not support iterate")
return nil, nil
}
func (vt *VersaTree) Prove(key []byte, proofDb ethdb.KeyValueWriter) error {
panic("versa tree not support prove")
return nil
}
// TODO:: debug code, will be deleted after stabilization
func (vt *VersaTree) CheckAccountTree() {
if !vt.accountTree {
panic("sub tree can't operate account")
}
}
// TODO:: debug code, will be deleted after stabilization
func (vt *VersaTree) CheckStorageTree() {
if vt.accountTree {
panic("root tree can't operate storage")
}
}

View File

@@ -20,6 +20,7 @@ import (
"errors"
"fmt"
versa "github.com/bnb-chain/versioned-state-database"
"github.com/crate-crypto/go-ipa/banderwagon"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/lru"
@@ -70,8 +71,33 @@ type Database interface {
// TrieDB returns the underlying trie database for managing trie nodes.
TrieDB() *triedb.Database
// Scheme returns triedb scheme, used to distinguish version triedb.
Scheme() string
// Flush used for version caching versa db to commit block state data.
Flush() error
// Release used for caching versa db to release resource.
Release() error
// Reset used for caching versa db to clean up meta data.
Reset()
// Copy used for caching versa db to copy db, main to transfer triedb with rw mode.
Copy() Database
// HasState returns the state data whether in the triedb.
HasState(version int64, root common.Hash) bool
// HasTreeExpired used for caching versa db, whether the state where the opened tree resides has been closed
HasTreeExpired(tr Trie) bool
// NoTries returns whether the database has tries storage.
NoTries() bool
SetVersion(version int64)
GetVersion() int64
}
// Trie is a Ethereum Merkle Patricia trie.
@@ -117,6 +143,8 @@ type Trie interface {
// to be moved to the stateWriter interface when the latter is ready.
UpdateContractCode(address common.Address, codeHash common.Hash, code []byte) error
WriteBatch(values map[string][]byte) error
// Hash returns the root hash of the trie. It does not write to the database and
// can be used even if the trie doesn't have one.
Hash() common.Hash
@@ -148,28 +176,43 @@ type Trie interface {
// concurrent use, but does not retain any recent trie nodes in memory. To keep some
// historical state in memory, use the NewDatabaseWithConfig constructor.
func NewDatabase(db ethdb.Database) Database {
return NewDatabaseWithConfig(db, nil)
return NewDatabaseWithConfig(db, nil, false)
}
// NewDatabaseWithConfig creates a backing store for state. The returned database
// is safe for concurrent use and retains a lot of collapsed RLP trie nodes in a
// large memory cache.
func NewDatabaseWithConfig(db ethdb.Database, config *triedb.Config) Database {
func NewDatabaseWithConfig(db ethdb.Database, config *triedb.Config, needCommit bool) Database {
noTries := config != nil && config.NoTries
triedb := triedb.NewDatabase(db, config)
if triedb.Scheme() == rawdb.VersionScheme {
if needCommit {
return NewVersaDatabase(db, triedb, versa.S_COMMIT)
}
return NewVersaDatabase(db, triedb, versa.S_RW)
}
return &cachingDB{
disk: db,
codeSizeCache: lru.NewCache[common.Hash, int](codeSizeCacheSize),
codeCache: lru.NewSizeConstrainedCache[common.Hash, []byte](codeCacheSize),
triedb: triedb.NewDatabase(db, config),
triedb: triedb,
noTries: noTries,
}
}
// NewDatabaseWithNodeDB creates a state database with an already initialized node database.
func NewDatabaseWithNodeDB(db ethdb.Database, triedb *triedb.Database) Database {
func NewDatabaseWithNodeDB(db ethdb.Database, triedb *triedb.Database, needCommit bool) Database {
noTries := triedb != nil && triedb.Config() != nil && triedb.Config().NoTries
if triedb.Scheme() == rawdb.VersionScheme {
if needCommit {
return NewVersaDatabase(db, triedb, versa.S_COMMIT)
}
return NewVersaDatabase(db, triedb, versa.S_RW)
}
return &cachingDB{
disk: db,
codeSizeCache: lru.NewCache[common.Hash, int](codeSizeCacheSize),
@@ -185,6 +228,8 @@ type cachingDB struct {
codeCache *lru.SizeConstrainedCache[common.Hash, []byte]
triedb *triedb.Database
noTries bool
//debug *DebugHashState
}
// OpenTrie opens the main account trie at a specific root hash.
@@ -197,8 +242,22 @@ func (db *cachingDB) OpenTrie(root common.Hash) (Trie, error) {
}
tr, err := trie.NewStateTrie(trie.StateTrieID(root), db.triedb)
if err != nil {
//if db.debug != nil {
// db.debug.OnError(fmt.Errorf("failed to open tree, root: %s, error: %s", root.String(), err.Error()))
//}
return nil, err
}
//ht := &HashTrie{
// trie: tr,
// root: root,
// address: common.Address{},
// owner: common.Hash{},
// debug: db.debug,
//}
//if db.debug != nil {
// db.debug.OnOpenTree(root, common.Hash{}, common.Address{})
//}
//return ht, nil
return tr, nil
}
@@ -214,10 +273,27 @@ func (db *cachingDB) OpenStorageTrie(stateRoot common.Hash, address common.Addre
if db.triedb.IsVerkle() {
return self, nil
}
tr, err := trie.NewStateTrie(trie.StorageTrieID(stateRoot, crypto.Keccak256Hash(address.Bytes()), root), db.triedb)
owner := crypto.Keccak256Hash(address.Bytes())
tr, err := trie.NewStateTrie(trie.StorageTrieID(stateRoot, owner, root), db.triedb)
if err != nil {
//if db.debug != nil {
// db.debug.OnError(fmt.Errorf("failed to storage open tree, stateRoot: %s, address: %s, root: %s, error: %s",
// stateRoot.String(), address.String(), root.String(), err.Error()))
//}
return nil, err
}
//ht := &HashTrie{
// trie: tr,
// root: stateRoot,
// statRoot: root,
// address: address,
// owner: owner,
// debug: db.debug,
//}
//if db.debug != nil {
// db.debug.OnOpenTree(root, owner, address)
//}
//return ht, nil
return tr, nil
}
@@ -235,6 +311,8 @@ func (db *cachingDB) CopyTrie(t Trie) Trie {
return t.Copy()
case *trie.EmptyTrie:
return t.Copy()
//case *HashTrie:
// return db.CopyTrie(t.trie)
default:
panic(fmt.Errorf("unknown trie type %T", t))
}
@@ -242,6 +320,9 @@ func (db *cachingDB) CopyTrie(t Trie) Trie {
// ContractCode retrieves a particular contract's code.
func (db *cachingDB) ContractCode(address common.Address, codeHash common.Hash) ([]byte, error) {
//if db.debug != nil {
// db.debug.OnGetCode(address, codeHash)
//}
code, _ := db.codeCache.Get(codeHash)
if len(code) > 0 {
return code, nil
@@ -290,3 +371,179 @@ func (db *cachingDB) DiskDB() ethdb.KeyValueStore {
func (db *cachingDB) TrieDB() *triedb.Database {
return db.triedb
}
func (db *cachingDB) Reset() {
return
}
func (db *cachingDB) Scheme() string {
return db.triedb.Scheme()
}
func (db *cachingDB) Flush() error {
return nil
}
func (db *cachingDB) Release() error {
//db.debug.flush()
//db.debug = nil
return nil
}
func (db *cachingDB) SetVersion(version int64) {
//db.debug = NewDebugHashState(db.disk)
//db.debug.Version = version
}
func (db *cachingDB) GetVersion() int64 {
//return db.debug.Version
return 0
}
func (db *cachingDB) Copy() Database {
return db
}
func (db *cachingDB) HasState(_ int64, root common.Hash) bool {
_, err := db.OpenTrie(root)
return err == nil
}
func (db *cachingDB) HasTreeExpired(_ Trie) bool {
return false
}
//type HashTrie struct {
// trie Trie
// root common.Hash
// statRoot common.Hash
// address common.Address
// owner common.Hash
//
// debug *DebugHashState
//}
//
//func (ht *HashTrie) GetKey(key []byte) []byte {
// return ht.trie.GetKey(key)
//}
//
//func (ht *HashTrie) GetAccount(address common.Address) (*types.StateAccount, error) {
// acc, err := ht.trie.GetAccount(address)
// if err != nil {
// if ht.debug != nil {
// ht.debug.OnError(fmt.Errorf("failed to get account, address: %s, error: %s", address.String(), err.Error()))
// }
// return nil, err
// }
// if ht.debug != nil {
// ht.debug.OnGetAccount(address, acc)
// }
// return acc, nil
//}
//
//func (ht *HashTrie) GetStorage(addr common.Address, key []byte) ([]byte, error) {
// val, err := ht.trie.GetStorage(addr, key)
// if err != nil {
// if ht.debug != nil {
// ht.debug.OnError(fmt.Errorf("failed to get storage, address: %s, error: %s", addr.String(), err.Error()))
// }
// return val, err
// }
// if ht.debug != nil {
// ht.debug.OnGetStorage(addr, key, val)
// }
// return val, err
//}
//
//func (ht *HashTrie) UpdateAccount(address common.Address, account *types.StateAccount) error {
// err := ht.trie.UpdateAccount(address, account)
// if err != nil {
// if ht.debug != nil {
// ht.debug.OnError(fmt.Errorf("failed to update account, address: %s, account: %s, error: %s",
// address.String(), account.String(), err.Error()))
// }
// return err
// }
// if ht.debug != nil {
// ht.debug.OnUpdateAccount(address, account)
// }
// return nil
//}
//
//func (ht *HashTrie) UpdateStorage(addr common.Address, key, value []byte) error {
// err := ht.trie.UpdateStorage(addr, key, value)
// if err != nil {
// if ht.debug != nil {
// ht.debug.OnError(fmt.Errorf("failed to update storage, address: %s, key: %s, val: %s, error: %s",
// addr.String(), common.Bytes2Hex(key), common.Bytes2Hex(value), err.Error()))
// }
// return err
// }
// if ht.debug != nil {
// ht.debug.OnUpdateStorage(addr, key, value)
// }
// return nil
//}
//
//func (ht *HashTrie) DeleteAccount(address common.Address) error {
// err := ht.trie.DeleteAccount(address)
// if err != nil {
// if ht.debug != nil {
// ht.debug.OnError(fmt.Errorf("failed to delete account, address: %s, error: %s", address.String(), err.Error()))
// }
// return err
// }
// if ht.debug != nil {
// ht.debug.OnDeleteAccount(address)
// }
// return nil
//}
//
//func (ht *HashTrie) DeleteStorage(addr common.Address, key []byte) error {
// err := ht.trie.DeleteStorage(addr, key)
// if err != nil {
// if ht.debug != nil {
// ht.debug.OnError(fmt.Errorf("failed to update storage, address: %s, key: %s, error: %s",
// addr.String(), common.Bytes2Hex(key), err.Error()))
// }
// return err
// }
// if ht.debug != nil {
// ht.debug.OnDeleteStorage(addr, key)
// }
// return nil
//}
//
//func (ht *HashTrie) UpdateContractCode(address common.Address, codeHash common.Hash, code []byte) error {
// return ht.trie.UpdateContractCode(address, codeHash, code)
//}
//
//func (ht *HashTrie) Hash() common.Hash {
// root := ht.trie.Hash()
// if ht.debug != nil {
// ht.debug.OnCalcHash(ht.address, root)
// }
// return root
//}
//
//func (ht *HashTrie) Commit(collectLeaf bool) (common.Hash, *trienode.NodeSet, error) {
// hash, set, err := ht.trie.Commit(collectLeaf)
// if err != nil {
// ht.debug.OnError(fmt.Errorf("failed to commit tree, address: %s, error: %s",
// ht.address.String(), err.Error()))
// return hash, set, err
// }
// if ht.debug != nil {
// ht.debug.OnCalcHash(ht.address, hash)
// ht.debug.OnCommitTree(ht.address, hash)
// }
// return hash, set, nil
//}
//
//func (ht *HashTrie) NodeIterator(startKey []byte) (trie.NodeIterator, error) {
// return ht.trie.NodeIterator(startKey)
//}
//
//func (ht *HashTrie) Prove(key []byte, proofDb ethdb.KeyValueWriter) error {
// return ht.trie.Prove(key, proofDb)
//}

View File

@@ -161,7 +161,7 @@ func (s *StateDB) DumpToCollector(c DumpCollector, conf *DumpConfig) (nextKey []
address = &addr
account.Address = address
}
obj := newObject(s, addr, &data)
obj := newObject(s, addr, &data, InvalidSateObjectVersion)
if !conf.SkipCode {
account.Code = obj.Code()
}

View File

@@ -34,4 +34,7 @@ var (
slotDeletionCount = metrics.NewRegisteredMeter("state/delete/storage/slot", nil)
slotDeletionSize = metrics.NewRegisteredMeter("state/delete/storage/size", nil)
slotDeletionSkip = metrics.NewRegisteredGauge("state/delete/storage/skip", nil)
accountIntermediateRootTimer = metrics.NewRegisteredTimer("state/account/intermediate/root/time", nil)
storageIntermediateRootTimer = metrics.NewRegisteredTimer("state/storage/intermediate/root/time", nil)
)

View File

@@ -232,6 +232,7 @@ func pruneAll(maindb ethdb.Database, g *core.Genesis) error {
}
log.Info("Database compaction finished", "elapsed", common.PrettyDuration(time.Since(cstart)))
}
// pruner should be not used to version db
statedb, _ := state.New(common.Hash{}, state.NewDatabase(maindb), nil)
for addr, account := range g.Alloc {
statedb.AddBalance(addr, uint256.MustFromBig(account.Balance))
@@ -382,7 +383,7 @@ func (p *BlockPruner) backUpOldDb(name string, cache, handles int, namespace str
log.Info("chainDB opened successfully")
// Get the number of items in old ancient db.
itemsOfAncient, err := chainDb.ItemAmountInAncient()
itemsOfAncient, err := chainDb.BlockStore().ItemAmountInAncient()
log.Info("the number of items in ancientDB is ", "itemsOfAncient", itemsOfAncient)
// If we can't access the freezer or it's empty, abort.
@@ -402,7 +403,7 @@ func (p *BlockPruner) backUpOldDb(name string, cache, handles int, namespace str
var oldOffSet uint64
if interrupt {
// The interrupt scecario within this function is specific for old and new ancientDB exsisted concurrently,
// The interrupt scecario within this function is specific for old and new ancientDB existed concurrently,
// should use last version of offset for oldAncientDB, because current offset is
// actually of the new ancientDB_Backup, but what we want is the offset of ancientDB being backup.
oldOffSet = rawdb.ReadOffSetOfLastAncientFreezer(chainDb)

View File

@@ -24,6 +24,7 @@ import (
"time"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/rawdb"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/metrics"
@@ -63,6 +64,7 @@ func (s Storage) Copy() Storage {
// - Finally, call commit to return the changes of storage trie and update account data.
type stateObject struct {
db *StateDB
version int64
address common.Address // address of ethereum account
addrHash common.Hash // hash of ethereum address of the account
origin *types.StateAccount // Account original data without any change applied, nil means it was not existent
@@ -99,7 +101,7 @@ func (s *stateObject) empty() bool {
}
// newObject creates a state object.
func newObject(db *StateDB, address common.Address, acct *types.StateAccount) *stateObject {
func newObject(db *StateDB, address common.Address, acct *types.StateAccount, version int64) *stateObject {
var (
origin = acct
created = acct == nil // true if the account was not existent
@@ -112,9 +114,9 @@ func newObject(db *StateDB, address common.Address, acct *types.StateAccount) *s
if db != nil && db.storagePool != nil {
storageMap = db.GetStorage(address)
}
return &stateObject{
db: db,
version: version,
address: address,
addrHash: crypto.Keccak256Hash(address[:]),
origin: origin,
@@ -158,8 +160,18 @@ func (s *stateObject) getTrie() (Trie, error) {
// s.trie = s.db.prefetcher.trie(s.addrHash, s.data.Root)
// }
// if s.trie == nil {
tr, err := s.db.db.OpenStorageTrie(s.db.originalRoot, s.address, s.data.Root, s.db.trie)
var (
tr Trie
err error
)
if s.version == InvalidSateObjectVersion {
tr, err = s.db.db.OpenStorageTrie(s.db.originalRoot, s.address, s.data.Root, s.db.trie)
} else {
tr, err = s.db.db.(*cachingVersaDB).openStorageTreeWithVersion(s.version, s.db.originalRoot, s.address, s.data.Root)
}
if err != nil {
panic(fmt.Sprintf("open storage storage failed, root version: %d, storage version: %d, addrss: %s, storage root: %s, error: %s", s.db.db.GetVersion(), s.version, s.address.String(), s.data.Root.String(), err.Error()))
return nil, err
}
s.trie = tr
@@ -223,12 +235,21 @@ func (s *stateObject) GetCommittedState(key common.Hash) common.Hash {
return common.Hash{}
}
// If no live objects are available, attempt to use snapshots
defer func(start time.Time) {
stateDBGetTimer.UpdateSince(start)
stateDBGetQPS.Mark(1)
stateDBGetStorageTimer.UpdateSince(start)
stateDBGetStorageQPS.Mark(1)
}(time.Now())
var (
enc []byte
err error
value common.Hash
)
if s.db.snap != nil {
panic("snap is not nil")
start := time.Now()
enc, err = s.db.snap.Storage(s.addrHash, crypto.Keccak256Hash(key.Bytes()))
if metrics.EnabledExpensive {
@@ -302,6 +323,30 @@ func (s *stateObject) finalise(prefetch bool) {
}
}
func (s *stateObject) IsContractAccount() bool {
return s.data.Root.Cmp(types.EmptyRootHash) != 0 ||
bytes.Compare(s.data.CodeHash, types.EmptyCodeHash.Bytes()) != 0
}
func (s *stateObject) IsAccountChanged() bool {
if s.origin == nil {
return true
}
if s.data.Nonce != s.origin.Nonce {
return true
}
if s.data.Balance.Cmp(s.origin.Balance) != 0 {
return true
}
if s.data.Root.Cmp(s.origin.Root) != 0 {
return true
}
if bytes.Compare(s.data.CodeHash, s.origin.CodeHash) != 0 {
return true
}
return false
}
// updateTrie is responsible for persisting cached storage changes into the
// object's storage trie. In case the storage trie is not yet loaded, this
// function will load the trie automatically. If any issues arise during the
@@ -312,10 +357,28 @@ func (s *stateObject) updateTrie() (Trie, error) {
// Make sure all dirty slots are finalized into the pending storage area
s.finalise(false)
// Short circuit if nothing changed, don't bother with hashing anything
if len(s.pendingStorage) == 0 {
return s.trie, nil
// fix 33740 blocks issue, add 1002 contract balance, but not update 1002
// storage tree, the case lead to 1002 account version mismatch with 1002
// storage tree version, occurs 53409 block open 1002 storage tree error.
if s.db.db.Scheme() == rawdb.VersionScheme {
if len(s.pendingStorage) == 0 {
// transferring balance to a contract or upgrading the code, but
// without updating the storage key, a commit is still required to
// increment the version number of the storage tree.
if !s.IsContractAccount() {
return s.trie, nil
}
//if !s.IsAccountChanged() {
// return s.trie, nil
//}
}
} else {
// Short circuit if nothing changed, don't bother with hashing anything
if len(s.pendingStorage) == 0 {
return s.trie, nil
}
}
// Track the amount of time wasted on updating the storage trie
if metrics.EnabledExpensive {
defer func(start time.Time) {
@@ -335,6 +398,10 @@ func (s *stateObject) updateTrie() (Trie, error) {
s.db.setError(err)
return nil, err
}
if len(s.pendingStorage) == 0 {
return s.trie, nil
}
// Insert all the pending storage updates into the trie
usedStorage := make([][]byte, 0, len(s.pendingStorage))
dirtyStorage := make(map[common.Hash][]byte)
@@ -350,11 +417,14 @@ func (s *stateObject) updateTrie() (Trie, error) {
}
dirtyStorage[key] = v
}
//storages := make(map[string][]byte)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for key, value := range dirtyStorage {
//TODO:: add version schema check
if len(value) == 0 {
if err := tr.DeleteStorage(s.address, key[:]); err != nil {
s.db.setError(err)
@@ -366,6 +436,16 @@ func (s *stateObject) updateTrie() (Trie, error) {
}
s.db.StorageUpdated += 1
}
//if len(value) == 0 {
// storages[string(key[:])] = nil
//} else {
// v, _ := rlp.EncodeToBytes(value)
// storages[string(key[:])] = v
//}
//if err := tr.WriteBatch(storages); err != nil {
// s.db.setError(err)
//}
// Cache the items for preloading
usedStorage = append(usedStorage, common.CopyBytes(key[:]))
}
@@ -452,6 +532,11 @@ func (s *stateObject) updateRoot() {
// The returned set can be nil if nothing to commit. This function assumes all
// storage mutations have already been flushed into trie by updateRoot.
func (s *stateObject) commit() (*trienode.NodeSet, error) {
if s.IsContractAccount() && s.trie == nil && s.db.db.GetVersion() != 0 {
panic(fmt.Sprintf("not open contract account, owner: %s, r_version: %d, c_version: %d, root: %s",
s.address.String(), s.db.db.GetVersion(), s.version, s.Root().String()))
}
// Short circuit if trie is not even loaded, don't bother with committing anything
if s.trie == nil {
s.origin = s.data.Copy()
@@ -511,6 +596,7 @@ func (s *stateObject) setBalance(amount *uint256.Int) {
}
func (s *stateObject) deepCopy(db *StateDB) *stateObject {
//TODO:: debug code, deleted in the future
obj := &stateObject{
db: db,
address: s.address,

View File

@@ -54,6 +54,16 @@ type revision struct {
journalIndex int
}
var (
stateDBGetTimer = metrics.NewRegisteredTimer("statedb/get/time", nil)
stateDBGetQPS = metrics.NewRegisteredMeter("statedb/get/qps", nil)
stateDBGetAccountTimer = metrics.NewRegisteredTimer("statedb/account/get/time", nil)
stateDBGetAccountQPS = metrics.NewRegisteredMeter("statedb/account/get/qps", nil)
stateDBGetStorageTimer = metrics.NewRegisteredTimer("statedb/storage/get/time", nil)
stateDBGetStorageQPS = metrics.NewRegisteredMeter("statedb/storage/get/qps", nil)
)
// StateDB structs within the ethereum protocol are used to store anything
// within the merkle trie. StateDBs take care of caching and storing
// nested states. It's the general query interface to retrieve:
@@ -166,12 +176,18 @@ func NewWithSharedPool(root common.Hash, db Database, snaps *snapshot.Tree) (*St
if err != nil {
return nil, err
}
statedb.storagePool = NewStoragePool()
//statedb.storagePool = NewStoragePool()
return statedb, nil
}
// New creates a new state from a given trie.
func New(root common.Hash, db Database, snaps *snapshot.Tree) (*StateDB, error) {
if db.Scheme() == rawdb.VersionScheme && snaps != nil {
panic("statedb snapshot must be nil in version db.")
}
// clean up previous traces
db.Reset()
sdb := &StateDB{
db: db,
originalRoot: root,
@@ -196,6 +212,7 @@ func New(root common.Hash, db Database, snaps *snapshot.Tree) (*StateDB, error)
sdb.snap = sdb.snaps.Snapshot(root)
}
// It should only one to open account tree
tr, err := db.OpenTrie(root)
// return error when 1. failed to open trie and 2. the snap is nil or the snap is not nil and done verification
if err != nil && (sdb.snap == nil || sdb.snap.Verified()) {
@@ -222,6 +239,9 @@ func (s *StateDB) TransferPrefetcher(prev *StateDB) {
prev.prefetcherLock.Lock()
fetcher = prev.prefetcher
prev.prefetcher = nil
if fetcher != nil {
panic("TransferPrefetcher is not nil")
}
prev.prefetcherLock.Unlock()
s.prefetcherLock.Lock()
@@ -243,6 +263,8 @@ func (s *StateDB) StartPrefetcher(namespace string) {
s.prefetcher = nil
}
if s.snap != nil {
// TODO:: debug code , will be deleted in the future
panic("snapshot is not nill, will start prefetch")
parent := s.snap.Parent()
if parent != nil {
s.prefetcher = newTriePrefetcher(s.db, s.originalRoot, parent.Root(), namespace)
@@ -305,6 +327,7 @@ func (s *StateDB) EnablePipeCommit() {
if s.snap != nil && s.snaps.Layers() > 1 {
// after big merge, disable pipeCommit for now,
// because `s.db.TrieDB().Update` should be called after `s.trie.Commit(true)`
panic("snapshot is not nil")
s.pipeCommit = false
}
}
@@ -323,6 +346,8 @@ func (s *StateDB) MarkFullProcessed() {
func (s *StateDB) setError(err error) {
if s.dbErr == nil {
s.dbErr = err
} else {
s.dbErr = fmt.Errorf(s.dbErr.Error()+", ", err.Error())
}
}
@@ -338,6 +363,8 @@ func (s *StateDB) Error() error {
// Not thread safe
func (s *StateDB) Trie() (Trie, error) {
if s.trie == nil {
// TODO:: debug code, will be deleted in the future.
panic("state get trie is nil")
err := s.WaitPipeVerification()
if err != nil {
return nil, err
@@ -665,6 +692,7 @@ func (s *StateDB) updateStateObject(obj *stateObject) {
if err := s.trie.UpdateAccount(addr, &obj.data); err != nil {
s.setError(fmt.Errorf("updateStateObject (%x) error: %v", addr[:], err))
}
if obj.dirtyCode {
s.trie.UpdateContractCode(obj.Address(), common.BytesToHash(obj.CodeHash()), obj.code)
}
@@ -692,6 +720,7 @@ func (s *StateDB) deleteStateObject(obj *stateObject) {
}
// Delete the account from the trie
addr := obj.Address()
if err := s.trie.DeleteAccount(addr); err != nil {
s.setError(fmt.Errorf("deleteStateObject (%x) error: %v", addr[:], err))
}
@@ -716,9 +745,18 @@ func (s *StateDB) getDeletedStateObject(addr common.Address) *stateObject {
if obj := s.stateObjects[addr]; obj != nil {
return obj
}
defer func(start time.Time) {
stateDBGetTimer.UpdateSince(start)
stateDBGetQPS.Mark(1)
stateDBGetAccountTimer.UpdateSince(start)
stateDBGetAccountQPS.Mark(1)
}(time.Now())
// If no live objects are available, attempt to use snapshots
var data *types.StateAccount
if s.snap != nil {
panic("snapshot is not nil")
start := time.Now()
acc, err := s.snap.Account(crypto.HashData(s.hasher, addr.Bytes()))
if metrics.EnabledExpensive {
@@ -743,9 +781,12 @@ func (s *StateDB) getDeletedStateObject(addr common.Address) *stateObject {
}
}
version := InvalidSateObjectVersion
// If snapshot unavailable or reading from it failed, load from the database
if data == nil {
if s.trie == nil {
// TODO:: debug code, will be deleted in the future.
panic("getDeletedStateObject get trie is nil")
tr, err := s.db.OpenTrie(s.originalRoot)
if err != nil {
s.setError(errors.New("failed to open trie tree"))
@@ -755,7 +796,12 @@ func (s *StateDB) getDeletedStateObject(addr common.Address) *stateObject {
}
start := time.Now()
var err error
data, err = s.trie.GetAccount(addr)
if vtr, ok := s.trie.(*VersaTree); ok {
version, data, err = vtr.getAccountWithVersion(addr)
} else {
data, err = s.trie.GetAccount(addr)
}
if metrics.EnabledExpensive {
s.AccountReads += time.Since(start)
}
@@ -768,7 +814,7 @@ func (s *StateDB) getDeletedStateObject(addr common.Address) *stateObject {
}
}
// Insert into the live set
obj := newObject(s, addr, data)
obj := newObject(s, addr, data, version)
s.setStateObject(obj)
return obj
}
@@ -790,7 +836,7 @@ func (s *StateDB) getOrNewStateObject(addr common.Address) *stateObject {
// the given address, it is overwritten and returned as the second return value.
func (s *StateDB) createObject(addr common.Address) (newobj, prev *stateObject) {
prev = s.getDeletedStateObject(addr) // Note, prev might have been deleted, we need that!
newobj = newObject(s, addr, nil)
newobj = newObject(s, addr, nil, InvalidSateObjectVersion)
if prev == nil {
s.journal.append(createObjectChange{account: &addr})
} else {
@@ -859,10 +905,12 @@ func (s *StateDB) CopyDoPrefetch() *StateDB {
// If doPrefetch is true, it tries to reuse the prefetcher, the copied StateDB will do active trie prefetch.
// otherwise, just do inactive copy trie prefetcher.
func (s *StateDB) copyInternal(doPrefetch bool) *StateDB {
db := s.db.Copy()
tr := db.CopyTrie(s.trie)
// Copy all the basic fields, initialize the memory ones
state := &StateDB{
db: s.db,
trie: s.db.CopyTrie(s.trie),
db: db,
trie: tr,
// noTrie:s.noTrie,
// expectedRoot: s.expectedRoot,
// stateRoot: s.stateRoot,
@@ -933,8 +981,8 @@ func (s *StateDB) copyInternal(doPrefetch bool) *StateDB {
// along with their original values.
state.accounts = copySet(s.accounts)
state.storages = copy2DSet(s.storages)
state.accountsOrigin = copySet(state.accountsOrigin)
state.storagesOrigin = copy2DSet(state.storagesOrigin)
state.accountsOrigin = copySet(s.accountsOrigin)
state.storagesOrigin = copy2DSet(s.storagesOrigin)
// Deep copy the logs occurred in the scope of block
for hash, logs := range s.logs {
@@ -1003,6 +1051,7 @@ func (s *StateDB) GetRefund() uint64 {
func (s *StateDB) WaitPipeVerification() error {
// Need to wait for the parent trie to commit
if s.snap != nil {
panic("snapshot is not nil")
if valid := s.snap.WaitAndGetVerifyRes(); !valid {
return errors.New("verification on parent snap failed")
}
@@ -1108,6 +1157,7 @@ func (s *StateDB) PopulateSnapAccountAndStorage() {
for addr := range s.stateObjectsPending {
if obj := s.stateObjects[addr]; !obj.deleted {
if s.snap != nil {
panic("snapshot is not nil")
s.populateSnapStorage(obj)
s.accounts[obj.addrHash] = types.SlimAccountRLP(obj.data)
}
@@ -1147,6 +1197,10 @@ func (s *StateDB) populateSnapStorage(obj *stateObject) bool {
}
func (s *StateDB) AccountsIntermediateRoot() {
defer func(start time.Time) {
storageIntermediateRootTimer.UpdateSince(start)
}(time.Now())
tasks := make(chan func())
finishCh := make(chan struct{})
defer close(finishCh)
@@ -1174,7 +1228,6 @@ func (s *StateDB) AccountsIntermediateRoot() {
wg.Add(1)
tasks <- func() {
obj.updateRoot()
// Cache the data until commit. Note, this update mechanism is not symmetric
// to the deletion, because whereas it is enough to track account updates
// at commit time, deletions need tracking at transaction boundary level to
@@ -1191,6 +1244,9 @@ func (s *StateDB) AccountsIntermediateRoot() {
}
func (s *StateDB) StateIntermediateRoot() common.Hash {
defer func(start time.Time) {
accountIntermediateRootTimer.UpdateSince(start)
}(time.Now())
// If there was a trie prefetcher operating, it gets aborted and irrevocably
// modified after we start retrieving tries. Remove it from the statedb after
// this round of use.
@@ -1210,6 +1266,8 @@ func (s *StateDB) StateIntermediateRoot() common.Hash {
}
}
if s.trie == nil {
// TODO:: debug code, will be deleted in the future.
panic("StateIntermediateRoot get trie is nil")
tr, err := s.db.OpenTrie(s.originalRoot)
if err != nil {
panic(fmt.Sprintf("failed to open trie tree %s", s.originalRoot))
@@ -1369,6 +1427,7 @@ func (s *StateDB) deleteStorage(addr common.Address, addrHash common.Hash, root
// generated, or it's internally corrupted. Fallback to the slow
// one just in case.
if s.snap != nil {
panic("snapshot is not nil")
aborted, size, slots, nodes, err = s.fastDeleteStorage(addrHash, root)
}
if s.snap == nil || err != nil {
@@ -1422,7 +1481,8 @@ func (s *StateDB) handleDestruction(nodes *trienode.MergedNodeSet) (map[common.A
// considerable time and storage deletion isn't supported in hash mode, thus
// preemptively avoiding unnecessary expenses.
incomplete := make(map[common.Address]struct{})
if s.db.TrieDB().Scheme() == rawdb.HashScheme {
// Only pbss need handler incomplete destruction storage trie
if s.db.TrieDB().Scheme() != rawdb.PathScheme {
return incomplete, nil
}
for addr, prev := range s.stateObjectsDestruct {
@@ -1498,6 +1558,7 @@ func (s *StateDB) Commit(block uint64, failPostCommitFunc func(), postCommitFunc
)
if s.snap != nil {
panic("snapshot is not nil")
diffLayer = &types.DiffLayer{}
}
if s.pipeCommit {
@@ -1613,18 +1674,33 @@ func (s *StateDB) Commit(block uint64, failPostCommitFunc func(), postCommitFunc
origin = types.EmptyRootHash
}
if root != origin {
if s.db.Scheme() == rawdb.VersionScheme {
// flush and release will occur regardless of whether the root changes
start := time.Now()
set := triestate.New(s.accountsOrigin, s.storagesOrigin, incomplete)
if err := s.db.TrieDB().Update(root, origin, block, nodes, set); err != nil {
if err := s.db.Flush(); err != nil {
return err
}
//if err := s.db.Release(); err != nil {
// return err
//}
s.originalRoot = root
if metrics.EnabledExpensive {
s.TrieDBCommits += time.Since(start)
}
if s.onCommit != nil {
s.onCommit(set)
} else {
if root != origin {
start := time.Now()
set := triestate.New(s.accountsOrigin, s.storagesOrigin, incomplete)
if err := s.db.TrieDB().Update(root, origin, block, nodes, set); err != nil {
return err
}
s.originalRoot = root
if metrics.EnabledExpensive {
s.TrieDBCommits += time.Since(start)
}
if s.onCommit != nil {
s.onCommit(set)
}
}
}
}
@@ -1663,8 +1739,21 @@ func (s *StateDB) Commit(block uint64, failPostCommitFunc func(), postCommitFunc
// Write any contract code associated with the state object
if obj.code != nil && obj.dirtyCode {
rawdb.WriteCode(codeWriter, common.BytesToHash(obj.CodeHash()), obj.code)
//switch d := s.db.(type) {
//case *cachingVersaDB:
// if d.debug != nil {
// d.debug.OnUpdateCode(obj.address, common.BytesToHash(obj.CodeHash()))
// }
//case *cachingDB:
// if d.debug != nil {
// d.debug.OnUpdateCode(obj.address, common.BytesToHash(obj.CodeHash()))
// }
//default:
// panic("caching db type error")
//}
obj.dirtyCode = false
if s.snap != nil {
panic("snapshot is not nil")
diffLayer.Codes = append(diffLayer.Codes, types.DiffCode{
Hash: common.BytesToHash(obj.CodeHash()),
Code: obj.code,
@@ -1690,6 +1779,7 @@ func (s *StateDB) Commit(block uint64, failPostCommitFunc func(), postCommitFunc
func() error {
// If snapshotting is enabled, update the snapshot tree with this new version
if s.snap != nil {
panic("snapshot is not nil")
if metrics.EnabledExpensive {
defer func(start time.Time) { s.SnapshotCommits += time.Since(start) }(time.Now())
}

View File

@@ -0,0 +1,207 @@
package state
import (
"encoding/json"
"fmt"
"github.com/ethereum/go-ethereum/common"
)
var (
VersionState = "version"
HashState = "hash"
)
type DebugStateDiff struct {
DiffUpdateAccount map[string][]*VersaAccountInfo
DiffDeleteAccount map[string][]common.Address
DiffUpdateStorage map[string][]*VersaStorageInfo
DiffDeleteStorage map[string][]*VersaStorageInfo
DiffCalcHash map[string]map[common.Address]common.Hash
OwnerMap map[common.Address]common.Hash
DiffErrs map[string][]string
}
func (df *DebugStateDiff) diffUpdateAccount(vs []*VersaAccountInfo, hs []*VersaAccountInfo) {
count := len(vs)
if count > len(hs) {
count = len(hs)
}
idx := 0
for ; idx < count; idx++ {
if vs[idx].Address.Cmp(hs[idx].Address) != 0 {
break
}
if vs[idx].Account.Nonce != hs[idx].Account.Nonce {
break
}
if vs[idx].Account.Balance.Cmp(hs[idx].Account.Balance) != 0 {
break
}
if vs[idx].Account.Root.Cmp(hs[idx].Account.Root) != 0 {
break
}
if common.BytesToHash(vs[idx].Account.CodeHash).Cmp(common.BytesToHash(hs[idx].Account.CodeHash)) != 0 {
break
}
}
if idx < len(vs) {
df.DiffUpdateAccount[VersionState] = vs[idx:]
}
if idx < len(hs) {
df.DiffUpdateAccount[HashState] = hs[idx:]
}
return
}
func (df *DebugStateDiff) diffDeleteAccount(vs []common.Address, hs []common.Address) {
count := len(vs)
if count > len(hs) {
count = len(hs)
}
idx := 0
for ; idx < count; idx++ {
if vs[idx].Cmp(hs[idx]) != 0 {
break
}
}
if idx < len(vs) {
df.DiffDeleteAccount[VersionState] = vs[idx:]
}
if idx < len(hs) {
df.DiffDeleteAccount[HashState] = hs[idx:]
}
return
}
func (df *DebugStateDiff) diffUpdateStorage(vs []*VersaStorageInfo, hs []*VersaStorageInfo) {
count := len(vs)
if count > len(hs) {
count = len(hs)
}
idx := 0
for ; idx < count; idx++ {
if vs[idx].Address.Cmp(hs[idx].Address) != 0 {
break
}
if vs[idx].Key != hs[idx].Key {
break
}
if vs[idx].Val != hs[idx].Val {
break
}
}
if idx < len(vs) {
df.DiffUpdateStorage[VersionState] = vs[idx:]
}
if idx < len(hs) {
df.DiffUpdateStorage[HashState] = hs[idx:]
}
return
}
func (df *DebugStateDiff) diffDeleteStorage(vs []*VersaStorageInfo, hs []*VersaStorageInfo) {
count := len(vs)
if count > len(hs) {
count = len(hs)
}
idx := 0
for ; idx < count; idx++ {
if vs[idx].Address.Cmp(hs[idx].Address) != 0 {
break
}
if vs[idx].Key != hs[idx].Key {
break
}
}
if idx < len(vs) {
df.DiffDeleteStorage[VersionState] = vs[idx:]
}
if idx < len(hs) {
df.DiffDeleteStorage[HashState] = hs[idx:]
}
return
}
func (df *DebugStateDiff) diffCalcHash(vs map[common.Address]common.Hash, hs map[common.Address]common.Hash) {
record := make(map[common.Address]struct{})
for address, vch := range vs {
record[address] = struct{}{}
hch, ok := hs[address]
if !ok {
df.DiffCalcHash[VersionState][address] = vch
}
if vch.Cmp(hch) != 0 {
df.DiffCalcHash[VersionState][address] = vch
df.DiffCalcHash[HashState][address] = hch
}
}
for address := range record {
delete(vs, address)
delete(hs, address)
}
for address, hash := range vs {
df.DiffCalcHash[VersionState][address] = hash
}
for address, hash := range hs {
df.DiffCalcHash[HashState][address] = hash
}
}
func GenerateDebugStateDiff(vs *DebugVersionState, hs *DebugHashState) string {
diff := &DebugStateDiff{
DiffUpdateAccount: make(map[string][]*VersaAccountInfo),
DiffDeleteAccount: make(map[string][]common.Address),
DiffUpdateStorage: make(map[string][]*VersaStorageInfo),
DiffDeleteStorage: make(map[string][]*VersaStorageInfo),
DiffCalcHash: make(map[string]map[common.Address]common.Hash),
OwnerMap: make(map[common.Address]common.Hash),
DiffErrs: make(map[string][]string),
}
diff.DiffUpdateAccount[VersionState] = make([]*VersaAccountInfo, 0)
diff.DiffUpdateAccount[HashState] = make([]*VersaAccountInfo, 0)
diff.DiffDeleteAccount[VersionState] = make([]common.Address, 0)
diff.DiffDeleteAccount[HashState] = make([]common.Address, 0)
diff.DiffUpdateStorage[VersionState] = make([]*VersaStorageInfo, 0)
diff.DiffUpdateStorage[HashState] = make([]*VersaStorageInfo, 0)
diff.DiffDeleteStorage[VersionState] = make([]*VersaStorageInfo, 0)
diff.DiffDeleteStorage[HashState] = make([]*VersaStorageInfo, 0)
diff.DiffCalcHash[VersionState] = make(map[common.Address]common.Hash)
diff.DiffCalcHash[HashState] = make(map[common.Address]common.Hash)
diff.DiffErrs[VersionState] = make([]string, 0)
diff.DiffErrs[HashState] = make([]string, 0)
diff.diffUpdateAccount(vs.UpdateAccounts, hs.UpdateAccounts)
diff.diffDeleteAccount(vs.DeleteAccounts, hs.DeleteAccounts)
diff.diffUpdateStorage(vs.UpdateStorage, hs.UpdateStorage)
diff.diffDeleteStorage(vs.DeleteStorage, hs.DeleteStorage)
diff.diffCalcHash(vs.CalcHash, hs.CalcHash)
for address, _ := range diff.DiffCalcHash[HashState] {
diff.OwnerMap[address] = hs.StorageAddr2Owner[address]
}
for address, _ := range diff.DiffCalcHash[VersionState] {
diff.OwnerMap[address] = vs.StorageAddr2Owner[address]
}
if len(vs.Errs) != 0 || len(hs.Errs) != 0 {
diff.DiffErrs[VersionState] = vs.Errs
diff.DiffErrs[HashState] = hs.Errs
}
data, err := json.Marshal(diff)
if err != nil {
panic(fmt.Sprintf("failed to json encode debug info, err: %s", err.Error()))
}
return string(data)
}

View File

@@ -0,0 +1,216 @@
package state
import (
"encoding/json"
"fmt"
"sort"
"strconv"
"sync"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/ethdb"
)
type DebugHashState struct {
disk ethdb.KeyValueStore
lock sync.Mutex
Version int64
AccessTrees map[common.Address][]common.Hash
CommitTrees map[common.Address][]common.Hash
CalcHash map[common.Address]common.Hash
GetAccounts []*VersaAccountInfo
UpdateAccounts []*VersaAccountInfo
DeleteAccounts []common.Address
GetStorage []*VersaStorageInfo
UpdateStorage []*VersaStorageInfo
DeleteStorage []*VersaStorageInfo
StorageAddr2Owner map[common.Address]common.Hash
GetCode map[common.Address][]common.Hash
UpdateCode map[common.Address][]common.Hash
Errs []string
}
func NewDebugHashState(disk ethdb.KeyValueStore) *DebugHashState {
return &DebugHashState{
disk: disk,
AccessTrees: make(map[common.Address][]common.Hash),
CommitTrees: make(map[common.Address][]common.Hash),
CalcHash: make(map[common.Address]common.Hash),
GetAccounts: make([]*VersaAccountInfo, 0),
UpdateAccounts: make([]*VersaAccountInfo, 0),
DeleteAccounts: make([]common.Address, 0),
GetStorage: make([]*VersaStorageInfo, 0),
UpdateStorage: make([]*VersaStorageInfo, 0),
DeleteStorage: make([]*VersaStorageInfo, 0),
StorageAddr2Owner: make(map[common.Address]common.Hash),
GetCode: make(map[common.Address][]common.Hash),
UpdateCode: make(map[common.Address][]common.Hash),
Errs: make([]string, 0),
}
}
func (hs *DebugHashState) OnOpenTree(root common.Hash, owner common.Hash, address common.Address) {
hs.lock.Lock()
defer hs.lock.Unlock()
if _, ok := hs.AccessTrees[address]; !ok {
hs.AccessTrees[address] = make([]common.Hash, 0)
}
hs.AccessTrees[address] = append(hs.AccessTrees[address], root)
hs.StorageAddr2Owner[address] = owner
}
func (hs *DebugHashState) OnGetAccount(addr common.Address, acc *types.StateAccount) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.GetAccounts = append(hs.GetAccounts, &VersaAccountInfo{
Address: addr,
Account: acc,
})
}
func (hs *DebugHashState) OnUpdateAccount(addr common.Address, acc *types.StateAccount) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.UpdateAccounts = append(hs.UpdateAccounts, &VersaAccountInfo{
Address: addr,
Account: acc,
})
}
func (hs *DebugHashState) OnDeleteAccount(address common.Address) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.DeleteAccounts = append(hs.DeleteAccounts, address)
}
func (hs *DebugHashState) OnGetStorage(address common.Address, key []byte, val []byte) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.GetStorage = append(hs.GetStorage, &VersaStorageInfo{
Address: address,
Key: common.Bytes2Hex(key),
Val: common.Bytes2Hex(val),
})
}
func (hs *DebugHashState) OnUpdateStorage(address common.Address, key []byte, val []byte) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.UpdateStorage = append(hs.UpdateStorage, &VersaStorageInfo{
Address: address,
Key: common.Bytes2Hex(key),
Val: common.Bytes2Hex(val),
})
}
func (hs *DebugHashState) OnDeleteStorage(address common.Address, key []byte) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.DeleteStorage = append(hs.DeleteStorage, &VersaStorageInfo{
Address: address,
Key: common.Bytes2Hex(key),
})
}
func (hs *DebugHashState) OnGetCode(addr common.Address, codeHash common.Hash) {
hs.lock.Lock()
defer hs.lock.Unlock()
if _, ok := hs.GetCode[addr]; !ok {
hs.GetCode[addr] = make([]common.Hash, 0)
}
hs.GetCode[addr] = append(hs.GetCode[addr], codeHash)
}
func (hs *DebugHashState) OnUpdateCode(addr common.Address, codeHash common.Hash) {
hs.lock.Lock()
defer hs.lock.Unlock()
if _, ok := hs.UpdateCode[addr]; !ok {
hs.UpdateCode[addr] = make([]common.Hash, 0)
}
hs.UpdateCode[addr] = append(hs.UpdateCode[addr], codeHash)
}
func (hs *DebugHashState) OnCalcHash(addr common.Address, root common.Hash) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.CalcHash[addr] = root
}
func (hs *DebugHashState) OnCommitTree(addr common.Address, root common.Hash) {
hs.lock.Lock()
defer hs.lock.Unlock()
if _, ok := hs.CommitTrees[addr]; !ok {
hs.CommitTrees[addr] = make([]common.Hash, 0)
}
hs.CommitTrees[addr] = append(hs.CommitTrees[addr], root)
}
func (hs *DebugHashState) OnError(err error) {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.Errs = append(hs.Errs, err.Error())
}
func (hs *DebugHashState) flush() {
hs.lock.Lock()
defer hs.lock.Unlock()
hs.sortItems()
data, err := json.Marshal(hs)
if err != nil {
panic(fmt.Sprintf("failed to json encode debug info, err: %s", err.Error()))
}
err = hs.disk.Put(DebugHashStateKey(hs.Version), data)
if err != nil {
panic(fmt.Sprintf("failed to put debug version state into disk, err: %s", err.Error()))
}
}
func (hs *DebugHashState) sortItems() {
sort.Slice(hs.GetAccounts, func(i, j int) bool {
return hs.GetAccounts[i].Address.Cmp(hs.GetAccounts[j].Address) < 0
})
sort.Slice(hs.UpdateAccounts, func(i, j int) bool {
return hs.UpdateAccounts[i].Address.Cmp(hs.UpdateAccounts[j].Address) < 0
})
sort.Slice(hs.DeleteAccounts, func(i, j int) bool {
return hs.DeleteAccounts[i].Cmp(hs.DeleteAccounts[j]) < 0
})
sort.Slice(hs.GetStorage, func(i, j int) bool {
if hs.GetStorage[i].Address.Cmp(hs.GetStorage[j].Address) == 0 {
return hs.GetStorage[i].Key < hs.GetStorage[j].Key
}
return hs.GetStorage[i].Address.Cmp(hs.GetStorage[j].Address) < 0
})
sort.Slice(hs.UpdateStorage, func(i, j int) bool {
if hs.UpdateStorage[i].Address.Cmp(hs.UpdateStorage[j].Address) == 0 {
return hs.UpdateStorage[i].Key < hs.UpdateStorage[j].Key
}
return hs.UpdateStorage[i].Address.Cmp(hs.UpdateStorage[j].Address) < 0
})
sort.Slice(hs.DeleteStorage, func(i, j int) bool {
if hs.DeleteStorage[i].Address.Cmp(hs.DeleteStorage[j].Address) == 0 {
return hs.DeleteStorage[i].Key < hs.DeleteStorage[j].Key
}
return hs.DeleteStorage[i].Address.Cmp(hs.DeleteStorage[j].Address) < 0
})
}
func DebugHashStateKey(version int64) []byte {
key := "debug_hash_prefix" + strconv.FormatInt(version, 10)
return []byte(key)
}

View File

@@ -0,0 +1,290 @@
package state
import (
"encoding/json"
"fmt"
"sort"
"strconv"
"sync"
versa "github.com/bnb-chain/versioned-state-database"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/log"
)
var (
DiffVersionCount = 0
DiskVersionCount = 0
)
type VersaAccountInfo struct {
Address common.Address
Account *types.StateAccount
}
type VersaStorageInfo struct {
Handler versa.TreeHandler
Address common.Address
Key string
Val string
}
type DebugVersionState struct {
disk ethdb.KeyValueStore
versionDB versa.Database
lock sync.Mutex
Version int64
PreState *versa.StateInfo
PostState *versa.StateInfo
AccessTrees map[common.Address][]*versa.TreeInfo
CommitTrees map[common.Address][]*versa.TreeInfo
CalcHash map[common.Address]common.Hash
GetAccounts []*VersaAccountInfo
UpdateAccounts []*VersaAccountInfo
DeleteAccounts []common.Address
GetStorage []*VersaStorageInfo
UpdateStorage []*VersaStorageInfo
DeleteStorage []*VersaStorageInfo
StorageAddr2Owner map[common.Address]common.Hash
GetCode map[common.Address][]common.Hash
UpdateCode map[common.Address][]common.Hash
Errs []string
}
func NewDebugVersionState(disk ethdb.KeyValueStore, versionDB versa.Database) *DebugVersionState {
return &DebugVersionState{
disk: disk,
versionDB: versionDB,
AccessTrees: make(map[common.Address][]*versa.TreeInfo, 0),
CommitTrees: make(map[common.Address][]*versa.TreeInfo, 0),
CalcHash: make(map[common.Address]common.Hash),
GetAccounts: make([]*VersaAccountInfo, 0),
UpdateAccounts: make([]*VersaAccountInfo, 0),
DeleteAccounts: make([]common.Address, 0),
GetStorage: make([]*VersaStorageInfo, 0),
UpdateStorage: make([]*VersaStorageInfo, 0),
DeleteStorage: make([]*VersaStorageInfo, 0),
StorageAddr2Owner: make(map[common.Address]common.Hash),
GetCode: make(map[common.Address][]common.Hash, 0),
UpdateCode: make(map[common.Address][]common.Hash, 0),
Errs: make([]string, 0),
}
}
func (ds *DebugVersionState) SetVersion(version int64) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.Version = version
}
func (ds *DebugVersionState) OnOpenState(handler versa.StateHandler) {
ds.lock.Lock()
defer ds.lock.Unlock()
stateInfo, err := ds.versionDB.GetStateInfo(handler)
if err != nil {
panic(fmt.Sprintf("failed to get state info on open state, err: %s", err.Error()))
}
ds.PreState = stateInfo
}
func (ds *DebugVersionState) OnOpenTree(handler versa.TreeHandler, owner common.Hash, address common.Address) {
ds.lock.Lock()
defer ds.lock.Unlock()
treeInfo, err := ds.versionDB.GetTreeInfo(handler)
if err != nil {
panic(fmt.Sprintf("failed to get tree info on open tree, err: %s", err.Error()))
}
if _, ok := ds.AccessTrees[address]; !ok {
ds.AccessTrees[address] = make([]*versa.TreeInfo, 0)
}
ds.AccessTrees[address] = append(ds.AccessTrees[address], treeInfo)
ds.StorageAddr2Owner[address] = owner
}
func (ds *DebugVersionState) OnGetAccount(addr common.Address, acc *types.StateAccount) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.GetAccounts = append(ds.GetAccounts, &VersaAccountInfo{
Address: addr,
Account: acc,
})
}
func (ds *DebugVersionState) OnUpdateAccount(addr common.Address, acc *types.StateAccount) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.UpdateAccounts = append(ds.UpdateAccounts, &VersaAccountInfo{
Address: addr,
Account: acc,
})
}
func (ds *DebugVersionState) OnDeleteAccount(address common.Address) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.DeleteAccounts = append(ds.DeleteAccounts, address)
}
func (ds *DebugVersionState) OnGetStorage(handler versa.TreeHandler, address common.Address, key []byte, val []byte) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.GetStorage = append(ds.GetStorage, &VersaStorageInfo{
Handler: handler,
Address: address,
Key: common.Bytes2Hex(key),
Val: common.Bytes2Hex(val),
})
}
func (ds *DebugVersionState) OnUpdateStorage(handler versa.TreeHandler, address common.Address, key []byte, val []byte) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.UpdateStorage = append(ds.UpdateStorage, &VersaStorageInfo{
Handler: handler,
Address: address,
Key: common.Bytes2Hex(key),
Val: common.Bytes2Hex(val),
})
}
func (ds *DebugVersionState) OnDeleteStorage(handler versa.TreeHandler, address common.Address, key []byte) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.DeleteStorage = append(ds.DeleteStorage, &VersaStorageInfo{
Handler: handler,
Address: address,
Key: common.Bytes2Hex(key),
})
}
func (ds *DebugVersionState) OnGetCode(addr common.Address, codeHash common.Hash) {
ds.lock.Lock()
defer ds.lock.Unlock()
if _, ok := ds.GetCode[addr]; !ok {
ds.GetCode[addr] = make([]common.Hash, 0)
}
ds.GetCode[addr] = append(ds.GetCode[addr], codeHash)
}
func (ds *DebugVersionState) OnUpdateCode(addr common.Address, codeHash common.Hash) {
ds.lock.Lock()
defer ds.lock.Unlock()
if _, ok := ds.UpdateCode[addr]; !ok {
ds.UpdateCode[addr] = make([]common.Hash, 0)
}
ds.UpdateCode[addr] = append(ds.UpdateCode[addr], codeHash)
}
func (ds *DebugVersionState) OnCalcHash(addr common.Address, root common.Hash) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.CalcHash[addr] = root
}
func (ds *DebugVersionState) OnCommitTree(addr common.Address, handler versa.TreeHandler) {
ds.lock.Lock()
defer ds.lock.Unlock()
treeInfo, err := ds.versionDB.GetTreeInfo(handler)
if err != nil {
panic(fmt.Sprintf("failed to get tree info on commit tree, err: %s", err.Error()))
}
if _, ok := ds.CommitTrees[addr]; !ok {
ds.CommitTrees[addr] = make([]*versa.TreeInfo, 0)
}
ds.CommitTrees[addr] = append(ds.CommitTrees[addr], treeInfo)
}
func (ds *DebugVersionState) OnError(err error) {
ds.lock.Lock()
defer ds.lock.Unlock()
ds.Errs = append(ds.Errs, err.Error())
}
func (ds *DebugVersionState) OnCloseState(handler versa.StateHandler) {
ds.lock.Lock()
defer ds.lock.Unlock()
stateInfo, err := ds.versionDB.GetStateInfo(handler)
if err != nil {
panic(fmt.Sprintf("failed to get state info on close state, err: %s", err.Error()))
}
ds.PostState = stateInfo
oldDiskVersionCount := DiskVersionCount
if ds.PreState.Root.Cmp(ds.PostState.Root) != 0 {
DiffVersionCount++
if ds.PostState.IsDiskVersion {
DiskVersionCount++
}
}
if ds.Version%1000 == 0 || oldDiskVersionCount != DiskVersionCount {
log.Info("version state info", "current block", ds.Version, "diff version count", DiffVersionCount, "disk version count", DiskVersionCount)
}
ds.sortItems()
data, err := json.Marshal(ds)
if err != nil {
panic(fmt.Sprintf("failed to json encode debug info, err: %s", err.Error()))
}
err = ds.disk.Put(DebugVersionStateKey(ds.Version), data)
if err != nil {
panic(fmt.Sprintf("failed to put debug version state into disk, err: %s", err.Error()))
}
if len(ds.Errs) != 0 {
log.Info("version state occurs error", "debug info", string(data))
log.Crit("exit....")
}
}
func (ds *DebugVersionState) sortItems() {
sort.Slice(ds.GetAccounts, func(i, j int) bool {
return ds.GetAccounts[i].Address.Cmp(ds.GetAccounts[j].Address) < 0
})
sort.Slice(ds.UpdateAccounts, func(i, j int) bool {
return ds.UpdateAccounts[i].Address.Cmp(ds.UpdateAccounts[j].Address) < 0
})
sort.Slice(ds.DeleteAccounts, func(i, j int) bool {
return ds.DeleteAccounts[i].Cmp(ds.DeleteAccounts[j]) < 0
})
sort.Slice(ds.GetStorage, func(i, j int) bool {
if ds.GetStorage[i].Address.Cmp(ds.GetStorage[j].Address) == 0 {
return ds.GetStorage[i].Key < ds.GetStorage[j].Key
}
return ds.GetStorage[i].Address.Cmp(ds.GetStorage[j].Address) < 0
})
sort.Slice(ds.UpdateStorage, func(i, j int) bool {
if ds.UpdateStorage[i].Address.Cmp(ds.UpdateStorage[j].Address) == 0 {
return ds.UpdateStorage[i].Key < ds.UpdateStorage[j].Key
}
return ds.UpdateStorage[i].Address.Cmp(ds.UpdateStorage[j].Address) < 0
})
sort.Slice(ds.DeleteStorage, func(i, j int) bool {
if ds.DeleteStorage[i].Address.Cmp(ds.DeleteStorage[j].Address) == 0 {
return ds.DeleteStorage[i].Key < ds.DeleteStorage[j].Key
}
return ds.DeleteStorage[i].Address.Cmp(ds.DeleteStorage[j].Address) < 0
})
}
func DebugVersionStateKey(version int64) []byte {
key := "debug_version_prefix" + strconv.FormatInt(version, 10)
return []byte(key)
}

View File

@@ -25,7 +25,7 @@ import (
)
// NewStateSync creates a new state trie download scheduler.
func NewStateSync(root common.Hash, database ethdb.KeyValueReader, onLeaf func(keys [][]byte, leaf []byte) error, scheme string) *trie.Sync {
func NewStateSync(root common.Hash, database ethdb.Database, onLeaf func(keys [][]byte, leaf []byte) error, scheme string) *trie.Sync {
// Register the storage slot callback if the external callback is specified.
var onSlot func(keys [][]byte, path []byte, leaf []byte, parent common.Hash, parentPath []byte) error
if onLeaf != nil {

View File

@@ -268,7 +268,7 @@ func testIterativeStateSync(t *testing.T, count int, commit bool, bypath bool, s
}
}
batch := dstDb.NewBatch()
if err := sched.Commit(batch); err != nil {
if err := sched.Commit(batch, nil); err != nil {
t.Fatalf("failed to commit data: %v", err)
}
batch.Write()
@@ -369,7 +369,7 @@ func testIterativeDelayedStateSync(t *testing.T, scheme string) {
nodeProcessed = len(nodeResults)
}
batch := dstDb.NewBatch()
if err := sched.Commit(batch); err != nil {
if err := sched.Commit(batch, nil); err != nil {
t.Fatalf("failed to commit data: %v", err)
}
batch.Write()
@@ -469,7 +469,7 @@ func testIterativeRandomStateSync(t *testing.T, count int, scheme string) {
}
}
batch := dstDb.NewBatch()
if err := sched.Commit(batch); err != nil {
if err := sched.Commit(batch, nil); err != nil {
t.Fatalf("failed to commit data: %v", err)
}
batch.Write()
@@ -575,7 +575,7 @@ func testIterativeRandomDelayedStateSync(t *testing.T, scheme string) {
}
}
batch := dstDb.NewBatch()
if err := sched.Commit(batch); err != nil {
if err := sched.Commit(batch, nil); err != nil {
t.Fatalf("failed to commit data: %v", err)
}
batch.Write()
@@ -688,7 +688,7 @@ func testIncompleteStateSync(t *testing.T, scheme string) {
}
}
batch := dstDb.NewBatch()
if err := sched.Commit(batch); err != nil {
if err := sched.Commit(batch, nil); err != nil {
t.Fatalf("failed to commit data: %v", err)
}
batch.Write()

View File

@@ -81,6 +81,7 @@ type triePrefetcher struct {
// newTriePrefetcher
func newTriePrefetcher(db Database, root, rootParent common.Hash, namespace string) *triePrefetcher {
panic("prefetcher not support")
prefix := triePrefetchMetricsPrefix + namespace
p := &triePrefetcher{
db: db,

View File

@@ -101,6 +101,7 @@ func (p *statePrefetcher) Prefetch(block *types.Block, statedb *state.StateDB, c
// the transaction messages using the statedb, but any changes are discarded. The
// only goal is to pre-cache transaction signatures and snapshot clean state. Only used for mining stage
func (p *statePrefetcher) PrefetchMining(txs TransactionsByPriceAndNonce, header *types.Header, gasLimit uint64, statedb *state.StateDB, cfg vm.Config, interruptCh <-chan struct{}, txCurr **types.Transaction) {
panic("prefetcher not support")
var signer = types.MakeSigner(p.config, header.Number, header.Time)
txCh := make(chan *types.Transaction, 2*prefetchThread)

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,15 @@
package bruno
import _ "embed"
// contract codes for Mainnet upgrade
var (
//go:embed mainnet/ValidatorContract
MainnetValidatorContract string
)
// contract codes for Chapel upgrade
var (
//go:embed chapel/ValidatorContract
ChapelValidatorContract string
)

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,19 @@
package euler
import _ "embed"
// contract codes for Mainnet upgrade
var (
//go:embed mainnet/ValidatorContract
MainnetValidatorContract string
//go:embed mainnet/SlashContract
MainnetSlashContract string
)
// contract codes for Chapel upgrade
var (
//go:embed chapel/ValidatorContract
ChapelValidatorContract string
//go:embed chapel/SlashContract
ChapelSlashContract string
)

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,59 @@
package feynman
import _ "embed"
// contract codes for Mainnet upgrade
var (
//go:embed mainnet/ValidatorContract
MainnetValidatorContract string
//go:embed mainnet/SlashContract
MainnetSlashContract string
//go:embed mainnet/TokenHubContract
MainnetTokenHubContract string
//go:embed mainnet/GovHubContract
MainnetGovHubContract string
//go:embed mainnet/CrossChainContract
MainnetCrossChainContract string
//go:embed mainnet/StakingContract
MainnetStakingContract string
//go:embed mainnet/StakeHubContract
MainnetStakeHubContract string
//go:embed mainnet/StakeCreditContract
MainnetStakeCreditContract string
//go:embed mainnet/GovernorContract
MainnetGovernorContract string
//go:embed mainnet/GovTokenContract
MainnetGovTokenContract string
//go:embed mainnet/TimelockContract
MainnetTimelockContract string
//go:embed mainnet/TokenRecoverPortalContract
MainnetTokenRecoverPortalContract string
)
// contract codes for Chapel upgrade
var (
//go:embed chapel/ValidatorContract
ChapelValidatorContract string
//go:embed chapel/SlashContract
ChapelSlashContract string
//go:embed chapel/TokenHubContract
ChapelTokenHubContract string
//go:embed chapel/GovHubContract
ChapelGovHubContract string
//go:embed chapel/CrossChainContract
ChapelCrossChainContract string
//go:embed chapel/StakingContract
ChapelStakingContract string
//go:embed chapel/StakeHubContract
ChapelStakeHubContract string
//go:embed chapel/StakeCreditContract
ChapelStakeCreditContract string
//go:embed chapel/GovernorContract
ChapelGovernorContract string
//go:embed chapel/GovTokenContract
ChapelGovTokenContract string
//go:embed chapel/TimelockContract
ChapelTimelockContract string
//go:embed chapel/TokenRecoverPortalContract
ChapelTokenRecoverPortalContract string
)

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,11 @@
package feynman_fix
import _ "embed"
// contract codes for Chapel upgrade
var (
//go:embed chapel/ValidatorContract
ChapelValidatorContract string
//go:embed chapel/StakeHubContract
ChapelStakeHubContract string
)

Some files were not shown because too many files have changed in this diff Show More