Commit Graph

71 Commits

Author SHA1 Message Date
zjubfd
1ded097733
[R4R]implement diff sync (#376)
* implement block process part of light sync

* add difflayer protocol

* handle difflayer and refine light processor

* add testcase for diff protocol

* make it faster

* allow validator to light sync

* change into diff sync

* ligth sync: download difflayer (#2)

* ligth sync: download difflayer

Signed-off-by: kyrie-yl <lei.y@binance.com>

* download diff layer: fix according to the comments

Signed-off-by: kyrie-yl <lei.y@binance.com>

* download diff layer: update

Signed-off-by: kyrie-yl <lei.y@binance.com>

* download diff layer: fix accroding comments

Signed-off-by: kyrie-yl <lei.y@binance.com>

Co-authored-by: kyrie-yl <lei.y@binance.com>

* update light sync to diff sync

* raise the max diff limit

* add switcher of snap protocol

* fix test case

* make commit concurrently

* remove peer for diff cache when peer closed

* consensus tuning

* add test code

* remove extra message

* fix testcase and lint

make diff block configable

wait code write

fix testcase

resolve comments

resolve comment

* resolve comments

* resolve comments

* resolve comment

* fix mistake

Co-authored-by: kyrie-yl <83150977+kyrie-yl@users.noreply.github.com>
Co-authored-by: kyrie-yl <lei.y@binance.com>
2021-09-28 16:03:38 +08:00
Péter Szilágyi
1a83114c74
all: update author list and licenses 2019-07-22 12:17:27 +03:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong
80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
rjl493456442
b6cac42e9f
core/rawdb: add file lock for freezer 2019-05-16 10:39:31 +03:00
Péter Szilágyi
006c21efc7
cmd, core, eth, les, node: chain freezer on top of db rework 2019-05-16 10:39:29 +03:00
Martin Holst Swende
59e1953246 core, ethdb, trie: mode dirty data to clean cache on flush (#19307)
This PR is a more advanced form of the dirty-to-clean cacher (#18995),
where we reuse previous database write batches as datasets to uncache,
saving a dirty-trie-iteration and a dirty-trie-rlp-reencoding per block.
2019-03-26 15:48:31 +01:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
Péter Szilágyi
d6225ab846
cmd/utils, eth: relinquish GC cache to read cache in archive mode 2019-02-07 14:36:25 +02:00
Dmitrij Koniajev
6a7695e367 ethdb, rpc: support building on js/wasm (#17709)
The changes allow building WebAssembly applications which use ethclient.Client.
2018-10-16 00:47:25 +02:00
gary rong
5d1d1a808d consensus, ethdb, metrics: implement forced-meter (#17667) 2018-09-17 15:32:34 +03:00
gary rong
e8824f6e74 vendor, ethdb: resume write operation asap (#17144)
* vendor: update leveldb

* ethdb: remove useless warning log
2018-07-12 11:07:51 +03:00
gary rong
a4a2343cdc ethdb, core: implement delete for db batch (#17101) 2018-07-02 11:16:30 +03:00
gary rong
f991995918 ethdb: gracefullly handle quit channel (#16794)
* ethdb: gratefullly handle quit channel

* ethdb: minor polish
2018-06-11 16:11:48 +03:00
gary rong
6ce21a4744 vendor, ethdb: print warning log if leveldb is performing compaction (#16766)
* vendor: update leveldb package

* ethdb: print warning log if db is performing compaction

* ethdb: update annotation and log
2018-05-22 11:12:52 +03:00
gary rong
6b2b328cdb ethdb: add leveldb write delay statistic (#16499) 2018-04-16 11:18:24 +03:00
Felix Lange
0fac705ed0
compression/rle: delete RLE compression (#16468) 2018-04-09 13:47:06 +02:00
Péter Szilágyi
495bdb0c71
cmd: export preimages in RLP, support GZIP, uniform with block export 2018-03-26 14:08:01 +03:00
rjl493456442
b6b6f52ec8
cmd: implement preimage dump and import cmds 2018-03-26 12:51:46 +03:00
Kurkó Mihály
39c16c8a1e cmd, ethdb, vendor: integrate leveldb iostats (#16277)
* cmd, dashboard, ethdb, vendor: send iostats to dashboard

* ethdb: change names

* ethdb: handle parsing errors

* ethdb: handle iostats syntax error

* ethdb: r -> w
2018-03-08 14:59:00 +02:00
Anton Evangelatov
ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Péter Szilágyi
3e89b80ccb
ethdb: reset the batch size too on reset 2018-01-30 19:12:28 +02:00
Martin Holst Swende
017b9f7eac core, ethdb: reuse database batches (#15989)
* leveldb: Update leveldb to 211f780 (poolfix)

* core, ethdb: reuse database batches
2018-01-30 19:03:31 +02:00
Felix Lange
10181b57a9 core, eth/downloader: commit block data using batches (#15115)
* ethdb: add Putter interface and Has method

* ethdb: improve docs and add IdealBatchSize

* ethdb: remove memory batch lock

Batches are not safe for concurrent use.

* core: use ethdb.Putter for Write* functions

This covers the easy cases.

* core/state: simplify StateSync

* trie: optimize local node check

* ethdb: add ValueSize to Batch

* core: optimize HasHeader check

This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.

* core: write fast sync block data in batches

Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.

* eth/downloader: commit larger state batches

Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.

* core: optimize HasBlock check

This avoids a random database read to get the number.

* core: use numberCache in HasHeader

numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.

* core: write imported block data using a batch

Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.

This commit also removes posting of logs when a forked block is imported.

* core: fix DB write error handling

* ethdb: use RLock for Has

* core: fix HasBlock comment
2017-09-09 19:03:07 +03:00
Péter Szilágyi
9184249b39 Logger updates 3 (#3730)
* accounts, cmd, eth, ethdb: port logs over to new system

* ethdb: drop concept of cache distribution between dbs

* eth: fix some log nitpicks to make them nicer
2017-03-02 14:06:16 +01:00
Péter Szilágyi
d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Nick Johnson
d30d7800e0 ethdb: Implement interface for prefixed operations to the DB (#3536) 2017-01-11 13:26:09 +01:00
zsfelfoldi
7db7109a5b cmd, eth: added light client and light server modes 2016-11-09 02:12:53 +01:00
Felix Lange
44bc2e80dd Merge pull request #2914 from fjl/node-coinhabit
cmd/utils, node: make datadir reusable for bzzd
2016-09-29 14:24:37 +02:00
Péter Szilágyi
710435b51b core, eth, trie: reuse trie journals in all our code 2016-09-28 11:27:31 +03:00
Felix Lange
52ede09b17 ethdb: add accessor for database directory 2016-09-16 15:24:31 +02:00
Felix Lange
84d11c19fd eth: remove dapp database remains 2016-08-17 17:39:03 +02:00
villesundell
c7442ef0d2 ethdb, p2p/discover: replace "alloted" with "allotted" (#2785) 2016-07-12 11:42:47 +02:00
Péter Szilágyi
e90958cd29 cmd, eth, ethdb, node: prioritise chaindata for resources, bump cache 2016-03-09 10:33:39 +02:00
Péter Szilágyi
f1ddb1a7ad ethdb: fully disable metrics if not requested (oops?) 2016-02-23 15:47:04 +02:00
Felix Lange
b252589960 ethdb: remove Flush 2015-09-14 23:36:30 +02:00
Felix Lange
8b32f10f16 ethdb: add NewBatch 2015-09-14 23:36:30 +02:00
Jeffrey Wilcke
a23478c0be core, eth, trie, xeth: merged state, chain, extra databases in one 2015-08-07 22:29:02 +02:00
Jeffrey Wilcke
1fad8798ec Merge pull request #1515 from fjl/license-fixes
all: fix license headers one more time
2015-07-28 04:29:42 -07:00
Felix Lange
bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Jeffrey Wilcke
8f56eea77d ethdb, trie: removed RLE compression 2015-07-23 12:57:33 +02:00
Jeffrey Wilcke
0cff61beda Merge pull request #1510 from fjl/license-fixes
all: license fixes
2015-07-23 01:43:11 -07:00
Felix Lange
3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Péter Szilágyi
c7e7778f2a cmd, core, eth, ethdb: cache flag to allocate memory for db internal use 2015-07-22 14:00:52 +03:00
Péter Szilágyi
eb2d168710 eth, ethdb: fix a data race during startup/shutdown 2015-07-09 12:44:07 +03:00
Felix Lange
ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00
Péter Szilágyi
ccbb56b4f2 cmd/geth, eth, ethdb: monitor database compactions 2015-06-30 00:51:29 +02:00
obscuren
2a5a55efaf ethdb: accessor for LDB. TODO remove this interface 2015-06-29 18:51:48 +02:00
Péter Szilágyi
803b3c4a82 eth, ethdb: measure database operation latencies too 2015-06-24 18:34:05 +03:00
Péter Szilágyi
792b0ddccd core, eth, eth/fetcher, ethdb: polish metrics gathering a bit 2015-06-24 18:34:04 +03:00