This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
This changes the error code returned by the RPC server in certain situations:
- handler panic: code -32603
- result marshaling error: code -32603
- attempt to subscribe via HTTP: code -32001
In all of the above cases, the server previously returned the default error
code -32000.
Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The JSON-RPC spec requires the "version" field to be exactly "2.0",
so we should verify that. This change is not backwards-compatible with
sloppy client implementations, but I decided to go ahead with it anyway
because the failure will be caught via the returned error.
This adds a generic mechanism for 'dial options' in the RPC client,
and also implements a specific dial option for the JWT authentication
mechanism used by the engine API. Some real tests for the server-side
authentication handling are also added.
Co-authored-by: Joshua Gutow <jgutow@optimism.io>
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes http.Server.ReadHeaderTimeout configurable separately
from ReadTimeout for RPC servers. The default is set to the same as
ReadTimeout, which in order to cause no change in existing deployments.
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
* rpc, node: refactor request validation and add jwt validation
* node, rpc: fix error message, ignore engine api in RegisterAPIs
* node: make authenticated port configurable
* eth/catalyst: enable unauthenticated version of engine api
* node: rework obtainjwtsecret (backport later)
* cmd/geth: added auth port flag
* node: happy lint, happy life
* node: refactor authenticated api
Modifies the authentication mechanism to use default values
* node: trim spaces and newline away from secret
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
When talking to an HTTP2 server, there are situations where it needs to
"rewind" the Request.Body. To allow this, we have to set up the Request.GetBody
function to return a brand new instance of the body.
If not set, we can end up with the following error:
http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
See this commit for more information: cffdcf672a
This replaces the sketchy and undocumented string context keys for HTTP requests
with a defined interface. Using string keys with context is discouraged because
they may clash with keys created by other packages.
We added these keys to make connection metadata available in the signer, so this
change also updates signer/core to use the new PeerInfo API.
* all: work for eth1/2 transtition
* consensus/beacon, eth: change beacon difficulty to 0
* eth: updates
* all: add terminalBlockDifficulty config, fix rebasing issues
* eth: implemented merge interop spec
* internal/ethapi: update to v1.0.0.alpha.2
This commit updates the code to the new spec, moving payloadId into
it's own object. It also fixes an issue with finalizing an empty blockhash.
It also properly sets the basefee
* all: sync polishes, other fixes + refactors
* core, eth: correct semantics for LeavePoW, EnterPoS
* core: fixed rebasing artifacts
* core: light: performance improvements
* core: use keyed field (f)
* core: eth: fix compilation issues + tests
* eth/catalyst: dbetter error codes
* all: move Merger to consensus/, remove reliance on it in bc
* all: renamed EnterPoS and LeavePoW to ReachTDD and FinalizePoS
* core: make mergelogs a function
* core: use InsertChain instead of InsertBlock
* les: drop merger from lightchain object
* consensus: add merger
* core: recoverAncestors in catalyst mode
* core: fix nitpick
* all: removed merger from beacon, use TTD, nitpicks
* consensus: eth: add docstring, removed unnecessary code duplication
* consensus/beacon: better comment
* all: easy to fix nitpicks by karalabe
* consensus/beacon: verify known headers to be sure
* core: comments
* core: eth: don't drop peers who advertise blocks, nitpicks
* core: never add beacon blocks to the future queue
* core: fixed nitpicks
* consensus/beacon: simplify IsTTDReached check
* consensus/beacon: correct IsTTDReached check
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This avoids quadratic time complexity in the lookup of the batch element
corresponding to an RPC response. Unfortunately, the new approach
requires additional memory for the mapping from ID to index.
Fixes#22805
* core: fix warning flagging the use of DeepEqual on error
* apply the same change everywhere possible
* revert change that was committed by mistake
* fix build error
* Update config.go
* revert changes to ConfigCompatError
* review feedback
Co-authored-by: Felix Lange <fjl@twurst.com>
Currently rpc.BlockNumber is marshalled to JSON as a numeric value, which is
wrong because BlockNumber.UnmarshalJSON() wants it to either be hex-encoded
or string "earliest"/"latest"/"pending". As a result, the call chain
rpc.BlockNumberOrHashWithNumber(123) -> json.Marshal() -> json.Unmarshal()
fails with error "cannot unmarshal object into Go value of type string".
The new error type is returned by client operations contains details of
the response error code and response body.
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a rare issue where the client subscription forwarding loop
would attempt send on the subscription's channel after Unsubscribe has
returned, leading to a panic if the subscription channel was already
closed by the user. Example:
sub, _ := client.Subscribe(..., channel, ...)
sub.Unsubscribe()
close(channel)
The race occurred because Unsubscribe called quitWithServer to tell the
forwarding loop to stop sending on sub.channel, but did not wait for the
loop to actually come down. This is fixed by adding an additional channel
to track the shutdown, on which Unsubscribe now waits.
Fixes#22322
* trie: fix tests to work on 32-bit systems
* les: make test work on 32-bit platform
* cmd/geth: fix windows-issues on tests
* trie: improve balance
* cmd/geth: make account tests less verbose + less mem intense
* rpc: make debug-level log output less verbose
* cmd/geth: lint
* Only compare hostnames in ws.origins
Also using a helper function for ToLower consolidates all preparation steps in one function for more maintainable consistency.
Spaces => tabs
Remove a semicolon
Add space at start of comment
Remove parens around conditional
Handle case wehre parsed hostname is empty
When passing a single word like "localhost" the parsed hostname is an empty string. Handle this and the error-parsing case together as default, and the nonempty hostname case in the conditional.
Refactor with new originIsAllowed functions
Adds originIsAllowed() & ruleAllowsOrigin(); removes prepOriginForComparison
Remove blank line
Added tests for simple allowed-orign rule
which does not specify a protocol or port, just a hostname
Fix copy-paste: `:=` => `=`
Remove parens around conditional
Remove autoadded whitespace on blank lines
Compare scheme, hostname, and port with rule
if the rule specifies those portions.
Remove one autoadded trailing whitespace
Better handle case where only origin host is given
e.g. "localhost"
Remove parens around conditional
Refactor: attemptWebsocketConnectionFromOrigin DRY
Include return type on helper function
Provide srv obj in helper fn
Provide srv to helper fn
Remove stray underscore
Remove blank line
parent 93e666b4c1e7e49b8406dc83ed93f4a02ea49ac1
author wbt <wbt@users.noreply.github.com> 1598559718 -0400
committer Martin Holst Swende <martin@swende.se> 1605602257 +0100
gpgsig -----BEGIN PGP SIGNATURE-----
iQFFBAABCAAvFiEEypmrtbNuJK1doP1AaDtDjAWl3fAFAl+zi9ARHG1hcnRpbkBz
d2VuZGUuc2UACgkQaDtDjAWl3fDRiwgAoMtzU8dwRV7Q9xkCwWEx9Wz2f3n6jUr2
VWBycDKGKwRkPPOER3oc9kzjGU/P1tFlK07PjfnAKZ9KWzxpDcJZwYM3xCBurG7A
16y4YsQnzgPNONv3xIkdi3RZtDBIiPFFEmdZFFvZ/jKexfI6JIYPngCAoqdTIFb9
On/aPvvVWQn1ExfmarsvvJ7kUDUG77tZipuacEH5FfFsfelBWOEYPe+I9ToUHskv
+qO6rOkV1Ojk8eBc6o0R1PnApwCAlEhJs7aM/SEOg4B4ZJJneiFuEXBIG9+0yS2I
NOicuDPLGucOB5nBsfIKI3USPeE+3jxdT8go2lN5Nrhm6MimoILDsQ==
=sgUp
-----END PGP SIGNATURE-----
Refactor: drop err var for more concise test lines
Add several tests for new WebSocket origin checks
Remove autoadded whitespace on blank lines
Restore TestWebsocketOrigins originally-named test
and rename the others to be helpers rather than full tests
Remove autoadded whitespace on blank line
Temporarily comment out new test sets
Uncomment test around origin rule with scheme
Remove tests without scheme on browser origin
per https://github.com/ethereum/go-ethereum/pull/21481/files#r479371498
Uncomment tests with port; remove some blank lines
Handle when browser does not specify scheme/port
Uncomment test for including scheme & port in rule
Add IP tests
* node: more tests + table-driven, ws origin changes
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/ethapi: return revert reason for eth_call
* internal/ethapi: moved revert reason logic to doCall
* accounts/abi/bind/backends: added revert reason logic to simulated backend
* internal/ethapi: fixed linting error
* internal/ethapi: check if require reason can be unpacked
* internal/ethapi: better error logic
* internal/ethapi: simplify logic
* internal/ethapi: return vmError()
* internal/ethapi: move handling of revert out of docall
* graphql: removed revert logic until spec change
* rpc: internal/ethapi: added custom error types
* graphql: use returndata instead of return
Return() checks if there is an error. If an error is found, we return nil.
For most use cases it can be beneficial to return the output even if there
was an error. This code should be changed anyway once the spec supports
error reasons in graphql responses
* accounts/abi/bind/backends: added tests for revert reason
* internal/ethapi: add errorCode to revert error
* internal/ethapi: add errorCode of 3 to revertError
* internal/ethapi: unified estimateGasErrors, simplified logic
* internal/ethapi: unified handling of errors in DoEstimateGas
* rpc: print error data field
* accounts/abi/bind/backends: unify simulatedBackend and RPC
* internal/ethapi: added binary data to revertError data
* internal/ethapi: refactored unpacking logic into newRevertError
* accounts/abi/bind/backends: fix EstimateGas
* accounts, console, internal, rpc: minor error interface cleanups
* Revert "accounts, console, internal, rpc: minor error interface cleanups"
This reverts commit 2d3ef53c5304e429a04983210a417c1f4e0dafb7.
* re-apply the good parts of 2d3ef53c53
* rpc: add test for returning server error data from client
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This corrects the call to eth_getBlockByNumber, which previously
returned this error:
can't get latest block: missing value for required argument 1
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes it possible to run geth with JSON-RPC over HTTP and
WebSocket on the same TCP port. The default port for WebSocket
is still 8546.
geth --rpc --rpcport 8545 --ws --wsport 8545
This also removes a lot of deprecated API surface from package rpc.
The rpc package is now purely about serving JSON-RPC and no longer
provides a way to start an HTTP server.
The leaks were mostly in unit tests, and could all be resolved by
adding suitably-sized channel buffers or by restructuring the test
to not send on a channel after an error has occurred.
There is an unavoidable goroutine leak in Console.Interactive: when
we receive a signal, the line reader cannot be unblocked and will get
stuck. This leak is now documented and I've tried to make it slightly
less bad by adding a one-element buffer to the output channels of
the line-reading loop. Should the reader eventually awake from its
blocked state (i.e. when stdin is closed), at least it won't get stuck
trying to send to the interpreter loop which has quit long ago.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a couple of metrics for tracking the timing
and frequency of method calls:
- rpc/requests gauge counts all requests
- rpc/success gauge counts requests which return err == nil
- rpc/failure gauge counts requests which return err != nil
- rpc/duration/all timer tracks timing of all requests
- rpc/duration/<method>/<success/failure> tracks per-method timing
This just prevents a false negative ERROR warning when, for some unknown
reason, a user attempts to turn on the module rpc even though it's already going
to be on.
This removes the error added in #20597 in favor of a log message at
error level. Failing to start broke a bunch of people's setups and is
probably not the right thing to do for this check.
This change makes the client attempt to reconnect when a write fails.
We already had reconnect support, but the reconnect would previously
happen on the next call after an error. Being more eager leads to a
smoother experience overall.
This commit intents to replicate the DialHTTPWithClient function which allows
creating a RPC Client using a custom dialer but for websockets.
We introduce a new DialWebsocketWithDialer function which allows the caller
to instantiate a new websocket client using a custom dialer.
* rpc: remove 'exported or builtin' restriction for parameters
There is no technial reason for this restriction because package reflect
can create values of any type. Requiring parameters and return values to
be exported causes a lot of noise in package exports.
* rpc: fix staticcheck warnings
* rpc: improve codec abstraction
rpc.ServerCodec is an opaque interface. There was only one way to get a
codec using existing APIs: rpc.NewJSONCodec. This change exports
newCodec (as NewFuncCodec) and NewJSONCodec (as NewCodec). It also makes
all codec methods non-public to avoid showing internals in godoc.
While here, remove codec options in tests because they are not
supported anymore.
* p2p/simulations: use github.com/gorilla/websocket
This package was the last remaining user of golang.org/x/net/websocket.
Migrating to the new library wasn't straightforward because it is no
longer possible to treat WebSocket connections as a net.Conn.
* vendor: delete golang.org/x/net/websocket
* rpc: fix godoc comments and run gofmt
This change adds support for gzip encoding on HTTP responses.
Gzip encoding is used when the client sets the 'accept-encoding: gzip' header.
Original change by @brianosaurus, with fixes from @SjonHortensius.
* rpc: implement websockets with github.com/gorilla/websocket
This change makes package rpc use the github.com/gorilla/websocket
package for WebSockets instead of golang.org/x/net/websocket. The new
library is more robust and supports all WebSocket features including
continuation frames.
There are new tests for two issues with the previously-used library:
- TestWebsocketClientPing checks handling of Ping frames.
- TestWebsocketLargeCall checks whether the request size limit is
applied correctly.
* rpc: raise HTTP/WebSocket request size limit to 5MB
* rpc: remove default origin for client connections
The client used to put the local hostname into the Origin header because
the server wanted an origin to accept the connection, but that's silly:
Origin is for browsers/websites. The nobody would whitelist a particular
hostname.
Now that the server doesn't need Origin anymore, don't bother setting
one for clients. Users who need an origin can use DialWebsocket to
create a client with arbitrary origin if needed.
* vendor: put golang.org/x/net/websocket back
* rpc: don't set Origin header for empty (default) origin
* rpc: add HTTP status code to handshake error
This makes it easier to debug failing connections.
* ethstats: use github.com/gorilla/websocket
* rpc: fix lint
This PR updates a comment about the maximum client subscription buffer
to reflect changes made previously, and fixes a test that wouldn't fail
when wantError == true but execution did not return an error.
When cancelling the context for a call on a HTTP-based client while the
call is running, the select in requestOp.wait may hit the <-context.Done()
case instead of the <-op.resp case. This doesn't happen often -- our
cancel test hasn't caught this even though it ran thousands of times
on CI since the RPC client was added.
Fixes#19714
New APIs added:
client.RegisterName(namespace, service) // makes service available to server
client.Notify(ctx, method, args...) // sends a notification
ClientFromContext(ctx) // to get a client in handler method
This is essentially a rewrite of the server-side code. JSON-RPC
processing code is now the same on both server and client side. Many
minor issues were fixed in the process and there is a new test suite for
JSON-RPC spec compliance (and non-compliance in some cases).
List of behavior changes:
- Method handlers are now called with a per-request context instead of a
per-connection context. The context is canceled right after the method
returns.
- Subscription error channels are always closed when the connection
ends. There is no need to also wait on the Notifier's Closed channel
to detect whether the subscription has ended.
- Client now omits "params" instead of sending "params": null when there
are no arguments to a call. The previous behavior was not compliant
with the spec. The server still accepts "params": null.
- Floating point numbers are allowed as "id". The spec doesn't allow
them, but we handle request "id" as json.RawMessage and guarantee that
the same number will be sent back.
- Logging is improved significantly. There is now a message at DEBUG
level for each RPC call served.
* Initial work on a graphql API
* Added receipts, and more transaction fields.
* Finish receipts, add logs
* Add transactionCount to block
* Add types and .
* Update Block type to be compatible with ethql
* Rename nonce to transactionCount in Account, to be compatible with ethql
* Update transaction, receipt and log to match ethql
* Add query operator, for a range of blocks
* Added ommerCount to Block
* Add transactionAt and ommerAt to Block
* Added sendRawTransaction mutation
* Add Call and EstimateGas to graphQL API
* Refactored to use hexutil.Bytes instead of HexBytes
* Replace BigNum with hexutil.Big
* Refactor call and estimateGas to use ethapi struct type
* Replace ethgraphql.Address with common.Address
* Replace ethgraphql.Hash with common.Hash
* Converted most quantities to Long instead of Int
* Add support for logs
* Fix bug in runFilter
* Restructured Transaction to work primarily with headers, so uncle data is reported properly
* Add gasPrice API
* Add protocolVersion API
* Add syncing API
* Moved schema into its own source file
* Move some single use args types into anonymous structs
* Add doc-comments
* Fixed backend fetching to use context
* Added (very) basic tests
* Add documentation to the graphql schema
* Fix reversion for formatting of big numbers
* Correct spelling error
* s/BigInt/Long/
* Update common/types.go
* Fixes in response to review
* Fix lint error
* Updated calls on private functions
* Fix typo in graphql.go
* Rollback ethapi breaking changes for graphql support
Co-Authored-By: Arachnid <arachnid@notdot.net>
Notifier tracks whether subscription are 'active'. A subscription
becomes active when the subscription ID has been sent to the client. If
the client sends notifications in the request handler before the
subscription becomes active they are dropped. The tests tried to work
around this problem by always waiting 5s before sending the first
notification.
Fix it by buffering notifications until the subscription becomes active.
This speeds up all subscription tests.
Also fix TestSubscriptionMultipleNamespaces to wait for three messages
per subscription instead of six. The test now finishes just after all
notifications have been received and doesn't hit the 30s timeout anymore.
* signer: remove local path disclosure from extapi
* signer: show more data in cli ui
* rpc: make http server forward UA and Origin via Context
* signer, clef/core: ui changes + display UA and Origin
* signer: cliui - indicate less trust in remote headers, see https://github.com/ethereum/go-ethereum/issues/17637
* signer: prevent possibility swap KV-entries in aes_gcm storage, fixes#17635
* signer: remove ecrecover from external API
* signer,clef: default reject instead of warn + valideate new passwords. fixes#17632 and #17631
* signer: check calldata length even if no ABI signature is present
* signer: fix failing testcase
* clef: remove account import from external api
* signer: allow space in passwords, improve error messsage
* signer/storage: fix typos
