Improve contract fetching

This merge contains the following set of changes:
  * [Oracle, Improvement] Periodic check for RPC sync state (#656)

CONFIGURATION.md

@@ -8,11 +8,11 @@ COMMON_HOME_RPC_URL | The HTTPS URL(s) used to communicate to the RPC nodes in t
 COMMON_FOREIGN_RPC_URL | The HTTPS URL(s) used to communicate to the RPC nodes in the Foreign network. Several URLs can be specified, delimited by spaces. If the connection to one of these nodes is lost the next URL is used for connection. | URL(s)
 COMMON_HOME_BRIDGE_ADDRESS | The address of the bridge contract address in the Home network. It is used to listen to events from and send validators' transactions to the Home network. | hexidecimal beginning with "0x"
 COMMON_FOREIGN_BRIDGE_ADDRESS | The  address of the bridge contract address in the Foreign network. It is used to listen to events from and send validators' transactions to the Foreign network. | hexidecimal beginning with "0x"
-COMMON_HOME_GAS_PRICE_SUPPLIER_URL | The URL used to get a JSON response from the gas price prediction oracle for the Home network. The gas price provided by the oracle is used to send the validator's transactions to the RPC node. Since it is assumed that the Home network has a predefined gas price (e.g. the gas price in the Core of POA.Network is `1 GWei`), the gas price oracle parameter can be omitted for such networks. | URL
+COMMON_HOME_GAS_PRICE_SUPPLIER_URL | The URL used to get a JSON response from the gas price prediction oracle for the Home network. The gas price provided by the oracle is used to send the validator's transactions to the RPC node. Since it is assumed that the Home network has a predefined gas price (e.g. the gas price in the Core of POA.Network is `1 GWei`), the gas price oracle parameter can be omitted for such networks. Set to `eip1559-gas-estimation` if you want to use EIP1559 RPC-based gas estimation. | URL
 COMMON_HOME_GAS_PRICE_SPEED_TYPE | Assuming the gas price oracle responds with the following JSON structure: `{"fast": 20.0, "block_time": 12.834, "health": true, "standard": 6.0, "block_number": 6470469, "instant": 71.0, "slow": 1.889}`, this parameter specifies the desirable transaction speed. The speed type can be omitted when `COMMON_HOME_GAS_PRICE_SUPPLIER_URL` is not used. | `instant` / `fast` / `standard` / `slow`
 COMMON_HOME_GAS_PRICE_FALLBACK | The gas price (in Wei) that is used if both the oracle and the fall back gas price specified in the Home Bridge contract are not available. | integer
 COMMON_HOME_GAS_PRICE_FACTOR | A value that will multiply the gas price of the oracle to convert it to gwei. If the oracle API returns gas prices in gwei then this can be set to `1`. Also, it could be used to intentionally pay more gas than suggested by the oracle to guarantee the transaction verification. E.g. `1.25` or `1.5`. | integer
-COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL | The URL used to get a JSON response from the gas price prediction oracle for the Foreign network. The provided gas price is used to send the validator's transactions to the RPC node. If the Foreign network is Ethereum Foundation mainnet, the oracle URL can be: https://gasprice.poa.network. Otherwise this parameter can be omitted. Set to `gas-price-oracle` if you want to use npm `gas-price-oracle` package for retrieving gas price from multiple sources. | URL
+COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL | The URL used to get a JSON response from the gas price prediction oracle for the Foreign network. The provided gas price is used to send the validator's transactions to the RPC node. If the Foreign network is Ethereum Foundation mainnet, the oracle URL can be: https://gasprice.poa.network. Otherwise this parameter can be omitted. Set to `gas-price-oracle` if you want to use npm `gas-price-oracle` package for retrieving gas price from multiple sources. Set to `eip1559-gas-estimation` if you want to use EIP1559 RPC-based gas estimation. | URL
 COMMON_FOREIGN_GAS_PRICE_SPEED_TYPE | Assuming the gas price oracle responds with the following JSON structure: `{"fast": 20.0, "block_time": 12.834, "health": true, "standard": 6.0, "block_number": 6470469, "instant": 71.0, "slow": 1.889}`, this parameter specifies the desirable transaction speed. The speed type can be omitted when `COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL`is not used. | `instant` / `fast` / `standard` / `slow`
 COMMON_FOREIGN_GAS_PRICE_FALLBACK | The gas price (in Wei) used if both the oracle and fall back gas price specified in the Foreign Bridge contract are not available. | integer
 COMMON_FOREIGN_GAS_PRICE_FACTOR | A value that will multiply the gas price of the oracle to convert it to gwei. If the oracle API returns gas prices in gwei then this can be set to `1`. Also, it could be used to intentionally pay more gas than suggested by the oracle to guarantee the transaction verification. E.g. `1.25` or `1.5`. | integer
@@ -53,8 +53,14 @@ ORACLE_SHUTDOWN_CONTRACT_METHOD | Method signature to be used in the side chain
 ORACLE_FOREIGN_RPC_BLOCK_POLLING_LIMIT | Max length for the block range used in `eth_getLogs` requests for polling contract events for the Foreign chain. Infinite, if not provided. | `integer`
 ORACLE_HOME_RPC_BLOCK_POLLING_LIMIT | Max length for the block range used in `eth_getLogs` requests for polling contract events for the Home chain. Infinite, if not provided. | `integer`
 ORACLE_JSONRPC_ERROR_CODES | Override default JSON rpc error codes that can trigger RPC fallback to the next URL from the list (or a retry in case of a single RPC URL). Default is `-32603,-32002,-32005`. Should be a comma-separated list of negative integers. | `string`
-ORACLE_VALIDATOR_KEYSTORE_PATH | Path to the keystore v3 json file with the encrypted validator key. | `string`  
+ORACLE_HOME_EVENTS_REPROCESSING | If set to `true`, home events happened in the past will be refetched and processed once again, to ensure that nothing was missed on the first pass. | `bool`
-ORACLE_VALIDATOR_KEYSTORE_PASSWORD | Password from the provided keystore file, oracle won't startup properly, if the provided password is invalid | `string`
+ORACLE_HOME_EVENTS_REPROCESSING_BATCH_SIZE | Batch size for one `eth_getLogs` request when reprocessing old logs in the home chain. Defaults to `1000` | `integer`
+ORACLE_HOME_EVENTS_REPROCESSING_BLOCK_DELAY | Block confirmations number, after which old logs are being reprocessed in the home chain. Defaults to `500` | `integer`
+ORACLE_HOME_RPC_SYNC_STATE_CHECK_INTERVAL | Interval for checking JSON RPC sync state, by requesting the latest block number. Oracle will switch to the fallback JSON RPC in case sync process is stuck | `integer`
+ORACLE_FOREIGN_EVENTS_REPROCESSING  | If set to `true`, foreign events happened in the past will be refetched and processed once again, to ensure that nothing was missed on the first pass. | `bool`
+ORACLE_FOREIGN_EVENTS_REPROCESSING_BATCH_SIZE | Batch size for one `eth_getLogs` request when reprocessing old logs in the foreign chain. Defaults to `500` | `integer`
+ORACLE_FOREIGN_EVENTS_REPROCESSING_BLOCK_DELAY | Block confirmations number, after which old logs are being reprocessed in the foreign chain. Defaults to `250` | `integer`
+ORACLE_FOREIGN_RPC_SYNC_STATE_CHECK_INTERVAL | Interval for checking JSON RPC sync state, by requesting the latest block number. Oracle will switch to the fallback JSON RPC in case sync process is stuck | `integer`
 
 
 ## Monitor configuration

README.md

@@ -19,7 +19,7 @@ Sub-repositories maintained within this monorepo are listed below.
 
 | Sub-repository | Description |
 | --- | --- |
-| [Oracle](oracle/README.md) | Oracle responsible for listening to bridge related events and authorizing asset transfers. |
+| [Oracle](oracle/README.md) | Responsible for listening to bridge related events and authorizing asset transfers. |
 | [Monitor](monitor/README.md) | Tool for checking balances and unprocessed events in bridged networks. |
 | [Deployment](deployment/README.md) | Ansible playbooks for deploying cross-chain bridges. |
 | [Oracle-E2E](oracle-e2e/README.md) | End to end tests for the Oracle |

alm/src/components/ConfirmationsContainer.tsx

@@ -1,9 +1,9 @@
-import React from 'react'
+import React, { useEffect, useState } from 'react'
 import { TransactionReceipt } from 'web3-eth'
 import { useMessageConfirmations } from '../hooks/useMessageConfirmations'
 import { MessageObject } from '../utils/web3'
 import styled from 'styled-components'
-import { CONFIRMATIONS_STATUS } from '../config/constants'
+import { CONFIRMATIONS_STATUS, VALIDATOR_CONFIRMATION_STATUS } from '../config/constants'
 import { CONFIRMATIONS_STATUS_LABEL, CONFIRMATIONS_STATUS_LABEL_HOME } from '../config/descriptions'
 import { SimpleLoading } from './commons/Loading'
 import { ValidatorsConfirmations } from './ValidatorsConfirmations'
@@ -54,7 +54,9 @@ export const ConfirmationsContainer = ({
     home: { name: homeName },
     foreign: { name: foreignName }
   } = useStateProvider()
-  const { requiredSignatures, validatorList } = useValidatorContract({ fromHome, receipt })
+  const src = useValidatorContract(fromHome, receipt ? receipt.blockNumber : 0)
+  const [executionBlockNumber, setExecutionBlockNumber] = useState(0)
+  const dst = useValidatorContract(!fromHome, executionBlockNumber || 'latest')
   const { blockConfirmations } = useBlockConfirmations({ fromHome, receipt })
   const {
     confirmations,
@@ -71,11 +73,21 @@ export const ConfirmationsContainer = ({
     fromHome,
     homeStartBlock,
     foreignStartBlock,
-    requiredSignatures,
+    requiredSignatures: src.requiredSignatures,
-    validatorList,
+    validatorList: src.validatorList,
+    targetValidatorList: dst.validatorList,
     blockConfirmations
   })
 
+  useEffect(
+    () => {
+      if (executionBlockNumber || executionData.status !== VALIDATOR_CONFIRMATION_STATUS.EXECUTION_SUCCESS) return
+
+      setExecutionBlockNumber(executionData.blockNumber)
+    },
+    [executionData.status, executionBlockNumber, executionData.blockNumber]
+  )
+
   const statusLabel = fromHome ? CONFIRMATIONS_STATUS_LABEL_HOME : CONFIRMATIONS_STATUS_LABEL
 
   const parseDescription = () => {
@@ -114,20 +126,22 @@ export const ConfirmationsContainer = ({
           </MultiLine>
         </StatusDescription>
         <ValidatorsConfirmations
-          confirmations={confirmations}
+          confirmations={fromHome ? confirmations.filter(c => dst.validatorList.includes(c.validator)) : confirmations}
-          requiredSignatures={requiredSignatures}
+          requiredSignatures={dst.requiredSignatures}
-          validatorList={validatorList}
+          validatorList={dst.validatorList}
           waitingBlocksResolved={waitingBlocksResolved}
         />
         {signatureCollected && (
           <ExecutionConfirmation
-            messageData={message.data}
+            message={message}
             executionData={executionData}
             isHome={!fromHome}
-            signatureCollected={signatureCollected}
+            confirmations={confirmations}
             setExecutionData={setExecutionData}
             executionEventsFetched={executionEventsFetched}
             setPendingExecution={setPendingExecution}
+            dstRequiredSignatures={dst.requiredSignatures}
+            dstValidatorList={dst.validatorList}
           />
         )}
       </StyledConfirmationContainer>

alm/src/components/ExecutionConfirmation.tsx

@@ -4,38 +4,47 @@ import { useWindowWidth } from '@react-hook/window-size'
 import { SEARCHING_TX, VALIDATOR_CONFIRMATION_STATUS, ALM_HOME_TO_FOREIGN_MANUAL_EXECUTION } from '../config/constants'
 import { SimpleLoading } from './commons/Loading'
 import styled from 'styled-components'
-import { ExecutionData } from '../hooks/useMessageConfirmations'
+import { ConfirmationParam, ExecutionData } from '../hooks/useMessageConfirmations'
 import { GreyLabel, RedLabel, SuccessLabel } from './commons/Labels'
 import { ExplorerTxLink } from './commons/ExplorerTxLink'
 import { Thead, AgeTd, StatusTd } from './commons/Table'
 import { ManualExecutionButton } from './ManualExecutionButton'
 import { useStateProvider } from '../state/StateProvider'
+import { matchesRule, MessageObject, WarnRule } from '../utils/web3'
+import { WarningAlert } from './commons/WarningAlert'
+import { ErrorAlert } from './commons/ErrorAlert'
 
 const StyledExecutionConfirmation = styled.div`
   margin-top: 30px;
 `
 
 export interface ExecutionConfirmationParams {
-  messageData: string
+  message: MessageObject
   executionData: ExecutionData
   setExecutionData: Function
-  signatureCollected: boolean | string[]
+  confirmations: ConfirmationParam[]
   isHome: boolean
   executionEventsFetched: boolean
   setPendingExecution: Function
+  dstRequiredSignatures: number
+  dstValidatorList: string[]
 }
 
 export const ExecutionConfirmation = ({
-  messageData,
+  message,
   executionData,
   setExecutionData,
-  signatureCollected,
+  confirmations,
   isHome,
   executionEventsFetched,
-  setPendingExecution
+  setPendingExecution,
+  dstRequiredSignatures,
+  dstValidatorList
 }: ExecutionConfirmationParams) => {
   const { foreign } = useStateProvider()
   const [safeExecutionAvailable, setSafeExecutionAvailable] = useState(false)
+  const [error, setError] = useState('')
+  const [warning, setWarning] = useState('')
   const availableManualExecution =
     !isHome &&
     (executionData.status === VALIDATOR_CONFIRMATION_STATUS.WAITING ||
@@ -67,9 +76,27 @@ export const ExecutionConfirmation = ({
     [availableManualExecution, foreign.bridgeContract]
   )
 
+  useEffect(
+    () => {
+      if (!message.data || !executionData || !availableManualExecution) return
+
+      try {
+        const fileName = 'warnRules'
+        const rules: WarnRule[] = require(`../snapshots/${fileName}.json`)
+        for (let rule of rules) {
+          if (matchesRule(rule, message)) {
+            setWarning(rule.message)
+            return
+          }
+        }
+      } catch (e) {}
+    },
+    [availableManualExecution, executionData, message, message.data, setWarning]
+  )
+
   const getExecutionStatusElement = (validatorStatus = '') => {
     switch (validatorStatus) {
-      case VALIDATOR_CONFIRMATION_STATUS.SUCCESS:
+      case VALIDATOR_CONFIRMATION_STATUS.EXECUTION_SUCCESS:
         return <SuccessLabel>{validatorStatus}</SuccessLabel>
       case VALIDATOR_CONFIRMATION_STATUS.FAILED:
         return <RedLabel>{validatorStatus}</RedLabel>
@@ -87,6 +114,8 @@ export const ExecutionConfirmation = ({
 
   return (
     <StyledExecutionConfirmation>
+      {error && <ErrorAlert onClick={() => setError('')} error={error} />}
+      {warning && <WarningAlert onClick={() => setWarning('')} error={warning} />}
       <table>
         <Thead>
           <tr>
@@ -125,10 +154,13 @@ export const ExecutionConfirmation = ({
               <td>
                 <ManualExecutionButton
                   safeExecutionAvailable={safeExecutionAvailable}
-                  messageData={messageData}
+                  messageData={message.data}
                   setExecutionData={setExecutionData}
-                  signatureCollected={signatureCollected as string[]}
+                  confirmations={confirmations}
                   setPendingExecution={setPendingExecution}
+                  setError={setError}
+                  requiredSignatures={dstRequiredSignatures}
+                  validatorList={dstValidatorList}
                 />
               </td>
             )}

alm/src/components/MainPage.tsx

@@ -8,7 +8,6 @@ import { TransactionReceipt } from 'web3-eth'
 import { InfoAlert } from './commons/InfoAlert'
 import { ExplorerTxLink } from './commons/ExplorerTxLink'
 import { FOREIGN_NETWORK_NAME, HOME_NETWORK_NAME } from '../config/constants'
-import { ErrorAlert } from './commons/ErrorAlert'
 
 const StyledMainPage = styled.div`
   text-align: center;
@@ -52,7 +51,7 @@ export interface FormSubmitParams {
 
 export const MainPage = () => {
   const history = useHistory()
-  const { home, foreign, error, setError } = useStateProvider()
+  const { home, foreign } = useStateProvider()
   const [networkName, setNetworkName] = useState('')
   const [receipt, setReceipt] = useState<Maybe<TransactionReceipt>>(null)
   const [showInfoAlert, setShowInfoAlert] = useState(false)
@@ -132,7 +131,6 @@ export const MainPage = () => {
             </AlertP>
           </InfoAlert>
         )}
-        {error && <ErrorAlert onClick={() => setError('')} error={error} />}
         <Route exact path={['/']} children={<Form onSubmit={onFormSubmit} />} />
         <Route
           path={['/:chainId/:txHash/:messageIdParam', '/:chainId/:txHash']}

alm/src/components/ManualExecutionButton.tsx

@@ -14,6 +14,7 @@ import { useStateProvider } from '../state/StateProvider'
 import { signatureToVRS, packSignatures } from '../utils/signatures'
 import { getSuccessExecutionData } from '../utils/getFinalizationEvent'
 import { TransactionReceipt } from 'web3-eth'
+import { ConfirmationParam } from '../hooks/useMessageConfirmations'
 
 const ActionButton = styled.button`
   color: var(--button-color);
@@ -30,22 +31,75 @@ interface ManualExecutionButtonParams {
   safeExecutionAvailable: boolean
   messageData: string
   setExecutionData: Function
-  signatureCollected: string[]
+  confirmations: ConfirmationParam[]
   setPendingExecution: Function
+  setError: Function
+  requiredSignatures: number
+  validatorList: string[]
 }
 
 export const ManualExecutionButton = ({
   safeExecutionAvailable,
   messageData,
   setExecutionData,
-  signatureCollected,
+  confirmations,
-  setPendingExecution
+  setPendingExecution,
+  setError,
+  requiredSignatures,
+  validatorList
 }: ManualExecutionButtonParams) => {
-  const { foreign, setError } = useStateProvider()
+  const { foreign } = useStateProvider()
   const { library, activate, account, active } = useWeb3React()
   const [manualExecution, setManualExecution] = useState(false)
   const [allowFailures, setAllowFailures] = useState(false)
-  const notReady = !foreign.bridgeContract || !signatureCollected || !signatureCollected.length
+  const [ready, setReady] = useState(false)
+  const [title, setTitle] = useState('Loading')
+  const [validSignatures, setValidSignatures] = useState<string[]>([])
+
+  useEffect(
+    () => {
+      if (
+        !foreign.bridgeContract ||
+        !foreign.web3 ||
+        !confirmations ||
+        !confirmations.length ||
+        !requiredSignatures ||
+        !validatorList ||
+        !validatorList.length
+      )
+        return
+
+      const signatures = []
+      for (let i = 0; i < confirmations.length && signatures.length < requiredSignatures; i++) {
+        const sig = confirmations[i].signature
+        if (!sig) {
+          continue
+        }
+        const { v, r, s } = signatureToVRS(sig)
+        const signer = foreign.web3.eth.accounts.recover(messageData, `0x${v}`, `0x${r}`, `0x${s}`)
+        if (validatorList.includes(signer)) {
+          signatures.push(sig)
+        }
+      }
+
+      if (signatures.length >= requiredSignatures) {
+        setValidSignatures(signatures.slice(0, requiredSignatures))
+        setTitle('Execute')
+        setReady(true)
+      } else {
+        setTitle('Unavailable')
+      }
+    },
+    [
+      foreign.bridgeContract,
+      foreign.web3,
+      validatorList,
+      requiredSignatures,
+      messageData,
+      setValidSignatures,
+      confirmations
+    ]
+  )
 
   useEffect(
     () => {
@@ -73,9 +127,9 @@ export const ManualExecutionButton = ({
         return
       }
 
-      if (!library || !foreign.bridgeContract || !signatureCollected || !signatureCollected.length) return
+      if (!library || !foreign.bridgeContract || !foreign.web3 || !validSignatures || !validSignatures.length) return
 
-      const signatures = packSignatures(signatureCollected.map(signatureToVRS))
+      const signatures = packSignatures(validSignatures.map(signatureToVRS))
       const messageId = messageData.slice(0, 66)
       const bridge = foreign.bridgeContract
       const executeMethod =
@@ -140,19 +194,20 @@ export const ManualExecutionButton = ({
       foreign.bridgeContract,
       setError,
       messageData,
-      signatureCollected,
       setExecutionData,
       setPendingExecution,
       safeExecutionAvailable,
-      allowFailures
+      allowFailures,
+      foreign.web3,
+      validSignatures
     ]
   )
 
   return (
     <div>
       <div className="is-center">
-        <ActionButton disabled={notReady} className="button outline" onClick={() => setManualExecution(true)}>
+        <ActionButton disabled={!ready} className="button outline" onClick={() => setManualExecution(true)}>
-          Execute
+          {title}
         </ActionButton>
       </div>
       {safeExecutionAvailable && (

alm/src/components/ValidatorsConfirmations.tsx

@@ -1,7 +1,7 @@
 import React from 'react'
 import { formatTimestamp, formatTxHash, getExplorerTxUrl } from '../utils/networks'
 import { useWindowWidth } from '@react-hook/window-size'
-import { SEARCHING_TX, VALIDATOR_CONFIRMATION_STATUS } from '../config/constants'
+import { RECENT_AGE, SEARCHING_TX, VALIDATOR_CONFIRMATION_STATUS } from '../config/constants'
 import { SimpleLoading } from './commons/Loading'
 import styled from 'styled-components'
 import { ConfirmationParam } from '../hooks/useMessageConfirmations'
@@ -31,7 +31,9 @@ export const ValidatorsConfirmations = ({
   const getValidatorStatusElement = (validatorStatus = '') => {
     switch (validatorStatus) {
       case VALIDATOR_CONFIRMATION_STATUS.SUCCESS:
-        return <SuccessLabel>{validatorStatus}</SuccessLabel>
+      case VALIDATOR_CONFIRMATION_STATUS.MANUAL:
+      case VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID:
+        return <SuccessLabel>{VALIDATOR_CONFIRMATION_STATUS.SUCCESS}</SuccessLabel>
       case VALIDATOR_CONFIRMATION_STATUS.FAILED:
         return <RedLabel>{validatorStatus}</RedLabel>
       case VALIDATOR_CONFIRMATION_STATUS.PENDING:
@@ -58,26 +60,28 @@ export const ValidatorsConfirmations = ({
           </tr>
         </Thead>
         <tbody>
-          {validatorList.map((validator, i) => {
+          {confirmations.map((confirmation, i) => {
-            const filteredConfirmation = confirmations.filter(c => c.validator === validator)
+            const displayedStatus = confirmation.status
-            const confirmation = filteredConfirmation.length > 0 ? filteredConfirmation[0] : null
+            const explorerLink = getExplorerTxUrl(confirmation.txHash, true)
-            const displayedStatus = confirmation && confirmation.status ? confirmation.status : ''
+            let elementIfNoTimestamp: any = <SimpleLoading />
-            const explorerLink = confirmation && confirmation.txHash ? getExplorerTxUrl(confirmation.txHash, true) : ''
+            switch (displayedStatus) {
-            const elementIfNoTimestamp =
+              case '':
-              displayedStatus !== VALIDATOR_CONFIRMATION_STATUS.WAITING &&
+              case VALIDATOR_CONFIRMATION_STATUS.UNDEFINED:
-              displayedStatus !== VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED ? (
+                if (waitingBlocksResolved) {
-                (displayedStatus === VALIDATOR_CONFIRMATION_STATUS.UNDEFINED || displayedStatus === '') &&
+                  elementIfNoTimestamp = SEARCHING_TX
-                waitingBlocksResolved ? (
+                }
-                  SEARCHING_TX
+                break
-                ) : (
+              case VALIDATOR_CONFIRMATION_STATUS.WAITING:
-                  <SimpleLoading />
+              case VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED:
-                )
+                elementIfNoTimestamp = ''
-              ) : (
+                break
-                ''
+              case VALIDATOR_CONFIRMATION_STATUS.MANUAL:
-              )
+                elementIfNoTimestamp = RECENT_AGE
+                break
+            }
             return (
               <tr key={i}>
-                <td>{windowWidth < 850 ? formatTxHash(validator) : validator}</td>
+                <td>{windowWidth < 850 ? formatTxHash(confirmation.validator) : confirmation.validator}</td>
                 <StatusTd className="text-center">{getValidatorStatusElement(displayedStatus)}</StatusTd>
                 <AgeTd className="text-center">
                   {confirmation && confirmation.timestamp > 0 ? (
@@ -94,7 +98,7 @@ export const ValidatorsConfirmations = ({
         </tbody>
       </table>
       <RequiredConfirmations>
-        {requiredSignatures} of {validatorList.length} confirmations required
+        At least <strong>{requiredSignatures}</strong> of <strong>{validatorList.length}</strong> confirmations required
       </RequiredConfirmations>
     </div>
   )

alm/src/components/commons/ErrorAlert.tsx

@@ -33,7 +33,7 @@ export const ErrorAlert = ({ onClick, error }: { onClick: () => void; error: str
   }
   return (
     <div className="row is-center">
-      <StyledErrorAlert className="col-10 is-vertical-align row">
+      <StyledErrorAlert className="col-12 is-vertical-align row">
         <InfoIcon color="var(--failed-color)" />
         <TextContainer className="col-10">
           {text}

alm/src/components/commons/WarningAlert.tsx

@@ -0,0 +1,34 @@
+import React from 'react'
+import styled from 'styled-components'
+import { InfoIcon } from './InfoIcon'
+import { CloseIcon } from './CloseIcon'
+
+const StyledErrorAlert = styled.div`
+  border: 1px solid var(--warning-color);
+  border-radius: 4px;
+  margin-bottom: 20px;
+  padding-top: 10px;
+`
+
+const CloseIconContainer = styled.div`
+  cursor: pointer;
+`
+
+const TextContainer = styled.div`
+  white-space: pre-wrap;
+  flex-direction: column;
+`
+
+export const WarningAlert = ({ onClick, error }: { onClick: () => void; error: string }) => {
+  return (
+    <div className="row is-center">
+      <StyledErrorAlert className="col-12 is-vertical-align row">
+        <InfoIcon color="var(--warning-color)" />
+        <TextContainer className="col-10">{error}</TextContainer>
+        <CloseIconContainer className="col-1 is-vertical-align is-center" onClick={onClick}>
+          <CloseIcon color="var(--warning-color)" />
+        </CloseIconContainer>
+      </StyledErrorAlert>
+    </div>
+  )
+}

alm/src/config/constants.ts

@@ -54,14 +54,19 @@ export const CONFIRMATIONS_STATUS = {
 }
 
 export const VALIDATOR_CONFIRMATION_STATUS = {
-  SUCCESS: 'Success',
+  SUCCESS: 'Confirmed',
+  MANUAL: 'Manual',
+  EXECUTION_SUCCESS: 'Executed',
   FAILED: 'Failed',
+  FAILED_VALID: 'Failed valid',
   PENDING: 'Pending',
   WAITING: 'Waiting',
   NOT_REQUIRED: 'Not required',
   UNDEFINED: 'UNDEFINED'
 }
 
+export const RECENT_AGE = 'Recent'
+
 export const SEARCHING_TX = 'Searching Transaction...'
 
 export const INCORRECT_CHAIN_ERROR = `Incorrect chain chosen. Switch to ${FOREIGN_NETWORK_NAME} in the wallet.`

alm/src/hooks/useMessageConfirmations.ts

@@ -29,17 +29,16 @@ export interface useMessageConfirmationsParams {
   foreignStartBlock: Maybe<number>
   requiredSignatures: number
   validatorList: string[]
+  targetValidatorList: string[]
   blockConfirmations: number
 }
 
-export interface BasicConfirmationParam {
+export interface ConfirmationParam {
   validator: string
   status: string
-}
-
-export interface ConfirmationParam extends BasicConfirmationParam {
   txHash: string
   timestamp: number
+  signature?: string
 }
 
 export interface ExecutionData {
@@ -48,6 +47,7 @@ export interface ExecutionData {
   txHash: string
   timestamp: number
   executionResult: boolean
+  blockNumber: number
 }
 
 export const useMessageConfirmations = ({
@@ -58,6 +58,7 @@ export const useMessageConfirmations = ({
   foreignStartBlock,
   requiredSignatures,
   validatorList,
+  targetValidatorList,
   blockConfirmations
 }: useMessageConfirmationsParams) => {
   const { home, foreign } = useStateProvider()
@@ -65,7 +66,7 @@ export const useMessageConfirmations = ({
   const [status, setStatus] = useState(CONFIRMATIONS_STATUS.UNDEFINED)
   const [waitingBlocks, setWaitingBlocks] = useState(false)
   const [waitingBlocksResolved, setWaitingBlocksResolved] = useState(false)
-  const [signatureCollected, setSignatureCollected] = useState<boolean | string[]>(false)
+  const [signatureCollected, setSignatureCollected] = useState(false)
   const [executionEventsFetched, setExecutionEventsFetched] = useState(false)
   const [collectedSignaturesEvent, setCollectedSignaturesEvent] = useState<Maybe<EventData>>(null)
   const [executionData, setExecutionData] = useState<ExecutionData>({
@@ -73,7 +74,8 @@ export const useMessageConfirmations = ({
     validator: '',
     txHash: '',
     timestamp: 0,
-    executionResult: false
+    executionResult: false,
+    blockNumber: 0
   })
   const [waitingBlocksForExecution, setWaitingBlocksForExecution] = useState(false)
   const [waitingBlocksForExecutionResolved, setWaitingBlocksForExecutionResolved] = useState(false)
@@ -140,10 +142,9 @@ export const useMessageConfirmations = ({
   // The collected signature event is only fetched once the signatures are collected on tx from home to foreign, to calculate if
   // the execution tx on the foreign network is waiting for block confirmations
   // This is executed if the message is in Home to Foreign direction only
-  const hasCollectedSignatures = !!signatureCollected // true or string[]
   useEffect(
     () => {
-      if (!fromHome || !receipt || !home.web3 || !home.bridgeContract || !hasCollectedSignatures) return
+      if (!fromHome || !receipt || !home.web3 || !home.bridgeContract || !signatureCollected) return
 
       let timeoutId: number
       let isCancelled = false
@@ -179,7 +180,7 @@ export const useMessageConfirmations = ({
         isCancelled = true
       }
     },
-    [fromHome, home.bridgeContract, home.web3, message.data, receipt, hasCollectedSignatures]
+    [fromHome, home.bridgeContract, home.web3, message.data, receipt, signatureCollected]
   )
 
   // Check if the responsible validator is waiting for block confirmations to execute the message on foreign network
@@ -252,6 +253,35 @@ export const useMessageConfirmations = ({
       let timeoutId: number
       let isCancelled = false
 
+      if (fromHome) {
+        if (!targetValidatorList || !targetValidatorList.length) return
+        const msgHash = home.web3.utils.sha3(message.data)!
+        const allValidators = [...validatorList, ...targetValidatorList].filter((v, i, s) => s.indexOf(v) === i)
+        const manualConfirmations = []
+        for (let i = 0; i < allValidators.length; i++) {
+          try {
+            const overrideSignatures: {
+              [key: string]: string
+            } = require(`../snapshots/signatures_${allValidators[i]}.json`)
+            if (overrideSignatures[msgHash]) {
+              console.log(`Adding manual signature from ${allValidators[i]}`)
+              manualConfirmations.push({
+                status: VALIDATOR_CONFIRMATION_STATUS.MANUAL,
+                validator: allValidators[i],
+                timestamp: 0,
+                txHash: '',
+                signature: overrideSignatures[msgHash]
+              })
+            } else {
+              console.log(`No manual signature from ${allValidators[i]} was found`)
+            }
+          } catch (e) {
+            console.log(`Signatures overrides are not present for ${allValidators[i]}`)
+          }
+        }
+        setConfirmations(manualConfirmations)
+      }
+
       getConfirmationsForTx(
         message.data,
         home.web3,
@@ -284,7 +314,8 @@ export const useMessageConfirmations = ({
       home.bridgeContract,
       requiredSignatures,
       waitingBlocksResolved,
-      homeStartBlock
+      homeStartBlock,
+      targetValidatorList
     ]
   )
 
@@ -343,7 +374,10 @@ export const useMessageConfirmations = ({
   // Sets the message status based in the collected information
   useEffect(
     () => {
-      if (executionData.status === VALIDATOR_CONFIRMATION_STATUS.SUCCESS && existsConfirmation(confirmations)) {
+      if (
+        executionData.status === VALIDATOR_CONFIRMATION_STATUS.EXECUTION_SUCCESS &&
+        existsConfirmation(confirmations)
+      ) {
         const newStatus = executionData.executionResult
           ? CONFIRMATIONS_STATUS.SUCCESS
           : CONFIRMATIONS_STATUS.SUCCESS_MESSAGE_FAILED

alm/src/hooks/useValidatorContract.ts

@@ -4,19 +4,13 @@ import Web3 from 'web3'
 import { getRequiredSignatures, getValidatorAddress, getValidatorList } from '../utils/contract'
 import { BRIDGE_VALIDATORS_ABI } from '../abis'
 import { useStateProvider } from '../state/StateProvider'
-import { TransactionReceipt } from 'web3-eth'
 import { foreignSnapshotProvider, homeSnapshotProvider, SnapshotProvider } from '../services/SnapshotProvider'
 import { FOREIGN_EXPLORER_API, HOME_EXPLORER_API } from '../config/constants'
 
-export interface useValidatorContractParams {
+export const useValidatorContract = (isHome: boolean, blockNumber: number | 'latest') => {
-  fromHome: boolean
-  receipt: Maybe<TransactionReceipt>
-}
-
-export const useValidatorContract = ({ receipt, fromHome }: useValidatorContractParams) => {
   const [validatorContract, setValidatorContract] = useState<Maybe<Contract>>(null)
   const [requiredSignatures, setRequiredSignatures] = useState(0)
-  const [validatorList, setValidatorList] = useState([])
+  const [validatorList, setValidatorList] = useState<string[]>([])
 
   const { home, foreign } = useStateProvider()
 
@@ -29,34 +23,34 @@ export const useValidatorContract = ({ receipt, fromHome }: useValidatorContract
 
   const callRequiredSignatures = async (
     contract: Maybe<Contract>,
-    receipt: TransactionReceipt,
+    blockNumber: number | 'latest',
     setResult: Function,
     snapshotProvider: SnapshotProvider,
     web3: Web3,
     api: string
   ) => {
     if (!contract) return
-    const result = await getRequiredSignatures(contract, receipt.blockNumber, snapshotProvider, web3, api)
+    const result = await getRequiredSignatures(contract, blockNumber, snapshotProvider, web3, api)
     setResult(result)
   }
 
   const callValidatorList = async (
     contract: Maybe<Contract>,
-    receipt: TransactionReceipt,
+    blockNumber: number | 'latest',
     setResult: Function,
     snapshotProvider: SnapshotProvider,
     web3: Web3,
     api: string
   ) => {
     if (!contract) return
-    const result = await getValidatorList(contract, receipt.blockNumber, snapshotProvider, web3, api)
+    const result = await getValidatorList(contract, blockNumber, snapshotProvider, web3, api)
     setResult(result)
   }
 
-  const web3 = fromHome ? home.web3 : foreign.web3
+  const web3 = isHome ? home.web3 : foreign.web3
-  const api = fromHome ? HOME_EXPLORER_API : FOREIGN_EXPLORER_API
+  const api = isHome ? HOME_EXPLORER_API : FOREIGN_EXPLORER_API
-  const bridgeContract = fromHome ? home.bridgeContract : foreign.bridgeContract
+  const bridgeContract = isHome ? home.bridgeContract : foreign.bridgeContract
-  const snapshotProvider = fromHome ? homeSnapshotProvider : foreignSnapshotProvider
+  const snapshotProvider = isHome ? homeSnapshotProvider : foreignSnapshotProvider
 
   useEffect(
     () => {
@@ -68,11 +62,11 @@ export const useValidatorContract = ({ receipt, fromHome }: useValidatorContract
 
   useEffect(
     () => {
-      if (!web3 || !receipt) return
+      if (!web3 || !blockNumber) return
-      callRequiredSignatures(validatorContract, receipt, setRequiredSignatures, snapshotProvider, web3, api)
+      callRequiredSignatures(validatorContract, blockNumber, setRequiredSignatures, snapshotProvider, web3, api)
-      callValidatorList(validatorContract, receipt, setValidatorList, snapshotProvider, web3, api)
+      callValidatorList(validatorContract, blockNumber, setValidatorList, snapshotProvider, web3, api)
     },
-    [validatorContract, receipt, web3, snapshotProvider, api]
+    [validatorContract, blockNumber, web3, snapshotProvider, api]
   )
 
   return {

alm/src/state/StateProvider.tsx

@@ -1,4 +1,4 @@
-import React, { createContext, ReactNode, useState } from 'react'
+import React, { createContext, ReactNode } from 'react'
 import { useNetwork } from '../hooks/useNetwork'
 import {
   HOME_RPC_URL,
@@ -25,8 +25,6 @@ export interface StateContext {
   home: BaseNetworkParams
   foreign: BaseNetworkParams
   loading: boolean
-  error: string
-  setError: Function
 }
 
 const initialState = {
@@ -44,9 +42,7 @@ const initialState = {
     bridgeAddress: FOREIGN_BRIDGE_ADDRESS,
     bridgeContract: null
   },
-  loading: true,
+  loading: true
-  error: '',
-  setError: () => {}
 }
 
 const StateContext = createContext<StateContext>(initialState)
@@ -58,7 +54,6 @@ export const StateProvider = ({ children }: { children: ReactNode }) => {
     homeWeb3: homeNetwork.web3,
     foreignWeb3: foreignNetwork.web3
   })
-  const [error, setError] = useState('')
 
   const value = {
     home: {
@@ -73,9 +68,7 @@ export const StateProvider = ({ children }: { children: ReactNode }) => {
       bridgeContract: foreignBridge,
       ...foreignNetwork
     },
-    loading: homeNetwork.loading || foreignNetwork.loading,
+    loading: homeNetwork.loading || foreignNetwork.loading
-    error,
-    setError
   }
 
   return <StateContext.Provider value={value}>{children}</StateContext.Provider>

alm/src/themes/GlobalStyle.tsx

@@ -28,5 +28,7 @@ export const GlobalStyle = createGlobalStyle<{ theme: ThemeType }>`
     --not-required-bg-color: ${props => props.theme.notRequired.backgroundColor};
     --failed-color: ${props => props.theme.failed.textColor};
     --failed-bg-color: ${props => props.theme.failed.backgroundColor};
+    --warning-color: ${props => props.theme.warning.textColor};
+    --warning-bg-color: ${props => props.theme.warning.backgroundColor};
   }
 `

alm/src/themes/Light.ts

@@ -17,6 +17,10 @@ const theme = {
   failed: {
     textColor: '#de4437',
     backgroundColor: 'rgba(222,68,55,.1)'
+  },
+  warning: {
+    textColor: '#ffa758',
+    backgroundColor: 'rgba(222,68,55,.1)'
   }
 }
 export default theme

alm/src/utils/__tests__/explorer.test.ts

@@ -14,37 +14,40 @@ const messageData = '0x123456'
 const OTHER_HASH = 'aabbccdd'
 const bridgeAddress = '0xFe446bEF1DbF7AFE24E81e05BC8B271C1BA9a560'
 const otherAddress = '0xD4075FB57fCf038bFc702c915Ef9592534bED5c1'
+const validator1 = '0x45b96809336A8b714BFbdAB3E4B5e0fe5d839908'
+const validator2 = '0xAe8bFfc8BBc6AAa9E21ED1E4e4957fe798BEA25f'
+const validator3 = '0x285A6eB779be4db94dA65e2F3518B1c5F0f71244'
 
 describe('getFailedTransactions', () => {
   test('should only return failed transactions', async () => {
     const to = otherAddress
     const transactions = [
-      { isError: '0', to },
+      { isError: '0', to, from: validator1 },
-      { isError: '1', to },
+      { isError: '1', to, from: validator1 },
-      { isError: '0', to },
+      { isError: '0', to, from: validator2 },
-      { isError: '1', to },
+      { isError: '1', to, from: validator2 },
-      { isError: '1', to }
+      { isError: '1', to, from: validator3 }
     ]
 
     const fetchAccountTransactions = jest.fn().mockImplementation(() => transactions)
-    const result = await getFailedTransactions('', to, 0, 1, '', fetchAccountTransactions)
+    const result = await getFailedTransactions(validator1, to, 0, 1, '', fetchAccountTransactions)
-    expect(result.length).toEqual(3)
+    expect(result.length).toEqual(1)
   })
 })
 describe('getSuccessTransactions', () => {
   test('should only return success transactions', async () => {
     const to = otherAddress
     const transactions = [
-      { isError: '0', to },
+      { isError: '0', to, from: validator1 },
-      { isError: '1', to },
+      { isError: '1', to, from: validator1 },
-      { isError: '0', to },
+      { isError: '0', to, from: validator2 },
-      { isError: '1', to },
+      { isError: '1', to, from: validator2 },
-      { isError: '1', to }
+      { isError: '1', to, from: validator3 }
     ]
 
     const fetchAccountTransactions = jest.fn().mockImplementation(() => transactions)
-    const result = await getSuccessTransactions('', to, 0, 1, '', fetchAccountTransactions)
+    const result = await getSuccessTransactions(validator1, to, 0, 1, '', fetchAccountTransactions)
-    expect(result.length).toEqual(2)
+    expect(result.length).toEqual(1)
   })
 })
 describe('filterValidatorSignatureTransaction', () => {

alm/src/utils/__tests__/getConfirmationsForTx.test.ts

@@ -5,7 +5,7 @@ import Web3 from 'web3'
 import { Contract } from 'web3-eth-contract'
 import { APIPendingTransaction, APITransaction } from '../explorer'
 import { VALIDATOR_CONFIRMATION_STATUS } from '../../config/constants'
-import { BasicConfirmationParam } from '../../hooks/useMessageConfirmations'
+import { ConfirmationParam } from '../../hooks/useMessageConfirmations'
 
 jest.mock('../validatorConfirmationHelpers')
 
@@ -18,6 +18,9 @@ const messageData = '0x111111111'
 const web3 = {
   utils: {
     soliditySha3Raw: (data: string) => `0xaaaa${data.replace('0x', '')}`
+  },
+  eth: {
+    accounts: new Web3().eth.accounts
   }
 } as Web3
 const validator1 = '0x45b96809336A8b714BFbdAB3E4B5e0fe5d839908'
@@ -25,7 +28,7 @@ const validator2 = '0xAe8bFfc8BBc6AAa9E21ED1E4e4957fe798BEA25f'
 const validator3 = '0x285A6eB779be4db94dA65e2F3518B1c5F0f71244'
 const validatorList = [validator1, validator2, validator3]
 const signature =
-  '0x519d704bceed17423daa79c20531cc34fc27a4be6e53fc5069a8023019188ca4519d704bceed17423daa79c20531cc34fc27a4be6e53fc5069a8023019188ca4'
+  '0x6f5b74905669999f1abdb52e1e215506907e1849aac7b31854da458b33a5954e15b165007c3703cfd16e61ca46a96a56727ed11fa47be359d3834515accd016e1b'
 const bridgeContract = {
   methods: {
     signature: () => ({
@@ -61,19 +64,19 @@ describe('getConfirmationsForTx', () => {
       validator,
       status: validator !== validator3 ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
     }))
-    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
       txHash: '',
       timestamp: 0
     }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
       timestamp: 0
     }))
-    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
@@ -110,9 +113,8 @@ describe('getConfirmationsForTx', () => {
     expect(setResult).toBeCalledTimes(2)
     expect(getValidatorConfirmation).toBeCalledTimes(1)
     expect(getSuccessExecutionTransaction).toBeCalledTimes(1)
-    expect(setSignatureCollected).toBeCalledTimes(2)
+    expect(setSignatureCollected).toBeCalledTimes(1)
     expect(setSignatureCollected.mock.calls[0][0]).toEqual(true)
-    expect(setSignatureCollected.mock.calls[1][0]).toEqual([signature, signature])
 
     expect(getValidatorFailedTransaction).toBeCalledTimes(1)
     expect(setFailedConfirmations).toBeCalledTimes(1)
@@ -135,7 +137,7 @@ describe('getConfirmationsForTx', () => {
       expect.arrayContaining([
         { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
         { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED, txHash: '', timestamp: 0 }
       ])
     )
   })
@@ -144,19 +146,19 @@ describe('getConfirmationsForTx', () => {
       validator,
       status: validator === validator3 ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
     }))
-    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
       txHash: '',
       timestamp: 0
     }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
       timestamp: 0
     }))
-    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
@@ -208,19 +210,19 @@ describe('getConfirmationsForTx', () => {
       validator,
       status: validator !== validator3 ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
     }))
-    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
       txHash: validatorData.validator !== validator3 ? '0x123' : '',
       timestamp: validatorData.validator !== validator3 ? 123 : 0
     }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
       timestamp: 0
     }))
-    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
@@ -257,9 +259,8 @@ describe('getConfirmationsForTx', () => {
     expect(setResult).toBeCalledTimes(3)
     expect(getValidatorConfirmation).toBeCalledTimes(1)
     expect(getSuccessExecutionTransaction).toBeCalledTimes(1)
-    expect(setSignatureCollected).toBeCalledTimes(2)
+    expect(setSignatureCollected).toBeCalledTimes(1)
     expect(setSignatureCollected.mock.calls[0][0]).toEqual(true)
-    expect(setSignatureCollected.mock.calls[1][0]).toEqual([signature, signature])
 
     expect(getValidatorFailedTransaction).toBeCalledTimes(1)
     expect(setFailedConfirmations).toBeCalledTimes(1)
@@ -281,16 +282,16 @@ describe('getConfirmationsForTx', () => {
     )
     expect(res2).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res3).toEqual(
       expect.arrayContaining([
         { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
         { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED, txHash: '', timestamp: 0 }
       ])
     )
   })
@@ -304,22 +305,22 @@ describe('getConfirmationsForTx', () => {
           ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS
           : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
     }))
-    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
       txHash: validatorData.validator !== validator3 && validatorData.validator !== validator4 ? '0x123' : '',
       timestamp: validatorData.validator !== validator3 && validatorData.validator !== validator4 ? 123 : 0
     }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status:
         validatorData.validator === validator3
-          ? VALIDATOR_CONFIRMATION_STATUS.FAILED
+          ? VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID
           : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: validatorData.validator === validator3 ? '0x123' : '',
       timestamp: validatorData.validator === validator3 ? 123 : 0
     }))
-    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
@@ -356,9 +357,8 @@ describe('getConfirmationsForTx', () => {
     expect(setResult).toBeCalledTimes(4)
     expect(getValidatorConfirmation).toBeCalledTimes(1)
     expect(getSuccessExecutionTransaction).toBeCalledTimes(1)
-    expect(setSignatureCollected).toBeCalledTimes(2)
+    expect(setSignatureCollected).toBeCalledTimes(1)
     expect(setSignatureCollected.mock.calls[0][0]).toEqual(true)
-    expect(setSignatureCollected.mock.calls[1][0]).toEqual([signature, signature])
 
     expect(getValidatorFailedTransaction).toBeCalledTimes(1)
     expect(setFailedConfirmations).toBeCalledTimes(1)
@@ -382,26 +382,26 @@ describe('getConfirmationsForTx', () => {
     )
     expect(res2).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
         { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res3).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID, txHash: '0x123', timestamp: 123 },
-        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED }
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res4).toEqual(
       expect.arrayContaining([
         { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
         { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID, txHash: '0x123', timestamp: 123 },
-        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED }
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED, txHash: '', timestamp: 0 }
       ])
     )
   })
@@ -414,22 +414,22 @@ describe('getConfirmationsForTx', () => {
       validator,
       status: validator === validator1 ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
     }))
-    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
       txHash: validatorData.validator === validator1 ? '0x123' : '',
       timestamp: validatorData.validator === validator1 ? 123 : 0
     }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status:
         validatorData.validator === validator2
-          ? VALIDATOR_CONFIRMATION_STATUS.FAILED
+          ? VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID
           : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: validatorData.validator === validator2 ? '0x123' : '',
       timestamp: validatorData.validator === validator2 ? 123 : 0
     }))
-    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status:
         validatorData.validator === validator3
@@ -492,22 +492,22 @@ describe('getConfirmationsForTx', () => {
     )
     expect(res2).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
         { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x123', timestamp: 123 }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res3).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
         { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x123', timestamp: 123 }
       ])
     )
     expect(res4).toEqual(
       expect.arrayContaining([
         { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID, txHash: '0x123', timestamp: 123 },
         { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x123', timestamp: 123 }
       ])
     )
@@ -521,13 +521,13 @@ describe('getConfirmationsForTx', () => {
       validator,
       status: validator === validator1 ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
     }))
-    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getSuccessExecutionTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
       txHash: validatorData.validator === validator1 ? '0x123' : '',
       timestamp: validatorData.validator === validator1 ? 123 : 0
     }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status:
         validatorData.validator !== validator1
@@ -536,7 +536,7 @@ describe('getConfirmationsForTx', () => {
       txHash: validatorData.validator !== validator1 ? '0x123' : '',
       timestamp: validatorData.validator !== validator1 ? 123 : 0
     }))
-    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+    getValidatorPendingTransaction.mockImplementation(() => async (validatorData: ConfirmationParam) => ({
       validator: validatorData.validator,
       status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
       txHash: '',
@@ -596,9 +596,9 @@ describe('getConfirmationsForTx', () => {
     )
     expect(res2).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res3).toEqual(
@@ -610,9 +610,13 @@ describe('getConfirmationsForTx', () => {
     )
   })
   test('should remove pending state after transaction mined', async () => {
-    // Validator1 success
+    const validator4 = '0x9d2dC11C342F4eF3C5491A048D0f0eBCd2D8f7C3'
-    // Validator2 failed
+    const validatorList = [validator1, validator2, validator3, validator4]
-    // Validator3 Pending
+
+    // Validator1 success (ts=100)
+    // Validator2 failed (ts=200)
+    // Validator3 Pending (ts=300)
+    // Validator4 Excess confirmation (Failed) (ts=400)
 
     getValidatorConfirmation
       .mockImplementationOnce(() => async (validator: string) => ({
@@ -623,41 +627,57 @@ describe('getConfirmationsForTx', () => {
       .mockImplementation(() => async (validator: string) => ({
         validator,
         status:
-          validator !== validator2 ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
+          validator === validator1 || validator === validator3
+            ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS
+            : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
       }))
     getSuccessExecutionTransaction
-      .mockImplementationOnce(() => async (validatorData: BasicConfirmationParam) => ({
+      .mockImplementationOnce(() => async (validatorData: ConfirmationParam) => ({
         validator: validatorData.validator,
         status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
-        txHash: validatorData.validator === validator1 ? '0x123' : '',
+        txHash: validatorData.validator === validator1 ? '0x100' : '',
-        timestamp: validatorData.validator === validator1 ? 123 : 0
+        timestamp: validatorData.validator === validator1 ? 100 : 0
       }))
-      .mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
+      .mockImplementation(() => async (validatorData: ConfirmationParam) => ({
         validator: validatorData.validator,
         status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
-        txHash: validatorData.validator !== validator2 ? '0x123' : '',
+        txHash:
-        timestamp: validatorData.validator !== validator2 ? 123 : 0
+          validatorData.validator === validator1 ? '0x100' : validatorData.validator === validator3 ? '0x300' : '',
+        timestamp: validatorData.validator === validator1 ? 100 : validatorData.validator === validator3 ? 300 : ''
+      }))
+    getValidatorFailedTransaction
+      .mockImplementationOnce(() => async (validatorData: ConfirmationParam) => ({
+        validator: validatorData.validator,
+        status:
+          validatorData.validator === validator2
+            ? VALIDATOR_CONFIRMATION_STATUS.FAILED
+            : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
+        txHash: validatorData.validator === validator2 ? '0x200' : '',
+        timestamp: validatorData.validator === validator2 ? 200 : 0
+      }))
+      .mockImplementation(() => async (validatorData: ConfirmationParam) => ({
+        validator: validatorData.validator,
+        status:
+          validatorData.validator === validator2 || validatorData.validator === validator4
+            ? validatorData.validator === validator2
+              ? VALIDATOR_CONFIRMATION_STATUS.FAILED
+              : VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID
+            : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
+        txHash:
+          validatorData.validator === validator2 ? '0x200' : validatorData.validator === validator4 ? '0x400' : '',
+        timestamp: validatorData.validator === validator2 ? 200 : validatorData.validator === validator4 ? 400 : ''
       }))
-    getValidatorFailedTransaction.mockImplementation(() => async (validatorData: BasicConfirmationParam) => ({
-      validator: validatorData.validator,
-      status:
-        validatorData.validator === validator2
-          ? VALIDATOR_CONFIRMATION_STATUS.FAILED
-          : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
-      txHash: validatorData.validator === validator2 ? '0x123' : '',
-      timestamp: validatorData.validator === validator2 ? 123 : 0
-    }))
     getValidatorPendingTransaction
-      .mockImplementationOnce(() => async (validatorData: BasicConfirmationParam) => ({
+      .mockImplementationOnce(() => async (validatorData: ConfirmationParam) => ({
         validator: validatorData.validator,
         status:
           validatorData.validator === validator3
             ? VALIDATOR_CONFIRMATION_STATUS.PENDING
             : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
-        txHash: validatorData.validator === validator3 ? '0x123' : '',
+        txHash: validatorData.validator === validator3 ? '0x300' : '',
-        timestamp: validatorData.validator === validator3 ? 123 : 0
+        timestamp: validatorData.validator === validator3 ? 300 : 0
       }))
-      .mockImplementationOnce(() => async (validatorData: BasicConfirmationParam) => ({
+      .mockImplementationOnce(() => async (validatorData: ConfirmationParam) => ({
         validator: validatorData.validator,
         status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
         txHash: '',
@@ -698,7 +718,7 @@ describe('getConfirmationsForTx', () => {
 
     expect(getValidatorFailedTransaction).toBeCalledTimes(1)
     expect(setFailedConfirmations).toBeCalledTimes(1)
-    expect(setFailedConfirmations.mock.calls[0][0]).toEqual(false)
+    expect(setFailedConfirmations.mock.calls[0][0]).toEqual(true)
 
     expect(getValidatorPendingTransaction).toBeCalledTimes(1)
     expect(setPendingConfirmations).toBeCalledTimes(1)
@@ -712,28 +732,32 @@ describe('getConfirmationsForTx', () => {
       expect.arrayContaining([
         { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
         { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res2).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x100', timestamp: 100 },
         { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x123', timestamp: 123 }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res3).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x100', timestamp: 100 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x123', timestamp: 123 }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x300', timestamp: 300 },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res4).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x100', timestamp: 100 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x200', timestamp: 200 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x123', timestamp: 123 }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.PENDING, txHash: '0x300', timestamp: 300 },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
 
@@ -761,14 +785,13 @@ describe('getConfirmationsForTx', () => {
     expect(setResult).toBeCalledTimes(7)
     expect(getValidatorConfirmation).toBeCalledTimes(2)
     expect(getSuccessExecutionTransaction).toBeCalledTimes(2)
-    expect(setSignatureCollected).toBeCalledTimes(3)
+    expect(setSignatureCollected).toBeCalledTimes(2)
     expect(setSignatureCollected.mock.calls[0][0]).toEqual(false)
     expect(setSignatureCollected.mock.calls[1][0]).toEqual(true)
-    expect(setSignatureCollected.mock.calls[2][0]).toEqual([signature, signature])
 
     expect(getValidatorFailedTransaction).toBeCalledTimes(2)
     expect(setFailedConfirmations).toBeCalledTimes(2)
-    expect(setFailedConfirmations.mock.calls[0][0]).toEqual(false)
+    expect(setFailedConfirmations.mock.calls[0][0]).toEqual(true)
     expect(setFailedConfirmations.mock.calls[1][0]).toEqual(false)
 
     expect(getValidatorPendingTransaction).toBeCalledTimes(1)
@@ -781,23 +804,26 @@ describe('getConfirmationsForTx', () => {
     const res7 = setResult.mock.calls[6][0](res6)
     expect(res5).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x100', timestamp: 100 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x200', timestamp: 200 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x300', timestamp: 300 },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res6).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x100', timestamp: 100 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x200', timestamp: 200 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x300', timestamp: 300 },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED }
       ])
     )
     expect(res7).toEqual(
       expect.arrayContaining([
-        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 },
+        { validator: validator1, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x100', timestamp: 100 },
-        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x123', timestamp: 123 },
+        { validator: validator2, status: VALIDATOR_CONFIRMATION_STATUS.FAILED, txHash: '0x200', timestamp: 200 },
-        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x123', timestamp: 123 }
+        { validator: validator3, status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS, txHash: '0x300', timestamp: 300 },
+        { validator: validator4, status: VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID, txHash: '0x400', timestamp: 400 }
       ])
     )
   })

alm/src/utils/__tests__/getFinalizationEvent.test.ts

@@ -84,10 +84,11 @@ describe('getFinalizationEvent', () => {
     expect(setResult).toBeCalledTimes(1)
     expect(setResult.mock.calls[0][0]).toEqual({
       validator: validator1,
-      status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
+      status: VALIDATOR_CONFIRMATION_STATUS.EXECUTION_SUCCESS,
       txHash,
       timestamp,
-      executionResult: true
+      executionResult: true,
+      blockNumber: 5523145
     })
 
     expect(getFailedExecution).toBeCalledTimes(0)
@@ -237,7 +238,8 @@ describe('getFinalizationEvent', () => {
       status: VALIDATOR_CONFIRMATION_STATUS.PENDING,
       txHash,
       timestamp: expect.any(Number),
-      executionResult: false
+      executionResult: false,
+      blockNumber: 0
     })
 
     expect(getFailedExecution).toBeCalledTimes(0)
@@ -294,7 +296,8 @@ describe('getFinalizationEvent', () => {
       status: VALIDATOR_CONFIRMATION_STATUS.FAILED,
       txHash,
       timestamp: expect.any(Number),
-      executionResult: false
+      executionResult: false,
+      blockNumber: expect.any(Number)
     })
 
     expect(getFailedExecution).toBeCalledTimes(1)

alm/src/utils/contract.ts

@@ -22,6 +22,16 @@ export const getRequiredBlockConfirmations = async (
   web3: Web3 | null = null,
   api: string = ''
 ) => {
+  let blockConfirmations
+
+  try {
+    blockConfirmations = await contract.methods.requiredBlockConfirmations().call()
+  } catch {}
+
+  if (blockConfirmations) {
+    return parseInt(blockConfirmations)
+  }
+
   const eventsFromSnapshot = snapshotProvider.requiredBlockConfirmationEvents(blockNumber)
   const snapshotBlockNumber = snapshotProvider.snapshotBlockNumber()
 
@@ -35,16 +45,10 @@ export const getRequiredBlockConfirmations = async (
 
   const events = [...eventsFromSnapshot, ...contractEvents]
 
-  let blockConfirmations
+  // Use the value from last event before the transaction
-  if (events.length > 0) {
+  const event = events[events.length - 1]
-    // Use the value from last event before the transaction
+  blockConfirmations = event.returnValues.requiredBlockConfirmations
-    const event = events[events.length - 1]
+
-    blockConfirmations = event.returnValues.requiredBlockConfirmations
-  } else {
-    // This is a special case where RequiredBlockConfirmationChanged was not emitted during initialization in early versions of AMB
-    // of Sokol - Kovan. In this case the current value is used.
-    blockConfirmations = await contract.methods.requiredBlockConfirmations().call()
-  }
   return parseInt(blockConfirmations)
 }
 
@@ -52,11 +56,25 @@ export const getValidatorAddress = (contract: Contract) => contract.methods.vali
 
 export const getRequiredSignatures = async (
   contract: Contract,
-  blockNumber: number,
+  blockNumber: number | 'latest',
   snapshotProvider: SnapshotProvider,
   web3: Web3 | null = null,
   api: string = ''
 ) => {
+  let requiredSignatures
+
+  try {
+    requiredSignatures = await contract.methods.requiredSignatures().call()
+  } catch {}
+
+  if (requiredSignatures) {
+    return parseInt(requiredSignatures)
+  }
+
+  if (blockNumber === 'latest') {
+    return contract.methods.requiredSignatures().call()
+  }
+
   const eventsFromSnapshot = snapshotProvider.requiredSignaturesEvents(blockNumber)
   const snapshotBlockNumber = snapshotProvider.snapshotBlockNumber()
 
@@ -72,17 +90,25 @@ export const getRequiredSignatures = async (
 
   // Use the value form last event before the transaction
   const event = events[events.length - 1]
-  const { requiredSignatures } = event.returnValues
+  ;({ requiredSignatures } = event.returnValues)
   return parseInt(requiredSignatures)
 }
 
 export const getValidatorList = async (
   contract: Contract,
-  blockNumber: number,
+  blockNumber: number | 'latest',
   snapshotProvider: SnapshotProvider,
   web3: Web3 | null = null,
   api: string = ''
 ) => {
+  try {
+    const currentList = await contract.methods.validatorList().call()
+
+    if (currentList) {
+      return currentList
+    }
+  } catch {}
+
   const addedEventsFromSnapshot = snapshotProvider.validatorAddedEvents(blockNumber)
   const removedEventsFromSnapshot = snapshotProvider.validatorRemovedEvents(blockNumber)
   const snapshotBlockNumber = snapshotProvider.snapshotBlockNumber()

alm/src/utils/explorer.ts

@@ -12,6 +12,7 @@ import Web3 from 'web3'
 import { Contract } from 'web3-eth-contract'
 
 export interface APITransaction {
+  from: string
   timeStamp: string
   isError: string
   input: string
@@ -54,7 +55,7 @@ export const fetchAccountTransactions = async ({ account, startBlock, endBlock,
   url.searchParams.append('module', 'account')
   url.searchParams.append('action', 'txlist')
   url.searchParams.append('address', account)
-  url.searchParams.append('filterby', 'from')
+  url.searchParams.append('filterby', 'to')
   url.searchParams.append('startblock', startBlock.toString())
   url.searchParams.append('endblock', endBlock.toString())
 
@@ -64,7 +65,7 @@ export const fetchAccountTransactions = async ({ account, startBlock, endBlock,
     return []
   }
 
-  return result.result
+  return result.result || []
 }
 
 export const fetchPendingTransactions = async ({
@@ -180,7 +181,9 @@ export const getLogs = async (
     if (topics[i] !== null) {
       url.searchParams.append(`topic${i}`, topics[i] as string)
       for (let j = 0; j < i; j++) {
-        url.searchParams.append(`topic${j}_${i}_opr`, 'and')
+        if (topics[j] !== null) {
+          url.searchParams.append(`topic${j}_${i}_opr`, 'and')
+        }
       }
     }
   }
@@ -194,7 +197,7 @@ export const getLogs = async (
   }))
 }
 
-const filterReceiver = (to: string) => (tx: APITransaction) => tx.to.toLowerCase() === to.toLowerCase()
+const filterSender = (from: string) => (tx: APITransaction) => tx.from.toLowerCase() === from.toLowerCase()
 
 export const getFailedTransactions = async (
   account: string,
@@ -204,9 +207,9 @@ export const getFailedTransactions = async (
   api: string,
   getAccountTransactionsMethod = getAccountTransactions
 ): Promise<APITransaction[]> => {
-  const transactions = await getAccountTransactionsMethod({ account, startBlock, endBlock, api })
+  const transactions = await getAccountTransactionsMethod({ account: to, startBlock, endBlock, api })
 
-  return transactions.filter(t => t.isError !== '0').filter(filterReceiver(to))
+  return transactions.filter(t => t.isError !== '0').filter(filterSender(account))
 }
 
 export const getSuccessTransactions = async (
@@ -217,9 +220,9 @@ export const getSuccessTransactions = async (
   api: string,
   getAccountTransactionsMethod = getAccountTransactions
 ): Promise<APITransaction[]> => {
-  const transactions = await getAccountTransactionsMethod({ account, startBlock, endBlock, api })
+  const transactions = await getAccountTransactionsMethod({ account: to, startBlock, endBlock, api })
 
-  return transactions.filter(t => t.isError === '0').filter(filterReceiver(to))
+  return transactions.filter(t => t.isError === '0').filter(filterSender(account))
 }
 
 export const filterValidatorSignatureTransaction = (

alm/src/utils/getConfirmationsForTx.ts

@@ -2,26 +2,37 @@ import Web3 from 'web3'
 import { Contract } from 'web3-eth-contract'
 import { HOME_RPC_POLLING_INTERVAL, VALIDATOR_CONFIRMATION_STATUS } from '../config/constants'
 import { GetTransactionParams, APITransaction, APIPendingTransaction, GetPendingTransactionParams } from './explorer'
-import { getAffirmationsSigned, getMessagesSigned } from './contract'
 import {
   getValidatorConfirmation,
   getValidatorFailedTransaction,
   getValidatorPendingTransaction,
   getSuccessExecutionTransaction
 } from './validatorConfirmationHelpers'
-import { BasicConfirmationParam, ConfirmationParam } from '../hooks/useMessageConfirmations'
+import { ConfirmationParam } from '../hooks/useMessageConfirmations'
+import { signatureToVRS } from './signatures'
 
-const mergeConfirmations = (oldConfirmations: BasicConfirmationParam[], newConfirmations: BasicConfirmationParam[]) => {
+const mergeConfirmations = (oldConfirmations: ConfirmationParam[], newConfirmations: ConfirmationParam[]) => {
   const confirmations = [...oldConfirmations]
   newConfirmations.forEach(validatorData => {
     const index = confirmations.findIndex(e => e.validator === validatorData.validator)
+    if (index === -1) {
+      confirmations.push(validatorData)
+      return
+    }
     const currentStatus = confirmations[index].status
     const newStatus = validatorData.status
     if (
-      (validatorData as ConfirmationParam).txHash ||
+      validatorData.txHash ||
+      !!validatorData.signature ||
       (newStatus !== currentStatus && newStatus !== VALIDATOR_CONFIRMATION_STATUS.UNDEFINED)
     ) {
-      confirmations[index] = validatorData
+      confirmations[index] = {
+        status: validatorData.status,
+        validator: validatorData.validator,
+        timestamp: confirmations[index].timestamp || validatorData.timestamp,
+        txHash: confirmations[index].txHash || validatorData.txHash,
+        signature: confirmations[index].signature || validatorData.signature
+      }
     }
   })
   return confirmations
@@ -45,19 +56,17 @@ export const getConfirmationsForTx = async (
   setPendingConfirmations: Function,
   getSuccessTransactions: (args: GetTransactionParams) => Promise<APITransaction[]>
 ) => {
-  const confirmationContractMethod = fromHome ? getMessagesSigned : getAffirmationsSigned
-
   const hashMsg = web3.utils.soliditySha3Raw(messageData)
   let validatorConfirmations = await Promise.all(
-    validatorList.map(getValidatorConfirmation(web3, hashMsg, bridgeContract, confirmationContractMethod))
+    validatorList.map(getValidatorConfirmation(web3, hashMsg, bridgeContract, fromHome))
   )
 
-  const updateConfirmations = (confirmations: BasicConfirmationParam[]) => {
+  const updateConfirmations = (confirmations: ConfirmationParam[]) => {
     if (confirmations.length === 0) {
       return
     }
     validatorConfirmations = mergeConfirmations(validatorConfirmations, confirmations)
-    setResult((currentConfirmations: BasicConfirmationParam[]) => {
+    setResult((currentConfirmations: ConfirmationParam[]) => {
       if (currentConfirmations && currentConfirmations.length) {
         return mergeConfirmations(currentConfirmations, confirmations)
       }
@@ -67,11 +76,37 @@ export const getConfirmationsForTx = async (
 
   const successConfirmations = validatorConfirmations.filter(c => c.status === VALIDATOR_CONFIRMATION_STATUS.SUCCESS)
   const notSuccessConfirmations = validatorConfirmations.filter(c => c.status !== VALIDATOR_CONFIRMATION_STATUS.SUCCESS)
-  const hasEnoughSignatures = successConfirmations.length === requiredSignatures
+  const hasEnoughSignatures = successConfirmations.length >= requiredSignatures
 
   updateConfirmations(validatorConfirmations)
   setSignatureCollected(hasEnoughSignatures)
 
+  if (hasEnoughSignatures) {
+    setPendingConfirmations(false)
+    if (fromHome) {
+      // fetch collected signatures for possible manual processing
+      const signatures = await Promise.all(
+        Array.from(Array(requiredSignatures).keys()).map(i => bridgeContract.methods.signature(hashMsg, i).call())
+      )
+      const confirmations = signatures.flatMap(sig => {
+        const { v, r, s } = signatureToVRS(sig)
+        const address = web3.eth.accounts.recover(messageData, `0x${v}`, `0x${r}`, `0x${s}`)
+        return successConfirmations.filter(c => c.validator === address).map(c => ({ ...c, signature: sig }))
+      })
+      updateConfirmations(confirmations)
+    }
+  }
+
+  // get transactions from success signatures
+  const successConfirmationWithData = await Promise.all(
+    successConfirmations.map(
+      getSuccessExecutionTransaction(web3, bridgeContract, fromHome, messageData, startBlock, getSuccessTransactions)
+    )
+  )
+
+  const successConfirmationWithTxFound = successConfirmationWithData.filter(v => v.txHash !== '')
+  updateConfirmations(successConfirmationWithTxFound)
+
   // If signatures not collected, look for pending transactions
   if (!hasEnoughSignatures) {
     // Check if confirmation is pending
@@ -84,16 +119,6 @@ export const getConfirmationsForTx = async (
     )
     updateConfirmations(validatorPendingConfirmations)
     setPendingConfirmations(validatorPendingConfirmations.length > 0)
-  } else {
-    setPendingConfirmations(false)
-    if (fromHome) {
-      // fetch collected signatures for possible manual processing
-      setSignatureCollected(
-        await Promise.all(
-          Array.from(Array(requiredSignatures).keys()).map(i => bridgeContract.methods.signature(hashMsg, i).call())
-        )
-      )
-    }
   }
 
   const undefinedConfirmations = validatorConfirmations.filter(
@@ -103,13 +128,27 @@ export const getConfirmationsForTx = async (
   // Check if confirmation failed
   const validatorFailedConfirmationsChecks = await Promise.all(
     undefinedConfirmations.map(
-      getValidatorFailedTransaction(bridgeContract, messageData, startBlock, getFailedTransactions)
+      getValidatorFailedTransaction(web3, bridgeContract, messageData, startBlock, getFailedTransactions)
     )
   )
-  const validatorFailedConfirmations = validatorFailedConfirmationsChecks.filter(
+  let validatorFailedConfirmations = validatorFailedConfirmationsChecks.filter(
-    c => c.status === VALIDATOR_CONFIRMATION_STATUS.FAILED
+    c => c.status === VALIDATOR_CONFIRMATION_STATUS.FAILED || c.status === VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID
+  )
+  if (hasEnoughSignatures && !fromHome) {
+    const lastTS = Math.max(...successConfirmationWithTxFound.map(c => c.timestamp || 0))
+    validatorFailedConfirmations = validatorFailedConfirmations.map(
+      c =>
+        c.timestamp < lastTS
+          ? c
+          : {
+              ...c,
+              status: VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID
+            }
+    )
+  }
+  setFailedConfirmations(
+    !hasEnoughSignatures && validatorFailedConfirmations.some(c => c.status === VALIDATOR_CONFIRMATION_STATUS.FAILED)
   )
-  setFailedConfirmations(validatorFailedConfirmations.length > validatorList.length - requiredSignatures)
   updateConfirmations(validatorFailedConfirmations)
 
   const missingConfirmations = validatorConfirmations.filter(
@@ -120,21 +159,13 @@ export const getConfirmationsForTx = async (
     // If signatures collected, it should set other signatures not found as not required
     const notRequiredConfirmations = missingConfirmations.map(c => ({
       validator: c.validator,
-      status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED
+      status: VALIDATOR_CONFIRMATION_STATUS.NOT_REQUIRED,
+      timestamp: 0,
+      txHash: ''
     }))
     updateConfirmations(notRequiredConfirmations)
   }
 
-  // get transactions from success signatures
-  const successConfirmationWithData = await Promise.all(
-    successConfirmations.map(
-      getSuccessExecutionTransaction(web3, bridgeContract, fromHome, messageData, startBlock, getSuccessTransactions)
-    )
-  )
-
-  const successConfirmationWithTxFound = successConfirmationWithData.filter(v => v.txHash !== '')
-  updateConfirmations(successConfirmationWithTxFound)
-
   // retry if not all signatures are collected and some confirmations are still missing
   // or some success transactions were not fetched successfully
   if (

alm/src/utils/getFinalizationEvent.ts

@@ -59,11 +59,12 @@ export const getSuccessExecutionData = async (
     const validatorAddress = web3.utils.toChecksumAddress(txReceipt.from)
 
     return {
-      status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
+      status: VALIDATOR_CONFIRMATION_STATUS.EXECUTION_SUCCESS,
       validator: validatorAddress,
       txHash: event.transactionHash,
       timestamp: blockTimestamp,
-      executionResult: event.returnValues.status
+      executionResult: event.returnValues.status,
+      blockNumber: event.blockNumber
     }
   }
   return null
@@ -115,7 +116,8 @@ export const getFinalizationEvent = async (
           validator: validator,
           txHash: pendingTx.hash,
           timestamp: nowTimestamp,
-          executionResult: false
+          executionResult: false,
+          blockNumber: 0
         })
         setPendingExecution(true)
       } else {
@@ -144,7 +146,8 @@ export const getFinalizationEvent = async (
               validator: validator,
               txHash: failedTx.hash,
               timestamp,
-              executionResult: false
+              executionResult: false,
+              blockNumber: parseInt(failedTx.blockNumber)
             })
             setFailedExecution(true)
           }

alm/src/utils/validatorConfirmationHelpers.ts

@@ -1,38 +1,45 @@
 import Web3 from 'web3'
 import { Contract } from 'web3-eth-contract'
-import { BasicConfirmationParam, ConfirmationParam } from '../hooks/useMessageConfirmations'
+import { ConfirmationParam } from '../hooks/useMessageConfirmations'
 import validatorsCache from '../services/ValidatorsCache'
 import { CACHE_KEY_FAILED, CACHE_KEY_SUCCESS, VALIDATOR_CONFIRMATION_STATUS } from '../config/constants'
 import { APIPendingTransaction, APITransaction, GetTransactionParams, GetPendingTransactionParams } from './explorer'
 import { homeBlockNumberProvider } from '../services/BlockNumberProvider'
+import { getAffirmationsSigned, getMessagesSigned } from './contract'
 
 export const getValidatorConfirmation = (
   web3: Web3,
   hashMsg: string,
   bridgeContract: Contract,
-  confirmationContractMethod: Function
+  fromHome: boolean
-) => async (validator: string): Promise<BasicConfirmationParam> => {
+) => async (validator: string): Promise<ConfirmationParam> => {
   const hashSenderMsg = web3.utils.soliditySha3Raw(validator, hashMsg)
 
-  const signatureFromCache = validatorsCache.get(hashSenderMsg)
+  const fromCache = validatorsCache.getData(hashSenderMsg)
-  if (signatureFromCache) {
+  if (fromCache) {
-    return {
+    return fromCache
-      validator,
-      status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS
-    }
   }
 
+  const confirmationContractMethod = fromHome ? getMessagesSigned : getAffirmationsSigned
   const confirmed = await confirmationContractMethod(bridgeContract, hashSenderMsg)
-  const status = confirmed ? VALIDATOR_CONFIRMATION_STATUS.SUCCESS : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
 
   // If validator confirmed signature, we cache the result to avoid doing future requests for a result that won't change
   if (confirmed) {
-    validatorsCache.set(hashSenderMsg, confirmed)
+    const confirmation: ConfirmationParam = {
+      status: VALIDATOR_CONFIRMATION_STATUS.SUCCESS,
+      validator,
+      timestamp: 0,
+      txHash: ''
+    }
+    validatorsCache.setData(hashSenderMsg, confirmation)
+    return confirmation
   }
 
   return {
+    status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
     validator,
-    status
+    timestamp: 0,
+    txHash: ''
   }
 }
 
@@ -43,7 +50,7 @@ export const getSuccessExecutionTransaction = (
   messageData: string,
   startBlock: number,
   getSuccessTransactions: (args: GetTransactionParams) => Promise<APITransaction[]>
-) => async (validatorData: BasicConfirmationParam): Promise<ConfirmationParam> => {
+) => async (validatorData: ConfirmationParam): Promise<ConfirmationParam> => {
   const { validator } = validatorData
   const validatorCacheKey = `${CACHE_KEY_SUCCESS}${validatorData.validator}-${messageData}`
   const fromCache = validatorsCache.getData(validatorCacheKey)
@@ -87,11 +94,12 @@ export const getSuccessExecutionTransaction = (
 }
 
 export const getValidatorFailedTransaction = (
+  web3: Web3,
   bridgeContract: Contract,
   messageData: string,
   startBlock: number,
   getFailedTransactions: (args: GetTransactionParams) => Promise<APITransaction[]>
-) => async (validatorData: BasicConfirmationParam): Promise<ConfirmationParam> => {
+) => async (validatorData: ConfirmationParam): Promise<ConfirmationParam> => {
   const validatorCacheKey = `${CACHE_KEY_FAILED}${validatorData.validator}-${messageData}`
   const failedFromCache = validatorsCache.getData(validatorCacheKey)
 
@@ -106,30 +114,33 @@ export const getValidatorFailedTransaction = (
     startBlock,
     endBlock: homeBlockNumberProvider.get() || 0
   })
-  const newStatus =
-    failedTransactions.length > 0 ? VALIDATOR_CONFIRMATION_STATUS.FAILED : VALIDATOR_CONFIRMATION_STATUS.UNDEFINED
-
-  let txHashTimestamp = 0
-  let txHash = ''
   // If validator signature failed, we cache the result to avoid doing future requests for a result that won't change
   if (failedTransactions.length > 0) {
     const failedTx = failedTransactions[0]
-    txHashTimestamp = parseInt(failedTx.timeStamp)
+    const confirmation: ConfirmationParam = {
-    txHash = failedTx.hash
+      status: VALIDATOR_CONFIRMATION_STATUS.FAILED,
-
-    validatorsCache.setData(validatorCacheKey, {
       validator: validatorData.validator,
-      status: newStatus,
+      txHash: failedTx.hash,
-      txHash,
+      timestamp: parseInt(failedTx.timeStamp)
-      timestamp: txHashTimestamp
+    }
-    })
+
+    if (failedTx.input && failedTx.input.length > 10) {
+      try {
+        const res = web3.eth.abi.decodeParameters(['bytes', 'bytes'], `0x${failedTx.input.slice(10)}`)
+        confirmation.signature = res[0]
+        confirmation.status = VALIDATOR_CONFIRMATION_STATUS.FAILED_VALID
+        console.log(`Adding manual signature from failed message from ${validatorData.validator}`)
+      } catch {}
+    }
+    validatorsCache.setData(validatorCacheKey, confirmation)
+    return confirmation
   }
 
   return {
+    status: VALIDATOR_CONFIRMATION_STATUS.UNDEFINED,
     validator: validatorData.validator,
-    status: newStatus,
+    txHash: '',
-    txHash,
+    timestamp: 0
-    timestamp: txHashTimestamp
   }
 }
 
@@ -137,7 +148,7 @@ export const getValidatorPendingTransaction = (
   bridgeContract: Contract,
   messageData: string,
   getPendingTransactions: (args: GetPendingTransactionParams) => Promise<APIPendingTransaction[]>
-) => async (validatorData: BasicConfirmationParam): Promise<ConfirmationParam> => {
+) => async (validatorData: ConfirmationParam): Promise<ConfirmationParam> => {
   const failedTransactions = await getPendingTransactions({
     account: validatorData.validator,
     to: bridgeContract.options.address,

alm/src/utils/web3.ts

@@ -10,6 +10,37 @@ import { SnapshotProvider } from '../services/SnapshotProvider'
 export interface MessageObject {
   id: string
   data: string
+  sender?: string
+  executor?: string
+  obToken?: string
+  obReceiver?: string
+}
+
+export interface WarnRule {
+  message: string
+  sender?: string
+  executor?: string
+  obToken?: string
+  obReceiver?: string
+}
+
+export const matchesRule = (rule: WarnRule, msg: MessageObject) => {
+  if (!msg.executor || !msg.sender) {
+    return false
+  }
+  if (!!rule.executor && rule.executor.toLowerCase() !== msg.executor.toLowerCase()) {
+    return false
+  }
+  if (!!rule.sender && rule.sender.toLowerCase() !== msg.sender.toLowerCase()) {
+    return false
+  }
+  if (!!rule.obToken && (!msg.obToken || rule.obToken.toLowerCase() !== msg.obToken.toLowerCase())) {
+    return false
+  }
+  if (!!rule.obReceiver && (!msg.obReceiver || rule.obReceiver.toLowerCase() !== msg.obReceiver.toLowerCase())) {
+    return false
+  }
+  return true
 }
 
 const rawGetWeb3 = (url: string) => new Web3(new Web3.providers.HttpProvider(url))
@@ -26,15 +57,33 @@ export const filterEventsByAbi = (
   const eventHash = web3.eth.abi.encodeEventSignature(eventAbi)
   const events = txReceipt.logs.filter(e => e.address === bridgeAddress && e.topics[0] === eventHash)
 
+  if (!eventAbi || !eventAbi.inputs || !eventAbi.inputs.length) {
+    return []
+  }
+  const inputs = eventAbi.inputs
   return events.map(e => {
-    let decodedLogs: { [p: string]: string } = {
+    const { messageId, encodedData } = web3.eth.abi.decodeLog(inputs, e.data, [e.topics[1]])
-      messageId: '',
+    let sender, executor, obToken, obReceiver
-      encodedData: ''
+    if (encodedData.length >= 160) {
+      sender = `0x${encodedData.slice(66, 106)}`
+      executor = `0x${encodedData.slice(106, 146)}`
+      const dataOffset =
+        160 + (parseInt(encodedData.slice(154, 156), 16) + parseInt(encodedData.slice(156, 158), 16)) * 2 + 8
+      if (encodedData.length >= dataOffset + 64) {
+        obToken = `0x${encodedData.slice(dataOffset + 24, dataOffset + 64)}`
+      }
+      if (encodedData.length >= dataOffset + 128) {
+        obReceiver = `0x${encodedData.slice(dataOffset + 88, dataOffset + 128)}`
+      }
     }
-    if (eventAbi && eventAbi.inputs && eventAbi.inputs.length) {
+    return {
-      decodedLogs = web3.eth.abi.decodeLog(eventAbi.inputs, e.data, [e.topics[1]])
+      id: messageId || '',
+      data: encodedData || '',
+      sender,
+      executor,
+      obToken,
+      obReceiver
     }
-    return { id: decodedLogs.messageId, data: decodedLogs.encodedData }
   })
 }
 

commons/package.json

@@ -8,6 +8,7 @@
     "test": "NODE_ENV=test mocha"
   },
   "dependencies": {
+    "@mycrypto/gas-estimation": "^1.1.0",
     "gas-price-oracle": "^0.1.5",
     "web3-utils": "^1.3.0",
     "node-fetch": "^2.1.2"

commons/utils.js

@@ -1,5 +1,6 @@
 const { toWei, toBN, BN } = require('web3-utils')
 const { GasPriceOracle } = require('gas-price-oracle')
+const { estimateFees } = require('@mycrypto/gas-estimation')
 const fetch = require('node-fetch')
 const { BRIDGE_MODES } = require('./constants')
 const { REWARDABLE_VALIDATORS_ABI } = require('./abis')
@@ -176,12 +177,20 @@ const gasPriceWithinLimits = (gasPrice, limits) => {
 const normalizeGasPrice = (oracleGasPrice, factor, limits = null) => {
   let gasPrice = oracleGasPrice * factor
   gasPrice = gasPriceWithinLimits(gasPrice, limits)
-  return toBN(toWei(gasPrice.toFixed(2).toString(), 'gwei'))
+  return toWei(gasPrice.toFixed(2).toString(), 'gwei')
 }
 
-const gasPriceFromSupplier = async (url, options = {}) => {
+const gasPriceFromSupplier = async (web3, url, options = {}) => {
   try {
     let json
+    if (url === 'eip1559-gas-estimation') {
+      const { maxFeePerGas, maxPriorityFeePerGas } = await estimateFees(web3)
+      const res = { maxFeePerGas: maxFeePerGas.toString(10), maxPriorityFeePerGas: maxPriorityFeePerGas.toString(10) }
+      options.logger &&
+        options.logger.debug &&
+        options.logger.debug(res, 'Gas price updated using eip1559-gas-estimation')
+      return res
+    }
     if (url === 'gas-price-oracle') {
       json = await gasPriceOracle.fetchGasPricesOffChain()
     } else if (url) {
@@ -205,7 +214,7 @@ const gasPriceFromSupplier = async (url, options = {}) => {
       options.logger.debug &&
       options.logger.debug({ oracleGasPrice, normalizedGasPrice }, 'Gas price updated using the API')
 
-    return normalizedGasPrice
+    return { gasPrice: normalizedGasPrice }
   } catch (e) {
     options.logger && options.logger.error && options.logger.error(`Gas Price API is not available. ${e.message}`)
   }
@@ -214,11 +223,11 @@ const gasPriceFromSupplier = async (url, options = {}) => {
 
 const gasPriceFromContract = async (bridgeContract, options = {}) => {
   try {
-    const gasPrice = await bridgeContract.methods.gasPrice().call()
+    const gasPrice = (await bridgeContract.methods.gasPrice().call()).toString()
     options.logger &&
       options.logger.debug &&
       options.logger.debug({ gasPrice }, 'Gas price updated using the contracts')
-    return gasPrice
+    return { gasPrice }
   } catch (e) {
     options.logger &&
       options.logger.error &&

deployment-e2e/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.7
+FROM python:3.7-stretch
 RUN curl -fsSL https://get.docker.com | sh
-RUN pip3 install docker molecule[docker,ansible] pytest pytest-testinfra flake8
+RUN pip3 install docker molecule==2.22rc1 molecule[docker] flake8
 WORKDIR mono/deployment-e2e

deployment-e2e/molecule/monitor/molecule.yml

@@ -3,6 +3,12 @@ dependency:
   name: galaxy
 driver:
   name: docker
+lint:
+  name: yamllint
+  enabled: True
+  options:
+    config-data:
+      ignore: ../../hosts.yml
 platforms:
   - name: monitor-host
     groups:
@@ -16,6 +22,11 @@ platforms:
       - /var/run/docker.sock:/var/run/docker.sock
 provisioner:
   name: ansible
+  lint:
+    name: ansible-lint
+    enabled: True
+    options:
+      r: ["bug"]
   playbooks:
     prepare: ../prepare.yml
     converge: ./converge.yml
@@ -26,11 +37,14 @@ provisioner:
         syslog_server_port: "udp://127.0.0.1:514"
 verifier:
   name: testinfra
+  lint:
+    name: flake8
   additional_files_or_dirs:
     - ../../tests/*
 scenario:
   name: monitor
   test_sequence:
+    - lint
     - cleanup
     - destroy
     - dependency

deployment-e2e/molecule/multiple/molecule.yml

@@ -3,6 +3,12 @@ dependency:
   name: galaxy
 driver:
   name: docker
+lint:
+  name: yamllint
+  enabled: True
+  options:
+    config-data:
+      ignore: ../../hosts.yml
 platforms:
   - name: multiple-host
     groups:
@@ -17,6 +23,11 @@ platforms:
       - /var/run/docker.sock:/var/run/docker.sock
 provisioner:
   name: ansible
+  lint:
+    name: ansible-lint
+    enabled: True
+    options:
+      r: ["bug"]
   playbooks:
     prepare: ../prepare.yml
     converge: ../monitor/converge.yml
@@ -28,11 +39,14 @@ provisioner:
         syslog_server_port: "udp://127.0.0.1:514"
 verifier:
   name: testinfra
+  lint:
+    name: flake8
   additional_files_or_dirs:
     - ../../tests/*
 scenario:
   name: multiple
   test_sequence:
+    - lint
     - cleanup
     - destroy
     - dependency

deployment-e2e/molecule/oracle/molecule.yml

@@ -3,12 +3,18 @@ dependency:
   name: galaxy
 driver:
   name: docker
+lint:
+  name: yamllint
+  enabled: True
+  options:
+    config-data:
+      ignore: ../../hosts.yml
 platforms:
   - name: oracle-host
     groups:
       - example
     children:
-      - oracle_swarm
+      - oracle
     image: ubuntu:16.04
     privileged: true
     network_mode: host
@@ -16,22 +22,29 @@ platforms:
       - /var/run/docker.sock:/var/run/docker.sock
 provisioner:
   name: ansible
+  lint:
+    name: ansible-lint
+    enabled: True
+    options:
+      r: ["bug"]
   playbooks:
     prepare: ../prepare.yml
     converge: ../../../deployment/site.yml
   inventory:
     host_vars:
       oracle-host:
-        ORACLE_VALIDATOR_KEYSTORE_PATH: "../../../e2e-commons/keystore.json"
+        ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY: "6c48435bd464a53ed66ed62127c4dba8af75cf1a99a8ebe2680599948fbfbc6d"
-        ORACLE_VALIDATOR_KEYSTORE_PASSWORD: "12345678"
         syslog_server_port: "udp://127.0.0.1:514"
 verifier:
   name: testinfra
+  lint:
+    name: flake8
   additional_files_or_dirs:
     - ../../tests/*
 scenario:
   name: oracle
   test_sequence:
+    - lint
     - cleanup
     - destroy
     - dependency

deployment-e2e/molecule/oracle/tests/test_oracle.py

@@ -3,21 +3,22 @@ import pytest
 import testinfra.utils.ansible_runner
 
 testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
-    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('oracle_swarm')
+    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('oracle')
 
 
 @pytest.mark.parametrize("name", [
-    ("oracle_rabbit"),
+    ("oracle_rabbit_1"),
-    ("oracle_redis"),
+    ("oracle_redis_1"),
-    ("oracle_bridge_request"),
+    ("oracle_bridge_request_1"),
-    ("oracle_bridge_collected"),
+    ("oracle_bridge_collected_1"),
-    ("oracle_bridge_affirmation"),
+    ("oracle_bridge_affirmation_1"),
-    ("oracle_bridge_senderhome"),
+    ("oracle_bridge_senderhome_1"),
-    ("oracle_bridge_senderforeign"),
+    ("oracle_bridge_senderforeign_1"),
-    ("oracle_bridge_shutdown"),
+    ("oracle_bridge_shutdown_1"),
 ])
 def test_docker_containers(host, name):
-    assert host.docker(name) is not None
+    container = host.docker(name)
+    assert container.is_running
 
 
 @pytest.mark.parametrize("service", [

deployment-e2e/molecule/ultimate-amb/molecule.yml

@@ -25,6 +25,8 @@ provisioner:
         ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY: "8e829f695aed89a154550f30262f1529582cc49dc30eff74a6b491359e0230f9"
 verifier:
   name: testinfra
+  lint:
+    name: flake8
 scenario:
   name: ultimate-amb
   test_sequence:

deployment-e2e/molecule/ultimate-erc-to-native/molecule.yml

@@ -27,6 +27,8 @@ provisioner:
         ORACLE_FOREIGN_START_BLOCK: 1
 verifier:
   name: testinfra
+  lint:
+    name: flake8
 scenario:
   name: ultimate-erc-to-native
   test_sequence:

deployment/hosts.yml.example

@@ -7,13 +7,6 @@ sokol-kovan:
           ansible_user: ubuntu
           ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
           #syslog_server_port: "udp://127.0.0.1:514"
-    oracle_swarm:
-      hosts:
-        127.0.0.1:
-          ansible_user: ubuntu
-          ORACLE_VALIDATOR_KEYSTORE_PATH: "/path/to/keystore.json"
-          ORACLE_VALIDATOR_KEYSTORE_PASSWORD: "12345678"
-          #syslog_server_port: "udp://127.0.0.1:514"
     monitor:
       hosts:
         127.0.0.1:

deployment/requirements.txt

@@ -0,0 +1,4 @@
+# pre-release because it contains "CI Fixes for ansible 2.8"
+molecule==2.22rc1
+docker
+flake8

deployment/roles/common/files/daemon.json

@@ -1,4 +1,4 @@
 {
-    "live-restore": false,
+    "live-restore": true,
     "no-new-privileges": true
 }

deployment/roles/common/tasks/dependencies.yml

@@ -31,7 +31,6 @@
     owner: "root"
     group: "root"
     mode: "0755"
-  when: skip_compose is undefined
 
 - name: Upgrade pip version
   shell: pip3 install --upgrade pip==19.3.1
@@ -46,9 +45,6 @@
     group: docker
     createhome: yes
 
-- name: reset ssh connection to allow user changes to affect ansible user
-  meta: reset_connection
-
 - name: Install auditd
   apt:
     name: auditd

deployment/roles/oracle_swarm/defaults/main.yml

@@ -1,9 +0,0 @@
----
-bridge_path: "/home/{{ compose_service_user }}/bridge"
-bridge_data_path: "/home/{{ compose_service_user }}/bridge_data"
-ORACLE_ALLOW_HTTP_FOR_RPC: no
-ORACLE_QUEUE_URL: amqp://rabbit
-ORACLE_REDIS_URL: redis://redis
-keyfile_path: "/root/.key"
-keystore_path: "/root/.keystore.json"
-oracle_image: poanetwork/tokenbridge-oracle:latest

deployment/roles/oracle_swarm/meta/main.yml

@@ -1,3 +0,0 @@
----
-dependencies:
-  - { role: common, skip_repo: true, skip_compose: true }

deployment/roles/oracle_swarm/tasks/jumpbox.yml

@@ -1,8 +0,0 @@
----
-- name: Pull the containers images
-  community.docker.docker_image:
-    name: "{{ oracle_image }}"
-    source: pull
-  when: skip_pull is undefined
-  vars:
-    ansible_python_interpreter: /usr/bin/python3

deployment/roles/oracle_swarm/tasks/logging.yml

@@ -1,41 +0,0 @@
----
-- name: Set the oracle's containers local logs configuration file
-  template:
-    src: 31-oracle-docker.conf.j2
-    dest: /etc/rsyslog.d/31-oracle-docker.conf
-    owner: root
-    group: root
-    mode: 0644
-
-- name: Set the redis container local logs configuration file
-  template:
-    src: 32-redis-docker.conf.j2
-    dest: /etc/rsyslog.d/32-redis-docker.conf
-    owner: root
-    group: root
-    mode: 0644
-
-- name: Set the rabbit MQ container local logs configuration file
-  template:
-    src: 33-rabbit-docker.conf.j2
-    dest: /etc/rsyslog.d/33-rabbit-docker.conf
-    owner: root
-    group: root
-    mode: 0644
-
-- name: Set the log configuration file to send container logs to remote server
-  template:
-    src: 36-oracle-remote-logging.conf.j2
-    dest: /etc/rsyslog.d/36-oracle-remote-logging.conf
-    owner: root
-    group: root
-    mode: 0644
-  when: syslog_server_port is defined
-
-- name: Discarding unwanted messages in rsyslog
-  blockinfile:
-    path: /etc/rsyslog.conf
-    insertbefore: "# Where to place spool and state files"
-    marker: "#{mark} add string to discarding unwanted messages"
-    content: ':msg, contains, "ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY" ~'
-  notify: restart rsyslog

deployment/roles/oracle_swarm/tasks/main.yml

@@ -1,6 +0,0 @@
----
-- include_tasks: pre_config.yml
-- include_tasks: logging.yml
-- include_tasks: jumpbox.yml
-- include_tasks: post_config.yml
-- include_tasks: servinstall.yml

deployment/roles/oracle_swarm/tasks/post_config.yml

@@ -1,46 +0,0 @@
----
-- name: Get blocks
-  become_user: "{{ compose_service_user }}"
-  shell: docker run --env-file .env --rm {{ oracle_image }} scripts/getValidatorStartBlocks.js
-  args:
-    chdir: "{{ bridge_path }}/oracle"
-  register: BLOCKS
-  when: (ORACLE_HOME_START_BLOCK is not defined) or (ORACLE_FOREIGN_START_BLOCK is not defined)
-
-- name: Write blocks
-  blockinfile:
-    path: "{{ bridge_path }}/oracle/.env"
-    marker: "## {mark} Calculated by scripts/getValidatorStartBlocks.js"
-    block: |
-      ORACLE_HOME_START_BLOCK={{ (BLOCKS.stdout | from_json).homeStartBlock }}
-      ORACLE_FOREIGN_START_BLOCK={{ (BLOCKS.stdout | from_json).foreignStartBlock }}
-  when: (ORACLE_HOME_START_BLOCK is not defined) or (ORACLE_FOREIGN_START_BLOCK is not defined)
-
-- name: Copy keystore file
-  copy:
-    src: "{{ ORACLE_VALIDATOR_KEYSTORE_PATH }}"
-    dest: "{{ keystore_path }}"
-    owner: root
-    group: root
-    mode: 0600
-
-- name: Create swarm secret
-  community.docker.docker_secret:
-    name: oracle_keystore
-    state: present
-    data_src: "{{ keystore_path }}"
-  vars:
-    ansible_python_interpreter: /usr/bin/python3
-
-- name: Remove unencrypted keystore file
-  file:
-    path: "{{ keystore_path }}"
-    state: absent
-
-- name: Install .key config
-  template:
-    src: key.j2
-    dest: "{{ keyfile_path }}"
-    owner: root
-    group: root
-    mode: 0600

deployment/roles/oracle_swarm/tasks/pre_config.yml

@@ -1,47 +0,0 @@
----
-- name: Init docker swarm
-  community.docker.docker_swarm:
-    state: present
-    autolock_managers: yes
-    listen_addr: 127.0.0.1:2377
-  vars:
-    ansible_python_interpreter: /usr/bin/python3
-
-- name: Get unlock token
-  community.docker.docker_swarm_info:
-    unlock_key: yes
-  register: result
-  vars:
-    ansible_python_interpreter: /usr/bin/python3
-
-- name: Print unlock token
-  debug:
-    var: result.swarm_unlock_key
-
-- name: Create oracle directory
-  file:
-    path: "{{ bridge_path }}/oracle"
-    state: directory
-    mode: '0755'
-
-- name: Create rabbitmq directory
-  file:
-    path: "{{ bridge_data_path }}/{{ item }}"
-    state: directory
-    mode: '0775'
-  loop:
-    - rabbitmq
-    - redis
-
-- name: Install .env config
-  template:
-    src: .env.j2
-    dest: "{{ bridge_path }}/oracle/.env"
-    owner: "{{ compose_service_user }}"
-    mode: '0640'
-
-- name: Install docker-compose file
-  template:
-    src: docker-compose.yml.j2
-    dest: "{{ bridge_path }}/oracle/docker-compose.yml"
-    mode: '0755'

deployment/roles/oracle_swarm/tasks/servinstall.yml

@@ -1,19 +0,0 @@
-# This role creates a poabridge service which is designed to manage docker-compose bridge deployment.
-# /etc/init.d/poabridge start, status, stop, restart - does what the services usually do in such cases.
----
-- name: "Set poabridge service"
-  template:
-    src: poabridge.j2
-    dest: "/etc/init.d/poabridge"
-    owner: root
-    mode: 755
-
-- name: "Enable the service"
-  service:
-    name: "poabridge"
-    state: started
-    enabled: yes
-    use: service
-
-- name: Start the service
-  shell: service poabridge start

deployment/roles/oracle_swarm/templates/.env.j2

@@ -1,86 +0,0 @@
-## General settings
-ORACLE_BRIDGE_MODE={{ ORACLE_BRIDGE_MODE }}
-{% if ORACLE_LOG_LEVEL | default('') != '' %}
-ORACLE_LOG_LEVEL={{ ORACLE_LOG_LEVEL }}
-{% endif %}
-
-## Home contract
-COMMON_HOME_RPC_URL={{ COMMON_HOME_RPC_URL }}
-COMMON_HOME_BRIDGE_ADDRESS={{ COMMON_HOME_BRIDGE_ADDRESS }}
-ORACLE_HOME_RPC_POLLING_INTERVAL={{ ORACLE_HOME_RPC_POLLING_INTERVAL }}
-
-## Foreign contract
-COMMON_FOREIGN_RPC_URL={{ COMMON_FOREIGN_RPC_URL }}
-{% if ORACLE_FOREIGN_ARCHIVE_RPC_URL | default('') != '' %}
-ORACLE_FOREIGN_ARCHIVE_RPC_URL={{ ORACLE_FOREIGN_ARCHIVE_RPC_URL }}
-{% endif %}
-COMMON_FOREIGN_BRIDGE_ADDRESS={{ COMMON_FOREIGN_BRIDGE_ADDRESS }}
-ORACLE_FOREIGN_RPC_POLLING_INTERVAL={{ ORACLE_FOREIGN_RPC_POLLING_INTERVAL }}
-
-{% if ORACLE_TX_REDUNDANCY | default('') != '' %}
-ORACLE_TX_REDUNDANCY={{ ORACLE_TX_REDUNDANCY }}
-{% endif %}
-
-## Gasprice
-{% if COMMON_HOME_GAS_PRICE_SUPPLIER_URL | default('') != '' %}
-COMMON_HOME_GAS_PRICE_SUPPLIER_URL={{ COMMON_HOME_GAS_PRICE_SUPPLIER_URL }}
-{% endif %}
-{% if COMMON_HOME_GAS_PRICE_SPEED_TYPE | default('') != '' %}
-COMMON_HOME_GAS_PRICE_SPEED_TYPE={{ COMMON_HOME_GAS_PRICE_SPEED_TYPE }}
-{% endif %}
-COMMON_HOME_GAS_PRICE_FALLBACK={{ COMMON_HOME_GAS_PRICE_FALLBACK }}
-{% if ORACLE_HOME_GAS_PRICE_UPDATE_INTERVAL | default('') != '' %}
-ORACLE_HOME_GAS_PRICE_UPDATE_INTERVAL={{ ORACLE_HOME_GAS_PRICE_UPDATE_INTERVAL }}
-{% endif %}
-{% if COMMON_HOME_GAS_PRICE_FACTOR | default('') != '' %}
-COMMON_HOME_GAS_PRICE_FACTOR={{ COMMON_HOME_GAS_PRICE_FACTOR }}
-{% endif %}
-
-{% if COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL | default('') != '' %}
-COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL={{ COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL }}
-{% endif %}
-{% if COMMON_FOREIGN_GAS_PRICE_SPEED_TYPE | default('') != '' %}
-COMMON_FOREIGN_GAS_PRICE_SPEED_TYPE={{ COMMON_FOREIGN_GAS_PRICE_SPEED_TYPE }}
-{% endif %}
-COMMON_FOREIGN_GAS_PRICE_FALLBACK={{ COMMON_FOREIGN_GAS_PRICE_FALLBACK }}
-{% if ORACLE_FOREIGN_GAS_PRICE_UPDATE_INTERVAL | default('') != '' %}
-ORACLE_FOREIGN_GAS_PRICE_UPDATE_INTERVAL={{ ORACLE_FOREIGN_GAS_PRICE_UPDATE_INTERVAL }}
-{% endif %}
-{% if COMMON_FOREIGN_GAS_PRICE_FACTOR | default('') != '' %}
-COMMON_FOREIGN_GAS_PRICE_FACTOR={{ COMMON_FOREIGN_GAS_PRICE_FACTOR }}
-{% endif %}
-
-## Transport configuration
-ORACLE_ALLOW_HTTP_FOR_RPC={{ "yes" if ORACLE_ALLOW_HTTP_FOR_RPC else "no" }}
-ORACLE_QUEUE_URL={{ ORACLE_QUEUE_URL }}
-ORACLE_REDIS_URL={{ ORACLE_REDIS_URL }}
-{% if ORACLE_FOREIGN_TX_RESEND_INTERVAL | default('') != '' %}
-ORACLE_FOREIGN_TX_RESEND_INTERVAL={{ ORACLE_FOREIGN_TX_RESEND_INTERVAL }}
-{% endif %}
-{% if ORACLE_HOME_TX_RESEND_INTERVAL | default('') != '' %}
-ORACLE_HOME_TX_RESEND_INTERVAL={{ ORACLE_HOME_TX_RESEND_INTERVAL }}
-{% endif %}
-
-## Emergency shutdown configuration
-{% if ORACLE_SHUTDOWN_SERVICE_URL | default('') != '' %}
-ORACLE_SHUTDOWN_SERVICE_URL={{ ORACLE_SHUTDOWN_SERVICE_URL }}
-{% endif %}
-{% if ORACLE_SHUTDOWN_SERVICE_POLLING_INTERVAL | default('') != '' %}
-ORACLE_SHUTDOWN_SERVICE_POLLING_INTERVAL={{ ORACLE_SHUTDOWN_SERVICE_POLLING_INTERVAL }}
-{% endif %}
-{% if ORACLE_SIDE_RPC_URL | default('') != '' %}
-ORACLE_SIDE_RPC_URL={{ ORACLE_SIDE_RPC_URL }}
-{% endif %}
-{% if ORACLE_SHUTDOWN_CONTRACT_ADDRESS | default('') != '' %}
-ORACLE_SHUTDOWN_CONTRACT_ADDRESS={{ ORACLE_SHUTDOWN_CONTRACT_ADDRESS }}
-{% endif %}
-{% if ORACLE_SHUTDOWN_CONTRACT_METHOD | default('') != '' %}
-ORACLE_SHUTDOWN_CONTRACT_METHOD={{ ORACLE_SHUTDOWN_CONTRACT_METHOD }}
-{% endif %}
-
-{% if ORACLE_HOME_START_BLOCK | default('') != '' %}
-ORACLE_HOME_START_BLOCK={{ ORACLE_HOME_START_BLOCK }}
-{% endif %}
-{% if ORACLE_FOREIGN_START_BLOCK | default('') != '' %}
-ORACLE_FOREIGN_START_BLOCK={{ ORACLE_FOREIGN_START_BLOCK }}
-{% endif %}

deployment/roles/oracle_swarm/templates/31-oracle-docker.conf.j2

@@ -1,11 +0,0 @@
-$FileCreateMode 0644
-template(name="DockerLogFileName_Oracle" type="list") {
-   constant(value="/var/log/docker/")
-   property(name="syslogtag" securepath="replace" regex.type="ERE" regex.submatch="1" regex.expression="bridge_(.*)\\/[a-zA-Z0-9]+\\[")
-   constant(value="/docker.log")
-}
-
-if $programname startswith 'oracle_bridge_' then \
-  ?DockerLogFileName_Oracle
-
-$FileCreateMode 0600

deployment/roles/oracle_swarm/templates/32-redis-docker.conf.j2

@@ -1,11 +0,0 @@
-$FileCreateMode 0644
-template(name="DockerLogFileName_Redis" type="list") {
-   constant(value="/var/log/docker/")
-   property(name="syslogtag" securepath="replace" regex.type="ERE" regex.submatch="1" regex.expression="oracle_(.*redis.*)\\/[a-zA-Z0-9]+\\[")
-   constant(value="/docker.log")
-}
-
-if $programname contains 'oracle' and $programname contains 'redis' then \
-  ?DockerLogFileName_Redis
-
-$FileCreateMode 0600

deployment/roles/oracle_swarm/templates/33-rabbit-docker.conf.j2

@@ -1,11 +0,0 @@
-$FileCreateMode 0644
-template(name="DockerLogFileName_Rabbit" type="list") {
-   constant(value="/var/log/docker/")
-   property(name="syslogtag" securepath="replace" regex.type="ERE" regex.submatch="1" regex.expression="oracle_(.*rabbit.*)\\/[a-zA-Z0-9]+\\[")
-   constant(value="/docker.log")
-}
-
-if $programname contains 'oracle' and $programname contains 'rabbit' then \
-  ?DockerLogFileName_Rabbit
-
-$FileCreateMode 0600

deployment/roles/oracle_swarm/templates/36-oracle-remote-logging.conf.j2

@@ -1,15 +0,0 @@
-if $programname startswith 'oracle_bridge_' then {
-action(
-   type="omfwd"
-   protocol="{{ syslog_server_port.split(":")[0] }}"
-   target="{{ (syslog_server_port.split(":")[1])[2:] }}"
-   port="{{ syslog_server_port.split(":")[2] }}"
-   template="RemoteForwardFormat"
-   queue.SpoolDirectory="/var/spool/rsyslog"
-   queue.FileName="remote"
-   queue.MaxDiskSpace="1g"
-   queue.SaveOnShutdown="on"
-   queue.Type="LinkedList"
-   ResendLastMSGOnReconnect="on"
-   )
-}

deployment/roles/oracle_swarm/templates/docker-compose.yml.j2

@@ -1,144 +0,0 @@
-version: '3.9'
-x-deploy: &x-deploy
-  resources:
-    limits:
-      cpus: '0.3'
-      memory: 500M
-    reservations:
-      memory: 100M
-x-keystore-access: &x-keystore-access
-  environment:
-    ORACLE_VALIDATOR_KEYSTORE_PATH: /run/secrets/oracle_keystore
-    ORACLE_VALIDATOR_KEYSTORE_PASSWORD:
-  secrets:
-    - oracle_keystore
-x-logging: &x-logging
-  driver: 'syslog'
-  options: {tag: '{{ '{{.Name}}/{{.ID}}' }}' }
-services:
-  rabbit:
-    image: rabbitmq:3
-    hostname: rabbit
-    deploy: *x-deploy
-    logging: *x-logging
-    environment: [ 'RABBITMQ_NODENAME=node@rabbit' ]
-    networks:
-      - net_rabbit_bridge_request
-      - net_rabbit_bridge_collected
-      - net_rabbit_bridge_affirmation
-      - net_rabbit_bridge_senderhome
-      - net_rabbit_bridge_senderforeign
-    volumes: [ '{{ bridge_data_path }}/rabbitmq:/var/lib/rabbitmq/mnesia' ]
-  redis:
-    image: redis:4
-    hostname: redis
-    deploy: *x-deploy
-    logging: *x-logging
-    command: [ redis-server, --appendonly, 'yes' ]
-    networks:
-      - net_db_bridge_request
-      - net_db_bridge_collected
-      - net_db_bridge_affirmation
-      - net_db_bridge_senderhome
-      - net_db_bridge_senderforeign
-      - net_db_bridge_shutdown
-    volumes: [ '{{ bridge_data_path }}/redis:/data' ]
-  bridge_request:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    logging: *x-logging
-    env_file: ./.env
-    <<: *x-keystore-access
-    entrypoint: yarn watcher:signature-request
-    networks:
-      - net_db_bridge_request
-      - net_rabbit_bridge_request
-  bridge_collected:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    env_file: ./.env
-    entrypoint: yarn watcher:collected-signatures
-    networks:
-      - net_db_bridge_collected
-      - net_rabbit_bridge_collected
-  bridge_affirmation:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    logging: *x-logging
-    env_file: ./.env
-    entrypoint: yarn watcher:affirmation-request
-    networks:
-      - net_db_bridge_affirmation
-      - net_rabbit_bridge_affirmation
-  bridge_senderhome:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    env_file: ./.env
-    <<: *x-keystore-access
-    entrypoint: yarn sender:home
-    networks:
-      - net_db_bridge_senderhome
-      - net_rabbit_bridge_senderhome
-  bridge_senderforeign:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    logging: *x-logging
-    env_file: ./.env
-    <<: *x-keystore-access
-    entrypoint: yarn sender:foreign
-    networks:
-      - net_db_bridge_senderforeign
-      - net_rabbit_bridge_senderforeign
-  bridge_shutdown:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    env_file: ./.env
-    entrypoint: yarn manager:shutdown
-    networks:
-      - net_db_bridge_shutdown
-{% if ORACLE_BRIDGE_MODE == "ERC_TO_NATIVE" %}
-  bridge_transfer:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    logging: *x-logging
-    env_file: ./.env
-    entrypoint: yarn watcher:transfer
-    networks:
-      - net_db_bridge_transfer
-      - net_rabbit_bridge_transfer
-{% endif %}
-{% if ORACLE_BRIDGE_MODE == "ARBITRARY_MESSAGE" %}
-  bridge_information:
-    image: {{ oracle_image }}
-    deploy: *x-deploy
-    logging: *x-logging
-    env_file: ./.env
-    entrypoint: yarn watcher:information-request
-    networks:
-      - net_db_bridge_information
-      - net_rabbit_bridge_information
-{% endif %}
-networks:
-  net_db_bridge_request:
-  net_db_bridge_collected:
-  net_db_bridge_affirmation:
-  net_db_bridge_senderhome:
-  net_db_bridge_senderforeign:
-  net_db_bridge_shutdown:
-{% if ORACLE_BRIDGE_MODE == "ERC_TO_NATIVE" %}
-  net_db_bridge_transfer:
-  net_rabbit_bridge_transfer:
-{% endif %}
-{% if ORACLE_BRIDGE_MODE == "ARBITRARY_MESSAGE" %}
-  net_db_bridge_information:
-  net_rabbit_bridge_information:
-{% endif %}
-  net_rabbit_bridge_request:
-  net_rabbit_bridge_collected:
-  net_rabbit_bridge_affirmation:
-  net_rabbit_bridge_senderhome:
-  net_rabbit_bridge_senderforeign:
-
-secrets:
-  oracle_keystore:
-    external: true

deployment/roles/oracle_swarm/templates/key.j2

@@ -1,2 +0,0 @@
-## Validator-specific options
-ORACLE_VALIDATOR_KEYSTORE_PASSWORD={{ ORACLE_VALIDATOR_KEYSTORE_PASSWORD }}

deployment/roles/oracle_swarm/templates/poabridge.j2

@@ -1,66 +0,0 @@
-#! /bin/bash
-
-### BEGIN INIT INFO
-# Provides:          poabridge
-# Required-Start:    $remote_fs $syslog
-# Required-Stop:     $remote_fs $syslog
-# Default-Start:     2 3 4 5
-# Default-Stop:      0 1 6
-# Short-Description: Start daemon at boot time
-# Description:       Enable service provided by daemon.
-### END INIT INFO
-
-WORKDIR="{{ '/home/' + compose_service_user | default('poadocker') + '/' + bridge_path + '/oracle' if bridge_path[:1] != "/" else bridge_path + '/oracle' }}"
-
-#Getting path to private key file and variable name for parsing key file
-source {{ keyfile_path }}
-
-start(){
-    echo "Starting bridge.."
-    cd $WORKDIR
-    sudo -u "{{ compose_service_user }}" docker stack rm oracle
-    sudo -u "{{ compose_service_user }}" "ORACLE_VALIDATOR_KEYSTORE_PASSWORD=$ORACLE_VALIDATOR_KEYSTORE_PASSWORD" docker stack deploy oracle -c docker-compose.yml
-}
-
-stop(){
-    echo "Stopping bridge.."
-    cd $WORKDIR
-    sudo -u "{{ compose_service_user }}" docker stack rm oracle
-    sleep 2
-}
-
-status(){
-    echo "Bridge status:"
-    cd $WORKDIR
-    sudo -u "{{ compose_service_user }}" docker service ls
-}
-
-
-case "$1" in
-
-  start)
-      start
-    ;;
-
-  stop)
-      stop
-    ;;
-
-  status)
-      status
-    ;;
-
-  restart)
-    echo "Restarting bridge.."
-    stop
-    start
-    ;;
-
-  *)
-    echo $"Usage: $0 {start|stop|restart|status}"
-    exit 1
-    ;;
-
-esac
-
-exit 0

deployment/site.yml

@@ -4,11 +4,6 @@
   become: true
   roles:
     - { role: oracle }
-- name: Install Oracle as a Docker Swarm service
-  hosts: oracle_swarm
-  become: true
-  roles:
-    - { role: oracle_swarm }
 - name: Install Monitor
   hosts: monitor
   become: true

e2e-commons/components-envs/oracle-amb.env

@@ -23,3 +23,9 @@ ORACLE_HOME_START_BLOCK=1
 ORACLE_FOREIGN_START_BLOCK=1
 ORACLE_HOME_TO_FOREIGN_BLOCK_LIST=/mono/oracle/access-lists/block_list.txt
 ORACLE_FOREIGN_ARCHIVE_RPC_URL=http://parity2:8545
+ORACLE_HOME_EVENTS_REPROCESSING=false
+ORACLE_HOME_EVENTS_REPROCESSING_BATCH_SIZE=10
+ORACLE_HOME_EVENTS_REPROCESSING_BLOCK_DELAY=10
+ORACLE_FOREIGN_EVENTS_REPROCESSING=true
+ORACLE_FOREIGN_EVENTS_REPROCESSING_BATCH_SIZE=10
+ORACLE_FOREIGN_EVENTS_REPROCESSING_BLOCK_DELAY=10

e2e-commons/components-envs/oracle-erc20-native.env

@@ -22,3 +22,9 @@ ORACLE_ALLOW_HTTP_FOR_RPC=yes
 ORACLE_HOME_START_BLOCK=1
 ORACLE_FOREIGN_START_BLOCK=1
 ORACLE_HOME_TO_FOREIGN_BLOCK_LIST=/mono/oracle/access-lists/block_list.txt
+ORACLE_HOME_EVENTS_REPROCESSING=true
+ORACLE_HOME_EVENTS_REPROCESSING_BATCH_SIZE=10
+ORACLE_HOME_EVENTS_REPROCESSING_BLOCK_DELAY=10
+ORACLE_FOREIGN_EVENTS_REPROCESSING=true
+ORACLE_FOREIGN_EVENTS_REPROCESSING_BATCH_SIZE=10
+ORACLE_FOREIGN_EVENTS_REPROCESSING_BLOCK_DELAY=10

e2e-commons/keystore.json

@@ -1 +0,0 @@
-[{"version":3,"id":"e7e64a1b-5e61-4c17-a473-963d2bbb59e5","address":"d138a69eb2da1c3518e792737c820b23cce62e4b","crypto":{"ciphertext":"f6ddf0b2638fb9fd5777de2aa07937b5ee9bc17acc74c8e6e6580e2dfd0d3de6","cipherparams":{"iv":"bcdbc5af4582887e5cdcf264e8d5b80d"},"cipher":"aes-128-ctr","kdf":"scrypt","kdfparams":{"dklen":32,"salt":"f9e621918990e64e278e0fb8cf0343219e1cceaea8547d50fae452ad8f42f231","n":8192,"r":8,"p":1},"mac":"34149cd0b3ddea52588825d403fb75cfb8b864b616d455f75f2de001cc2601ed"}}]

oracle/README.md

@@ -1,10 +1,10 @@
 # POA TokenBridge / Oracle
-Oracle responsible for listening to bridge related events and authorizing asset transfers.
+An oracle responsible for listening to bridge related events and authorizing asset transfers.
 
 ## Overview
 Please refer to the [POA TokenBridge](../README.md) overview first of all.
 
-The Oracle is deployed on specified validator nodes (only nodes whose private keys correspond to addresses specified in the smart contracts) in the network. It connects to two chains via a Remote Procedure Call (RPC) and is responsible for:
+The oracle is deployed on specified validator nodes (only nodes whose private keys correspond to addresses specified in the smart contracts) in the network. It connects to two chains via a Remote Procedure Call (RPC) and is responsible for:
 - listening to events related to bridge contracts
 - sending transactions to authorize asset transfers
 
@@ -69,7 +69,7 @@ For more information on the Redis/RabbitMQ requirements, see [#90](/../../issues
    }
    ```
 
-## Install and configure the Oracle
+## Install and configure the oracle
 
 1. [Initialize](../README.md#initializing-the-monorepository) the monorepository.
 

oracle/config/base.config.js

@@ -7,16 +7,22 @@ const {
   HOME_AMB_ABI,
   FOREIGN_AMB_ABI
 } = require('../../commons')
-const { web3Home, web3Foreign } = require('../src/services/web3')
+const {
-const { add0xPrefix, privateKeyToAddress, loadKeystore } = require('../src/utils/utils')
+  web3Home,
+  web3Foreign,
+  web3HomeRedundant,
+  web3HomeFallback,
+  web3ForeignRedundant,
+  web3ForeignFallback,
+  web3ForeignArchive
+} = require('../src/services/web3')
+const { add0xPrefix, privateKeyToAddress } = require('../src/utils/utils')
 const { EXIT_CODES } = require('../src/utils/constants')
 
 const {
   ORACLE_BRIDGE_MODE,
   ORACLE_VALIDATOR_ADDRESS,
   ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY,
-  ORACLE_VALIDATOR_KEYSTORE_PATH,
-  ORACLE_VALIDATOR_KEYSTORE_PASSWORD,
   ORACLE_MAX_PROCESSING_TIME,
   COMMON_HOME_BRIDGE_ADDRESS,
   COMMON_FOREIGN_BRIDGE_ADDRESS,
@@ -25,7 +31,15 @@ const {
   ORACLE_HOME_START_BLOCK,
   ORACLE_FOREIGN_START_BLOCK,
   ORACLE_HOME_RPC_BLOCK_POLLING_LIMIT,
-  ORACLE_FOREIGN_RPC_BLOCK_POLLING_LIMIT
+  ORACLE_FOREIGN_RPC_BLOCK_POLLING_LIMIT,
+  ORACLE_HOME_EVENTS_REPROCESSING,
+  ORACLE_HOME_EVENTS_REPROCESSING_BATCH_SIZE,
+  ORACLE_HOME_EVENTS_REPROCESSING_BLOCK_DELAY,
+  ORACLE_HOME_RPC_SYNC_STATE_CHECK_INTERVAL,
+  ORACLE_FOREIGN_EVENTS_REPROCESSING,
+  ORACLE_FOREIGN_EVENTS_REPROCESSING_BATCH_SIZE,
+  ORACLE_FOREIGN_EVENTS_REPROCESSING_BLOCK_DELAY,
+  ORACLE_FOREIGN_RPC_SYNC_STATE_CHECK_INTERVAL
 } = process.env
 
 let homeAbi
@@ -59,11 +73,19 @@ const homeConfig = {
   bridgeAddress: COMMON_HOME_BRIDGE_ADDRESS,
   bridgeABI: homeAbi,
   pollingInterval: parseInt(ORACLE_HOME_RPC_POLLING_INTERVAL, 10),
+  syncCheckInterval: parseInt(ORACLE_HOME_RPC_SYNC_STATE_CHECK_INTERVAL, 10) || 60000,
   startBlock: parseInt(ORACLE_HOME_START_BLOCK, 10) || 0,
   blockPollingLimit: parseInt(ORACLE_HOME_RPC_BLOCK_POLLING_LIMIT, 10),
   web3: web3Home,
+  web3Redundant: web3HomeRedundant,
+  web3Fallback: web3HomeFallback,
   bridgeContract: homeContract,
-  eventContract: homeContract
+  eventContract: homeContract,
+  reprocessingOptions: {
+    enabled: ORACLE_HOME_EVENTS_REPROCESSING === 'true',
+    batchSize: parseInt(ORACLE_HOME_EVENTS_REPROCESSING_BATCH_SIZE, 10) || 1000,
+    blockDelay: parseInt(ORACLE_HOME_EVENTS_REPROCESSING_BLOCK_DELAY, 10) || 500
+  }
 }
 
 const foreignContract = new web3Foreign.eth.Contract(foreignAbi, COMMON_FOREIGN_BRIDGE_ADDRESS)
@@ -72,18 +94,26 @@ const foreignConfig = {
   bridgeAddress: COMMON_FOREIGN_BRIDGE_ADDRESS,
   bridgeABI: foreignAbi,
   pollingInterval: parseInt(ORACLE_FOREIGN_RPC_POLLING_INTERVAL, 10),
+  syncCheckInterval: parseInt(ORACLE_FOREIGN_RPC_SYNC_STATE_CHECK_INTERVAL, 10) || 60000,
   startBlock: parseInt(ORACLE_FOREIGN_START_BLOCK, 10) || 0,
   blockPollingLimit: parseInt(ORACLE_FOREIGN_RPC_BLOCK_POLLING_LIMIT, 10),
   web3: web3Foreign,
+  web3Redundant: web3ForeignRedundant,
+  web3Fallback: web3ForeignFallback,
+  web3Archive: web3ForeignArchive || web3Foreign,
   bridgeContract: foreignContract,
-  eventContract: foreignContract
+  eventContract: foreignContract,
+  reprocessingOptions: {
+    enabled: ORACLE_FOREIGN_EVENTS_REPROCESSING === 'true',
+    batchSize: parseInt(ORACLE_FOREIGN_EVENTS_REPROCESSING_BATCH_SIZE, 10) || 500,
+    blockDelay: parseInt(ORACLE_FOREIGN_EVENTS_REPROCESSING_BLOCK_DELAY, 10) || 250
+  }
 }
 
 const maxProcessingTime =
   parseInt(ORACLE_MAX_PROCESSING_TIME, 10) || 4 * Math.max(homeConfig.pollingInterval, foreignConfig.pollingInterval)
 
 let validatorPrivateKey
-let validatorAddress = ORACLE_VALIDATOR_ADDRESS
 if (ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY) {
   validatorPrivateKey = add0xPrefix(ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY)
   const derived = privateKeyToAddress(validatorPrivateKey)
@@ -93,22 +123,12 @@ if (ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY) {
     )
     process.exit(EXIT_CODES.INCOMPATIBILITY)
   }
-  validatorAddress = derived
-} else if (ORACLE_VALIDATOR_KEYSTORE_PATH) {
-  try {
-    const keystore = loadKeystore(ORACLE_VALIDATOR_KEYSTORE_PATH, ORACLE_VALIDATOR_KEYSTORE_PASSWORD)
-    validatorPrivateKey = keystore.privateKey
-    validatorAddress = keystore.address
-  } catch (e) {
-    console.error(`Can't load keystore file: ${e.message}`)
-    process.exit(EXIT_CODES.INCOMPATIBILITY)
-  }
 }
 
 module.exports = {
   eventFilter: {},
   validatorPrivateKey,
-  validatorAddress,
+  validatorAddress: ORACLE_VALIDATOR_ADDRESS || privateKeyToAddress(validatorPrivateKey),
   maxProcessingTime,
   shutdownKey: 'oracle-shutdown',
   home: homeConfig,

oracle/config/foreign-mev-sender.config.js

@@ -0,0 +1,37 @@
+const baseConfig = require('./base.config')
+
+const { DEFAULT_TRANSACTION_RESEND_INTERVAL } = require('../src/utils/constants')
+const { MEV_HELPER_ABI } = require('../src/utils/mev')
+const { web3Foreign, getFlashbotsProvider } = require('../src/services/web3')
+
+const {
+  ORACLE_FOREIGN_TX_RESEND_INTERVAL,
+  ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS,
+  ORACLE_MEV_FOREIGN_MIN_GAS_PRICE,
+  ORACLE_MEV_FOREIGN_FLAT_MINER_FEE,
+  ORACLE_MEV_FOREIGN_MAX_PRIORITY_FEE_PER_GAS,
+  ORACLE_MEV_FOREIGN_MAX_FEE_PER_GAS,
+  ORACLE_MEV_FOREIGN_BUNDLES_BLOCK_RANGE
+} = process.env
+
+const contract = new baseConfig.foreign.web3.eth.Contract(MEV_HELPER_ABI, ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS)
+
+module.exports = {
+  ...baseConfig,
+  pollingInterval: baseConfig.foreign.pollingInterval,
+  mevForeign: {
+    contractAddress: ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS,
+    contract,
+    minGasPrice: ORACLE_MEV_FOREIGN_MIN_GAS_PRICE,
+    flatMinerFee: ORACLE_MEV_FOREIGN_FLAT_MINER_FEE,
+    maxPriorityFeePerGas: ORACLE_MEV_FOREIGN_MAX_PRIORITY_FEE_PER_GAS,
+    maxFeePerGas: ORACLE_MEV_FOREIGN_MAX_FEE_PER_GAS,
+    bundlesPerIteration: Math.max(parseInt(ORACLE_MEV_FOREIGN_BUNDLES_BLOCK_RANGE, 10) || 5, 1),
+    getFlashbotsProvider
+  },
+  mevJobsRedisKey: `${baseConfig.id}-collected-signatures-mev:mevJobs`,
+  id: 'mev-sender-foreign',
+  name: 'mev-sender-foreign',
+  web3: web3Foreign,
+  resendInterval: parseInt(ORACLE_FOREIGN_TX_RESEND_INTERVAL, 10) || DEFAULT_TRANSACTION_RESEND_INTERVAL
+}

oracle/config/foreign-sender.config.js

@@ -1,18 +1,14 @@
 const baseConfig = require('./base.config')
 
 const { DEFAULT_TRANSACTION_RESEND_INTERVAL } = require('../src/utils/constants')
-const { web3Foreign, web3ForeignRedundant, web3ForeignFallback } = require('../src/services/web3')
 
 const { ORACLE_FOREIGN_TX_RESEND_INTERVAL } = process.env
 
 module.exports = {
   ...baseConfig,
+  main: baseConfig.foreign,
   queue: 'foreign-prioritized',
-  oldQueue: 'foreign',
   id: 'foreign',
   name: 'sender-foreign',
-  web3: web3Foreign,
-  web3Redundant: web3ForeignRedundant,
-  web3Fallback: web3ForeignFallback,
   resendInterval: parseInt(ORACLE_FOREIGN_TX_RESEND_INTERVAL, 10) || DEFAULT_TRANSACTION_RESEND_INTERVAL
 }

oracle/config/home-sender.config.js

@@ -1,18 +1,14 @@
 const baseConfig = require('./base.config')
 
 const { DEFAULT_TRANSACTION_RESEND_INTERVAL } = require('../src/utils/constants')
-const { web3Home, web3HomeRedundant, web3HomeFallback } = require('../src/services/web3')
 
 const { ORACLE_HOME_TX_RESEND_INTERVAL } = process.env
 
 module.exports = {
   ...baseConfig,
+  main: baseConfig.home,
   queue: 'home-prioritized',
-  oldQueue: 'home',
   id: 'home',
   name: 'sender-home',
-  web3: web3Home,
-  web3Redundant: web3HomeRedundant,
-  web3Fallback: web3HomeFallback,
   resendInterval: parseInt(ORACLE_HOME_TX_RESEND_INTERVAL, 10) || DEFAULT_TRANSACTION_RESEND_INTERVAL
 }

oracle/config/information-request-watcher.config.js

@@ -1,11 +1,9 @@
 const baseConfig = require('./base.config')
-const { web3ForeignArchive } = require('../src/services/web3')
 
 const id = `${baseConfig.id}-information-request`
 
 module.exports = {
   ...baseConfig,
-  web3ForeignArchive: web3ForeignArchive || baseConfig.foreign.web3,
   main: baseConfig.home,
   event: 'UserRequestForInformation',
   sender: 'home',

oracle/config/mev-collected-signatures-watcher.config.js

@@ -0,0 +1,30 @@
+const baseConfig = require('./base.config')
+const { MEV_HELPER_ABI } = require('../src/utils/mev')
+
+const {
+  ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS,
+  ORACLE_MEV_FOREIGN_MIN_GAS_PRICE,
+  ORACLE_MEV_FOREIGN_FLAT_MINER_FEE,
+  ORACLE_MEV_FOREIGN_MAX_PRIORITY_FEE_PER_GAS,
+  ORACLE_MEV_FOREIGN_MAX_FEE_PER_GAS
+} = process.env
+
+const id = `${baseConfig.id}-collected-signatures-mev`
+
+const contract = new baseConfig.foreign.web3.eth.Contract(MEV_HELPER_ABI, ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS)
+
+module.exports = {
+  ...baseConfig,
+  mevForeign: {
+    contractAddress: ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS,
+    contract,
+    minGasPrice: ORACLE_MEV_FOREIGN_MIN_GAS_PRICE,
+    flatMinerFee: ORACLE_MEV_FOREIGN_FLAT_MINER_FEE,
+    maxPriorityFeePerGas: ORACLE_MEV_FOREIGN_MAX_PRIORITY_FEE_PER_GAS,
+    maxFeePerGas: ORACLE_MEV_FOREIGN_MAX_FEE_PER_GAS
+  },
+  main: baseConfig.home,
+  event: 'CollectedSignatures',
+  name: `watcher-${id}`,
+  id
+}

oracle/docker-compose-helpers.yml

@@ -1,6 +1,14 @@
 ---
 version: '2.4'
 services:
+  redis:
+    cpus: 0.1
+    mem_limit: 500m
+    command: [ redis-server, --appendonly, 'yes' ]
+    hostname: redis
+    image: redis:4
+    restart: unless-stopped
+    volumes: [ '~/bridge_data/helpers/redis:/data' ]
   interestFetcher:
     cpus: 0.1
     mem_limit: 500m
@@ -13,3 +21,41 @@ services:
       INTERVAL: 300000
     restart: unless-stopped
     entrypoint: yarn helper:interestFether
+  mevWatcher:
+    cpus: 0.1
+    mem_limit: 500m
+    image: poanetwork/tokenbridge-oracle:latest
+    env_file: ./.env
+    environment:
+      NODE_ENV: production
+      ORACLE_VALIDATOR_ADDRESS: ${ORACLE_VALIDATOR_ADDRESS}
+      ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS: 'TBD'
+      ORACLE_MEV_FOREIGN_MIN_GAS_PRICE: '50000000000' # 50 gwei
+      ORACLE_MEV_FOREIGN_FLAT_MINER_FEE: '1500000000000000' # 0.0015 eth = 300k gas * 5 gwei
+      ORACLE_MEV_FOREIGN_MAX_PRIORITY_FEE_PER_GAS: '0' # 0 gwei
+      ORACLE_MEV_FOREIGN_MAX_FEE_PER_GAS: '1000000000000' # 1000 gwei
+      ORACLE_FOREIGN_RPC_POLLING_INTERVAL: '15000' # CollectedSignatures event polling interval
+      ORACLE_HOME_START_BLOCK: 'TBD'
+      ORACLE_HOME_SKIP_MANUAL_LANE: 'true'
+    restart: unless-stopped
+    entrypoint: yarn mev:watcher:collected-signatures
+  mevSender:
+    cpus: 0.1
+    mem_limit: 500m
+    image: poanetwork/tokenbridge-oracle:latest
+    env_file: ./.env
+    environment:
+      NODE_ENV: production
+      ORACLE_VALIDATOR_ADDRESS: ${ORACLE_VALIDATOR_ADDRESS}
+      ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY: ${ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY}
+      ORACLE_MEV_FOREIGN_HELPER_CONTRACT_ADDRESS: 'TBD'
+      ORACLE_MEV_FOREIGN_MIN_GAS_PRICE: '50000000000' # 50 gwei
+      ORACLE_MEV_FOREIGN_FLAT_MINER_FEE: '1500000000000000' # 0.0015 eth = 300k gas * 5 gwei
+      ORACLE_MEV_FOREIGN_MAX_PRIORITY_FEE_PER_GAS: '0' # 0 gwei
+      ORACLE_MEV_FOREIGN_MAX_FEE_PER_GAS: '1000000000000' # 1000 gwei
+      ORACLE_MEV_FOREIGN_FLASHBOTS_RPC_URL: 'https://relay-goerli.flashbots.net'
+      ORACLE_MEV_FOREIGN_FLASHBOTS_AUTH_SIGNING_KEY: 82db7175932f4e6c8e45283b78b54fd5f195149378ec90d95b8fd0ec8bdadf1d
+      ORACLE_MEV_FOREIGN_BUNDLES_BLOCK_RANGE: '5'
+      ORACLE_FOREIGN_RPC_POLLING_INTERVAL: '70000' # time between sending different batches of MEV bundles (~= 5 blocks * 14 seconds)
+    restart: unless-stopped
+    entrypoint: yarn mev:sender:foreign

oracle/package.json

@@ -19,6 +19,9 @@
     "confirm:information-request": "./scripts/start-worker.sh confirmRelay information-request-watcher",
     "manager:shutdown": "./scripts/start-worker.sh shutdownManager shutdown-manager",
     "helper:interestFether": "node ./scripts/interestFetcher.js",
+    "helper:signPendingMessages": "node ./scripts/signPendingMessages.js",
+    "mev:watcher:collected-signatures": "./scripts/start-worker.sh mevWatcher mev-collected-signatures-watcher",
+    "mev:sender:foreign": "./scripts/start-worker.sh mevSender foreign-mev-sender",
     "dev": "concurrently -n 'watcher:signature-request,watcher:collected-signatures,watcher:affirmation-request,watcher:transfer, sender:home,sender:foreign' -c 'red,green,yellow,blue,magenta,cyan' 'yarn watcher:signature-request' 'yarn watcher:collected-signatures' 'yarn watcher:affirmation-request' 'yarn watcher:transfer' 'yarn sender:home' 'yarn sender:foreign'",
     "test": "NODE_ENV=test mocha",
     "test:watch": "NODE_ENV=test mocha --watch --reporter=min",
@@ -28,17 +31,19 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
+    "@flashbots/ethers-provider-bundle": "^0.4.3",
     "amqp-connection-manager": "^2.0.0",
     "amqplib": "^0.5.2",
     "bignumber.js": "^7.2.1",
     "dotenv": "^5.0.1",
+    "ethers": "^5.5.3",
     "ioredis": "^3.2.2",
     "node-fetch": "^2.1.2",
     "pino": "^4.17.3",
     "pino-pretty": "^2.0.1",
     "promise-limit": "^2.7.0",
     "promise-retry": "^1.1.1",
-    "web3": "^1.3.0"
+    "web3": "^1.6.0"
   },
   "devDependencies": {
     "bn-chai": "^1.0.1",

oracle/scripts/erc20_to_native/sendForeign.js

@@ -36,7 +36,7 @@ async function main() {
         data,
         nonce,
         gasPrice: FOREIGN_TEST_TX_GAS_PRICE,
-        amount: '0',
+        value: '0',
         gasLimit,
         to: bridgeableTokenAddress,
         web3: web3Foreign,

oracle/scripts/erc20_to_native/sendHome.js

@@ -29,7 +29,7 @@ async function main() {
         data: '0x',
         nonce,
         gasPrice: HOME_TEST_TX_GAS_PRICE,
-        amount: HOME_MIN_AMOUNT_PER_TX,
+        value: web3Home.utils.toWei(HOME_MIN_AMOUNT_PER_TX),
         gasLimit: 100000,
         to: COMMON_HOME_BRIDGE_ADDRESS,
         web3: web3Home,

oracle/scripts/interestFetcher.js

@@ -54,7 +54,7 @@ async function main() {
       nonce,
       gasPrice,
       gasLimit: Math.round(gasLimit * 1.5),
-      amount: '0',
+      value: '0',
       chainId,
       web3: web3Home
     })

oracle/scripts/signPendingMessages.js

@@ -0,0 +1,83 @@
+require('dotenv').config()
+
+const {
+  COMMON_HOME_BRIDGE_ADDRESS,
+  COMMON_FOREIGN_BRIDGE_ADDRESS,
+  ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY,
+  ORACLE_HOME_START_BLOCK,
+  ORACLE_HOME_END_BLOCK,
+  ORACLE_BRIDGE_MODE
+} = process.env
+
+const fs = require('fs')
+const promiseLimit = require('promise-limit')
+
+const { web3Home, web3Foreign } = require('../src/services/web3')
+const { getBridgeABIs, getPastEvents, parseAMBMessage, BRIDGE_MODES } = require('../../commons')
+const { setLogger } = require('../src/services/injectedLogger')
+
+const mockLogger = { debug: () => {}, info: () => {}, error: () => {}, child: () => mockLogger }
+setLogger(mockLogger)
+
+const limit = promiseLimit(50)
+
+const output = process.argv[2]
+
+async function main() {
+  const { HOME_ABI, FOREIGN_ABI } = getBridgeABIs(ORACLE_BRIDGE_MODE)
+  const wallet = web3Home.eth.accounts.wallet.add(ORACLE_VALIDATOR_ADDRESS_PRIVATE_KEY)
+  const homeBridge = new web3Home.eth.Contract(HOME_ABI, COMMON_HOME_BRIDGE_ADDRESS)
+  const foreignBridge = new web3Foreign.eth.Contract(FOREIGN_ABI, COMMON_FOREIGN_BRIDGE_ADDRESS)
+  const fromBlock = parseInt(ORACLE_HOME_START_BLOCK, 10) || 0
+  let toBlock = parseInt(ORACLE_HOME_END_BLOCK, 10)
+  if (!toBlock) {
+    toBlock = await web3Home.eth.getBlockNumber()
+  }
+  console.log(`Getting CollectedSignatures events from block ${fromBlock} to block ${toBlock}`)
+  const events = await getPastEvents(homeBridge, { event: 'CollectedSignatures', fromBlock, toBlock })
+  console.log(`Found ${events.length} CollectedSignatures events`)
+  console.log('Getting messages')
+  let messages = await Promise.all(
+    events.map((event, i) => () => getMessage(ORACLE_BRIDGE_MODE, homeBridge, foreignBridge, event, i)).map(limit)
+  )
+  messages = messages.filter(x => x)
+  console.log(`Filtered ${messages.length} pending messages`)
+  const result = {}
+  messages.forEach(msg => {
+    result[msg.msgHash] = wallet.sign(msg.message).signature
+  })
+
+  console.log('Writing results')
+  if (output === '-') {
+    console.log(JSON.stringify(result))
+  } else {
+    fs.writeFileSync(output, JSON.stringify(result))
+  }
+}
+
+async function getMessage(bridgeMode, homeBridge, foreignBridge, event, i) {
+  if (i % 50 === 0) {
+    console.log(`Processing event #${i}`)
+  }
+  const msgHash = event.returnValues.messageHash
+  const message = await homeBridge.methods.message(msgHash).call()
+
+  let msgId
+  if (bridgeMode === BRIDGE_MODES.ARBITRARY_MESSAGE) {
+    msgId = parseAMBMessage(message).messageId
+  } else {
+    msgId = `0x${message.slice(106, 170)}`
+  }
+  const alreadyProcessed = await foreignBridge.methods.relayedMessages(msgId).call()
+
+  if (alreadyProcessed) {
+    return null
+  }
+
+  return {
+    msgHash,
+    message
+  }
+}
+
+main()

oracle/src/confirmRelay.js

@@ -153,11 +153,11 @@ async function main({ sendJob, txHashes }) {
 }
 
 async function sendJobTx(jobs) {
-  await GasPrice.start(chain, true)
-  const gasPrice = GasPrice.getPrice().toString(10)
-
   const { web3 } = config.sender === 'foreign' ? config.foreign : config.home
 
+  await GasPrice.start(chain, web3, true)
+  const gasPriceOptions = GasPrice.gasPriceOptions()
+
   const chainId = await getChainId(web3)
   let nonce = await getNonce(web3, config.validatorAddress)
 
@@ -174,13 +174,13 @@ async function sendJobTx(jobs) {
       const txHash = await sendTx({
         data: job.data,
         nonce,
-        gasPrice,
+        value: '0',
-        amount: '0',
         gasLimit,
         privateKey: config.validatorPrivateKey,
         to: job.to,
         chainId,
-        web3
+        web3,
+        gasPriceOptions
       })
 
       nonce++
@@ -197,7 +197,7 @@ async function sendJobTx(jobs) {
 
       if (e.message.toLowerCase().includes('insufficient funds')) {
         const currentBalance = await web3.eth.getBalance(config.validatorAddress)
-        const minimumBalance = gasLimit.multipliedBy(gasPrice)
+        const minimumBalance = gasLimit.multipliedBy(gasPriceOptions.gasPrice || gasPriceOptions.maxFeePerGas)
         logger.error(
           `Insufficient funds: ${currentBalance}. Stop processing messages until the balance is at least ${minimumBalance}.`
         )

oracle/src/events/processAMBCollectedSignatures/estimateGas.js

@@ -4,7 +4,6 @@ const { AlreadyProcessedError, IncompatibleContractError, InvalidValidatorError
 const logger = require('../../services/logger').child({
   module: 'processCollectedSignatures:estimateGas'
 })
-const { parseAMBHeader } = require('../../utils/message')
 
 const web3 = new Web3()
 const { toBN } = Web3.utils
@@ -22,15 +21,9 @@ async function estimateGas({
   address
 }) {
   try {
-    const gasEstimate = await foreignBridge.methods.executeSignatures(message, signatures).estimateGas({
+    return await foreignBridge.methods.executeSignatures(message, signatures).estimateGas({
       from: address
     })
-    const msgGasLimit = parseAMBHeader(message).gasLimit
-
-    // + estimateExtraGas(len)
-    // is not needed here, since estimateGas will already take into account gas
-    // needed for memory expansion, message processing, etc.
-    return gasEstimate + msgGasLimit
   } catch (e) {
     if (e instanceof HttpListProviderError) {
       throw e

oracle/src/events/processAMBCollectedSignaturesMEV/index.js

@@ -0,0 +1,183 @@
+require('dotenv').config()
+const promiseLimit = require('promise-limit')
+const { HttpListProviderError } = require('../../services/HttpListProvider')
+const { getValidatorContract } = require('../../tx/web3')
+const rootLogger = require('../../services/logger')
+const { signatureToVRS, packSignatures } = require('../../utils/message')
+const { readAccessListFile, isRevertError } = require('../../utils/utils')
+const { parseAMBMessage } = require('../../../../commons')
+const estimateGas = require('../processAMBCollectedSignatures/estimateGas')
+const { AlreadyProcessedError, IncompatibleContractError, InvalidValidatorError } = require('../../utils/errors')
+const { MAX_CONCURRENT_EVENTS, EXTRA_GAS_ABSOLUTE } = require('../../utils/constants')
+
+const limit = promiseLimit(MAX_CONCURRENT_EVENTS)
+
+const { ORACLE_HOME_TO_FOREIGN_ALLOWANCE_LIST, ORACLE_HOME_TO_FOREIGN_BLOCK_LIST } = process.env
+const ORACLE_HOME_SKIP_MANUAL_LANE = process.env.ORACLE_HOME_SKIP_MANUAL_LANE === 'true'
+
+function processCollectedSignaturesBuilder(config) {
+  const { home, foreign, mevForeign } = config
+
+  let validatorContract = null
+
+  return async function processCollectedSignatures(signatures) {
+    const txToSend = []
+
+    if (validatorContract === null) {
+      validatorContract = await getValidatorContract(foreign.bridgeContract, foreign.web3)
+    }
+
+    rootLogger.debug(`Processing ${signatures.length} CollectedSignatures events`)
+    const callbacks = signatures
+      .map(colSignature => async () => {
+        const { messageHash, NumberOfCollectedSignatures } = colSignature.returnValues
+
+        const logger = rootLogger.child({
+          eventTransactionHash: colSignature.transactionHash
+        })
+
+        logger.info(`Processing CollectedSignatures ${colSignature.transactionHash}`)
+        const message = await home.bridgeContract.methods.message(messageHash).call()
+        const parsedMessage = parseAMBMessage(message)
+
+        if (ORACLE_HOME_TO_FOREIGN_ALLOWANCE_LIST || ORACLE_HOME_TO_FOREIGN_BLOCK_LIST) {
+          const sender = parsedMessage.sender.toLowerCase()
+          const executor = parsedMessage.executor.toLowerCase()
+
+          if (ORACLE_HOME_TO_FOREIGN_ALLOWANCE_LIST) {
+            const allowanceList = await readAccessListFile(ORACLE_HOME_TO_FOREIGN_ALLOWANCE_LIST, logger)
+            if (!allowanceList.includes(executor) && !allowanceList.includes(sender)) {
+              logger.info(
+                { sender, executor },
+                'Validator skips a message. Neither sender nor executor addresses are in the allowance list.'
+              )
+              return
+            }
+          } else if (ORACLE_HOME_TO_FOREIGN_BLOCK_LIST) {
+            const blockList = await readAccessListFile(ORACLE_HOME_TO_FOREIGN_BLOCK_LIST, logger)
+            if (blockList.includes(executor)) {
+              logger.info({ executor }, 'Validator skips a message. Executor address is in the block list.')
+              return
+            }
+            if (blockList.includes(sender)) {
+              logger.info({ sender }, 'Validator skips a message. Sender address is in the block list.')
+              return
+            }
+          }
+        }
+
+        if (ORACLE_HOME_SKIP_MANUAL_LANE && parsedMessage.decodedDataType.manualLane) {
+          logger.info(
+            { dataType: parsedMessage.dataType },
+            'Validator skips a message. Message was forwarded to the manual lane by the extension'
+          )
+          return
+        }
+
+        logger.debug({ NumberOfCollectedSignatures }, 'Number of signatures to get')
+
+        const requiredSignatures = []
+        requiredSignatures.length = NumberOfCollectedSignatures
+        requiredSignatures.fill(0)
+
+        const signaturesArray = []
+        const [v, r, s] = [[], [], []]
+        logger.debug('Getting message signatures')
+        const signaturePromises = requiredSignatures.map(async (el, index) => {
+          logger.debug({ index }, 'Getting message signature')
+          const signature = await home.bridgeContract.methods.signature(messageHash, index).call()
+          const vrs = signatureToVRS(signature)
+          v.push(vrs.v)
+          r.push(vrs.r)
+          s.push(vrs.s)
+          signaturesArray.push(vrs)
+        })
+
+        await Promise.all(signaturePromises)
+        const signatures = packSignatures(signaturesArray)
+        logger.info(`Processing messageId: ${parsedMessage.messageId}`)
+
+        let gasEstimate
+        try {
+          logger.debug('Estimate gas')
+          gasEstimate = await estimateGas({
+            foreignBridge: foreign.bridgeContract,
+            validatorContract,
+            v,
+            r,
+            s,
+            signatures,
+            message,
+            numberOfCollectedSignatures: NumberOfCollectedSignatures,
+            messageId: parsedMessage.messageId,
+            address: config.validatorAddress
+          })
+          logger.debug({ gasEstimate }, 'Gas estimated')
+        } catch (e) {
+          if (e instanceof HttpListProviderError) {
+            throw new Error('RPC Connection Error: submitSignature Gas Estimate cannot be obtained.')
+          } else if (e instanceof AlreadyProcessedError) {
+            logger.info(`Already processed CollectedSignatures ${colSignature.transactionHash}`)
+            return
+          } else if (e instanceof IncompatibleContractError || e instanceof InvalidValidatorError) {
+            logger.error(`The message couldn't be processed; skipping: ${e.message}`)
+            return
+          } else {
+            logger.error(e, 'Unknown error while processing transaction')
+            throw e
+          }
+        }
+
+        const executeData = foreign.bridgeContract.methods.executeSignatures(message, signatures).encodeABI()
+        const profit = await estimateProfit(
+          mevForeign.contract,
+          mevForeign.minGasPrice,
+          executeData,
+          mevForeign.flatMinerFee
+        )
+        if (profit === '0') {
+          logger.error('No MEV opportunity found when testing with min gas price, skipping job')
+          return
+        }
+        logger.info(`Estimated profit of ${profit} when simulating with ${mevForeign.minGasPrice} gas price`)
+
+        txToSend.push({
+          profit,
+          executeData,
+          data: mevForeign.contract.methods.execute(executeData).encodeABI(),
+          gasEstimate,
+          extraGas: EXTRA_GAS_ABSOLUTE,
+          maxFeePerGas: mevForeign.maxFeePerGas,
+          maxPriorityFeePerGas: mevForeign.maxPriorityFeePerGas,
+          transactionReference: colSignature.transactionHash,
+          to: mevForeign.contractAddress,
+          value: mevForeign.flatMinerFee
+        })
+      })
+      .map(promise => limit(promise))
+
+    await Promise.all(callbacks)
+
+    return txToSend
+  }
+}
+
+async function estimateProfit(contract, gasPrice, data, minerFee) {
+  return contract.methods
+    .estimateProfit(gasPrice, data)
+    .call({ value: minerFee })
+    .then(
+      res => res.toString(),
+      e => {
+        if (isRevertError(e)) {
+          return '0'
+        }
+        throw e
+      }
+    )
+}
+
+module.exports = {
+  processCollectedSignaturesBuilder,
+  estimateProfit
+}

oracle/src/events/processAMBInformationRequests/index.js

@@ -35,11 +35,13 @@ Object.keys(asyncCalls).forEach(method => {
 })
 
 function processInformationRequestsBuilder(config) {
-  const { home, foreign, web3ForeignArchive } = config
+  const { home, foreign } = config
 
   let validatorContract = null
   let blockFinder = null
 
+  foreign.web3Archive.currentProvider.startSyncStateChecker(foreign.syncCheckInterval)
+
   return async function processInformationRequests(informationRequests) {
     const txToSend = []
 
@@ -49,13 +51,15 @@ function processInformationRequestsBuilder(config) {
 
     if (blockFinder === null) {
       rootLogger.debug('Initializing block finder')
-      blockFinder = await makeBlockFinder('foreign', foreign.web3)
+      blockFinder = await makeBlockFinder('foreign', foreign.web3Archive)
     }
 
+    // latest foreign block is requested from an archive RPC, to ensure that it is synced with the network
+    // block confirmations can be requested from the regular JSON RPC
     const foreignBlockNumber =
-      (await getBlockNumber(foreign.web3)) - (await getRequiredBlockConfirmations(foreign.bridgeContract))
+      (await getBlockNumber(foreign.web3Archive)) - (await getRequiredBlockConfirmations(foreign.bridgeContract))
     const homeBlock = await getBlock(home.web3, informationRequests[0].blockNumber)
-    const lastForeignBlock = await getBlock(foreign.web3, foreignBlockNumber)
+    const lastForeignBlock = await getBlock(foreign.web3Archive, foreignBlockNumber)
 
     if (homeBlock.timestamp > lastForeignBlock.timestamp) {
       rootLogger.debug(
@@ -85,7 +89,7 @@ function processInformationRequestsBuilder(config) {
         logger.info({ requestSelector, method: asyncCallMethod, data }, 'Processing async request')
 
         const call = asyncCalls[asyncCallMethod]
-        let [status, result] = await call(web3ForeignArchive, data, foreignClosestBlock).catch(e => {
+        let [status, result] = await call(foreign.web3Archive, data, foreignClosestBlock).catch(e => {
           if (e instanceof HttpListProviderError) {
             throw e
           }

oracle/src/events/processAffirmationRequests/estimateGas.js

@@ -6,11 +6,9 @@ const logger = require('../../services/logger').child({
 
 async function estimateGas({ web3, homeBridge, validatorContract, recipient, value, txHash, address }) {
   try {
-    const gasEstimate = await homeBridge.methods.executeAffirmation(recipient, value, txHash).estimateGas({
+    return await homeBridge.methods.executeAffirmation(recipient, value, txHash).estimateGas({
       from: address
     })
-
-    return gasEstimate
   } catch (e) {
     if (e instanceof HttpListProviderError) {
       throw e

oracle/src/events/processCollectedSignatures/estimateGas.js

@@ -20,8 +20,7 @@ async function estimateGas({
   signatures
 }) {
   try {
-    const gasEstimate = await foreignBridge.methods.executeSignatures(message, signatures).estimateGas()
+    return await foreignBridge.methods.executeSignatures(message, signatures).estimateGas()
-    return gasEstimate
   } catch (e) {
     if (e instanceof HttpListProviderError) {
       throw e

oracle/src/events/processSignatureRequests/estimateGas.js

@@ -6,10 +6,9 @@ const logger = require('../../services/logger').child({
 
 async function estimateGas({ web3, homeBridge, validatorContract, signature, message, address }) {
   try {
-    const gasEstimate = await homeBridge.methods.submitSignature(signature, message).estimateGas({
+    return await homeBridge.methods.submitSignature(signature, message).estimateGas({
       from: address
     })
-    return gasEstimate
   } catch (e) {
     if (e instanceof HttpListProviderError) {
       throw e

oracle/src/mevSender.js

@@ -0,0 +1,159 @@
+require('../env')
+const path = require('path')
+const BigNumber = require('bignumber.js')
+const { redis } = require('./services/redisClient')
+const logger = require('./services/logger')
+const { sendTx } = require('./tx/sendTx')
+const { getNonce, getChainId, getBlock } = require('./tx/web3')
+const { addExtraGas, checkHTTPS, watchdog } = require('./utils/utils')
+const { EXIT_CODES, EXTRA_GAS_PERCENTAGE, MAX_GAS_LIMIT } = require('./utils/constants')
+const { estimateProfit } = require('./events/processAMBCollectedSignaturesMEV')
+
+if (process.argv.length < 3) {
+  logger.error('Please check the number of arguments, config file was not provided')
+  process.exit(EXIT_CODES.GENERAL_ERROR)
+}
+
+const config = require(path.join('../config/', process.argv[2]))
+
+const { web3, mevForeign, validatorAddress } = config
+
+let chainId = 0
+let flashbotsProvider
+
+async function initialize() {
+  try {
+    const checkHttps = checkHTTPS(process.env.ORACLE_ALLOW_HTTP_FOR_RPC, logger)
+
+    web3.currentProvider.urls.forEach(checkHttps(config.id))
+
+    chainId = await getChainId(web3)
+    flashbotsProvider = await mevForeign.getFlashbotsProvider(chainId)
+    return runMain()
+  } catch (e) {
+    logger.error(e.message)
+    process.exit(EXIT_CODES.GENERAL_ERROR)
+  }
+}
+
+async function runMain() {
+  try {
+    if (redis.status === 'ready') {
+      if (config.maxProcessingTime) {
+        await watchdog(main, config.maxProcessingTime, () => {
+          logger.fatal('Max processing time reached')
+          process.exit(EXIT_CODES.MAX_TIME_REACHED)
+        })
+      } else {
+        await main()
+      }
+    }
+  } catch (e) {
+    logger.error(e)
+  }
+
+  setTimeout(runMain, config.pollingInterval)
+}
+
+async function main() {
+  try {
+    const jobs = Object.values(await redis.hgetall(config.mevJobsRedisKey)).map(JSON.parse)
+    const totalJobs = jobs.length
+
+    if (totalJobs === 0) {
+      logger.debug('Nothing to process')
+      return
+    }
+
+    const { baseFeePerGas: pendingBaseFee, number: pendingBlockNumber } = await getBlock(web3, 'pending')
+    const bestJob = pickBestJob(jobs, pendingBaseFee)
+
+    if (!bestJob) {
+      logger.info({ totalJobs, pendingBaseFee }, 'No suitable job was found, waiting for a lower gas price')
+      return
+    }
+
+    const jobLogger = logger.child({ eventTransactionHash: bestJob.transactionReference })
+
+    const maxProfit = await estimateProfit(
+      mevForeign.contract,
+      mevForeign.minGasPrice,
+      bestJob.executeData,
+      bestJob.value
+    )
+
+    if (maxProfit === '0') {
+      jobLogger.info(`No MEV opportunity found when testing with min gas price ${mevForeign.minGasPrice}, removing job`)
+      await redis.hdel(config.mevJobsRedisKey, bestJob.transactionReference)
+      return
+    }
+    jobLogger.info(`Estimated profit of ${maxProfit} when simulating with ${mevForeign.minGasPrice} gas price`)
+    bestJob.profit = maxProfit
+
+    if (new BigNumber(pendingBaseFee).gt(mevForeign.minGasPrice)) {
+      const profit = await estimateProfit(mevForeign.contract, pendingBaseFee, bestJob.executeData, bestJob.value)
+      if (profit === '0') {
+        jobLogger.info(
+          `No MEV opportunity found when testing with current gas price ${pendingBaseFee}, waiting for lower gas price`
+        )
+        bestJob.maxFeePerGas = pendingBaseFee
+        await redis.hset(config.mevJobsRedisKey, bestJob.transactionReference, JSON.stringify(bestJob))
+        return
+      }
+      jobLogger.info(`Estimated profit of ${profit} when simulating with ${pendingBaseFee} gas price`)
+    }
+
+    let gasLimit
+    if (typeof bestJob.extraGas === 'number') {
+      gasLimit = addExtraGas(bestJob.gasEstimate + bestJob.extraGas, 0, MAX_GAS_LIMIT)
+    } else {
+      gasLimit = addExtraGas(bestJob.gasEstimate, EXTRA_GAS_PERCENTAGE, MAX_GAS_LIMIT)
+    }
+
+    const nonce = await getNonce(web3, validatorAddress)
+    jobLogger.info(
+      { nonce, fromBlock: pendingBlockNumber, toBlock: pendingBlockNumber + mevForeign.bundlesPerIteration - 1 },
+      'Sending MEV bundles'
+    )
+    const txHash = await sendTx({
+      data: bestJob.data,
+      nonce,
+      value: bestJob.value,
+      gasLimit,
+      privateKey: config.validatorPrivateKey,
+      to: bestJob.to,
+      chainId,
+      web3,
+      gasPriceOptions: {
+        maxFeePerGas: bestJob.maxFeePerGas,
+        maxPriorityFeePerGas: bestJob.maxPriorityFeePerGas
+      },
+      mevOptions: {
+        provider: flashbotsProvider,
+        fromBlock: pendingBlockNumber,
+        toBlock: pendingBlockNumber + mevForeign.bundlesPerIteration - 1,
+        logger
+      }
+    })
+
+    jobLogger.info({ txHash }, `Tx generated ${txHash} for event Tx ${bestJob.transactionReference}`)
+
+    await redis.hset(config.mevJobsRedisKey, bestJob.transactionReference, JSON.stringify(bestJob))
+    jobLogger.debug(`Finished processing msg`)
+  } catch (e) {
+    logger.error(e)
+  }
+}
+
+function pickBestJob(jobs, feePerGas) {
+  const feePerGasBN = new BigNumber(feePerGas)
+  let best = null
+  jobs.forEach(job => {
+    if (feePerGasBN.lt(job.maxFeePerGas) && (!best || new BigNumber(best.profit).lt(job.profit))) {
+      best = job
+    }
+  })
+  return best
+}
+
+initialize()

oracle/src/mevWatcher.js

@@ -0,0 +1,251 @@
+require('../env')
+const path = require('path')
+const { redis } = require('./services/redisClient')
+const logger = require('./services/logger')
+const { getBlockNumber, getRequiredBlockConfirmations, getEvents } = require('./tx/web3')
+const { checkHTTPS, watchdog, syncForEach } = require('./utils/utils')
+const { processCollectedSignaturesBuilder } = require('./events/processAMBCollectedSignaturesMEV')
+const {
+  EXIT_CODES,
+  BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT,
+  MAX_HISTORY_BLOCK_TO_REPROCESS
+} = require('./utils/constants')
+
+if (process.argv.length < 3) {
+  logger.error('Please check the number of arguments, config file was not provided')
+  process.exit(EXIT_CODES.GENERAL_ERROR)
+}
+
+const config = require(path.join('../config/', process.argv[2]))
+
+const processAMBCollectedSignaturesMEV = processCollectedSignaturesBuilder(config)
+
+const {
+  web3,
+  bridgeContract,
+  eventContract,
+  startBlock,
+  pollingInterval,
+  chain,
+  reprocessingOptions,
+  blockPollingLimit
+} = config.main
+const lastBlockRedisKey = `${config.id}:lastProcessedBlock`
+const lastReprocessedBlockRedisKey = `${config.id}:lastReprocessedBlock`
+const seenEventsRedisKey = `${config.id}:seenEvents`
+const mevJobsRedisKey = `${config.id}:mevJobs`
+let lastProcessedBlock = Math.max(startBlock - 1, 0)
+let lastReprocessedBlock
+let lastSeenBlockNumber = 0
+let sameBlockNumberCounter = 0
+
+async function initialize() {
+  try {
+    const checkHttps = checkHTTPS(process.env.ORACLE_ALLOW_HTTP_FOR_RPC, logger)
+
+    web3.currentProvider.urls.forEach(checkHttps(chain))
+
+    await getLastProcessedBlock()
+    await getLastReprocessedBlock()
+    runMain({ sendToQueue: saveJobsToRedis })
+  } catch (e) {
+    logger.error(e)
+    process.exit(EXIT_CODES.GENERAL_ERROR)
+  }
+}
+
+async function runMain({ sendToQueue }) {
+  try {
+    if (redis.status === 'ready') {
+      if (config.maxProcessingTime) {
+        await watchdog(() => main({ sendToQueue }), config.maxProcessingTime, () => {
+          logger.fatal('Max processing time reached')
+          process.exit(EXIT_CODES.MAX_TIME_REACHED)
+        })
+      } else {
+        await main({ sendToQueue })
+      }
+    }
+  } catch (e) {
+    logger.error(e)
+  }
+
+  setTimeout(() => {
+    runMain({ sendToQueue })
+  }, pollingInterval)
+}
+
+async function saveJobsToRedis(jobs) {
+  return syncForEach(jobs, job => redis.hset(mevJobsRedisKey, job.transactionReference, JSON.stringify(job)))
+}
+
+async function getLastProcessedBlock() {
+  const result = await redis.get(lastBlockRedisKey)
+  logger.debug({ fromRedis: result, fromConfig: lastProcessedBlock }, 'Last Processed block obtained')
+  lastProcessedBlock = result ? parseInt(result, 10) : lastProcessedBlock
+}
+
+async function getLastReprocessedBlock() {
+  if (reprocessingOptions.enabled) {
+    const result = await redis.get(lastReprocessedBlockRedisKey)
+    if (result) {
+      lastReprocessedBlock = Math.max(parseInt(result, 10), lastProcessedBlock - MAX_HISTORY_BLOCK_TO_REPROCESS)
+    } else {
+      lastReprocessedBlock = lastProcessedBlock
+    }
+    logger.debug({ block: lastReprocessedBlock }, 'Last reprocessed block obtained')
+  } else {
+    // when reprocessing is being enabled not for the first time,
+    // we do not want to process blocks for which we didn't recorded seen events,
+    // instead, we want to start from the current block.
+    // Thus we should delete this reprocessing pointer once it is disabled.
+    await redis.del(lastReprocessedBlockRedisKey)
+  }
+}
+
+function updateLastProcessedBlock(lastBlockNumber) {
+  lastProcessedBlock = lastBlockNumber
+  return redis.set(lastBlockRedisKey, lastProcessedBlock)
+}
+
+function updateLastReprocessedBlock(lastBlockNumber) {
+  lastReprocessedBlock = lastBlockNumber
+  return redis.set(lastReprocessedBlockRedisKey, lastReprocessedBlock)
+}
+
+function processEvents(events) {
+  switch (config.id) {
+    case 'amb-collected-signatures-mev':
+      return processAMBCollectedSignaturesMEV(events)
+    default:
+      return []
+  }
+}
+
+const eventKey = e => `${e.transactionHash}-${e.logIndex}`
+
+async function reprocessOldLogs(sendToQueue) {
+  const fromBlock = lastReprocessedBlock + 1
+  const toBlock = lastReprocessedBlock + reprocessingOptions.batchSize
+  const events = await getEvents({
+    contract: eventContract,
+    event: config.event,
+    fromBlock,
+    toBlock,
+    filter: config.eventFilter
+  })
+  const alreadySeenEvents = await getSeenEvents(fromBlock, toBlock)
+  const missingEvents = events.filter(e => !alreadySeenEvents[eventKey(e)])
+  if (missingEvents.length === 0) {
+    logger.debug('No missed events were found')
+  } else {
+    logger.info(`Found ${missingEvents.length} ${config.event} missed events`)
+    const job = await processEvents(missingEvents)
+    logger.info('Missed events transactions to send:', job.length)
+    if (job.length) {
+      await sendToQueue(job)
+    }
+  }
+
+  await updateLastReprocessedBlock(toBlock)
+  await deleteSeenEvents(0, toBlock)
+}
+
+async function getSeenEvents(fromBlock, toBlock) {
+  const keys = await redis.zrangebyscore(seenEventsRedisKey, fromBlock, toBlock)
+  const res = {}
+  keys.forEach(k => {
+    res[k] = true
+  })
+  return res
+}
+
+function deleteSeenEvents(fromBlock, toBlock) {
+  return redis.zremrangebyscore(seenEventsRedisKey, fromBlock, toBlock)
+}
+
+function addSeenEvents(events) {
+  return redis.zadd(seenEventsRedisKey, ...events.flatMap(e => [e.blockNumber, eventKey(e)]))
+}
+
+async function getLastBlockToProcess(web3, bridgeContract) {
+  const [lastBlockNumber, requiredBlockConfirmations] = await Promise.all([
+    getBlockNumber(web3),
+    getRequiredBlockConfirmations(bridgeContract)
+  ])
+
+  if (lastBlockNumber < lastSeenBlockNumber) {
+    sameBlockNumberCounter = 0
+    logger.warn({ lastBlockNumber, lastSeenBlockNumber }, 'Received block number less than already seen block')
+    web3.currentProvider.switchToFallbackRPC()
+  } else if (lastBlockNumber === lastSeenBlockNumber) {
+    sameBlockNumberCounter++
+    if (sameBlockNumberCounter > 1) {
+      logger.info({ lastBlockNumber, sameBlockNumberCounter }, 'Received the same block number more than twice')
+      if (sameBlockNumberCounter >= BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT) {
+        sameBlockNumberCounter = 0
+        logger.warn(
+          { lastBlockNumber, n: BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT },
+          'Received the same block number for too many times. Probably node is not synced anymore'
+        )
+        web3.currentProvider.switchToFallbackRPC()
+      }
+    }
+  } else {
+    sameBlockNumberCounter = 0
+    lastSeenBlockNumber = lastBlockNumber
+  }
+  return lastBlockNumber - requiredBlockConfirmations
+}
+
+async function main({ sendToQueue }) {
+  try {
+    const lastBlockToProcess = await getLastBlockToProcess(web3, bridgeContract)
+
+    if (reprocessingOptions.enabled) {
+      if (lastReprocessedBlock + reprocessingOptions.batchSize + reprocessingOptions.blockDelay < lastBlockToProcess) {
+        await reprocessOldLogs(sendToQueue)
+        return
+      }
+    }
+
+    if (lastBlockToProcess <= lastProcessedBlock) {
+      logger.debug('All blocks already processed')
+      return
+    }
+
+    const fromBlock = lastProcessedBlock + 1
+    const rangeEndBlock = blockPollingLimit ? fromBlock + blockPollingLimit : lastBlockToProcess
+    const toBlock = Math.min(lastBlockToProcess, rangeEndBlock)
+
+    const events = await getEvents({
+      contract: eventContract,
+      event: config.event,
+      fromBlock,
+      toBlock,
+      filter: config.eventFilter
+    })
+    logger.info(`Found ${events.length} ${config.event} events`)
+
+    if (events.length) {
+      const job = await processEvents(events)
+      logger.info('Transactions to send:', job.length)
+
+      if (job.length) {
+        await sendToQueue(job)
+      }
+      if (reprocessingOptions.enabled) {
+        await addSeenEvents(events)
+      }
+    }
+
+    logger.debug({ lastProcessedBlock: toBlock.toString() }, 'Updating last processed block')
+    await updateLastProcessedBlock(toBlock)
+  } catch (e) {
+    logger.error(e)
+  }
+
+  logger.debug('Finished')
+}
+
+initialize()

oracle/src/sender.js

@@ -9,6 +9,8 @@ const { sendTx } = require('./tx/sendTx')
 const { getNonce, getChainId } = require('./tx/web3')
 const {
   addExtraGas,
+  applyMinGasFeeBump,
+  chooseGasPriceOptions,
   checkHTTPS,
   syncForEach,
   waitForFunds,
@@ -19,7 +21,7 @@ const {
   isInsufficientBalanceError,
   isNonceError
 } = require('./utils/utils')
-const { EXIT_CODES, EXTRA_GAS_PERCENTAGE, MAX_GAS_LIMIT } = require('./utils/constants')
+const { EXIT_CODES, EXTRA_GAS_PERCENTAGE, MAX_GAS_LIMIT, MIN_GAS_PRICE_BUMP_FACTOR } = require('./utils/constants')
 
 const { ORACLE_TX_REDUNDANCY } = process.env
 
@@ -30,8 +32,8 @@ if (process.argv.length < 3) {
 
 const config = require(path.join('../config/', process.argv[2]))
 
-const { web3, web3Fallback } = config
+const { web3, web3Fallback, syncCheckInterval } = config.main
-const web3Redundant = ORACLE_TX_REDUNDANCY === 'true' ? config.web3Redundant : web3
+const web3Redundant = ORACLE_TX_REDUNDANCY === 'true' ? config.main.web3Redundant : web3
 
 const nonceKey = `${config.id}:nonce`
 let chainId = 0
@@ -41,8 +43,9 @@ async function initialize() {
     const checkHttps = checkHTTPS(process.env.ORACLE_ALLOW_HTTP_FOR_RPC, logger)
 
     web3.currentProvider.urls.forEach(checkHttps(config.id))
+    web3.currentProvider.startSyncStateChecker(syncCheckInterval)
 
-    GasPrice.start(config.id)
+    GasPrice.start(config.id, web3)
 
     chainId = await getChainId(web3)
     connectQueue()
@@ -55,7 +58,6 @@ async function initialize() {
 function connectQueue() {
   connectSenderToQueue({
     queueName: config.queue,
-    oldQueueName: config.oldQueue,
     resendInterval: config.resendInterval,
     cb: options => {
       if (config.maxProcessingTime) {
@@ -121,7 +123,7 @@ async function main({ msg, ackMsg, nackMsg, channel, scheduleForRetry, scheduleT
 
     const txArray = JSON.parse(msg.content)
     logger.debug(`Msg received with ${txArray.length} Tx to send`)
-    const gasPrice = GasPrice.getPrice().toString(10)
+    const gasPriceOptions = GasPrice.gasPriceOptions()
 
     let nonce
     let insufficientFunds = false
@@ -147,6 +149,7 @@ async function main({ msg, ackMsg, nackMsg, channel, scheduleForRetry, scheduleT
       }
 
       try {
+        const newGasPriceOptions = chooseGasPriceOptions(gasPriceOptions, job.gasPriceOptions)
         if (isResend) {
           const tx = await web3Fallback.eth.getTransaction(job.txHash)
 
@@ -159,24 +162,26 @@ async function main({ msg, ackMsg, nackMsg, channel, scheduleForRetry, scheduleT
             nonce = await readNonce(true)
           }
 
-          logger.info(`Transaction ${job.txHash} was not mined, updating gasPrice: ${job.gasPrice} -> ${gasPrice}`)
+          const oldGasPrice = JSON.stringify(job.gasPriceOptions)
+          const newGasPrice = JSON.stringify(newGasPriceOptions)
+          logger.info(`Transaction ${job.txHash} was not mined, updating gasPrice: ${oldGasPrice} -> ${newGasPrice}`)
         }
         logger.info(`Sending transaction with nonce ${nonce}`)
         const txHash = await sendTx({
           data: job.data,
           nonce,
-          gasPrice,
+          value: '0',
-          amount: '0',
           gasLimit,
           privateKey: config.validatorPrivateKey,
           to: job.to,
           chainId,
-          web3: web3Redundant
+          web3: web3Redundant,
+          gasPriceOptions: newGasPriceOptions
         })
         const resendJob = {
           ...job,
           txHash,
-          gasPrice
+          gasPriceOptions: newGasPriceOptions
         }
         resendJobs.push(resendJob)
 
@@ -194,8 +199,8 @@ async function main({ msg, ackMsg, nackMsg, channel, scheduleForRetry, scheduleT
 
         if (isGasPriceError(e)) {
           logger.info('Replacement transaction underpriced, forcing gas price update')
-          GasPrice.start(config.id)
+          GasPrice.start(config.id, web3)
-          failedTx.push(job)
+          failedTx.push(applyMinGasFeeBump(job, MIN_GAS_PRICE_BUMP_FACTOR))
         } else if (isResend || isSameTransactionError(e)) {
           resendJobs.push(job)
         } else {
@@ -208,7 +213,7 @@ async function main({ msg, ackMsg, nackMsg, channel, scheduleForRetry, scheduleT
         if (isInsufficientBalanceError(e)) {
           insufficientFunds = true
           const currentBalance = await web3.eth.getBalance(config.validatorAddress)
-          minimumBalance = gasLimit.multipliedBy(gasPrice)
+          minimumBalance = gasLimit.multipliedBy(gasPriceOptions.gasPrice || gasPriceOptions.maxFeePerGas)
           logger.error(
             `Insufficient funds: ${currentBalance}. Stop processing messages until the balance is at least ${minimumBalance}.`
           )

oracle/src/services/HttpListProvider.js

@@ -1,5 +1,6 @@
 const fetch = require('node-fetch')
 const promiseRetry = require('promise-retry')
+const { utils } = require('web3')
 const { FALLBACK_RPC_URL_SWITCH_TIMEOUT } = require('../utils/constants')
 
 const { onInjected } = require('./injectedLogger')
@@ -39,19 +40,54 @@ function HttpListProvider(urls, options = {}) {
   this.options = { ...defaultOptions, ...options }
   this.currentIndex = 0
   this.lastTimeUsedPrimary = 0
+  this.latestBlock = 0
+  this.syncStateCheckerIntervalId = 0
 
   onInjected(logger => {
     this.logger = logger.child({ module: `HttpListProvider:${this.options.name}` })
   })
 }
 
-HttpListProvider.prototype.switchToFallbackRPC = function() {
+HttpListProvider.prototype.startSyncStateChecker = function(syncCheckInterval) {
-  if (this.urls.length < 2) {
+  if (this.urls.length > 1 && syncCheckInterval > 0 && this.syncStateCheckerIntervalId === 0) {
+    this.syncStateCheckerIntervalId = setInterval(this.checkLatestBlock.bind(this), syncCheckInterval)
+  }
+}
+
+HttpListProvider.prototype.checkLatestBlock = function() {
+  const payload = { jsonrpc: '2.0', id: 1, method: 'eth_blockNumber', params: [] }
+  this.send(payload, (error, result) => {
+    if (error) {
+      this.logger.warn({ oldBlock: this.latestBlock }, 'Failed to request latest block from all RPC urls')
+    } else if (result.error) {
+      this.logger.warn(
+        { oldBlock: this.latestBlock, error: result.error.message },
+        'Failed to make eth_blockNumber request due to unknown error, switching to fallback RPC'
+      )
+      this.switchToFallbackRPC()
+    } else {
+      const blockNumber = utils.hexToNumber(result.result)
+      if (blockNumber > this.latestBlock) {
+        this.logger.debug({ oldBlock: this.latestBlock, newBlock: blockNumber }, 'Updating latest block number')
+        this.latestBlock = blockNumber
+      } else {
+        this.logger.warn(
+          { oldBlock: this.latestBlock, newBlock: blockNumber },
+          'Latest block on the node was not updated since last request, switching to fallback RPC'
+        )
+        this.switchToFallbackRPC()
+      }
+    }
+  })
+}
+
+HttpListProvider.prototype.switchToFallbackRPC = function(index) {
+  const prevIndex = this.currentIndex
+  const newIndex = index || (prevIndex + 1) % this.urls.length
+  if (this.urls.length < 2 || prevIndex === newIndex) {
     return
   }
 
-  const prevIndex = this.currentIndex
-  const newIndex = (prevIndex + 1) % this.urls.length
   this.logger.info(
     { index: newIndex, oldURL: this.urls[prevIndex], newURL: this.urls[newIndex] },
     'Switching to fallback JSON-RPC URL'
@@ -80,11 +116,7 @@ HttpListProvider.prototype.send = async function send(payload, callback) {
 
     // if some of URLs failed to respond, current URL index is updated to the first URL that responded
     if (currentIndex !== index) {
-      this.logger.info(
+      this.switchToFallbackRPC(index)
-        { index, oldURL: this.urls[currentIndex], newURL: this.urls[index] },
-        'Switching to fallback JSON-RPC URL'
-      )
-      this.currentIndex = index
     }
     callback(null, result)
   } catch (e) {

oracle/src/services/amqpClient.js

@@ -40,23 +40,9 @@ function connectWatcherToQueue({ queueName, cb }) {
   cb({ sendToQueue, channel: channelWrapper })
 }
 
-function connectSenderToQueue({ queueName, oldQueueName, cb, resendInterval }) {
+function connectSenderToQueue({ queueName, cb, resendInterval }) {
   const deadLetterExchange = `${queueName}-retry`
 
-  async function resendMessagesToNewQueue(channel) {
-    logger.info(`Trying to check messages in the old non-priority queue ${queueName}`)
-    while (true) {
-      const msg = await channel.get(oldQueueName)
-      if (msg === false) {
-        logger.info(`No messages in the old queue ${oldQueueName} left`)
-        break
-      }
-      logger.debug(`Message in the old queue ${oldQueueName} was found, redirecting it to the new queue ${queueName}`)
-      await channel.sendToQueue(queueName, msg.content, { persistent: true, priority: SENDER_QUEUE_SEND_PRIORITY })
-      await channel.ack(msg)
-    }
-  }
-
   const channelWrapper = connection.createChannel({
     json: true
   })
@@ -64,7 +50,6 @@ function connectSenderToQueue({ queueName, oldQueueName, cb, resendInterval }) {
   channelWrapper.addSetup(async channel => {
     await channel.assertExchange(deadLetterExchange, 'fanout', { durable: true })
     await channel.assertQueue(queueName, { durable: true, maxPriority: SENDER_QUEUE_MAX_PRIORITY })
-    await channel.assertQueue(oldQueueName, { durable: true }).then(() => resendMessagesToNewQueue(channel))
     await channel.bindQueue(queueName, deadLetterExchange)
     await channel.prefetch(1)
     await channel.consume(queueName, msg =>

oracle/src/services/gasPrice.js

@@ -20,21 +20,21 @@ const {
   COMMON_HOME_GAS_PRICE_FACTOR
 } = process.env
 
-let cachedGasPrice = null
+let cachedGasPriceOptions = null
 
 let fetchGasPriceInterval = null
 
-const fetchGasPrice = async (speedType, factor, bridgeContract, gasPriceSupplierUrl) => {
+const fetchGasPrice = async (speedType, factor, web3, bridgeContract, gasPriceSupplierUrl) => {
   const contractOptions = { logger }
   const supplierOptions = { speedType, factor, limits: GAS_PRICE_BOUNDARIES, logger }
-  cachedGasPrice =
+  cachedGasPriceOptions =
-    (await gasPriceFromSupplier(gasPriceSupplierUrl, supplierOptions)) ||
+    (await gasPriceFromSupplier(web3, gasPriceSupplierUrl, supplierOptions)) ||
     (await gasPriceFromContract(bridgeContract, contractOptions)) ||
-    cachedGasPrice
+    cachedGasPriceOptions
-  return cachedGasPrice
+  return cachedGasPriceOptions
 }
 
-async function start(chainId, fetchOnce) {
+async function start(chainId, web3, fetchOnce) {
   clearInterval(fetchGasPriceInterval)
 
   let contract = null
@@ -49,7 +49,7 @@ async function start(chainId, fetchOnce) {
     updateInterval = ORACLE_HOME_GAS_PRICE_UPDATE_INTERVAL || DEFAULT_UPDATE_INTERVAL
     factor = Number(COMMON_HOME_GAS_PRICE_FACTOR) || DEFAULT_GAS_PRICE_FACTOR
 
-    cachedGasPrice = COMMON_HOME_GAS_PRICE_FALLBACK
+    cachedGasPriceOptions = { gasPrice: COMMON_HOME_GAS_PRICE_FALLBACK }
   } else if (chainId === 'foreign') {
     contract = foreign.bridgeContract
     gasPriceSupplierUrl = COMMON_FOREIGN_GAS_PRICE_SUPPLIER_URL
@@ -57,7 +57,7 @@ async function start(chainId, fetchOnce) {
     updateInterval = ORACLE_FOREIGN_GAS_PRICE_UPDATE_INTERVAL || DEFAULT_UPDATE_INTERVAL
     factor = Number(COMMON_FOREIGN_GAS_PRICE_FACTOR) || DEFAULT_GAS_PRICE_FACTOR
 
-    cachedGasPrice = COMMON_FOREIGN_GAS_PRICE_FALLBACK
+    cachedGasPriceOptions = { gasPrice: COMMON_FOREIGN_GAS_PRICE_FALLBACK }
   } else {
     throw new Error(`Unrecognized chainId '${chainId}'`)
   }
@@ -67,21 +67,21 @@ async function start(chainId, fetchOnce) {
   }
 
   if (fetchOnce) {
-    await fetchGasPrice(speedType, factor, contract, gasPriceSupplierUrl)
+    await fetchGasPrice(speedType, factor, web3, contract, gasPriceSupplierUrl)
   } else {
     fetchGasPriceInterval = await setIntervalAndRun(
-      () => fetchGasPrice(speedType, factor, contract, gasPriceSupplierUrl),
+      () => fetchGasPrice(speedType, factor, web3, contract, gasPriceSupplierUrl),
       updateInterval
     )
   }
 }
 
-function getPrice() {
+function gasPriceOptions() {
-  return cachedGasPrice
+  return cachedGasPriceOptions
 }
 
 module.exports = {
   start,
-  getPrice,
+  gasPriceOptions,
   fetchGasPrice
 }

oracle/src/services/web3.js

@@ -1,4 +1,6 @@
 const Web3 = require('web3')
+const ethers = require('ethers')
+const flashbots = require('@flashbots/ethers-provider-bundle')
 const { HttpListProvider } = require('./HttpListProvider')
 const { SafeEthLogsProvider } = require('./SafeEthLogsProvider')
 const { RedundantHttpListProvider } = require('./RedundantHttpListProvider')
@@ -9,6 +11,8 @@ const {
   COMMON_FOREIGN_RPC_URL,
   ORACLE_SIDE_RPC_URL,
   ORACLE_FOREIGN_ARCHIVE_RPC_URL,
+  ORACLE_MEV_FOREIGN_FLASHBOTS_RPC_URL,
+  ORACLE_MEV_FOREIGN_FLASHBOTS_AUTH_SIGNING_KEY,
   ORACLE_RPC_REQUEST_TIMEOUT,
   ORACLE_HOME_RPC_POLLING_INTERVAL,
   ORACLE_FOREIGN_RPC_POLLING_INTERVAL
@@ -94,6 +98,15 @@ if (foreignUrls.length > 1) {
   web3ForeignRedundant = new Web3(redundantProvider)
 }
 
+let getFlashbotsProvider
+if (ORACLE_MEV_FOREIGN_FLASHBOTS_RPC_URL) {
+  const provider = new ethers.providers.JsonRpcProvider(foreignUrls[0])
+  const authSigner = new ethers.Wallet(ORACLE_MEV_FOREIGN_FLASHBOTS_AUTH_SIGNING_KEY, provider)
+
+  getFlashbotsProvider = chainId =>
+    flashbots.FlashbotsBundleProvider.create(provider, authSigner, ORACLE_MEV_FOREIGN_FLASHBOTS_RPC_URL, chainId)
+}
+
 module.exports = {
   web3Home,
   web3Foreign,
@@ -102,5 +115,6 @@ module.exports = {
   web3HomeRedundant,
   web3ForeignRedundant,
   web3HomeFallback,
-  web3ForeignFallback
+  web3ForeignFallback,
+  getFlashbotsProvider
 }

oracle/src/tx/sendTx.js

@@ -1,25 +1,38 @@
-const { toWei } = require('web3').utils
+async function sendTx(opts) {
-
+  const { privateKey, data, nonce, gasPrice, gasPriceOptions, value, gasLimit, to, chainId, web3, mevOptions } = opts
-async function sendTx({ privateKey, data, nonce, gasPrice, amount, gasLimit, to, chainId, web3 }) {
+  const gasOpts = gasPriceOptions || { gasPrice }
   const serializedTx = await web3.eth.accounts.signTransaction(
     {
       nonce: Number(nonce),
       chainId,
       to,
       data,
-      value: toWei(amount),
+      value,
-      gasPrice,
+      gas: gasLimit,
-      gas: gasLimit
+      ...gasOpts
     },
     privateKey
   )
 
-  return new Promise((res, rej) =>
+  if (!mevOptions) {
-    web3.eth
+    return new Promise((res, rej) =>
-      .sendSignedTransaction(serializedTx.rawTransaction)
+      web3.eth
-      .once('transactionHash', res)
+        .sendSignedTransaction(serializedTx.rawTransaction)
-      .once('error', rej)
+        .once('transactionHash', res)
+        .once('error', rej)
+    )
+  }
+
+  mevOptions.logger.debug(
+    { rawTx: serializedTx.rawTransaction, txHash: serializedTx.transactionHash },
+    'Signed MEV helper transaction'
   )
+
+  for (let blockNumber = mevOptions.fromBlock; blockNumber <= mevOptions.toBlock; blockNumber++) {
+    mevOptions.logger.debug({ txHash: serializedTx.transactionHash, blockNumber }, 'Sending MEV bundle transaction')
+    await mevOptions.provider.sendRawBundle([serializedTx.rawTransaction], blockNumber)
+  }
+  return Promise.resolve(serializedTx.transactionHash)
 }
 
 module.exports = {

oracle/src/tx/web3.js

@@ -87,7 +87,7 @@ async function getEvents({ contract, event, fromBlock, toBlock, filter }) {
     )
     const pastEvents = await contract.getPastEvents(event, { fromBlock, toBlock, filter })
     logger.debug({ contractAddress, event, count: pastEvents.length }, 'Past events obtained')
-    return pastEvents
+    return pastEvents.sort((a, b) => a.blockNumber - b.blockNumber || a.transactionIndex - b.transactionIndex)
   } catch (e) {
     logger.error(e.message)
     throw new Error(`${event} events cannot be obtained`)

oracle/src/utils/constants.js

@@ -5,6 +5,7 @@ module.exports = {
   MIN_AMB_HEADER_LENGTH: 32 + 20 + 20 + 4 + 2 + 1 + 2,
   MAX_GAS_LIMIT: 10000000,
   MAX_CONCURRENT_EVENTS: 50,
+  MAX_HISTORY_BLOCK_TO_REPROCESS: 10000,
   RETRY_CONFIG: {
     retries: 20,
     factor: 1.4,
@@ -23,9 +24,9 @@ module.exports = {
     MIN: 1,
     MAX: 1000
   },
+  MIN_GAS_PRICE_BUMP_FACTOR: 0.1,
   DEFAULT_TRANSACTION_RESEND_INTERVAL: 20 * 60 * 1000,
   FALLBACK_RPC_URL_SWITCH_TIMEOUT: 60 * 60 * 1000,
-  BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT: 10,
   SENDER_QUEUE_MAX_PRIORITY: 10,
   SENDER_QUEUE_SEND_PRIORITY: 5,
   SENDER_QUEUE_CHECK_STATUS_PRIORITY: 1,

oracle/src/utils/mev.js

@@ -0,0 +1,43 @@
+const MEV_HELPER_ABI = [
+  {
+    constant: false,
+    inputs: [
+      {
+        name: '_data',
+        type: 'bytes'
+      }
+    ],
+    name: 'execute',
+    outputs: [],
+    payable: false,
+    stateMutability: 'nonpayable',
+    type: 'function'
+  },
+  {
+    constant: false,
+    inputs: [
+      {
+        name: '_gasPrice',
+        type: 'uint256'
+      },
+      {
+        name: '_data',
+        type: 'bytes'
+      }
+    ],
+    name: 'estimateProfit',
+    outputs: [
+      {
+        name: '',
+        type: 'uint256'
+      }
+    ],
+    payable: true,
+    stateMutability: 'nonpayable',
+    type: 'function'
+  }
+]
+
+module.exports = {
+  MEV_HELPER_ABI
+}

oracle/src/utils/utils.js

@@ -2,6 +2,9 @@ const fs = require('fs')
 const BigNumber = require('bignumber.js')
 const promiseRetry = require('promise-retry')
 const Web3 = require('web3')
+const { GAS_PRICE_BOUNDARIES } = require('./constants')
+
+const { toBN, toWei } = Web3.utils
 
 const retrySequence = [1, 2, 3, 5, 8, 13, 21, 34, 55, 60]
 
@@ -34,8 +37,8 @@ const promiseRetryForever = f => promiseRetry(f, { forever: true, factor: 1 })
 async function waitForFunds(web3, address, minimumBalance, cb, logger) {
   promiseRetryForever(async retry => {
     logger.debug('Getting balance of validator account')
-    const newBalance = web3.utils.toBN(await web3.eth.getBalance(address))
+    const newBalance = toBN(await web3.eth.getBalance(address))
-    if (newBalance.gte(web3.utils.toBN(minimumBalance.toString(10)))) {
+    if (newBalance.gte(toBN(minimumBalance.toString(10)))) {
       logger.debug({ balance: newBalance, minimumBalance }, 'Validator has minimum necessary balance')
       cb(newBalance)
     } else {
@@ -64,6 +67,48 @@ function addExtraGas(gas, extraPercentage, maxGasLimit = Infinity) {
   return BigNumber.min(maxGasLimit, gasWithExtra)
 }
 
+function applyMinGasFeeBump(job, bumpFactor = 0.1) {
+  if (!job.gasPriceOptions) {
+    return job
+  }
+  const { gasPrice, maxFeePerGas, maxPriorityFeePerGas } = job.gasPriceOptions
+  const maxGasPrice = toWei(GAS_PRICE_BOUNDARIES.MAX.toString(), 'gwei')
+  if (gasPrice) {
+    return {
+      ...job,
+      gasPriceOptions: {
+        gasPrice: addExtraGas(gasPrice, bumpFactor, maxGasPrice).toString()
+      }
+    }
+  }
+  if (maxFeePerGas && maxPriorityFeePerGas) {
+    return {
+      ...job,
+      gasPriceOptions: {
+        maxFeePerGas: addExtraGas(maxFeePerGas, bumpFactor, maxGasPrice).toString(),
+        maxPriorityFeePerGas: addExtraGas(maxPriorityFeePerGas, bumpFactor, maxGasPrice).toString()
+      }
+    }
+  }
+  return job
+}
+
+function chooseGasPriceOptions(a, b) {
+  if (!a) {
+    return b
+  }
+  if (a && b && a.gasPrice && b.gasPrice) {
+    return { gasPrice: BigNumber.max(a.gasPrice, b.gasPrice).toString() }
+  }
+  if (a && b && a.maxFeePerGas && b.maxFeePerGas && a.maxPriorityFeePerGas && b.maxPriorityFeePerGas) {
+    return {
+      maxFeePerGas: BigNumber.max(a.maxFeePerGas, b.maxFeePerGas).toString(),
+      maxPriorityFeePerGas: BigNumber.max(a.maxPriorityFeePerGas, b.maxPriorityFeePerGas).toString()
+    }
+  }
+  return a
+}
+
 async function setIntervalAndRun(f, interval) {
   const handler = setInterval(f, interval)
   await f()
@@ -99,11 +144,6 @@ function privateKeyToAddress(privateKey) {
   return privateKey ? new Web3().eth.accounts.privateKeyToAccount(add0xPrefix(privateKey)).address : null
 }
 
-function loadKeystore(keystorePath, password) {
-  const keystore = JSON.parse(fs.readFileSync(keystorePath).toString())
-  return new Web3().eth.accounts.wallet.decrypt(keystore, password)[0]
-}
-
 function isGasPriceError(e) {
   const message = e.message.toLowerCase()
   return message.includes('replacement transaction underpriced')
@@ -129,7 +169,8 @@ function isNonceError(e) {
     message.includes('transaction nonce is too low') ||
     message.includes('nonce too low') ||
     message.includes('transaction with same nonce in the queue') ||
-    message.includes('oldnonce')
+    message.includes('oldnonce') ||
+    message.includes(`the tx doesn't have the correct nonce`)
   )
 }
 
@@ -188,6 +229,8 @@ module.exports = {
   waitForFunds,
   waitForUnsuspend,
   addExtraGas,
+  chooseGasPriceOptions,
+  applyMinGasFeeBump,
   setIntervalAndRun,
   watchdog,
   add0xPrefix,
@@ -200,6 +243,5 @@ module.exports = {
   getRetrySequence,
   promiseAny,
   readAccessListFile,
-  zipToObject,
+  zipToObject
-  loadKeystore
 }

oracle/src/watcher.js

@@ -6,7 +6,7 @@ const logger = require('./services/logger')
 const { getShutdownFlag } = require('./services/shutdownState')
 const { getBlockNumber, getRequiredBlockConfirmations, getEvents } = require('./tx/web3')
 const { checkHTTPS, watchdog } = require('./utils/utils')
-const { EXIT_CODES, BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT } = require('./utils/constants')
+const { EXIT_CODES, MAX_HISTORY_BLOCK_TO_REPROCESS } = require('./utils/constants')
 
 if (process.argv.length < 3) {
   logger.error('Please check the number of arguments, config file was not provided')
@@ -26,19 +26,33 @@ const processAMBInformationRequests = require('./events/processAMBInformationReq
 
 const { getTokensState } = require('./utils/tokenState')
 
-const { web3, bridgeContract, eventContract, startBlock, pollingInterval, chain } = config.main
+const {
+  web3,
+  bridgeContract,
+  eventContract,
+  startBlock,
+  pollingInterval,
+  chain,
+  reprocessingOptions,
+  blockPollingLimit,
+  syncCheckInterval
+} = config.main
 const lastBlockRedisKey = `${config.id}:lastProcessedBlock`
+const lastReprocessedBlockRedisKey = `${config.id}:lastReprocessedBlock`
+const seenEventsRedisKey = `${config.id}:seenEvents`
 let lastProcessedBlock = Math.max(startBlock - 1, 0)
-let lastSeenBlockNumber = 0
+let lastReprocessedBlock
-let sameBlockNumberCounter = 0
 
 async function initialize() {
   try {
     const checkHttps = checkHTTPS(process.env.ORACLE_ALLOW_HTTP_FOR_RPC, logger)
 
     web3.currentProvider.urls.forEach(checkHttps(chain))
+    web3.currentProvider.startSyncStateChecker(syncCheckInterval)
 
     await getLastProcessedBlock()
+    await getLastReprocessedBlock()
+    await checkConditions()
     connectWatcherToQueue({
       queueName: config.queue,
       cb: runMain
@@ -76,11 +90,34 @@ async function getLastProcessedBlock() {
   lastProcessedBlock = result ? parseInt(result, 10) : lastProcessedBlock
 }
 
+async function getLastReprocessedBlock() {
+  if (reprocessingOptions.enabled) {
+    const result = await redis.get(lastReprocessedBlockRedisKey)
+    if (result) {
+      lastReprocessedBlock = Math.max(parseInt(result, 10), lastProcessedBlock - MAX_HISTORY_BLOCK_TO_REPROCESS)
+    } else {
+      lastReprocessedBlock = lastProcessedBlock
+    }
+    logger.debug({ block: lastReprocessedBlock }, 'Last reprocessed block obtained')
+  } else {
+    // when reprocessing is being enabled not for the first time,
+    // we do not want to process blocks for which we didn't recorded seen events,
+    // instead, we want to start from the current block.
+    // Thus we should delete this reprocessing pointer once it is disabled.
+    await redis.del(lastReprocessedBlockRedisKey)
+  }
+}
+
 function updateLastProcessedBlock(lastBlockNumber) {
   lastProcessedBlock = lastBlockNumber
   return redis.set(lastBlockRedisKey, lastProcessedBlock)
 }
 
+function updateLastReprocessedBlock(lastBlockNumber) {
+  lastReprocessedBlock = lastBlockNumber
+  return redis.set(lastReprocessedBlockRedisKey, lastReprocessedBlock)
+}
+
 function processEvents(events) {
   switch (config.id) {
     case 'erc-native-signature-request':
@@ -114,33 +151,76 @@ async function checkConditions() {
   }
 }
 
+const eventKey = e => `${e.transactionHash}-${e.logIndex}`
+
+async function reprocessOldLogs(sendToQueue) {
+  const fromBlock = lastReprocessedBlock + 1
+  let toBlock = lastReprocessedBlock + reprocessingOptions.batchSize
+  const events = await getEvents({
+    contract: eventContract,
+    event: config.event,
+    fromBlock,
+    toBlock,
+    filter: config.eventFilter
+  })
+  const alreadySeenEvents = await getSeenEvents(fromBlock, toBlock)
+  const missingEvents = events.filter(e => !alreadySeenEvents[eventKey(e)])
+  if (missingEvents.length === 0) {
+    logger.debug('No missed events were found')
+  } else {
+    logger.info(`Found ${missingEvents.length} ${config.event} missed events`)
+    let job
+    if (config.id === 'amb-information-request') {
+      // obtain block number and events from the earliest block
+      const batchBlockNumber = missingEvents[0].blockNumber
+      const batchEvents = missingEvents.filter(event => event.blockNumber === batchBlockNumber)
+
+      // if there are some other events in the later blocks,
+      // adjust lastReprocessedBlock so that these events will be processed again on the next iteration
+      if (batchEvents.length < missingEvents.length) {
+        // pick event outside from the batch
+        toBlock = missingEvents[batchEvents.length].blockNumber - 1
+      }
+
+      job = await processAMBInformationRequests(batchEvents)
+      if (job === null) {
+        return
+      }
+    } else {
+      job = await processEvents(missingEvents)
+    }
+    logger.info('Missed events transactions to send:', job.length)
+    if (job.length) {
+      await sendToQueue(job)
+    }
+  }
+
+  await updateLastReprocessedBlock(toBlock)
+  await deleteSeenEvents(0, toBlock)
+}
+
+async function getSeenEvents(fromBlock, toBlock) {
+  const keys = await redis.zrangebyscore(seenEventsRedisKey, fromBlock, toBlock)
+  const res = {}
+  keys.forEach(k => {
+    res[k] = true
+  })
+  return res
+}
+
+function deleteSeenEvents(fromBlock, toBlock) {
+  return redis.zremrangebyscore(seenEventsRedisKey, fromBlock, toBlock)
+}
+
+function addSeenEvents(events) {
+  return redis.zadd(seenEventsRedisKey, ...events.flatMap(e => [e.blockNumber, eventKey(e)]))
+}
+
 async function getLastBlockToProcess(web3, bridgeContract) {
   const [lastBlockNumber, requiredBlockConfirmations] = await Promise.all([
     getBlockNumber(web3),
     getRequiredBlockConfirmations(bridgeContract)
   ])
-
-  if (lastBlockNumber < lastSeenBlockNumber) {
-    sameBlockNumberCounter = 0
-    logger.warn({ lastBlockNumber, lastSeenBlockNumber }, 'Received block number less than already seen block')
-    web3.currentProvider.switchToFallbackRPC()
-  } else if (lastBlockNumber === lastSeenBlockNumber) {
-    sameBlockNumberCounter++
-    if (sameBlockNumberCounter > 1) {
-      logger.info({ lastBlockNumber, sameBlockNumberCounter }, 'Received the same block number more than twice')
-      if (sameBlockNumberCounter >= BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT) {
-        sameBlockNumberCounter = 0
-        logger.warn(
-          { lastBlockNumber, n: BLOCK_NUMBER_PROGRESS_ITERATIONS_LIMIT },
-          'Received the same block number for too many times. Probably node is not synced anymore'
-        )
-        web3.currentProvider.switchToFallbackRPC()
-      }
-    }
-  } else {
-    sameBlockNumberCounter = 0
-    lastSeenBlockNumber = lastBlockNumber
-  }
   return lastBlockNumber - requiredBlockConfirmations
 }
 
@@ -158,24 +238,29 @@ async function main({ sendToQueue }) {
 
     const lastBlockToProcess = await getLastBlockToProcess(web3, bridgeContract)
 
+    if (reprocessingOptions.enabled) {
+      if (lastReprocessedBlock + reprocessingOptions.batchSize + reprocessingOptions.blockDelay < lastBlockToProcess) {
+        await reprocessOldLogs(sendToQueue)
+        return
+      }
+    }
+
     if (lastBlockToProcess <= lastProcessedBlock) {
       logger.debug('All blocks already processed')
       return
     }
 
-    await checkConditions()
-
     const fromBlock = lastProcessedBlock + 1
-    const rangeEndBlock = config.blockPollingLimit ? fromBlock + config.blockPollingLimit : lastBlockToProcess
+    const rangeEndBlock = blockPollingLimit ? fromBlock + blockPollingLimit : lastBlockToProcess
     let toBlock = Math.min(lastBlockToProcess, rangeEndBlock)
 
-    const events = (await getEvents({
+    let events = await getEvents({
       contract: eventContract,
       event: config.event,
       fromBlock,
       toBlock,
       filter: config.eventFilter
-    })).sort((a, b) => a.blockNumber - b.blockNumber)
+    })
     logger.info(`Found ${events.length} ${config.event} events`)
 
     if (events.length) {
@@ -192,9 +277,10 @@ async function main({ sendToQueue }) {
         if (batchEvents.length < events.length) {
           // pick event outside from the batch
           toBlock = events[batchEvents.length].blockNumber - 1
+          events = batchEvents
         }
 
-        job = await processAMBInformationRequests(batchEvents)
+        job = await processAMBInformationRequests(events)
         if (job === null) {
           return
         }
@@ -206,6 +292,9 @@ async function main({ sendToQueue }) {
       if (job.length) {
         await sendToQueue(job)
       }
+      if (reprocessingOptions.enabled) {
+        await addSeenEvents(events)
+      }
     }
 
     logger.debug({ lastProcessedBlock: toBlock.toString() }, 'Updating last processed block')

oracle/test/gasPrice.test.js

@@ -71,10 +71,10 @@ describe('gasPrice', () => {
       await gasPrice.start('home')
 
       // when
-      await gasPrice.fetchGasPrice('standard', 1, null, null)
+      await gasPrice.fetchGasPrice('standard', 1, null, null, null)
 
       // then
-      expect(gasPrice.getPrice()).to.equal('101000000000')
+      expect(gasPrice.gasPriceOptions()).to.eql({ gasPrice: '101000000000' })
     })
 
     it('should fetch gas from supplier', async () => {
@@ -82,10 +82,10 @@ describe('gasPrice', () => {
       await gasPrice.start('home')
 
       // when
-      await gasPrice.fetchGasPrice('standard', 1, null, 'url')
+      await gasPrice.fetchGasPrice('standard', 1, null, null, 'url')
 
       // then
-      expect(gasPrice.getPrice().toString()).to.equal('103000000000')
+      expect(gasPrice.gasPriceOptions()).to.eql({ gasPrice: '103000000000' })
     })
 
     it('should fetch gas from contract', async () => {
@@ -101,10 +101,10 @@ describe('gasPrice', () => {
       }
 
       // when
-      await gasPrice.fetchGasPrice('standard', 1, bridgeContractMock, null)
+      await gasPrice.fetchGasPrice('standard', 1, null, bridgeContractMock, null)
 
       // then
-      expect(gasPrice.getPrice().toString()).to.equal('102000000000')
+      expect(gasPrice.gasPriceOptions()).to.eql({ gasPrice: '102000000000' })
     })
 
     it('should fetch the gas price from the oracle first', async () => {
@@ -120,10 +120,10 @@ describe('gasPrice', () => {
       }
 
       // when
-      await gasPrice.fetchGasPrice('standard', 1, bridgeContractMock, 'url')
+      await gasPrice.fetchGasPrice('standard', 1, null, bridgeContractMock, 'url')
 
       // then
-      expect(gasPrice.getPrice().toString()).to.equal('103000000000')
+      expect(gasPrice.gasPriceOptions()).to.eql({ gasPrice: '103000000000' })
     })
 
     it('log error using the logger', async () => {
@@ -131,7 +131,7 @@ describe('gasPrice', () => {
       await gasPrice.start('home')
 
       // when
-      await gasPrice.fetchGasPrice('standard', 1, null, null)
+      await gasPrice.fetchGasPrice('standard', 1, null, null, null)
 
       // then
       expect(fakeLogger.warn.calledOnce).to.equal(true) // one warning

oracle/test/utils.test.js

@@ -3,7 +3,13 @@ const chai = require('chai')
 const chaiAsPromised = require('chai-as-promised')
 const BigNumber = require('bignumber.js')
 const proxyquire = require('proxyquire')
-const { addExtraGas, syncForEach, promiseAny } = require('../src/utils/utils')
+const {
+  addExtraGas,
+  applyMinGasFeeBump,
+  chooseGasPriceOptions,
+  syncForEach,
+  promiseAny
+} = require('../src/utils/utils')
 
 chai.use(chaiAsPromised)
 chai.should()
@@ -173,4 +179,43 @@ describe('utils', () => {
       await promiseAny(array.map(f)).should.be.rejected
     })
   })
+
+  describe('applyMinGasFeeBump', () => {
+    it('should bump pre-eip1559 fee', () => {
+      const job = { gasPriceOptions: { gasPrice: '100000000000' } }
+      const newJob = applyMinGasFeeBump(job)
+      expect(newJob.gasPriceOptions.gasPrice).to.be.equal('110000000000')
+    })
+
+    it('should bump eip1559 fee', () => {
+      const job = { gasPriceOptions: { maxFeePerGas: '100000000000', maxPriorityFeePerGas: '20000000000' } }
+      const newJob = applyMinGasFeeBump(job)
+      expect(newJob.gasPriceOptions.maxFeePerGas).to.be.equal('110000000000')
+      expect(newJob.gasPriceOptions.maxPriorityFeePerGas).to.be.equal('22000000000')
+    })
+  })
+
+  describe('chooseGasPriceOptions', () => {
+    it('should choose max pre-eip1559 fee', () => {
+      const opts1 = { gasPrice: '100000000000' }
+      const opts2 = { gasPrice: '101000000000' }
+      expect(chooseGasPriceOptions(opts1, opts2).gasPrice).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(opts2, opts1).gasPrice).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(opts2, undefined).gasPrice).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(undefined, opts2).gasPrice).to.be.equal('101000000000')
+    })
+
+    it('should choose max eip1559 fee', () => {
+      const opts1 = { maxFeePerGas: '100000000000', maxPriorityFeePerGas: '21000000000' }
+      const opts2 = { maxFeePerGas: '101000000000', maxPriorityFeePerGas: '20000000000' }
+      expect(chooseGasPriceOptions(opts1, opts2).maxFeePerGas).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(opts1, opts2).maxPriorityFeePerGas).to.be.equal('21000000000')
+      expect(chooseGasPriceOptions(opts2, opts1).maxFeePerGas).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(opts2, opts1).maxPriorityFeePerGas).to.be.equal('21000000000')
+      expect(chooseGasPriceOptions(opts2, undefined).maxFeePerGas).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(opts2, undefined).maxPriorityFeePerGas).to.be.equal('20000000000')
+      expect(chooseGasPriceOptions(undefined, opts2).maxFeePerGas).to.be.equal('101000000000')
+      expect(chooseGasPriceOptions(undefined, opts2).maxPriorityFeePerGas).to.be.equal('20000000000')
+    })
+  })
 })

package.json

@@ -45,5 +45,8 @@
     "compile:contracts": "yarn workspace tokenbridge-contracts run compile",
     "install:deploy": "cd contracts/deploy && npm install --unsafe-perm --silent",
     "postinstall": "test -n \"$NOYARNPOSTINSTALL\" || ln -sf $(pwd)/node_modules/openzeppelin-solidity/ contracts/node_modules/openzeppelin-solidity"
+  },
+  "resolutions": {
+    "**/@mycrypto/eth-scan": "3.5.3"
   }
 }