Troubleshooting

Storage errors

DB initilaize failure

Aergo uses Badger DB as its internal storage. If the DB creation fails for Badger at initial boot, boot process will fail.

In this case, you can see the following error when checking the server’s log.

failed to initialize chaindb
failed to initialize statedb

You need to make sure that you have write access to the path you want to create the data directory on the config file If the database server was rebooted while it was running, it is possible that the DB datafile has been crashed. In this case, a reinstallation is required.

check previously running processes

The DB initialize problem is most likely to occur when a new aergosvr is run without shutting down the existing aergosvr completely. In this case, you can check with the following log.

panic: Fail to Create New DB: Cannot acquire directory lock on "config data directory". 
Another process is using this Badger database.: resource temporarily unavailable`

In this case, terminate the existing aergosvr completely and restart it.

Miscellaneous DB errors

If an error occurs in DB, the following log is output.

Database Error: description...

In this case, it’s a good idea to first check to see if there is a problem with the filesystem .If there is no problem with the filesystem, the DB datafile can be seen as corrupt. In this case, reinstallation is required.

Disk performance problem

The BP node generates a block every 1 second. When a block is created, it is stored in the DB and Disk I / O occurs at this time.

If the disk I / O is too slow, the block created by the BP will not be added to the chain. This is because the block generated by the node is propagated to another node but the block of the corresponding height is already generated in the next BP. Therefore, the block arriving late comes to be discarded.

Check the disk I / O execution time through dd. This allows you to check for disk errors.

Network trouble

peer connection failure

If there is a problem with the network configuration, the peer connections may not be normal. you can check peer connections as follows.

1. You need to make sure that the entire peer set in the config is connected. If there is an unconnected peer, check the network configuration with the peer.

   You can check the connection status of peers with the following command:

   aergocli -p $ServerPort getpeers
   [
           {
                   "Address": "127.0.0.1",
                   "PeerID": "16Uiu2HAkxUPj8zUnSkBzP8f4mw9WvJ4s5DeMVarzsqBuyrDy6dmD",
                   "Port": "11002",
                   "State": "RUNNING"
           }
   ]
   

network performance problem

aergo uses DPOS as a consensus algorithm. Blocks generated by one node must be transmitted to other nodes at a high speed. If the block transmission is too slow due to the network problem, the block created by the BP will not be included in the chain.

You can track the block creation time and the time passed to other nodes.

Oct 30 15:50:57.009 |INFO| block produced BP=16Uiu2HAmF5fpuk9ZWQbzHUuu9C1jwtqKnZdtkQYYDapA7AyoibWP caller=/Users/everjs/go/src/github.com/aergoio/aergo/consensus/impl/dpos/blockfactory.go:213 confirms=2 id=6mdjTYFEKXJ4UQ3xdKEu5RybnVWh3Hio6VG8a5SAXPbp lpb=1356 module=dpos no=1358 sroot=9hbmSFxzGj58YQuM2642zyo8jDAuu5qTWNG8JCMb78iZO`

Oct 30 15:50:57.012 |INFO| added block successfully. best=1358

Oct 30 15:50:57.013 |DEBUG| add block chainservice blockNo=1358 caller=/Users/ev
erjs/go/src/github.com/aergoio/aergo/chain/chainservice.go:252 hash=6mdjTYFEKXJ4UQ3xdKEu5RybnVWh3Hio6VG8a5SAXPbp module=chain

Server Process Hang

The aergo server was developed using the proto actor framework. Each module in the server receives a request using a message, performs an action on it, and returns a response.

At this time, if a specific module does not operate normally and hangs, it can be checked with the following command.

aergocli -p 10001 node
{
        "AccountsSvc": {
                "status": "started",
                "acc_processed_msg": 202,
                "msg_queue_len": 0,
                "msg_latency": "55.623µs",
                "error": "",
                "actor": null
        },

...

}

If a particular module hangs, the value of msg_queue_len field is high.

The details can be analyzed through the log by increasing the log level of the module.

Tracking status of Transaction

A transaction is added to a mempool and then executed when it is included in a block. Transactions are not included in block if validation fails due to nonce, balance, etc.. All transactions in the block will have a receipt. If the VM fails to execute contract, the error is recorded in the receipt. Therefore, the state of transaction may be as follows.

• State of transaction
  • pending in mempool
  • included in block
    • sucess
    • fail if contract

check status of transaction

Use the following command to track the status of the transaction.

aergocli -p $PORT gettx $TXHASH

If the transaction stays in the pending state in the mempool, the following status is output.

aergocli -p $PORT gettx $TXHASH
Pending:  {
 "Hash": "BTY3zrsBkVHTqFbpqKtwbXwgfpGwRoiAmKRrdFKfZRna",
 "Body": {
  "Nonce": 101,
  "Account": "AmM5wCL3XhDngGAyYk6pbd2AYKSomVAKRqLBhnP8QsBMRRoA62fv",
  "Recipient": "AmNeY5fDqvGAfpYYLxhy7AhSsFzCSL96iYBApR2aHvRzMY5D34Ky",
  "Amount": 2560000,
  "Payload": "",
  "Limit": 0,
  "Price": 0,
  "Type": 0,
  "Sign": "381yXZRAPSGAFbGYndN5xU3s9RXHddFbCxyutNtUYvyT4DbDt7jWyia4oM6CLGtXYeQqtQBvqTaNU2gwpdrCm5FjwFP6fZCQ"
 }
}

check account

First, check the nonce and balance of the corresponding account. The status of the current account can be seen by the command below.

aergocli -p $PORT  getstate --address $ADDRESS

Ensure that the nonce and balance of transaction are consistent with the account status.

check status of mempool

There are too many transactions in mempool, which can take a long time to be included in the block. To see the current status of mempool, try the following command.

aergocli -p $PORT  node

In the cache_len field, you can see the total number of tx’s in mempool.

check receipt

If the transaction is contained in a block, you can use the receipt to check the execution result. Receipt can be seen using transaction hash

aergocli -p $PORT receipt get $TXHASH
{
 "contractAddress": "AmNeY5fDqvGAfpYYLxhy7AhSsFzCSL96iYBApR2aHvRzMY5D34Ky",
 "status": "CREATED",
 "ret": ""
}