Emergency guide for Gnosis staking - what to do if anything goes wrong

How to consider next steps and possibly stop the staking and withdraw locked funds?

Disclaimer

This guide is for informational purposes only. The author nor website owner does not guarantee accuracy of the information in this guide and is not responsible for any damages or losses incurred by following the guide.

Introduction

Staking, or validating, involves participating in the security and operation of Gnosis blockchain network. Validators are rewarded for performing this continuous task, but they can also be penalized for failing to participate. The goal is to always be an active and valid member of the network. To do so, some necessary actions may be taken from time to time to keep getting rewards and protect locked capital.

An action is necessary especially in 3 cases:

  1. Validator(s) are down or have a poor performance.

    It's essential to use some kind of monitoring of your validators performance to be sure that they are online and valid (you earn standard rewards). Otherwise, you are penalized. You can do the monitoring either on the local level, or better, externally through online services such as the Stakers.space validator monitor or Gnosischa.in dashboard. Both services can notify you imediatelly in case of a validator failure, allowing you to address as soon as possible.

  2. An important client update is released.

    You can sign-up for Github notifications onnew client versions releases, or you can get these notifications from online services such as the Stakers.space validator monitor or Gnosischa.in dashboard. again. In a case of important releases there is required to process the client update on your staking machine to remain on the proper fork chain.

  3. You lost access to your staking server.

    If you have your server physically accessible, you can re-initializate your node, otherwise you must exit the validators.

Why the action is required?

As mentioned already, for a reason of capital protection. Each validator is associated with a locked collateral for that you may be deducted if the validator does not comply properly with the network. For that reason, in a long-run, it's unprecedented to keep you validator either online and valid, or exited.

If you do not do anythng, a penalties are gradually applied and deducted from a locked collateral. Once the colaterral falls to 50%, the validator will be exited, remaining funds unlocked and send to a withdrawal wallet.

Validator is down or have a poor performance

This issue can be caused for numerous reaons:

  • Internet connection issue
  • Power Failure
  • Client issue
  • Hardware issue

Step by step guide to diagnose and fix the issue deal with following:

  1. Connect to your staking node

    • If you are able to log-in and access the server, skip at step 2 - Check MOTD.
    • If you are not able to connect your server, check the following:
      • Computer is turned on (= no Power Failure) and loaded (no Hardware issue)
      • There's no internet connection issue

      If both are correct, skip at step 2 - Check MOTD.

  2. Check MOTD (Message of the Day)

    MOTD is the first screen you see after succesfull log-in to your staking machine. Check especially following:

    • Uptime: Make sure there hasn't been a recent power outage.
    • Last login: Make sure there hasn't been unauthorized access to your staking node.
    • SSD usage: Make sure there is enough free disk space.

  3. Check server time

    Check that server time is being siynchronized

    timedatectl status
    System clock synchronized must have value yes. If not, run timedatectl set-ntp true.

  4. Check clients statuses

    Usually, Loaded state instead of Active on a client service may occur after a system reboot with missing autostart setup. Status of any service may be checked with a use of following command:

    systemctl status <ServiceName>

    ToDo: Check statuses of all staking services running on the server. If any service is not active, launch it with command sudo systemctl start <ServiceName>.

  5. Check clients logs for issues

    Most common issue is an issue of any client. You can check log of any client service with following command:

    journalctl -f -u <ServiceName>

    Based on the log you may find out several issues such as:

    • No peers → check internet connection issue
    • Stucked client (no execution/consensus client listening) → try restart the client
    • DB issue/corruption = try stop and start the client. If it does not help, remove client database and resync.

  6. Perform the Best possible action

    You should consider how long it will take you to fix the problem and run the validator(s) again. If you are not to do this within maximally few weeks, you should consider exiting your validators and launching new once your staking node will be ready again.

  • Check clients logs for issues
  • Check number of connected peers. If you sue VPN, consider switching at a different VPN server.
  • Check hardware components usage and temperature
  • Check server time

An important client update was released (Update is required)

Update the client. Firstly, find out what clients are used and in what way. If native mode is used, an update may be used based on the following guides for updating following popular clients and services:

Execution clients:

Consensus clients:

Services clients:

No/Lost access to staking server

First rule of staking Gnosis is to avoid running the same validator on two machines at the same time. Such beahviour would lead to double attestation and a significant penalty called "slashing" associated with forced exit. From here:

  • If you have physical access to your server and you can be sure the validators on it will never go online through the server (e.g. due to damaged disk or removed keystores from the disk), you can relaunch the validators on another machine.
  • If you have no control above the server with keystores (e.g. someone stole it), you must exit validators.

(Re)Generating keystores

Validators are represented by keystores. Keystores (or validation keys) are generated from a seed. The seed, as well as keystores, is generated through Gnosis Chain validator's data generator client. The software has more use-cases:

  • Generation of a seed + first keystores from the seed (+ deposit keys file)
  • From the seed, regeneration of keystores or generation of more keystores (+ deposit keys file)

During a generation process, you will have to provide following:

  • An information, whether you want to generate new keystores (you want to generate seed as well), or generate keystores from a seed. The second option is right in this case, as you need to regenerate lost keystores. To do so, use a command 
    ./deposit existing-mnemonic

    On request, provde the seed based on which the keystores will be regenerated.

  • --num_validators - information for how many validators the keystores should be generated
  • --validator_indices - keystore index. As a seed may be associated with many validator keystores, each keystore is being identified through own index. Each index generates always the same keystores, and this is the way how keystores can be regenerated. Simply, set keystores value of the keystores you lost. On Stakers.space account, we allow to save encrypted information about used indexes for each instance. If you do not know the the index (starts at 0 in default, - so if you had 2 validators, it was 0 and 1 in standard setup), you can generate e.g. 1000 keystores for indexes starting at --validator_start_index 0 (= from 0 to 1000). With a new keystores generation, you will be requested to provide a new keystore password. As this password serves for decryption / encryption of the generated keys only, it does not need to be same as before - simply set your own password.

Keep the seed and validator keystores with password private. Do not share neither of these with anyone, as knowledge either of the seed or owning the keystores with password allows to start a validator on any server and leads to a risk of slashing (happens when the same validator contributes to the network more times at the same time (= is launched paralelly on 2 machines or instances)).

For loading a validator to a validator client on the staking server, you will need the validator keystore(s) with the password. The seed serves especially as a backup for regenerating the keystores (if you would lose them or the password).

So far you have either the seed or the keystore(s) with passoword, you can move your validators between staking machines as well as exit them. However, if you lose both, you are no more access for your validators.

It's always a good practice to have both and do not rely at one only. Keystore(s) with passwrds can be reinitialized from seed anytime in a case you lose them. However, if you lose your seed, there's no way to find out it from the generated keystores. Thus, be always sure you have your seed, and if not, consider exiting your validators and reinitialized them from a new seed you will keep better than the previous.

Exiting Gnosis validator(s)

There are numerous ways to axit Gnosis validator:

Broadcast presigned exit message

If you have presigned exit messages, or you are able to generate them, you can simply broadcast them to the beaconchain through the https://gnosischa.in/tools/broadcast tool on the Gnosischa.in website.

Through the tool, you can send signed messages to the network just like from a server.

The presigned messages for validation keys can be created using the tool Ethdo. The signed keys are then simply uploaded to the broadcasting tool.

Guide

You can generate presigned withdrawal keys yourself according to the guide, or you can request them from your staking manager who operates a node for you.

Exit from the server where the validators are active / registered

In a case you have access to a server your validators are active on and your beaconchain client is synchronized, you can process the exit right from the consensus client on the server, see below:

Exit from a different server than where the validators are active

You can follow the Full staking guide to launch an Execution client and Beacon client on a new machine. Then import your keystores (guide in Validators part), but instead of configuring the validator client and starting it, simply run the exit command.

From a speed and simplicity, I recommend to use a combination of Nethermind + Lodestar client.