Operational Tasks

General Management

Introduction

Validator performance is critical to the security and stability of the Taker Chain network. As a validator, your role involves optimizing performance, minimizing latency, and ensuring high availability, especially during periods of high demand. Proper configuration, proactive monitoring, and robust security practices are essential to avoid penalties such as slashing and to maintain a reliable validator node.

This guide outlines the key operational tasks for managing your validator, including performance tuning, security hardening, and real-time monitoring. By following these practices, you can ensure optimal performance and protect your validator from potential risks.

Configuration Optimization

To ensure your validator operates efficiently, you can apply several performance optimizations at the system level. These configurations reduce latency, improve responsiveness, and maintain consistency during high-demand periods.

Deactivate Simultaneous Multithreading (SMT)

Validators on Taker Chain primarily perform single-threaded operations for critical tasks. Disabling SMT (also known as Hyper-Threading on Intel CPUs or 2-way SMT on AMD CPUs) can improve single-core performance and reduce processing latency.

Steps to Disable SMT

  1. Loop through all CPU cores and deactivate virtual cores:

    for cpunum in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | \
cut -s -d, -f2- | tr ',' '\n' | sort -un)
do
echo 0 > /sys/devices/system/cpu/cpu$cpunum/online
done

  2. Permanently disable SMT by adding nosmt=force to the GRUB configuration:

    sudo nano /etc/default/grub

    Add the following to the GRUB_CMDLINE_LINUX_DEFAULT line:

    GRUB_CMDLINE_LINUX_DEFAULT="nosmt=force"

  3. Update GRUB and reboot:

    sudo update-grub
sudo reboot

  4. Verify SMT is disabled:

    lscpu --extended

Deactivate Automatic NUMA Balancing

For multi-CPU setups, disabling NUMA (Non-Uniform Memory Access) balancing ensures that processes remain on the same CPU node, reducing latency caused by memory access across nodes.

Steps to Disable NUMA Balancing

  1. Temporarily disable NUMA balancing:

    sysctl kernel.numa_balancing=0

  2. Permanently disable NUMA balancing by editing the GRUB configuration:

    sudo nano /etc/default/grub

    Add the following to the GRUB_CMDLINE_LINUX_DEFAULT line:

    GRUB_CMDLINE_LINUX_DEFAULT="numa_balancing=disable"

  3. Update GRUB and reboot:

    sudo update-grub
sudo reboot

  4. Confirm NUMA balancing is disabled:

    sysctl -a | grep 'kernel.numa_balancing'

Optimize Spectre and Meltdown Mitigations

Spectre and Meltdown mitigations in modern Linux kernels can slightly reduce performance. If your system does not require these mitigations for security reasons, you can selectively disable them to improve performance.

Steps to Optimize Mitigations

  1. Edit the GRUB configuration:

    sudo nano /etc/default/grub

    Add the following to the GRUB_CMDLINE_LINUX_DEFAULT line:

    GRUB_CMDLINE_LINUX_DEFAULT="spec_store_bypass_disable=prctl spectre_v2_user=prctl"

  2. Update GRUB and reboot:

    sudo update-grub
sudo reboot

  3. Verify the changes: Check the kernel boot parameters to ensure the mitigations are disabled.

Monitor Your Node

Real-time monitoring is essential to ensure your validator operates reliably and remains in sync with the network. The following tools can help you monitor node performance and set up alerts for potential issues:

  • Prometheus: Collects and queries time-series data.

  • Grafana: Provides interactive dashboards for visualizing metrics.

  • Alertmanager: Manages and routes alerts based on Prometheus data.

Set Up Prometheus

  1. Install Prometheus:

    • Download the latest Prometheus release from the official repository.

    • Extract and move the binaries to /usr/local/bin/.

  2. Configure Prometheus:

    • Create a configuration file (/etc/prometheus/prometheus.yml) to define global settings, scrape intervals, and targets.

    • Example configuration:

      global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'taker_validator'
    static_configs:
      - targets: ['localhost:9615']

  3. Run Prometheus:

    • Start Prometheus manually or configure it as a systemd service to ensure it starts on boot.

  4. Verify Installation:

    • Access the Prometheus web interface at http://<SERVER_IP>:9090.

Set Up Grafana

  1. Install Grafana:

    • Add the Grafana repository and install the latest version.

  2. Configure Data Sources:

    • Add Prometheus as a data source in Grafana.

  3. Create Dashboards:

    • Import pre-built dashboards or create custom dashboards to visualize node metrics.

  4. Access Grafana:

    • Open the Grafana web interface at http://<SERVER_IP>:3000.

Set Up Alertmanager

  1. Install Alertmanager:

    • Download and configure Alertmanager.

  2. Configure Alerts:

    • Define alert rules in Prometheus (/etc/prometheus/rules.yml) to monitor critical metrics, such as node downtime or high CPU usage.

  3. Integrate with Grafana:

    • Add Alertmanager as a data source in Grafana to visualize and manage alerts.

Secure Your Validator

Security is paramount for validators, as any compromise could result in slashing or loss of funds.

Key Management

  • Session Keys: Generate and securely store session keys using the rotateKeys RPC call or external tools.

  • Node Keys: Ensure your node key is static and securely stored to maintain a consistent network identity.

System Security Best Practices

  • Use a non-root user for all validator operations.

  • Regularly apply OS security updates.

  • Enable a firewall and restrict access to essential ports (e.g., P2P port 30333).

  • Use SSH key-based authentication and disable password-based login.

  • Configure automatic backups for critical data.

Validator Best Practices

  • Run only the Taker Chain binary on your validator server.

  • Use bare-metal servers for better performance and reliability.

  • Automate node provisioning and configuration using tools like Ansible or Terraform.

  • Set up a robust alert system and define clear escalation protocols for handling issues.

Upgrade a Validator Node

Introduction

Upgrading a validator node on Taker Chain is essential to stay up-to-date with network updates, maintain compliance, and ensure optimal performance. Whether you are performing routine software upgrades or major server changes, a carefully planned upgrade process is critical to avoid downtime, slashing, or network disruptions.

This guide provides step-by-step instructions for managing session keys, securely transitioning between validator nodes, and performing maintenance operations. By following these instructions, you can seamlessly substitute an active validator server while maintaining uninterrupted service. The entire process may take several hours, so plan accordingly.

Prerequisites

Before upgrading your validator node, ensure the following:

  • You have a fully functional validator setup with all required binaries installed. Refer to the Set Up a Validator and Validator Requirements sections for additional guidance.

  • Your infrastructure supports running a secondary validator instance temporarily during the upgrade process.

  • You understand the importance of session keys and their delayed effect when switching validators.

Session Keys

Session keys are critical for signing validator operations and linking your validator node to your staking proxy account. Key points to remember:

  • Delayed Effect: Any changes to session keys only take effect after the current session ends and two additional sessions have passed.

  • Key Management: Always generate new session keys for each validator instance. Never reuse or duplicate session keys, as this can lead to equivocation slashing.

For more details on session keys, refer to the Keys section of the documentation.

Keystore

The keystore folder on your validator server contains private keys used for signing network-level transactions. Proper keystore management is crucial to avoid slashing penalties:

  • No Duplication: Never copy or transfer the keystore folder between validator instances. Doing so could result in multiple validators signing with the same keys, leading to equivocation slashing.

  • Default Keystore Path:

    /home/<user>/.local/share/polkadot/chains/<chain>/keystore

  • Best Practice: Always generate new session keys for each validator instance to ensure secure and independent operation.

Upgrade Using Backup Validator

To perform maintenance or upgrades, you can temporarily switch between two validator nodes. The process involves two validators:

  • Validator A: The original active validator.

  • Validator B: A backup node used for maintenance purposes.

Session N

  1. Start Validator B:

    • Launch a secondary node (Validator B) and wait until it is fully synced with the network.

    • Start Validator B with the --validator flag to enable validator mode.

  2. Generate Session Keys for Validator B:

    • Create new session keys specifically for Validator B.

  3. Submit the set_key Extrinsic:

    • Use your staking proxy account to submit a set_key extrinsic, linking Validator B's session keys to your staking setup.

  4. Record the Session:

    • Note the session in which you executed the set_key extrinsic.

  5. Wait for Session Changes:

    • Allow the current session to end and wait for two additional full sessions for the new keys to take effect.

  6. Keep Validator A Running:

    • It is critical to keep Validator A operational during this waiting period, as the set_key extrinsic does not take effect immediately. Turning off Validator A too early may result in chilling or slashing penalties.

Session N+3

At this stage, Validator B becomes the active validator, allowing you to safely perform maintenance tasks on Validator A.

Steps to Transition Back to Validator A

  1. Start Validator A:

    • Launch Validator A, sync the blockchain database, and ensure it is running with the --validator flag.

  2. Generate New Session Keys for Validator A:

    • Create fresh session keys for Validator A.

  3. Submit the set_key Extrinsic:

    • Use your staking proxy account to submit a set_key extrinsic with Validator A's new session keys.

  4. Record the Session:

    • Note the session in which you executed the set_key extrinsic.

  5. Keep Validator B Active:

    • Keep Validator B operational until the session during which you executed the set_key extrinsic completes, plus two additional full sessions.

  6. Verify the Transition:

    • Check the logs to confirm that Validator A has successfully taken over. You should see messages indicating the authority set change, such as:

      2019-10-28 21:44:13 Applying authority set change scheduled at block #450092
2019-10-28 21:44:13 Applying GRANDPA set change to new set with 20 authorities

  7. Stop Validator B:

    • Once Validator A has successfully resumed its role, you can safely stop Validator B.

Best Practices for Validator Upgrades

  • Plan Ahead: Ensure you understand the process and schedule the upgrade during a low-traffic period.

  • Monitor Progress: Use monitoring tools like Prometheus and Grafana to track the status of your validators during the transition.

  • Test Backup Nodes: Regularly test your backup validator setup to ensure it is ready for use when needed.

  • Avoid Downtime: Always keep at least one validator operational to avoid penalties or slashing.

Pause Validating

Introduction

If you need to temporarily step back from Taker Chain staking activities without fully unbonding your funds, chilling provides an efficient solution. Chilling removes your node from active validation or nomination in the next era while keeping your funds bonded. This is particularly useful for planned downtimes, maintenance, or temporary pauses in staking participation.

This guide details how to chill as a validator or nominator, the steps involved, and the implications of chilling on your staking status and nominations.

Chilling Your Node

Chilling pauses your active staking participation and sets your account to inactive in the next era. However, your funds remain bonded, making it easier to resume staking activities when you’re ready.

Staking Election Timing Considerations

The timing of your chilling decision affects your eligibility for the next staking election:

  • Chilled During the Previous Era:

    • Your account will not participate in the current era election and will remain inactive until reactivated.

  • Chilled During the Current Era:

    • Your account will not be selected for the next era's election.

  • Chilled After the Current Era:

    • If your account was active during the previous era and is chilled now, it may still be selected for the current era.

Chilling as a Nominator

When you chill as a nominator, your active nominations are reset. Key points to keep in mind:

  • Manual Reselection:

    • Upon re-entering the nominating process, you must manually select validators to support. These can be the same validators as before or a new set.

  • Bond Preservation:

    • Your account remains bonded while chilled, preserving your staked funds without requiring a full unbonding process.

  • Reactivating Nominations:

    • When ready to start nominating again, issue a new nomination call to activate your bond with a fresh set of validators. This allows you to maintain your stake while adjusting your involvement in active staking.

Chilling as a Validator

When you chill as a validator, your active validator status is paused. The following effects and considerations apply:

  • Existing Nominators:

    • Nominators who are bonded to you remain bonded, but your validator bond will no longer appear as an active choice for new or revised nominations until reactivated.

    • If nominators adjust their stakes while you are chilled, they will not be able to nominate your validator until it resumes activity.

  • Reactivation Requirements:

    • Upon reactivating as a validator, you must reconfigure your validator preferences (e.g., commission rate and other parameters).

    • These preferences can match your previous configuration or be updated as desired.

  • Eligibility for Nominations:

    • Reactivating as a validator restores your eligibility for nominations and allows you to rejoin the active validator set.

Chill Other

Historically, the staking system included constraints to manage the number of nominators and validators. One such mechanism was the chillOther extrinsic, which allowed users to chill accounts that no longer met minimum staking requirements.

Changes with Referendum #90

  • With the passage of Referendum #90, the maxNominatorCount on Polkadot was set to None, effectively removing the limit on how many nominators and validators can participate.

  • As a result, the ChillThreshold (a parameter that defined when users could chill one another) will never be met.

  • Impact: The chillOther extrinsic no longer has any effect and is effectively obsolete.

Best Practices for Chilling

  • Plan Ahead:

    • If you anticipate downtime or maintenance, chill your account in advance to avoid disruptions in staking participation.

  • Monitor Timing:

    • Understand how the timing of your chilling decision affects your eligibility for upcoming staking elections.

  • Communicate with Nominators:

    • Notify your nominators if you plan to chill as a validator, so they can make informed decisions about their nominations.

  • Secure Reactivation:

    • When reactivating, double-check your session keys and validator configurations to ensure a smooth transition back into active staking.

By using chilling effectively, you can temporarily pause your staking participation while keeping your funds bonded and avoiding the need for a full unbonding process. This flexibility ensures that you can manage planned downtimes or adjust your staking strategy without incurring penalties or losing your staking position.

PreviousOnboarding and OffboardingNextNominator

Last updated