Fail over to the DR cluster

In the event of catastrophic failure that has impacted the full cluster due to Data Center or Region failure, the user can initiate the failover to the Disaster Recovery (DR) cluster.

Starting from TigerGraph 4.3, failover to a Disaster Recovery (DR) cluster is simplified into a single command. The new --promote flag for gadmin crr stop replaces the previous multi-step manual process, making DR promotion faster and more reliable. Update existing failover scripts to use this new command.

gadmin crr stop --promote --dump-checkpoint /path/to/checkpoint.json -y

Key Behavior Changes and Enhancements

1. Single Command Operation

The gadmin crr stop --promote command now orchestrates the entire promotion process. It replaces four legacy commands and automates the following actions:

  • Stops the CRR connector.

  • Updates the configuration parameter: System.CrossRegionReplication.Enabled=false

  • Applies configuration changes and restarts dependent services.

2. Automated Safety Checks for Data Consistency

Before stopping replication, the command automatically verifies synchronization between the DR and Primary clusters.

  • If replication lag is detected, the system displays a warning indicating the DR cluster is not fully synchronized.

  • Promoting a lagging DR cluster can result in data loss. If lag is detected, the command pauses and requests explicit confirmation before proceeding.

  • If no lag exists, promotion continues automatically without user intervention.

3. Integrated Replication Checkpointing

The --dump-checkpoint flag generates a replication checkpoint file that records precise replication offsets at the moment of promotion.

  • When the DR cluster is fully synchronized, the checkpoint file is saved automatically to the specified path.

  • This checkpoint enables a fast-switch operation when converting the old Primary cluster into a new DR cluster—avoiding a full backup and restore.

  • See Set Up a New DR Cluster After Failover for instructions on using the checkpoint for a fast-switch.

4. Forced Promotion for Emergency Scenarios

In urgent cases, administrators can use the --force flag along with --promote to bypass synchronization checks.

gadmin crr stop --promote --force -y

Use this option only during emergencies. Forced promotion may lead to data loss and will require a full backup and restore to re-establish the DR cluster. An emergency refers to scenarios where the Primary Cluster is not available or has experienced a critical failure, and requires immediate promotion of the Disaster Recovery (DR) cluster to Primary to maintain service continuity.

Users must update existing failover scripts and operational runbooks. The older four-step failover process is now deprecated and should be replaced with the single gadmin crr stop --promote command.

This update improves safety, reduces operational complexity, and ensures more reliable failover operations.

Set Up a New DR Cluster After Failover

Overview

TigerGraph 4.3 introduces a new fast-switch capability that allows administrators to quickly convert the former Primary cluster into a new DR cluster. This method leverages the replication checkpoint generated during failover, reducing downtime and avoiding a full backup and restore.

After promoting the DR cluster to Primary, the former Primary becomes offline. To re-establish high availability, configure the old Primary as the new DR cluster.

Principle: Checkpoint Validation for Fast-Switch

The fast-switch mechanism depends on whether the DR cluster was fully synchronized with the Primary at the time of failover. This is determined by validating the replication checkpoint file created during promotion.

gadmin crr stop --promote --dump-checkpoint <path>

Validation outcomes:

  • Validation passes: Clusters were fully synchronized. A fast-switch occurs, converting the cluster to a DR instance instantly without data transfer.

  • Validation fails: The failover occurred during replication lag (e.g., when --force was used or when new data is inserted into the old Primary Cluster after the failover). A fast-switch is not possible, and a full backup and restore is required.

Method 1: Fast-Switch Using a Replication Checkpoint (Recommended)

Use this method when:

  • The failover was clean (no replication lag detected).

  • A valid replication checkpoint file (e.g., checkpoint.json) was created successfully.

Steps

  1. Configure the old Primary cluster as the new DR cluster.

    # Set the IPs of the new Primary cluster
gadmin config set System.CrossRegionReplication.PrimaryKafkaIPs <new_primary_ip1,...>

# Set the Kafka port of the new Primary cluster
gadmin config set System.CrossRegionReplication.PrimaryKafkaPort <new_primary_kafka_port>

# Set the topic prefix. Add an additional ".Primary" suffix for each failover.
gadmin config set System.CrossRegionReplication.TopicPrefix <prefix>

# Apply configuration changes
gadmin config apply

  2. Perform the fast-switch restore.

    gadmin backup restore --dr --dr-checkpoint /path/to/checkpoint.json

    If validation succeeds, the cluster is reconfigured as a DR instance and replication resumes from the exact checkpoint position.

Method 2: Full Backup and Restore (Fallback Method)

Use this method when:

  • The checkpoint validation fails (clusters were not synchronized).

  • The checkpoint file is missing.

  • The DR cluster is being deployed on new hardware.

Steps

  1. Create a backup on the new Primary cluster.

    gadmin backup create <backup_name>

  2. Configure the new DR cluster.

    Follow the configuration steps in Method 1 to point the DR cluster to the new Primary.

  3. Restore the backup to the new DR cluster.

    gadmin backup restore <backup_tag> --dr