CRR Monitoring

Starting from TigerGraph 4.3, the Disaster Recovery (DR) cluster supports Cross-Region Replication (CRR) status, metrics collection and query capabilities.

The CRR metrics collection interval can be configured using the following command (default: 60s):

gadmin config entry System.Metrics.CRRIntervalSec

1. CLI-based CRR status query

Users can run a gadmin command to get detailed CRR metrics.

gadmin crr status [flags]

The command provides comprehensive CRR status information, including:

  • Overall CRR system status

  • Topic replication and replay lag

  • MirrorMaker 2 connector status

1.1. Flags

The following flags can be used with the gadmin crr status command.

Flag Description

--json

Outputs CRR status in JSON format.

--realtime

Retrieves real-time status instead of cached data.

-v, --verbose

Displays detailed metrics, including source and target offsets and timestamps.

1.2. CRR Topic Metrics

The following table provides the definitions of topic metrics.

Field Name Description Notes Shown When

replicationLagRecords

Number of records the DR topic is behind the corresponding PR topic.

Calculated as sourceOffset - replicatedOffset

always

replayLagRecords

Number of records the replay process in DR is behind the replicated topic.

Calculated as targetOffset - replayedOffset

always

replicationLagSeconds

Time lag in seconds between the latest record in PR and its replication in DR.

Calculated as sourceTimestamp - replicatedTimestamp

always

replayLagSeconds

Time lag in seconds between the latest record in DR and its replay.

Calculated as targetTimestamp - replayedTimestamp

always

replicationStalledSeconds

Duration in seconds that replication from PR to DR has made no forward progress.

0 means replication is actively progressing.

always

replayStalledSeconds

Duration in seconds that replay within DR has made no forward progress.

0 means replay is actively progressing.

always

sourceOffset

Latest offset available in the PR Kafka cluster.

Highest offset produced in the PR topic.

--verbose

replicatedOffset

Offset up to which data has been replicated from PR to DR.

Highest offset confirmed written to the DR topic.

--verbose

targetOffset

Latest offset available in the DR Kafka topic.

Highest offset physically present in the DR topic.

--verbose

replayedOffset

Offset up to which the replay service in DR has processed records.

Highest offset successfully replayed in DR.

--verbose

sourceTimestamp

Timestamp of the record at sourceOffset in PR.

Based on Kafka message timestamp.

--verbose

replicatedTimestamp

Timestamp of the record at replicatedOffset in DR.

Based on Kafka message timestamp.

--verbose

targetTimestamp

Timestamp of the record at targetOffset in DR.

Based on Kafka message timestamp.

--verbose

replayedTimestamp

Timestamp of the record at replayedOffset in DR.

Based on Kafka message timestamp.

--verbose

2. OpenMetrics Integration

CRR topic metrics can also be collected in OpenMetrics format using the following API:

curl http://localhost:14240/informant/metrics

The following table defines the CRR metrics available in OpenMetrics format.

Type Name Label Description

Metric

tigergraph_crr_replication_lag_records

topic

Replication lag (records)

Metric

tigergraph_crr_replay_lag_records

topic

Replay lag (records)

Metric

tigergraph_crr_replication_lag_seconds

topic

Replication lag (seconds)

Metric

tigergraph_crr_replay_lag_seconds

topic

Replay lag (seconds)

Metric

tigergraph_crr_replication_stalled_seconds

topic

Duration replication has stalled

Metric

tigergraph_crr_replay_stalled_seconds

topic

Duration replay has stalled

Metric

tigergraph_crr_source_offset

topic

Source offset in PR

Metric

tigergraph_crr_replicated_offset

topic

Replicated offset in DR

Metric

tigergraph_crr_target_offset

topic

Target offset in DR topic

Metric

tigergraph_crr_replayed_offset

topic

Replayed offset in DR

Metric

tigergraph_crr_source_timestamp

topic

Timestamp of source record in PR

Metric

tigergraph_crr_replicated_timestamp

topic

Timestamp of replicated record in DR

Metric

tigergraph_crr_target_timestamp

topic

Timestamp of target record in DR

Metric

tigergraph_crr_replayed_timestamp

topic

Timestamp of replayed record in DR

TigerGraph provides raw metrics for CRR topic Replication and Replay, but does not include built-in alerting.

Replication and replay lag may vary depending on cluster configuration, network bandwidth, and latency. Users are advised to define alerting rules in their own monitoring systems (for example, Prometheus).

For example, a CRR topic can be considered abnormal if any of the following conditions occur:

  • tigergraph_crr_replication_lag_records > 20

  • tigergraph_crr_replay_lag_records > 20

  • tigergraph_crr_replication_stalled_seconds > 60

  • tigergraph_crr_replay_stalled_seconds > 60

These thresholds can be adjusted according to the environment and operational requirements.