---
title: "How to check and correct substantial NTP drift"
slug: "how-to-check-and-correct-substantial-ntp-drift"
description: "Resolve time drift issues in CNode or DNode hardware by adjusting NTP settings and restarting chronyd for quick synchronization."
updated: 2026-10-27T19:37:32Z
published: 2026-10-28T19:52:20Z
canonical: "docs.vastdata.com/how-to-check-and-correct-substantial-ntp-drift"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How to check and correct substantial NTP drift

## Summary

If CNode or DNode hardware has been idle for long periods, the onboard RTC clock may drift substantially from the current time. Once a substantial deviation (>5-10 min) occurs, `chronyd` will not be able to step or slew the system clock in a reasonable amount of wall clock time (it takes weeks to years) to bring the nodes into time sync, and potentially will not be able to accept the response from an NTP server. The quickest solution is to:

1. Disable NTP on the affected nodes.
2. Manually set the time.
3. Re-enable NTP.
4. Restart `chronyd` on ALL nodes.

This will bring the node to within a few minutes of the current NTP time and speed up synchronization.

## Symptoms

During an Install, Expansion, Upgrade, Node Replacement or other task, the installer is asked to run the following commands:

### Configure the time zone and time

```plaintext
# Restart chronyd. This ensures chrony is actually using the configured NTP servers and not the OS default.
clush -a sudo systemctl restart chronyd

# Set UTC and confirm time is in sync (NTP should be working).
clush -a sudo timedatectl set-timezone UTC
clush -aB date
```

Example Output

```plaintext
---------------
172.16.3.[1-12,100-105] (18)
---------------
Sat Apr 30 03:38:18 GMT 2022
```

## Issue

In some circumstances, the new node(s) may have substantial time drift.

```plaintext
clush -aB date
```

Example Output

```plaintext
---------------
172.16.3.[1-8,100-105] (14)
---------------
Sat Apr 30 03:38:18 GMT 2022

---------------
172.16.3.[9-12] (4)
---------------
Sat Apr 30 03:05:18 GMT 2022
```

As shown in the example above, nodes 9-12 have drifted by 33 minutes from the current NTP time.

## Solution

First, double-check all configurations and verify that NTP servers are reachable.

### Confirm `chrony` config

Confirm that all configs have the same `chrony` servers and they are correct:

```plaintext
clush -aB grep ^server /etc/chrony.conf
```

If they aren't the same or they are wrong, make them the same and/or correct them and restart `chronyd` again.

### Ping the NTP Server(s)

***Change NTP1/2 to your customer’s NTP server addresses.***

```plaintext
ping -c 4 -i 0.2 -M do -s 1000 NTP1 |grep loss
ping -c 4 -i 0.2 -M do -s 1000 NTP2 |grep loss
```

If you cannot reach the NTP servers, there may be an issue on the Management Network.

### Disable NTP, manually set the time, then re-enable NTP and restart `chronyd`

```plaintext
# Disable NTP on all affected nodes
clush -w 172.16.3.[9-12] 'sudo timedatectl set-ntp false'

#Manually set the 
clush -w 172.16.3.[9-12] 'sudo timedatectl set-time "2019-06-22 13:41:00"'

# Enable NTP on all affected nodes
clush -w 172.16.3.[9-12] 'sudo timedatectl set-ntp true'

# Restart chronyd on all nodes to force an NTP sync.
clush -a 'sudo systemctl restart chronyd'

Check you work
clush -aB date
```

Example Output

```plaintext
---------------
172.16.3.[1-12,100-105] (18)
---------------
Sat Apr 30 03:38:18 GMT 2022
```

Drifts of < 5 minutes should resolve within 3-5 minutes of restarting `chronyd`. If issues persist after following these steps, collect all logs, SSH terminal history, and contact CS.