Time Synchronization
Why This Matters
A financial trading system executes a transaction at 14:32:07.003. The audit log on a different server records it at 14:32:08.147. A third server shows 14:31:59.891. When regulators ask for the exact sequence of events, nobody can say which happened first. This is not a hypothetical scenario -- it has caused real regulatory fines.
Or consider this: Kerberos authentication, used in Active Directory environments, rejects tickets if the time difference between client and server exceeds 5 minutes. Your users cannot log in to their desktops because a server's clock drifted by 6 minutes over the weekend. All because nobody set up time synchronization.
Accurate time is not a luxury. It is a dependency for:
- Log correlation: Matching events across multiple servers
- Authentication: Kerberos, TOTP (two-factor auth), TLS certificates
- Distributed systems: Database replication, consensus protocols
- Compliance: Financial regulations, healthcare audit trails
- Backups: Determining which files have changed
- Cron jobs: Running tasks at the right time
Getting time right is one of those things that seems trivial until it goes wrong. This chapter ensures it never goes wrong on your systems.
Try This Right Now
Check your system's time configuration in 30 seconds:
$ timedatectl
Expected output:
Local time: Sat 2025-06-15 14:32:07 UTC
Universal time: Sat 2025-06-15 14:32:07 UTC
RTC time: Sat 2025-06-15 14:32:07
Time zone: UTC (UTC, +0000)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
The critical lines:
- System clock synchronized: yes -- your clock is being synchronized
- NTP service: active -- an NTP client is running
- RTC in local TZ: no -- the hardware clock stores UTC (correct for servers)
If "System clock synchronized" shows "no," your system's time is drifting without correction. Fix it by the end of this chapter.
UTC vs. Local Time
┌──────────────────────────────────────────────────────────────┐
│ UTC vs LOCAL TIME │
│ │
│ UTC (Coordinated Universal Time) │
│ ───────────────────────────────── │
│ • The global reference time │
│ • Does not change for daylight saving │
│ • Standard for servers, logs, databases │
│ • Eliminates ambiguity when correlating events │
│ │
│ LOCAL TIME │
│ ────────── │
│ • UTC + timezone offset │
│ • Changes with daylight saving time │
│ • Convenient for humans │
│ • Problematic for automation and logging │
│ │
│ BEST PRACTICE: │
│ • Servers: set timezone to UTC │
│ • Logs: always log in UTC │
│ • Databases: store timestamps in UTC │
│ • Display: convert to local time for users in the app │
│ │
└──────────────────────────────────────────────────────────────┘
Setting the Timezone
# View current timezone
$ timedatectl show --property=Timezone
Timezone=UTC
# List available timezones
$ timedatectl list-timezones | grep America
America/Chicago
America/Denver
America/Los_Angeles
America/New_York
...
# Set timezone to UTC (recommended for servers)
$ sudo timedatectl set-timezone UTC
# Set timezone to a local zone (if needed)
$ sudo timedatectl set-timezone America/New_York
# The timezone is stored as a symlink
$ ls -la /etc/localtime
lrwxrwxrwx 1 root root 27 Jun 15 10:00 /etc/localtime -> /usr/share/zoneinfo/UTC
Think About It: Why is daylight saving time a problem for servers? Imagine a cron job scheduled for 2:30 AM on the night clocks "spring forward" from 2:00 to 3:00. What happens? What about when clocks "fall back" and 2:30 AM happens twice?
Hardware Clock vs. System Clock
Your Linux system maintains two separate clocks:
┌──────────────────────────────────────────────────────────────┐
│ HARDWARE CLOCK vs SYSTEM CLOCK │
│ │
│ HARDWARE CLOCK (RTC) SYSTEM CLOCK │
│ ──────────────────── ──────────── │
│ • Physical chip on motherboard • Maintained by the kernel │
│ • Runs when system is off • Only while system is on │
│ • Battery-backed (CMOS) • Initialized from RTC │
│ • Low precision (drifts) at boot │
│ • Accessed via /dev/rtc • Corrected by NTP │
│ • Set with hwclock • Set with timedatectl │
│ │
│ BOOT SEQUENCE: │
│ 1. System powers on │
│ 2. Kernel reads hardware clock → sets system clock │
│ 3. NTP client starts → corrects system clock │
│ 4. System clock is now accurate │
│ │
│ SHUTDOWN: │
│ 1. System writes system clock → hardware clock │
│ 2. System powers off │
│ 3. Hardware clock keeps ticking (drifts) │
│ │
└──────────────────────────────────────────────────────────────┘
Working with the Hardware Clock
# Read the hardware clock
$ sudo hwclock --show
2025-06-15 14:32:07.123456+00:00
# Sync system clock to hardware clock
$ sudo hwclock --hctosys
# Sync hardware clock to system clock
$ sudo hwclock --systohc
# Check if hardware clock stores UTC or local time
$ timedatectl | grep "RTC in local TZ"
RTC in local TZ: no # Good -- RTC stores UTC
# Set RTC to UTC (if it is wrong)
$ sudo timedatectl set-local-rtc 0
Distro Note: On dual-boot systems with Windows, Windows historically sets the RTC to local time while Linux prefers UTC. If you dual-boot, either configure Windows to use UTC for the RTC, or set
timedatectl set-local-rtc 1on Linux (not recommended for servers).
NTP Protocol Basics
The Network Time Protocol (NTP) synchronizes clocks over the network. It has been doing this reliably since 1985 and is one of the oldest Internet protocols still in active use.
How NTP Works
┌──────────────────────────────────────────────────────────────┐
│ NTP HIERARCHY (STRATA) │
│ │
│ Stratum 0: Reference clocks │
│ (GPS receivers, atomic clocks, radio clocks) │
│ │ │
│ ▼ │
│ Stratum 1: Primary time servers │
│ (directly connected to stratum 0) │
│ time.nist.gov, ntp.ubuntu.com │
│ │ │
│ ▼ │
│ Stratum 2: Secondary servers │
│ (synchronized to stratum 1) │
│ pool.ntp.org servers │
│ │ │
│ ▼ │
│ Stratum 3: Your servers │
│ (synchronized to stratum 2) │
│ │ │
│ ▼ │
│ Stratum 4: Your clients │
│ (synchronized to your servers) │
│ │
│ Each stratum adds a tiny bit of inaccuracy. │
│ Stratum 16 = unsynchronized (not trustworthy). │
│ │
└──────────────────────────────────────────────────────────────┘
The NTP Exchange
┌──────────┐ ┌──────────┐
│ Client │ │ Server │
│ │ │ │
│ t1 ───┤──── request packet ────────► │ │
│ │ │ t2 │
│ │ │ │
│ │ │ t3 │
│ t4 ◄──┤──── response packet ◄────────┤ │
│ │ │ │
└──────────┘ └──────────┘
The client records 4 timestamps:
t1 = when client sent the request
t2 = when server received the request
t3 = when server sent the response
t4 = when client received the response
Offset = ((t2 - t1) + (t3 - t4)) / 2
Delay = (t4 - t1) - (t3 - t2)
The client adjusts its clock by the calculated offset.
NTP does not simply "jump" the clock. It gradually slews (speeds up or slows down) the system clock to converge on the correct time, avoiding the problems that sudden jumps cause for running applications. Only when the offset is very large (over 128ms by default) does it step the clock.
chronyd vs. ntpd
Two NTP client implementations are common on Linux:
┌──────────────────────────────────────────────────────────────┐
│ chronyd vs ntpd │
│ │
│ chronyd (chrony) ntpd (classic NTP) │
│ ──────────────── ──────────────── │
│ Modern, actively developed Classic, mature │
│ Fast initial sync (<seconds) Slow initial sync │
│ Handles intermittent network Needs stable connection │
│ Good for VMs and laptops Designed for servers │
│ Low memory footprint Larger footprint │
│ Default on RHEL, Fedora, Ubuntu Legacy, being replaced │
│ │
│ RECOMMENDATION: Use chronyd unless you have a specific │
│ reason to use ntpd. It is the default on modern distros. │
│ │
└──────────────────────────────────────────────────────────────┘
Configuring chrony
Installation
# Debian/Ubuntu
$ sudo apt install -y chrony
# Fedora/RHEL/AlmaLinux/Rocky
$ sudo dnf install -y chrony
# Start and enable
$ sudo systemctl enable --now chronyd
Configuration File
$ sudo cat /etc/chrony/chrony.conf
A typical configuration:
# NTP servers to synchronize with
pool 2.pool.ntp.org iburst
# Record the rate at which the system clock gains/loses time
driftfile /var/lib/chrony/drift
# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second
makestep 1.0 3
# Enable kernel synchronization of the real-time clock (RTC)
rtcsync
# Specify directory for log files
logdir /var/log/chrony
Customizing NTP Servers
For better accuracy and reliability, use servers geographically close to you:
# Use NTP pool for your region
pool 0.us.pool.ntp.org iburst
pool 1.us.pool.ntp.org iburst
pool 2.us.pool.ntp.org iburst
pool 3.us.pool.ntp.org iburst
# Or use specific servers
server time.cloudflare.com iburst
server time.google.com iburst
The iburst option sends multiple requests at startup for faster initial synchronization.
After editing the configuration:
$ sudo systemctl restart chronyd
NTP Pool Servers
The NTP Pool Project (pool.ntp.org) is a cluster of thousands of volunteer time servers. When you configure pool.ntp.org, DNS returns different server IPs on each query, distributing load:
# See which pool servers your system resolved to
$ chronyc sources
For enterprise environments, consider running your own internal NTP server that synchronizes with the pool, and have all internal machines sync from it. This reduces external dependencies and ensures consistency.
Verifying Time Synchronization
chronyc: The chrony Control Tool
# Show current time sources
$ chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp1.example.com 2 6 377 34 +142us[ +187us] +/- 15ms
^+ ntp2.example.com 2 6 377 35 -523us[ -478us] +/- 18ms
^+ ntp3.example.com 3 6 377 34 +1124us[+1169us] +/- 22ms
^- ntp4.example.com 3 6 377 36 -2351us[-2306us] +/- 45ms
Understanding the output:
^*= current best source (the one being used)^+= acceptable source, ready to be selected^-= acceptable source, but too far from best^?= source not yet evaluated^x= source deemed unreliable- Stratum = distance from reference clock
- Poll = polling interval (log2 seconds: 6 = 64 seconds)
- Reach = reachability register (377 = last 8 attempts all successful)
- Last sample = offset from this source
# Show detailed tracking information
$ chronyc tracking
Reference ID : C0A80101 (ntp1.example.com)
Stratum : 3
Ref time (UTC) : Sat Jun 15 14:32:07 2025
System time : 0.000000142 seconds fast of NTP time
Last offset : +0.000000187 seconds
RMS offset : 0.000000523 seconds
Frequency : 3.128 ppm slow
Residual freq : +0.001 ppm
Skew : 0.012 ppm
Root delay : 0.015234 seconds
Root dispersion : 0.001234 seconds
Update interval : 64.0 seconds
Leap status : Normal
Key fields:
- System time: How far the system clock is from NTP time (should be microseconds)
- Frequency: How much the local clock drifts (in parts per million)
- Leap status: Should be "Normal" (changes during leap second events)
# Show sources with statistics
$ chronyc sourcestats
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
ntp1.example.com 25 14 26m -0.003 0.017 +142us 63us
ntp2.example.com 25 12 26m +0.012 0.023 -523us 89us
Quick Health Check
# One-liner: is time synchronized?
$ timedatectl show --property=NTPSynchronized --value
yes
# Chrony: am I synced?
$ chronyc tracking | grep "System time"
System time : 0.000000142 seconds fast of NTP time
If the system time offset is in microseconds, you are in excellent shape. Milliseconds are acceptable. Seconds mean something is wrong.
Hands-On: Setting Up chrony from Scratch
Let us configure time synchronization on a fresh system.
Step 1: Install chrony:
$ sudo apt install -y chrony # Debian/Ubuntu
# or
$ sudo dnf install -y chrony # RHEL/Fedora
Step 2: Configure NTP sources:
$ sudo tee /etc/chrony/chrony.conf << 'EOF'
# NTP servers
pool 0.pool.ntp.org iburst
pool 1.pool.ntp.org iburst
pool 2.pool.ntp.org iburst
pool 3.pool.ntp.org iburst
# Drift file
driftfile /var/lib/chrony/drift
# Allow initial large time correction
makestep 1.0 3
# Sync hardware clock
rtcsync
# Logging
logdir /var/log/chrony
log measurements statistics tracking
EOF
Step 3: Restart and verify:
$ sudo systemctl restart chronyd
$ chronyc sources
Wait 30-60 seconds for initial synchronization, then check:
$ chronyc tracking
Step 4: Verify with timedatectl:
$ timedatectl
Confirm:
NTP service: activeSystem clock synchronized: yes
Step 5: Test resilience -- force a time offset and watch chrony correct it:
# Check current time
$ date
# Deliberately set wrong time (chrony will correct it)
$ sudo date -s "14:00:00"
# Watch chrony fix it (within seconds due to makestep)
$ watch -n1 chronyc tracking
Safety Warning: Manually setting the system time on a production server can cause issues with running applications, especially databases. The
makestepdirective in chrony handles large initial offsets gracefully, but do not manually manipulate time on production systems.
Running Your Own NTP Server
For environments with many machines, running an internal NTP server reduces external dependencies and ensures all machines agree on time:
# On the NTP server, add to chrony.conf:
$ sudo tee -a /etc/chrony/chrony.conf << 'EOF'
# Allow NTP clients on the local network
allow 10.0.0.0/8
allow 192.168.0.0/16
# Serve time even when not synchronized (optional, for isolated networks)
# local stratum 10
EOF
$ sudo systemctl restart chronyd
# On client machines, point to your internal server:
$ sudo tee /etc/chrony/chrony.conf << 'EOF'
server ntp.internal.example.com iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
EOF
$ sudo systemctl restart chronyd
┌──────────────────────────────────────────────────────────────┐
│ INTERNAL NTP ARCHITECTURE │
│ │
│ Internet │
│ ┌─────────────────────┐ │
│ │ pool.ntp.org │ │
│ │ (stratum 2) │ │
│ └────────┬────────────┘ │
│ │ │
│ ─────────┼──────────── Firewall ──────────────────── │
│ │ │
│ ┌────────▼────────────┐ │
│ │ Internal NTP │ │
│ │ ntp.internal │ │
│ │ (stratum 3) │ │
│ └────────┬────────────┘ │
│ │ │
│ ┌─────┼─────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Server Server Server │
│ 1 2 3 (stratum 4) │
│ │
│ Only the internal NTP server needs internet access. │
│ All clients sync from it. │
│ │
└──────────────────────────────────────────────────────────────┘
Think About It: You have an isolated network with no internet access. How would you provide accurate time to the servers? Consider GPS receivers, PTP grandmaster clocks, and the
local stratumdirective.
PTP: Precision Time Protocol
For applications requiring sub-microsecond accuracy (financial trading, telecom, scientific instruments), NTP's millisecond-level accuracy is not enough. PTP (IEEE 1588) provides nanosecond-level synchronization.
┌──────────────────────────────────────────────────────────────┐
│ NTP vs PTP ACCURACY │
│ │
│ Protocol Typical Accuracy Use Case │
│ ──────── ──────────────── ──────── │
│ NTP 1-50 ms General servers, logs │
│ Chrony 0.01-1 ms Good server deployments │
│ PTP < 1 microsecond Financial, telecom, science │
│ │
│ PTP requires hardware support: │
│ • PTP-capable network cards (hardware timestamping) │
│ • PTP-capable switches (transparent or boundary clocks) │
│ • PTP grandmaster clock (GPS-disciplined) │
│ │
└──────────────────────────────────────────────────────────────┘
PTP on Linux is implemented by linuxptp:
# Install linuxptp
$ sudo apt install -y linuxptp # Debian/Ubuntu
$ sudo dnf install -y linuxptp # RHEL/Fedora
# Check if your NIC supports hardware timestamping
$ ethtool -T eth0 | grep -i ptp
PTP Hardware Clock: 0
# Run ptp4l (PTP daemon)
$ sudo ptp4l -i eth0 -m
PTP is specialized and requires compatible hardware throughout the network path. For most Linux administration tasks, chrony with NTP provides more than sufficient accuracy.
Debug This
Users report that Kerberos authentication is failing intermittently. The error message says "Clock skew too great." You check the affected server:
$ timedatectl
Local time: Sat 2025-06-15 14:38:12 UTC
Universal time: Sat 2025-06-15 14:38:12 UTC
RTC time: Sat 2025-06-15 14:38:12
Time zone: UTC (UTC, +0000)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no
What is wrong and how do you fix it?
Diagnosis:
NTP service: inactive-- no NTP client is runningSystem clock synchronized: no-- the clock is drifting freely
Fix:
# Install and start chrony
$ sudo apt install -y chrony
$ sudo systemctl enable --now chronyd
# Verify synchronization
$ chronyc sources
$ timedatectl
# Force immediate sync if needed
$ sudo chronyc makestep
Root cause: The server was installed without NTP configured. Over days or weeks, the hardware clock drifted by more than 5 minutes (the default Kerberos tolerance), causing authentication failures.
Prevention: Include chrony installation in your server provisioning playbook (see Chapter 68) and monitor NTP synchronization status with your monitoring stack (see Chapter 70).
What Just Happened?
┌──────────────────────────────────────────────────────────────┐
│ CHAPTER 75 RECAP │
│──────────────────────────────────────────────────────────────│
│ │
│ Accurate time is critical for logs, authentication, │
│ distributed systems, and compliance. │
│ │
│ Key concepts: │
│ • UTC for servers, local time for display only │
│ • Hardware clock (RTC): persists when off, drifts │
│ • System clock: maintained by kernel, corrected by NTP │
│ • NTP: network protocol for time synchronization │
│ • Stratum: distance from reference clock (lower = better) │
│ │
│ Tools: │
│ • timedatectl: view/set time, timezone, NTP status │
│ • chronyd: modern NTP client (recommended) │
│ • chronyc: control and query chrony │
│ • hwclock: manage hardware clock │
│ │
│ Essential commands: │
│ • timedatectl (check status) │
│ • chronyc sources (see NTP servers) │
│ • chronyc tracking (see sync accuracy) │
│ • chronyc makestep (force immediate correction) │
│ │
│ PTP provides nanosecond accuracy for specialized needs. │
│ │
└──────────────────────────────────────────────────────────────┘
Try This
Exercise 1: Time Audit
Run timedatectl and chronyc sources on every Linux system you have access to. Document which are synchronized and which are not. Fix any that are not.
Exercise 2: NTP Server List
Configure chrony to use a specific set of NTP servers (not the default pool). Choose servers geographically close to you from the NTP pool project. Verify they are being used with chronyc sources.
Exercise 3: Drift Measurement
Stop chrony, wait 24 hours, then check how far your clock has drifted:
$ sudo systemctl stop chronyd
# Wait 24 hours
$ chronyc tracking # Before restart
$ sudo systemctl start chronyd
This shows your hardware clock's natural drift rate.
Exercise 4: Internal NTP Server
Set up a chrony server that serves time to your local network. Configure a second machine to sync from it instead of the public pool. Verify with chronyc sources on the client.
Bonus Challenge
Write a monitoring check (shell script or Prometheus alert rule) that verifies:
- chronyd is running
- The system clock offset is less than 100 milliseconds
- At least one NTP source is reachable (Reach > 0)
- The stratum is less than 10
Integrate this into your monitoring stack from Chapter 70.