Scheduling & Automation

Why This Matters

Every system administrator and developer has tasks that need to run on a schedule: database backups at midnight, log rotation every week, security scans every Sunday, certificate renewals before they expire, health checks every five minutes.

You could set a reminder and do these things manually. But you are human -- you will forget, you will be on vacation, you will be asleep. The machine never forgets. The machine never sleeps.

Linux provides several tools for scheduling automated tasks: cron (the classic scheduler), anacron (for machines that are not always on), at (for one-time future tasks), and systemd timers (the modern alternative). This chapter covers all of them, compares their strengths, and teaches you best practices for automation that runs reliably for years.

Try This Right Now

# See what is currently scheduled for your user
crontab -l 2>/dev/null || echo "No crontab for $(whoami)"

# See all system-wide cron jobs
ls /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.weekly/ /etc/cron.monthly/ 2>/dev/null

# See all active systemd timers
systemctl list-timers --all --no-pager

# Schedule a one-time command for 2 minutes from now (if 'at' is installed)
echo "echo 'Hello from the future!' >> /tmp/at-test.txt" | at now + 2 minutes 2>/dev/null || echo "Install 'at': sudo apt install at"

# Check what's queued
atq 2>/dev/null

cron: The Classic Scheduler

cron is the traditional Unix job scheduler. It has been around since the 1970s, and it is available on every Linux system. The cron daemon (crond or cron) wakes up every minute, checks the schedule, and runs any jobs that are due.

crontab: User Cron Tables

Each user can have their own crontab (cron table):

# View your crontab
crontab -l

# Edit your crontab
crontab -e

# Remove your crontab entirely
crontab -r

WARNING: crontab -r removes your entire crontab without confirmation. If you have important jobs, back them up first: crontab -l > ~/crontab-backup.txt

Distro Note: On some systems, the default editor for crontab -e is vi. To change it: export EDITOR=nano (or add it to your ~/.bashrc).

Crontab Syntax

Each line in a crontab follows this format:

*  *  *  *  *  command-to-run
│  │  │  │  │
│  │  │  │  └── Day of Week   (0-7, 0 and 7 = Sunday)
│  │  │  └───── Month         (1-12)
│  │  └──────── Day of Month  (1-31)
│  └─────────── Hour          (0-23)
└────────────── Minute        (0-59)

+----------------------------------------------------------------+
|  Field         | Range   | Special Values                      |
+----------------------------------------------------------------+
|  Minute        | 0-59    | * (every), */5 (every 5)           |
|  Hour          | 0-23    | * (every), 1-5 (range)             |
|  Day of Month  | 1-31    | * (every), 1,15 (list)             |
|  Month         | 1-12    | * (every), jan-dec                  |
|  Day of Week   | 0-7     | * (every), mon-fri                  |
+----------------------------------------------------------------+

Common Schedules

# Every minute
* * * * * /path/to/script.sh

# Every 5 minutes
*/5 * * * * /path/to/script.sh

# Every hour at minute 0
0 * * * * /path/to/script.sh

# Every day at 2:30 AM
30 2 * * * /path/to/script.sh

# Every Monday at 9 AM
0 9 * * 1 /path/to/script.sh

# Every weekday at 6 PM
0 18 * * 1-5 /path/to/script.sh

# First day of every month at midnight
0 0 1 * * /path/to/script.sh

# Every 15 minutes during business hours (9-17)
*/15 9-17 * * 1-5 /path/to/script.sh

# Twice a day at 8 AM and 8 PM
0 8,20 * * * /path/to/script.sh

# Every Sunday at 3 AM
0 3 * * 0 /path/to/script.sh

# January 1st at midnight
0 0 1 1 * /path/to/script.sh

Special Strings

Some cron implementations support shorthand:

@reboot    /path/to/script.sh    # Run once at startup
@yearly    /path/to/script.sh    # 0 0 1 1 *
@monthly   /path/to/script.sh    # 0 0 1 * *
@weekly    /path/to/script.sh    # 0 0 * * 0
@daily     /path/to/script.sh    # 0 0 * * *
@hourly    /path/to/script.sh    # 0 * * * *

Environment in cron

Cron jobs run in a minimal environment. Your $PATH, aliases, and shell functions are not available. This is the number one source of cron bugs.

# BAD: relies on $PATH to find python3
* * * * * python3 /opt/myapp/script.py

# GOOD: use absolute paths
* * * * * /usr/bin/python3 /opt/myapp/script.py

# Or set PATH in the crontab
PATH=/usr/local/bin:/usr/bin:/bin
* * * * * python3 /opt/myapp/script.py

You can also set other environment variables at the top of the crontab:

# Set environment at top of crontab
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=admin@example.com
HOME=/home/myuser

# Jobs follow
30 2 * * * /opt/scripts/backup.sh

Think About It: Why does cron use a minimal environment instead of sourcing the user's shell profile? Think about what could go wrong if cron inherited a user's full interactive environment.

Hands-On: Setting Up a Cron Job

Step 1: Create a Script

mkdir -p ~/scripts
cat > ~/scripts/system-snapshot.sh << 'SCRIPT'
#!/bin/bash
set -euo pipefail

LOGFILE="/tmp/system-snapshots.log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

{
    echo "=== System Snapshot: $TIMESTAMP ==="
    echo "Uptime: $(uptime -p)"
    echo "Load: $(cat /proc/loadavg)"
    echo "Memory: $(free -h | awk '/Mem:/ {print $3, "/", $2}')"
    echo "Disk: $(df -h / | awk 'NR==2 {print $3, "/", $2, "("$5" used)"}')"
    echo ""
} >> "$LOGFILE"
SCRIPT
chmod +x ~/scripts/system-snapshot.sh

# Test it
~/scripts/system-snapshot.sh
cat /tmp/system-snapshots.log

Step 2: Schedule It

# Add a cron job to run every 15 minutes
(crontab -l 2>/dev/null; echo "*/15 * * * * $HOME/scripts/system-snapshot.sh") | crontab -

# Verify
crontab -l

Step 3: Check Cron Logs

# On Debian/Ubuntu
grep CRON /var/log/syslog | tail -5

# On RHEL/Fedora
grep CRON /var/log/cron | tail -5

# Or via journald
journalctl -u cron.service --since "30 min ago" --no-pager 2>/dev/null || \
journalctl -u crond.service --since "30 min ago" --no-pager 2>/dev/null

Step 4: Clean Up

# Remove just that one job
crontab -l | grep -v 'system-snapshot' | crontab -

# Verify it is gone
crontab -l

System-Wide Cron

Beyond per-user crontabs, there are system-wide cron locations:

/etc/crontab

The system crontab has an extra field -- the username:

# /etc/crontab format:
# min hour dom month dow USER command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || run-parts --report /etc/cron.daily
47 6    * * 7   root    test -x /usr/sbin/anacron || run-parts --report /etc/cron.weekly
52 6    1 * *   root    test -x /usr/sbin/anacron || run-parts --report /etc/cron.monthly

/etc/cron.d/

Individual cron files for system services (same format as /etc/crontab):

ls /etc/cron.d/

Drop-In Directories

Scripts placed in these directories run at the indicated frequency:

/etc/cron.hourly/    # Runs every hour
/etc/cron.daily/     # Runs every day
/etc/cron.weekly/    # Runs every week
/etc/cron.monthly/   # Runs every month

Scripts in these directories must be executable and should not have a .sh extension on some systems (run-parts may skip files with dots in the name).

# List daily cron jobs
ls -la /etc/cron.daily/

# You'll see things like:
# logrotate
# man-db
# apt-compat

anacron: For Machines That Sleep

cron expects the machine to be running 24/7. If a daily job is scheduled for 3 AM and the machine is off at 3 AM, the job simply does not run. anacron fixes this.

anacron tracks when jobs last ran and executes them when the machine comes back online. It is ideal for laptops and desktops that are not always on.

# Check anacron configuration
cat /etc/anacrontab

Typical contents:

# period  delay  job-identifier  command
1         5      cron.daily      run-parts /etc/cron.daily
7         10     cron.weekly     run-parts /etc/cron.weekly
@monthly  15     cron.monthly    run-parts /etc/cron.monthly

anacron checks timestamps in /var/spool/anacron/. If a job has not run within its period, anacron runs it (after the specified delay).

# See when jobs last ran
cat /var/spool/anacron/cron.daily
# 20250310

# Force anacron to run all pending jobs
sudo anacron -f

at: One-Time Scheduled Commands

The at command schedules a command to run once at a specific time in the future.

# Install if needed
sudo apt install at        # Debian/Ubuntu
sudo dnf install at        # Fedora/RHEL

Scheduling with at

# Run at a specific time
at 14:30 << 'JOB'
echo "Reminder: meeting in 30 minutes" | mail -s "Meeting" user@example.com
JOB

# Run at a relative time
at now + 30 minutes << 'JOB'
/opt/scripts/cleanup.sh
JOB

# Other time formats
at midnight
at noon
at teatime        # 4:00 PM
at 9:00 AM tomorrow
at now + 2 hours
at 3:00 PM next Friday

Managing at Jobs

# List pending jobs
atq

# View a specific job's commands
at -c 5    # Show job number 5

# Remove a pending job
atrm 5     # Remove job number 5

batch: Run When Load is Low

batch is like at but waits until the system load drops below a threshold:

batch << 'JOB'
/opt/scripts/heavy-processing.sh
JOB

This is useful for resource-intensive tasks that should not interfere with normal operations.

systemd Timers: The Modern Way

systemd timers are a powerful, modern alternative to cron. We introduced them in Chapter 16. Here we go deeper.

Why Use systemd Timers?

Creating a systemd Timer

You need two files: a .service (what to run) and a .timer (when to run).

Service file (/etc/systemd/system/disk-report.service):

[Unit]
Description=Generate Disk Usage Report

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disk-report.sh
User=root
StandardOutput=journal
StandardError=journal

Timer file (/etc/systemd/system/disk-report.timer):

[Unit]
Description=Run Disk Report Every 6 Hours

[Timer]
OnCalendar=*-*-* 00/6:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

The script (/usr/local/bin/disk-report.sh):

#!/bin/bash
set -euo pipefail
echo "=== Disk Report: $(date) ==="
df -h | grep -E '^/dev/'
echo ""
du -sh /var/log/ /var/cache/ /tmp/ 2>/dev/null

sudo chmod +x /usr/local/bin/disk-report.sh
sudo systemctl daemon-reload
sudo systemctl enable --now disk-report.timer

OnCalendar Expressions

OnCalendar=*-*-* 02:00:00          # Daily at 2 AM
OnCalendar=Mon *-*-* 09:00:00      # Mondays at 9 AM
OnCalendar=*-*-* 00/6:00:00        # Every 6 hours (0, 6, 12, 18)
OnCalendar=*-*-* *:00/15:00        # Every 15 minutes
OnCalendar=*-*-1 00:00:00          # First of each month
OnCalendar=Mon..Fri *-*-* 08:00:00 # Weekdays at 8 AM
OnCalendar=hourly                   # Shorthand for *-*-* *:00:00
OnCalendar=daily                    # Shorthand for *-*-* 00:00:00
OnCalendar=weekly                   # Shorthand for Mon *-*-* 00:00:00

Validate your schedule:

systemd-analyze calendar "Mon..Fri *-*-* 08:00:00"
systemd-analyze calendar "*-*-* *:00/15:00"
systemd-analyze calendar "daily"

Monotonic Timers (Relative)

Instead of calendar-based, trigger relative to system events:

[Timer]
OnBootSec=15min          # 15 minutes after boot
OnStartupSec=30min       # 30 minutes after systemd started
OnUnitActiveSec=1h       # 1 hour after the service last ran
OnUnitInactiveSec=30min  # 30 minutes after the service became inactive

Managing Timers

# List all timers and their next firing time
systemctl list-timers --all --no-pager

# Check a specific timer
systemctl status disk-report.timer

# Manually trigger the associated service (for testing)
sudo systemctl start disk-report.service

# View the service's output
journalctl -u disk-report.service --no-pager -n 20

# Disable a timer
sudo systemctl disable --now disk-report.timer

Hands-On: Migrating from cron to systemd Timer

Let us convert a common cron job to a systemd timer.

The Original Cron Job

# In crontab: run backup every night at 2 AM
0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

Step 1: Create the Service

sudo tee /etc/systemd/system/nightly-backup.service << 'UNIT'
[Unit]
Description=Nightly Backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=root
StandardOutput=journal
StandardError=journal
SyslogIdentifier=nightly-backup

# Security hardening
NoNewPrivileges=yes
ProtectHome=read-only
PrivateTmp=yes
UNIT

Step 2: Create the Timer

sudo tee /etc/systemd/system/nightly-backup.timer << 'UNIT'
[Unit]
Description=Run Nightly Backup at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=600

[Install]
WantedBy=timers.target
UNIT

Step 3: Enable and Test

sudo systemctl daemon-reload
sudo systemctl enable --now nightly-backup.timer

# Verify the timer is scheduled
systemctl list-timers nightly-backup.timer --no-pager

# Test by running the service manually
sudo systemctl start nightly-backup.service

# Check the output
journalctl -u nightly-backup.service --no-pager

Step 4: Remove the Old Cron Job

crontab -l | grep -v 'backup.sh' | crontab -

Comparing cron vs systemd Timers

When to use cron:

• Simple, quick one-liners
• Systems without systemd
• When minimal setup overhead matters

When to use systemd timers:

• Production services needing reliability
• Jobs with dependencies on other services
• When you need resource control or sandboxing
• When logging and debugging matter

Think About It: You have a job that sends daily reports. It takes about 30 seconds to run. Which would you choose, cron or a systemd timer, and why?

Automation Best Practices

1. Always Log Output

# Cron: redirect both stdout and stderr
0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

# systemd: automatic, but set SyslogIdentifier
[Service]
StandardOutput=journal
SyslogIdentifier=my-backup

2. Handle Errors Gracefully

#!/bin/bash
set -euo pipefail

cleanup() {
    echo "[$(date)] Script exiting with code $?" >> /var/log/myjob.log
}
trap cleanup EXIT

echo "[$(date)] Starting backup..." >> /var/log/myjob.log
# ... actual work ...
echo "[$(date)] Backup complete." >> /var/log/myjob.log

3. Prevent Overlapping Runs (Locking)

If a job takes longer than its interval, you can end up with multiple instances running simultaneously. Use a lock file:

#!/bin/bash
set -euo pipefail

LOCKFILE="/var/lock/mybackup.lock"

# Use flock for atomic locking
exec 200>"$LOCKFILE"
if ! flock -n 200; then
    echo "Another instance is already running. Exiting."
    exit 1
fi

# Your actual work here
echo "Running backup at $(date)"
sleep 60  # Simulating long-running job
echo "Backup complete at $(date)"

# Lock is automatically released when the script exits

Or use flock directly on the command line:

# In crontab:
*/5 * * * * flock -n /var/lock/myjob.lock /opt/scripts/myjob.sh

4. Use Timeouts

Prevent a job from running forever:

# Cron: use timeout command
0 2 * * * timeout 3600 /opt/scripts/backup.sh

# systemd: built-in timeout
[Service]
TimeoutStartSec=3600

5. Monitor Your Jobs

# Create a simple monitoring wrapper
#!/bin/bash
set -euo pipefail

JOB_NAME="nightly-backup"
START_TIME=$(date +%s)

/opt/scripts/backup.sh
EXIT_CODE=$?

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

if [[ $EXIT_CODE -ne 0 ]]; then
    echo "ALERT: $JOB_NAME failed with exit code $EXIT_CODE after ${DURATION}s" | \
        mail -s "Job Failure: $JOB_NAME" admin@example.com
fi

echo "$JOB_NAME completed in ${DURATION}s with exit code $EXIT_CODE"

6. Set MAILTO (cron)

cron can email the output of jobs:

# At the top of crontab
MAILTO=admin@example.com

# Or to disable email for a specific job
0 2 * * * /opt/scripts/backup.sh > /dev/null 2>&1

Debug This: Cron Job Not Running

You set up a cron job but it never fires. Here is your debugging checklist:

1. Is cron running?

   systemctl status cron.service 2>/dev/null || systemctl status crond.service
   

2. Is the crontab syntax correct?

   crontab -l
   # Common mistake: 6 fields instead of 5 (user field only in /etc/crontab)
   

3. Can the script be found and executed?

   # Use absolute paths!
   which python3
   ls -la /opt/scripts/backup.sh
   

4. Check cron logs:

   grep CRON /var/log/syslog | tail -20
   journalctl -u cron.service --since "1 hour ago" --no-pager
   

5. Check permissions:

   # Script must be executable
   chmod +x /opt/scripts/backup.sh
   
   # Check /etc/cron.allow and /etc/cron.deny
   cat /etc/cron.allow 2>/dev/null
   cat /etc/cron.deny 2>/dev/null
   

6. Test the command manually as the cron user:

   # Run with cron's minimal environment
   env -i /bin/bash --noprofile --norc -c '/opt/scripts/backup.sh'
   

7. Check for environment issues:

   # Add this as the first line in your cron script
   env > /tmp/cron-env-debug.txt
   # Then compare with your interactive environment
   

What Just Happened?

+------------------------------------------------------------------+
|                     CHAPTER 24 RECAP                              |
+------------------------------------------------------------------+
|                                                                  |
|  cron:                                                           |
|  - Classic scheduler, minute-level precision                     |
|  - Syntax: min hour dom month dow command                       |
|  - Use crontab -e to edit, crontab -l to list                  |
|  - Always use absolute paths in cron jobs                       |
|                                                                  |
|  anacron:                                                        |
|  - Catches up on missed jobs after downtime                     |
|  - Ideal for laptops and desktops                               |
|                                                                  |
|  at:                                                             |
|  - One-time future execution                                    |
|  - at now + 30 minutes, at 3:00 PM tomorrow                    |
|                                                                  |
|  systemd timers:                                                 |
|  - Modern, with logging, dependencies, sandboxing               |
|  - OnCalendar for schedules, Persistent for catch-up            |
|  - Two files: .service + .timer                                 |
|                                                                  |
|  Best practices:                                                 |
|  - Always log output                                            |
|  - Use flock to prevent overlapping runs                        |
|  - Set timeouts for long-running jobs                           |
|  - Monitor job success/failure                                  |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Cron Basics

Set up a cron job that runs every 5 minutes and appends the current date and load average to /tmp/load-monitor.log. Let it run for 30 minutes, then check the log. Remove the cron job when done.

Exercise 2: systemd Timer

Convert the cron job from Exercise 1 to a systemd timer. Use OnUnitActiveSec=5min for the timing. Compare the logging experience: which is easier to check for errors?

Exercise 3: Locking

Create two cron jobs that run the same script at the same time (every minute). The script should sleep for 90 seconds. Observe what happens without locking (two instances run simultaneously). Then add flock and verify only one runs at a time.

Exercise 4: Catch-Up Behavior

Create a systemd timer with Persistent=true set to run every hour. Stop the timer for 3 hours, then restart it. Does it catch up on the missed runs? Now try the same with a cron job. What is the difference?

Bonus Challenge

Build a complete automation suite: a systemd timer that runs a script every 6 hours. The script should: (1) check disk usage and warn if any filesystem is over 80%, (2) check for failed systemd services, (3) report the top 5 CPU-consuming processes, and (4) send the report to a log file. Add flock for safety, a timeout of 60 seconds, and proper error handling. Include the service file, timer file, and script.