Signals Deep Dive

Why This Matters

You press Ctrl+C and a program stops. You close a terminal and background jobs die. You run sudo systemctl reload nginx and Nginx picks up its new configuration without dropping a single connection. You type kill -9 and a stubborn process vanishes.

All of these work through signals -- the kernel's mechanism for delivering notifications to processes. Signals are how Linux says "hey, something just happened that you need to deal with." They are the nervous system of process management, and understanding them is the difference between blindly typing kill -9 and surgically managing processes like a professional.

This chapter covers what signals are, which ones matter most, how to trap them in your own scripts, and how production services use them for graceful reloads and shutdowns.

Try This Right Now

# See all available signals on your system
kill -l

# Start a process and send it different signals
sleep 300 &
PID=$!
echo "Started sleep with PID $PID"

# Send SIGTERM (signal 15) -- the polite termination
kill $PID
# The process is gone

# Start another
sleep 300 &
PID=$!

# Send SIGSTOP (pause the process)
kill -STOP $PID
ps -p $PID -o pid,stat,cmd
# State should show 'T' (stopped)

# Resume it
kill -CONT $PID
ps -p $PID -o pid,stat,cmd
# State should be back to 'S' (sleeping)

# Clean up
kill $PID

What Are Signals?

A signal is an asynchronous notification sent to a process. Think of it like tapping someone on the shoulder -- the process was doing something else, and now it has to respond.

Signals can come from:

The kernel -- hardware faults (SIGSEGV), timer expired, child process died
Another process -- via the kill system call (despite the name, it sends any signal)
The terminal -- key combinations like Ctrl+C, Ctrl+Z, Ctrl+\
The process itself -- a program can send signals to itself

When a signal arrives, the process can:

Handle it -- run a custom signal handler function
Ignore it -- pretend it never happened
Let the default action occur -- each signal has a defined default

Two signals are special: SIGKILL (9) and SIGSTOP (19) can NEVER be caught, handled, or ignored. The kernel enforces them directly.

The Signal Table: Signals You Need to Know

+--------+----------+---------+-----------------------------------------------+
| Number | Name     | Default | Purpose                                       |
+--------+----------+---------+-----------------------------------------------+
|   1    | SIGHUP   | Term    | Terminal hangup; daemons: reload config        |
|   2    | SIGINT   | Term    | Interrupt from keyboard (Ctrl+C)               |
|   3    | SIGQUIT  | Core    | Quit from keyboard (Ctrl+\), produces core dump|
|   6    | SIGABRT  | Core    | Abort signal from abort() call                 |
|   9    | SIGKILL  | Term    | Force kill (CANNOT be caught or ignored)       |
|  10    | SIGUSR1  | Term    | User-defined signal 1                          |
|  11    | SIGSEGV  | Core    | Invalid memory reference (segfault)            |
|  12    | SIGUSR2  | Term    | User-defined signal 2                          |
|  13    | SIGPIPE  | Term    | Broken pipe (write to pipe with no reader)     |
|  14    | SIGALRM  | Term    | Timer alarm from alarm()                       |
|  15    | SIGTERM  | Term    | Graceful termination request                   |
|  17    | SIGCHLD  | Ignore  | Child process stopped or terminated            |
|  18    | SIGCONT  | Cont    | Resume if stopped                              |
|  19    | SIGSTOP  | Stop    | Pause process (CANNOT be caught or ignored)    |
|  20    | SIGTSTP  | Stop    | Stop from terminal (Ctrl+Z)                    |
+--------+----------+---------+-----------------------------------------------+

Distro Note: Signal numbers can differ between architectures (e.g., SIGUSR1 is 10 on x86 but 16 on MIPS). Always use signal names, not numbers, in scripts and commands for portability. Write kill -TERM instead of kill -15.

Default Actions Explained

Term -- Terminate the process
Core -- Terminate and produce a core dump file (for debugging)
Stop -- Pause (suspend) the process
Cont -- Resume a stopped process
Ignore -- Do nothing

Signals in Detail

SIGTERM (15) -- The Gentleman's Termination

This is the default signal sent by kill. It is a polite request: "please shut down." Well-written programs catch SIGTERM, clean up resources (close files, release locks, flush buffers), and exit cleanly.

# These are equivalent:
kill 12345
kill -15 12345
kill -TERM 12345
kill -SIGTERM 12345

SIGKILL (9) -- The Executioner

SIGKILL terminates a process immediately. The process gets no chance to handle it, no chance to clean up. The kernel simply removes it.

kill -9 12345
kill -KILL 12345

When to use SIGKILL:

The process is not responding to SIGTERM
The process is stuck and must be removed
You are dealing with a compromised process

When NOT to use SIGKILL:

As your first attempt (always try SIGTERM first)
On database processes (risk of data corruption)
On processes holding locks (locks may become stale)

WARNING: kill -9 is the sledgehammer. It leaves temporary files behind, does not close database connections cleanly, does not release file locks, and does not flush write buffers. Use it only after SIGTERM has failed.

SIGINT (2) -- The Keyboard Interrupt

This is what Ctrl+C sends. Programs can catch it to clean up gracefully:

# Start a process
sleep 300
# Press Ctrl+C
# The process receives SIGINT and terminates

SIGQUIT (3) -- Quit with Core Dump

Ctrl+\ sends SIGQUIT. Like SIGINT, but also produces a core dump for debugging:

# Start a process
sleep 300
# Press Ctrl+\
# Output: Quit (core dumped)

SIGHUP (1) -- Hang Up / Reload

Originally meant "the terminal connection was lost" (the modem hung up). Today it has two common uses:

Terminal hangup: When you close a terminal, SIGHUP is sent to all processes in that terminal
Daemon reload: By convention, sending SIGHUP to a daemon tells it to re-read its configuration file without restarting

# Reload Nginx configuration
sudo kill -HUP $(cat /run/nginx.pid)
# Or equivalently:
sudo systemctl reload nginx  # This sends SIGHUP under the hood

# Reload sshd
sudo kill -HUP $(pgrep -x sshd | head -1)

SIGSTOP (19) and SIGCONT (18) -- Pause and Resume

SIGSTOP pauses a process. SIGCONT resumes it. SIGSTOP cannot be caught -- this is how debuggers freeze processes.

# Pause a process
kill -STOP 12345

# Resume it
kill -CONT 12345

The terminal equivalent of SIGSTOP is SIGTSTP (20), sent by Ctrl+Z. Unlike SIGSTOP, SIGTSTP CAN be caught and handled.

SIGUSR1 (10) and SIGUSR2 (12) -- User Defined

These have no predefined meaning. Programs define what they do:

# Example: dd reports progress on SIGUSR1
dd if=/dev/urandom of=/tmp/testfile bs=1M count=1000 &
DD_PID=$!

# Send SIGUSR1 to get a progress report
kill -USR1 $DD_PID
# dd outputs bytes transferred so far

# Clean up
kill $DD_PID
rm -f /tmp/testfile

Other examples:

Nginx uses SIGUSR1 to reopen log files (log rotation)
Some programs use SIGUSR2 to toggle debug mode

SIGPIPE (13) -- Broken Pipe

Sent when a process writes to a pipe but the reading end has been closed:

# This triggers SIGPIPE:
yes | head -1
# 'yes' keeps writing, but 'head' exits after 1 line
# The kernel sends SIGPIPE to 'yes'

SIGCHLD (17) -- Child Status Changed

Sent to a parent when a child process stops, continues, or terminates. The parent should call wait() to collect the child's exit status. If it doesn't, the child becomes a zombie.

Think About It: When you run systemctl reload nginx, Nginx re-reads its config and applies it without dropping connections. How is this possible? (Hint: Nginx's master process catches SIGHUP, re-reads the config, spawns new worker processes with the new config, and gracefully shuts down old workers.)

Sending Signals: kill, killall, pkill

kill -- By PID

# Send SIGTERM (default)
kill 12345

# Send a specific signal
kill -HUP 12345
kill -SIGUSR1 12345

# Send to multiple PIDs
kill 12345 12346 12347

# Send signal 0 -- checks if process exists (sends nothing)
kill -0 12345
echo $?  # 0 = exists, 1 = doesn't exist

killall -- By Name

# Kill all processes named "python3"
killall python3

# Send SIGHUP to all nginx processes
killall -HUP nginx

# Interactive mode -- confirm before each kill
killall -i python3

# Only kill processes owned by a specific user
killall -u alice python3

pkill -- By Pattern

# Kill processes matching a pattern
pkill -f "python train"

# Send SIGHUP to processes matching a pattern
pkill -HUP -f "gunicorn"

# Kill processes owned by a user
pkill -u testuser

# Kill the oldest matching process
pkill -o -f "worker"

# Kill the newest matching process
pkill -n -f "worker"

WARNING: Be very careful with killall and pkill. killall python3 kills ALL python3 processes, not just yours (if you are root). Always verify what will be matched first with pgrep:

# See what pkill would match BEFORE actually killing
pgrep -a -f "python train"

Trapping Signals in Bash

The trap builtin lets your scripts catch signals and run custom code in response. This is how you write scripts that clean up after themselves:

Basic Trap Syntax

trap 'commands' SIGNAL_LIST

Example: Cleanup on Exit

#!/bin/bash
TMPFILE=$(mktemp)
echo "Working in $TMPFILE"

# Clean up on exit, interrupt, or termination
trap 'echo "Cleaning up..."; rm -f "$TMPFILE"; exit' EXIT INT TERM

# Do some work
echo "data" > "$TMPFILE"
sleep 30

echo "Done"

Example: Graceful Shutdown Script

#!/bin/bash

RUNNING=true

cleanup() {
    echo "Received shutdown signal. Cleaning up..."
    RUNNING=false
    # Close database connections, flush caches, etc.
    echo "Cleanup complete. Exiting."
    exit 0
}

trap cleanup SIGTERM SIGINT

echo "Service started. PID: $$"

# Main loop
while $RUNNING; do
    # Do work here
    echo "Working... $(date)"
    sleep 5
done

Test it:

# Terminal 1:
bash graceful_shutdown.sh

# Terminal 2:
kill $(pgrep -f graceful_shutdown)
# Watch Terminal 1 -- it should print cleanup messages

Example: Ignore a Signal

#!/bin/bash
# Ignore SIGINT (Ctrl+C) -- use with caution!
trap '' SIGINT

echo "You cannot Ctrl+C me! (Use Ctrl+\ or kill from another terminal)"
sleep 60

Example: Trap on SIGHUP for Config Reload

#!/bin/bash
CONFIG_FILE="/etc/myapp/config.conf"

load_config() {
    echo "Loading configuration from $CONFIG_FILE..."
    # In a real script, you would source the config file here
    # source "$CONFIG_FILE"
    echo "Configuration reloaded at $(date)"
}

trap load_config SIGHUP

load_config  # Initial load

echo "Running as PID $$. Send SIGHUP to reload config."

while true; do
    # Main application loop
    sleep 10
done

Test it:

# Terminal 1:
bash config_daemon.sh

# Terminal 2:
kill -HUP $(pgrep -f config_daemon)
# Watch Terminal 1 -- it should print "Configuration reloaded"

Trap Gotchas

# List all active traps
trap -p

# Reset a trap to default behavior
trap - SIGINT

# The EXIT trap fires on ANY exit (normal, error, signal)
trap 'echo "Goodbye"' EXIT

# Traps in subshells: subshells inherit traps but
# modifications in subshells do not affect the parent
(trap 'echo "sub"' SIGINT)
# The parent's SIGINT trap is unchanged

Think About It: If you trap SIGTERM in a script but your script calls kill -9 $$, will the trap run? Why or why not?

How Daemons Handle SIGHUP: A Deep Look

The SIGHUP reload pattern is one of the most important signal conventions in Linux. Here is how it typically works with a real service like Nginx:

                    SIGHUP
                      |
                      v
  +-----------------------------------+
  |  Nginx Master Process (PID 100)   |
  |  1. Catches SIGHUP                |
  |  2. Re-reads nginx.conf           |
  |  3. Validates new configuration   |
  |  4. If valid:                     |
  |     a. Spawns new worker processes|
  |     b. Signals old workers to     |
  |        finish current requests    |
  |     c. Old workers exit after     |
  |        completing in-flight work  |
  |  5. If invalid:                   |
  |     a. Logs error                 |
  |     b. Keeps running with old     |
  |        configuration              |
  +-----------------------------------+
         |                    |
         v                    v
  +-------------+     +-------------+
  | Old Worker  |     | New Worker  |
  | (finishing  |     | (new config)|
  |  requests)  |     |             |
  +-------------+     +-------------+

This is why systemctl reload nginx does not cause downtime -- existing connections are served to completion by old workers while new connections go to new workers with the updated config.

Services and Their Signal Conventions

Service	SIGHUP	SIGUSR1	SIGUSR2
Nginx	Reload config	Reopen logs	Upgrade binary
Apache	Graceful restart	Reopen logs	Graceful restart
PostgreSQL	Reload config	--	--
sshd	Reload config	--	--
rsyslog	Reload config	--	--

# The systemctl commands that use signals under the hood:
sudo systemctl reload nginx     # Sends SIGHUP
sudo systemctl stop nginx       # Sends SIGTERM, then SIGKILL after timeout
sudo systemctl kill nginx       # Sends SIGTERM by default
sudo systemctl kill -s HUP nginx  # Sends SIGHUP explicitly

Signal Handling Best Practices

For Scripts

Always trap EXIT for cleanup: Temporary files, lock files, PID files -- clean them up.

trap 'rm -f /tmp/myapp.lock' EXIT

Use named signals, not numbers: kill -TERM is portable; kill -15 may not be.
Do not trap SIGKILL or SIGSTOP: You cannot. If you try, the trap is silently ignored.
Keep trap handlers short: A signal can arrive at any time. Your handler should do minimal work -- set a flag, then let the main loop check the flag.

# Good: set a flag
SHUTDOWN=false
trap 'SHUTDOWN=true' SIGTERM

while ! $SHUTDOWN; do
    do_work
done
cleanup

Propagate signals to child processes in scripts:

trap 'kill 0' SIGTERM SIGINT
# 'kill 0' sends the signal to the entire process group

For System Administration

Always try SIGTERM before SIGKILL: Give processes time to clean up.
Use systemctl for managed services: Let systemd handle signal delivery.
Use SIGHUP for config reloads: Check if the service supports it first (man service_name).
Use kill -0 to check if a process is alive:

if kill -0 "$PID" 2>/dev/null; then
    echo "Process $PID is still running"
else
    echo "Process $PID is gone"
fi

Debug This: A Script That Won't Die

You have a script running that ignores Ctrl+C:

#!/bin/bash
trap '' SIGINT SIGTERM
echo "I am invincible! PID: $$"
while true; do sleep 1; done

How do you stop it?

Diagnosis:

# Ctrl+C won't work (SIGINT is trapped and ignored)
# kill PID won't work (SIGTERM is trapped and ignored)

# Option 1: SIGKILL -- cannot be trapped
kill -9 $(pgrep -f "I am invincible")

# Option 2: SIGQUIT -- they forgot to trap it
kill -QUIT $(pgrep -f "I am invincible")

# Option 3: SIGSTOP -- pause it, then SIGKILL
kill -STOP $(pgrep -f "I am invincible")
kill -9 $(pgrep -f "I am invincible")

The lesson: trapping SIGINT and SIGTERM can be useful for cleanup, but ignoring them entirely (empty trap handler) is an anti-pattern. Always do something useful in the handler, and always exit afterward.

Hands-On: Signal Playground

Create a script that demonstrates signal handling:

#!/bin/bash
# signal_playground.sh

echo "Signal Playground - PID: $$"
echo "Send me signals and watch what happens!"
echo ""

trap 'echo "[$(date +%T)] Caught SIGHUP -- reloading config..."' HUP
trap 'echo "[$(date +%T)] Caught SIGUSR1 -- toggling debug mode"' USR1
trap 'echo "[$(date +%T)] Caught SIGUSR2 -- dumping status"' USR2
trap 'echo "[$(date +%T)] Caught SIGINT -- Ctrl+C pressed"; exit 0' INT
trap 'echo "[$(date +%T)] Caught SIGTERM -- shutting down gracefully"; exit 0' TERM
trap 'echo "[$(date +%T)] EXIT trap -- final cleanup done"' EXIT

echo "Waiting for signals... (try these in another terminal)"
echo "  kill -HUP $$"
echo "  kill -USR1 $$"
echo "  kill -USR2 $$"
echo "  kill -INT $$     (or press Ctrl+C)"
echo "  kill -TERM $$"
echo ""

while true; do
    sleep 1
done

Run it and experiment from another terminal:

# Terminal 1:
bash signal_playground.sh

# Terminal 2:
PID=<the PID shown>
kill -HUP $PID
kill -USR1 $PID
kill -USR2 $PID
kill -TERM $PID

Signals and Process Groups

When you press Ctrl+C, SIGINT is sent not just to one process but to the entire foreground process group. This is why a pipeline like cat file | grep pattern | sort is killed entirely by one Ctrl+C.

# See process group IDs
ps -eo pid,pgid,sid,cmd | head -20

# Send a signal to an entire process group
kill -TERM -12345   # Note the negative PID -- means "process group 12345"

This is also how kill 0 works in a trap handler -- it sends the signal to every process in the current process group.

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 11 Recap: Signals Deep Dive                             |
|------------------------------------------------------------------|
|                                                                  |
|  - Signals are asynchronous notifications to processes.          |
|  - SIGTERM (15): graceful termination request (default kill).    |
|  - SIGKILL (9): force kill, CANNOT be caught -- last resort.    |
|  - SIGINT (2): keyboard interrupt (Ctrl+C).                     |
|  - SIGHUP (1): hangup / reload config for daemons.              |
|  - SIGSTOP/SIGCONT: pause and resume (SIGSTOP can't be caught). |
|  - SIGUSR1/SIGUSR2: user-defined, application-specific.         |
|  - trap in bash lets scripts handle signals.                     |
|  - Always trap EXIT for cleanup of temp files and locks.         |
|  - Daemons use SIGHUP to reload config without downtime.        |
|  - Use signal names (SIGTERM) not numbers (15) for portability. |
|  - Try SIGTERM first; only SIGKILL as last resort.              |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Signal Identification

Run kill -l and identify the signal number for: SIGHUP, SIGINT, SIGTERM, SIGKILL, SIGUSR1, SIGCHLD, SIGSTOP, SIGCONT. Then write the keyboard shortcut that sends SIGINT, SIGQUIT, and SIGTSTP.

Exercise 2: Trap Practice

Write a script that creates three temporary files on startup and removes them all when it receives SIGINT or SIGTERM, or when it exits normally. Test it by killing it with different signals and verifying the temp files are gone.

Exercise 3: Process Group Experiment

Start a pipeline: sleep 100 | sleep 200 | sleep 300 &. Find the process group ID for all three processes (use ps -eo pid,pgid,cmd | grep sleep). Send SIGTERM to the process group using kill -TERM -PGID. Verify all three are gone.

Exercise 4: SIGHUP Reload

Write a simple script that reads a "config file" (just a text file with a key=value pair) on startup and re-reads it whenever it receives SIGHUP. Test it by changing the config file and sending SIGHUP.

Bonus Challenge

Write a bash script that acts as a simple process monitor. It takes a command as arguments, runs it, and if the command dies unexpectedly (exit code not 0), it restarts it automatically. Use trap to handle SIGCHLD or simply loop with wait. The monitor itself should handle SIGTERM gracefully by forwarding it to the child process and then exiting.

Linux Book: From First Boot to Production