Linux Book

From First Boot to Production

Welcome to Linux Book — a hands-on guide that takes you from your first ls command to confidently managing production Linux systems. This is not a reference manual you skim and shelve. Every chapter puts commands in your hands, explains what happens underneath, and builds real-world muscle memory.

Who This Book Is For

• Aspiring sysadmins who want to go beyond clicking buttons in a GUI
• DevOps engineers who need deep Linux fluency, not just copy-pasted scripts
• Developers who deploy to Linux but never really learned it properly
• Students and career-changers building a foundation for cloud, security, or infrastructure roles
• Anyone who's tired of blindly following tutorials without understanding why

You don't need programming experience. You don't need a computer science degree. You need curiosity and a terminal.

What Makes This Book Different

Every tool covered is open source. No proprietary lock-in, no vendor-specific tricks. What you learn here works on any Linux distribution, any cloud provider, any hardware.

Every chapter follows the same rhythm:

1. "Why This Matters" — A real-world scenario that shows you exactly when you'd need this skill
2. "Try This Right Now" — Copy-paste commands you can run immediately
3. Concept deep-dives — Clear explanations with ASCII diagrams you can actually read
4. "Think About It" — Mid-chapter questions that make you stop and reason
5. Hands-on blocks — Step-by-step walkthroughs with real command output
6. "Debug This" — Broken scenarios for you to diagnose (because that's the real job)
7. "What Just Happened?" — Recap boxes that crystallize each section
8. "Try This" — End-of-chapter exercises with bonus challenges

Where commands differ between distributions (Debian/Ubuntu vs RHEL/Fedora vs Arch), you'll see Distro Notes calling out the differences. Where commands can destroy data, you'll see safety warnings before you run anything dangerous.

What You'll Master

┌──────────────────────────────────────────────────────────┐
│                    Linux Book                          │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  Part I     The Ground Floor                             │
│             What Linux is, choosing a distro,            │
│             installing it, meeting the shell             │
│                                                          │
│  Part II    Filesystem & "Everything Is a File"          │
│             Hierarchy, permissions, disks, inodes        │
│                                                          │
│  Part III   Users, Processes, Signals & IPC              │
│             Access control, job control, the kernel,     │
│             how Linux boots                              │
│                                                          │
│  Part IV    systemd & Service Management                 │
│             Units, services, journald logging            │
│                                                          │
│  Part V     Bash, Regex & Text Processing                │
│             Shell mastery, scripting, sed, awk,          │
│             cron, automation                             │
│                                                          │
│  Part VI    Essential Tools                              │
│             Vim, tmux, Git for operations                │
│                                                          │
│  Part VII   Networking Fundamentals                      │
│             OSI/TCP-IP, subnetting, DNS, DHCP            │
│                                                          │
│  Part VIII  Linux Networking in Practice                  │
│             Interfaces, firewalls, routing, SSH,         │
│             WireGuard VPN                                │
│                                                          │
│  Part IX    Security, PKI & Cryptography                 │
│             Hardening, TLS/SSL, OpenSSL, ACME,           │
│             SELinux, AppArmor                            │
│                                                          │
│  Part X     Web Servers & Load Balancing                 │
│             HTTP, Nginx, Apache, HAProxy                 │
│                                                          │
│  Part XI    Storage, Backup & Recovery                   │
│             LVM, RAID, NFS, backup strategies,           │
│             disaster recovery                            │
│                                                          │
│  Part XII   Performance & Monitoring                     │
│             top/htop, memory, disk I/O, network,         │
│             resource limits                              │
│                                                          │
│  Part XIII  Package Management & Software                │
│             apt/dnf/pacman, compiling from source,       │
│             shared libraries, custom kernels             │
│                                                          │
│  Part XIV   Containers & Virtualization                  │
│             Cgroups, namespaces, Docker, Podman,         │
│             LXC/LXD, orchestration                      │
│                                                          │
│  Part XV    Configuration Management & DevOps            │
│             IaC concepts, Ansible, CI/CD,                │
│             Prometheus + Grafana                         │
│                                                          │
│  Part XVI   Linux in the Real World                      │
│             Enterprise, embedded, cloud, databases,      │
│             NTP, troubleshooting methodology             │
│                                                          │
│  Appendices Command reference, config files,             │
│             lab setup, glossary, further reading         │
│                                                          │
└──────────────────────────────────────────────────────────┘

How to Use This Book

If you're brand new to Linux: Start at Chapter 1 and go straight through. Each chapter builds on the previous ones.

If you have some Linux experience: Jump to whatever interests you. Each Part is reasonably self-contained, and cross-references point you to prerequisites when needed.

If you're studying for a certification (RHCSA, LFCS, etc.): This book covers the practical skills those exams test. Use the appendices as quick references.

Set up a lab. Appendix C walks you through creating a safe practice environment. You can use a virtual machine, a spare laptop, WSL2 on Windows, or a cheap cloud instance. The important thing is: type the commands. Reading about Linux is like reading about swimming — you have to get in the water.

Conventions

Throughout this book:

• monospace text indicates commands, file paths, or configuration values
• Bold text marks important terms on first use
• Commands prefixed with $ run as a regular user; commands prefixed with # require root
• Output blocks show real terminal output — what you see should match

Distro Note: Boxes like this highlight where commands or paths differ between Debian/Ubuntu, RHEL/Fedora, and Arch Linux.

Warning: Boxes like this appear before commands that can destroy data or break your system. Read them before you type.

Think About It: Boxes like this pose questions to help you reason through concepts before we explain them.

Let's begin.

What Is Linux and Why It's Everywhere

Why This Matters

Picture this: you open your phone to check the weather, stream music on your commute, tap your card at a payment terminal, and then sit down at work to deploy code to a cloud server. Every single one of those actions just touched Linux. The phone runs Android (built on the Linux kernel). The music service runs on Linux servers. The payment terminal likely runs embedded Linux. The cloud server? Almost certainly Linux.

Linux is not some niche operating system for hobbyists. It powers 96% of the world's top one million web servers, all of the world's top 500 supercomputers, the International Space Station, Tesla's infotainment systems, and the majority of the world's cloud infrastructure. When you learn Linux, you are learning the operating system that runs the modern world.

This chapter answers the most fundamental question: what exactly is Linux, where did it come from, and why should you invest your time mastering it?

Try This Right Now

If you already have access to any Linux system (a server, a VM, WSL2, or even an Android phone with Termux), open a terminal and type:

$ uname -a

You should see something like:

Linux myhost 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

That single line tells you:

• You are running Linux (the kernel)
• Your hostname is myhost
• The kernel version is 6.1.0-18-amd64
• It was compiled with SMP (symmetric multiprocessing) and PREEMPT (preemptive scheduling)
• The distribution is Debian
• The architecture is x86_64 (64-bit Intel/AMD)

If you do not have a Linux system yet, that is perfectly fine. Chapter 3 walks you through setting one up. For now, read on and come back to run the commands later.

What Linux Actually Is

When people say "Linux," they usually mean one of two things, and the difference matters.

The Linux Kernel

In the strictest sense, Linux is a kernel -- the core piece of software that sits between your hardware and everything else. The kernel's job is enormous and critical:

┌─────────────────────────────────────────────────┐
│              User Applications                   │
│         (Firefox, vim, python, nginx)            │
├─────────────────────────────────────────────────┤
│              System Libraries                    │
│           (glibc, libssl, libpthread)            │
├─────────────────────────────────────────────────┤
│           System Calls Interface                 │
│      (open, read, write, fork, exec, ...)        │
├─────────────────────────────────────────────────┤
│             THE LINUX KERNEL                     │
│  ┌───────────┬──────────┬──────────────────┐     │
│  │ Process   │ Memory   │ Filesystem       │     │
│  │ Scheduler │ Manager  │ Layer (VFS)      │     │
│  ├───────────┼──────────┼──────────────────┤     │
│  │ Network   │ Device   │ Security         │     │
│  │ Stack     │ Drivers  │ (SELinux, etc.)  │     │
│  └───────────┴──────────┴──────────────────┘     │
├─────────────────────────────────────────────────┤
│                  HARDWARE                        │
│       (CPU, RAM, Disk, Network Card, GPU)        │
└─────────────────────────────────────────────────┘

The kernel handles:

• Process management -- creating, scheduling, and terminating programs
• Memory management -- allocating RAM to programs, swapping to disk
• Filesystem access -- reading and writing files on disks
• Device drivers -- talking to hardware (network cards, USB devices, GPUs)
• Networking -- implementing TCP/IP, routing packets
• Security -- enforcing permissions, capabilities, and mandatory access controls

Without the kernel, your applications have no way to talk to the hardware. With a broken kernel, nothing works. The kernel is the foundation.

The Linux Operating System (Distribution)

A kernel alone is not very useful. You need a shell to type commands in, utilities to copy files and manage processes, a package manager to install software, an init system to boot the machine, and thousands of other tools. When all of these are bundled together with the Linux kernel, you get a Linux distribution (or "distro").

┌──────────────────────────────────────────────┐
│          A Linux Distribution                 │
│                                               │
│   Linux Kernel                                │
│   + GNU Core Utilities (ls, cp, grep, etc.)   │
│   + Shell (bash, zsh)                         │
│   + Init System (systemd, OpenRC)             │
│   + Package Manager (apt, dnf, pacman)        │
│   + Libraries (glibc, openssl)                │
│   + Desktop Environment (optional)            │
│   + Default Applications                      │
│   + Configuration & Branding                  │
└──────────────────────────────────────────────┘

When someone says "I run Linux," they almost always mean "I run a Linux distribution." Ubuntu, Fedora, Debian, Arch Linux, Red Hat Enterprise Linux -- these are all distributions. They all share the same kernel (or a version of it) but differ in what tools they bundle, how they manage packages, and what their default configurations look like.

Think About It: If all Linux distributions share the same kernel, why are there hundreds of different distributions? What would be different about each one?

A Brief History: How We Got Here

Understanding where Linux came from helps you understand why it works the way it does.

Unix: The Ancestor (1969)

The story starts at AT&T's Bell Labs in 1969. Ken Thompson and Dennis Ritchie created Unix, an operating system built around a few powerful ideas:

• Everything is a file -- devices, processes, even network connections can be accessed like files
• Small, sharp tools -- each program does one thing well
• Text as a universal interface -- programs communicate through plain text streams
• Pipelines -- you can chain small tools together to do complex things

These ideas were radical in 1969 and they remain the philosophical backbone of Linux today. When you pipe grep into sort into uniq later in this book, you are using a design philosophy that is over fifty years old -- because it works.

GNU: The Free Software Movement (1983)

By the early 1980s, Unix had become proprietary and expensive. Richard Stallman, a programmer at MIT, decided that software should be free -- not "free" as in price, but "free" as in freedom. In 1983, he launched the GNU Project (a recursive acronym: "GNU's Not Unix") to create a completely free Unix-like operating system.

By 1991, the GNU Project had built almost everything needed for a complete operating system: a compiler (GCC), a shell (Bash), core utilities (ls, cp, mv, grep), text editors (Emacs), and libraries (glibc). They had everything except the most critical piece: a working kernel.

Linux: The Missing Kernel (1991)

On August 25, 1991, a 21-year-old Finnish computer science student named Linus Torvalds posted this message to the comp.os.minix newsgroup:

"Hello everybody out there using minix - I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones."

That "hobby" became the Linux kernel. When combined with the GNU tools, the result was a complete, free, Unix-like operating system. This is why some people call it GNU/Linux -- the kernel is Linux, but much of the userspace came from the GNU Project. In practice, most people just say "Linux."

The Timeline

1969  Unix created at Bell Labs
1983  Richard Stallman launches the GNU Project
1985  Free Software Foundation (FSF) established
1989  GNU General Public License (GPL) v1 released
1991  Linus Torvalds releases Linux kernel 0.01
1992  Linux re-licensed under GPL v2
1993  Debian and Slackware (first major distros) appear
1994  Linux kernel 1.0 released
1996  Linux kernel 2.0 -- SMP support
2003  RHEL (Red Hat Enterprise Linux) launched
2004  Ubuntu 4.10 ("Warty Warthog") released
2007  Android announced (built on Linux kernel)
2008  Linux kernel used in first Android phones
2011  Linux kernel 3.0 released
2013  All Top500 supercomputers run Linux
2015  Linux kernel 4.0 -- live patching
2019  Microsoft ships WSL2 with a real Linux kernel
2020  Linux kernel 5.x series
2022  Linux kernel 6.0 released
2024  Linux kernel 6.x continues active development

Think About It: Linus Torvalds said his project "won't be big and professional." It now runs on everything from wristwatches to supercomputers. What do you think allowed a hobby project to grow into the dominant operating system?

The GPL and Open Source Philosophy

Linux is released under the GNU General Public License version 2 (GPL v2). This is not just a legal technicality -- it is the reason Linux exists in its current form.

The GPL v2 says:

1. Freedom to use -- You can run the software for any purpose
2. Freedom to study -- You can read and modify the source code
3. Freedom to share -- You can distribute copies
4. Freedom to improve -- You can distribute your modifications

The critical requirement: if you distribute a modified version of GPL-licensed software, you must also distribute the source code under the same license. This is called copyleft -- it ensures that Linux and its derivatives remain free forever.

This is why:

• You can download the entire Linux kernel source code right now, for free
• Android, which is built on Linux, must make its kernel modifications public
• Companies like Red Hat, Canonical, and SUSE can sell support and services around Linux, but cannot lock up the code itself
• You can build your own Linux distribution tomorrow if you want to

Open Source vs Free Software

You will hear both terms. They overlap heavily but differ in emphasis:

• Free Software (Stallman's philosophy): emphasizes user freedom as a moral imperative
• Open Source (coined 1998): emphasizes the practical benefits of collaborative development

Linux lives comfortably in both camps. For this book, what matters is this: every tool we cover is open source. You can inspect its code, modify it, and learn from it. No black boxes.

Where Linux Runs

The scope of Linux adoption is staggering. Let us look at the numbers.

Servers and Cloud

• 96.3% of the top one million web servers run Linux
• 90%+ of public cloud workloads run on Linux
• All major cloud providers (AWS, Google Cloud, Azure) offer Linux as the primary server OS
• AWS's own infrastructure runs on a custom Linux distribution (Amazon Linux)
• Google runs a custom Linux internally across billions of containers

When you visit nearly any website -- Google, Netflix, Amazon, Wikipedia, Reddit -- you are talking to Linux servers.

Supercomputers

• 100% of the world's top 500 supercomputers run Linux
• This has been the case since November 2017
• Before Linux dominated, the list included Unix, Windows, and custom operating systems
• Linux won because it is free, customizable, and scales from one core to millions

Mobile Devices

• Android runs on the Linux kernel
• Android holds roughly 72% of the global mobile OS market
• That means the majority of smartphones in the world run Linux

Embedded Systems and IoT

• Smart TVs (Samsung's Tizen, LG's webOS -- both Linux-based)
• Routers and network equipment
• Industrial control systems
• Automotive infotainment (Tesla, Toyota, BMW)
• Digital signage, ATMs, point-of-sale terminals

Space

• The International Space Station migrated from Windows to Linux (Debian) in 2013
• NASA's Mars Ingenuity helicopter runs Linux
• SpaceX uses Linux on its flight computers

Desktop

Linux has roughly 4% of the desktop market. This is small compared to Windows and macOS, but it represents tens of millions of users. The desktop is also where you will likely do most of your learning.

┌────────────────────────────────────────────┐
│         Where Linux Runs                    │
│                                             │
│  Supercomputers     ████████████████ 100%   │
│  Cloud Servers      ███████████████░  90%+  │
│  Web Servers        ███████████████░  96%   │
│  Mobile (Android)   ███████████░░░░░  72%   │
│  Embedded/IoT       ████████░░░░░░░░  ~50%  │
│  Desktop            █░░░░░░░░░░░░░░░  ~4%   │
│                                             │
└────────────────────────────────────────────┘

Think About It: Linux dominates servers, supercomputers, and phones but has only about 4% of the desktop market. What factors might explain this difference?

Why Learn Linux?

Career Demand

Nearly every job posting for system administration, DevOps engineering, site reliability engineering, cloud engineering, cybersecurity, or backend development lists Linux as a required or preferred skill. This is not a trend -- it has been the case for decades and the demand is growing.

Certifications like the RHCSA (Red Hat Certified System Administrator), LFCS (Linux Foundation Certified System Administrator), and AWS Solutions Architect all require Linux knowledge.

Understanding Over Clicking

On Windows or macOS, you often configure things through graphical menus. You click, toggle, and hope. On Linux, configuration is done through text files and commands. This seems harder at first, but it has profound advantages:

• Reproducibility -- you can script everything
• Version control -- you can track every change with Git
• Remote management -- you can manage a server 10,000 miles away over SSH
• Automation -- you can configure a thousand servers identically
• Debugging -- when something breaks, you can read the configuration and logs to understand exactly what happened

Control and Transparency

Linux hides nothing from you. If you want to know what your operating system is doing right now:

$ ps aux      # show every running process
$ top         # watch CPU and memory usage live
$ ss -tlnp    # see which programs are listening on network ports
$ dmesg       # read kernel messages
$ journalctl  # read system logs

On proprietary systems, much of this is hidden behind opaque interfaces. On Linux, it is all text, all accessible, all yours.

Freedom

With Linux, you are not locked into any vendor's ecosystem. You can:

• Choose your desktop environment (or use none)
• Choose your init system
• Choose your package manager
• Build the system from scratch if you want (see: Linux From Scratch, Gentoo)
• Run it on hardware that proprietary vendors have abandoned
• Audit every line of code that runs on your machine

This is not just philosophical. If a vendor discontinues a product, Linux keeps running. If a government mandates backdoors in proprietary software, you can verify that your Linux system has none.

Kernel vs Userspace: A Critical Distinction

One concept that will recur throughout this book is the boundary between kernel space and user space.

┌──────────────────────────────────────────────┐
│              USER SPACE                       │
│                                               │
│  Your programs, shells, scripts, daemons      │
│  They CANNOT directly access hardware         │
│  They MUST ask the kernel via system calls     │
│                                               │
├──────────────────────────────────────────────┤
│          SYSTEM CALL BOUNDARY                 │
│     (the controlled gateway)                  │
├──────────────────────────────────────────────┤
│              KERNEL SPACE                     │
│                                               │
│  The kernel, device drivers, kernel modules   │
│  Full access to hardware and memory           │
│  Runs with highest privilege                  │
│                                               │
├──────────────────────────────────────────────┤
│              HARDWARE                         │
└──────────────────────────────────────────────┘

• Kernel space: The kernel runs with full access to the hardware. A bug here can crash the entire system.
• User space: Everything else -- your shell, your web browser, your web server. Programs here cannot directly touch hardware. They must make system calls to ask the kernel to do things on their behalf.

This separation is what makes Linux stable. If Firefox crashes, the kernel keeps running. If a Python script goes rogue and tries to access memory it should not, the kernel terminates it. The kernel is the bouncer.

You will encounter this boundary again and again: when we discuss processes (Chapter 10), the kernel (Chapter 13), and containers (Chapter 62-66).

Hands-On: Exploring Your (or Any) Linux System

If you have a Linux system available, try these commands. Each one reveals something about the system.

What kernel are you running?

$ uname -r
6.1.0-18-amd64

The kernel version follows a pattern: major.minor.patch. The 6.1.0 tells you this is major version 6, minor version 1, patch 0.

What distribution is this?

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
ID=debian
...

This file exists on virtually every modern Linux distribution and gives you standardized information about which distro and version you are running.

How long has the system been running?

$ uptime
 14:32:07 up 47 days,  3:21,  2 users,  load average: 0.15, 0.10, 0.08

This tells you the system has been running for 47 days without a reboot. Linux servers commonly run for months or years between reboots.

What CPU does the system have?

$ lscpu | head -15
Architecture:            x86_64
CPU op-mode(s):          32-bit, 64-bit
Address sizes:           46 bits physical, 48 bits virtual
Byte Order:              Little Endian
CPU(s):                  4
On-line CPU(s) list:     0-3
Vendor ID:               GenuineIntel
Model name:              Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
...

How much memory is available?

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       2.1Gi       3.4Gi       256Mi       2.2Gi       5.1Gi
Swap:          2.0Gi          0B       2.0Gi

The -h flag means "human-readable" -- you will see this flag on many Linux commands.

How much disk space?

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   12G   35G  26% /

Every one of these commands is a small tool that does one thing well. That is the Unix philosophy at work.

The Linux Development Model

The Linux kernel is the largest collaborative software project in human history. Some numbers to appreciate the scale:

• Over 30 million lines of code in the kernel source
• Thousands of contributors per release cycle
• A new stable release approximately every 9-10 weeks
• Companies contributing include Intel, Google, Red Hat, Samsung, Microsoft, Meta, Amazon, and hundreds more
• Linus Torvalds remains the project's Benevolent Dictator For Life (BDFL) with final say on what gets merged

The development process is remarkably open:

1. Developers submit patches to public mailing lists
2. Patches are reviewed by maintainers and other developers
3. Accepted patches flow through subsystem maintainer trees
4. Linus merges from maintainer trees during the merge window
5. A series of release candidates (rc1, rc2, ...) are tested
6. A stable release is tagged

Every email, every patch, every discussion is public. You can read the kernel mailing list archives and see exactly why every change was made.

What Just Happened?

┌─────────────────────────────────────────────────────┐
│                                                      │
│  In this chapter, you learned:                       │
│                                                      │
│  - Linux is a KERNEL, not an operating system.       │
│    A distribution = kernel + userspace tools.         │
│                                                      │
│  - Linux descends from Unix (1969), combines the     │
│    GNU tools (1983) with Linus Torvalds' kernel      │
│    (1991), and is licensed under the GPL v2.          │
│                                                      │
│  - Linux runs everywhere: 100% of supercomputers,    │
│    96% of web servers, 72% of mobile phones           │
│    (Android), the ISS, Mars helicopters, and cars.    │
│                                                      │
│  - The GPL ensures Linux remains free and open.       │
│    You can read, modify, and redistribute the code.   │
│                                                      │
│  - The kernel/userspace boundary is a key concept:    │
│    programs ask the kernel to access hardware via      │
│    system calls.                                      │
│                                                      │
│  - Learning Linux gives you career skills, system     │
│    understanding, and freedom from vendor lock-in.    │
│                                                      │
│  - Commands you met: uname, cat, uptime, lscpu,      │
│    free, df                                          │
│                                                      │
└─────────────────────────────────────────────────────┘

Try This

Exercises

1. Research exercise: Go to kernel.org and find the current stable kernel version. Compare it to the version shown by uname -r on any Linux system you have access to. Is the system up to date?

2. Exploration exercise: If you have a Linux system, run cat /etc/os-release and identify: the distribution name, the version number, and the ID. Write these down -- you will need them in Chapter 2.

3. Reading exercise: Read the original Linus Torvalds newsgroup post from 1991 (search for "linux history linus newsgroup 1991"). What hardware was he targeting? What features was he not planning to support?

4. Thinking exercise: We said "everything is a file" is a Unix/Linux philosophy. What might it mean for a device (like a hard disk or a keyboard) to be represented as a file? What advantages might this give?

5. Career exercise: Search a job board for "Linux system administrator" or "DevOps engineer" positions. List five specific Linux skills that appear in multiple job descriptions.

Bonus Challenge

Find three devices in your home that likely run Linux (hint: your router is almost certainly one). For each device, research what Linux-based operating system it uses and what kernel version it likely runs.

What's Next

Now that you understand what Linux is and why it matters, the next question is: which Linux should you use? There are hundreds of distributions, and Chapter 2 will help you navigate this landscape and choose the right one for your goals.

Choosing Your Linux

Why This Matters

You have just accepted a new job as a junior DevOps engineer. On day one, your team lead says: "Spin up two servers -- one Ubuntu 22.04 for the web tier, one Rocky Linux 9 for the database tier." You nod confidently, but inside you are thinking: Why two different distributions? What is the difference? Does it matter?

It does matter. Distributions differ in their package managers, release schedules, default configurations, security models, community support, and commercial backing. Choosing the wrong distribution for a task can mean fighting your tools instead of using them. Choosing the right one means your environment works with you.

This chapter maps the Linux distribution landscape so you can make informed choices -- whether you are setting up a learning environment, a personal desktop, or a production server fleet.

Try This Right Now

If you have any Linux system available, let us immediately figure out what distribution and family you are running:

$ cat /etc/os-release

NAME="Ubuntu"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
ID=ubuntu
ID_LIKE=debian
VERSION_ID="22.04"
PRETTY_NAME="Ubuntu 22.04.3 LTS"

Two key fields:

• ID tells you the exact distribution (ubuntu)
• ID_LIKE tells you the family it belongs to (debian)

Now check which package manager is available:

$ which apt dnf pacman zypper 2>/dev/null
/usr/bin/apt

Whatever comes back tells you which distribution family you are in. If you see apt, you are in the Debian family. If dnf, the Red Hat family. If pacman, the Arch family. If zypper, the SUSE family.

What Is a Distribution?

In Chapter 1, we established that Linux is a kernel. A distribution (distro) takes that kernel and adds everything needed to make a usable operating system:

┌─────────────────────────────────────────────────┐
│            Linux Distribution                    │
│                                                  │
│  ┌────────────────────────────────────────────┐  │
│  │  Kernel (shared across all distros)        │  │
│  │  (possibly with distro-specific patches)   │  │
│  └────────────────────────────────────────────┘  │
│                                                  │
│  ┌────────────────────────────────────────────┐  │
│  │  Core Userspace:                           │  │
│  │  - Init system (systemd, OpenRC, runit)    │  │
│  │  - Shell (bash, zsh)                       │  │
│  │  - Core utilities (coreutils)              │  │
│  │  - System libraries (glibc, musl)          │  │
│  └────────────────────────────────────────────┘  │
│                                                  │
│  ┌────────────────────────────────────────────┐  │
│  │  Package Management:                       │  │
│  │  - Package manager (apt, dnf, pacman)      │  │
│  │  - Software repositories                   │  │
│  │  - Dependency resolution                   │  │
│  └────────────────────────────────────────────┘  │
│                                                  │
│  ┌────────────────────────────────────────────┐  │
│  │  Extras (varies by distro):                │  │
│  │  - Desktop environment (GNOME, KDE, none)  │  │
│  │  - Default applications                    │  │
│  │  - Configuration tools                     │  │
│  │  - Branding and theming                    │  │
│  └────────────────────────────────────────────┘  │
│                                                  │
└─────────────────────────────────────────────────┘

Every distribution makes choices: which kernel version to ship, which init system to use, how to manage packages, what default software to include, and how frequently to release updates. These choices create the identity and trade-offs of each distribution.

The Major Distribution Families

There are hundreds of Linux distributions, but nearly all descend from a handful of families. Understanding the families is more important than memorizing individual distros.

The Debian Family

Debian (1993)
├── Ubuntu (2004)
│   ├── Linux Mint
│   ├── Pop!_OS
│   ├── Elementary OS
│   └── Kubuntu, Xubuntu, Lubuntu (flavors)
├── Kali Linux (security/pentesting)
├── Raspberry Pi OS
└── Devuan (Debian without systemd)

Debian is the grandfather of this family. Founded in 1993 by Ian Murdock, it is one of the oldest distributions still in active development. Debian prioritizes stability and freedom above all else.

Key characteristics:

• Package format: .deb
• Package manager: apt (Advanced Package Tool), dpkg for low-level operations
• Release model: Point release, roughly every 2 years
• Philosophy: rock-solid stability, software freedom
• Releases are named after Toy Story characters (Bookworm, Bullseye, Buster)

Ubuntu, created by Canonical in 2004, is the most popular Debian derivative. It takes Debian's foundation and adds polish, newer software, and commercial support. Ubuntu has its own release cadence:

• Regular releases every 6 months (April and October)
• LTS (Long Term Support) releases every 2 years, supported for 5 years
• LTS versions are what you use on servers (e.g., Ubuntu 22.04 LTS, Ubuntu 24.04 LTS)

When to choose Debian/Ubuntu:

• Ubuntu LTS is the most common choice for cloud servers and containers
• Debian stable is ideal when you want maximum stability and minimal surprises
• Enormous package repositories (over 59,000 packages in Debian)
• Most online tutorials and guides target Ubuntu

Distro Note: When this book shows commands using apt, they apply to Debian, Ubuntu, and all their derivatives.

The Red Hat Family

Red Hat Enterprise Linux (RHEL)
├── CentOS Stream (upstream preview of RHEL)
├── Rocky Linux (community RHEL rebuild)
├── AlmaLinux (community RHEL rebuild)
├── Oracle Linux (Oracle's RHEL rebuild)
└── Amazon Linux (AWS's RHEL-like distro)

Fedora (community upstream of RHEL)

Red Hat Enterprise Linux (RHEL) is the dominant distribution in enterprise and corporate environments. Red Hat (now owned by IBM) sells subscriptions that include support, security patches, and certification.

Fedora is the community distribution that serves as a testing ground for technologies that eventually make it into RHEL. Fedora ships newer software and is more cutting-edge.

Key characteristics:

• Package format: .rpm (RPM Package Manager)
• Package manager: dnf (Dandified YUM), rpm for low-level operations
• Release model: RHEL has point releases with 10-year support cycles; Fedora releases every ~6 months
• Philosophy: enterprise stability, commercial support, certification

The CentOS Story: For years, CentOS was a free, community rebuild of RHEL -- binary-compatible, just without the Red Hat branding and support. In 2020, Red Hat controversially shifted CentOS to "CentOS Stream," an upstream preview of RHEL rather than a downstream rebuild. This broke the free-as-in-RHEL promise, and two community projects emerged to fill the gap:

• Rocky Linux -- founded by one of the original CentOS creators
• AlmaLinux -- backed by CloudLinux

Both aim to be 1:1 compatible with RHEL, free of charge.

When to choose RHEL/Fedora family:

• Enterprise environments that require commercial support and certification
• When your organization already uses RHEL (consistency matters)
• When preparing for the RHCSA or RHCE certifications
• Fedora for desktop users who want cutting-edge software with good stability

Distro Note: When this book shows commands using dnf, they apply to Fedora, RHEL, Rocky Linux, AlmaLinux, and CentOS Stream.

The Arch Family

Arch Linux (2002)
├── Manjaro
├── EndeavourOS
├── Garuda Linux
└── SteamOS 3.0 (Valve's gaming OS)

Arch Linux takes a radically different approach: minimalism and user control. Arch gives you nothing by default -- no desktop environment, no hand-holding, no automatic configuration. You build the system yourself, piece by piece.

Key characteristics:

• Package format: .pkg.tar.zst
• Package manager: pacman
• Release model: Rolling release -- there are no "versions." You install once and continuously update
• Philosophy: simplicity, user-centricity, cutting-edge software
• The Arch Wiki is widely regarded as the best Linux documentation on the internet (useful even for non-Arch users)
• AUR (Arch User Repository) -- a community repository with packages for virtually any software

When to choose Arch:

• When you want to deeply understand how Linux works (the installation teaches you)
• When you want the very latest software versions
• Desktop use by experienced users
• NOT recommended for production servers (rolling releases are unpredictable for stability)

Distro Note: When this book shows commands using pacman, they apply to Arch Linux and its derivatives like Manjaro and EndeavourOS.

The SUSE Family

SUSE Linux Enterprise (SLE)
├── SUSE Linux Enterprise Server (SLES)
└── SUSE Linux Enterprise Desktop (SLED)

openSUSE
├── openSUSE Leap (point release, based on SLE)
└── openSUSE Tumbleweed (rolling release)

SUSE is another major enterprise distribution, particularly popular in Europe. openSUSE is its community counterpart.

Key characteristics:

• Package format: .rpm
• Package manager: zypper
• YaST (Yet another Setup Tool) -- a comprehensive system administration tool
• Release model: Leap is point release; Tumbleweed is rolling
• Strong in European enterprise, particularly Germany

When to choose SUSE:

• Enterprise environments that already use SUSE infrastructure
• When you want YaST for administration
• openSUSE Tumbleweed for a polished rolling-release desktop

Other Notable Distributions

• Alpine Linux -- tiny (5 MB base image), uses musl libc instead of glibc, and apk package manager. Extremely popular for Docker containers
• Gentoo -- compiles everything from source for maximum optimization. Uses emerge package manager. Deep learning experience
• Slackware -- one of the oldest surviving distributions (1993). Minimalist, does not auto-resolve dependencies
• NixOS -- declarative system configuration. Entire OS defined in a config file. Revolutionary approach to reproducibility
• Void Linux -- independent distribution using runit init system instead of systemd

Think About It: Why do you think there are so many Linux distributions? In most other software ecosystems, there is one dominant option. What about Linux encourages this diversity?

Package Managers: The Heart of a Distribution

The package manager is arguably the single most important difference between distribution families. It determines how you install, update, and remove software.

What a Package Manager Does

┌─────────────────────────────────────────┐
│         Package Manager                  │
│                                          │
│  1. Downloads software from repositories │
│  2. Resolves dependencies automatically  │
│  3. Installs files to correct locations  │
│  4. Tracks what is installed             │
│  5. Handles upgrades and removals        │
│  6. Verifies package integrity (GPG)     │
│                                          │
└─────────────────────────────────────────┘

Without a package manager, installing software means manually downloading source code, resolving dependencies by hand, compiling, and tracking what you installed. Package managers automate all of this.

Side-by-Side Comparison

The syntax differs, but the concepts are identical. Once you understand one package manager deeply, learning another takes hours, not days. Chapter 57 dives much deeper into package management.

Think About It: Why do you think Linux uses centralized package repositories instead of having users download installers from individual websites (like Windows traditionally does)? What security and reliability advantages does this provide?

Release Models: Rolling vs Point Release

This is one of the most consequential choices a distribution makes.

Point Release (Fixed Release)

Examples: Debian, Ubuntu LTS, RHEL, Rocky Linux, openSUSE Leap

Timeline:
──────────────────────────────────────────────────────────>

  Ubuntu 22.04 LTS        Ubuntu 24.04 LTS
  ┌────────────────┐      ┌────────────────┐
  │ Released        │      │ Released        │
  │ Apr 2022        │      │ Apr 2024        │
  │                 │      │                 │
  │ Security fixes  │      │ Security fixes  │
  │ only until      │      │ only until      │
  │ Apr 2027        │      │ Apr 2029        │
  └────────────────┘      └────────────────┘

• Software versions are frozen at release time
• Only security patches and critical bug fixes are backported
• Major software upgrades require upgrading to the next release
• Pros: predictable, stable, well-tested combinations of software
• Cons: software versions can feel old; you may need newer features

Point release distributions are the standard for servers. You do not want your production database server spontaneously updating to a new major version of PostgreSQL.

Rolling Release

Examples: Arch Linux, openSUSE Tumbleweed, Gentoo, Void Linux

Timeline:
──────────────────────────────────────────────────────────>
  Arch Linux
  ┌──────────────────────────────────────────────────────┐
  │ Install once, update forever                          │
  │                                                       │
  │ Kernel 6.1 → 6.2 → 6.3 → 6.4 → 6.5 → 6.6 → ...    │
  │ Firefox 110 → 111 → 112 → 113 → 114 → 115 → ...    │
  │ Python 3.11 → 3.12 → ...                            │
  │                                                       │
  │ Every update brings the latest stable version         │
  └──────────────────────────────────────────────────────┘

• Software is continuously updated to the latest stable versions
• There are no release "versions" -- you are always on the latest
• Pros: always up to date, no painful upgrade cycles
• Cons: updates can introduce breaking changes, less predictable for servers

Semi-Rolling / Hybrid Models

Some distributions blend both approaches:

• Fedora: releases every ~6 months but moves fast between releases. Not technically rolling, but much more current than RHEL or Debian stable
• CentOS Stream: continuously updated, slightly ahead of RHEL. A rolling preview of the next RHEL point release
• Ubuntu non-LTS: new release every 6 months with only 9 months of support. Good for desktops, risky for servers

Choosing for Your Use Case

For Learning (This Book)

Recommended: Ubuntu LTS or Fedora

Either of these will work perfectly for every exercise in this book. Both are well-documented, widely used, and easy to install. Ubuntu LTS is the safer default because more tutorials and guides target it.

If you want to challenge yourself and learn deeply, Arch Linux is an incredible teacher -- the installation process alone teaches you partitioning, bootloaders, and system configuration.

For Desktop Daily Use

For Servers (Production)

For Specific Roles

Hands-On: Comparing Distributions Side by Side

You do not need to install every distribution to compare them. Let us use Docker to quickly peek inside different distributions.

Note: If you do not have Docker yet, that is fine -- just read through this section. You will install Docker in Chapter 63. This is here so you can come back to it later.

# Peek inside Ubuntu
$ docker run --rm -it ubuntu:22.04 bash
root@abc123:/# cat /etc/os-release | head -5
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
ID=ubuntu

root@abc123:/# apt --version
apt 2.4.11 (amd64)

root@abc123:/# exit

# Peek inside Fedora
$ docker run --rm -it fedora:39 bash
[root@def456 /]# cat /etc/os-release | head -5
NAME="Fedora Linux"
VERSION="39 (Container Image)"
ID=fedora
VERSION_ID=39
PRETTY_NAME="Fedora Linux 39 (Container Image)"

[root@def456 /]# dnf --version
4.18.0
...

[root@def456 /]# exit

# Peek inside Alpine (notice how tiny it is)
$ docker run --rm -it alpine:3.19 sh
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.0
PRETTY_NAME="Alpine Linux v3.19"

/ # apk --version
apk-tools 2.14.0

/ # exit

# Peek inside Arch
$ docker run --rm -it archlinux:latest bash
[root@ghi789 /]# cat /etc/os-release | head -3
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch

[root@ghi789 /]# pacman --version
 .---.                  Pacman v6.0.2
...

Same kernel underneath. Different userspace. Different tools. Same fundamental concepts.

Debug This

Your colleague says: "I'm setting up a new production server and I want to use Arch Linux because it has the latest packages."

What is wrong with this reasoning? What would you recommend instead, and why?

Understanding Support Lifecycles

When choosing a distribution for production, the support lifecycle matters enormously.

Distribution Support Lifecycles (approximate):
─────────────────────────────────────────────────────────>
                                                    time

Ubuntu LTS:      ├──── 5 years standard ────┤
                 ├──── 10 years with ESM ────────────┤

Debian stable:   ├──── 3 years full ────┤
                 ├──── 5 years with LTS ──────┤

RHEL:            ├──── 10 years full support ────────────────┤
                 ├──── 13 years with ELS ────────────────────────┤

Fedora:          ├── ~13 months ──┤

Arch:            ├── rolling (forever, but no guarantees) ──────>

For a server that needs to run reliably for years, you want a distribution with a long support lifecycle. This ensures you receive security patches without needing to upgrade to a completely new release.

Think About It: Imagine you are managing 500 servers. What happens when your distribution reaches end-of-life? How does this affect your choice of distribution?

What Just Happened?

┌─────────────────────────────────────────────────────┐
│                                                      │
│  In this chapter, you learned:                       │
│                                                      │
│  - A distribution = kernel + package manager +       │
│    system tools + configuration + philosophy.         │
│                                                      │
│  - Major families: Debian (apt), Red Hat (dnf),      │
│    Arch (pacman), SUSE (zypper).                     │
│                                                      │
│  - Point-release distros (Debian, Ubuntu LTS, RHEL)  │
│    freeze versions for stability. Rolling-release     │
│    distros (Arch, Tumbleweed) update continuously.    │
│                                                      │
│  - For servers: Ubuntu LTS, Debian, RHEL, Rocky.     │
│    For desktops: Ubuntu, Fedora, Mint, Arch.          │
│    For containers: Alpine, Debian slim.               │
│    For learning: Ubuntu LTS or Fedora.                │
│                                                      │
│  - Package managers differ in syntax but share        │
│    the same concepts: install, remove, update,        │
│    search, query.                                     │
│                                                      │
│  - Support lifecycles matter for production.          │
│    RHEL: 10+ years. Ubuntu LTS: 5-10 years.          │
│    Arch: rolling with no guarantees.                  │
│                                                      │
└─────────────────────────────────────────────────────┘

Try This

Exercises

1. Identification exercise: Run cat /etc/os-release on every Linux system you have access to (VM, server, WSL, Docker containers). Record the ID, ID_LIKE, and VERSION_ID for each.

2. Comparison exercise: Visit the websites of Ubuntu (ubuntu.com), Fedora (fedoraproject.org), and Arch Linux (archlinux.org). For each, find: the latest release version, the supported architectures, and the default desktop environment.

3. Package manager exercise: Using the comparison table in this chapter, write the equivalent commands for all three package managers to: (a) install the curl package, (b) search for packages related to "postgresql", and (c) remove the curl package.

4. Research exercise: Go to distrowatch.com and look at the top 10 distributions by page hit ranking. How many of them are Debian-based? How many are independently developed?

5. Decision exercise: For each scenario below, choose a distribution and justify your choice:

   • A web server that must run for 5 years with minimal maintenance
   • A developer's personal laptop
   • A Raspberry Pi running as a home media server
   • A Docker base image for a microservice
   • A security professional's testing workstation

Bonus Challenge

Install two different distributions in virtual machines (Chapter 3 shows you how) and perform the same task in both -- for example, install Nginx, start it, and verify it is running. Note every difference in commands and paths. This is the fastest way to internalize how distributions differ.

What's Next

You have chosen your distribution (or at least narrowed it down). Now it is time to actually install it. Chapter 3 walks you through multiple installation methods: virtual machines, WSL2 on Windows, bare metal, and cloud instances.

Installing Linux

Why This Matters

You cannot learn Linux from a book alone. You need a running system where you can break things, fix them, break them again, and gradually build confidence. The first real step on that path is installation.

The good news: installing Linux has never been easier. You do not need to wipe your existing operating system. You do not need dedicated hardware. You have at least five different ways to get a working Linux environment, ranging from "takes five minutes, no risk" to "full bare-metal installation." This chapter covers them all.

By the end of this chapter, you will have a working Linux system ready for every exercise in this book.

Try This Right Now

If you are on Windows 10/11, you can have a working Linux system in under five minutes:

# Open PowerShell as Administrator and run:
wsl --install

That single command installs WSL2 with Ubuntu. After a reboot, you will have a full Linux environment. If you already did this, try:

wsl --status

If you are on macOS or already run Linux, skip ahead to the VM section or the bare-metal section.

Installation Options Overview

Here are your options, from simplest to most involved:

┌─────────────────────────────────────────────────────────┐
│              Installation Options                        │
│                                                          │
│  Option              Risk Level    Time     Best For     │
│  ─────────────────────────────────────────────────────   │
│  WSL2 (Windows)      None          5 min    Learning,    │
│                                              dev work    │
│                                                          │
│  Cloud Instance       None          5 min    Server      │
│  (AWS/GCP/etc.)                              skills      │
│                                                          │
│  Virtual Machine      None          30 min   Full        │
│  (VirtualBox/QEMU)                           experience  │
│                                                          │
│  Live USB             None          10 min   Testing     │
│  (no install)                                hardware    │
│                                                          │
│  Dual Boot            Medium        45 min   Daily use   │
│                                              + Windows   │
│                                                          │
│  Bare Metal           High          30 min   Full        │
│  (replace OS)                                commitment  │
│                                                          │
└─────────────────────────────────────────────────────────┘

For this book, any of these options works. A VM or WSL2 is recommended because you can take snapshots, experiment freely, and destroy and rebuild your environment without consequence.

Option 1: WSL2 on Windows

Windows Subsystem for Linux 2 (WSL2) runs a real Linux kernel inside a lightweight virtual machine managed by Windows. It is not emulation -- it is genuine Linux.

Prerequisites

• Windows 10 version 2004 or later (Build 19041+) or Windows 11
• Hardware virtualization enabled in BIOS (usually enabled by default)
• At least 4 GB RAM (8 GB recommended)

Step-by-Step Installation

Step 1: Enable WSL

Open PowerShell as Administrator:

wsl --install

This command:

• Enables the WSL feature
• Enables the Virtual Machine Platform
• Downloads and installs the Linux kernel
• Sets WSL2 as the default version
• Installs Ubuntu as the default distribution

Step 2: Reboot

# After the command completes:
Restart-Computer

Step 3: First Launch

After reboot, Ubuntu will launch automatically (or find it in the Start menu). You will be prompted to create a username and password:

Installing, this may take a few minutes...
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: yourname
New password:
Retype new password:
passwd: password updated successfully
Installation successful!

Warning: This username and password are for Linux only. They are separate from your Windows login. Choose a short username with no spaces. You will type this password often -- make it something you can remember.

Step 4: Verify

$ uname -a
Linux DESKTOP-ABC123 5.15.153.1-microsoft-standard-WSL2 #1 SMP x86_64 GNU/Linux

$ cat /etc/os-release | head -3
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"

$ echo "Hello from Linux!"
Hello from Linux!

You now have a working Linux system.

Installing Additional Distributions in WSL2

# List available distributions
wsl --list --online

# Install a specific one
wsl --install -d Debian
wsl --install -d Fedora

# List your installed distributions
wsl --list --verbose

WSL2 Tips

• Access Windows files from Linux: They are at /mnt/c/, /mnt/d/, etc.
• Access Linux files from Windows: In Explorer, type \\wsl$ in the address bar
• Run Linux commands from PowerShell: wsl ls -la /home
• Run Windows commands from Linux: /mnt/c/Windows/System32/notepad.exe
• Keep your Linux files in the Linux filesystem (under /home/yourname), not on /mnt/c/. The Linux filesystem is much faster for Linux operations.

WSL2 Limitations

WSL2 is excellent for learning and development, but has some limitations:

• No systemd by default (though it can be enabled in recent versions)
• No direct hardware access (USB passthrough requires extra tools)
• Networking is bridged through Windows
• Not suitable for testing boot processes, kernel modules, or hardware interaction

For these topics, use a virtual machine instead.

Distro Note: WSL2 defaults to Ubuntu, but you can install Debian, Fedora, openSUSE, Kali, Alpine, and others. Check wsl --list --online for the current list.

Option 2: Virtual Machine with VirtualBox

A virtual machine (VM) gives you a complete, isolated Linux system with full control. You can take snapshots, experiment with boot configurations, and even simulate hardware failures.

Prerequisites

• Any operating system (Windows, macOS, or Linux) as the host
• At least 8 GB RAM (to give 2-4 GB to the VM)
• At least 25 GB free disk space
• VirtualBox downloaded from virtualbox.org

Note: VirtualBox is open source (GPLv2). Other options include GNOME Boxes (Linux), QEMU/KVM (Linux, covered below), and VMware Workstation Player (free for personal use but not open source).

Step 1: Download the Linux ISO

Go to your chosen distribution's website and download the installation ISO:

• Ubuntu: ubuntu.com/download/server (Ubuntu Server LTS, ~2 GB)
• Fedora: fedoraproject.org (Fedora Server or Workstation, ~2 GB)
• Debian: debian.org/distrib (netinst ISO, ~600 MB)

For this walkthrough, we use Ubuntu Server 22.04 LTS.

Step 2: Create the Virtual Machine

1. Open VirtualBox and click New

2. Configure:

   • Name: ubuntu-lab
   • Type: Linux
   • Version: Ubuntu (64-bit)
   • Memory: 2048 MB (2 GB) minimum, 4096 MB recommended
   • Hard disk: Create a virtual hard disk now
   • Disk size: 25 GB (dynamically allocated)
   • Disk type: VDI (VirtualBox Disk Image)

3. Before starting, go to Settings:

   • System > Processor: 2 CPUs
   • Network > Adapter 1: NAT (for internet) or Bridged Adapter (for network visibility)
   • Storage: Under "Controller: IDE," click the empty disk icon, then select "Choose a disk file" and point to your downloaded ISO

Step 3: Install Ubuntu Server

Start the VM. You will boot from the ISO into the Ubuntu installer.

1. Language: English
2. Keyboard layout: Your keyboard layout
3. Installation type: Ubuntu Server
4. Network: The installer auto-detects. Accept defaults
5. Storage/Partitioning: Use the entire disk (guided). The installer will suggest:

  /dev/sda  25 GB
  ├── /dev/sda1   1 GB   /boot/efi  (EFI System Partition)
  ├── /dev/sda2   1.7 GB /boot      (boot partition)
  └── /dev/sda3   22 GB  / (root)   (main partition, ext4)

Accept the defaults for now. We will cover partitioning in depth in Chapter 7.

6. Profile setup:

   • Your name: Your Name
   • Server name: ubuntu-lab
   • Username: yourname
   • Password: choose something memorable

7. SSH Setup: Check "Install OpenSSH server" -- you will need this

8. Featured snaps: Skip all of these for now

9. Wait for installation to complete, then select Reboot Now

Step 4: First Boot

After reboot, you will see a login prompt:

ubuntu-lab login: yourname
Password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-91-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Mon Jan 15 10:30:00 UTC 2024

  System load:  0.08              Processes:             112
  Usage of /:   18.2% of 21.5GB  Users logged in:       0
  Memory usage: 12%               IPv4 address for eth0: 10.0.2.15
  Swap usage:   0%

yourname@ubuntu-lab:~$

You are in.

Step 5: Take a Snapshot

Before doing anything else, take a snapshot in VirtualBox. This gives you a known-good state you can revert to at any time:

1. In VirtualBox Manager, select your VM
2. Click Snapshots (top right)
3. Click Take (camera icon)
4. Name it "fresh-install"

Warning: Always take a snapshot before experimenting with anything potentially destructive. This is your safety net. You can restore to any snapshot in seconds.

Option 3: QEMU/KVM (Linux Host)

If your host system already runs Linux, QEMU/KVM is the preferred virtualization solution. It is fully open source and offers near-native performance.

Install QEMU/KVM

# Debian/Ubuntu
$ sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients \
    bridge-utils virt-manager

# Fedora
$ sudo dnf install @virtualization

# Arch
$ sudo pacman -S qemu-full libvirt virt-manager dnsmasq

Verify KVM Support

$ lscpu | grep Virtualization
Virtualization:                  VT-x

$ ls /dev/kvm
/dev/kvm

If /dev/kvm exists, you have hardware virtualization support.

Create a VM with virt-manager (GUI)

1. Launch virt-manager
2. Click Create a new virtual machine
3. Select Local install media (ISO)
4. Browse to your downloaded ISO
5. Allocate 2 GB RAM, 2 CPUs
6. Create a 25 GB disk
7. Name your VM and click Finish

Create a VM from the Command Line

# Create a disk image
$ qemu-img create -f qcow2 ubuntu-lab.qcow2 25G

# Boot from ISO to install
$ qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -smp 2 \
    -hda ubuntu-lab.qcow2 \
    -cdrom ubuntu-22.04-live-server-amd64.iso \
    -boot d \
    -net nic -net user,hostfwd=tcp::2222-:22

# After installation, boot without ISO
$ qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -smp 2 \
    -hda ubuntu-lab.qcow2 \
    -net nic -net user,hostfwd=tcp::2222-:22

The hostfwd option forwards port 2222 on your host to port 22 on the VM, so you can SSH in:

$ ssh -p 2222 yourname@localhost

Option 4: Cloud Instance

If you have an account with any cloud provider, you can launch a Linux instance in minutes.

AWS (Free Tier)

# Using the AWS CLI
$ aws ec2 run-instances \
    --image-id ami-0c7217cdde317cfec \
    --instance-type t2.micro \
    --key-name your-keypair \
    --security-group-ids sg-xxxxx

# SSH in
$ ssh -i your-key.pem ubuntu@<public-ip>

DigitalOcean, Linode, Vultr

These providers offer $5-10/month VMs with one-click Linux provisioning. Many offer free credits for new accounts. The setup is typically:

1. Create an account
2. Create a "Droplet" / "Linode" / "Instance"
3. Choose Ubuntu 22.04 LTS
4. Choose the smallest size
5. Add your SSH key
6. SSH in

This approach is excellent for practicing remote server administration, which is how you will manage Linux servers in the real world.

Option 5: Live USB (Try Without Installing)

A Live USB lets you boot a full Linux system from a USB drive without touching your hard disk. This is great for testing hardware compatibility before committing to an installation.

Create a Live USB

What you need:

• A USB drive (4 GB minimum, 8 GB recommended)
• A Linux ISO (Ubuntu Desktop is best for this)
• A tool to write the ISO to USB

Warning: Writing an ISO to a USB drive will ERASE ALL DATA on that drive. Back up any files on the USB before proceeding.

On Linux:

# Identify your USB drive (CAREFULLY -- do not pick the wrong device!)
$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 256.0G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0 255.5G  0 part /
sdb      8:16   1   7.5G  0 disk          <-- This is the USB drive

# Write the ISO (replace /dev/sdb with YOUR USB device)
$ sudo dd if=ubuntu-22.04-desktop-amd64.iso of=/dev/sdb bs=4M status=progress
$ sync

Warning: The dd command is sometimes called "disk destroyer" for good reason. If you specify the wrong of= target, you will overwrite your hard disk. Triple-check the device name with lsblk before running dd.

On Windows:

Use Rufus (rufus.ie) -- a free, open-source tool:

1. Download and run Rufus
2. Select your USB drive
3. Select the ISO file
4. Click Start

On macOS:

Use balenaEtcher (balena.io/etcher) -- also free and open source.

Boot from USB

1. Insert the USB drive
2. Reboot your computer
3. Enter the boot menu (usually F12, F2, Esc, or Del during startup -- varies by manufacturer)
4. Select the USB drive
5. Choose "Try Ubuntu" (or equivalent) to boot without installing

Option 6: Bare Metal Installation

Installing Linux as the sole operating system on a computer gives you the best performance and full hardware access. This is ideal if you have a spare laptop or desktop.

Warning: Installing Linux on bare metal will ERASE your existing operating system and all data on the target disk (unless you set up dual boot). Back up everything first.

The process is the same as a VM install, but booting from a USB drive on real hardware:

1. Create a Live USB (see above)
2. Boot from USB
3. Choose "Install Ubuntu" instead of "Try Ubuntu"
4. Follow the installer (same steps as the VM walkthrough above)
5. When asked about disk partitioning, "Erase disk and install" is the simplest option

Understanding Partitioning Basics

During installation, you encounter disk partitioning. Here is what you need to know now (Chapter 7 goes deep).

What Is a Partition?

A partition divides a physical disk into logical sections. Each partition can have its own filesystem and purpose.

Common Partition Layouts

Simple (recommended for learning):

┌──────────────────────────────────────────────┐
│  /dev/sda                                     │
│                                               │
│  ┌────────┬──────────────────────────────┐    │
│  │ sda1   │ sda2                         │    │
│  │ /boot  │ / (root)                     │    │
│  │ ~1 GB  │ remaining space              │    │
│  │ (EFI)  │ (ext4 filesystem)            │    │
│  └────────┴──────────────────────────────┘    │
└──────────────────────────────────────────────┘

Production server (typical):

┌──────────────────────────────────────────────┐
│  /dev/sda                                     │
│                                               │
│  ┌──────┬──────┬────────┬────────┬────────┐  │
│  │ sda1 │ sda2 │ sda3   │ sda4   │ sda5   │  │
│  │/boot │ /    │ /home  │ /var   │ swap   │  │
│  │512MB │ 20GB │ 50GB   │ 20GB   │ 4GB    │  │
│  │(EFI) │      │        │(logs)  │        │  │
│  └──────┴──────┴────────┴────────┴────────┘  │
└──────────────────────────────────────────────┘

Why separate partitions?

• /boot or /boot/efi -- keeps boot files on a simple filesystem the bootloader can read
• / (root) -- the core operating system
• /home -- user files (separate so you can reinstall the OS without losing your data)
• /var -- logs and variable data (separate so a full disk from runaway logs does not crash the OS)
• swap -- virtual memory space (used when RAM is full)

For a learning VM, the simple layout is perfect. Accept the installer's defaults.

Filesystem Types

The installer will ask about filesystem types. The short answer: use ext4. It is the default, well-tested, and reliable.

Think About It: Why might you put /var/log on a separate partition from /? What happens if log files fill up all available disk space on the root partition?

Post-Install First Steps

You have a running Linux system. Now what? Run through these steps to make sure everything is working and up to date.

Step 1: Update Your System

The very first thing to do on any new Linux install is update all packages:

# Debian/Ubuntu
$ sudo apt update && sudo apt upgrade -y

# Fedora/RHEL/Rocky
$ sudo dnf upgrade -y

# Arch
$ sudo pacman -Syu

Distro Note: apt update refreshes the package list (what is available). apt upgrade installs newer versions. These are two separate steps on Debian/Ubuntu. On Fedora, dnf upgrade does both. On Arch, pacman -Syu does both.

Step 2: Check Networking

# Do you have an IP address?
$ ip addr show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP>
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0

# Can you reach the internet?
$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=11.8 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=12.1 ms

# Can DNS resolution work?
$ ping -c 3 google.com
PING google.com (142.250.80.46) 56(84) bytes of data.
64 bytes from lax17s64-in-f14.1e100.net: icmp_seq=1 ttl=117 time=12.5 ms

If ping 8.8.8.8 works but ping google.com does not, you have a DNS problem. Check /etc/resolv.conf.

Step 3: Install Essential Tools

# Debian/Ubuntu
$ sudo apt install -y curl wget vim git htop tree net-tools

# Fedora/RHEL/Rocky
$ sudo dnf install -y curl wget vim git htop tree net-tools

# Arch
$ sudo pacman -S curl wget vim git htop tree net-tools

These are tools you will use throughout this book.

Step 4: Check Disk Space

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        22G  4.2G   17G  21% /
/dev/sda1       512M  5.3M  507M   2% /boot/efi
tmpfs           994M     0  994M   0% /dev/shm

Make sure you have plenty of space available on /.

Step 5: Verify Your User Can Use sudo

$ sudo whoami
root

If this returns root, your user has sudo privileges. If it returns an error, you need to add your user to the sudo group:

# This requires logging in as root first
# Debian/Ubuntu
$ su -
# usermod -aG sudo yourname

# Fedora/RHEL/Rocky
$ su -
# usermod -aG wheel yourname

Distro Note: The sudo group is called sudo on Debian/Ubuntu and wheel on Fedora/RHEL. Same concept, different name.

Step 6: Set Your Timezone

# Check current timezone
$ timedatectl
               Local time: Mon 2024-01-15 10:45:00 UTC
           Universal time: Mon 2024-01-15 10:45:00 UTC
                 RTC time: Mon 2024-01-15 10:45:00
                Time zone: Etc/UTC (UTC, +0000)

# List available timezones
$ timedatectl list-timezones | grep America
America/Chicago
America/Denver
America/Los_Angeles
America/New_York
...

# Set your timezone
$ sudo timedatectl set-timezone America/New_York

# Verify
$ date
Mon Jan 15 05:45:00 EST 2024

Verifying Your Installation: A Checklist

Run through this checklist to confirm your system is ready for the rest of this book:

# 1. Kernel is running
$ uname -r
6.1.0-18-amd64

# 2. You know your distribution
$ cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Ubuntu 22.04.3 LTS"

# 3. Networking works
$ curl -s ifconfig.me
203.0.113.42

# 4. Package manager works
$ apt list --installed 2>/dev/null | wc -l    # Debian/Ubuntu
587

# 5. sudo works
$ sudo echo "I have sudo access"
I have sudo access

# 6. Disk space is sufficient
$ df -h / | awk 'NR==2 {print "Available:", $4}'
Available: 17G

# 7. Memory is sufficient
$ free -h | awk '/Mem:/ {print "Available:", $7}'
Available: 1.5Gi

# 8. Essential tools are installed
$ which bash curl vim git ssh
/usr/bin/bash
/usr/bin/curl
/usr/bin/vim
/usr/bin/git
/usr/bin/ssh

If all of these pass, you are ready.

Debug This

You installed Ubuntu in a VM. You can log in at the console, but when you try to SSH from your host machine, it does not connect:

$ ssh yourname@10.0.2.15
ssh: connect to host 10.0.2.15 port 22: Connection refused

What could be wrong? Work through these possibilities:

# On the VM console
$ systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
     Active: active (running)

$ sudo ufw status
Status: active
...

$ sudo ufw allow ssh

What Just Happened?

┌─────────────────────────────────────────────────────┐
│                                                      │
│  In this chapter, you:                               │
│                                                      │
│  - Learned 6 ways to run Linux: WSL2, VirtualBox,    │
│    QEMU/KVM, cloud instance, live USB, bare metal.   │
│                                                      │
│  - Set up WSL2 on Windows with a single command      │
│    (wsl --install).                                   │
│                                                      │
│  - Created a VirtualBox VM with Ubuntu Server,       │
│    walked through the installer step by step.         │
│                                                      │
│  - Learned partitioning basics: /, /boot, /home,     │
│    /var, swap -- and why separate partitions matter.  │
│                                                      │
│  - Completed post-install setup: updated packages,   │
│    verified networking, installed essential tools,    │
│    set timezone, confirmed sudo access.              │
│                                                      │
│  - Ran a verification checklist to confirm your      │
│    system is ready for the rest of this book.         │
│                                                      │
│  Commands you met: wsl, apt update, apt upgrade,     │
│  ip addr, ping, curl, df, free, timedatectl,         │
│  systemctl, ufw, dd, lsblk, sudo                    │
│                                                      │
└─────────────────────────────────────────────────────┘

Try This

Exercises

1. Installation exercise: Install Linux using at least one method from this chapter. If you are on Windows, start with WSL2. If you want the full experience, create a VirtualBox VM.

2. Snapshot exercise (VM users): Take a snapshot of your fresh install. Then intentionally break something (delete /etc/hostname, for example). Restore the snapshot and verify everything is back to normal.

3. Post-install exercise: Run through every step in the "Post-Install First Steps" section. Record the output of each command. Compare with a classmate or colleague -- are your outputs different? Why?

4. Partitioning exercise: Run lsblk and df -h on your system. Draw a diagram of your disk layout similar to the ASCII diagrams in this chapter. What partitions exist? What filesystem does each use?

5. Multi-distro exercise: If you have VirtualBox or Docker, install a second distribution (e.g., if you installed Ubuntu, try Fedora). Run through the same post-install steps. Note every command that differs.

Bonus Challenge

Install Arch Linux in a virtual machine following the official Arch Installation Guide on the Arch Wiki. This is a rite of passage for Linux enthusiasts. The Arch installer gives you nothing -- no GUI, no automation. You will manually partition disks, install the base system, configure the bootloader, and set up networking. It is difficult, educational, and deeply satisfying when it works.

What's Next

You have a running Linux system. You are staring at a blinking cursor. What now? Chapter 4 introduces the shell -- the interface between you and the full power of Linux. This is where the real journey begins.

The Shell: Your Superpower

Why This Matters

Imagine this: a web server is down, hundreds of users are affected, and you need to find the problem and fix it -- fast. You SSH into the server. There is no graphical interface. No mouse. No menus. Just a blinking cursor. The only tool you have is the shell.

In this moment, the shell is either your greatest asset or your greatest obstacle. If you know your way around it, you can diagnose the problem in seconds: check running processes, read log files, restart services, inspect network connections -- all without leaving the keyboard. If you do not, you are helpless.

The shell is the single most important skill in this book. Every chapter after this one assumes you are comfortable typing commands, reading output, and navigating the filesystem. Master this chapter, and everything else falls into place.

Try This Right Now

Open your Linux terminal (whether it is WSL2, a VM, SSH to a server, or a native Linux desktop) and type:

$ echo "I am $(whoami) on $(hostname), running $(uname -s) kernel $(uname -r)"

You should see something like:

I am yourname on ubuntu-lab, running Linux kernel 6.1.0-18-amd64

You just used four commands (echo, whoami, hostname, uname) combined into a single line using command substitution ($(...) runs a command and inserts its output). This is the kind of thing the shell makes effortless.

Terminal vs Shell vs Console: Clearing the Confusion

These three terms are used loosely, but they mean different things.

┌─────────────────────────────────────────────────────┐
│                                                      │
│  TERMINAL EMULATOR                                   │
│  ┌─────────────────────────────────────────────────┐ │
│  │                                                  │ │
│  │  The window that displays text. It handles       │ │
│  │  fonts, colors, scrolling, and keyboard input.   │ │
│  │                                                  │ │
│  │  Examples: GNOME Terminal, Konsole, Alacritty,   │ │
│  │  iTerm2, Windows Terminal, xterm                 │ │
│  │                                                  │ │
│  │  ┌──────────────────────────────────────────┐    │ │
│  │  │                                           │    │ │
│  │  │  SHELL                                    │    │ │
│  │  │                                           │    │ │
│  │  │  The program that interprets your          │    │ │
│  │  │  commands. It reads what you type,         │    │ │
│  │  │  executes programs, and shows output.      │    │ │
│  │  │                                           │    │ │
│  │  │  Examples: bash, zsh, fish, sh, dash       │    │ │
│  │  │                                           │    │ │
│  │  └──────────────────────────────────────────┘    │ │
│  │                                                  │ │
│  └─────────────────────────────────────────────────┘ │
│                                                      │
│  CONSOLE                                             │
│  The physical or virtual text interface               │
│  (e.g., the screen when no GUI is running,           │
│  or Ctrl+Alt+F1 through F6 on Linux)                 │
│                                                      │
└─────────────────────────────────────────────────────┘

• Terminal emulator: the application that provides the window. It is just a container.
• Shell: the program running inside the terminal that actually interprets your commands.
• Console: historically the physical terminal attached to a machine. Today, it usually refers to the virtual terminals you can switch to with Ctrl+Alt+F1 through F6 on a Linux system.

When you open "Terminal" on your Linux desktop, you are launching a terminal emulator that starts a shell (usually Bash) inside it.

Which Shell Are You Using?

# What shell is running right now?
$ echo $SHELL
/bin/bash

# What version?
$ bash --version
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)

# What shells are available on this system?
$ cat /etc/shells
/bin/sh
/bin/bash
/usr/bin/bash
/bin/zsh
/usr/bin/zsh
/bin/dash

The Major Shells

Bash (Bourne Again Shell) -- the default on most Linux distributions. This is what we use throughout this book. It is the lingua franca of Linux.

Zsh (Z Shell) -- the default on macOS since Catalina. Compatible with Bash but adds features like better tab completion, spelling correction, and plugin frameworks (Oh My Zsh). Popular on developer workstations.

Fish (Friendly Interactive Shell) -- designed for usability. Features autosuggestions, syntax highlighting out of the box, and a web-based configuration interface. Not POSIX-compatible, so scripts written for Bash may not work in Fish.

Dash -- a minimal POSIX-compliant shell. Used as /bin/sh on Debian/Ubuntu for speed. Not meant for interactive use.

sh (Bourne Shell) -- the original Unix shell. On modern Linux, /bin/sh is usually a symlink to dash or bash.

For this book, use Bash. It is everywhere, and understanding Bash gives you the foundation to learn any other shell quickly.

Think About It: If Bash is the default shell on most Linux distributions, why do alternative shells like Zsh and Fish exist? What might they offer that Bash does not?

Anatomy of a Command

Every shell command follows the same basic structure:

command   [options]   [arguments]

  │          │            │
  │          │            └── What to act on (files, strings, etc.)
  │          │
  │          └── Modify the command's behavior (flags/switches)
  │
  └── The program to run

Examples:

ls                          # command only (no options, no arguments)
ls -l                       # command + option
ls -l /home                 # command + option + argument
ls -la /home /tmp           # command + multiple options + multiple arguments
cp -r /source /destination  # command + option + two arguments

Options: Short and Long Form

Most commands support both short (single dash, single letter) and long (double dash, word) options:

ls -a                 # short form: show all files (including hidden)
ls --all              # long form: same thing

ls -l                 # short form: long listing format
ls --format=long      # long form: same thing

ls -la                # short forms can be combined: -l + -a
ls -l -a              # equivalent: separate short options
ls --all --format=long  # long forms cannot be combined

The -- Convention

A double dash by itself (--) means "end of options, everything after this is an argument":

# What if you have a file named "-l" and want to delete it?
$ rm -l         # ERROR: rm interprets -l as an option
$ rm -- -l      # CORRECT: -- tells rm that -l is a filename

Your First Commands

Let us work through the essential commands, one at a time. Type each one.

pwd -- Where Am I?

$ pwd
/home/yourname

pwd stands for Print Working Directory. It tells you your current location in the filesystem. You always have a current location, and it matters because relative paths are relative to this location.

ls -- What Is Here?

# List contents of current directory
$ ls
Desktop  Documents  Downloads  Music  Pictures

# Long format: permissions, owner, size, date
$ ls -l
total 20
drwxr-xr-x 2 yourname yourname 4096 Jan 15 10:30 Desktop
drwxr-xr-x 2 yourname yourname 4096 Jan 15 10:30 Documents
drwxr-xr-x 2 yourname yourname 4096 Jan 15 10:30 Downloads
drwxr-xr-x 2 yourname yourname 4096 Jan 15 10:30 Music
drwxr-xr-x 2 yourname yourname 4096 Jan 15 10:30 Pictures

# Show hidden files (files starting with .)
$ ls -a
.  ..  .bash_history  .bashrc  .profile  Desktop  Documents  ...

# Combine: long format + hidden + human-readable sizes
$ ls -lah
total 44K
drwxr-xr-x 7 yourname yourname 4.0K Jan 15 10:30 .
drwxr-xr-x 3 root     root     4.0K Jan 14 09:00 ..
-rw------- 1 yourname yourname  256 Jan 15 10:45 .bash_history
-rw-r--r-- 1 yourname yourname  220 Jan 14 09:00 .bash_logout
-rw-r--r-- 1 yourname yourname 3.5K Jan 14 09:00 .bashrc
drwxr-xr-x 2 yourname yourname 4.0K Jan 15 10:30 Desktop
...

# List a specific directory
$ ls -l /etc/

Key flags to remember:

• -l -- long format (permissions, owner, size, date)
• -a -- all files, including hidden (dotfiles)
• -h -- human-readable sizes (K, M, G instead of bytes)
• -t -- sort by modification time (newest first)
• -r -- reverse sort order
• -R -- recursive (list subdirectories too)

cd -- Move Around

# Go to a directory
$ cd /var/log
$ pwd
/var/log

# Go to your home directory (three equivalent ways)
$ cd
$ cd ~
$ cd $HOME

# Go to the previous directory
$ cd -
/var/log

# Go up one level
$ cd ..
$ pwd
/var

# Go up two levels
$ cd ../..
$ pwd
/

Important concepts:

• . means "current directory"
• .. means "parent directory"
• ~ means "home directory" (/home/yourname)
• - means "previous directory" (where you just were)

cat -- Read File Contents

# Display a file's contents
$ cat /etc/hostname
ubuntu-lab

# Display with line numbers
$ cat -n /etc/passwd | head -5
     1  root:x:0:0:root:/root:/bin/bash
     2  daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
     3  bin:x:2:2:bin:/bin:/usr/sbin/nologin
     4  sys:x:3:3:sys:/dev:/usr/sbin/nologin
     5  sync:x:4:65534:sync:/bin:/bin/sync

cat stands for "concatenate." Its original purpose was to concatenate multiple files, but it is most commonly used to display a single file.

Think About It: cat dumps the entire file to the screen. What happens if the file is thousands of lines long? What might be a better tool for reading long files?

echo -- Print Text

# Print a simple message
$ echo "Hello, Linux!"
Hello, Linux!

# Print the value of a variable
$ echo $HOME
/home/yourname

$ echo "My shell is $SHELL"
My shell is /bin/bash

# Print without a trailing newline
$ echo -n "no newline here"
no newline here$

# Print with escape characters
$ echo -e "Line one\nLine two\nLine three"
Line one
Line two
Line three

echo is the simplest way to produce output. You will use it constantly in scripts and one-liners.

man -- The Manual Pages

$ man ls

This opens the manual page for ls. The man page tells you:

• What the command does
• Every option and flag
• Examples (sometimes)
• Related commands

Navigating man pages:

• Space or f -- next page
• b -- previous page
• /pattern -- search for "pattern"
• n -- next search result
• N -- previous search result
• q -- quit

Man page sections:

# Man pages are organized into sections
$ man 1 ls      # Section 1: User commands
$ man 5 passwd  # Section 5: File formats (/etc/passwd format)
$ man 8 mount   # Section 8: System administration commands

Getting Help: Three Methods

You will need help constantly. That is normal. Here are three ways to get it.

Method 1: man pages (comprehensive)

$ man grep       # full manual
$ man -k "copy files"  # search man pages by keyword

Method 2: --help flag (quick reference)

$ ls --help
Usage: ls [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
...

Almost every command supports --help. The output is shorter than a man page and usually sufficient for quick questions.

Method 3: info pages (detailed, GNU-style)

$ info coreutils   # detailed GNU documentation

info pages are more detailed than man pages for GNU tools but use a different navigation scheme. Most people prefer man pages.

Which to Use?

• Quick question ("What flag makes ls sort by size?"): ls --help | grep size
• Learning a new command: man command
• Deep reference: info command or the online documentation

Tab Completion: Your Fingers Will Thank You

Tab completion is one of the most important productivity features of the shell. Press Tab to auto-complete commands, filenames, and paths.

Hands-On: Try Tab Completion

# Type "cd /e" then press Tab
$ cd /e<TAB>
$ cd /etc/              # auto-completed!

# Type "cd /etc/sys" then press Tab
$ cd /etc/sys<TAB>
$ cd /etc/sysctl.d/     # if only one match, it completes

# Type "cd /etc/s" then press Tab twice
$ cd /etc/s<TAB><TAB>
security/  shadow     shells     skel/      ssh/       ssl/       subgid     subuid     sudoers    sudoers.d/ sysctl.d/  systemd/
# Multiple matches! Shows all possibilities

# Type "cat /etc/hos" then press Tab
$ cat /etc/hos<TAB>
$ cat /etc/hostname     # completed!

# Tab also works for commands
$ sys<TAB><TAB>
systemctl    systemd-analyze    systemd-cat    ...

Rules of tab completion:

• One match: auto-completes immediately
• Multiple matches: press Tab twice to see all options
• No matches: nothing happens (no beep, no output)

In Bash, install bash-completion for even better tab completion (it can complete package names, git branches, systemctl units, and more):

# Debian/Ubuntu
$ sudo apt install bash-completion

# Fedora/RHEL
$ sudo dnf install bash-completion

# Arch
$ sudo pacman -S bash-completion

Distro Note: Most distributions install bash-completion by default. If tab completion for commands like systemctl and git already works for subcommands and arguments, you already have it.

Command History: The Shell Remembers

The shell keeps a history of every command you type. This is enormously useful.

Basic History Usage

# Show recent commands
$ history
  1  ls -la
  2  cd /etc
  3  cat hostname
  4  ping google.com
  5  sudo apt update
  ...

# Show last 10 commands
$ history 10

# Re-run command number 42
$ !42

# Re-run the last command
$ !!

# Re-run the last command that started with "sudo"
$ !sudo

# Search history interactively: press Ctrl+R, then type
$ (press Ctrl+R)
(reverse-i-search)`ping': ping -c 3 google.com
# Press Enter to run it, or Ctrl+R again for the next match

History Configuration

# Where is history stored?
$ echo $HISTFILE
/home/yourname/.bash_history

# How many commands are kept?
$ echo $HISTSIZE
1000

# You can increase these in ~/.bashrc:
# HISTSIZE=10000
# HISTFILESIZE=20000

Essential Keyboard Shortcuts

These work in Bash and most other shells:

┌───────────────────────────────────────────────────┐
│  Navigation                                        │
│  Ctrl+A      Move cursor to beginning of line      │
│  Ctrl+E      Move cursor to end of line             │
│  Alt+B       Move back one word                     │
│  Alt+F       Move forward one word                  │
│  Ctrl+←      Move back one word (some terminals)    │
│  Ctrl+→      Move forward one word                  │
│                                                     │
│  Editing                                            │
│  Ctrl+U      Delete from cursor to beginning        │
│  Ctrl+K      Delete from cursor to end              │
│  Ctrl+W      Delete the word before cursor          │
│  Alt+D       Delete the word after cursor            │
│  Ctrl+Y      Paste what was last deleted (yank)     │
│                                                     │
│  History                                            │
│  Ctrl+R      Reverse search history                 │
│  Ctrl+P      Previous command (same as Up arrow)    │
│  Ctrl+N      Next command (same as Down arrow)      │
│  ↑ / ↓       Navigate through history               │
│                                                     │
│  Control                                            │
│  Ctrl+C      Cancel current command                 │
│  Ctrl+D      Exit shell (or signal end of input)    │
│  Ctrl+L      Clear the screen (same as `clear`)     │
│  Ctrl+Z      Suspend current process (background)   │
│                                                     │
└───────────────────────────────────────────────────┘

Practice these until they are muscle memory. The difference between a beginner and an experienced Linux user often comes down to how fluently they navigate the command line.

Environment Variables

Environment variables are named values that the shell and programs use for configuration. They are a fundamental part of how Linux works.

Viewing Environment Variables

# See all environment variables
$ env
HOME=/home/yourname
USER=yourname
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
TERM=xterm-256color
...

# See a specific variable
$ echo $HOME
/home/yourname

$ echo $USER
yourname

$ echo $SHELL
/bin/bash

Setting Variables

# Set a variable (no spaces around =)
$ GREETING="Hello, Linux"
$ echo $GREETING
Hello, Linux

# This variable exists only in the current shell
# To make it available to child processes, export it:
$ export GREETING="Hello, Linux"

# Set and export in one step
$ export MY_APP_PORT=8080
$ echo $MY_APP_PORT
8080

Warning: There are no spaces around the = sign. NAME=value works. NAME = value fails with a confusing error (the shell tries to run NAME as a command with = and value as arguments).

Important Environment Variables

PATH: How the Shell Finds Commands

When you type ls, how does the shell know where the ls program is? The answer is the PATH variable.

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

PATH is a colon-separated list of directories. When you type a command, the shell searches these directories in order, left to right, looking for an executable file with that name.

You type: ls

Shell searches:
  /usr/local/sbin/ls  -- not found
  /usr/local/bin/ls   -- not found
  /usr/sbin/ls        -- not found
  /usr/bin/ls         -- FOUND! Runs /usr/bin/ls

Finding Where a Command Lives

# which: shows the path to the command that would run
$ which ls
/usr/bin/ls

$ which python3
/usr/bin/python3

# type: shows more detail (aliases, builtins, etc.)
$ type ls
ls is aliased to `ls --color=auto'

$ type cd
cd is a shell builtin

$ type python3
python3 is /usr/bin/python3

# whereis: finds the binary, source, and man page
$ whereis ls
ls: /usr/bin/ls /usr/share/man/man1/ls.1.gz

Modifying PATH

# Add a directory to PATH (temporary, current session only)
$ export PATH="$HOME/bin:$PATH"
$ echo $PATH
/home/yourname/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# To make it permanent, add the export line to ~/.bashrc:
$ echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc

Think About It: What happens if you have two different programs both named python3, one in /usr/bin/ and one in /usr/local/bin/? Which one runs when you type python3? How would you change which one runs?

Hands-On: Putting It All Together

Let us chain together everything you have learned in a real scenario.

Scenario: Investigate an Unknown Linux System

You have just SSH'd into a server you have never seen before. Gather information:

# Who am I?
$ whoami
yourname

# What machine is this?
$ hostname
prod-web-03

# What distribution?
$ cat /etc/os-release | head -2
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"

# What kernel?
$ uname -r
5.15.0-91-generic

# How long has it been running?
$ uptime
 14:32:07 up 127 days,  3:21,  1 user,  load average: 0.45, 0.38, 0.35

# How much memory?
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.1Gi       312Mi       128Mi       1.4Gi       1.3Gi
Swap:          2.0Gi       128Mi       1.9Gi

# How much disk space?
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   32G   15G  69% /

# What is running? (top 5 CPU consumers)
$ ps aux --sort=-%cpu | head -6
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data  1234 12.3  5.2 512000 201000 ?       Sl   Jan01 245:32 nginx: worker
mysql     5678  8.7 15.3 1024000 590000 ?      Sl   Jan01 170:45 /usr/sbin/mysqld
root      9012  1.2  0.5  65000  19000 ?       Ss   Jan01  23:10 /usr/sbin/sshd
...

# What ports are open?
$ ss -tlnp
State  Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
LISTEN 0      511    0.0.0.0:80           0.0.0.0:*         users:(("nginx",pid=1233))
LISTEN 0      511    0.0.0.0:443          0.0.0.0:*         users:(("nginx",pid=1233))
LISTEN 0      151    127.0.0.1:3306       0.0.0.0:*         users:(("mysqld",pid=5678))
LISTEN 0      128    0.0.0.0:22           0.0.0.0:*         users:(("sshd",pid=9012))

In under a minute, using only the commands from this chapter, you have determined that this is an Ubuntu web server running Nginx and MySQL, with 69% disk usage (might need attention), and it has been up for 127 days. This is the power of the shell.

Debug This

You are trying to run a program you just downloaded, but the shell cannot find it:

$ my-tool --version
bash: my-tool: command not found

$ ls ~/downloads/my-tool
/home/yourname/downloads/my-tool

The file exists. Why can the shell not find it? Fix it.

# Option A: Run with full path
$ ~/downloads/my-tool --version

# Option B: Run with relative path
$ ./downloads/my-tool --version   # if you are in ~

# Option C: Add the directory to PATH
$ export PATH="$HOME/downloads:$PATH"
$ my-tool --version

$ ls -l ~/downloads/my-tool
-rw-r--r-- 1 yourname yourname 102400 Jan 15 10:00 my-tool

$ chmod +x ~/downloads/my-tool
$ ~/downloads/my-tool --version

The Shell Startup Files

When Bash starts, it reads configuration files that set up your environment. Understanding which files are read and when is important.

Login Shell vs Non-Login Shell

┌─────────────────────────────────────┐
│         Login Shell                  │
│  (SSH, console login, su -)          │
│                                      │
│  Reads:                              │
│  1. /etc/profile                     │
│  2. ~/.bash_profile                  │
│     (or ~/.bash_login                │
│      or ~/.profile)                  │
│                                      │
├─────────────────────────────────────┤
│     Non-Login (Interactive) Shell    │
│  (opening a new terminal window)     │
│                                      │
│  Reads:                              │
│  1. /etc/bash.bashrc  (Debian)       │
│     /etc/bashrc       (RHEL)         │
│  2. ~/.bashrc                        │
│                                      │
└─────────────────────────────────────┘

In practice, most people put their customizations in ~/.bashrc because ~/.bash_profile typically sources ~/.bashrc anyway. Check yours:

$ cat ~/.bash_profile
# If this file exists, it often contains:
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

What Goes in ~/.bashrc?

# Aliases (shortcuts)
alias ll='ls -lah'
alias la='ls -A'
alias ..='cd ..'
alias ...='cd ../..'

# Custom PATH
export PATH="$HOME/bin:$PATH"

# Default editor
export EDITOR=vim

# History settings
HISTSIZE=10000
HISTFILESIZE=20000

# Custom prompt (covered more in Chapter 18)
PS1='\u@\h:\w\$ '

After editing ~/.bashrc, reload it:

$ source ~/.bashrc
# or equivalently
$ . ~/.bashrc

Distro Note: On Debian/Ubuntu, the system-wide bashrc is /etc/bash.bashrc. On Fedora/RHEL, it is /etc/bashrc. Both serve the same purpose.

Combining Commands

The shell has several ways to combine commands. Here is a quick preview (Chapter 18 goes deeper):

# Run commands sequentially (regardless of success/failure)
$ command1 ; command2

# Run command2 only if command1 succeeds
$ command1 && command2

# Run command2 only if command1 fails
$ command1 || command2

# Pipe: send output of command1 as input to command2
$ command1 | command2

# Examples:
$ mkdir mydir && cd mydir        # create dir, then enter it
$ sudo apt update && sudo apt upgrade -y   # update, then upgrade
$ cat /etc/passwd | grep yourname          # find your line in passwd
$ ls /nonexistent || echo "Directory not found"   # handle failure

What Just Happened?

┌─────────────────────────────────────────────────────┐
│                                                      │
│  In this chapter, you learned:                       │
│                                                      │
│  - The terminal is the window. The shell is the      │
│    program inside it. Bash is the default shell.     │
│                                                      │
│  - Command anatomy: command [options] [arguments].   │
│    Options modify behavior, arguments specify        │
│    targets.                                          │
│                                                      │
│  - Essential commands: pwd, ls, cd, cat, echo, man.  │
│    These are your daily tools.                       │
│                                                      │
│  - Tab completion saves keystrokes and prevents      │
│    typos. Use it relentlessly.                       │
│                                                      │
│  - Command history (Ctrl+R, !!, history) lets you    │
│    reuse and search past commands.                   │
│                                                      │
│  - Environment variables (HOME, USER, PATH, SHELL)   │
│    configure the shell and programs.                 │
│                                                      │
│  - PATH tells the shell where to find commands.      │
│    Modify it to add custom tool locations.           │
│                                                      │
│  - Getting help: man, --help, info. man is your      │
│    primary reference.                                │
│                                                      │
│  - Keyboard shortcuts (Ctrl+A, Ctrl+E, Ctrl+R,      │
│    Ctrl+C, Ctrl+L) make you fast.                    │
│                                                      │
│  - ~/.bashrc is where you customize your shell.      │
│                                                      │
│  - Commands can be combined with ;, &&, ||, and |.   │
│                                                      │
└─────────────────────────────────────────────────────┘

Try This

Exercises

1. Navigation exercise: Starting from your home directory, use cd and ls to explore these directories: /etc, /var/log, /usr/bin, /tmp, /proc. For each, use ls to see what is inside and pwd to confirm your location. Return home with cd (no arguments).

2. man page exercise: Read the man page for ls (man ls). Find the option that sorts files by size. Find the option that shows file sizes in human-readable format. Run ls with both options on /usr/bin.

3. History exercise: Use history to find a command you ran earlier. Re-run it using !number. Practice using Ctrl+R to search your history for the word "ls".

4. PATH exercise: Run echo $PATH and identify each directory listed. Use ls to see what is in /usr/local/bin. How many executables are in /usr/bin? (Hint: ls /usr/bin | wc -l)

5. Variable exercise: Set a variable MY_NAME to your name. Echo it. Export it. Start a new Bash session (bash) and verify the variable is still set. Exit the sub-shell (exit). Now try without export -- what happens in the sub-shell?

6. Help exercise: For each of these commands, find out what they do using ONLY man or --help: wc, sort, head, tail, touch. Write a one-sentence description of each.

7. Keyboard shortcuts exercise: Type a long command (do not press Enter). Practice: move to the beginning with Ctrl+A, move to the end with Ctrl+E, delete the entire line with Ctrl+U, bring it back with Ctrl+Y. Clear the screen with Ctrl+L.

Bonus Challenge

Customize your ~/.bashrc with at least three aliases that will save you time. For example:

alias update='sudo apt update && sudo apt upgrade -y'
alias ports='ss -tlnp'
alias myip='curl -s ifconfig.me && echo'

Source your .bashrc and test each alias. Think about what commands you type most often and create aliases for them.

What's Next

You can navigate, list files, read file contents, get help, and customize your shell. In Chapter 5, we go deeper into the filesystem -- the hierarchy of directories that organizes everything in Linux, and the profound idea that in Linux, everything is a file.

The Linux Filesystem Hierarchy

Why This Matters

You have just joined a team as a junior systems administrator. Your first ticket reads: "Application logs are eating up disk space on the web servers. Clean up and figure out where they're writing." You SSH in and stare at the root of the filesystem. There are twenty-odd directories -- /var, /tmp, /usr, /opt, /etc -- and you have no idea where to look.

This is not a hypothetical situation. Every Linux professional hits this wall on day one. Unlike Windows, where programs scatter files across C:\Program Files, C:\Users, and the Registry, Linux follows a well-defined plan called the Filesystem Hierarchy Standard (FHS). Learn it once, and you will always know where to find configuration files, log files, binaries, libraries, and temporary data -- on any distribution, on any server, anywhere in the world.

By the end of this chapter, the Linux directory tree will feel like your own neighborhood.

Try This Right Now

Open a terminal and run these commands. Do not worry about understanding every detail yet -- we will cover each one shortly.

# See the top-level directory structure
ls /

# Get a tree view (install tree if needed)
tree -L 1 /

# How big is each top-level directory?
du -sh /* 2>/dev/null | sort -rh | head -15

# Where is the 'ls' command actually stored?
which ls
file $(which ls)

You should see something like this from ls /:

bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

Each of those directories has a specific purpose. Let us walk through every single one.

The Root of Everything: /

In Linux, there is one single directory tree. Everything -- every file, every device, every running process -- lives somewhere under / (called "root"). There are no drive letters like C: or D:. If you plug in a USB drive, it appears as a directory under /mnt or /media. If you add a second hard disk, it gets mounted somewhere inside this same tree.

/                         <-- The root of the entire filesystem
├── bin/                  <-- Essential user commands
├── boot/                 <-- Bootloader and kernel
├── dev/                  <-- Device files
├── etc/                  <-- System configuration
├── home/                 <-- User home directories
├── lib/                  <-- Essential shared libraries
├── media/                <-- Removable media mount points
├── mnt/                  <-- Temporary mount points
├── opt/                  <-- Optional/third-party software
├── proc/                 <-- Process and kernel info (virtual)
├── root/                 <-- Root user's home directory
├── run/                  <-- Runtime variable data
├── sbin/                 <-- Essential system binaries
├── srv/                  <-- Service data (web, FTP)
├── sys/                  <-- Kernel and device info (virtual)
├── tmp/                  <-- Temporary files
├── usr/                  <-- User programs and data
└── var/                  <-- Variable data (logs, mail, spool)

This layout is not arbitrary. It is defined by the Filesystem Hierarchy Standard (FHS), maintained by the Linux Foundation. Most major distributions (Debian, Ubuntu, Fedora, RHEL, SUSE, Arch) follow it, with minor variations.

Think About It: Why does Linux use a single unified tree instead of drive letters? What advantages does this give you as a sysadmin managing servers with dozens of disks?

Walking the Tree: Directory by Directory

/bin -- Essential User Binaries

This directory contains the commands that every user needs, even in single-user (rescue) mode. Think of the absolute basics: ls, cp, mv, rm, cat, echo, mkdir, chmod, grep.

ls /bin | head -20

On modern systems (Fedora, Ubuntu 20.04+, Arch), /bin is actually a symbolic link to /usr/bin. This is called the UsrMerge -- we will explain why shortly.

# Check if /bin is a symlink
ls -ld /bin
# Likely output: lrwxrwxrwx 1 root root 7 ... /bin -> usr/bin

Distro Note: On older CentOS 6 / Debian 8 systems, /bin and /usr/bin are separate directories. On modern Fedora, Ubuntu 20.04+, and Arch, they have been merged. If you are working on an older system, the distinction matters for rescue scenarios.

/sbin -- System Binaries

Similar to /bin, but these are commands that typically require root privileges: fdisk, mkfs, iptables, reboot, shutdown, ip.

ls /sbin | head -20

# Try running one as a regular user
fdisk -l
# You'll likely get "Permission denied" or empty output

# Now with sudo
sudo fdisk -l

On merged systems, /sbin is a symlink to /usr/sbin.

/etc -- Configuration Files

This is where the system-wide configuration lives. Every major service stores its config here. The name historically comes from "et cetera" -- it was originally a catch-all, but today it is strictly for configuration.

# List the top-level config files and directories
ls /etc

# Some important files you should know about:
cat /etc/hostname          # System hostname
cat /etc/os-release        # Distribution info
cat /etc/passwd            # User accounts (not actually passwords!)
cat /etc/fstab             # Filesystem mount table
ls /etc/ssh/               # SSH server/client config
ls /etc/systemd/           # systemd unit overrides

Key principle: configuration files in /etc are plain text. You can read, edit, back up, and version-control them. This is a major advantage of Linux -- there is no opaque binary registry.

# Find all config files modified in the last 24 hours
find /etc -type f -mtime -1 2>/dev/null

/home -- User Home Directories

Every regular user gets a directory here. Your personal files, shell configuration (.bashrc, .profile), SSH keys (.ssh/), and application settings all live under your home directory.

ls /home

# Your home directory
echo $HOME
ls -la ~

# Important dot-files
ls -la ~/.bashrc ~/.profile ~/.ssh/ 2>/dev/null

The ~ (tilde) character is a shortcut for your home directory. ~alice means /home/alice.

/root -- Root User's Home

The root user does not live in /home/root. Instead, root's home directory is /root. Why? Because /home might be on a separate partition that has not mounted yet during rescue operations. The root user needs a home directory available on the root filesystem.

sudo ls -la /root

/var -- Variable Data

This is where data that changes during normal system operation lives. This is where you look when logs are eating disk space (remember our opening scenario?).

# Key subdirectories
ls /var

# System logs -- your first stop for troubleshooting
ls /var/log/
sudo tail -20 /var/log/syslog      # Debian/Ubuntu
sudo tail -20 /var/log/messages     # RHEL/Fedora

# Package manager cache
du -sh /var/cache/apt 2>/dev/null    # Debian/Ubuntu
du -sh /var/cache/dnf 2>/dev/null    # Fedora

# Mail spool
ls /var/mail/ 2>/dev/null

# Variable runtime data
ls /var/run    # Usually a symlink to /run

Distro Note: On Debian/Ubuntu the main system log is /var/log/syslog. On RHEL/Fedora/CentOS it is /var/log/messages. On systems using only journald, use journalctl instead.

Common /var subdirectories:

/tmp -- Temporary Files

Any user or process can create files here. Files in /tmp are typically cleared on reboot (and on many systems, periodically by systemd-tmpfiles or tmpwatch).

ls /tmp

# Create a temp file
echo "test" > /tmp/mytest.txt
cat /tmp/mytest.txt

# Check if /tmp is a tmpfs (RAM-based filesystem)
df -h /tmp
mount | grep /tmp

Safety Warning: Never store anything important in /tmp. It may be cleared at any time. Also, because it is world-writable, be cautious about security -- other users can see filenames (though not necessarily contents) in /tmp.

/usr -- User Programs (The Big One)

/usr is typically the largest directory on the system. Despite its name, it is not about users -- it stands for "Unix System Resources" (retroactively). It contains the bulk of installed software.

ls /usr

du -sh /usr

/usr/
├── bin/          <-- Most user commands (merged with /bin)
├── sbin/         <-- Most admin commands (merged with /sbin)
├── lib/          <-- Libraries for /usr/bin and /usr/sbin
├── lib64/        <-- 64-bit libraries
├── include/      <-- C/C++ header files
├── share/        <-- Architecture-independent data (docs, man pages)
├── local/        <-- Locally installed software (not from package manager)
└── src/          <-- Source code (kernel headers, etc.)

The key subdirectory to know is /usr/local/. When you compile and install software from source (using make install), it goes here by default. This keeps it separate from package-manager-installed software.

# Software from your package manager
which nginx     # /usr/bin/nginx or /usr/sbin/nginx

# Software you compiled yourself
ls /usr/local/bin/

/opt -- Optional Software

Third-party commercial or self-contained software packages often install here. Examples: Google Chrome (/opt/google/chrome), some monitoring agents, vendor-specific tools.

ls /opt 2>/dev/null

# Each application gets its own subdirectory
# /opt/vendor_name/application_name

Think About It: When should software go in /usr/local versus /opt? The convention is: /usr/local for software that follows the traditional Unix layout (bin, lib, share), and /opt for self-contained packages that keep everything in one directory.

/boot -- Boot Files

Contains the Linux kernel, initial RAM disk (initramfs/initrd), and bootloader configuration (GRUB).

ls /boot

# You'll typically see:
# vmlinuz-*        -- The kernel
# initramfs-* or initrd-*  -- Initial RAM filesystem
# grub/ or grub2/  -- GRUB bootloader config

Safety Warning: Do not delete files in /boot unless you know exactly what you are doing. Deleting the wrong kernel image will make your system unbootable.

/dev -- Device Files

This is where the "everything is a file" philosophy shines. Hardware devices are represented as files here. Your hard disk is /dev/sda. Your terminal is /dev/tty. Random numbers come from /dev/urandom.

ls /dev | head -30

# Your disks
ls /dev/sd*  2>/dev/null   # SATA/SCSI disks
ls /dev/nvme* 2>/dev/null  # NVMe drives
ls /dev/vd*  2>/dev/null   # Virtual disks (VMs)

# Special devices
echo "Hello" > /dev/null   # The black hole -- discards everything
head -c 16 /dev/urandom | xxd   # Random bytes

# Your terminal
tty                        # Shows your terminal device
echo "Hello" > $(tty)      # Writes to your own terminal

We will dive deep into /dev in Chapter 8 when we discuss device files and the VFS.

/proc -- Process Information (Virtual)

This directory does not exist on disk. It is a virtual filesystem generated by the kernel in real time. Each numbered directory corresponds to a running process (by PID). It also exposes kernel parameters.

# List running processes
ls /proc | head -20

# Info about process 1 (init/systemd)
sudo cat /proc/1/cmdline | tr '\0' ' ' ; echo
sudo ls /proc/1/fd       # File descriptors

# System information
cat /proc/cpuinfo | head -20
cat /proc/meminfo | head -10
cat /proc/version
cat /proc/uptime

# Kernel tunable parameters
cat /proc/sys/net/ipv4/ip_forward

/sys -- System and Device Info (Virtual)

Another virtual filesystem, introduced in Linux 2.6. While /proc focuses on processes, /sys exposes a structured view of the kernel's device model: hardware, drivers, buses, and kernel subsystems.

# Block devices
ls /sys/block/

# CPU information
ls /sys/devices/system/cpu/

# Brightness control (on laptops)
cat /sys/class/backlight/*/brightness 2>/dev/null

# Network interface info
ls /sys/class/net/
cat /sys/class/net/eth0/address 2>/dev/null

/mnt and /media -- Mount Points

/mnt is traditionally used for temporarily mounting filesystems (an NFS share, an extra disk). /media is used by desktop environments for automatically mounting removable media (USB drives, CDs).

ls /mnt
ls /media

# Manually mount an ISO
sudo mkdir -p /mnt/iso
sudo mount -o loop some-image.iso /mnt/iso

/srv -- Service Data

Meant for data served by the system: web server document roots, FTP files, etc. In practice, many administrators use /var/www for web content instead, but the FHS recommends /srv.

/run -- Runtime Data

A tmpfs filesystem that holds runtime information: PID files, sockets, lock files. It is cleared on every boot. On many systems, /var/run is a symlink to /run.

ls /run
# You'll see: systemd/, user/, lock, utmp, etc.

# PID files for running services
cat /run/sshd.pid 2>/dev/null

The "Everything Is a File" Philosophy

This is one of the most important ideas in Unix and Linux. In Linux:

• A regular file is a file
• A directory is a file (that contains a list of other files)
• A hardware device is a file (/dev/sda)
• A running process's info is a file (/proc/1234/status)
• A network socket can be a file
• A pipe between two processes is a file
• Even kernel parameters are files (/proc/sys/*)

This means you can use the same tools -- cat, echo, read, write -- to interact with hardware, processes, and kernel settings, not just text files.

# Read CPU info like a file
cat /proc/cpuinfo | grep "model name" | head -1

# Write to a kernel parameter like a file
cat /proc/sys/net/ipv4/ip_forward         # Read it
sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'  # Write to it

# Read from a device like a file
sudo head -c 512 /dev/sda | xxd | head    # Read raw disk bytes

Safety Warning: The command head -c 512 /dev/sda reads raw bytes from your disk. It is safe (read-only). But never write to /dev/sda directly unless you want to destroy data. Commands like dd if=/dev/zero of=/dev/sda will wipe your disk instantly with no confirmation.

Hands-On: Exploring the Filesystem

Let us put this knowledge to work. Follow along on your own system.

Exercise 1: Finding Where Things Live

# Where is the bash binary?
which bash
# /usr/bin/bash

# Where does bash keep its config?
ls /etc/bash*
# /etc/bash.bashrc  /etc/bash_completion  etc.

# Where are bash's man pages?
man -w bash
# /usr/share/man/man1/bash.1.gz

Notice the pattern: binary in /usr/bin, config in /etc, docs in /usr/share.

Exercise 2: Using find to Search the Tree

# Find all files named "sshd_config"
sudo find / -name "sshd_config" 2>/dev/null

# Find all log files larger than 100MB
sudo find /var/log -size +100M 2>/dev/null

# Find config files modified in the last hour
sudo find /etc -type f -mmin -60 2>/dev/null

# Find all SUID binaries (security audit)
sudo find / -perm -4000 -type f 2>/dev/null

Exercise 3: Using tree for Visual Exploration

# Install tree if needed
# Debian/Ubuntu: sudo apt install tree
# Fedora: sudo dnf install tree
# Arch: sudo pacman -S tree

# Two levels deep from root
tree -L 2 / 2>/dev/null | head -50

# Show only directories
tree -d -L 2 /etc | head -40

# Show with file sizes
tree -sh /var/log 2>/dev/null | head -30

Exercise 4: Checking Disk Usage by Directory

# Top-level disk usage
du -sh /* 2>/dev/null | sort -rh

# What's eating space in /var?
sudo du -sh /var/*/ 2>/dev/null | sort -rh

# What's eating space in /var/log?
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -10

The UsrMerge: Why /bin Points to /usr/bin

On modern distributions, you will notice that /bin, /sbin, /lib, and /lib64 are symlinks:

ls -ld /bin /sbin /lib
# lrwxrwxrwx 1 root root 7 ... /bin -> usr/bin
# lrwxrwxrwx 1 root root 8 ... /sbin -> usr/sbin
# lrwxrwxrwx 1 root root 7 ... /lib -> usr/lib

Historically, the split between /bin and /usr/bin existed because /usr might be on a separate disk that was not available during early boot. The essential boot commands had to live in /bin on the root filesystem. Modern systems mount all filesystems before user space starts (via initramfs), so the split is no longer necessary. Merging simplifies packaging and avoids confusing questions like "should this binary go in /bin or /usr/bin?"

Debug This

A colleague tells you: "I installed a custom-compiled application yesterday but I can't find it. I used ./configure && make && sudo make install with default settings. Where did it go?"

Try to answer before reading the solution.

ls /usr/local/bin/
ls /usr/local/lib/
ls /usr/local/etc/

echo $PATH | tr ':' '\n' | grep local

What Just Happened?

+------------------------------------------------------------------+
|                    CHAPTER 5 RECAP                                |
+------------------------------------------------------------------+
|                                                                    |
|  - Linux has ONE directory tree, rooted at /                      |
|  - The FHS defines where everything goes                          |
|  - /bin, /sbin    -> Essential commands (often merged to /usr)    |
|  - /etc           -> System-wide configuration (plain text!)      |
|  - /home          -> User home directories                        |
|  - /var           -> Variable data: logs, caches, spool           |
|  - /tmp           -> Temporary files (cleared on reboot)          |
|  - /usr           -> Bulk of installed programs and libraries     |
|  - /usr/local     -> Locally compiled software                    |
|  - /opt           -> Self-contained third-party software          |
|  - /proc, /sys    -> Virtual filesystems (kernel info)            |
|  - /dev           -> Device files (everything is a file!)         |
|  - /boot          -> Kernel and bootloader                        |
|  - /mnt, /media   -> Mount points for extra filesystems           |
|                                                                    |
|  Tools: ls, find, tree, du, which, file, cat, mount              |
|                                                                    |
+------------------------------------------------------------------+

Try This

Exercises

1. Map your system. Run du -sh /* 2>/dev/null | sort -rh and identify which three directories consume the most space. Can you explain why?

2. Track down a service. Pick a service running on your system (e.g., sshd or cron). Find its binary (using which), its configuration file (in /etc), its log file (in /var/log), and its PID file (in /run).

3. Explore /proc. Find your shell's PID with echo $$, then explore /proc/<PID>/. Read status, cmdline, environ, and list fd/. What do you discover?

4. Find the largest log file. Using find and du, locate the single largest file in /var/log. What service is writing it?

5. Predict the location. Without searching, predict where each of these lives, then verify:

   • The timezone configuration
   • The system's DNS resolver settings
   • The top command binary
   • The kernel's view of mounted filesystems

Bonus Challenge

Write a shell script called fs-audit.sh that:

• Lists each top-level directory under / with its size
• Counts the number of files in /etc
• Shows the 5 largest files in /var/log
• Reports whether /tmp is a tmpfs or a regular directory
• Checks whether /bin is a symlink or a real directory

#!/bin/bash
echo "=== Filesystem Audit ==="
echo ""
echo "--- Top-level directory sizes ---"
du -sh /* 2>/dev/null | sort -rh
echo ""
echo "--- Number of files in /etc ---"
find /etc -type f 2>/dev/null | wc -l
echo ""
echo "--- 5 largest files in /var/log ---"
sudo find /var/log -type f -exec du -sh {} + 2>/dev/null | sort -rh | head -5
echo ""
echo "--- /tmp filesystem type ---"
df -T /tmp | tail -1
echo ""
echo "--- Is /bin a symlink? ---"
if [ -L /bin ]; then
    echo "Yes: $(ls -l /bin | awk '{print $NF}')"
else
    echo "No, /bin is a real directory"
fi

Next up: Chapter 6 -- Files, Permissions & Ownership. Now that you know where everything lives, you need to understand who can access it and how.

Files, Permissions & Ownership

Why This Matters

It is 2 AM. A deployment just went out and the application is returning 403 Forbidden errors. The developer says "it works on my machine." You SSH into the production server, check the application directory, and discover that the deployment script set the wrong file permissions -- the web server process cannot read the files it needs to serve. You run one chmod command and the site comes back to life.

Permission problems are among the most common issues in Linux administration. They are also among the quickest to fix -- if you understand how they work. This chapter gives you a thorough, hands-on understanding of Linux file types, permission bits, ownership, and the more advanced features like SUID, sticky bits, and ACLs.

Try This Right Now

# Look at permissions on some files
ls -l /etc/passwd
ls -l /etc/shadow
ls -la ~/

# Create a test file and inspect it
touch /tmp/permtest
ls -l /tmp/permtest

# What is your current umask?
umask

# Who are you?
id
whoami
groups

Look at the output of ls -l /etc/passwd:

-rw-r--r-- 1 root root 2847 Jan 15 10:30 /etc/passwd

That string -rw-r--r-- encodes the file type and permissions. By the end of this chapter, you will be able to read and set these in your sleep.

File Types in Linux

Linux has seven file types. The first character in the ls -l output tells you which type you are looking at.

Let us see each one:

# Regular file
ls -l /etc/hostname
# -rw-r--r-- ...

# Directory
ls -ld /etc
# drwxr-xr-x ...

# Symbolic link
ls -l /bin
# lrwxrwxrwx ... /bin -> usr/bin

# Block device (disk)
ls -l /dev/sda 2>/dev/null || ls -l /dev/vda 2>/dev/null
# brw-rw---- ...

# Character device (terminal)
ls -l /dev/tty
# crw-rw-rw- ...

# Socket
ls -l /run/systemd/journal/dev-log 2>/dev/null
# srw-rw-rw- ...

# Named pipe
mkfifo /tmp/mypipe
ls -l /tmp/mypipe
# prw-r--r-- ...
rm /tmp/mypipe

Think About It: Why does Linux distinguish between block devices and character devices? Think about how you access a hard disk (in blocks of data) versus a keyboard (one character at a time).

Understanding Permission Bits

Every file has three sets of permission bits:

  Owner   Group   Others
  r w x   r w x   r w x

Let us decode the string -rw-r--r--:

-    rw-    r--    r--
|    |      |      |
|    |      |      +-- Others: read only
|    |      +--------- Group: read only
|    +---------------- Owner: read + write
+--------------------- Type: regular file

Hands-On: See Permissions in Action

# Create a test directory
mkdir -p /tmp/permlab
cd /tmp/permlab

# Create a file
echo "Hello, permissions!" > testfile.txt
ls -l testfile.txt
# -rw-r--r-- 1 youruser yourgroup 20 ... testfile.txt

# Remove read permission for yourself
chmod u-r testfile.txt
cat testfile.txt
# cat: testfile.txt: Permission denied

# Restore read, remove for others
chmod u+r,o-r testfile.txt
ls -l testfile.txt
# -rw-r----- 1 youruser yourgroup 20 ... testfile.txt

# Make it executable
chmod u+x testfile.txt
ls -l testfile.txt
# -rwxr----- 1 youruser yourgroup 20 ... testfile.txt

Directory Permissions Are Different

This is a common source of confusion. For directories:

• r (read): You can list the directory contents with ls
• w (write): You can create, rename, or delete files inside
• x (execute): You can enter the directory with cd and access files inside

# Create a test directory
mkdir /tmp/dirtest
echo "secret" > /tmp/dirtest/file.txt

# Remove execute permission
chmod a-x /tmp/dirtest
ls /tmp/dirtest          # Works (read still allowed)
cat /tmp/dirtest/file.txt  # FAILS -- can't traverse directory

# Restore execute, remove read
chmod a+x,a-r /tmp/dirtest
ls /tmp/dirtest          # FAILS -- can't list
cat /tmp/dirtest/file.txt  # Works (if you know the filename)

# Clean up
chmod 755 /tmp/dirtest

Think About It: A directory has r permission but not x. Can you list files in it? Can you read those files? What about the reverse -- x but no r?

Octal (Numeric) Notation

Sysadmins almost always use the octal shorthand instead of the symbolic rwx notation. Each permission bit is a power of 2:

r = 4
w = 2
x = 1

So:  rwx = 4+2+1 = 7
     rw- = 4+2+0 = 6
     r-x = 4+0+1 = 5
     r-- = 4+0+0 = 4
     --- = 0+0+0 = 0

Three digits for owner, group, others:

chmod 755 file  -->  rwxr-xr-x   (owner: full, group/others: read+execute)
chmod 644 file  -->  rw-r--r--   (owner: read+write, group/others: read only)
chmod 700 file  -->  rwx------   (owner: full, nobody else)
chmod 600 file  -->  rw-------   (owner: read+write, nobody else)
chmod 777 file  -->  rwxrwxrwx   (everyone: full access -- almost never do this!)

Common permissions you will use every day:

# Practice: set permissions using octal
chmod 644 /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt

chmod 600 /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt

chmod 755 /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt

Safety Warning: Never use chmod 777 on production systems. It means "every user on this system can read, write, and execute this file." If someone tells you to run chmod -R 777 /, they are either joking or about to destroy your system's security.

Ownership: chown and chgrp

Every file has an owner (user) and a group. You change them with chown and chgrp.

# View ownership
ls -l /tmp/permlab/testfile.txt
# -rwxr-xr-x 1 youruser yourgroup 20 ... testfile.txt
#               ^^^^^^^^ ^^^^^^^^^
#               owner    group

# Change owner (requires root)
sudo chown root /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt

# Change group
sudo chgrp root /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt

# Change both at once: user:group
sudo chown youruser:yourgroup /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt

# Recursively change ownership of a directory
sudo chown -R www-data:www-data /var/www/html 2>/dev/null

Real-World Scenario: Web Server Permissions

A web server (Nginx or Apache) runs as a specific user, typically www-data (Debian/Ubuntu) or nginx (RHEL/Fedora). The web content files must be readable by that user.

# Typical web directory setup
sudo mkdir -p /var/www/mysite
sudo chown -R www-data:www-data /var/www/mysite
sudo chmod -R 755 /var/www/mysite
sudo chmod 644 /var/www/mysite/*.html 2>/dev/null

Distro Note: The web server user is www-data on Debian/Ubuntu, nginx on RHEL/Fedora (when using Nginx), and apache on RHEL/Fedora (when using Apache httpd).

The umask: Default Permission Control

When you create a new file or directory, the permissions are not set to 777. They are filtered through your umask (user file-creation mask).

# Check your current umask
umask
# Typical output: 0022

# What does 0022 mean?
# Default for files:   666 - 022 = 644 (rw-r--r--)
# Default for dirs:    777 - 022 = 755 (rwxr-xr-x)

# Verify
touch /tmp/umask-test-file
mkdir /tmp/umask-test-dir
ls -l /tmp/umask-test-file
ls -ld /tmp/umask-test-dir

The umask subtracts permissions. Common umask values:

# Temporarily change umask
umask 077
touch /tmp/private-file
ls -l /tmp/private-file
# -rw------- 1 youruser yourgroup 0 ... /tmp/private-file

# Reset to default
umask 022

To make umask changes permanent, add umask 027 to your ~/.bashrc or set it system-wide in /etc/login.defs or /etc/profile.

Special Permission Bits: SUID, SGID, and Sticky

Beyond the basic rwx bits, there are three special bits that you will encounter in production systems.

SUID (Set User ID) -- The 4 Bit

When set on an executable, the program runs as the file's owner, not the user who invoked it. This is how passwd can modify /etc/shadow even when run by a regular user.

# Notice the 's' in the owner's execute position
ls -l /usr/bin/passwd
# -rwsr-xr-x 1 root root ... /usr/bin/passwd
#    ^
#    SUID bit

# Find all SUID binaries on the system
sudo find / -perm -4000 -type f 2>/dev/null

Setting SUID:

# Using octal (4 prepended)
chmod 4755 somefile

# Using symbolic
chmod u+s somefile

Safety Warning: SUID binaries are a security-sensitive feature. A SUID root program with a vulnerability can give attackers root access. Regular security audits should check for unexpected SUID binaries.

SGID (Set Group ID) -- The 2 Bit

On an executable: the program runs with the file's group privileges. On a directory: new files created inside inherit the directory's group (instead of the creator's primary group). This is extremely useful for shared project directories.

# Create a shared project directory
sudo mkdir /tmp/project
sudo chgrp developers /tmp/project 2>/dev/null || sudo chgrp staff /tmp/project
sudo chmod 2775 /tmp/project
ls -ld /tmp/project
# drwxrwsr-x ... /tmp/project
#         ^
#         SGID bit (s in group execute position)

# New files inherit the group
touch /tmp/project/newfile
ls -l /tmp/project/newfile
# The group will be 'developers' (or 'staff'), not your primary group

Sticky Bit -- The 1 Bit

When set on a directory, only the file's owner (or root) can delete or rename files inside, even if others have write permission. The classic example is /tmp.

ls -ld /tmp
# drwxrwxrwt ... /tmp
#          ^
#          Sticky bit (t in others execute position)

# Without the sticky bit, any user could delete any other user's files in /tmp
# With it, you can only delete your own files

Setting the sticky bit:

chmod 1777 /tmp/shared-dir    # Octal
chmod +t /tmp/shared-dir      # Symbolic

Summary of Special Bits

Octal    Symbolic    Position              Effect
4xxx     u+s         Owner execute: s/S    SUID -- run as file owner
2xxx     g+s         Group execute: s/S    SGID -- run as file group / inherit group
1xxx     +t          Other execute: t/T    Sticky -- only owner can delete

When you see an uppercase S or T instead of lowercase, it means the special bit is set but the underlying execute bit is NOT set (an unusual and often incorrect configuration).

# Uppercase S means SUID without execute -- usually a mistake
chmod 4644 /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt
# -rwSr--r-- ... (capital S = SUID set but no execute)

# Fix it
chmod 4755 /tmp/permlab/testfile.txt
ls -l /tmp/permlab/testfile.txt
# -rwsr-xr-x ... (lowercase s = SUID with execute)

Access Control Lists (ACLs)

Traditional Unix permissions are limited: one owner, one group, and everyone else. What if you need to give user alice read access and user bob read+write access to the same file, without creating a special group? ACLs solve this.

Checking ACL Support

# Check if the filesystem supports ACLs
mount | grep -E 'ext4|xfs|btrfs'
# Most modern filesystems support ACLs by default

# Install ACL tools if needed
# Debian/Ubuntu: sudo apt install acl
# Fedora/RHEL: sudo dnf install acl
# (usually pre-installed)

Using getfacl and setfacl

# Create a test file
echo "ACL test" > /tmp/acltest.txt

# View current ACL
getfacl /tmp/acltest.txt
# file: tmp/acltest.txt
# owner: youruser
# group: yourgroup
# user::rw-
# group::r--
# other::r--

# Grant specific user read+write access
sudo setfacl -m u:nobody:rw /tmp/acltest.txt

# View updated ACL
getfacl /tmp/acltest.txt
# Notice the new line: user:nobody:rw-

# ls -l now shows a '+' indicating ACLs are present
ls -l /tmp/acltest.txt
# -rw-rw-r--+ 1 youruser yourgroup ... /tmp/acltest.txt
#           ^
#           The + means ACLs are set

# Grant a group specific permissions
sudo setfacl -m g:adm:r /tmp/acltest.txt

# Set a default ACL on a directory (affects new files created inside)
mkdir /tmp/acl-dir
setfacl -d -m u:nobody:rwx /tmp/acl-dir
getfacl /tmp/acl-dir

# Remove a specific ACL entry
setfacl -x u:nobody /tmp/acltest.txt

# Remove all ACLs
setfacl -b /tmp/acltest.txt

ACL Quick Reference

Think About It: When would you use ACLs instead of traditional Unix permissions? Think about a web development team where the designer needs read-only access to CSS files, the developer needs full access, and the deployment bot needs read+execute.

Real-World Permission Scenarios

Scenario 1: SSH Key Permissions

SSH is very strict about permissions. If your key files are too permissive, SSH will refuse to use them.

# These are the required permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa            # Private key: owner read/write ONLY
chmod 644 ~/.ssh/id_rsa.pub        # Public key: readable
chmod 600 ~/.ssh/authorized_keys   # authorized_keys: owner read/write ONLY
chmod 644 ~/.ssh/config            # SSH config: readable

# If you get "Permissions too open" errors, this is why

Scenario 2: Shared Team Directory

# Create a shared directory for the 'devteam' group
sudo groupadd devteam
sudo usermod -aG devteam alice
sudo usermod -aG devteam bob

sudo mkdir /opt/project
sudo chown root:devteam /opt/project
sudo chmod 2775 /opt/project
# 2 = SGID (new files inherit the group)
# 775 = rwxrwxr-x (owner and group can write, others can read)

Scenario 3: Fixing the Web Server 403 Error

# The problem: web server can't read files
# Step 1: Check what user the web server runs as
ps aux | grep -E 'nginx|apache|httpd' | head -3

# Step 2: Check current permissions
ls -la /var/www/html/

# Step 3: Fix ownership
sudo chown -R www-data:www-data /var/www/html/

# Step 4: Fix permissions
sudo find /var/www/html -type d -exec chmod 755 {} \;
sudo find /var/www/html -type f -exec chmod 644 {} \;

Debug This

You have a script called deploy.sh with the following permissions:

-rw-r--r-- 1 deploy deploy 1542 Jan 20 14:30 deploy.sh

When you try to run it, you get "Permission denied." You run chmod 777 deploy.sh and it works, but your security team flags this as a violation. What is the minimum permission change needed?

chmod u+x deploy.sh
# Result: -rwxr--r-- (owner can execute, no extra access for others)

# Or if the whole team needs to run it:
chmod 755 deploy.sh
# Result: -rwxr-xr-x (everyone can execute, only owner can modify)

What Just Happened?

+------------------------------------------------------------------+
|                    CHAPTER 6 RECAP                                |
+------------------------------------------------------------------+
|                                                                    |
|  File Types: regular(-) dir(d) link(l) block(b) char(c)          |
|              socket(s) pipe(p)                                    |
|                                                                    |
|  Permission Bits:    r=4   w=2   x=1                              |
|  Three sets:         Owner | Group | Others                       |
|  Common octals:      644 (files), 755 (dirs), 600 (private)      |
|                                                                    |
|  Commands:                                                         |
|    chmod   -- Change permissions (symbolic or octal)              |
|    chown   -- Change owner (and optionally group)                 |
|    chgrp   -- Change group                                        |
|    umask   -- Set default permission mask                         |
|                                                                    |
|  Special bits:                                                     |
|    SUID (4xxx)  -- Run as file owner                              |
|    SGID (2xxx)  -- Run as file group / inherit group in dirs      |
|    Sticky (1xxx) -- Only owner can delete in shared dirs          |
|                                                                    |
|  ACLs: getfacl / setfacl for fine-grained access control         |
|                                                                    |
+------------------------------------------------------------------+

Try This

Exercises

1. Decode these permissions. Without using a computer, write out what each octal means in rwx notation:

   • 750
   • 4711
   • 1777
   • 2755
   • 0600

2. Permission detective. Run ls -l /usr/bin/passwd, ls -l /etc/shadow, and ls -l /etc/passwd. Explain why passwd has the SUID bit and how this allows regular users to change their password.

3. Directory permissions lab. Create a directory where:

   • The owner can do everything
   • The group can read and enter but not create files
   • Others have no access at all What octal value is this?

4. ACL challenge. Create a file. Using ACLs, give three different users three different levels of access (one gets r, another rw, another rwx). Verify with getfacl.

5. Security audit. Find all SUID and SGID binaries on your system. How many are there? Research what three of them do and explain why they need the special bit.

Bonus Challenge

Write a script that takes a directory as an argument and produces a permission report:

• Lists all files with their octal permissions
• Flags any files with 777 permissions
• Flags any SUID/SGID files
• Reports the total number of files for each permission level

#!/bin/bash
# permission-audit.sh - Audit file permissions in a directory
DIR="${1:-.}"
echo "=== Permission Audit: $DIR ==="
echo ""

echo "--- Files with 777 permissions (DANGER!) ---"
find "$DIR" -perm 777 -type f 2>/dev/null

echo ""
echo "--- SUID files ---"
find "$DIR" -perm -4000 -type f 2>/dev/null

echo ""
echo "--- SGID files ---"
find "$DIR" -perm -2000 -type f 2>/dev/null

echo ""
echo "--- Permission distribution ---"
find "$DIR" -type f -printf '%m\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20

echo ""
echo "--- World-writable files ---"
find "$DIR" -perm -o+w -type f 2>/dev/null

Next up: Chapter 7 -- Disks, Partitions & Filesystems. Now that you know about files and permissions, let us look at the physical storage underneath it all.

Disks, Partitions & Filesystems

Why This Matters

A monitoring alert fires: "Disk usage on /var has crossed 90%." You need to add a new disk, partition it, create a filesystem, and mount it -- all without rebooting, all without losing data, and ideally within the next fifteen minutes before the application crashes because it cannot write logs.

This is not a rare event. Disk management is a bread-and-butter skill for any Linux administrator. Whether you are provisioning a new server, expanding storage for a growing database, or preparing a fresh cloud instance, you need to understand how Linux sees physical disks, how you carve them into partitions, how you put a filesystem on top, and how you make that filesystem accessible to the rest of the system.

Try This Right Now

# What block devices does the system have?
lsblk

# How much disk space is used?
df -h

# What filesystems are mounted?
mount | column -t | head -20

# What is the disk usage of your current directory?
du -sh .

# What are the UUIDs of your partitions?
sudo blkid

These five commands form the core toolkit for day-to-day disk management. Let us understand every one of them deeply.

Block Devices: How Linux Sees Disks

A block device is a piece of hardware (or virtualized hardware) that stores data in fixed-size blocks and supports random access. Hard drives, SSDs, NVMe drives, USB sticks, and virtual disks are all block devices.

Linux represents block devices as files in /dev/:

Device type       Naming scheme         Example
-----------       -------------         -------
SATA/SCSI/SAS     /dev/sdX              /dev/sda, /dev/sdb
NVMe              /dev/nvmeXnY          /dev/nvme0n1
VirtIO (VMs)      /dev/vdX              /dev/vda, /dev/vdb
MMC/SD cards      /dev/mmcblkX          /dev/mmcblk0
Floppy (legacy)   /dev/fdX              /dev/fd0

Partitions are numbered:

/dev/sda          <-- Entire first SATA disk
/dev/sda1         <-- First partition on sda
/dev/sda2         <-- Second partition on sda
/dev/sda3         <-- Third partition on sda

/dev/nvme0n1      <-- Entire first NVMe drive
/dev/nvme0n1p1    <-- First partition on nvme0n1
/dev/nvme0n1p2    <-- Second partition on nvme0n1

lsblk -- Your Best Friend

lsblk lists all block devices in a tree format, showing the relationship between disks, partitions, and mount points.

lsblk

Typical output:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0    50G  0 disk
├─sda1        8:1    0     1G  0 part /boot
├─sda2        8:2    0    45G  0 part /
└─sda3        8:3    0     4G  0 part [SWAP]
sdb           8:16   0   100G  0 disk
└─sdb1        8:17   0   100G  0 part /data

Useful lsblk options:

# Show filesystem type and UUIDs
lsblk -f

# Show sizes in bytes
lsblk -b

# Show all devices including empty ones
lsblk -a

# Output specific columns
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID

MBR vs GPT: Partition Table Formats

Before you create partitions, the disk needs a partition table -- a data structure at the beginning of the disk that records the layout.

MBR (Master Boot Record)

• The original PC partition scheme from 1983
• Supports up to 4 primary partitions (or 3 primary + 1 extended with unlimited logical partitions)
• Maximum disk size: 2 TB
• Stores the partition table in the first 512 bytes of the disk
• Used with BIOS boot

GPT (GUID Partition Table)

• Modern standard, part of UEFI specification
• Supports up to 128 partitions by default (no extended/logical nonsense)
• Maximum disk size: 9.4 ZB (zettabytes -- effectively unlimited)
• Stores redundant copies of the partition table (beginning and end of disk)
• Includes CRC32 checksums for integrity
• Used with UEFI boot (but can also work with BIOS via "BIOS boot partition")

+-------------------------------------------------------------------+
|                     MBR vs GPT Comparison                         |
+-------------------------------------------------------------------+
|                                                                     |
|   MBR (Legacy)              GPT (Modern)                          |
|   +-----------+             +--+--+--+--+--+--+--+--+--+--+      |
|   | MBR | P1 | P2 | P3 | P4 |   | GP | P1 | P2 | .. | Pn | GP |  |
|   +-----------+             +--+--+--+--+--+--+--+--+--+--+      |
|   Max 4 primary             Up to 128 partitions                   |
|   Max 2 TB disk             Max 9.4 ZB disk                       |
|   No redundancy             Backup table at disk end              |
|   BIOS boot                 UEFI boot (usually)                   |
|                                                                     |
+-------------------------------------------------------------------+

Think About It: When should you still use MBR? Mainly for legacy systems that boot with BIOS (not UEFI), or for very small disks where compatibility with older tools matters. For anything new, use GPT.

Partitioning Tools: fdisk and parted

fdisk -- Interactive Disk Partitioning

fdisk is the classic Linux partitioning tool. It now supports both MBR and GPT. It is interactive but straightforward.

Safety Warning: Partitioning a disk that contains data will destroy that data. Always double-check the device name. Running fdisk /dev/sda when you meant /dev/sdb could wipe your operating system. Use lsblk first to confirm which disk is which.

# ALWAYS verify your target disk first
lsblk

# View existing partition table (safe, read-only)
sudo fdisk -l /dev/sdb

# Start interactive partitioning (ONLY on a disk you want to modify)
sudo fdisk /dev/sdb

Inside fdisk, the key commands are:

Hands-On: Partitioning with fdisk (Using a Loop Device)

We will safely practice by creating a virtual disk using a loop device. This will not touch your real disks.

# Create a 500MB file to use as a virtual disk
dd if=/dev/zero of=/tmp/practice-disk.img bs=1M count=500

# Attach it as a loop device
sudo losetup -fP /tmp/practice-disk.img
LOOPDEV=$(sudo losetup -j /tmp/practice-disk.img | cut -d: -f1)
echo "Our virtual disk is: $LOOPDEV"

# Verify it appears
lsblk $LOOPDEV

# Now partition it with fdisk
sudo fdisk $LOOPDEV

Inside fdisk, follow these steps:

Command (m for help): g        <-- Create GPT partition table
Command (m for help): n        <-- New partition
Partition number: 1            <-- Accept default
First sector: [Enter]          <-- Accept default
Last sector: +200M             <-- 200MB partition
Command (m for help): n        <-- New partition
Partition number: 2            <-- Accept default
First sector: [Enter]          <-- Accept default
Last sector: [Enter]           <-- Use remaining space
Command (m for help): p        <-- Print to verify
Command (m for help): w        <-- Write and exit

# Verify the partitions
lsblk $LOOPDEV

# You should see:
# loop0        7:0    0   500M  0 loop
# ├─loop0p1  259:0    0   200M  0 part
# └─loop0p2  259:1    0   298M  0 part

parted -- Non-Interactive Partitioning

parted is more powerful than fdisk and can be used non-interactively, which makes it better for scripting.

# View disk info
sudo parted $LOOPDEV print

# Non-interactive examples (DON'T run these -- just reference):
# Create GPT label
# sudo parted /dev/sdb mklabel gpt

# Create a partition from 1MiB to 500MiB
# sudo parted /dev/sdb mkpart primary ext4 1MiB 500MiB

Distro Note: On Fedora/RHEL, parted is installed by default. On minimal Debian/Ubuntu installs, you may need sudo apt install parted.

Creating Filesystems with mkfs

A partition is just a raw chunk of disk space. Before you can store files on it, you need to put a filesystem on it. The mkfs command creates filesystems.

Common Linux Filesystems

# Create ext4 filesystem on our practice partitions
sudo mkfs.ext4 ${LOOPDEV}p1

# Create XFS filesystem
sudo mkfs.xfs ${LOOPDEV}p2

# Verify
sudo blkid ${LOOPDEV}p1 ${LOOPDEV}p2
lsblk -f $LOOPDEV

You should see output like:

NAME       FSTYPE LABEL UUID                                 MOUNTPOINT
loop0
├─loop0p1  ext4         a1b2c3d4-e5f6-7890-abcd-ef1234567890
└─loop0p2  xfs          f1e2d3c4-b5a6-7890-abcd-ef0987654321

mkfs Options

# ext4 with a label
sudo mkfs.ext4 -L "mydata" /dev/sdXn

# ext4 with specific block size
sudo mkfs.ext4 -b 4096 /dev/sdXn

# XFS with a label
sudo mkfs.xfs -L "bigdata" /dev/sdXn

# Btrfs with a label
sudo mkfs.btrfs -L "snapshots" /dev/sdXn

Safety Warning: mkfs destroys all existing data on the partition. There is no "Are you sure?" prompt for most variants. Triple-check the device name before running it.

Mounting and Unmounting Filesystems

A filesystem on a partition is useless until it is mounted -- attached to a directory in the filesystem tree.

mount -- Attach a Filesystem

# Create mount points
sudo mkdir -p /mnt/part1
sudo mkdir -p /mnt/part2

# Mount our practice partitions
sudo mount ${LOOPDEV}p1 /mnt/part1
sudo mount ${LOOPDEV}p2 /mnt/part2

# Verify
df -h /mnt/part1 /mnt/part2
mount | grep loop

# Use the filesystems
echo "Hello from ext4!" | sudo tee /mnt/part1/hello.txt
echo "Hello from XFS!" | sudo tee /mnt/part2/hello.txt
cat /mnt/part1/hello.txt
cat /mnt/part2/hello.txt

umount -- Detach a Filesystem

# Unmount (note: it's "umount", not "unmount"!)
sudo umount /mnt/part1
sudo umount /mnt/part2

If you get "device is busy":

# Find what is using the mount point
sudo lsof +D /mnt/part1
# or
sudo fuser -mv /mnt/part1

# Force unmount (use with caution)
sudo umount -l /mnt/part1    # Lazy unmount -- detaches immediately,
                              # cleans up when no longer in use

Mount Options

# Mount read-only
sudo mount -o ro ${LOOPDEV}p1 /mnt/part1

# Mount with no-exec (prevent running executables)
sudo mount -o noexec ${LOOPDEV}p1 /mnt/part1

# Mount with nosuid (ignore SUID bits)
sudo mount -o nosuid ${LOOPDEV}p1 /mnt/part1

# Combine options
sudo mount -o ro,noexec,nosuid ${LOOPDEV}p1 /mnt/part1

# Remount with different options (without unmounting)
sudo mount -o remount,rw /mnt/part1

Persistent Mounts with /etc/fstab

Mounts created with the mount command are temporary -- they disappear on reboot. To make mounts persistent, add them to /etc/fstab.

# View your current fstab
cat /etc/fstab

fstab Format

Each line in /etc/fstab has six fields:

<device>     <mount point>  <fstype>  <options>       <dump>  <pass>
UUID=xxxx    /data          ext4      defaults        0       2
/dev/sdb1    /backup        xfs       defaults,noatime 0      2

Using UUIDs (Best Practice)

Device names like /dev/sdb1 can change if you add or remove disks. UUIDs are unique identifiers that never change.

# Find UUIDs
sudo blkid

# Example output:
# /dev/sda1: UUID="a1b2c3d4-..." TYPE="ext4"
# /dev/sda2: UUID="e5f6a7b8-..." TYPE="swap"

Adding an Entry to fstab

# Get the UUID of your partition
UUID=$(sudo blkid -s UUID -o value ${LOOPDEV}p1)
echo "UUID is: $UUID"

# Add to fstab (be VERY careful editing this file)
echo "UUID=$UUID  /mnt/part1  ext4  defaults  0  2" | sudo tee -a /etc/fstab

# Test without rebooting
sudo mount -a

# Verify
df -h /mnt/part1

Safety Warning: A syntax error in /etc/fstab can prevent your system from booting. Always test with sudo mount -a after editing. If you are editing remotely, keep a second SSH session open as a safety net. Also consider using sudo findmnt --verify to check for errors.

# Validate fstab syntax
sudo findmnt --verify --tab-file /etc/fstab

Think About It: Why does the pass field matter? Think about what happens if the system loses power while writing to disk, and the filesystem needs to be checked for consistency before it can be safely used.

Checking Disk Space: df and du

df -- Disk Free (Filesystem Level)

Shows how much space is used and available on each mounted filesystem.

# Human-readable sizes
df -h

# Show filesystem types
df -hT

# Check specific mount point
df -h /var

# Show only real filesystems (exclude tmpfs, devtmpfs, etc.)
df -h -x tmpfs -x devtmpfs -x squashfs

Example output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        45G   12G   31G  28% /
/dev/sda1       976M  150M  760M  17% /boot
/dev/sdb1       100G   45G   55G  45% /data

du -- Disk Usage (Directory Level)

Shows the disk space used by files and directories.

# Size of current directory
du -sh .

# Size of each subdirectory
du -sh /var/*/ 2>/dev/null | sort -rh | head -10

# Size of a specific directory, 1 level deep
du -h --max-depth=1 /var 2>/dev/null | sort -rh

# Find the largest files in a directory
du -ah /var/log 2>/dev/null | sort -rh | head -10

# Total size of a directory
du -sh /usr

Practical: Finding What Is Eating Your Disk

# Step 1: Which filesystem is full?
df -h

# Step 2: Find the biggest directories
sudo du -h --max-depth=1 / 2>/dev/null | sort -rh | head -10

# Step 3: Drill down into the culprit
sudo du -h --max-depth=1 /var 2>/dev/null | sort -rh | head -10

# Step 4: Find individual large files
sudo find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -10

# Step 5: Check for deleted files still held open
sudo lsof +L1 2>/dev/null

Swap Space

Swap is disk space used as an extension of RAM. When physical memory is full, the kernel moves inactive pages to swap. While slower than RAM, it prevents out-of-memory crashes.

Checking Current Swap

# View swap
free -h
swapon --show
cat /proc/swaps

Creating a Swap Partition

# On a real disk (example -- do not run blindly):
# sudo mkswap /dev/sda3
# sudo swapon /dev/sda3

Creating a Swap File

This is often easier than creating a swap partition, especially on cloud instances.

# Create a 1GB swap file
sudo fallocate -l 1G /swapfile
# or: sudo dd if=/dev/zero of=/swapfile bs=1M count=1024

# Set correct permissions (MUST be 600)
sudo chmod 600 /swapfile

# Format as swap
sudo mkswap /swapfile

# Enable it
sudo swapon /swapfile

# Verify
swapon --show
free -h

To make it permanent, add to /etc/fstab:

/swapfile    none    swap    sw    0    0

Swappiness

The vm.swappiness parameter controls how aggressively the kernel uses swap (0-100).

# Check current value (default is usually 60)
cat /proc/sys/vm/swappiness

# Reduce swappiness (prefer RAM over swap)
sudo sysctl vm.swappiness=10

# Make permanent
echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.d/99-swappiness.conf

Hands-On: Complete Disk Workflow

Let us bring everything together. We will use our practice loop device to walk through the full lifecycle.

# Step 1: Create a virtual disk
dd if=/dev/zero of=/tmp/lab-disk.img bs=1M count=200

# Step 2: Attach as loop device
sudo losetup -fP /tmp/lab-disk.img
DISK=$(sudo losetup -j /tmp/lab-disk.img | cut -d: -f1)
echo "Working with: $DISK"

# Step 3: Create a GPT partition table and one partition
echo -e "g\nn\n1\n\n\nw" | sudo fdisk $DISK

# Step 4: Create an ext4 filesystem with a label
sudo mkfs.ext4 -L "labdata" ${DISK}p1

# Step 5: Create a mount point
sudo mkdir -p /mnt/labdata

# Step 6: Mount it
sudo mount ${DISK}p1 /mnt/labdata

# Step 7: Verify
lsblk $DISK
df -h /mnt/labdata
mount | grep labdata

# Step 8: Use it
echo "Disk management is easy!" | sudo tee /mnt/labdata/test.txt
cat /mnt/labdata/test.txt

# Step 9: Check the filesystem details
sudo tune2fs -l ${DISK}p1 | head -20    # ext4 info
sudo dumpe2fs ${DISK}p1 2>/dev/null | head -30

# Step 10: Clean up
sudo umount /mnt/labdata
sudo losetup -d $DISK
rm /tmp/lab-disk.img

Debug This

A developer reports: "I added a 100GB disk to the server and added it to /etc/fstab, but after rebooting the server, it boots into emergency mode." You check the console and see filesystem mount errors.

What are the likely causes?

# Edit fstab to fix the error
vi /etc/fstab

# Or use nofail option to prevent boot failure
UUID=xxxx  /data  ext4  defaults,nofail  0  2

What Just Happened?

+------------------------------------------------------------------+
|                    CHAPTER 7 RECAP                                |
+------------------------------------------------------------------+
|                                                                    |
|  Block Devices:                                                    |
|    /dev/sdX (SATA)  /dev/nvmeXnY (NVMe)  /dev/vdX (VirtIO)     |
|                                                                    |
|  Partition Tables: MBR (legacy, 2TB max) vs GPT (modern)         |
|                                                                    |
|  Partitioning:  fdisk (interactive)  parted (scriptable)         |
|                                                                    |
|  Filesystems: ext4 (default), XFS (large), Btrfs (snapshots)    |
|  Create with: mkfs.ext4, mkfs.xfs, mkfs.btrfs                   |
|                                                                    |
|  Mounting: mount/umount, /etc/fstab for persistence              |
|  Always use UUIDs in fstab, not device names                     |
|  Use 'nofail' for non-critical mounts                            |
|                                                                    |
|  Space checks: df -h (filesystem), du -sh (directory)            |
|                                                                    |
|  Swap: mkswap + swapon, swap files for flexibility               |
|                                                                    |
|  Tools: lsblk, blkid, fdisk, parted, mkfs, mount, umount,       |
|         df, du, swapon, free, findmnt                            |
|                                                                    |
+------------------------------------------------------------------+

Try This

Exercises

1. Explore your system. Run lsblk -f and df -hT. Identify every physical disk, partition, filesystem type, and mount point on your system. Draw it as a tree diagram.

2. Practice safe partitioning. Create a 1GB virtual disk with dd, partition it into three parts using fdisk (a 200MB ext4, a 500MB XFS, and the rest as swap). Create the filesystems, mount them, and verify.

3. fstab lab. For each partition in exercise 2, add an fstab entry using UUIDs. Use the nofail option. Test with sudo mount -a.

4. Disk usage investigation. Write a one-liner that finds the 10 largest files on your system (hint: find / -type f -printf '%s %p\n' | sort -rn | head -10).

5. Swap file. Create a 512MB swap file, enable it, verify it shows in free -h, then disable and remove it.

Bonus Challenge

Write a script called disk-report.sh that generates a comprehensive disk report:

#!/bin/bash
echo "=== Disk Report: $(hostname) -- $(date) ==="
echo ""

echo "--- Block Devices ---"
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID

echo ""
echo "--- Filesystem Usage ---"
df -hT -x tmpfs -x devtmpfs -x squashfs

echo ""
echo "--- Largest Directories Under / ---"
sudo du -h --max-depth=1 / 2>/dev/null | sort -rh | head -10

echo ""
echo "--- Swap Status ---"
free -h | grep -i swap
swapon --show

echo ""
echo "--- Partitions Over 80% Full ---"
df -h | awk 'NR>1 && int($5)>80 {print $0}'

echo ""
echo "--- fstab Entries ---"
grep -v '^#' /etc/fstab | grep -v '^$'

Next up: Chapter 8 -- Links, Inodes & the VFS. You now understand the physical storage layer. It is time to peek behind the curtain and see how Linux actually tracks files internally.

Links, Inodes & the VFS

Why This Matters

You delete a 2GB log file, but df -h shows no change in disk usage. You check and the file is gone -- ls confirms it. Yet the space is not freed. What is going on?

The answer lies in a concept called inodes -- the hidden data structures that the filesystem uses to track every file. The file you deleted was still held open by a running process, and as long as that process keeps its file descriptor open, the inode (and the data blocks it points to) stays alive. Understanding inodes, links, and how Linux's Virtual Filesystem layer ties everything together gives you the ability to diagnose puzzles like this in seconds.

This chapter pulls back the curtain on how Linux tracks files internally, how hard links and soft links work (and why they behave differently), and how the VFS lets you interact with wildly different systems -- physical disks, RAM-based pseudo-filesystems, network shares -- through a single uniform interface.

Try This Right Now

# See the inode number of a file
ls -i /etc/hostname

# Get detailed inode information
stat /etc/hostname

# See inode usage on your filesystem
df -i

# Create and compare hard and soft links
echo "original content" > /tmp/original.txt
ln /tmp/original.txt /tmp/hardlink.txt
ln -s /tmp/original.txt /tmp/softlink.txt

ls -li /tmp/original.txt /tmp/hardlink.txt /tmp/softlink.txt

Look at the output of that last command carefully. The original file and the hard link share the same inode number. The soft link has a different inode. This single observation is the key to understanding everything in this chapter.

What Is an Inode?

An inode (index node) is a data structure on disk that stores all the metadata about a file -- everything except the file's name and its actual data. Every file and directory on an ext4, XFS, or Btrfs filesystem has exactly one inode.

What an Inode Stores

+------------------------------------------+
|              INODE #12345                 |
+------------------------------------------+
| File type:       regular file             |
| Permissions:     rw-r--r-- (644)          |
| Owner (UID):     1000                     |
| Group (GID):     1000                     |
| Size:            4096 bytes               |
| Timestamps:                               |
|   - atime:       last accessed            |
|   - mtime:       last modified            |
|   - ctime:       last status change       |
| Link count:      1                        |
| Pointers to data blocks:                  |
|   Block 0 -> disk sector 48392           |
|   Block 1 -> disk sector 48393           |
|   Block 2 -> disk sector 48400           |
+------------------------------------------+

Notice what is not in the inode: the filename. The filename lives in the directory that contains the file. A directory is essentially a table mapping names to inode numbers.

Directory: /home/alice/
+-------------------+--------+
| Filename          | Inode  |
+-------------------+--------+
| .                 | 23001  |
| ..                | 2      |
| notes.txt         | 23042  |
| report.pdf        | 23108  |
| scripts/          | 23200  |
+-------------------+--------+

The stat Command

stat shows you everything the inode contains:

stat /etc/hostname

  File: /etc/hostname
  Size: 7             Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d    Inode: 131073      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-01-15 10:00:00.000000000 +0000
Modify: 2025-01-10 08:30:00.000000000 +0000
Change: 2025-01-10 08:30:00.000000000 +0000
 Birth: 2025-01-10 08:30:00.000000000 +0000

Key fields:

• Inode: The inode number (131073 in this example)
• Links: The hard link count (how many directory entries point to this inode)
• Access/Modify/Change: The three timestamps
  • atime: When the file was last read
  • mtime: When the file's contents were last modified
  • ctime: When the inode's metadata was last changed (permissions, owner, link count)

# See inode numbers alongside filenames
ls -i /etc/ | head -10

# Check inode usage (you can run out of inodes!)
df -i

Think About It: Can you run out of inodes even if you have plenty of disk space? Yes! If you create millions of tiny files (as some mail servers or cache systems do), you might exhaust inodes before disk space. On ext4, the inode count is set at filesystem creation time with mkfs.ext4 -N.

Hard Links

A hard link is an additional directory entry that points to the same inode as an existing file. The hard link and the original file are completely indistinguishable -- they are both equally "real" names for the same data.

# Create a file
echo "I am the original" > /tmp/original.txt
stat /tmp/original.txt | grep -E "Inode|Links"
# Inode: 12345   Links: 1

# Create a hard link
ln /tmp/original.txt /tmp/hardlink.txt
stat /tmp/original.txt | grep -E "Inode|Links"
# Inode: 12345   Links: 2    <-- Link count increased!

stat /tmp/hardlink.txt | grep -E "Inode|Links"
# Inode: 12345   Links: 2    <-- Same inode!

# Both names see the same content
cat /tmp/original.txt
cat /tmp/hardlink.txt

# Modify through one name, see it through the other
echo "modified!" >> /tmp/hardlink.txt
cat /tmp/original.txt
# I am the original
# modified!

What Happens When You Delete a Hard Link?

Deleting a file (with rm) actually removes a directory entry and decrements the inode's link count. The data is only freed when the link count reaches zero AND no processes have the file open.

# Check link count
stat /tmp/original.txt | grep Links
# Links: 2

# Delete the "original" -- only removes one name
rm /tmp/original.txt

# The hard link still works! Data is intact.
cat /tmp/hardlink.txt
# I am the original
# modified!

stat /tmp/hardlink.txt | grep Links
# Links: 1    <-- Count decreased, but > 0, so data survives

  BEFORE rm:                    AFTER rm:

  Directory entries:            Directory entries:
  "original.txt" --+           (deleted)
                    |---> Inode 12345    "hardlink.txt" ---> Inode 12345
  "hardlink.txt" --+           Links: 1
                   Links: 2             Data blocks: still allocated
                   Data blocks: allocated

Hard Link Restrictions

1. Cannot cross filesystems. A hard link must be on the same filesystem as the target because inode numbers are only unique within a filesystem.

2. Cannot link to directories (for regular users). This prevents circular references in the directory tree. Only the filesystem itself creates hard links to directories (. and ..).

# This will fail:
ln /tmp/somefile /home/somefile    # FAILS if /tmp and /home are on different filesystems

# This will also fail:
ln /tmp/mydir /tmp/mydir-link      # FAILS: can't hard link directories
# ln: /tmp/mydir: hard link not allowed for directory

Think About It: Why would circular directory hard links be catastrophic? Think about what happens to find, du, or any tool that walks the directory tree recursively.

Symbolic (Soft) Links

A symbolic link (symlink) is a special file that contains a text path pointing to another file. It is like a shortcut or alias. Unlike hard links, symlinks:

• Have their own inode
• Can cross filesystem boundaries
• Can point to directories
• Can point to files that do not exist (dangling link)

# Create a symlink
echo "target file content" > /tmp/target.txt
ln -s /tmp/target.txt /tmp/symlink.txt

# Inspect
ls -l /tmp/symlink.txt
# lrwxrwxrwx 1 user user 15 ... /tmp/symlink.txt -> /tmp/target.txt
#  ^                                                   ^^^^^^^^^^^^^^^^
#  'l' = link type                                     target path stored in the link

# Different inodes
ls -li /tmp/target.txt /tmp/symlink.txt
# 12345 -rw-r--r-- 1 user user 20 ... /tmp/target.txt
# 12346 lrwxrwxrwx 1 user user 15 ... /tmp/symlink.txt -> /tmp/target.txt

# Reading through the symlink gives you the target's content
cat /tmp/symlink.txt
# target file content

Dangling Symlinks

# Delete the target
rm /tmp/target.txt

# The symlink still exists but points nowhere
ls -l /tmp/symlink.txt
# lrwxrwxrwx 1 user user 15 ... /tmp/symlink.txt -> /tmp/target.txt

cat /tmp/symlink.txt
# cat: /tmp/symlink.txt: No such file or directory

# Find dangling symlinks
find /tmp -xtype l 2>/dev/null

Symlinks to Directories

# Symlinks can point to directories (unlike hard links)
ln -s /var/log /tmp/logs
ls /tmp/logs/
# Shows contents of /var/log/

Real-World Symlink Uses

Symlinks are everywhere in a modern Linux system:

# UsrMerge: /bin -> /usr/bin
ls -l /bin

# Alternatives system (Debian/Ubuntu)
ls -l /usr/bin/python3
ls -l /etc/alternatives/editor

# Library versioning
ls -l /usr/lib/x86_64-linux-gnu/libssl* 2>/dev/null || ls -l /usr/lib64/libssl* 2>/dev/null
# libssl.so -> libssl.so.3
# libssl.so.3 -> libssl.so.3.0.0

Hard Links vs Soft Links: The Complete Comparison

+---------------------------------------------------------------+
|              Hard Link vs Symbolic Link                       |
+---------------------------------------------------------------+
|                                                                 |
|  Feature              Hard Link        Symbolic Link           |
|  ---------------------------------------------------           |
|  Same inode as target  Yes              No (own inode)         |
|  Cross filesystems     No               Yes                    |
|  Link to directories   No*              Yes                    |
|  Survives target       Yes (data stays) No (dangling link)    |
|    deletion                                                     |
|  File type in ls -l    Same as target   'l' (link)            |
|  Size                  Same as target   Length of path string  |
|  Relative paths        N/A              Relative to link loc  |
|  Performance           Direct (fast)    Extra lookup (tiny)    |
|                                                                 |
|  * root can hard-link directories on some FS, but shouldn't   |
+---------------------------------------------------------------+

Hands-On: Seeing the Difference

mkdir -p /tmp/linklab && cd /tmp/linklab

# Create original file
echo "I am the data" > original.txt

# Create both types of links
ln original.txt hard.txt
ln -s original.txt soft.txt

# Compare inodes
ls -li original.txt hard.txt soft.txt
# original.txt and hard.txt: SAME inode number
# soft.txt: DIFFERENT inode number

# Compare with stat
stat original.txt | grep -E "Inode|Links|Size"
stat hard.txt | grep -E "Inode|Links|Size"
stat soft.txt | grep -E "Inode|Links|Size"

# Delete the original
rm original.txt

# Hard link survives
cat hard.txt
# I am the data

# Soft link is broken
cat soft.txt
# cat: soft.txt: No such file or directory

# The file type
file hard.txt
# hard.txt: ASCII text
file soft.txt
# soft.txt: broken symbolic link to original.txt

The Virtual Filesystem (VFS) Layer

Here is one of the most elegant ideas in Linux kernel design. Linux supports dozens of different filesystems: ext4, XFS, Btrfs, FAT32, NTFS, NFS, procfs, sysfs, tmpfs, and many more. Yet when you type cat /proc/cpuinfo, you use the same cat command you would use on a regular file. When you write to /sys/class/leds/led0/brightness, you use the same echo command. How?

The Virtual Filesystem Switch (VFS) is an abstraction layer in the kernel that provides a uniform interface for all filesystems. User programs never talk to a specific filesystem -- they make VFS system calls (open, read, write, close, stat), and the VFS routes each call to the appropriate filesystem driver.

  +----------------------------------------------+
  |          User Space Applications              |
  |   (cat, ls, cp, vim, python, nginx...)       |
  +----------------------------------------------+
              |  system calls: open, read, write, stat
              v
  +----------------------------------------------+
  |        VFS (Virtual Filesystem Switch)        |
  |  Uniform interface: inodes, dentries, files   |
  +----------------------------------------------+
      |          |          |          |          |
      v          v          v          v          v
  +------+  +------+  +------+  +------+  +------+
  | ext4 |  | XFS  |  | proc |  | sys  |  | NFS  |
  +------+  +------+  +------+  +------+  +------+
      |          |          |          |          |
      v          v          v          v          v
  [disk]    [disk]    [kernel]  [kernel]  [network]

The VFS defines a set of data structures that every filesystem must implement:

• superblock: Metadata about the entire filesystem (size, block size, state)
• inode: Metadata about a single file
• dentry: A directory entry (maps a name to an inode)
• file: An open file (tracks current position, open flags, etc.)

Each filesystem driver provides its own implementations of operations like "read inode from disk" or "create a new file." The VFS calls the right one.

# You can see all mounted filesystem types
mount | awk '{print $5}' | sort -u

# Or more cleanly
cat /proc/filesystems
# nodev  sysfs
# nodev  tmpfs
# nodev  proc
#        ext4
#        xfs
# nodev  devtmpfs
# ...

# "nodev" means the filesystem doesn't use a block device

Think About It: Why is the VFS architecture so powerful? Think about what it means for userspace programs. A program that works with files does not need to know or care whether it is reading from a local ext4 disk, a network NFS share, or a virtual /proc file. The VFS makes them all look the same.

/proc -- The Process Filesystem

/proc is a virtual filesystem -- none of its files exist on disk. The kernel generates their contents on the fly when you read them. It serves two purposes: exposing process information and providing kernel tuning parameters.

Process Information

Every running process has a directory in /proc named by its PID:

# Your current shell's PID
echo $$

# Explore your own process
ls /proc/$$/

# Key files in a process directory
cat /proc/$$/cmdline | tr '\0' ' '; echo   # Command line
cat /proc/$$/status | head -10              # Status info
cat /proc/$$/environ | tr '\0' '\n' | head -5  # Environment vars
ls -l /proc/$$/fd/                          # Open file descriptors
cat /proc/$$/maps | head -10                # Memory mappings
readlink /proc/$$/exe                       # Path to executable
readlink /proc/$$/cwd                       # Current working directory

System Information

# CPU information
cat /proc/cpuinfo | grep "model name" | uniq

# Memory
cat /proc/meminfo | head -5

# Kernel version
cat /proc/version

# Uptime (in seconds)
cat /proc/uptime

# Load averages
cat /proc/loadavg

# Mounted filesystems (from kernel's perspective)
cat /proc/mounts | head -10

# Command line the kernel was booted with
cat /proc/cmdline

Kernel Tuning via /proc/sys/

The /proc/sys/ directory contains files that map to kernel parameters. You can read and write them to tune system behavior at runtime.

# IP forwarding (routing)
cat /proc/sys/net/ipv4/ip_forward

# Maximum number of open files
cat /proc/sys/fs/file-max

# Maximum number of PIDs
cat /proc/sys/kernel/pid_max

# Hostname
cat /proc/sys/kernel/hostname

# Change a parameter at runtime (requires root)
sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'

# The sysctl command provides a friendlier interface
sysctl net.ipv4.ip_forward
sudo sysctl -w net.ipv4.ip_forward=1

/sys -- The Sysfs Filesystem

/sys (sysfs) exports the kernel's view of devices, drivers, buses, and other kernel objects as a filesystem hierarchy. Introduced in Linux 2.6, it replaced much of the hardware information that used to live in /proc.

# Block devices and their attributes
ls /sys/block/
cat /sys/block/sda/size 2>/dev/null    # Size in 512-byte sectors
cat /sys/block/sda/queue/scheduler 2>/dev/null  # I/O scheduler

# Network interfaces
ls /sys/class/net/
cat /sys/class/net/eth0/address 2>/dev/null   # MAC address
cat /sys/class/net/eth0/mtu 2>/dev/null       # MTU
cat /sys/class/net/eth0/operstate 2>/dev/null # up/down

# Power management
ls /sys/power/ 2>/dev/null
cat /sys/power/state 2>/dev/null

# CPU information
ls /sys/devices/system/cpu/
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null

# Modules (loaded kernel modules)
ls /sys/module/ | head -10

Writing to /sys

Some /sys files are writable, allowing you to change hardware and driver settings:

# Example: Change I/O scheduler for a disk
cat /sys/block/sda/queue/scheduler 2>/dev/null
# [mq-deadline] none
# sudo sh -c 'echo "none" > /sys/block/sda/queue/scheduler'

# Example: Control CPU frequency governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null

Device Files in /dev

As discussed in Chapter 5, /dev contains device files. Now that you understand inodes, let us look at how device files actually work.

Device files have a major number (identifies the driver) and a minor number (identifies the specific device within that driver):

ls -l /dev/sda /dev/null /dev/tty /dev/zero 2>/dev/null
# brw-rw---- 1 root disk    8,  0 ... /dev/sda   (block, major 8, minor 0)
# crw-rw-rw- 1 root root    1,  3 ... /dev/null  (char, major 1, minor 3)
# crw-rw-rw- 1 root tty     5,  0 ... /dev/tty   (char, major 5, minor 0)
# crw-rw-rw- 1 root root    1,  5 ... /dev/zero  (char, major 1, minor 5)

Block Devices vs Character Devices

Block Devices (b):
  - Accessed in blocks (typically 512 bytes or 4096 bytes)
  - Support random access (seek to any position)
  - Examples: hard drives, SSDs, USB drives
  - /dev/sda, /dev/nvme0n1, /dev/loop0

Character Devices (c):
  - Accessed one character (byte) at a time
  - Usually sequential access (some support seek)
  - Examples: terminals, serial ports, random number generators
  - /dev/tty, /dev/null, /dev/urandom

Special Device Files

# /dev/null -- The black hole. Discard any output.
echo "this goes nowhere" > /dev/null

# /dev/zero -- Infinite stream of zero bytes.
head -c 1024 /dev/zero | xxd | head

# /dev/urandom -- Infinite stream of random bytes.
head -c 32 /dev/urandom | xxd

# /dev/full -- Always reports "disk full" on write.
echo "test" > /dev/full
# bash: echo: write error: No space left on device

# /dev/random -- Random bytes (may block if entropy pool is empty, on older kernels)
head -c 32 /dev/random | xxd

The udev System

Modern Linux uses udev (managed by systemd as systemd-udevd) to dynamically create and manage device files. When you plug in a USB drive, udev detects the hardware event, creates the appropriate /dev entry, and can run rules to set permissions, create symlinks, or trigger scripts.

# View udev rules
ls /etc/udev/rules.d/
ls /usr/lib/udev/rules.d/ | head -20

# Monitor device events in real time (plug/unplug something)
sudo udevadm monitor --property
# (Press Ctrl+C to stop)

# Get device info
sudo udevadm info /dev/sda 2>/dev/null | head -20

Hands-On: Inode and Link Deep Dive

Exercise 1: Watching the Link Count

mkdir -p /tmp/inode-lab && cd /tmp/inode-lab

# Create a file and check its link count
echo "test data" > file.txt
stat file.txt | grep Links
# Links: 1

# Create hard links
ln file.txt link1.txt
ln file.txt link2.txt
ln file.txt link3.txt
stat file.txt | grep Links
# Links: 4

# All share the same inode
ls -li file.txt link1.txt link2.txt link3.txt
# All show the same inode number

# Delete some links
rm link1.txt link2.txt
stat file.txt | grep Links
# Links: 2

rm link3.txt
stat file.txt | grep Links
# Links: 1  (only the original name remains)

Exercise 2: Why Deleted Files Still Use Space

# Create a large file
dd if=/dev/zero of=/tmp/bigfile bs=1M count=100

# Check space
df -h /tmp

# Open the file in the background (keep a file descriptor open)
sleep 3600 < /tmp/bigfile &
BG_PID=$!

# Delete the file
rm /tmp/bigfile

# Check: ls shows it's gone
ls /tmp/bigfile 2>&1
# No such file or directory

# But space is NOT freed!
df -h /tmp
# (same usage as before)

# Find deleted-but-open files
sudo lsof +L1 2>/dev/null | grep bigfile

# The inode still exists because the process holds a file descriptor
# The link count in the directory is 0, but the kernel count includes open FDs

# Kill the background process
kill $BG_PID

# NOW the space is freed
df -h /tmp

This is exactly the mystery from our opening scenario. Now you know how to diagnose it.

Exercise 3: Directory Link Counts

# A directory's link count has a special meaning
mkdir -p /tmp/dir-links/sub1/sub2

# Check the link count of /tmp/dir-links
stat /tmp/dir-links | grep Links
# Links: 3

# Why 3?
#   1. The entry "dir-links" in /tmp
#   2. The "." entry inside /tmp/dir-links itself
#   3. The ".." entry inside /tmp/dir-links/sub1
#
# Formula: link_count = 2 + number_of_immediate_subdirectories

ls -la /tmp/dir-links
# drwxr-xr-x 3 user user ...  .      <-- "." is a hard link to itself
# drwxrwxrwt ... ...           ..     <-- ".." links to parent

# Add another subdirectory
mkdir /tmp/dir-links/sub3
stat /tmp/dir-links | grep Links
# Links: 4  (2 + 2 subdirectories)

Debug This

A user reports: "I created a symbolic link to /opt/app/config.yaml, but the application says the file does not exist, even though I can see it with ls -l."

You check:

$ ls -l /home/alice/config.yaml
lrwxrwxrwx 1 alice alice 25 Jan 15 10:00 /home/alice/config.yaml -> ../opt/app/config.yaml

What is wrong?

rm /home/alice/config.yaml
ln -s /opt/app/config.yaml /home/alice/config.yaml

ln -s ../../opt/app/config.yaml /home/alice/config.yaml

What Just Happened?

+------------------------------------------------------------------+
|                    CHAPTER 8 RECAP                                |
+------------------------------------------------------------------+
|                                                                    |
|  Inodes: The hidden data structure behind every file              |
|    - Stores metadata: permissions, timestamps, data block ptrs   |
|    - Does NOT store the filename (that's in the directory)        |
|    - Use stat to inspect, ls -i for inode numbers, df -i count   |
|                                                                    |
|  Hard Links: Additional name -> same inode                        |
|    - Same inode number, shared data                               |
|    - Survives "deletion" of other names (link count > 0)         |
|    - Cannot cross filesystems, cannot link to directories         |
|                                                                    |
|  Soft (Symbolic) Links: Separate file containing a path          |
|    - Different inode, stores target path as content               |
|    - Can cross filesystems, can link to directories               |
|    - Can become dangling (broken) if target is deleted            |
|                                                                    |
|  VFS: The kernel abstraction making all filesystems uniform       |
|    - ext4, XFS, proc, sys, NFS all behind one interface          |
|    - Userspace uses same syscalls for all filesystem types        |
|                                                                    |
|  /proc: Virtual FS for process info and kernel tuning             |
|  /sys:  Virtual FS for device/driver/bus information              |
|  /dev:  Device files (block, character) managed by udev           |
|                                                                    |
|  Tools: stat, ls -i, ln, ln -s, readlink, lsof +L1, df -i       |
|                                                                    |
+------------------------------------------------------------------+

Try This

Exercises

1. Inode investigation. Create a file, then create 5 hard links to it. Use stat to verify the link count at each step. Delete the links one by one and confirm the count decreases. At what point is the data actually freed?

2. Symlink maze. Create a chain of symlinks: a -> b -> c -> d -> real_file. Does reading a work? Now delete c. What happens when you read a? What error message do you get?

3. The /proc explorer. Write a script that takes a PID as an argument and prints:

   • The command that started the process
   • Its current working directory
   • The number of open file descriptors
   • Its memory usage (from /proc/<PID>/status)

4. Deleted-but-open files. Create a 50MB file, open it with tail -f in the background, delete the file, and verify with df that the space is not freed. Then kill tail and verify the space is freed. Use lsof +L1 to find the deleted-but-open file before killing the process.

5. Directory link count formula. Verify that a directory's link count equals 2 + (number of immediate subdirectories). Create a directory with 0, 1, 3, and 5 subdirectories, checking the link count each time.

Bonus Challenge

Write a script called link-analyzer.sh that takes a filename as an argument and reports:

• Whether it is a regular file, symlink, directory, device, etc.
• Its inode number
• Its link count
• If it is a symlink, the target path (and whether the target exists)
• If it is a regular file with link count > 1, find all other hard links to the same inode

#!/bin/bash
# link-analyzer.sh -- Analyze links and inodes for a given file
FILE="${1:?Usage: $0 <filename>}"

if [ ! -e "$FILE" ] && [ ! -L "$FILE" ]; then
    echo "Error: '$FILE' does not exist"
    exit 1
fi

echo "=== Link Analysis: $FILE ==="
echo ""

# File type
TYPE=$(stat -c %F "$FILE" 2>/dev/null || stat -f %HT "$FILE" 2>/dev/null)
echo "Type:       $TYPE"

# Inode number
INODE=$(stat -c %i "$FILE" 2>/dev/null)
echo "Inode:      $INODE"

# Link count
LINKS=$(stat -c %h "$FILE" 2>/dev/null)
echo "Link count: $LINKS"

# If symlink, show target
if [ -L "$FILE" ]; then
    TARGET=$(readlink "$FILE")
    echo "Symlink target: $TARGET"
    if [ -e "$FILE" ]; then
        echo "Target status: EXISTS"
        echo "Resolved path: $(readlink -f "$FILE")"
    else
        echo "Target status: BROKEN (dangling symlink)"
    fi
fi

# If regular file with multiple hard links, find siblings
if [ -f "$FILE" ] && [ "$LINKS" -gt 1 ]; then
    echo ""
    echo "--- Other hard links to inode $INODE ---"
    MOUNT=$(df --output=target "$FILE" 2>/dev/null | tail -1)
    if [ -n "$MOUNT" ]; then
        sudo find "$MOUNT" -inum "$INODE" 2>/dev/null | grep -v "^${FILE}$"
    fi
fi

echo ""
echo "--- Full stat output ---"
stat "$FILE"

This completes Part II of the book. You now understand how Linux organizes files on disk (Chapter 5), how access is controlled (Chapter 6), how physical storage is managed (Chapter 7), and the internal mechanisms of inodes, links, and the VFS (Chapter 8). Next, in Part III, we move to the dynamic side of Linux: users, processes, signals, and inter-process communication.

Users, Groups & Access Control

Why This Matters

Imagine you just set up a shared Linux server for your team. Three developers need to deploy code, two interns need read-only access to logs, and one DBA needs access only to the database directory. Without a solid understanding of users, groups, and access control, you will either lock everyone out or -- worse -- give everyone root access and pray nothing breaks.

Every file, every process, every socket on a Linux system is owned by a user and associated with a group. The entire security model of Linux rests on this foundation. When a web server gets compromised, it is the user/group model that determines whether the attacker can read /etc/shadow or pivot to other services. When a junior admin accidentally runs rm -rf /, it is the permission model that decides how much damage actually happens.

This chapter gives you the complete picture: how Linux identifies users, how groups organize access, how sudo elevates privileges safely, and how PAM ties authentication together behind the scenes.

Try This Right Now

Open a terminal and run these commands to see your own identity:

# Who am I?
whoami

# Full identity -- UID, GID, and all groups
id

# All users on this system (just usernames)
cut -d: -f1 /etc/passwd

# All groups on this system
cut -d: -f1 /etc/group

# Which groups do I belong to?
groups

You will see output like:

$ id
uid=1000(alice) gid=1000(alice) groups=1000(alice),27(sudo),999(docker)

That single line tells you everything the kernel needs to make access decisions about you.

Understanding UIDs and GIDs

Every user on a Linux system is identified by a User ID (UID) -- a number, not a name. The username is just a human-friendly label mapped to that number in /etc/passwd. The kernel does not care about "alice"; it cares about UID 1000.

Similarly, every group has a Group ID (GID).

The UID Landscape

+----------------------------------------------------+
|  UID Range         |  Purpose                       |
|--------------------|--------------------------------|
|  0                 |  root (superuser)              |
|  1 - 999           |  System/service accounts       |
|  1000 - 60000      |  Regular (human) users         |
|  65534             |  nobody (unmapped/overflow)     |
+----------------------------------------------------+

Distro Note: The boundary between system and regular UIDs varies. Debian/Ubuntu typically use 1000+ for regular users. RHEL/Fedora also use 1000+. Older systems may use 500+. Check /etc/login.defs for the UID_MIN and UID_MAX values on your system.

Why Does This Matter?

System accounts (UID 1-999) exist to run services. The www-data user runs your web server, mysql runs your database. These accounts typically have:

• No valid login shell (set to /usr/sbin/nologin or /bin/false)
• No home directory (or a non-standard one)
• No password

This is a deliberate security design: even if someone exploits your web server, they land in a restricted account that cannot log in interactively.

# See the shell assigned to www-data (if it exists)
grep www-data /etc/passwd

# See root's entry
grep ^root /etc/passwd

The Three Identity Files

Linux stores user and group information in three critical files. Let us examine each.

/etc/passwd -- The User Database

Despite its name, this file does NOT contain passwords (not anymore). It is world-readable and contains one line per user:

# Look at your own entry
grep "^$(whoami):" /etc/passwd

The format is seven colon-separated fields:

username:x:UID:GID:comment:home_directory:shell

Example:

alice:x:1000:1000:Alice Johnson:/home/alice:/bin/bash

The x in the password field means "look in /etc/shadow instead." In ancient Unix, the encrypted password was stored right here in this world-readable file. That was a terrible idea.

/etc/shadow -- The Password Vault

This file stores the actual password hashes and is readable only by root:

# This will fail as a regular user
cat /etc/shadow

# This works
sudo cat /etc/shadow

Format:

username:$algorithm$salt$hash:last_changed:min:max:warn:inactive:expire:reserved

Example:

alice:$6$rANd0mSaLt$kQ8Xp...long_hash...:19500:0:99999:7:::

The password hash format tells you the algorithm:

• $1$ -- MD5 (ancient, insecure)
• $5$ -- SHA-256
• $6$ -- SHA-512 (most common today)
• $y$ -- yescrypt (newer, used by Debian 12+, Fedora 35+)

Think About It: Why is the password hash stored in a separate file from /etc/passwd? What would happen if any user on the system could read everyone's password hashes?

/etc/group -- The Group Database

# See all groups
cat /etc/group

# See groups you belong to
grep "$(whoami)" /etc/group

Format:

groupname:x:GID:member1,member2,member3

Example:

sudo:x:27:alice,bob
docker:x:999:alice
developers:x:1001:alice,bob,charlie

A user has one primary group (from /etc/passwd, GID field) and zero or more supplementary groups (listed in /etc/group). When you create a file, it is owned by your primary group by default.

Hands-On: Managing Users

Creating Users

# Create a new user with defaults
sudo useradd testuser1

# Create a user with all the options you typically want
sudo useradd -m -s /bin/bash -c "Test User Two" testuser2

The flags explained:

• -m -- Create home directory
• -s /bin/bash -- Set login shell
• -c "Test User Two" -- Set the GECOS comment field

Distro Note: On Debian/Ubuntu, useradd does NOT create a home directory by default -- you need -m. On RHEL/Fedora, useradd creates a home directory by default. To be safe, always use -m explicitly.

# Verify the user was created
grep testuser2 /etc/passwd
id testuser2
ls -la /home/testuser2/

Setting Passwords

# Set a password for the new user
sudo passwd testuser2

You will be prompted to enter the password twice. The passwd command handles all the hashing and writes to /etc/shadow.

# Check the shadow entry (see that a hash now exists)
sudo grep testuser2 /etc/shadow

Creating System Users

System users are for running services, not for humans to log in:

# Create a system user for a hypothetical app
sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp

# Verify -- note the low UID
id myapp
grep myapp /etc/passwd

Modifying Users

# Change a user's shell
sudo usermod -s /bin/zsh testuser2

# Add a user to a supplementary group (KEEP existing groups with -a)
sudo usermod -aG sudo testuser2

# Change the user's comment
sudo usermod -c "Test Account" testuser2

# Lock an account (disable login without deleting)
sudo usermod -L testuser2

# Unlock an account
sudo usermod -U testuser2

WARNING: Using usermod -G WITHOUT -a will REMOVE the user from all other supplementary groups. Always use -aG to append to existing groups. This is one of the most common and dangerous mistakes in user management.

Deleting Users

# Delete user but keep their home directory
sudo userdel testuser1

# Delete user AND their home directory and mail spool
sudo userdel -r testuser2

WARNING: userdel -r permanently removes the user's home directory. In production, consider locking the account instead and archiving the home directory first.

Hands-On: Managing Groups

# Create a new group
sudo groupadd developers

# Create a group with a specific GID
sudo groupadd -g 2000 ops

# Add existing users to the group
sudo usermod -aG developers alice

# See group membership
getent group developers

# Remove a user from a group (use gpasswd)
sudo gpasswd -d alice developers

# Delete a group
sudo groupdel ops

Primary vs. Supplementary Groups

+--------------------------------------------------+
|  User: alice                                     |
|                                                  |
|  Primary Group: alice (GID 1000)                 |
|    - Assigned in /etc/passwd                     |
|    - Default group for new files                 |
|                                                  |
|  Supplementary Groups:                           |
|    - sudo (GID 27)     -- can use sudo           |
|    - docker (GID 999)  -- can use Docker          |
|    - developers (GID 1001)                       |
+--------------------------------------------------+

When alice creates a file, it is owned by alice:alice (user:primary_group). If she wants new files to be owned by the developers group, she can:

# Temporarily change primary group for this session
newgrp developers

# Now create a file -- it belongs to 'developers' group
touch /tmp/team-file
ls -l /tmp/team-file

Think About It: A developer says "I added myself to the docker group but I still get permission denied." What is the most likely cause? (Hint: group membership changes require a fresh login session.)

su vs. sudo: Elevating Privileges

su -- Switch User

su lets you become another user entirely. You need that user's password:

# Become root (need root's password)
su -

# Become another user
su - alice

# Run a single command as another user
su -c "whoami" alice

The - (or -l) flag is important: it gives you a full login shell with that user's environment. Without it, you keep your current environment, which can cause confusing path and variable issues.

sudo -- Superuser Do

sudo lets you run commands as root (or another user) using YOUR OWN password, if you are authorized:

# Run a single command as root
sudo apt update

# Open a root shell
sudo -i

# Run a command as a different user
sudo -u postgres psql

# See what you are allowed to do
sudo -l

su vs. sudo: When to Use Which

+-----------------------------------------------------------+
|  Feature          |  su                |  sudo              |
|-------------------|--------------------|---------------------|
|  Password needed  |  Target user's     |  Your own           |
|  Granularity      |  All or nothing    |  Per-command control |
|  Audit trail      |  Minimal           |  Full logging        |
|  Best for         |  Switching users   |  Admin tasks         |
+-----------------------------------------------------------+

In modern Linux administration, sudo is preferred almost universally. It provides better auditing (every sudo command is logged), does not require sharing the root password, and allows fine-grained control over what each user can do.

The sudoers File and visudo

The /etc/sudoers file controls who can use sudo and what they can do. NEVER edit it directly with a text editor -- always use visudo:

# Open sudoers for editing (uses your default EDITOR)
sudo visudo

visudo checks syntax before saving. A syntax error in sudoers can lock you out of sudo entirely.

sudoers Syntax

The basic format is:

who    where=(as_whom)    what

Examples:

# alice can run any command as any user on any host
alice   ALL=(ALL:ALL)   ALL

# bob can only restart nginx
bob     ALL=(root)      /usr/bin/systemctl restart nginx

# The %sudo group can run anything (the % means "group")
%sudo   ALL=(ALL:ALL)   ALL

# ops group can run any command without a password
%ops    ALL=(ALL)       NOPASSWD: ALL

# charlie can run specific commands without a password
charlie ALL=(root)      NOPASSWD: /usr/bin/systemctl restart myapp, /usr/bin/journalctl -u myapp

Drop-in Files

Rather than editing the main sudoers file, you can create files in /etc/sudoers.d/:

# Create a drop-in file for a team
sudo visudo -f /etc/sudoers.d/developers

Add content like:

%developers ALL=(root) /usr/bin/systemctl restart myapp, /usr/bin/tail -f /var/log/myapp/*

# Make sure the include directive exists in sudoers
sudo grep includedir /etc/sudoers
# You should see: @includedir /etc/sudoers.d

Distro Note: Debian/Ubuntu ship with the @includedir /etc/sudoers.d directive enabled by default. On RHEL/CentOS, it is also present but uses the older #includedir syntax (the # is NOT a comment here -- this is one of the strangest syntax decisions in Unix history).

Debug This: sudo Permission Problems

A user reports they cannot use sudo despite being in the sudo group:

# The user runs:
sudo whoami
# Output: alice is not in the sudoers file. This incident will be reported.

Diagnosis steps:

# 1. Check if the user is actually in the sudo group
id alice
# Look for "sudo" or "wheel" in the groups list

# 2. Did they log out and back in after being added?
# Group changes require a new session!

# 3. Check sudoers configuration
sudo visudo
# Look for: %sudo ALL=(ALL:ALL) ALL
# On RHEL: %wheel ALL=(ALL) ALL

# 4. Check for syntax errors in drop-in files
sudo visudo -c
# This checks ALL sudoers files for syntax errors

# 5. Check the log for details
sudo grep sudo /var/log/auth.log        # Debian/Ubuntu
sudo grep sudo /var/log/secure           # RHEL/Fedora

Distro Note: The admin group is called sudo on Debian/Ubuntu and wheel on RHEL/Fedora/Arch. The name goes back to the 1970s: "big wheel" was slang for someone with power.

PAM: Pluggable Authentication Modules

PAM is the framework that sits behind every authentication event on Linux. When you type your password at the login screen, when you su to another user, when you sudo -- PAM is handling it.

How PAM Works

+-----------------+     +--------+     +------------------+
|  Application    |---->|  PAM   |---->|  PAM Modules     |
|  (login, sudo,  |     |  Core  |     |  pam_unix.so     |
|   ssh, su)      |     |        |     |  pam_ldap.so     |
+-----------------+     +--------+     |  pam_google...so  |
                                       |  pam_limits.so   |
                                       +------------------+

Applications do not handle passwords themselves. They ask PAM, and PAM consults a chain of modules. This means you can change how authentication works (add two-factor, switch to LDAP) without modifying any application.

PAM Configuration

PAM config files live in /etc/pam.d/, one per service:

# See all PAM-aware services
ls /etc/pam.d/

# Look at the sudo PAM config
cat /etc/pam.d/sudo

A PAM config line has four fields:

type    control    module    [arguments]

• type: auth (verify identity), account (check access), password (change password), session (setup/teardown)
• control: required, requisite, sufficient, optional
• module: The shared library (e.g., pam_unix.so)

Example (/etc/pam.d/sudo):

auth    required    pam_unix.so
account required    pam_unix.so
session required    pam_limits.so

You do not need to master PAM internals right now, but understanding that it exists and how it is structured will help immensely when you need to configure LDAP authentication, set up two-factor auth, or debug "why can't this user log in?"

Hands-On: Practical Scenarios

Scenario 1: Onboarding a New Developer

# Create the user
sudo useradd -m -s /bin/bash -c "Dave Chen" dchen

# Set an initial password
sudo passwd dchen

# Add to the developers group
sudo usermod -aG developers dchen

# Give limited sudo access
sudo visudo -f /etc/sudoers.d/developers
# Add: %developers ALL=(root) /usr/bin/systemctl restart myapp

# Verify everything
id dchen
sudo -l -U dchen

Scenario 2: Offboarding an Employee

# Lock the account immediately
sudo usermod -L jsmith

# Kill any running processes
sudo pkill -u jsmith

# Expire the account
sudo usermod --expiredate 1 jsmith

# Archive home directory before deletion
sudo tar czf /backups/jsmith-home-$(date +%Y%m%d).tar.gz /home/jsmith

# Remove the user and home directory
sudo userdel -r jsmith

# Check for orphaned files anywhere on the system
sudo find / -nouser -o -nogroup 2>/dev/null

Scenario 3: Restricting a Contractor to a Specific Directory

# Create the user with a restricted shell
sudo useradd -m -s /bin/rbash contractor1
sudo passwd contractor1

# Create a limited PATH
sudo mkdir /home/contractor1/bin
sudo ln -s /usr/bin/ls /home/contractor1/bin/
sudo ln -s /usr/bin/cat /home/contractor1/bin/

# Set the restricted PATH in their .bashrc
echo 'export PATH=$HOME/bin' | sudo tee /home/contractor1/.bashrc
sudo chown contractor1:contractor1 /home/contractor1/.bashrc
sudo chmod 444 /home/contractor1/.bashrc

Password Policies and Account Aging

# See password aging info for a user
sudo chage -l alice

# Force password change on next login
sudo chage -d 0 alice

# Set password to expire in 90 days
sudo chage -M 90 alice

# Set minimum 7 days between password changes
sudo chage -m 7 alice

# Set account to expire on a specific date
sudo chage -E 2025-12-31 contractor1

You can set system-wide defaults in /etc/login.defs:

grep -E "^(PASS_MAX_DAYS|PASS_MIN_DAYS|PASS_WARN_AGE|UID_MIN|UID_MAX)" /etc/login.defs

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 9 Recap: Users, Groups & Access Control                 |
|------------------------------------------------------------------|
|                                                                  |
|  - Every user has a UID; every group has a GID.                  |
|  - UIDs 0-999 are system accounts; 1000+ are regular users.      |
|  - /etc/passwd stores user info (not passwords!).                |
|  - /etc/shadow stores password hashes (root-readable only).      |
|  - /etc/group maps group names to GIDs and members.              |
|  - useradd, usermod, userdel manage users.                       |
|  - groupadd, gpasswd manage groups.                              |
|  - sudo is preferred over su for administrative tasks.           |
|  - sudoers controls who can sudo; always edit with visudo.       |
|  - PAM handles all authentication behind the scenes.             |
|  - Group changes require logging out and back in.                |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: User Audit

Run awk -F: '$3 >= 1000 && $3 < 65534 {print $1, $3, $7}' /etc/passwd to list all regular users, their UIDs, and shells. Are there any with /bin/false or /usr/sbin/nologin? Why might a regular-range UID have a non-login shell?

Exercise 2: Group Design

Design a group structure for a team of 5 developers, 2 QA engineers, and 1 DBA. What groups would you create? Which sudo permissions would each group need? Write the sudoers rules.

Exercise 3: Shadow File Analysis

Run sudo awk -F: '{print $1, $3, $5}' /etc/shadow to see usernames, last-changed dates, and max days. Calculate the actual date each password was last changed (hint: the number is days since January 1, 1970 -- use date -d "1970-01-01 + N days").

Exercise 4: Lock and Investigate

Create a test user, set a password, lock the account with usermod -L, then examine /etc/shadow. What character was added to the password hash? Unlock it and check again.

Bonus Challenge

Write a bash script that takes a username as an argument and outputs a full "identity report": username, UID, GID, primary group name, all supplementary groups, home directory, shell, password status (locked/unlocked/no password), last password change, and account expiry date. Use only standard commands (id, passwd, chage, grep, awk).

Processes & Job Control

Why This Matters

You SSH into a production server and the monitoring alert says CPU usage is at 98%. Something is eating all your compute. Or maybe you started a long database migration over SSH and your connection dropped -- is the migration still running? Did it die? How do you reconnect to it?

Every command you type, every service running in the background, every daemon handling network requests -- they are all processes. Understanding how processes work, how to inspect them, how to control them, and how to keep them running when you walk away from the terminal is fundamental to Linux administration.

This chapter takes you from "what is a process?" to confidently managing foreground and background jobs, diagnosing runaway processes, and understanding the process lifecycle from birth to death.

Try This Right Now

# How many processes are running on your system right now?
ps aux | wc -l

# What is YOUR current shell's process ID?
echo $$

# What is your shell's parent process?
echo $PPID

# See the process tree rooted at PID 1
pstree -p 1 | head -30

# What is consuming the most CPU right now?
ps aux --sort=-%cpu | head -5

You will see that even a "quiet" Linux system has dozens or hundreds of processes running. Every one of them has a story.

What Is a Process?

A process is a running instance of a program. The program /usr/bin/bash is a file sitting on disk. When you launch a terminal, the kernel loads that program into memory, assigns it resources, and creates a process. If you open three terminals, you have three separate bash processes, each with its own memory, its own variables, its own PID.

Key Attributes of Every Process

+------------------------------------------+
|  Process Attributes                      |
|------------------------------------------|
|  PID    - Process ID (unique number)     |
|  PPID   - Parent Process ID             |
|  UID    - User who owns it              |
|  State  - Running, Sleeping, etc.       |
|  Nice   - Priority value                |
|  Memory - How much RAM it uses          |
|  CPU    - How much CPU time it uses     |
|  TTY    - Terminal it is attached to    |
|  CMD    - The command that started it   |
+------------------------------------------+

How Processes Are Born: fork() and exec()

When you type ls in your shell, here is what actually happens:

  Your bash shell (PID 1234)
       |
       | 1. fork() -- creates a COPY of itself
       |
       v
  Child bash (PID 5678)     <-- exact clone of parent
       |
       | 2. exec() -- replaces itself with the 'ls' program
       |
       v
  ls process (PID 5678)     <-- now running ls code
       |
       | 3. ls runs, outputs results
       |
       | 4. exit() -- process terminates
       |
       v
  Parent bash (PID 1234) collects exit status
  (the child is now gone)

This fork-then-exec pattern is how nearly every process on Linux is created. The very first process, PID 1 (init or systemd), is started by the kernel. Every other process is a descendant of PID 1.

# See the full process tree to verify this
pstree -p

Think About It: If every process is created by forking, and PID 1 is the ancestor of all, what happens to a process if its parent dies before it does? (We will answer this when we discuss zombie and orphan processes.)

Inspecting Processes with ps

ps is the workhorse command for viewing processes. It has two major syntax styles (because Unix history is messy):

BSD Style (No Dashes)

# The "classic" incantation -- show ALL processes with details
ps aux

Output columns:

USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1   0.0  0.1 169516 13200 ?        Ss   Feb10   0:07 /sbin/init
alice     1234   0.0  0.0  10056  3456 pts/0    Ss   09:15   0:00 -bash
alice     5678  45.2  2.1 987654 56789 pts/0    R+   09:30   5:12 python train.py

System V Style (With Dashes)

# Full-format listing
ps -ef

# Full-format with thread info
ps -eLf

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Feb10 ?        00:00:07 /sbin/init
alice     1234  1100  0 09:15 pts/0    00:00:00 -bash

Useful ps Recipes

# Find a specific process
ps aux | grep nginx

# Better: use pgrep (no grep-in-grep problem)
pgrep -a nginx

# Show process tree
ps auxf

# Show specific columns
ps -eo pid,ppid,user,%cpu,%mem,stat,cmd --sort=-%mem | head -20

# Show all processes by a specific user
ps -u alice

# Show process with specific PID
ps -p 1234 -o pid,ppid,stat,cmd

Process States

Every process is in one of several states. The STAT column in ps shows this:

+-------+------------------+------------------------------------------+
| Code  | State            | Meaning                                  |
+-------+------------------+------------------------------------------+
|  R    | Running          | Actively using CPU or ready to run        |
|  S    | Sleeping         | Waiting for an event (I/O, signal, etc.) |
|  D    | Uninterruptible  | Waiting for I/O (cannot be killed!)      |
|       | Sleep            |                                          |
|  Z    | Zombie           | Finished but parent hasn't collected      |
|       |                  | its exit status                          |
|  T    | Stopped          | Suspended (e.g., Ctrl+Z)                 |
|  t    | Traced           | Stopped by a debugger                    |
|  I    | Idle             | Kernel thread, idle                      |
+-------+------------------+------------------------------------------+

Additional modifiers appear after the main state:

+-------+----------------------------------------------+
| Mod   | Meaning                                      |
+-------+----------------------------------------------+
|  s    | Session leader                               |
|  +    | In the foreground process group               |
|  l    | Multi-threaded                                |
|  <    | High priority (negative nice value)           |
|  N    | Low priority (positive nice value)            |
+-------+----------------------------------------------+

So Ss means "sleeping, session leader." R+ means "running, foreground." Ssl means "sleeping, session leader, multi-threaded."

The Dangerous D State

A process in state D (uninterruptible sleep) cannot be killed, not even with SIGKILL. It is waiting for I/O to complete -- usually disk or NFS. If you see many processes stuck in D state, you likely have a storage problem (dead NFS mount, failing disk, etc.).

# Find processes in D state
ps aux | awk '$8 ~ /D/'

Zombie Processes

A zombie (Z) is not actually running. It is a dead process whose entry still sits in the process table because its parent has not called wait() to collect its exit status. Zombies consume almost no resources (just a PID table entry), but a large number of them indicates a buggy parent process.

# Find zombie processes
ps aux | awk '$8 == "Z"'

# Or
ps -eo pid,ppid,stat,cmd | grep Z

The fix for zombies is to fix or restart their parent process. Killing a zombie does nothing -- it is already dead.

Real-Time Monitoring with top and htop

top

# Launch top
top

Key top commands while running:

htop (Better Interactive Viewer)

# Install if not present
sudo apt install htop    # Debian/Ubuntu
sudo dnf install htop    # RHEL/Fedora

# Launch
htop

htop provides:

• Color-coded CPU/memory bars
• Mouse support
• Horizontal scrolling for long commands
• Tree view (press F5)
• Easy filtering (press F4)
• Easy process killing (press F9)

Distro Note: htop is not installed by default on most distributions. It is available in the standard repositories for all major distros. Install it -- you will use it constantly.

Exploring /proc/PID/

Every running process has a directory under /proc/ named after its PID. This is a virtual filesystem -- the kernel generates the contents on the fly.

# Explore your own shell's process info
ls /proc/$$/

# What command started this process?
cat /proc/$$/cmdline | tr '\0' ' '; echo

# What is its current working directory?
ls -l /proc/$$/cwd

# What is its executable?
ls -l /proc/$$/exe

# What environment variables does it have?
cat /proc/$$/environ | tr '\0' '\n' | head -10

# What files does it have open?
ls -l /proc/$$/fd/

# Its memory map
cat /proc/$$/maps | head -10

# Process status summary
cat /proc/$$/status | head -20

The status file is especially useful:

cat /proc/$$/status

Key fields:

• Name: -- process name
• State: -- current state
• Pid: / PPid: -- PID and parent PID
• Uid: / Gid: -- real, effective, saved, filesystem UIDs
• VmRSS: -- resident memory size
• Threads: -- number of threads
• voluntary_ctxt_switches: -- context switches

Think About It: The /proc/$$/fd/ directory shows all open file descriptors. File descriptor 0 is stdin, 1 is stdout, 2 is stderr. What do the symlinks point to when you are working in a terminal? What would they point to for a daemon process?

Foreground and Background Jobs

Running Commands in the Background

Add & to run a command in the background:

# Start a long-running command in the background
sleep 300 &
# Output: [1] 12345
# [1] is the job number, 12345 is the PID

Managing Jobs

# List all background jobs in this shell
jobs
# Output: [1]+  Running    sleep 300 &

# Bring a background job to the foreground
fg %1

# Suspend a foreground job (Ctrl+Z)
# Then see it in jobs list
jobs
# Output: [1]+  Stopped    sleep 300

# Resume it in the background
bg %1

# Resume it in the foreground
fg %1

The Job Control Workflow

                    Ctrl+Z
  FOREGROUND  ──────────────>  STOPPED
      ^                          |
      |                          |
    fg %N                      bg %N
      |                          |
      |                          v
      +──────── BACKGROUND <─────+
                   (& at start)

Practical Example

# Start a large file copy in the foreground
cp -r /data/bigdir /backup/bigdir

# Oh wait, this is taking forever. Suspend it:
# Press Ctrl+Z
# Output: [1]+  Stopped    cp -r /data/bigdir /backup/bigdir

# Resume it in the background so you can do other work
bg %1
# Output: [1]+ cp -r /data/bigdir /backup/bigdir &

# Now you can keep using the terminal
# Check on the job periodically
jobs

Keeping Processes Alive: nohup and disown

When you log out of a terminal, the shell sends SIGHUP to all its child processes. This kills them. That is fine for interactive commands, but terrible for long-running tasks.

nohup -- Plan Ahead

If you know beforehand that a command should survive logout:

# nohup redirects output to nohup.out and ignores SIGHUP
nohup python3 long_training.py &

# Or redirect output yourself
nohup ./backup_script.sh > /var/log/backup.log 2>&1 &

disown -- After the Fact

You already started a job and forgot nohup? Use disown:

# Start a long job
python3 train_model.py &
# Output: [1] 34567

# Remove it from the shell's job table
disown %1

# Now you can log out safely -- the process will keep running

# disown all background jobs
disown -a

# disown and also suppress SIGHUP
disown -h %1

Which to Use?

+-------------------------------------------------------------+
|  Situation                        |  Use                     |
|-----------------------------------|--------------------------|
|  Starting a new long-running job  |  nohup command &         |
|  Already running, forgot nohup    |  Ctrl+Z, bg, disown      |
|  Need to reconnect to output      |  Use tmux or screen       |
+-------------------------------------------------------------+

Think About It: Neither nohup nor disown lets you reconnect to the process output later. What tool from Chapter 26 (tmux) solves this problem? Why is tmux or screen often the better choice from the start?

Process Priority: nice and renice

Linux uses a priority system to decide how much CPU time each process gets. The "nice" value ranges from -20 (highest priority, least nice to other processes) to 19 (lowest priority, most nice):

  -20 ─────────── 0 ─────────── 19
  Highest priority    Default    Lowest priority
  (least nice)                   (most nice)

Setting Priority at Launch

# Run a CPU-heavy command at low priority
nice -n 10 make -j$(nproc)

# Run at high priority (requires root)
sudo nice -n -5 ./critical-task

Changing Priority of Running Processes

# Find the PID
pgrep -a my_script

# Lower its priority (any user can increase niceness)
renice 15 -p 12345

# Raise its priority (requires root)
sudo renice -10 -p 12345

# Renice all processes by a user
sudo renice 10 -u testuser

Practical Example

# You need to compile a project but don't want it to slow down
# the web server running on the same machine
nice -n 19 make -j$(nproc)

# The backup is running too fast and choking disk I/O
# Lower its priority
renice 15 -p $(pgrep rsync)

Killing Processes

The kill Command

Despite its name, kill actually sends signals. We will cover signals in depth in Chapter 11, but here are the essentials:

# Send SIGTERM (graceful termination) -- this is the default
kill 12345

# Send SIGKILL (forceful termination -- cannot be caught)
kill -9 12345
# Or equivalently:
kill -KILL 12345

# Send SIGTERM to all processes with a given name
killall nginx

# Same, with pattern matching
pkill -f "python.*train"

# Kill all processes owned by a user
pkill -u testuser

WARNING: kill -9 should be your LAST resort. It does not give the process a chance to clean up -- temporary files may be left behind, database connections may not be closed properly, data may be corrupted. Always try kill (SIGTERM) first, wait a few seconds, and only use kill -9 if the process refuses to die.

The Escalation Ladder

# Step 1: Ask nicely (SIGTERM)
kill 12345

# Step 2: Wait a moment
sleep 5

# Step 3: Check if it is still running
ps -p 12345

# Step 4: Only if still running, force kill
kill -9 12345

Debug This: Diagnosing a High-CPU Process

Your monitoring says the system load is 12 on a 4-core machine. Diagnose it:

# Step 1: See system load averages
uptime
# Output: load average: 12.05, 11.30, 9.45

# Step 2: What is consuming CPU?
ps aux --sort=-%cpu | head -10

# Step 3: Is it a single process or many?
# If one process shows 400% CPU, it is multi-threaded
# If many processes show 100%, you have too many competing

# Step 4: Investigate the top consumer
# Say PID 5678 is at 350% CPU
cat /proc/5678/status | grep -E "Name|State|Threads|Pid|PPid"

# Step 5: What files does it have open?
ls -l /proc/5678/fd/ | head -20

# Step 6: What is it actually doing? (trace its system calls)
sudo strace -p 5678 -c
# Press Ctrl+C after a few seconds to see a summary

# Step 7: What started it? Check full command line
cat /proc/5678/cmdline | tr '\0' ' '; echo

# Step 8: If it is runaway garbage, kill it gracefully
kill 5678
sleep 3
# If still alive:
kill -9 5678

Hands-On: Process Exploration Lab

# 1. Create some processes to observe
sleep 600 &
sleep 601 &
sleep 602 &

# 2. See them in your job list
jobs -l

# 3. See them in ps
ps aux | grep sleep

# 4. Look at their parent PID -- it should be your shell
echo "My shell PID: $$"
ps -p $(pgrep -f "sleep 60[0-2]") -o pid,ppid,stat,cmd

# 5. Suspend one
kill -STOP $(pgrep -f "sleep 600")
# Check its state changed to T
ps -p $(pgrep -f "sleep 600") -o pid,stat,cmd

# 6. Resume it
kill -CONT $(pgrep -f "sleep 600")
# Check its state is back to S
ps -p $(pgrep -f "sleep 600") -o pid,stat,cmd

# 7. Create a "zombie" (for educational purposes)
# This script forks a child that exits, but parent doesn't wait()
bash -c '(exit 0) & sleep 2; ps aux | grep -E "Z|defunct"' &

# 8. Clean up
kill $(pgrep -f "sleep 60[0-2]") 2>/dev/null

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 10 Recap: Processes & Job Control                       |
|------------------------------------------------------------------|
|                                                                  |
|  - A process is a running instance of a program.                 |
|  - Every process has a PID and a parent (PPID).                  |
|  - Processes are created via fork()+exec().                      |
|  - PID 1 (init/systemd) is the ancestor of all processes.       |
|  - ps aux and ps -ef show process listings.                      |
|  - Process states: R (running), S (sleeping), D (disk wait),    |
|    Z (zombie), T (stopped).                                     |
|  - Use & to background, Ctrl+Z to suspend, bg/fg to resume.    |
|  - nohup and disown keep processes alive after logout.           |
|  - nice/renice control scheduling priority (-20 to 19).         |
|  - kill sends signals; always try SIGTERM before SIGKILL.       |
|  - /proc/PID/ is a goldmine of per-process information.         |
|  - top/htop provide real-time monitoring.                        |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Process Genealogy

Run pstree -p $$ to see the process tree from your shell to PID 1. Count how many generations separate your shell from PID 1. What are the intermediate processes?

Exercise 2: Resource Hog

Write a command that consumes CPU (e.g., yes > /dev/null &). Launch three of them with different nice values (0, 10, 19). Use top to observe how CPU time is distributed. Which one gets the most? Kill them all when done.

Exercise 3: Job Control Mastery

Start five sleep 1000 commands in the background. Use jobs to list them. Bring the third one to the foreground, suspend it with Ctrl+Z, then resume it in the background. Kill the second one by job number (kill %2). Verify with jobs.

Exercise 4: /proc Deep Dive

Pick any running process and explore its /proc/PID/ directory. Read status, cmdline, environ, maps, and fd/. Write a one-paragraph summary of what this process is doing, based solely on what you found in /proc.

Bonus Challenge

Write a bash script called proc-report.sh that takes a PID as an argument and outputs: the command name, its parent's command name, its state, memory usage (RSS), number of open file descriptors, and how long it has been running. All information should come from /proc/PID/ and related files.

Signals Deep Dive

Why This Matters

You press Ctrl+C and a program stops. You close a terminal and background jobs die. You run sudo systemctl reload nginx and Nginx picks up its new configuration without dropping a single connection. You type kill -9 and a stubborn process vanishes.

All of these work through signals -- the kernel's mechanism for delivering notifications to processes. Signals are how Linux says "hey, something just happened that you need to deal with." They are the nervous system of process management, and understanding them is the difference between blindly typing kill -9 and surgically managing processes like a professional.

This chapter covers what signals are, which ones matter most, how to trap them in your own scripts, and how production services use them for graceful reloads and shutdowns.

Try This Right Now

# See all available signals on your system
kill -l

# Start a process and send it different signals
sleep 300 &
PID=$!
echo "Started sleep with PID $PID"

# Send SIGTERM (signal 15) -- the polite termination
kill $PID
# The process is gone

# Start another
sleep 300 &
PID=$!

# Send SIGSTOP (pause the process)
kill -STOP $PID
ps -p $PID -o pid,stat,cmd
# State should show 'T' (stopped)

# Resume it
kill -CONT $PID
ps -p $PID -o pid,stat,cmd
# State should be back to 'S' (sleeping)

# Clean up
kill $PID

What Are Signals?

A signal is an asynchronous notification sent to a process. Think of it like tapping someone on the shoulder -- the process was doing something else, and now it has to respond.

Signals can come from:

• The kernel -- hardware faults (SIGSEGV), timer expired, child process died
• Another process -- via the kill system call (despite the name, it sends any signal)
• The terminal -- key combinations like Ctrl+C, Ctrl+Z, Ctrl+\
• The process itself -- a program can send signals to itself

When a signal arrives, the process can:

1. Handle it -- run a custom signal handler function
2. Ignore it -- pretend it never happened
3. Let the default action occur -- each signal has a defined default

Two signals are special: SIGKILL (9) and SIGSTOP (19) can NEVER be caught, handled, or ignored. The kernel enforces them directly.

The Signal Table: Signals You Need to Know

+--------+----------+---------+-----------------------------------------------+
| Number | Name     | Default | Purpose                                       |
+--------+----------+---------+-----------------------------------------------+
|   1    | SIGHUP   | Term    | Terminal hangup; daemons: reload config        |
|   2    | SIGINT   | Term    | Interrupt from keyboard (Ctrl+C)               |
|   3    | SIGQUIT  | Core    | Quit from keyboard (Ctrl+\), produces core dump|
|   6    | SIGABRT  | Core    | Abort signal from abort() call                 |
|   9    | SIGKILL  | Term    | Force kill (CANNOT be caught or ignored)       |
|  10    | SIGUSR1  | Term    | User-defined signal 1                          |
|  11    | SIGSEGV  | Core    | Invalid memory reference (segfault)            |
|  12    | SIGUSR2  | Term    | User-defined signal 2                          |
|  13    | SIGPIPE  | Term    | Broken pipe (write to pipe with no reader)     |
|  14    | SIGALRM  | Term    | Timer alarm from alarm()                       |
|  15    | SIGTERM  | Term    | Graceful termination request                   |
|  17    | SIGCHLD  | Ignore  | Child process stopped or terminated            |
|  18    | SIGCONT  | Cont    | Resume if stopped                              |
|  19    | SIGSTOP  | Stop    | Pause process (CANNOT be caught or ignored)    |
|  20    | SIGTSTP  | Stop    | Stop from terminal (Ctrl+Z)                    |
+--------+----------+---------+-----------------------------------------------+

Distro Note: Signal numbers can differ between architectures (e.g., SIGUSR1 is 10 on x86 but 16 on MIPS). Always use signal names, not numbers, in scripts and commands for portability. Write kill -TERM instead of kill -15.

Default Actions Explained

• Term -- Terminate the process
• Core -- Terminate and produce a core dump file (for debugging)
• Stop -- Pause (suspend) the process
• Cont -- Resume a stopped process
• Ignore -- Do nothing

Signals in Detail

SIGTERM (15) -- The Gentleman's Termination

This is the default signal sent by kill. It is a polite request: "please shut down." Well-written programs catch SIGTERM, clean up resources (close files, release locks, flush buffers), and exit cleanly.

# These are equivalent:
kill 12345
kill -15 12345
kill -TERM 12345
kill -SIGTERM 12345

SIGKILL (9) -- The Executioner

SIGKILL terminates a process immediately. The process gets no chance to handle it, no chance to clean up. The kernel simply removes it.

kill -9 12345
kill -KILL 12345

When to use SIGKILL:

• The process is not responding to SIGTERM
• The process is stuck and must be removed
• You are dealing with a compromised process

When NOT to use SIGKILL:

• As your first attempt (always try SIGTERM first)
• On database processes (risk of data corruption)
• On processes holding locks (locks may become stale)

WARNING: kill -9 is the sledgehammer. It leaves temporary files behind, does not close database connections cleanly, does not release file locks, and does not flush write buffers. Use it only after SIGTERM has failed.

SIGINT (2) -- The Keyboard Interrupt

This is what Ctrl+C sends. Programs can catch it to clean up gracefully:

# Start a process
sleep 300
# Press Ctrl+C
# The process receives SIGINT and terminates

SIGQUIT (3) -- Quit with Core Dump

Ctrl+\ sends SIGQUIT. Like SIGINT, but also produces a core dump for debugging:

# Start a process
sleep 300
# Press Ctrl+\
# Output: Quit (core dumped)

SIGHUP (1) -- Hang Up / Reload

Originally meant "the terminal connection was lost" (the modem hung up). Today it has two common uses:

1. Terminal hangup: When you close a terminal, SIGHUP is sent to all processes in that terminal
2. Daemon reload: By convention, sending SIGHUP to a daemon tells it to re-read its configuration file without restarting

# Reload Nginx configuration
sudo kill -HUP $(cat /run/nginx.pid)
# Or equivalently:
sudo systemctl reload nginx  # This sends SIGHUP under the hood

# Reload sshd
sudo kill -HUP $(pgrep -x sshd | head -1)

SIGSTOP (19) and SIGCONT (18) -- Pause and Resume

SIGSTOP pauses a process. SIGCONT resumes it. SIGSTOP cannot be caught -- this is how debuggers freeze processes.

# Pause a process
kill -STOP 12345

# Resume it
kill -CONT 12345

The terminal equivalent of SIGSTOP is SIGTSTP (20), sent by Ctrl+Z. Unlike SIGSTOP, SIGTSTP CAN be caught and handled.

SIGUSR1 (10) and SIGUSR2 (12) -- User Defined

These have no predefined meaning. Programs define what they do:

# Example: dd reports progress on SIGUSR1
dd if=/dev/urandom of=/tmp/testfile bs=1M count=1000 &
DD_PID=$!

# Send SIGUSR1 to get a progress report
kill -USR1 $DD_PID
# dd outputs bytes transferred so far

# Clean up
kill $DD_PID
rm -f /tmp/testfile

Other examples:

• Nginx uses SIGUSR1 to reopen log files (log rotation)
• Some programs use SIGUSR2 to toggle debug mode

SIGPIPE (13) -- Broken Pipe

Sent when a process writes to a pipe but the reading end has been closed:

# This triggers SIGPIPE:
yes | head -1
# 'yes' keeps writing, but 'head' exits after 1 line
# The kernel sends SIGPIPE to 'yes'

SIGCHLD (17) -- Child Status Changed

Sent to a parent when a child process stops, continues, or terminates. The parent should call wait() to collect the child's exit status. If it doesn't, the child becomes a zombie.

Think About It: When you run systemctl reload nginx, Nginx re-reads its config and applies it without dropping connections. How is this possible? (Hint: Nginx's master process catches SIGHUP, re-reads the config, spawns new worker processes with the new config, and gracefully shuts down old workers.)

Sending Signals: kill, killall, pkill

kill -- By PID

# Send SIGTERM (default)
kill 12345

# Send a specific signal
kill -HUP 12345
kill -SIGUSR1 12345

# Send to multiple PIDs
kill 12345 12346 12347

# Send signal 0 -- checks if process exists (sends nothing)
kill -0 12345
echo $?  # 0 = exists, 1 = doesn't exist

killall -- By Name

# Kill all processes named "python3"
killall python3

# Send SIGHUP to all nginx processes
killall -HUP nginx

# Interactive mode -- confirm before each kill
killall -i python3

# Only kill processes owned by a specific user
killall -u alice python3

pkill -- By Pattern

# Kill processes matching a pattern
pkill -f "python train"

# Send SIGHUP to processes matching a pattern
pkill -HUP -f "gunicorn"

# Kill processes owned by a user
pkill -u testuser

# Kill the oldest matching process
pkill -o -f "worker"

# Kill the newest matching process
pkill -n -f "worker"

WARNING: Be very careful with killall and pkill. killall python3 kills ALL python3 processes, not just yours (if you are root). Always verify what will be matched first with pgrep:

# See what pkill would match BEFORE actually killing
pgrep -a -f "python train"

Trapping Signals in Bash

The trap builtin lets your scripts catch signals and run custom code in response. This is how you write scripts that clean up after themselves:

Basic Trap Syntax

trap 'commands' SIGNAL_LIST

Example: Cleanup on Exit

#!/bin/bash
TMPFILE=$(mktemp)
echo "Working in $TMPFILE"

# Clean up on exit, interrupt, or termination
trap 'echo "Cleaning up..."; rm -f "$TMPFILE"; exit' EXIT INT TERM

# Do some work
echo "data" > "$TMPFILE"
sleep 30

echo "Done"

Example: Graceful Shutdown Script

#!/bin/bash

RUNNING=true

cleanup() {
    echo "Received shutdown signal. Cleaning up..."
    RUNNING=false
    # Close database connections, flush caches, etc.
    echo "Cleanup complete. Exiting."
    exit 0
}

trap cleanup SIGTERM SIGINT

echo "Service started. PID: $$"

# Main loop
while $RUNNING; do
    # Do work here
    echo "Working... $(date)"
    sleep 5
done

Test it:

# Terminal 1:
bash graceful_shutdown.sh

# Terminal 2:
kill $(pgrep -f graceful_shutdown)
# Watch Terminal 1 -- it should print cleanup messages

Example: Ignore a Signal

#!/bin/bash
# Ignore SIGINT (Ctrl+C) -- use with caution!
trap '' SIGINT

echo "You cannot Ctrl+C me! (Use Ctrl+\ or kill from another terminal)"
sleep 60

Example: Trap on SIGHUP for Config Reload

#!/bin/bash
CONFIG_FILE="/etc/myapp/config.conf"

load_config() {
    echo "Loading configuration from $CONFIG_FILE..."
    # In a real script, you would source the config file here
    # source "$CONFIG_FILE"
    echo "Configuration reloaded at $(date)"
}

trap load_config SIGHUP

load_config  # Initial load

echo "Running as PID $$. Send SIGHUP to reload config."

while true; do
    # Main application loop
    sleep 10
done

Test it:

# Terminal 1:
bash config_daemon.sh

# Terminal 2:
kill -HUP $(pgrep -f config_daemon)
# Watch Terminal 1 -- it should print "Configuration reloaded"

Trap Gotchas

# List all active traps
trap -p

# Reset a trap to default behavior
trap - SIGINT

# The EXIT trap fires on ANY exit (normal, error, signal)
trap 'echo "Goodbye"' EXIT

# Traps in subshells: subshells inherit traps but
# modifications in subshells do not affect the parent
(trap 'echo "sub"' SIGINT)
# The parent's SIGINT trap is unchanged

Think About It: If you trap SIGTERM in a script but your script calls kill -9 $$, will the trap run? Why or why not?

How Daemons Handle SIGHUP: A Deep Look

The SIGHUP reload pattern is one of the most important signal conventions in Linux. Here is how it typically works with a real service like Nginx:

                    SIGHUP
                      |
                      v
  +-----------------------------------+
  |  Nginx Master Process (PID 100)   |
  |  1. Catches SIGHUP                |
  |  2. Re-reads nginx.conf           |
  |  3. Validates new configuration   |
  |  4. If valid:                     |
  |     a. Spawns new worker processes|
  |     b. Signals old workers to     |
  |        finish current requests    |
  |     c. Old workers exit after     |
  |        completing in-flight work  |
  |  5. If invalid:                   |
  |     a. Logs error                 |
  |     b. Keeps running with old     |
  |        configuration              |
  +-----------------------------------+
         |                    |
         v                    v
  +-------------+     +-------------+
  | Old Worker  |     | New Worker  |
  | (finishing  |     | (new config)|
  |  requests)  |     |             |
  +-------------+     +-------------+

This is why systemctl reload nginx does not cause downtime -- existing connections are served to completion by old workers while new connections go to new workers with the updated config.

Services and Their Signal Conventions

# The systemctl commands that use signals under the hood:
sudo systemctl reload nginx     # Sends SIGHUP
sudo systemctl stop nginx       # Sends SIGTERM, then SIGKILL after timeout
sudo systemctl kill nginx       # Sends SIGTERM by default
sudo systemctl kill -s HUP nginx  # Sends SIGHUP explicitly

Signal Handling Best Practices

For Scripts

1. Always trap EXIT for cleanup: Temporary files, lock files, PID files -- clean them up.

trap 'rm -f /tmp/myapp.lock' EXIT

2. Use named signals, not numbers: kill -TERM is portable; kill -15 may not be.

3. Do not trap SIGKILL or SIGSTOP: You cannot. If you try, the trap is silently ignored.

4. Keep trap handlers short: A signal can arrive at any time. Your handler should do minimal work -- set a flag, then let the main loop check the flag.

# Good: set a flag
SHUTDOWN=false
trap 'SHUTDOWN=true' SIGTERM

while ! $SHUTDOWN; do
    do_work
done
cleanup

5. Propagate signals to child processes in scripts:

trap 'kill 0' SIGTERM SIGINT
# 'kill 0' sends the signal to the entire process group

For System Administration

1. Always try SIGTERM before SIGKILL: Give processes time to clean up.

2. Use systemctl for managed services: Let systemd handle signal delivery.

3. Use SIGHUP for config reloads: Check if the service supports it first (man service_name).

4. Use kill -0 to check if a process is alive:

if kill -0 "$PID" 2>/dev/null; then
    echo "Process $PID is still running"
else
    echo "Process $PID is gone"
fi

Debug This: A Script That Won't Die

You have a script running that ignores Ctrl+C:

#!/bin/bash
trap '' SIGINT SIGTERM
echo "I am invincible! PID: $$"
while true; do sleep 1; done

How do you stop it?

Diagnosis:

# Ctrl+C won't work (SIGINT is trapped and ignored)
# kill PID won't work (SIGTERM is trapped and ignored)

# Option 1: SIGKILL -- cannot be trapped
kill -9 $(pgrep -f "I am invincible")

# Option 2: SIGQUIT -- they forgot to trap it
kill -QUIT $(pgrep -f "I am invincible")

# Option 3: SIGSTOP -- pause it, then SIGKILL
kill -STOP $(pgrep -f "I am invincible")
kill -9 $(pgrep -f "I am invincible")

The lesson: trapping SIGINT and SIGTERM can be useful for cleanup, but ignoring them entirely (empty trap handler) is an anti-pattern. Always do something useful in the handler, and always exit afterward.

Hands-On: Signal Playground

Create a script that demonstrates signal handling:

#!/bin/bash
# signal_playground.sh

echo "Signal Playground - PID: $$"
echo "Send me signals and watch what happens!"
echo ""

trap 'echo "[$(date +%T)] Caught SIGHUP -- reloading config..."' HUP
trap 'echo "[$(date +%T)] Caught SIGUSR1 -- toggling debug mode"' USR1
trap 'echo "[$(date +%T)] Caught SIGUSR2 -- dumping status"' USR2
trap 'echo "[$(date +%T)] Caught SIGINT -- Ctrl+C pressed"; exit 0' INT
trap 'echo "[$(date +%T)] Caught SIGTERM -- shutting down gracefully"; exit 0' TERM
trap 'echo "[$(date +%T)] EXIT trap -- final cleanup done"' EXIT

echo "Waiting for signals... (try these in another terminal)"
echo "  kill -HUP $$"
echo "  kill -USR1 $$"
echo "  kill -USR2 $$"
echo "  kill -INT $$     (or press Ctrl+C)"
echo "  kill -TERM $$"
echo ""

while true; do
    sleep 1
done

Run it and experiment from another terminal:

# Terminal 1:
bash signal_playground.sh

# Terminal 2:
PID=<the PID shown>
kill -HUP $PID
kill -USR1 $PID
kill -USR2 $PID
kill -TERM $PID

Signals and Process Groups

When you press Ctrl+C, SIGINT is sent not just to one process but to the entire foreground process group. This is why a pipeline like cat file | grep pattern | sort is killed entirely by one Ctrl+C.

# See process group IDs
ps -eo pid,pgid,sid,cmd | head -20

# Send a signal to an entire process group
kill -TERM -12345   # Note the negative PID -- means "process group 12345"

This is also how kill 0 works in a trap handler -- it sends the signal to every process in the current process group.

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 11 Recap: Signals Deep Dive                             |
|------------------------------------------------------------------|
|                                                                  |
|  - Signals are asynchronous notifications to processes.          |
|  - SIGTERM (15): graceful termination request (default kill).    |
|  - SIGKILL (9): force kill, CANNOT be caught -- last resort.    |
|  - SIGINT (2): keyboard interrupt (Ctrl+C).                     |
|  - SIGHUP (1): hangup / reload config for daemons.              |
|  - SIGSTOP/SIGCONT: pause and resume (SIGSTOP can't be caught). |
|  - SIGUSR1/SIGUSR2: user-defined, application-specific.         |
|  - trap in bash lets scripts handle signals.                     |
|  - Always trap EXIT for cleanup of temp files and locks.         |
|  - Daemons use SIGHUP to reload config without downtime.        |
|  - Use signal names (SIGTERM) not numbers (15) for portability. |
|  - Try SIGTERM first; only SIGKILL as last resort.              |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Signal Identification

Run kill -l and identify the signal number for: SIGHUP, SIGINT, SIGTERM, SIGKILL, SIGUSR1, SIGCHLD, SIGSTOP, SIGCONT. Then write the keyboard shortcut that sends SIGINT, SIGQUIT, and SIGTSTP.

Exercise 2: Trap Practice

Write a script that creates three temporary files on startup and removes them all when it receives SIGINT or SIGTERM, or when it exits normally. Test it by killing it with different signals and verifying the temp files are gone.

Exercise 3: Process Group Experiment

Start a pipeline: sleep 100 | sleep 200 | sleep 300 &. Find the process group ID for all three processes (use ps -eo pid,pgid,cmd | grep sleep). Send SIGTERM to the process group using kill -TERM -PGID. Verify all three are gone.

Exercise 4: SIGHUP Reload

Write a simple script that reads a "config file" (just a text file with a key=value pair) on startup and re-reads it whenever it receives SIGHUP. Test it by changing the config file and sending SIGHUP.

Bonus Challenge

Write a bash script that acts as a simple process monitor. It takes a command as arguments, runs it, and if the command dies unexpectedly (exit code not 0), it restarts it automatically. Use trap to handle SIGCHLD or simply loop with wait. The monitor itself should handle SIGTERM gracefully by forwarding it to the child process and then exiting.

Inter-Process Communication

Why This Matters

You type cat access.log | grep "404" | sort | uniq -c | sort -rn | head -10 and instantly get the top 10 most common 404 URLs from a log file. Five separate programs just cooperated seamlessly to produce that result. How?

Or consider this: your web browser talks to a local caching proxy. Your application connects to a PostgreSQL database running on the same machine. Your container runtime communicates with its daemon. None of these use the network -- they all use Inter-Process Communication (IPC) mechanisms built into the kernel.

Linux provides multiple IPC mechanisms, each suited to different scenarios. This chapter covers the ones you will encounter daily -- pipes, named pipes, redirections, process substitution -- and introduces the ones you need to know about for deeper work: shared memory, Unix domain sockets, and message queues.

Try This Right Now

# A pipeline: three processes communicating through pipes
echo "hello world" | tr 'a-z' 'A-Z' | rev
# Output: DLROW OLLEH

# How many processes were involved?
# Three: echo, tr, rev -- all connected by pipes

# See a pipe in action with /proc
sleep 100 | sleep 200 &
ls -l /proc/$(pgrep -n "sleep 100")/fd/
# fd/1 (stdout) will be a pipe
ls -l /proc/$(pgrep -n "sleep 200")/fd/
# fd/0 (stdin) will be a pipe

# Clean up
kill %1 2>/dev/null

Pipes: The Unix Superpower

The pipe (|) is the single most important IPC mechanism in Unix. It connects the standard output of one process to the standard input of the next:

+---------+    pipe    +---------+    pipe    +---------+
| Process |  stdout-->| Process |  stdout-->| Process |
|    A    |    stdin<--|    B    |    stdin<--|    C    |
+---------+           +---------+           +---------+

How Pipes Work Internally

When the shell sees cmd1 | cmd2, it:

1. Creates a pipe (a small kernel buffer, typically 64KB on Linux)
2. Forks two child processes
3. Connects cmd1's stdout (fd 1) to the write end of the pipe
4. Connects cmd2's stdin (fd 0) to the read end of the pipe
5. cmd1 writes data into the pipe; cmd2 reads data from the pipe

          Kernel Pipe Buffer (64KB)
         +------------------------+
cmd1 --> | data flows this way -> | --> cmd2
  fd 1   +------------------------+   fd 0
 (write)                             (read)

Key characteristics:

• Pipes are unidirectional -- data flows one way only
• Pipes are anonymous -- they exist only while the processes are running
• If the pipe buffer is full, the writer blocks until the reader consumes data
• If the reader exits, the writer gets SIGPIPE
• Pipes connect processes that share a common ancestor (usually the shell)

Pipeline Examples

# Count lines in a file
cat /etc/passwd | wc -l
# Better: wc -l < /etc/passwd (no useless 'cat')

# Find the 10 largest files in /var/log
du -sh /var/log/* 2>/dev/null | sort -rh | head -10

# Show unique shells used on the system
cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn

# Monitor a log file and filter for errors
tail -f /var/log/syslog | grep --line-buffered "error"

# Count processes per user
ps aux | awk '{print $1}' | sort | uniq -c | sort -rn

# Generate a random password
cat /dev/urandom | tr -dc 'A-Za-z0-9!@#$' | head -c 20; echo

Pipeline Exit Status

By default, the shell reports the exit status of the LAST command in a pipeline:

false | true
echo $?
# Output: 0 (true's exit code, not false's)

To get the exit status of every command in the pipeline:

# Enable pipefail -- the pipeline fails if ANY command fails
set -o pipefail

false | true
echo $?
# Output: 1 (false's exit code)

# Or check individual statuses with PIPESTATUS (bash-specific)
cat nonexistent_file 2>/dev/null | sort | head
echo "${PIPESTATUS[@]}"
# Output: 1 0 0

Think About It: Why does grep pattern file | wc -l work but might give the wrong answer if grep fails? How does set -o pipefail help in scripts?

Redirection: Controlling File Descriptors

Every process starts with three open file descriptors:

+-----+--------+-------------------+
| FD  | Name   | Default           |
+-----+--------+-------------------+
|  0  | stdin  | Keyboard / terminal|
|  1  | stdout | Terminal screen    |
|  2  | stderr | Terminal screen    |
+-----+--------+-------------------+

Redirection lets you change where these point.

Output Redirection

# Redirect stdout to a file (overwrites)
echo "hello" > output.txt

# Redirect stdout to a file (appends)
echo "world" >> output.txt

# Redirect stderr to a file
ls /nonexistent 2> errors.txt

# Redirect stderr to the same place as stdout
ls /nonexistent /tmp 2>&1

# Redirect both stdout and stderr to a file
ls /nonexistent /tmp > all_output.txt 2>&1
# Or the modern shorthand (bash 4+):
ls /nonexistent /tmp &> all_output.txt

# Append both stdout and stderr
command &>> logfile.txt

The Order Matters

This is a classic gotcha:

# WRONG: stderr goes to terminal, not the file
ls /nonexistent /tmp 2>&1 > output.txt
# Why? 2>&1 duplicates fd 2 to where fd 1 currently points (terminal)
# Then > output.txt redirects fd 1 to the file
# So fd 2 still points to the terminal!

# RIGHT: redirect stdout first, then stderr to the same place
ls /nonexistent /tmp > output.txt 2>&1
# > output.txt redirects fd 1 to the file
# 2>&1 duplicates fd 2 to where fd 1 now points (the file)

Input Redirection

# Read from a file instead of the keyboard
sort < unsorted_list.txt

# Here document -- inline multi-line input
cat << 'EOF'
This is line one.
This is line two.
Variables like $HOME are NOT expanded (single-quoted delimiter).
EOF

# Here document with variable expansion
cat << EOF
Your home directory is $HOME
Your shell is $SHELL
EOF

# Here string -- single-line input
grep "pattern" <<< "search in this string"

/dev/null -- The Black Hole

/dev/null is a special file that discards everything written to it and produces nothing when read:

# Discard stdout (keep only errors)
find / -name "*.conf" > /dev/null

# Discard stderr (keep only normal output)
find / -name "*.conf" 2>/dev/null

# Discard everything
command > /dev/null 2>&1

# Check if a command succeeds without seeing output
if grep -q "pattern" file.txt 2>/dev/null; then
    echo "Found it"
fi

Redirecting to Multiple Places with tee

tee reads stdin and writes to both stdout AND a file:

# See output AND save it to a file
make 2>&1 | tee build.log

# Append instead of overwrite
df -h | tee -a disk_report.txt

# Write to multiple files
echo "log entry" | tee file1.log file2.log file3.log

                   +-----------> Terminal (stdout)
                   |
stdin --> [ tee ] -+
                   |
                   +-----------> file.log

Named Pipes (FIFOs)

Regular pipes are anonymous -- they exist only within a pipeline. Named pipes (FIFOs) are visible in the filesystem and can connect unrelated processes:

# Create a named pipe
mkfifo /tmp/mypipe

# Check it
ls -l /tmp/mypipe
# prw-r--r-- 1 alice alice 0 ... /tmp/mypipe
# The 'p' at the start means "pipe"

Using Named Pipes

Named pipes are blocking: a writer blocks until a reader opens the pipe, and vice versa. You need two terminals (or background processes):

# Terminal 1: Write to the pipe (this will block until someone reads)
echo "Hello from Terminal 1" > /tmp/mypipe

# Terminal 2: Read from the pipe
cat < /tmp/mypipe
# Output: Hello from Terminal 1
# Both commands complete

Practical Use Case: Log Processing Pipeline

# Create a named pipe for log processing
mkfifo /tmp/log_pipe

# Terminal 1: Tail a log into the pipe
tail -f /var/log/syslog > /tmp/log_pipe 2>/dev/null &

# Terminal 2: Process logs from the pipe
cat /tmp/log_pipe | grep --line-buffered "error" | while read line; do
    echo "[ALERT] $line"
done

# Clean up
rm /tmp/log_pipe

Practical Use Case: Parallel Processing

# Create two named pipes
mkfifo /tmp/pipe_a /tmp/pipe_b

# Split processing: compress and checksum simultaneously
tee /tmp/pipe_a < largefile.bin | md5sum > checksum.txt &
gzip < /tmp/pipe_a > largefile.bin.gz &
wait

# Clean up
rm /tmp/pipe_a /tmp/pipe_b

Think About It: What happens if you open a named pipe for writing but no process ever opens it for reading? How is this different from writing to a regular file?

Process Substitution

Process substitution lets you use a process's output (or input) as if it were a file. This is a bash feature (not POSIX sh).

Output Process Substitution: <(command)

The <(command) syntax runs command and makes its output available as a file path:

# Compare the output of two commands
diff <(ls /dir1) <(ls /dir2)

# Compare sorted lists without creating temporary files
diff <(sort file1.txt) <(sort file2.txt)

# What is the "file"?
echo <(echo hello)
# Output: /dev/fd/63 (or similar -- it's a file descriptor)

This is incredibly powerful because many commands expect file arguments, not piped input:

# paste needs two files -- use process substitution
paste <(cut -d: -f1 /etc/passwd) <(cut -d: -f7 /etc/passwd)

# Feed two data streams to a command that expects files
join <(sort file1.txt) <(sort file2.txt)

# Load data from a command into a while loop without subshell issues
while read -r user shell; do
    echo "User $user uses $shell"
done < <(awk -F: '{print $1, $7}' /etc/passwd)

Input Process Substitution: >(command)

The >(command) syntax creates a file path that feeds into a command's stdin:

# Write to two destinations simultaneously
echo "log entry" | tee >(gzip > compressed.gz) >(wc -c > byte_count.txt)

# Send output to both a file and a log processor
some_command > >(tee output.log) 2> >(tee error.log >&2)

Why Not Just Use Pipes?

Process substitution solves problems that pipes cannot:

# Problem: compare output of two commands
# With pipes -- impossible (pipes are linear, not branching)
# With process substitution:
diff <(find /dir1 -type f | sort) <(find /dir2 -type f | sort)

# Problem: the while-read-pipe subshell issue
# This LOSES the variable outside the loop:
count=0
cat file.txt | while read line; do
    count=$((count + 1))
done
echo "$count"  # Prints 0! The while loop ran in a subshell

# Process substitution avoids the subshell:
count=0
while read line; do
    count=$((count + 1))
done < <(cat file.txt)
echo "$count"  # Prints the correct count

Shared Memory Overview

Shared memory is the fastest IPC mechanism. Two or more processes map the same region of physical memory into their address spaces:

  Process A                     Process B
  +----------+                 +----------+
  | Address  |                 | Address  |
  | Space    |                 | Space    |
  |          |                 |          |
  | Shared   |---+         +---| Shared   |
  | Region   |   |         |   | Region   |
  +----------+   |         |   +----------+
                  v         v
            +------------------+
            | Physical Memory  |
            | (shared segment) |
            +------------------+

There is no copying of data -- both processes read and write to the same memory. This makes it extremely fast but also means you need synchronization (mutexes, semaphores) to prevent data corruption.

POSIX Shared Memory

# List existing shared memory segments
ls /dev/shm/

# See System V shared memory segments
ipcs -m

# See all IPC resources (shared memory, semaphores, message queues)
ipcs -a

Shared memory is commonly used by:

• Database systems (PostgreSQL shared buffers)
• Web servers (shared worker state)
• Audio/video processing (passing frames between processes)
• tmpfs mounts (/dev/shm is a tmpfs)

# /dev/shm is a tmpfs mount -- a RAM-based filesystem
df -h /dev/shm
mount | grep shm

# You can use it for fast temporary storage
echo "fast data" > /dev/shm/temp_data
# But remember: it vanishes on reboot!

Distro Note: The default size of /dev/shm varies. Debian/Ubuntu defaults to 50% of RAM. RHEL/Fedora also defaults to 50% of RAM. You can resize it: sudo mount -o remount,size=2G /dev/shm.

Unix Domain Sockets

Unix domain sockets are like network sockets but for local communication only. They are faster than TCP/IP sockets (no network stack overhead) and support both stream and datagram modes.

# Find Unix domain sockets on your system
ss -xl  # or: ss -x

# Or look in common locations
ls -l /var/run/*.sock 2>/dev/null
ls -l /run/*.sock 2>/dev/null
ls -l /tmp/*.sock 2>/dev/null

Common examples you will encounter:

# Docker daemon socket
ls -l /var/run/docker.sock

# MySQL/MariaDB socket
ls -l /var/run/mysqld/mysqld.sock

# PostgreSQL socket
ls -l /var/run/postgresql/.s.PGSQL.5432

# systemd-journald socket
ls -l /run/systemd/journal/socket

# D-Bus system socket
ls -l /run/dbus/system_bus_socket

How They Differ from Pipes

+---------------------+------------------+--------------------+
| Feature             | Pipes            | Unix Sockets       |
+---------------------+------------------+--------------------+
| Direction           | Unidirectional   | Bidirectional      |
| Connections         | One-to-one       | Many-to-one        |
| Related processes   | Required (pipes) | Not required       |
| needed?             | Optional (FIFO)  |                    |
| File on disk        | No (pipes)       | Yes (socket file)  |
|                     | Yes (FIFOs)      |                    |
| Protocol support    | Byte stream only | Stream or datagram |
| Permissions         | Via fd inheritance| Via file permissions|
+---------------------+------------------+--------------------+

Practical Example: Communicating with Docker via Socket

# Docker CLI talks to dockerd via Unix socket
# You can do the same with curl:
sudo curl --unix-socket /var/run/docker.sock http://localhost/version
# Returns JSON with Docker version info

# List containers via the socket API
sudo curl --unix-socket /var/run/docker.sock http://localhost/containers/json

Creating a Simple Unix Domain Socket (with socat)

# Install socat if not present
sudo apt install socat    # Debian/Ubuntu
sudo dnf install socat    # RHEL/Fedora

# Terminal 1: Create a socket server
socat UNIX-LISTEN:/tmp/test.sock,fork EXEC:/bin/cat

# Terminal 2: Connect and send data
echo "Hello, socket!" | socat - UNIX-CONNECT:/tmp/test.sock

# Clean up
rm -f /tmp/test.sock

Message Queues Overview

Message queues allow processes to exchange discrete messages through a kernel-maintained queue. Unlike pipes (byte streams), messages maintain their boundaries.

  Process A                              Process B
  +----------+                          +----------+
  |  Send    |---> +----------------+ -->| Receive  |
  |  msg 1   |     | Kernel Message |   |  msg 1   |
  |  msg 2   |---> |    Queue       | -->|  msg 2   |
  |  msg 3   |---> |                | -->|  msg 3   |
  +----------+     +----------------+   +----------+

Key characteristics:

• Messages have types and priorities
• Messages maintain boundaries (unlike pipes, which are byte streams)
• The queue persists until explicitly removed (survives process death)
• Kernel enforces queue size limits

Viewing Message Queues

# List POSIX message queues
ls /dev/mqueue/ 2>/dev/null

# List System V message queues
ipcs -q

# Show all IPC objects with details
ipcs -a

Linux supports both System V IPC (older: shmget, msgget, semget) and POSIX IPC (newer: shm_open, mq_open, sem_open). New code should prefer POSIX. Use ipcs / ipcrm to manage System V resources, and browse /dev/shm / /dev/mqueue for POSIX resources.

Debug This: Mystery Broken Pipe

A user reports that their script fails with "Broken pipe" errors:

#!/bin/bash
generate_report | head -5
echo "Report generated successfully"

The script works but prints write error: Broken pipe to stderr.

Diagnosis:

generate_report produces many lines of output. head -5 reads only 5 lines and then exits, closing the read end of the pipe. When generate_report tries to write the next line, the kernel sends it SIGPIPE and it dies.

Solutions:

# Option 1: Suppress the error message
generate_report 2>/dev/null | head -5

# Option 2: Redirect stderr to /dev/null for just that command
{ generate_report 2>/dev/null; } | head -5

# Option 3: Trap SIGPIPE (if you control the script doing the writing)
trap '' SIGPIPE

# Option 4: In many cases, this is harmless -- the exit code
# of the pipeline is 0 (head succeeded), and the SIGPIPE
# is just the kernel cleaning up efficiently

Think About It: Is a broken pipe actually an error? Or is it the kernel's efficient way of saying "nobody is listening, so stop wasting effort"?

Hands-On: Building an IPC Pipeline

Let us build a practical log processing system using different IPC mechanisms:

# Step 1: Create a named pipe for log ingestion
mkfifo /tmp/log_pipe

# Step 2: Write a log producer (simulates an application logging)
for i in $(seq 1 100); do
    echo "$(date '+%Y-%m-%d %H:%M:%S') [$(shuf -e INFO WARN ERROR -n1)] Message $i"
    sleep 0.1
done > /tmp/log_pipe &
PRODUCER_PID=$!

# Step 3: Process the logs -- count by severity
cat /tmp/log_pipe | tee \
    >(grep "ERROR" >> /tmp/errors.log) \
    >(grep "WARN" >> /tmp/warnings.log) \
    > /tmp/all.log

# Step 4: Wait for the producer to finish
wait $PRODUCER_PID 2>/dev/null

# Step 5: Check results
echo "=== Error count ==="
wc -l < /tmp/errors.log
echo "=== Warning count ==="
wc -l < /tmp/warnings.log
echo "=== Total messages ==="
wc -l < /tmp/all.log

# Step 6: Compare error and warning files
diff <(cut -d']' -f2 /tmp/errors.log | sort) \
     <(cut -d']' -f2 /tmp/warnings.log | sort) | head -20

# Clean up
rm -f /tmp/log_pipe /tmp/errors.log /tmp/warnings.log /tmp/all.log

IPC Mechanism Selection Guide

+-------------------+-------------+----------+----------+-----------+
| Mechanism         | Direction   | Speed    | Related  | Best For  |
|                   |             |          | Procs?   |           |
+-------------------+-------------+----------+----------+-----------+
| Pipe (|)          | One-way     | Fast     | Yes      | Pipelines |
| Named Pipe (FIFO) | One-way    | Fast     | No       | Producer- |
|                   |             |          |          | consumer  |
| Unix Socket       | Two-way     | Fast     | No       | Client-   |
|                   |             |          |          | server    |
| Shared Memory     | Both read/  | Fastest  | No       | Large data|
|                   | write       |          |          | sharing   |
| Message Queue     | One-way     | Moderate | No       | Discrete  |
|                   | per queue   |          |          | messages  |
| Signals           | One-way     | Fast     | No       | Simple    |
|                   | (notify)    |          |          | events    |
| Files             | Both        | Slow     | No       | Persistent|
|                   |             | (disk)   |          | data      |
+-------------------+-------------+----------+----------+-----------+

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 12 Recap: Inter-Process Communication                   |
|------------------------------------------------------------------|
|                                                                  |
|  - Pipes (|) connect stdout of one process to stdin of next.     |
|  - Pipes are anonymous, unidirectional, and kernel-buffered.     |
|  - Redirections (>, >>, 2>&1, <) control file descriptors.      |
|  - /dev/null discards output; tee duplicates it.                |
|  - Named pipes (mkfifo) persist on disk and connect any two     |
|    processes.                                                    |
|  - Process substitution <() and >() treat command output as     |
|    file paths.                                                   |
|  - Shared memory is the fastest IPC (no data copying).          |
|  - Unix domain sockets provide bidirectional local IPC.          |
|  - Message queues preserve message boundaries.                   |
|  - Use pipefail in scripts to catch pipeline errors.            |
|  - Order matters in redirections: > file 2>&1 is correct.       |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Redirection Mastery

Write a command that runs find / -name "*.conf" and saves normal output to found.txt, errors to errors.txt, and also displays both on the terminal simultaneously. (Hint: you need tee and redirection.)

Exercise 2: Named Pipe Chat

Create a simple two-way chat system using two named pipes. Two terminals should be able to send messages to each other. Each terminal reads from one pipe and writes to the other.

Exercise 3: Process Substitution Power

Without creating any temporary files, find all files that exist in /etc but not in /usr/etc (or any two directories). Use diff with process substitution and find.

Exercise 4: Pipeline Analysis

Run cat /etc/passwd | cut -d: -f7 | sort | uniq -c | sort -rn. Then rewrite it without cat (using input redirection). Then use ${PIPESTATUS[@]} to verify all pipeline stages succeeded.

Bonus Challenge

Write a script that uses a named pipe to implement a simple job queue. One terminal acts as the "dispatcher" writing commands to the pipe, and another terminal acts as the "worker" reading and executing them one at a time. Include error handling and logging of each job's exit status.

The Kernel Up Close

Why This Matters

Every command you have run so far in this book -- every file you opened, every process you started, every network packet you sent -- went through the Linux kernel. The kernel is the one piece of software that sits between your programs and the hardware. It manages memory, schedules processes, handles disk I/O, drives network interfaces, and enforces security.

Yet most Linux users never look at the kernel directly. They interact with it through commands, system calls, and virtual filesystems without realizing it. This chapter pulls back the curtain. You will learn what the kernel actually does, how to inspect it, how to load and unload kernel modules, how to tune kernel behavior at runtime, and how to read the kernel's own log messages.

This knowledge is essential for performance tuning, hardware troubleshooting, security hardening, and understanding why things work the way they do.

Try This Right Now

# What kernel are you running?
uname -a

# Kernel version only
uname -r

# How long has this kernel been running?
uptime

# See kernel log messages (most recent)
dmesg | tail -20

# How many kernel modules are loaded?
lsmod | wc -l

# Peek at the kernel's view of your CPU
cat /proc/cpuinfo | head -20

# How much memory does the kernel see?
cat /proc/meminfo | head -10

Kernel vs. Userspace

The most fundamental distinction in Linux is between kernel space and user space.

+--------------------------------------------------+
|                User Space                        |
|                                                  |
|   +--------+  +--------+  +--------+  +------+  |
|   | bash   |  | nginx  |  | python |  | top  |  |
|   +--------+  +--------+  +--------+  +------+  |
|                                                  |
|   Applications, libraries (glibc), utilities     |
|                                                  |
+=======================+=========================+
|        System Call Interface (syscall)           |
+=======================+=========================+
|                                                  |
|                Kernel Space                      |
|                                                  |
|   +----------+  +---------+  +----------+       |
|   | Process   |  | Memory  |  | Network  |       |
|   | Scheduler |  | Manager |  | Stack    |       |
|   +----------+  +---------+  +----------+       |
|                                                  |
|   +----------+  +---------+  +----------+       |
|   | VFS      |  | Device  |  | Security |       |
|   |          |  | Drivers |  | (LSM)    |       |
|   +----------+  +---------+  +----------+       |
|                                                  |
+=======================+=========================+
|              Hardware                            |
|   CPU, RAM, Disk, Network, USB, GPU, ...         |
+--------------------------------------------------+

Why Two Spaces?

• Kernel space has unrestricted access to hardware. A bug here can crash the entire system.
• User space is restricted. A bug in your application cannot (usually) crash the kernel or affect other users.

The CPU enforces this split using hardware protection rings:

• Ring 0: Kernel mode (full hardware access)
• Ring 3: User mode (restricted)

When your program needs something that requires kernel privileges (opening a file, sending a network packet, allocating memory), it makes a system call.

System Calls: The Gateway

A system call (syscall) is how user-space programs request services from the kernel. Every meaningful operation eventually becomes a system call.

  Your Program (user space)
       |
       | printf("hello\n")
       |
       v
  C Library (glibc)
       |
       | write(1, "hello\n", 6)  <-- system call wrapper
       |
       v
  Kernel (kernel space)
       |
       | Actually writes bytes to the terminal device
       |
       v
  Hardware (terminal/screen)

Common System Calls

Watching System Calls with strace

strace lets you see every system call a process makes:

# Trace a simple command
strace ls /tmp 2>&1 | head -30

# Trace a running process
sudo strace -p $(pgrep nginx | head -1) -e trace=read,write

# Count system calls (summary mode)
strace -c ls /tmp

# Trace file-related calls only
strace -e trace=file ls /tmp

# Trace network-related calls only
strace -e trace=network curl -s example.com > /dev/null

# Trace with timestamps
strace -t ls /tmp 2>&1 | head -10

Example output from strace -c ls /tmp:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 25.00    0.000050          10         5           openat
 20.00    0.000040           5         8           mmap
 15.00    0.000030           4         7           close
 10.00    0.000020           3         6           fstat
 10.00    0.000020           4         5           read
  5.00    0.000010          10         1           getdents64
  ...

Think About It: When you run echo "hello", how many system calls happen? Try strace echo "hello" 2>&1 | wc -l to find out. Why are there so many for such a simple command?

Kernel Modules

The Linux kernel is modular. Rather than compiling every possible driver and feature into the kernel image, Linux loads functionality on demand through kernel modules. These are .ko (kernel object) files.

  +----------------------------------+
  |         Linux Kernel             |
  |                                  |
  |  Core (always loaded):           |
  |  - Process scheduler             |
  |  - Memory manager                |
  |  - VFS layer                     |
  |                                  |
  |  Modules (loaded on demand):     |
  |  +----------+ +----------+      |
  |  | ext4.ko  | | e1000.ko |      |
  |  +----------+ +----------+      |
  |  +----------+ +----------+      |
  |  | nf_tables| | usb_hid  |      |
  |  +----------+ +----------+      |
  +----------------------------------+

Listing Loaded Modules

# List all currently loaded modules
lsmod

# Output format:
# Module                  Size  Used by
# nf_tables             303104  0
# e1000                  151552  0
# ext4                   806912  1
# ...

The columns are:

• Module: Module name
• Size: Memory used (bytes)
• Used by: Count of dependents, and which modules depend on it

# Filter for a specific module
lsmod | grep ext4

# Count loaded modules
lsmod | wc -l

Getting Module Information

# Detailed info about a module
modinfo ext4

# Key fields:
# filename:       /lib/modules/.../ext4.ko
# license:        GPL
# description:    Fourth Extended Filesystem
# depends:        jbd2,mbcache,crc16
# parm:           ...  (module parameters)

# Just show the description
modinfo -d ext4

# Show module parameters
modinfo -p ext4

# Show the file path
modinfo -n ext4

Loading and Unloading Modules

# Load a module (resolves dependencies automatically)
sudo modprobe snd_dummy

# Verify it loaded
lsmod | grep snd_dummy

# Unload a module
sudo modprobe -r snd_dummy

# Load with parameters
sudo modprobe loop max_loop=64

WARNING: Be very careful loading and unloading kernel modules on production systems. Unloading a module that is in use can crash the system. modprobe -r will refuse if the module is in use, but forcing removal (rmmod -f) can cause a kernel panic.

Module Dependencies

Modules can depend on other modules. modprobe handles this automatically, but you can see the dependency tree:

# Show what a module depends on
modinfo ext4 | grep depends

# Show the full dependency tree
modprobe --show-depends ext4

Blacklisting Modules

Sometimes you need to prevent a module from loading (conflicting drivers, security):

# Create a blacklist file
sudo tee /etc/modprobe.d/blacklist-example.conf << 'EOF'
# Prevent the nouveau driver from loading (example)
blacklist nouveau
EOF

# After blacklisting, update initramfs
sudo update-initramfs -u      # Debian/Ubuntu
sudo dracut --force            # RHEL/Fedora

Distro Note: Module blacklisting syntax is the same across distributions, but the command to rebuild initramfs differs. Debian/Ubuntu use update-initramfs, RHEL/Fedora use dracut.

Exploring /proc -- The Process Filesystem

/proc is a virtual filesystem. Nothing in it exists on disk -- the kernel generates its contents on the fly when you read them. It is your window into the kernel's state.

System-Wide Information

# Kernel version
cat /proc/version

# CPU information
cat /proc/cpuinfo

# Memory statistics
cat /proc/meminfo

# Uptime (in seconds)
cat /proc/uptime

# Load average
cat /proc/loadavg

# Mounted filesystems
cat /proc/mounts

# Currently active partitions
cat /proc/partitions

# Network statistics
cat /proc/net/dev

# Open file count system-wide
cat /proc/sys/fs/file-nr

# Maximum number of open files
cat /proc/sys/fs/file-max

# Kernel command line (boot parameters)
cat /proc/cmdline

Per-Process Information

Each PID has its own directory (covered in Chapter 10, but here is the kernel-focused view):

# Pick a PID (your own shell)
PID=$$

# Command that started this process
cat /proc/$PID/cmdline | tr '\0' ' '; echo

# Process status (kernel's view)
cat /proc/$PID/status

# Memory map
cat /proc/$PID/maps | head -10

# Open file descriptors
ls -l /proc/$PID/fd/

# Limits applied to this process
cat /proc/$PID/limits

# cgroup membership
cat /proc/$PID/cgroup

# Namespace information
ls -l /proc/$PID/ns/

# Scheduling information
cat /proc/$PID/sched | head -20

Interesting /proc Files

# Random number entropy available
cat /proc/sys/kernel/random/entropy_avail

# Hostname
cat /proc/sys/kernel/hostname

# OS type
cat /proc/sys/kernel/ostype

# Swappiness (how aggressively kernel swaps)
cat /proc/sys/vm/swappiness

# IP forwarding enabled?
cat /proc/sys/net/ipv4/ip_forward

# Maximum number of processes
cat /proc/sys/kernel/pid_max

# Kernel taint flags (non-zero means something unusual)
cat /proc/sys/kernel/tainted

Think About It: /proc files have a size of 0 bytes according to ls -l, yet cat can read content from them. Why? What does this tell you about how /proc works?

Exploring /sys -- The Device Filesystem

/sys (sysfs) is another virtual filesystem, focused on devices and kernel subsystems:

# Block devices (disks)
ls /sys/block/

# Network devices and their MAC addresses
ls /sys/class/net/
cat /sys/class/net/eth0/address 2>/dev/null

# CPU frequency governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null

# Disk queue scheduler
cat /sys/block/sda/queue/scheduler 2>/dev/null

While /proc is a mix of process info and kernel state (and is older, from Linux 1.0), /sys is a cleaner, hierarchical view focused on devices and drivers (introduced in Linux 2.6). Both are virtual -- nothing on disk.

uname: Kernel Identity

# All information at once
uname -a

# Most commonly used flags
uname -r     # Kernel release: 6.1.0-18-amd64
uname -m     # Architecture: x86_64
uname -s     # Kernel name: Linux
uname -n     # Hostname

The kernel version string decoded:

  6.1.0-18-amd64
  | | |  |    |
  | | |  |    +-- Architecture variant
  | | |  +------- Distro patch level
  | | +---------- Patch version
  | +------------ Minor version
  +-------------- Major version

Kernel Parameters with sysctl

sysctl reads and writes kernel parameters at runtime. These correspond to files under /proc/sys/:

# List all kernel parameters
sysctl -a 2>/dev/null | head -20

# Read a specific parameter
sysctl net.ipv4.ip_forward
# Same as: cat /proc/sys/net/ipv4/ip_forward

# Set a parameter temporarily (until reboot)
sudo sysctl net.ipv4.ip_forward=1

# Set a parameter permanently
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-forwarding.conf
sudo sysctl --system    # Reload all sysctl config

Important sysctl Parameters

# Network
sysctl net.ipv4.ip_forward                    # IP routing
sysctl net.ipv4.tcp_syncookies                # SYN flood protection
sysctl net.core.somaxconn                     # Max socket listen backlog
sysctl net.ipv4.tcp_max_syn_backlog           # SYN queue size

# Virtual memory
sysctl vm.swappiness                          # Swap aggressiveness (0-100)
sysctl vm.dirty_ratio                         # % of RAM for dirty pages before sync
sysctl vm.overcommit_memory                   # Memory overcommit policy

# Kernel
sysctl kernel.pid_max                         # Maximum PID value
sysctl kernel.hostname                        # System hostname
sysctl kernel.panic                           # Seconds before reboot on panic (0=hang)

# File system
sysctl fs.file-max                            # Maximum open files system-wide
sysctl fs.inotify.max_user_watches            # inotify watch limit

Practical: Tuning for a Web Server

# Increase connection backlog for high-traffic servers
sudo sysctl net.core.somaxconn=65535
sudo sysctl net.ipv4.tcp_max_syn_backlog=65535

# Increase file descriptor limits
sudo sysctl fs.file-max=2097152

# Increase inotify watches (for file-watching dev tools)
sudo sysctl fs.inotify.max_user_watches=524288

# Make changes permanent
sudo tee /etc/sysctl.d/99-webserver.conf << 'EOF'
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288
EOF

sudo sysctl --system

dmesg: The Kernel Ring Buffer

dmesg displays the kernel ring buffer -- a circular log where the kernel writes messages about hardware detection, driver loading, errors, and other events:

# View all kernel messages
dmesg

# View with human-readable timestamps
dmesg -T

# View with color
dmesg --color=always | less -R

# Show only errors and warnings
dmesg --level=err,warn

# Follow new messages in real time (like tail -f)
dmesg -w

# Show messages since last boot
dmesg -T | head -50

# Clear the ring buffer (root only)
sudo dmesg -c

What to Look for in dmesg

# Hardware detection at boot
dmesg | grep -i "cpu\|memory\|disk\|network\|usb"

# Disk/storage messages
dmesg | grep -i "sd[a-z]\|nvme\|ext4\|xfs"

# Network interface detection
dmesg | grep -i "eth\|ens\|wlan\|link"

# Errors (these are important!)
dmesg --level=err

# Out of memory events
dmesg | grep -i "oom\|out of memory"

# USB device events
dmesg | grep -i usb

# Firewall drops (if logging is enabled)
dmesg | grep -i "iptables\|nftables\|DROP"

dmesg and journalctl

On systemd systems, kernel messages are also captured by journald:

# Kernel messages via journalctl
journalctl -k

# Kernel messages from current boot
journalctl -k -b 0

# Kernel messages from previous boot
journalctl -k -b -1

# Follow kernel messages
journalctl -kf

Think About It: The kernel ring buffer has a fixed size (typically 256KB-1MB). What happens when it fills up? How does this affect your ability to investigate boot problems hours after the system started?

Debug This: Identifying a Missing Driver

A new USB device is plugged in but does not work:

# Step 1: Check dmesg for recent USB events
dmesg -T | tail -30

# You might see something like:
# [timestamp] usb 1-1: new high-speed USB device number 4
# [timestamp] usb 1-1: New USB device found, idVendor=1234, idProduct=5678
# [timestamp] usb 1-1: New USB device strings: Mfr=1, Product=2, Serial=3

# Step 2: Check if a driver was loaded
dmesg -T | grep -i "driver\|module\|bound"

# Step 3: Find the vendor/product ID
lsusb
# Bus 001 Device 004: ID 1234:5678 Unknown Device

# Step 4: Search for a matching module
find /lib/modules/$(uname -r) -name "*.ko" | xargs modinfo 2>/dev/null | grep -B5 "1234"

# Step 5: Check if the module exists but isn't loaded
modprobe --show-depends relevant_module

# Step 6: Try loading it manually
sudo modprobe relevant_module

# Step 7: Check dmesg again
dmesg -T | tail -10

Hands-On: Kernel Exploration Lab

# 1. Determine your exact kernel version and architecture
uname -r
uname -m

# 2. How many system calls does the kernel support?
# (On x86_64 systems)
grep -c "^[0-9]" /usr/include/asm/unistd_64.h 2>/dev/null || \
ausyscall --dump 2>/dev/null | wc -l

# 3. What kernel modules are loaded for your filesystem?
lsmod | grep -E "ext4|xfs|btrfs"

# 4. What is the kernel's view of your disks?
cat /proc/partitions

# 5. Check kernel taint status (0 = clean, non-zero = something unusual)
cat /proc/sys/kernel/tainted

# 6. See the kernel command line (how it was booted)
cat /proc/cmdline

# 7. What interrupts are firing?
cat /proc/interrupts | head -20

# 8. Check the current swappiness
sysctl vm.swappiness

# 9. Temporarily change swappiness and verify
sudo sysctl vm.swappiness=10
sysctl vm.swappiness
# Reset it
sudo sysctl vm.swappiness=60

# 10. Trace system calls of a simple command
strace -c date 2>&1

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 13 Recap: The Kernel Up Close                           |
|------------------------------------------------------------------|
|                                                                  |
|  - The kernel manages hardware, processes, memory, and I/O.     |
|  - User space programs access the kernel via system calls.       |
|  - strace lets you watch system calls in real time.              |
|  - Kernel modules (.ko) load functionality on demand.            |
|  - lsmod, modprobe, modinfo manage modules.                     |
|  - /proc is a virtual filesystem exposing kernel state.          |
|  - /sys exposes device and driver information hierarchically.    |
|  - uname -r shows your kernel version.                           |
|  - sysctl reads and tunes kernel parameters at runtime.         |
|  - dmesg shows the kernel ring buffer (hardware, drivers, errors)|
|  - Kernel parameters can be made permanent in /etc/sysctl.d/.   |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: System Call Counting

Use strace -c on three different commands: ls /tmp, cat /etc/passwd, and curl -s example.com > /dev/null. Compare the number and types of system calls. Which command makes the most? Why?

Exercise 2: Module Investigation

Run lsmod and pick three modules you do not recognize. Use modinfo to learn about each one: what does it do, what license is it under, and what parameters does it accept?

Exercise 3: /proc Scavenger Hunt

Using only files in /proc, determine: (a) how many CPUs/cores the kernel sees, (b) total installed RAM, (c) current load average, (d) the kernel's command line boot parameters, and (e) how many file descriptors are currently in use system-wide.

Exercise 4: sysctl Tuning

Read the current values of vm.swappiness, net.ipv4.ip_forward, and fs.file-max. Change vm.swappiness to 10, verify the change took effect, then set it back to the original value. Write the appropriate line for /etc/sysctl.d/ to make it permanent.

Bonus Challenge

Write a script called kernel-report.sh that outputs a comprehensive report: kernel version, architecture, uptime, number of loaded modules, top 5 modules by memory usage, number of running processes, file descriptor usage, and any errors in the kernel ring buffer from the last hour. Format the output cleanly with headers and dividers.

Boot Process Demystified

Why This Matters

You press the power button. Sixty seconds later, you are staring at a login prompt. What just happened in those sixty seconds?

Understanding the boot process is not academic trivia. When a server refuses to boot, when a kernel update goes wrong, when you need to recover a system with a corrupted root filesystem, when a misconfigured GRUB entry leaves you staring at a blinking cursor -- that is when this knowledge saves you. Every minute a production server is down costs money. Knowing the boot sequence means you can diagnose where it broke and fix it.

This chapter walks through every stage, from the moment electricity hits the motherboard to the moment systemd presents you with a login prompt. You will learn to configure GRUB, inspect initramfs, analyze boot performance, and rescue a system that will not boot.

Try This Right Now

# How long did your system take to boot?
systemd-analyze

# Breakdown by stage
systemd-analyze blame | head -15

# Visual chain of the boot process
systemd-analyze critical-chain

# What target (runlevel) are you running?
systemctl get-default

# When was this system last booted?
who -b

# See the kernel's boot parameters
cat /proc/cmdline

# Check for boot errors
journalctl -b 0 -p err

The Boot Sequence: Bird's Eye View

+------------------------------------------------------------------+
|  1. FIRMWARE (BIOS or UEFI)                                     |
|     - Hardware initialization (POST)                             |
|     - Find and load the bootloader                               |
+------------------------------------------------------------------+
         |
         v
+------------------------------------------------------------------+
|  2. BOOTLOADER (GRUB2)                                          |
|     - Present boot menu                                          |
|     - Load the kernel and initramfs into memory                  |
+------------------------------------------------------------------+
         |
         v
+------------------------------------------------------------------+
|  3. KERNEL                                                       |
|     - Initialize hardware, memory, CPU                           |
|     - Mount initramfs as temporary root                          |
|     - Find and mount the real root filesystem                    |
|     - Start PID 1                                                |
+------------------------------------------------------------------+
         |
         v
+------------------------------------------------------------------+
|  4. INIT SYSTEM (systemd, PID 1)                                |
|     - Mount remaining filesystems                                |
|     - Start services according to the target                    |
|     - Present login prompt                                       |
+------------------------------------------------------------------+

Let us examine each stage in detail.

Stage 1: Firmware -- BIOS and UEFI

When you press the power button, the CPU starts executing code from a chip on the motherboard -- the firmware.

BIOS (Legacy)

The Basic Input/Output System is the older standard (1980s):

1. POST (Power-On Self-Test) -- checks CPU, RAM, and basic hardware
2. Looks for a bootable device based on the boot order (hard disk, USB, network)
3. Reads the first 512 bytes of the boot device -- the Master Boot Record (MBR)
4. Executes the bootloader code found in the MBR

MBR limitations:

• Only 446 bytes for bootloader code (tiny!)
• Maximum disk size: 2 TB
• Maximum 4 primary partitions

UEFI (Modern)

The Unified Extensible Firmware Interface is the modern replacement:

1. POST -- same hardware checks
2. Reads the EFI System Partition (ESP) -- a FAT32 partition, usually mounted at /boot/efi
3. Runs the bootloader EFI application (e.g., grubx64.efi)
4. Supports Secure Boot (only runs cryptographically signed bootloaders)

UEFI advantages:

• Supports disks larger than 2 TB (GPT partitioning)
• Faster boot (can skip legacy compatibility)
• Secure Boot protects against rootkits
• Can boot directly from EFI applications (no MBR needed)

# Check if your system uses UEFI or BIOS
ls /sys/firmware/efi 2>/dev/null && echo "UEFI boot" || echo "Legacy BIOS boot"

# If UEFI, see the boot entries
efibootmgr -v 2>/dev/null

# See the EFI System Partition
mount | grep efi
ls /boot/efi/EFI/ 2>/dev/null

Think About It: Why does the EFI System Partition use FAT32 and not ext4 or XFS? (Hint: the firmware needs to read it before any Linux driver is loaded.)

Stage 2: The GRUB2 Bootloader

GRUB2 (GRand Unified Bootloader version 2) is the standard bootloader on most Linux distributions. Its job is to load the kernel and initramfs into memory.

What GRUB Does

1. Presents a boot menu (if configured to show one)
2. Lets you select which kernel to boot, edit boot parameters, or enter a recovery shell
3. Loads the selected kernel image (vmlinuz-*) into memory
4. Loads the initial RAM filesystem (initramfs-* or initrd-*) into memory
5. Passes control to the kernel with the specified boot parameters

GRUB Configuration

# Main GRUB config file (DO NOT edit directly)
ls -l /boot/grub/grub.cfg      # Debian/Ubuntu
ls -l /boot/grub2/grub.cfg     # RHEL/Fedora

# Edit GRUB defaults instead
cat /etc/default/grub

The /etc/default/grub file contains the settings you should modify:

# Key settings in /etc/default/grub:

GRUB_DEFAULT=0                    # Default menu entry (0 = first)
GRUB_TIMEOUT=5                    # Seconds to show menu (0 = skip, -1 = wait forever)
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"  # Kernel params for default boot
GRUB_CMDLINE_LINUX=""             # Kernel params for ALL boots
GRUB_DISABLE_RECOVERY="false"    # Show recovery entries?

Common kernel parameters you might add:

Regenerating GRUB Configuration

After changing /etc/default/grub, regenerate the actual config:

# Debian/Ubuntu
sudo update-grub

# RHEL/Fedora
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# For UEFI systems on RHEL/Fedora
sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

Distro Note: Debian/Ubuntu provide the update-grub convenience wrapper. RHEL/Fedora require the full grub2-mkconfig command. Both do the same thing -- regenerate /boot/grub/grub.cfg from templates and /etc/default/grub.

WARNING: Never edit /boot/grub/grub.cfg directly. Your changes will be overwritten the next time update-grub runs (e.g., during a kernel update). Always edit /etc/default/grub and regenerate.

GRUB Interactive Editing

At the GRUB menu (press Shift during boot on BIOS, or Esc on UEFI):

• Press e to edit the selected entry's boot parameters
• Find the line starting with linux (the kernel command line)
• Add or modify parameters
• Press Ctrl+X or F10 to boot with those changes (temporary, one-time)
• Press c to drop to a GRUB command-line shell

This is invaluable for recovery. For example, to boot into a root shell:

# At the GRUB edit screen, add to the end of the 'linux' line:
init=/bin/bash
# Then press Ctrl+X to boot

GRUB Installation

# Install GRUB to a disk's MBR (BIOS)
sudo grub-install /dev/sda

# Install GRUB to EFI System Partition
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi

# Verify installation
sudo grub-install --recheck /dev/sda

Stage 3: The Kernel Loads

Once GRUB loads the kernel into memory and passes control, the kernel takes over:

Kernel Initialization Sequence

  1. Decompress kernel image (vmlinuz -> vmlinux)
  2. Initialize CPU and memory management
  3. Initialize essential hardware (console, timer, interrupts)
  4. Mount initramfs as temporary root (/)
  5. Run /init from initramfs
  6. initramfs loads necessary drivers (storage, filesystem)
  7. Mount the real root filesystem
  8. Pivot root from initramfs to real filesystem
  9. Execute /sbin/init (systemd) as PID 1

You can watch this process in dmesg:

# First few kernel messages -- decompression and initialization
dmesg | head -30

# Look for the root filesystem mounting
dmesg | grep -i "root\|mount\|ext4\|xfs"

# Look for the transition to userspace
dmesg | grep -i "init\|systemd\|pid 1"

initramfs: The Bridge

The kernel cannot mount your root filesystem without the right drivers. But those drivers are stored ON the root filesystem. Chicken-and-egg problem.

The solution is initramfs (initial RAM filesystem) -- a compressed archive containing just enough to find and mount the root filesystem:

+------------------------------------------+
|  initramfs contents:                     |
|  - Filesystem drivers (ext4, xfs, etc.)  |
|  - Storage drivers (SCSI, NVMe, RAID)   |
|  - LVM tools (if root is on LVM)        |
|  - LUKS tools (if root is encrypted)    |
|  - /init script (orchestrates mounting)  |
|  - udev rules for device detection      |
+------------------------------------------+

# See your initramfs files
ls -lh /boot/initrd.img-*    # Debian/Ubuntu
ls -lh /boot/initramfs-*     # RHEL/Fedora

# Examine initramfs contents
lsinitramfs /boot/initrd.img-$(uname -r) | head -30              # Debian/Ubuntu
lsinitrd /boot/initramfs-$(uname -r).img | head -30              # RHEL/Fedora

# Rebuild initramfs (if you've changed modules or drivers)
sudo update-initramfs -u -k $(uname -r)    # Debian/Ubuntu
sudo dracut --force                          # RHEL/Fedora

Think About It: If your root filesystem is on a simple SATA drive with ext4, the kernel might have those drivers built-in and could theoretically skip initramfs. But if your root is on an encrypted LVM volume on top of software RAID, initramfs is absolutely essential. Why?

Stage 4: systemd Takes Over (PID 1)

Once the kernel mounts the root filesystem, it executes /sbin/init, which on modern distributions is a symlink to systemd. systemd becomes PID 1 -- the parent of all processes.

Boot Targets (Modern Runlevels)

systemd uses targets instead of the old SysVinit runlevels:

+----------+-----------------------+-------------------------------+
| Runlevel | systemd Target        | Description                   |
+----------+-----------------------+-------------------------------+
|    0     | poweroff.target       | Halt the system               |
|    1     | rescue.target         | Single-user mode, root shell  |
|    2     | multi-user.target *   | (Debian: same as 3)           |
|    3     | multi-user.target     | Full multi-user, no GUI       |
|    4     | multi-user.target     | Unused (customizable)         |
|    5     | graphical.target      | Multi-user with GUI           |
|    6     | reboot.target         | Reboot                        |
+----------+-----------------------+-------------------------------+

# What target is currently active?
systemctl get-default

# Change the default target
sudo systemctl set-default multi-user.target    # Server (no GUI)
sudo systemctl set-default graphical.target     # Desktop (with GUI)

# Switch to a different target right now (without reboot)
sudo systemctl isolate rescue.target            # Single-user mode
sudo systemctl isolate multi-user.target        # Back to normal

What systemd Does During Boot

# See the dependency order of boot
systemd-analyze critical-chain

# Output (example):
# graphical.target @12.345s
# └─multi-user.target @12.300s
#   └─network-online.target @10.200s
#     └─NetworkManager-wait-online.service @5.100s +5.099s
#       └─NetworkManager.service @3.200s +1.899s
#         └─dbus.service @2.800s +0.399s
#           └─basic.target @2.700s
#             └─sockets.target @2.699s

Analyzing Boot Performance

# Total boot time breakdown
systemd-analyze
# Output: Startup finished in 2.5s (firmware) + 3.1s (loader) +
#         4.2s (kernel) + 12.3s (userspace) = 22.1s

# Which services took the longest?
systemd-analyze blame | head -20

# Generate an SVG boot chart (visual timeline)
systemd-analyze plot > boot-chart.svg
# Open in a browser to see a graphical timeline

# Check for units that are slowing down boot
systemd-analyze critical-chain --no-pager

# See target dependencies
systemctl list-dependencies multi-user.target | head -30

Boot Log

# View complete boot log for this boot
journalctl -b 0

# View boot log for the previous boot
journalctl -b -1

# Show only errors from this boot
journalctl -b 0 -p err

# Show kernel messages from boot
journalctl -b 0 -k

# List all recorded boots
journalctl --list-boots

Troubleshooting Boot Failures

Identifying Where Boot Failed

+----------------------------------------------------+
|  Symptom                    | Stage That Failed     |
|-----------------------------|-----------------------|
|  No display, no beep       | Hardware/BIOS         |
|  BIOS screen, then blank   | Bootloader (GRUB)     |
|  GRUB menu appears,        | Kernel loading or     |
|  then kernel panic          | initramfs             |
|  Kernel loads, services     | systemd / services    |
|  fail, no login             |                       |
|  Login appears but can't    | Authentication / PAM  |
|  log in                     |                       |
+----------------------------------------------------+

Rescue Mode

If the system boots far enough for GRUB to work:

Method 1: GRUB Kernel Parameter Edit

1. At the GRUB menu, press e
2. Find the line starting with linux
3. Add systemd.unit=rescue.target at the end
4. Press Ctrl+X to boot
5. You will get a root shell (may require root password)

Method 2: Emergency Mode

For more serious problems (filesystem issues):

1. At GRUB, press e
2. Add systemd.unit=emergency.target to the kernel line
3. Press Ctrl+X
4. Only the root filesystem is mounted (read-only)

# In emergency mode, remount root as read-write
mount -o remount,rw /

# Fix the problem (e.g., edit /etc/fstab)
vi /etc/fstab

# Reboot
reboot

Method 3: init=/bin/bash

When even systemd will not start:

1. At GRUB, press e
2. Replace ro quiet splash with rw init=/bin/bash
3. Press Ctrl+X
4. You drop directly to a bash shell as root, no authentication

# Root filesystem may be read-only; remount it
mount -o remount,rw /

# Reset a forgotten root password
passwd root

# Fix other issues as needed

# When done, remount read-only and reboot
mount -o remount,ro /
exec /sbin/reboot -f

WARNING: init=/bin/bash bypasses ALL authentication. Anyone with physical access to the GRUB menu can become root. This is why physical security matters, and why some organizations configure GRUB passwords.

Fixing a Broken GRUB

If GRUB itself is corrupted:

# Boot from a live USB/ISO
# Mount the installed system's partitions:
sudo mount /dev/sda2 /mnt          # Root partition
sudo mount /dev/sda1 /mnt/boot/efi # EFI partition (if UEFI)
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys

# Chroot into the installed system
sudo chroot /mnt

# Reinstall GRUB
grub-install /dev/sda              # BIOS
grub-install --target=x86_64-efi --efi-directory=/boot/efi  # UEFI

# Regenerate config
update-grub                        # Debian/Ubuntu
grub2-mkconfig -o /boot/grub2/grub.cfg  # RHEL/Fedora

# Exit chroot and reboot
exit
sudo reboot

Fixing a Broken /etc/fstab

A common boot failure: you added an entry to /etc/fstab with a typo, and now the system drops to an emergency shell on boot.

# In emergency mode:
# The error message will tell you which line failed
# Mount root as read-write
mount -o remount,rw /

# Edit fstab
vi /etc/fstab

# Either fix the typo or comment out the problematic line
# Save and reboot
reboot

Think About It: Why does a bad /etc/fstab entry prevent booting, even if the bad entry is for a non-root filesystem like /data? (Hint: look at the mount options -- is nofail set?)

Pro tip: always add nofail to non-critical mount entries in /etc/fstab:

/dev/sdb1  /data  ext4  defaults,nofail  0  2

This tells systemd: "Try to mount this, but do not fail the boot if it cannot be mounted."

Hands-On: Boot Analysis

# 1. Analyze your boot time
systemd-analyze

# 2. Find the slowest services
systemd-analyze blame | head -10

# 3. See the critical chain (the longest dependency path)
systemd-analyze critical-chain

# 4. Check for failed services during boot
systemctl --failed

# 5. See what the kernel was told at boot
cat /proc/cmdline

# 6. Examine the initramfs for your current kernel
file /boot/initrd.img-$(uname -r) 2>/dev/null || \
file /boot/initramfs-$(uname -r).img 2>/dev/null

# 7. Check GRUB configuration
cat /etc/default/grub

# 8. List available kernels in GRUB
grep menuentry /boot/grub/grub.cfg 2>/dev/null | head -10 || \
grep menuentry /boot/grub2/grub.cfg 2>/dev/null | head -10

# 9. Check for boot errors
journalctl -b 0 -p err --no-pager | head -30

# 10. See how many boots have been recorded
journalctl --list-boots

GRUB Password Protection

To prevent unauthorized users from editing GRUB entries (and bypassing authentication with init=/bin/bash):

# Generate a GRUB password hash
grub-mkpasswd-pbkdf2
# Enter and confirm a password
# Output: grub.pbkdf2.sha512.10000.HASH...

# Add to /etc/grub.d/40_custom
sudo tee -a /etc/grub.d/40_custom << 'EOF'
set superusers="admin"
password_pbkdf2 admin grub.pbkdf2.sha512.10000.YOUR_HASH_HERE
EOF

# Regenerate GRUB config
sudo update-grub

Now editing GRUB entries at boot will require the GRUB password.

Debug This: System Drops to Emergency Shell

After a reboot, the system drops to:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again
to boot into default mode.
Give root password for maintenance:

Diagnosis steps:

# 1. Enter root password (if set)

# 2. Check the journal for the cause
journalctl -xb --no-pager | grep -E "Failed|Error|error" | head -20

# 3. Common cause #1: /etc/fstab error
# Look for filesystem mount failures
systemctl --failed
# If a mount unit failed:
cat /etc/fstab
# Fix the bad line

# 4. Common cause #2: Disk UUID changed
blkid
# Compare UUIDs to what is in /etc/fstab

# 5. Common cause #3: Filesystem needs fsck
# The error message often says which device
fsck /dev/sda2

# 6. Remount root as read-write to make changes
mount -o remount,rw /

# 7. Make your fix, then:
systemctl default
# Or simply:
reboot

The Complete Boot Timeline

Here is the full sequence with approximate times for a modern SSD-based system:

  T+0.0s   Power button pressed
  T+0.5s   UEFI POST completes
  T+1.0s   UEFI finds and loads GRUB
  T+1.5s   GRUB loads kernel + initramfs
  T+2.0s   Kernel decompresses and initializes
  T+3.0s   initramfs mounts real root filesystem
  T+3.5s   systemd starts as PID 1
  T+4.0s   systemd mounts filesystems from /etc/fstab
  T+5.0s   systemd starts udev (device manager)
  T+6.0s   Network services start
  T+8.0s   Login services start (sshd, getty)
  T+10.0s  System is fully booted
            (graphical target may take longer)

# Verify this timeline on your system
systemd-analyze
# Output: Startup finished in 1.8s (firmware) + 2.3s (loader) +
#         3.5s (kernel) + 8.7s (userspace) = 16.3s

What Just Happened?

+------------------------------------------------------------------+
|  Chapter 14 Recap: Boot Process Demystified                      |
|------------------------------------------------------------------|
|                                                                  |
|  - Boot sequence: Firmware -> GRUB -> Kernel -> systemd          |
|  - BIOS is legacy (MBR, 2TB limit); UEFI is modern (GPT).       |
|  - GRUB2 loads the kernel and initramfs into memory.             |
|  - Edit /etc/default/grub, then run update-grub.                |
|  - initramfs provides drivers needed to mount the root FS.       |
|  - The kernel starts systemd as PID 1.                           |
|  - systemd targets replaced SysVinit runlevels.                  |
|  - systemd-analyze shows boot timing and bottlenecks.            |
|  - Rescue mode: add systemd.unit=rescue.target to kernel line.   |
|  - Emergency recovery: init=/bin/bash bypasses all auth.         |
|  - GRUB passwords prevent unauthorized boot parameter edits.     |
|  - Always use 'nofail' for non-critical mounts in /etc/fstab.   |
|  - journalctl -b 0 -p err shows boot errors.                    |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Boot Time Analysis

Run systemd-analyze blame | head -20 and identify the three slowest services. Research what each one does. Could any of them be disabled on a server that does not need a GUI?

Exercise 2: Kernel Command Line

Read /proc/cmdline. Look up each parameter. Try temporarily adding quiet (if not present) or removing it (if present) by editing the GRUB entry at boot time (press e, modify, Ctrl+X). Notice the difference in boot verbosity.

Exercise 3: initramfs Exploration

List the contents of your initramfs (lsinitramfs or lsinitrd). Find the filesystem driver modules included. Find the /init script. What does it do?

Exercise 4: Target Practice

Check your current default target (systemctl get-default). If it is graphical.target, switch to multi-user.target and reboot. Log in via the text console. Switch back and reboot again. Compare boot times between the two.

Bonus Challenge

Intentionally break your boot process in a VM (not on a real machine!) by adding a nonexistent UUID to /etc/fstab. Boot the VM and practice recovering:

1. Use GRUB edit mode to boot into rescue mode
2. Fix the /etc/fstab entry
3. Reboot and verify normal boot

Then, practice the GRUB recovery process:

1. From a live USB/ISO, mount the installed system's partitions
2. Chroot into the installed system
3. Reinstall and reconfigure GRUB
4. Reboot into the repaired system

systemd: The Init System

Why This Matters

Picture this: you deploy a web application at 2 AM, and the server reboots unexpectedly. When it comes back up, your database starts before the network is ready, your app starts before the database is listening, and your reverse proxy starts before your app is responding. Nothing works. Users see errors. Your phone rings.

This is the exact problem an init system solves. It is the very first process that runs on your Linux system (PID 1), and it is responsible for starting everything else in the right order, keeping services alive, and shutting things down cleanly. On virtually every modern Linux distribution, that init system is systemd.

Whether you are a developer deploying applications, a sysadmin managing servers, or someone learning Linux for the first time, understanding systemd is non-negotiable. It controls how your system boots, how services run, how logs are collected, and how your system shuts down.

Try This Right Now

Open a terminal and run these commands. No setup required:

# What is PID 1 on your system?
ps -p 1 -o comm=

# How long has your system been running?
systemctl status --no-pager | head -5

# List all running services
systemctl list-units --type=service --state=running --no-pager

# What target (runlevel) is your system in?
systemctl get-default

If ps -p 1 printed systemd, you are running systemd. That covers Ubuntu, Fedora, Debian, RHEL, Arch, openSUSE, and nearly every other mainstream distribution.

What Is an Init System?

When the Linux kernel finishes its own initialization, it does one final thing: it launches a single userspace process. This process gets PID 1, and it becomes the ancestor of every other process on the system.

This PID 1 process is the init system, and it has several critical responsibilities:

1. Start system services in the correct order (networking, logging, databases, etc.)
2. Manage dependencies between services (the database needs the network first)
3. Supervise running services and restart them if they crash
4. Handle system state transitions (booting, shutting down, rebooting)
5. Reap orphaned processes (adopt zombie children whose parents died)

+----------------------------------------------------------+
|                    Linux Kernel                           |
|  (hardware init, drivers, mount root filesystem)         |
+---------------------------+------------------------------+
                            |
                            v
                    +-------+-------+
                    |   PID 1       |
                    |   (systemd)   |
                    +-------+-------+
                            |
              +-------------+-------------+
              |             |             |
              v             v             v
        +---------+   +---------+   +---------+
        | sshd    |   | nginx   |   | cron    |
        | (svc)   |   | (svc)   |   | (svc)   |
        +---------+   +---------+   +---------+

A Brief History: SysVinit to systemd

SysVinit (The Old Way)

For decades, Linux used SysVinit, inherited from Unix System V. It worked like this:

• Shell scripts in /etc/init.d/ controlled each service
• Scripts were symlinked into numbered "runlevel" directories (/etc/rc3.d/, etc.)
• Services started sequentially, one after another
• The naming convention (S20sshd, S30apache) controlled start order

# Old SysVinit style (you may still see this on older systems)
/etc/init.d/apache2 start
/etc/init.d/apache2 stop
/etc/init.d/apache2 restart

The problems with SysVinit were real:

• Slow boot times because services started one at a time
• No dependency tracking — just numbered ordering and hope
• No process supervision — if a service crashed, nobody restarted it
• Shell scripts everywhere — fragile, inconsistent, hard to debug

systemd (The Modern Way)

Lennart Poettering and Kay Sievers created systemd in 2010. It was controversial (many Unix purists objected to its scope), but it solved genuine problems:

• Parallel service startup dramatically reduced boot times
• Declarative unit files replaced fragile shell scripts
• Dependency management ensured correct startup order
• Process supervision with automatic restart on failure
• Unified logging via the journal (journald)
• On-demand activation via socket and D-Bus activation

By 2015, every major distribution had adopted systemd.

Think About It: Why would parallel service startup require explicit dependency management? What could go wrong if you just started everything at the same time without tracking which services depend on which?

systemctl: Your Primary Interface

systemctl is the command you will use most often to interact with systemd. Think of it as the control panel for everything running on your system.

Starting and Stopping Services

# Start a service (takes effect immediately)
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service (stop + start)
sudo systemctl restart nginx

# Reload a service configuration without full restart
# (not all services support this)
sudo systemctl reload nginx

# Reload if supported, otherwise restart
sudo systemctl reload-or-restart nginx

Enabling and Disabling Services

Starting a service only affects the current session. If you reboot, it will not start automatically unless you enable it:

# Enable a service to start at boot
sudo systemctl enable nginx

# Disable a service from starting at boot
sudo systemctl disable nginx

# Enable AND start in one command
sudo systemctl enable --now nginx

# Disable AND stop in one command
sudo systemctl disable --now nginx

When you enable a service, systemd creates a symlink in the appropriate target directory. When you disable it, that symlink is removed.

# See what enabling actually does
sudo systemctl enable --now nginx
# Output: Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service
#         -> /usr/lib/systemd/system/nginx.service

Checking Service Status

# Detailed status of a service
systemctl status nginx

Here is what typical output looks like:

● nginx.service - A high performance web server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
     Active: active (running) since Mon 2025-03-10 14:22:01 UTC; 3h ago
       Docs: man:nginx(8)
    Process: 1234 ExecStartPre=/usr/bin/nginx -t -q (code=exited, status=0/SUCCESS)
   Main PID: 1235 (nginx)
      Tasks: 3 (limit: 4915)
     Memory: 8.2M
        CPU: 120ms
     CGroup: /system.slice/nginx.service
             ├─1235 "nginx: master process /usr/bin/nginx"
             ├─1236 "nginx: worker process"
             └─1237 "nginx: worker process"

Mar 10 14:22:01 server01 systemd[1]: Starting nginx.service...
Mar 10 14:22:01 server01 systemd[1]: Started nginx.service.

Let us break down each line:

Quick Status Checks

Sometimes you just need a yes/no answer:

# Is it running?
systemctl is-active nginx
# Output: active

# Is it enabled at boot?
systemctl is-enabled nginx
# Output: enabled

# Has it failed?
systemctl is-failed nginx
# Output: active   (meaning "not failed")

These commands return exit codes you can use in scripts:

if systemctl is-active --quiet nginx; then
    echo "nginx is running"
else
    echo "nginx is NOT running"
fi

Hands-On: Exploring Your System's Services

Let us explore what is running on your system right now.

Step 1: List All Running Services

systemctl list-units --type=service --state=running --no-pager

You will see output like:

UNIT                      LOAD   ACTIVE SUB     DESCRIPTION
cron.service              loaded active running Regular background program processing
dbus.service              loaded active running D-Bus System Message Bus
NetworkManager.service    loaded active running Network Manager
sshd.service              loaded active running OpenSSH Daemon
systemd-journald.service  loaded active running Journal Service
systemd-udevd.service     loaded active running Rule-based Manager for Device Events
...

Step 2: List Failed Services

systemctl list-units --type=service --state=failed --no-pager

On a healthy system, this should be empty. If you see failures, investigate with systemctl status <unit-name>.

Step 3: List All Installed Services (Running or Not)

systemctl list-unit-files --type=service --no-pager

This shows every service unit file installed on your system and whether it is enabled, disabled, static, or masked.

Step 4: View a Service's Unit File

systemctl cat sshd.service

This prints the actual unit file contents. We will study unit file anatomy in Chapter 16.

Distro Note: On Ubuntu/Debian, the SSH service is called ssh.service. On Fedora/RHEL/Arch, it is sshd.service. When in doubt, use tab completion: systemctl status ssh<TAB>.

Unit Types: Not Just Services

systemd does not just manage services. It manages many types of "units." Each unit type handles a different kind of system resource:

Listing Different Unit Types

# List all active timers
systemctl list-timers --no-pager

# List all mount units
systemctl list-units --type=mount --no-pager

# List all socket units
systemctl list-units --type=socket --no-pager

# List all targets
systemctl list-units --type=target --no-pager

Service Units

These are the most common. They manage long-running daemons:

systemctl list-units --type=service --no-pager | head -20

Socket Units

Socket units enable socket activation: systemd listens on a socket and only starts the actual service when a connection arrives. This saves resources.

# See which sockets systemd is listening on
systemctl list-sockets --no-pager

LISTEN                        UNIT                     ACTIVATES
/run/dbus/system_bus_socket   dbus.socket              dbus.service
/run/systemd/journal/socket   systemd-journald.socket  systemd-journald.service
[::]:22                       sshd.socket              sshd.service

Timer Units

Timer units are systemd's replacement for cron jobs. We will cover these in detail in Chapters 16 and 24.

# List all active timers and when they fire next
systemctl list-timers --all --no-pager

Mount Units

Every entry in /etc/fstab gets automatically converted to a mount unit:

# See mount units
systemctl list-units --type=mount --no-pager

Think About It: Why would systemd want to manage mount points as units instead of just reading /etc/fstab the old-fashioned way? Think about dependencies: what if a service needs a specific filesystem to be mounted before it can start?

Targets: The New Runlevels

In SysVinit, runlevels (0-6) defined what state the system was in. systemd replaces runlevels with targets -- units that group other units together.

Runlevel to Target Mapping

Checking and Changing the Default Target

# What target does your system boot into?
systemctl get-default
# Output: graphical.target   (desktop) or multi-user.target (server)

# Change default to text/server mode
sudo systemctl set-default multi-user.target

# Change default to graphical/desktop mode
sudo systemctl set-default graphical.target

Switching Targets at Runtime

# Switch to rescue mode (single-user, for recovery)
sudo systemctl isolate rescue.target

# Switch to multi-user (text mode)
sudo systemctl isolate multi-user.target

# Switch to graphical mode
sudo systemctl isolate graphical.target

WARNING: systemctl isolate rescue.target will kill most running services and drop you to a root shell. Do not run this on a remote server unless you have console access.

Understanding Target Dependencies

Targets are like dependency trees. multi-user.target depends on basic.target, which depends on sysinit.target, which depends on local-fs.target and others:

graphical.target
    └── multi-user.target
            └── basic.target
                    ├── sockets.target
                    ├── timers.target
                    ├── paths.target
                    ├── slices.target
                    └── sysinit.target
                            ├── local-fs.target
                            ├── swap.target
                            └── cryptsetup.target

You can visualize this:

# Show what a target "wants" (its dependencies)
systemctl list-dependencies multi-user.target --no-pager

# Show the full boot dependency tree
systemctl list-dependencies default.target --no-pager

Hands-On: Managing a Real Service

Let us work through a complete service management workflow using the SSH daemon.

Step 1: Check Current Status

systemctl status sshd.service

Distro Note: Use ssh.service on Debian/Ubuntu.

Step 2: Stop the Service

sudo systemctl stop sshd.service

WARNING: If you are connected via SSH, do NOT stop sshd. Your existing connection will survive, but you will not be able to open new ones. Use a different service for practice if you are remote.

Step 3: Verify It Stopped

systemctl is-active sshd.service
# Output: inactive

systemctl status sshd.service
# Active line now shows: inactive (dead)

Step 4: Start It Again

sudo systemctl start sshd.service

systemctl is-active sshd.service
# Output: active

Step 5: Check Boot Configuration

systemctl is-enabled sshd.service
# Output: enabled

Step 6: View Recent Logs

# Last 20 log entries for sshd
journalctl -u sshd.service -n 20 --no-pager

Masking Services: The Nuclear Option

Sometimes disabling a service is not enough. Another service or a system update might re-enable it. Masking a service makes it completely impossible to start:

# Mask a service (symlinks unit file to /dev/null)
sudo systemctl mask bluetooth.service

# Try to start it — it will refuse
sudo systemctl start bluetooth.service
# Failed to start bluetooth.service: Unit bluetooth.service is masked.

# Unmask it when you want to allow it again
sudo systemctl unmask bluetooth.service

Masking is useful when:

• You want to ensure a service never runs (security hardening)
• Two services conflict and you want to permanently disable one
• You are troubleshooting and want to eliminate a service entirely

# See what a mask looks like
ls -la /etc/systemd/system/bluetooth.service
# /etc/systemd/system/bluetooth.service -> /dev/null

Debug This: Why Won't My Service Start?

You install nginx and try to start it, but it fails:

sudo systemctl start nginx.service
# Job for nginx.service failed because the control process exited with error code.

Here is your debugging workflow:

Step 1: Check Status

systemctl status nginx.service --no-pager -l

The -l flag prevents line truncation. Look for error messages in the log section at the bottom.

Step 2: Check the Journal

journalctl -u nginx.service -n 50 --no-pager

Look for lines marked with err or crit priority.

Step 3: Check Configuration Syntax

# For nginx specifically
sudo nginx -t

Step 4: Check Port Conflicts

# Is something else using port 80?
sudo ss -tlnp | grep :80

Step 5: Check File Permissions

# Can the service user read its config?
ls -la /etc/nginx/nginx.conf

# Can it write to its log directory?
ls -la /var/log/nginx/

Step 6: Try Starting Manually

# Run the exact command from the unit file
systemctl cat nginx.service | grep ExecStart
# ExecStart=/usr/sbin/nginx -g 'daemon off;'

# Run it manually to see direct error output
sudo /usr/sbin/nginx -t

Common causes of service start failures:

• Configuration syntax errors in the service's config file
• Port already in use by another service
• Missing files or directories that the service expects
• Permission denied on config files, log directories, or PID files
• Missing dependencies (a library or another service)

Useful systemctl Commands Reference

# Reload systemd itself after editing unit files
sudo systemctl daemon-reload

# Show all properties of a unit
systemctl show nginx.service --no-pager

# Show a specific property
systemctl show nginx.service --property=MainPID
systemctl show nginx.service --property=ActiveState

# List all units that failed
systemctl --failed --no-pager

# Reset the "failed" state of a unit
sudo systemctl reset-failed nginx.service

# Show the boot time breakdown
systemd-analyze

# Show which services took the longest to start
systemd-analyze blame --no-pager

# Show the critical chain (boot bottlenecks)
systemd-analyze critical-chain --no-pager

# Verify a unit file for errors
systemd-analyze verify /etc/systemd/system/myapp.service

systemd-analyze: Understanding Boot Performance

systemd-analyze is a powerful tool for understanding what happens during boot:

# Overall boot time
systemd-analyze
# Startup finished in 2.5s (kernel) + 5.1s (userspace) = 7.6s

# Which services took the longest?
systemd-analyze blame --no-pager | head -10
# 3.2s   NetworkManager-wait-online.service
# 1.1s   snapd.service
# 0.8s   udisks2.service
# 0.5s   accounts-daemon.service
# ...

# Show the critical path through the boot
systemd-analyze critical-chain --no-pager

The critical chain shows the longest dependency path through boot. This is where to focus if you want to speed up boot times.

What Just Happened?

+------------------------------------------------------------------+
|                     CHAPTER 15 RECAP                              |
+------------------------------------------------------------------+
|                                                                  |
|  - The init system (PID 1) starts and supervises all services    |
|  - systemd replaced SysVinit with parallel startup,             |
|    dependency management, and process supervision                |
|  - systemctl is your main tool: start, stop, restart,           |
|    enable, disable, status                                       |
|  - Unit types: service, socket, timer, mount, target, etc.      |
|  - Targets replace runlevels: multi-user.target,                |
|    graphical.target, rescue.target                               |
|  - enable = start at boot; start = start now                    |
|  - mask = prevent a service from starting entirely              |
|  - systemd-analyze helps diagnose slow boots                    |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Service Inventory

List all enabled services on your system. How many are there? Pick three you do not recognize and look up what they do using systemctl cat and man.

systemctl list-unit-files --type=service --state=enabled --no-pager

Exercise 2: Boot Analysis

Run systemd-analyze blame and find the three slowest services. Research whether any can be safely disabled on your system.

Exercise 3: Target Exploration

Run systemctl list-dependencies graphical.target and trace the dependency tree. Draw it on paper. How many levels deep does it go?

Exercise 4: Service Lifecycle

Install a simple service (like apache2 or httpd), then practice the full lifecycle:

sudo apt install apache2        # or: sudo dnf install httpd
sudo systemctl start apache2    # or: httpd
sudo systemctl status apache2
sudo systemctl stop apache2
sudo systemctl enable apache2
sudo systemctl disable apache2
sudo systemctl mask apache2
sudo systemctl start apache2    # Watch it fail
sudo systemctl unmask apache2

Bonus Challenge

Use systemd-analyze critical-chain to find the boot bottleneck on your system. Can you reduce boot time by disabling unnecessary services? Document the before and after boot times.

Writing & Managing Services

Why This Matters

You have written an application -- maybe a Python web server, a Go API, or a Node.js background worker. It runs fine when you type the command in a terminal. But what happens when you log out? It dies. What happens when it crashes at 3 AM? It stays dead. What happens when the server reboots? Nobody starts it.

This is where writing your own systemd service unit comes in. A unit file is a simple text file that tells systemd how to start, stop, supervise, and restart your application. No shell script gymnastics. No screen sessions. No nohup hacks. Just a declarative configuration that systemd follows reliably, every time.

This chapter teaches you to write unit files from scratch, understand every directive that matters, configure restart policies, manage dependencies, and replace cron jobs with systemd timers.

Try This Right Now

Let us create a trivially simple service in under a minute:

# Create a tiny script
sudo tee /usr/local/bin/hello-service.sh << 'SCRIPT'
#!/bin/bash
while true; do
    echo "Hello from my service at $(date)"
    sleep 10
done
SCRIPT
sudo chmod +x /usr/local/bin/hello-service.sh

# Create a unit file for it
sudo tee /etc/systemd/system/hello.service << 'UNIT'
[Unit]
Description=My Hello Service

[Service]
ExecStart=/usr/local/bin/hello-service.sh

[Install]
WantedBy=multi-user.target
UNIT

# Load, start, and watch it
sudo systemctl daemon-reload
sudo systemctl start hello.service
journalctl -u hello.service -f

You should see "Hello from my service" messages appearing every 10 seconds. Press Ctrl+C to stop watching the log. The service keeps running.

# Clean up when done
sudo systemctl stop hello.service
sudo systemctl disable hello.service
sudo rm /etc/systemd/system/hello.service
sudo rm /usr/local/bin/hello-service.sh
sudo systemctl daemon-reload

Unit File Anatomy

Every systemd unit file has the same basic structure: sections (denoted by square brackets) containing key-value directives.

+---------------------------------------------------+
|  [Unit]           <-- Metadata & Dependencies     |
|  Description=...                                   |
|  After=...                                         |
|  Requires=...                                      |
|                                                    |
|  [Service]        <-- How to Run                  |
|  Type=...                                          |
|  ExecStart=...                                     |
|  Restart=...                                       |
|                                                    |
|  [Install]        <-- Boot Integration            |
|  WantedBy=...                                      |
+---------------------------------------------------+

The [Unit] Section

This section describes the unit and defines its relationships to other units.

[Unit]
Description=My Application Server
Documentation=https://example.com/docs
After=network.target postgresql.service
Requires=postgresql.service
Wants=redis.service

After vs Requires: A Critical Distinction

These two directives do different things, and confusing them is a very common mistake:

• After= controls ordering -- "start me after X has started"
• Requires= controls dependency -- "if X fails, I fail too"

You almost always want both together:

# WRONG: ordering without dependency
After=postgresql.service
# PostgreSQL starts first, but if it fails, your app starts anyway

# WRONG: dependency without ordering
Requires=postgresql.service
# They might start at the same time (parallel), causing race conditions

# RIGHT: both together
After=postgresql.service
Requires=postgresql.service
# PostgreSQL starts first, AND your app won't start if PostgreSQL fails

Think About It: When would you use Wants= instead of Requires=? Think of a case where you would prefer your application to start even if an optional dependency failed.

The [Service] Section

This is where you define how the service actually runs.

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/myapp
Environment=NODE_ENV=production
EnvironmentFile=/opt/myapp/.env
ExecStartPre=/opt/myapp/check-config.sh
ExecStart=/opt/myapp/server --port 8080
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

We will explore each important directive in detail below.

The [Install] Section

This section defines how the unit integrates with the boot process.

[Install]
WantedBy=multi-user.target

WantedBy=multi-user.target is the most common value. It means "start this service when the system reaches multi-user mode" -- which is the normal boot target for servers.

ExecStart, ExecStop, and ExecReload

ExecStart

The most important directive. This is the command that starts your service:

# Simple command
ExecStart=/usr/bin/python3 /opt/myapp/server.py

# With arguments
ExecStart=/usr/bin/node /opt/myapp/index.js --port 3000

# IMPORTANT: Must be an absolute path. This will NOT work:
# ExecStart=python3 server.py    <-- WRONG

Rules for ExecStart:

• Must use an absolute path to the executable
• For Type=simple and Type=forking, there can be only one ExecStart line
• For Type=oneshot, you can have multiple ExecStart lines

ExecStartPre and ExecStartPost

Run commands before or after the main process starts:

ExecStartPre=/opt/myapp/validate-config.sh
ExecStart=/opt/myapp/server
ExecStartPost=/opt/myapp/notify-started.sh

Prefix with - to ignore failures:

# If this check fails, still start the service
ExecStartPre=-/opt/myapp/optional-check.sh

ExecStop

How to stop the service. If not specified, systemd sends SIGTERM (and then SIGKILL after a timeout):

# Custom graceful shutdown
ExecStop=/opt/myapp/graceful-shutdown.sh

ExecReload

What to do when systemctl reload is called. Typically sends SIGHUP:

ExecReload=/bin/kill -HUP $MAINPID

$MAINPID is a special variable systemd sets to the PID of the main process.

Service Types: Type=

The Type= directive tells systemd how your service starts and how to track its main process. Getting this wrong is one of the most common sources of service management bugs.

Type=simple (Default)

systemd considers the service "started" as soon as ExecStart runs. The process specified by ExecStart is the main process.

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/myapp/server.py

Use when: your application runs in the foreground and does not fork.

Type=forking

For traditional daemons that fork a child process and then the parent exits. systemd considers the service started when the parent process exits.

[Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/opt/myapp/start.sh

Use when: your application daemonizes itself (forks into the background). You usually need PIDFile= so systemd can track the main process.

Type=oneshot

For services that do a single task and then exit. systemd waits for the process to finish before considering the unit "started."

[Service]
Type=oneshot
ExecStart=/opt/myapp/run-migration.sh
ExecStart=/opt/myapp/seed-database.sh
RemainAfterExit=yes

Use when: you need to run a setup task at boot (like loading firewall rules). With RemainAfterExit=yes, the unit shows as "active" even after the process exits.

Type=notify

The service sends a notification to systemd when it is ready. This is the most precise way to signal readiness.

[Service]
Type=notify
ExecStart=/opt/myapp/server

The application must call sd_notify(0, "READY=1") (using the systemd library) or write to the $NOTIFY_SOCKET. Many modern services support this (e.g., PostgreSQL, nginx with certain configurations).

Type=exec

Similar to simple, but systemd considers the service started only after the binary has been successfully executed (after the exec() system call). This catches cases where the binary does not exist or cannot be executed.

+---------------------------------------------------+
|  Type=simple   ->  Started immediately             |
|  Type=exec     ->  Started after exec() succeeds   |
|  Type=forking  ->  Started when parent exits       |
|  Type=oneshot  ->  Started when process exits      |
|  Type=notify   ->  Started when service signals    |
+---------------------------------------------------+

Think About It: You have an application that takes 30 seconds to warm up (loading data into memory, connecting to databases) before it can serve requests. Which Type would you choose, and why?

Restart Policies

One of systemd's most valuable features: automatic restart when a service crashes.

[Service]
Restart=on-failure
RestartSec=5

Restart= Options

For most services, you want either on-failure or always:

# Restart only on crashes (not on intentional stops)
Restart=on-failure
RestartSec=5

# Always restart (even after clean exit -- useful for workers)
Restart=always
RestartSec=5

Preventing Restart Loops

If a service is badly broken, you do not want systemd to restart it forever:

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5

This means: if the service fails 5 times within 300 seconds (5 minutes), stop trying. The service enters a "failed" state.

Distro Note: StartLimitIntervalSec and StartLimitBurst belong in the [Unit] section on older systemd versions (before 230). On modern systems, they work in either section, but [Unit] is more portable.

Hands-On: Writing a Custom Service

Let us write a proper service for a Python web application.

Step 1: Create the Application

sudo mkdir -p /opt/mywebapp
sudo tee /opt/mywebapp/app.py << 'PYTHON'
#!/usr/bin/env python3
"""A tiny HTTP server for demonstration."""
from http.server import HTTPServer, SimpleHTTPRequestHandler
import os
import signal
import sys

PORT = int(os.environ.get('PORT', 8080))

def graceful_shutdown(signum, frame):
    print(f"Received signal {signum}, shutting down gracefully...", flush=True)
    sys.exit(0)

signal.signal(signal.SIGTERM, graceful_shutdown)

print(f"Starting server on port {PORT}", flush=True)
server = HTTPServer(('', PORT), SimpleHTTPRequestHandler)
print(f"Server is ready and listening on port {PORT}", flush=True)
server.serve_forever()
PYTHON
sudo chmod +x /opt/mywebapp/app.py

Step 2: Create a Dedicated User

sudo useradd --system --no-create-home --shell /usr/sbin/nologin mywebapp

Step 3: Write the Unit File

sudo tee /etc/systemd/system/mywebapp.service << 'UNIT'
[Unit]
Description=My Python Web Application
After=network.target
Documentation=https://example.com/mywebapp

[Service]
Type=simple
User=mywebapp
Group=mywebapp
WorkingDirectory=/opt/mywebapp
Environment=PORT=8080
ExecStart=/usr/bin/python3 /opt/mywebapp/app.py
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes

StandardOutput=journal
StandardError=journal
SyslogIdentifier=mywebapp

[Install]
WantedBy=multi-user.target
UNIT

Step 4: Deploy and Start

# Reload systemd to pick up the new unit file
sudo systemctl daemon-reload

# Start and enable the service
sudo systemctl enable --now mywebapp.service

# Check status
systemctl status mywebapp.service

# Test it
curl http://localhost:8080/

Step 5: Test Restart Behavior

# Find the main PID
systemctl show mywebapp.service --property=MainPID
# MainPID=12345

# Kill it rudely (simulating a crash)
sudo kill -9 $(systemctl show mywebapp.service --property=MainPID --value)

# Wait a moment, then check -- it should have restarted
sleep 6
systemctl status mywebapp.service
# Notice the PID has changed and the service is active

Step 6: Check the Logs

# View all logs for this service
journalctl -u mywebapp.service --no-pager

# Follow logs in real time
journalctl -u mywebapp.service -f

Clean Up

sudo systemctl disable --now mywebapp.service
sudo rm /etc/systemd/system/mywebapp.service
sudo rm -rf /opt/mywebapp
sudo userdel mywebapp
sudo systemctl daemon-reload

Service Security Hardening

systemd provides powerful security directives that sandbox your service. Use them wherever possible:

[Service]
# Run as non-root
User=myapp
Group=myapp

# Cannot gain new privileges (e.g., via setuid binaries)
NoNewPrivileges=yes

# Make the entire filesystem read-only except specified paths
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /var/log/myapp

# Hide /home, /root, /run/user
ProtectHome=yes

# Private /tmp (isolated from other services)
PrivateTmp=yes

# Cannot modify kernel variables
ProtectKernelTunables=yes

# Cannot load kernel modules
ProtectKernelModules=yes

# Restrict network families
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# Restrict system calls
SystemCallFilter=@system-service

You can check the security score of any service:

systemd-analyze security mywebapp.service

This gives each service a score from 0 (fully exposed) to 10 (fully locked down) and shows which hardening directives are missing.

Dependencies Deep Dive

Ordering: After= and Before=

These control when units start relative to each other:

# My app starts AFTER network and database are ready
After=network.target postgresql.service

# My app starts BEFORE the monitoring agent
Before=monitoring-agent.service

Dependency: Requires= and Wants=

These control whether units must succeed:

# Hard dependency: if PostgreSQL fails to start, my app fails too
Requires=postgresql.service

# Soft dependency: try to start Redis, but my app works without it
Wants=redis.service

Combining Them

A complete dependency setup:

[Unit]
Description=My Application
After=network.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service

This means:

1. Start after network, PostgreSQL, and Redis
2. Fail if PostgreSQL is not running
3. Continue even if Redis is not running

BindsTo=

Stronger than Requires. If the bound unit stops at any time (not just at startup), this unit also stops:

[Unit]
BindsTo=postgresql.service
After=postgresql.service

Conflicts=

Ensures two units never run simultaneously:

[Unit]
Conflicts=apache2.service

If you start this service, apache2.service is stopped automatically.

systemd Timers: The Modern Cron

systemd timers are a powerful replacement for cron jobs. They offer better logging, dependency management, and resource control.

Timer Anatomy

A timer requires two files:

1. A .timer unit (the schedule)
2. A .service unit (the actual work)

Example: Run a Backup Every Day at 2 AM

The service file (/etc/systemd/system/backup.service):

[Unit]
Description=Daily Backup Job

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=backup
StandardOutput=journal

The timer file (/etc/systemd/system/backup.timer):

[Unit]
Description=Run Backup Daily at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

# Enable the timer (not the service)
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

# Check when it will fire next
systemctl list-timers backup.timer --no-pager

Timer Directives

OnCalendar Syntax

The OnCalendar format is DayOfWeek Year-Month-Day Hour:Minute:Second:

OnCalendar=*-*-* 02:00:00          # Every day at 2:00 AM
OnCalendar=Mon *-*-* 09:00:00      # Every Monday at 9:00 AM
OnCalendar=*-*-01 00:00:00         # First day of every month
OnCalendar=*-01-01 00:00:00        # January 1st every year
OnCalendar=hourly                   # Every hour
OnCalendar=daily                    # Every day at midnight
OnCalendar=weekly                   # Every Monday at midnight
OnCalendar=*-*-* *:00:00           # Every hour on the hour
OnCalendar=*-*-* *:*:00            # Every minute
OnCalendar=*-*-* 08..17:00:00      # Every hour from 8 AM to 5 PM

Validate your schedule with systemd-analyze:

# When will this fire next?
systemd-analyze calendar "Mon *-*-* 09:00:00"
# Next elapse: Mon 2025-03-17 09:00:00 UTC

# How about every 15 minutes?
systemd-analyze calendar "*-*-* *:00/15:00"

Relative Timers

Instead of calendar-based, run relative to events:

[Timer]
# 15 minutes after boot
OnBootSec=15min

# Every 30 minutes after the service last ran
OnUnitActiveSec=30min

Socket Activation Basics

Socket activation lets systemd listen on a port and start the service only when a connection arrives. This means:

• Services start on-demand, not at boot (faster boot)
• If no one connects, the service never runs (saves resources)
• systemd can restart a crashed service without losing queued connections

How Socket Activation Works

+----------+       +---------+       +----------+
| Client   | ----> | systemd | ----> | Service  |
| connects |       | (holds  |       | (started |
|          |       |  socket)|       |  on demand)|
+----------+       +---------+       +----------+

1. systemd creates the socket and listens
2. Client connects to the socket
3. systemd starts the service
4. systemd passes the socket file descriptor to the service
5. Service handles the connection

Example: Socket-Activated Service

Socket file (/etc/systemd/system/myapp.socket):

[Unit]
Description=My App Socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

Service file (/etc/systemd/system/myapp.service):

[Unit]
Description=My App Service
Requires=myapp.socket

[Service]
Type=simple
ExecStart=/opt/myapp/server

# Enable the socket (not the service directly)
sudo systemctl enable --now myapp.socket

# The service is not running yet
systemctl is-active myapp.service
# inactive

# Connect to the socket
curl http://localhost:8080/
# Now the service starts automatically

systemctl is-active myapp.service
# active

Debug This: Service Keeps Crashing

Your custom service starts but immediately dies, and systemd keeps restarting it:

● myapp.service - My Application
     Active: activating (auto-restart) (Result: exit-code)

The journal shows:

myapp.service: Main process exited, code=exited, status=1/FAILURE
myapp.service: Scheduled restart job, restart counter is at 4.

Here is your debugging checklist:

1. Check the full journal output:

   journalctl -u myapp.service -n 100 --no-pager
   

2. Run ExecStart manually to see errors directly:

   systemctl cat myapp.service | grep ExecStart
   # Then run that command as the same user:
   sudo -u appuser /opt/myapp/server
   

3. Check that the binary exists and is executable:

   ls -la /opt/myapp/server
   file /opt/myapp/server
   

4. Check that the user has correct permissions:

   sudo -u appuser ls -la /opt/myapp/
   sudo -u appuser cat /opt/myapp/config.yaml
   

5. Check environment variables:

   systemctl show myapp.service --property=Environment
   

6. Temporarily stop restart looping to debug:

   sudo systemctl stop myapp.service
   # Now you can investigate without it restarting
   

Where Unit Files Live

+------------------------------------------------------------------+
|  /usr/lib/systemd/system/     <- Vendor/package-provided units   |
|                                   (do NOT edit these directly)    |
|                                                                   |
|  /etc/systemd/system/         <- Admin-created units (your stuff)|
|                                   (this is where you create them)|
|                                                                   |
|  /run/systemd/system/         <- Runtime-only units              |
|                                   (disappear on reboot)          |
+------------------------------------------------------------------+

Priority: /etc > /run > /usr/lib

To override a vendor-provided unit without editing it directly:

# Create an override directory
sudo systemctl edit nginx.service
# This opens an editor for /etc/systemd/system/nginx.service.d/override.conf

Or manually:

sudo mkdir -p /etc/systemd/system/nginx.service.d/
sudo tee /etc/systemd/system/nginx.service.d/override.conf << 'OVERRIDE'
[Service]
LimitNOFILE=65535
OVERRIDE
sudo systemctl daemon-reload
sudo systemctl restart nginx

What Just Happened?

+------------------------------------------------------------------+
|                     CHAPTER 16 RECAP                              |
+------------------------------------------------------------------+
|                                                                  |
|  - Unit files have three sections: [Unit], [Service], [Install]  |
|  - ExecStart= must use absolute paths                           |
|  - Type= controls how systemd tracks your process:              |
|    simple, forking, oneshot, notify                              |
|  - Restart=on-failure with RestartSec= for automatic recovery   |
|  - After= controls order; Requires= controls dependency         |
|  - Use both together for proper dependency management            |
|  - systemd timers replace cron with OnCalendar= schedules       |
|  - Socket activation starts services on-demand                   |
|  - Put custom units in /etc/systemd/system/                     |
|  - Always run daemon-reload after editing unit files             |
|  - Security hardening: User=, ProtectSystem=, PrivateTmp=       |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Write a One-Shot Service

Write a Type=oneshot service that creates a /tmp/system-booted file containing the current timestamp. Enable it so it runs at boot.

Exercise 2: Build a Timer

Create a systemd timer that runs a script every 15 minutes. The script should append the current date and system load average (uptime) to a log file. Verify the timer with systemctl list-timers.

Exercise 3: Dependency Chain

Create three services: service-a, service-b, and service-c. Configure them so that service-c requires service-b, which requires service-a. Verify the ordering:

sudo systemctl start service-c.service
# Should automatically start a and b first

Exercise 4: Crash Recovery

Create a service that intentionally exits with an error after 5 seconds. Set up Restart=on-failure with RestartSec=3. Watch it restart using journalctl -f. Then add StartLimitBurst=3 and StartLimitIntervalSec=60 and observe what happens after the third failure.

Bonus Challenge

Convert one of your existing cron jobs to a systemd timer. Compare the two approaches. Which gives you better logging? Which is easier to debug when something goes wrong?

Logging with journald & syslog

Why This Matters

It is 3 AM. Your monitoring system pages you: "Website down." You SSH into the server, and the application is not running. You need answers, fast. When did it stop? What error caused it? Did something else fail first?

Every answer lives in the logs.

Logging is how Linux systems tell you what is happening, what went wrong, and what happened leading up to a failure. Without good logging, you are debugging blind. With good logging, you have a time machine that lets you replay exactly what happened on your system.

Modern Linux has two logging systems that work together: journald (systemd's structured journal) and rsyslog (the traditional syslog daemon). This chapter teaches you to query, filter, manage, and rotate logs so you always have the information you need when something breaks.

Try This Right Now

# See the last 20 log entries
journalctl -n 20 --no-pager

# Follow the log in real time (like tail -f but for everything)
journalctl -f
# Press Ctrl+C to stop

# See logs from the current boot only
journalctl -b --no-pager | head -30

# Find logs from a specific service
journalctl -u sshd.service -n 10 --no-pager

# See all ERROR-level messages from the last hour
journalctl -p err --since "1 hour ago" --no-pager

The Two Logging Systems

Modern Linux distributions run two logging systems side by side:

+----------------------------------------------------------+
|  Applications / Services / Kernel                         |
+----------+----------------------------+------------------+
           |                            |
           v                            v
  +--------+---------+       +----------+-----------+
  |    journald      |       |     rsyslog          |
  | (systemd journal)|       | (traditional syslog) |
  | Binary, indexed  |       | Plain text files     |
  | /run/log/journal |       | /var/log/syslog      |
  | or               |       | /var/log/messages    |
  | /var/log/journal |       | /var/log/auth.log    |
  +------------------+       +----------------------+

journald collects logs from:

• systemd services (stdout/stderr)
• The kernel (kmsg)
• Syslog messages
• Audit framework

rsyslog receives messages from:

• journald (forwarded)
• Direct syslog connections
• Remote log sources

On most modern systems, journald is the primary collector, and rsyslog writes the traditional text files that many tools expect.

journalctl: Your Log Investigation Tool

journalctl is the command-line tool for querying the systemd journal. It is incredibly powerful once you know its filtering options.

Basic Usage

# View all logs (oldest first, paged)
journalctl

# View all logs (newest first)
journalctl -r

# View last N entries
journalctl -n 50

# Follow new entries in real time
journalctl -f

# No pager (dump to stdout, useful for piping)
journalctl --no-pager

Filtering by Unit

This is the most common filter -- show logs from a specific service:

# Logs from nginx
journalctl -u nginx.service --no-pager

# Logs from multiple units
journalctl -u nginx.service -u php-fpm.service --no-pager

# Follow a specific service's logs
journalctl -u myapp.service -f

Filtering by Time

# Logs since a specific time
journalctl --since "2025-03-10 14:00:00" --no-pager

# Logs in a time range
journalctl --since "2025-03-10 14:00" --until "2025-03-10 15:00" --no-pager

# Relative time expressions
journalctl --since "1 hour ago" --no-pager
journalctl --since "30 min ago" --no-pager
journalctl --since yesterday --no-pager
journalctl --since today --no-pager

Filtering by Boot

# Current boot only
journalctl -b

# Previous boot
journalctl -b -1

# Two boots ago
journalctl -b -2

# List all recorded boots
journalctl --list-boots

Sample output from --list-boots:

-3 abc123... Sat 2025-03-08 10:15:22 — Sat 2025-03-08 22:01:15
-2 def456... Sun 2025-03-09 08:30:11 — Sun 2025-03-09 23:45:00
-1 ghi789... Mon 2025-03-10 07:00:05 — Mon 2025-03-10 23:59:59
 0 jkl012... Tue 2025-03-11 06:55:30 — Tue 2025-03-11 14:22:10

Think About It: Why would you want to look at logs from a previous boot? Think about what happens when a system crashes and reboots -- the clues to the crash are in the previous boot's logs, not the current one.

Filtering by Priority

Syslog priorities, from most to least severe:

# Show only errors and above (emerg, alert, crit, err)
journalctl -p err --no-pager

# Show warnings and above
journalctl -p warning --no-pager

# Show a specific priority range
journalctl -p warning..err --no-pager

# Combine with other filters
journalctl -p err -u nginx.service --since today --no-pager

Filtering by Other Fields

The journal stores structured data. You can filter on many fields:

# By process ID
journalctl _PID=1234 --no-pager

# By user ID
journalctl _UID=1000 --no-pager

# By executable path
journalctl _EXE=/usr/sbin/sshd --no-pager

# By hostname (useful in centralized logging)
journalctl _HOSTNAME=webserver01 --no-pager

# By kernel messages only
journalctl -k --no-pager
# or equivalently:
journalctl _TRANSPORT=kernel --no-pager

Output Formats

# Default format (human-readable)
journalctl -n 5

# Short with precise timestamps
journalctl -n 5 -o short-precise

# JSON format (for parsing)
journalctl -n 5 -o json --no-pager

# JSON, one entry per line (great for piping to jq)
journalctl -n 5 -o json-pretty --no-pager

# Verbose (show all fields)
journalctl -n 1 -o verbose --no-pager

# Export format (for backup/transfer)
journalctl -o export --no-pager > journal-export.bin

The verbose output is particularly useful for understanding what metadata the journal stores:

journalctl -n 1 -o verbose --no-pager

Tue 2025-03-11 14:22:01.123456 UTC [s=abc123;i=42;b=def456...]
    _TRANSPORT=syslog
    PRIORITY=6
    SYSLOG_IDENTIFIER=sshd
    _PID=1234
    _UID=0
    _GID=0
    _EXE=/usr/sbin/sshd
    _COMM=sshd
    _CMDLINE=sshd: user [priv]
    MESSAGE=Accepted publickey for user from 10.0.0.1 port 54321
    ...

Hands-On: Log Investigation Workflow

Let us practice a realistic log investigation. We will look at SSH authentication events.

Step 1: Find SSH Events

journalctl -u sshd.service --since today --no-pager

Distro Note: Use -u ssh.service on Ubuntu/Debian.

Step 2: Filter for Authentication Failures

journalctl -u sshd.service --since today --no-pager | grep -i "failed\|invalid\|error"

Or use the journal's native grep:

journalctl -u sshd.service --since today --no-pager --grep="Failed password"

Step 3: Count Events

# How many failed login attempts today?
journalctl -u sshd.service --since today --no-pager --grep="Failed password" | wc -l

Step 4: Extract Attacker IPs

journalctl -u sshd.service --since today --no-pager --grep="Failed password" \
  | grep -oP 'from \K[0-9.]+' | sort | uniq -c | sort -rn | head -10

This gives you the top 10 IP addresses attempting failed SSH logins.

Step 5: See the Full Context Around an Event

# Find an interesting timestamp, then look at everything around that time
journalctl --since "2025-03-11 14:20:00" --until "2025-03-11 14:25:00" --no-pager

Persistent Journal Storage

By default, many distributions store the journal only in /run/log/journal/, which means logs are lost on reboot. For production systems, you want persistent storage.

Check Your Current Setup

# Where is the journal stored?
journalctl --disk-usage

# Is it persistent?
ls -la /var/log/journal/ 2>/dev/null && echo "Persistent" || echo "Volatile"

Enable Persistent Storage

# Create the persistent journal directory
sudo mkdir -p /var/log/journal

# Set correct ownership
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Restart journald to start using persistent storage
sudo systemctl restart systemd-journald

# Verify
journalctl --disk-usage

Or configure it in the journal configuration:

sudo mkdir -p /etc/systemd/journald.conf.d/
sudo tee /etc/systemd/journald.conf.d/persistent.conf << 'CONF'
[Journal]
Storage=persistent
CONF
sudo systemctl restart systemd-journald

The Storage= options are:

Journal Size Management

The journal can grow large on busy systems. Configure size limits:

sudo tee /etc/systemd/journald.conf.d/size.conf << 'CONF'
[Journal]
# Maximum disk space the journal can use
SystemMaxUse=500M

# Maximum size of individual journal files
SystemMaxFileSize=50M

# Keep at least this much free disk space
SystemKeepFree=1G

# Maximum time to keep entries
MaxRetentionSec=30day
CONF
sudo systemctl restart systemd-journald

Manual Cleanup

# See current disk usage
journalctl --disk-usage
# Archived and active journals take up 1.2G in the file system.

# Remove entries older than 2 weeks
sudo journalctl --vacuum-time=2weeks

# Reduce journal to a specific size
sudo journalctl --vacuum-size=500M

# Remove entries beyond a number of journal files
sudo journalctl --vacuum-files=5

Think About It: What is the trade-off between keeping more log history and managing disk space? On a production server, how far back would you want to keep logs, and why?

rsyslog and /var/log

While journald is the modern standard, rsyslog and the traditional /var/log/ files are still important. Many tools, scripts, and monitoring systems expect plain text log files.

The /var/log Directory

ls /var/log/

Common log files:

Distro Note: Debian/Ubuntu use /var/log/syslog and /var/log/auth.log. RHEL and Fedora use /var/log/messages and /var/log/secure. The content is the same; only the filenames differ.

rsyslog Configuration

rsyslog's main configuration is in /etc/rsyslog.conf with additional files in /etc/rsyslog.d/.

# View main config
cat /etc/rsyslog.conf

The configuration uses rules in the format:

facility.priority    destination

Example rules:

# All auth messages go to auth.log
auth,authpriv.*      /var/log/auth.log

# Everything except auth goes to syslog
*.*;auth,authpriv.none    /var/log/syslog

# Kernel messages
kern.*               /var/log/kern.log

# Emergency messages to all logged-in users
*.emerg              :omusrmsg:*

Syslog Facilities

Testing Syslog

# Send a test message to syslog
logger "This is a test message from $(whoami)"

# Send with a specific facility and priority
logger -p local0.notice "Test from local0"

# Check it arrived
journalctl --since "1 min ago" --no-pager
tail -5 /var/log/syslog 2>/dev/null || tail -5 /var/log/messages 2>/dev/null

Log Rotation with logrotate

Text log files grow forever unless rotated. logrotate handles this automatically: it compresses old logs, removes ancient ones, and signals services to reopen their log files.

How logrotate Works

+------------------+    logrotate    +------------------+
| access.log       | -------------> | access.log       |  (current, fresh)
| (500 MB, 7 days) |                | access.log.1     |  (yesterday)
+------------------+                | access.log.2.gz  |  (2 days ago, compressed)
                                    | access.log.3.gz  |  (3 days ago, compressed)
                                    +------------------+

Configuration

Global settings are in /etc/logrotate.conf. Per-application configs are in /etc/logrotate.d/.

# See what configs exist
ls /etc/logrotate.d/

Example configuration for a custom application:

sudo tee /etc/logrotate.d/myapp << 'CONF'
/var/log/myapp/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp.service 2>/dev/null || true
    endscript
}
CONF

Let us break down each directive:

Testing logrotate

# Dry run (see what would happen without doing it)
sudo logrotate --debug /etc/logrotate.d/myapp

# Force a rotation right now
sudo logrotate --force /etc/logrotate.d/myapp

# Run the full logrotate (as cron normally does)
sudo logrotate /etc/logrotate.conf

Viewing the nginx Rotation Config

cat /etc/logrotate.d/nginx

Typical output:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then
            run-parts /etc/logrotate.d/httpd-prerotate
        fi
    endscript
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

Hands-On: Setting Up Complete Logging

Let us set up proper logging for a custom application.

Step 1: Create a Test Application That Logs

sudo mkdir -p /opt/logdemo
sudo tee /opt/logdemo/app.sh << 'SCRIPT'
#!/bin/bash
while true; do
    # Log to stdout (captured by journald)
    echo "[INFO] Processing request at $(date)"

    # Simulate occasional errors
    if (( RANDOM % 5 == 0 )); then
        echo "[ERROR] Something went wrong at $(date)" >&2
    fi

    sleep 5
done
SCRIPT
sudo chmod +x /opt/logdemo/app.sh

Step 2: Create a Service for It

sudo tee /etc/systemd/system/logdemo.service << 'UNIT'
[Unit]
Description=Logging Demo Application

[Service]
Type=simple
ExecStart=/opt/logdemo/app.sh
StandardOutput=journal
StandardError=journal
SyslogIdentifier=logdemo
Restart=on-failure

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl start logdemo.service

Step 3: Query the Logs

# All logs from our demo
journalctl -u logdemo.service --no-pager -n 20

# Only errors
journalctl -u logdemo.service -p err --no-pager

# Follow in real time
journalctl -u logdemo.service -f

# JSON output for parsing
journalctl -u logdemo.service -o json-pretty -n 5 --no-pager

Step 4: Clean Up

sudo systemctl stop logdemo.service
sudo rm /etc/systemd/system/logdemo.service
sudo rm -rf /opt/logdemo
sudo systemctl daemon-reload

Debug This: Where Did My Logs Go?

You check journalctl -u myapp.service and see nothing. The service is running. Where are the logs?

Diagnosis checklist:

1. Is the service actually producing output?

   systemctl cat myapp.service | grep -E "Standard(Output|Error)"
   

   If StandardOutput=null or StandardError=null, output is discarded.

2. Is the application logging to a file instead of stdout? Many applications write to their own log files. Check the app's configuration.

   ls -la /var/log/myapp/ 2>/dev/null
   

3. Is the journal full and dropping messages?

   journalctl --disk-usage
   journalctl -p warning --no-pager | grep -i "journal"
   

4. Is the service running under the wrong identifier?

   # Maybe it is logging under a different name
   journalctl --since "5 min ago" --no-pager | grep -i myapp
   

5. Are you looking at the right boot?

   # Make sure you are looking at the current boot
   journalctl -u myapp.service -b 0 --no-pager
   

Centralized Logging Concepts

In production, you rarely look at logs on individual servers. Instead, you send logs to a centralized system.

Why Centralize?

• Persistence: If a server dies, its local logs may be lost
• Correlation: See events from multiple servers in one place
• Search: Query across all servers at once
• Alerting: Trigger alerts on specific log patterns
• Compliance: Some regulations require centralized, tamper-proof logs

Common Approaches

+----------+     +----------+     +----------+
| Server A |     | Server B |     | Server C |
|  rsyslog | --> |          | --> |          |
|  journald|     |  rsyslog |     |  journald|
+----+-----+     +----+-----+     +----+-----+
     |                |                |
     +--------+-------+-------+--------+
              |               |
              v               v
     +-----------------+  +------------------+
     |  Central Syslog |  |  Elasticsearch   |
     |  Server         |  |  (ELK/OpenSearch)|
     +-----------------+  +------------------+

Forwarding with rsyslog

To send logs to a remote syslog server:

sudo tee /etc/rsyslog.d/50-remote.conf << 'CONF'
# Forward all logs to central server via TCP
*.* @@logserver.example.com:514

# Or via UDP (single @)
# *.* @logserver.example.com:514
CONF
sudo systemctl restart rsyslog

Forwarding with journald

journald can forward to a remote journal:

sudo tee /etc/systemd/journal-upload.conf << 'CONF'
[Upload]
URL=http://logserver.example.com:19532
CONF
sudo systemctl enable --now systemd-journal-upload.service

Open Source Centralized Logging Stacks

Grafana Loki is particularly popular because it is lightweight and integrates naturally with Grafana dashboards. It is designed to be "like Prometheus, but for logs."

What Just Happened?

+------------------------------------------------------------------+
|                     CHAPTER 17 RECAP                              |
+------------------------------------------------------------------+
|                                                                  |
|  - journalctl is your primary log investigation tool             |
|  - Filter by unit (-u), time (--since/--until), priority (-p),  |
|    and boot (-b)                                                 |
|  - Enable persistent journal storage for production systems      |
|  - Manage journal size with SystemMaxUse= and vacuum commands   |
|  - rsyslog writes traditional /var/log text files                |
|  - /var/log/syslog (Debian) or /var/log/messages (RHEL) for     |
|    general messages                                              |
|  - logrotate compresses and cleans old log files                 |
|  - Centralized logging (ELK, Loki, Graylog) for production     |
|  - logger command sends test messages to syslog                  |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Log Investigation

Use journalctl to answer these questions about your system:

• How many error-level messages occurred today?
• Which service produced the most log entries in the last hour?
• When did your system last boot?

Exercise 2: Persistent Journal

Check whether your journal is persistent. If not, enable persistent storage and verify that logs survive a reboot (if you can reboot your system).

Exercise 3: Custom logrotate

Create a logrotate configuration for a fictional application that writes to /var/log/myapp/app.log. Configure it to rotate weekly, keep 8 weeks of history, compress old files, and test with logrotate --debug.

Exercise 4: Priority Filtering

Use journalctl -p to list all critical and error messages from the last 7 days. Are there any patterns? Any services that appear repeatedly?

Bonus Challenge

Set up rsyslog to write all authentication-related log entries to a separate file at /var/log/auth-audit.log. Configure logrotate for this file. Then generate some authentication events (SSH logins, sudo commands) and verify they appear in both journalctl and your custom log file.

Bash In Depth

Why This Matters

You already know how to type commands into a terminal. But have you ever been confused by why echo $HOME prints your home directory while echo '$HOME' prints the literal text $HOME? Have you wondered why rm * deletes all files but rm "*" tries to delete a file literally named *? Or why echo {1..10} prints 1 2 3 4 5 6 7 8 9 10?

These behaviors are not random. Bash processes every command line through a precise sequence of expansions before executing it. Understanding this sequence is what separates someone who uses the shell from someone who truly controls it.

This chapter takes you deep into how Bash works: the expansion order, quoting rules, variables, arrays, and special parameters. Master these, and you will write commands and scripts that do exactly what you intend, every time.

Try This Right Now

Run each of these and observe the differences carefully:

# Brace expansion
echo {a,b,c}-{1,2}

# Tilde expansion
echo ~
echo ~root

# Parameter expansion
name="Linux"
echo "Hello, $name"
echo "Hello, ${name}!"
echo "Length: ${#name}"

# Command substitution
echo "Today is $(date +%A)"

# Arithmetic expansion
echo "2 + 3 = $((2 + 3))"

# Globbing
echo /etc/*.conf | tr ' ' '\n' | head -5

Now observe how quoting changes things:

echo $HOME         # Expanded
echo "$HOME"       # Expanded
echo '$HOME'       # NOT expanded -- literal text
echo \$HOME        # NOT expanded -- escaped

The Shell Expansion Order

When you type a command and press Enter, Bash does not simply pass your text to the program. It processes it through a specific sequence of expansions, in this exact order:

+-------------------------------------------------------------------+
|  1. Brace Expansion        {a,b,c}  {1..5}                       |
|  2. Tilde Expansion        ~  ~user                               |
|  3. Parameter Expansion    $var  ${var}  ${var:-default}          |
|  4. Command Substitution   $(cmd)  `cmd`                          |
|  5. Arithmetic Expansion   $((expr))                              |
|  6. Word Splitting         (on unquoted results of 3, 4, 5)      |
|  7. Pathname Expansion     *  ?  [abc]  (globbing)                |
+-------------------------------------------------------------------+

Each step operates on the output of the previous step. This ordering matters because it determines what you can combine and what you cannot.

Step 1: Brace Expansion

Braces generate multiple strings. This happens first, before any variable expansion:

# Comma-separated list
echo {cat,dog,fish}
# cat dog fish

# Sequence
echo {1..5}
# 1 2 3 4 5

# Sequence with step
echo {0..20..5}
# 0 5 10 15 20

# Letter sequence
echo {a..f}
# a b c d e f

# Combinations (cartesian product)
echo {web,db}-{01,02}
# web-01 web-02 db-01 db-02

# Practical: create multiple directories
mkdir -p project/{src,tests,docs}/{v1,v2}
# Creates: project/src/v1  project/src/v2  project/tests/v1 ...

# Practical: backup a file
cp config.yml{,.bak}
# Equivalent to: cp config.yml config.yml.bak

Key rule: Brace expansion happens before variable expansion. This means you cannot use a variable inside braces for sequence generation:

n=5
echo {1..$n}
# Output: {1..5}  <-- Not expanded! Braces happen first, $n isn't resolved yet

Step 2: Tilde Expansion

The tilde ~ expands to home directories:

echo ~                # /home/yourusername
echo ~root            # /root
echo ~nobody          # /nonexistent (or wherever nobody's home is)
echo ~/Documents      # /home/yourusername/Documents

This only works at the beginning of a word. A tilde in the middle of text is just a literal tilde.

Step 3: Parameter Expansion

This is where variables get replaced with their values. Bash offers far more than just $var:

name="hello world"

# Basic expansion
echo $name          # hello world
echo ${name}        # hello world (braces clarify boundaries)
echo "${name}ish"   # hello worldish

# String length
echo ${#name}       # 11

# Default values
echo ${unset_var:-default}    # default (use default if unset/empty)
echo ${unset_var:=default}    # default (AND assign the default)

# Substring extraction
str="Hello, World!"
echo ${str:7}       # World!
echo ${str:7:5}     # World

# Pattern removal
path="/home/user/documents/file.tar.gz"
echo ${path##*/}    # file.tar.gz  (remove longest prefix matching */)
echo ${path#*/}     # home/user/documents/file.tar.gz  (remove shortest prefix)
echo ${path%%.*}    # /home/user/documents/file  (remove longest suffix matching .*)
echo ${path%.*}     # /home/user/documents/file.tar  (remove shortest suffix)

# Substitution
echo ${path/user/admin}      # /home/admin/documents/file.tar.gz  (first match)
echo ${path//o/0}            # /h0me/user/d0cuments/file.tar.gz  (all matches)

# Case modification (Bash 4+)
greeting="hello world"
echo ${greeting^}    # Hello world  (capitalize first letter)
echo ${greeting^^}   # HELLO WORLD  (capitalize all)
upper="HELLO"
echo ${upper,}       # hELLO  (lowercase first letter)
echo ${upper,,}      # hello  (lowercase all)

Think About It: Given the file path /var/log/nginx/access.log, how would you extract just the filename access.log using parameter expansion? What about just the extension log?

Step 4: Command Substitution

Replace a command with its output:

# Modern syntax (preferred)
echo "Today is $(date +%Y-%m-%d)"

# Old syntax (backticks -- avoid for readability)
echo "Today is `date +%Y-%m-%d`"

# Nested command substitution (much cleaner with $() than backticks)
echo "Config dir: $(dirname $(readlink -f /etc/resolv.conf))"

# Assign to variable
file_count=$(ls /etc/*.conf 2>/dev/null | wc -l)
echo "Found $file_count conf files"

Always prefer $(...) over backticks. Backticks are harder to read and cannot nest cleanly.

Step 5: Arithmetic Expansion

Perform integer math directly:

echo $((2 + 3))         # 5
echo $((10 / 3))        # 3  (integer division!)
echo $((10 % 3))        # 1  (modulo)
echo $((2 ** 10))       # 1024  (exponentiation)

x=10
echo $((x + 5))         # 15  (no $ needed inside $(()))
echo $((x++))           # 10  (post-increment, x is now 11)
echo $x                 # 11

# Comparison (returns 1 for true, 0 for false)
echo $((5 > 3))         # 1
echo $((5 < 3))         # 0

WARNING: Bash arithmetic is integer only. $((10 / 3)) gives 3, not 3.333. For floating point, use bc or awk.

Step 6: Word Splitting

After parameter expansion, command substitution, and arithmetic expansion, Bash splits the results into separate words based on the IFS (Internal Field Separator) variable.

# Default IFS is space, tab, newline
files="file1.txt file2.txt file3.txt"
for f in $files; do        # Word splitting splits into three words
    echo "Processing: $f"
done

# This is why quoting is critical:
filename="my important file.txt"
touch "$filename"           # Creates ONE file: "my important file.txt"
touch $filename             # Creates THREE files: "my", "important", "file.txt"

Word splitting does not happen on text inside double quotes. This is why you should almost always quote your variables.

Step 7: Pathname Expansion (Globbing)

The final step expands wildcard patterns into matching filenames:

# * matches any string (including empty)
echo /etc/*.conf

# ? matches exactly one character
echo /etc/host?

# [abc] matches one character from the set
echo /dev/sd[a-c]

# [!abc] or [^abc] matches one character NOT in the set
echo /dev/sd[!a]

# ** matches directories recursively (needs shopt -s globstar)
shopt -s globstar
echo /etc/**/*.conf

Globbing only happens on unquoted text. This is another reason quoting matters:

echo *.txt        # Expands to matching files
echo "*.txt"      # Literal string: *.txt

Quoting Rules

Quoting controls which expansions happen. This is one of the most important things to understand in Bash.

No Quotes

All expansions happen. Word splitting and globbing happen.

echo $HOME/*.txt
# Expands $HOME, then globs for .txt files

Double Quotes (")

Parameter expansion, command substitution, and arithmetic expansion happen. Word splitting and globbing do not happen.

echo "$HOME/*.txt"
# Expands $HOME to /home/user, but *.txt stays literal
# Output: /home/user/*.txt

# Preserves whitespace in variables
greeting="  hello   world  "
echo $greeting         # hello world  (whitespace collapsed)
echo "$greeting"       #   hello   world   (whitespace preserved)

Single Quotes (')

Nothing is expanded. Everything is literal.

echo '$HOME is not expanded'
# Output: $HOME is not expanded

echo 'No $(commands) or $((math)) either'
# Output: No $(commands) or $((math)) either

Escape Character ()

Removes the special meaning of the next character:

echo \$HOME            # $HOME  (literal dollar sign)
echo "She said \"hi\"" # She said "hi"
echo 'It'\''s here'    # It's here (ending and reopening single quotes)

The $'...' Syntax

Allows escape sequences like \n, \t:

echo $'Line 1\nLine 2\tTabbed'
# Line 1
# Line 2	Tabbed

Summary Table

Hands-On: Quoting Pitfalls

Try each of these to internalize the differences:

# Setup
mkdir -p /tmp/quoting-lab
cd /tmp/quoting-lab
touch "file one.txt" "file two.txt" "file three.txt"

# WRONG: word splitting breaks filenames with spaces
for f in $(ls); do
    echo "File: $f"
done
# Output: each word separately (broken!)

# RIGHT: use glob instead of ls, with proper quoting
for f in *.txt; do
    echo "File: $f"
done
# Output: correct filenames

# The difference between $@ and $*
# Create a test script:
cat > /tmp/quoting-lab/test-args.sh << 'SCRIPT'
#!/bin/bash
echo '--- $* (unquoted) ---'
for arg in $*; do echo "  [$arg]"; done

echo '--- "$*" (quoted) ---'
for arg in "$*"; do echo "  [$arg]"; done

echo '--- $@ (unquoted) ---'
for arg in $@; do echo "  [$arg]"; done

echo '--- "$@" (quoted -- correct) ---'
for arg in "$@"; do echo "  [$arg]"; done
SCRIPT
chmod +x /tmp/quoting-lab/test-args.sh

# Test it with arguments that contain spaces
/tmp/quoting-lab/test-args.sh "hello world" "foo bar"

# Clean up
rm -rf /tmp/quoting-lab

Variables

Setting Variables

# No spaces around the = sign!
name="Alice"        # Correct
name = "Alice"      # WRONG -- Bash thinks "name" is a command

# No need to quote simple values
count=42

# Quote when value contains spaces or special characters
greeting="Hello, World!"
path="/home/user/my files"

Local Variables

By default, variables are local to the current shell:

color="blue"
echo $color          # blue

# Start a subshell
bash -c 'echo $color'  # (empty -- variable not inherited)

Exported Variables (Environment Variables)

Use export to make a variable available to child processes:

export DATABASE_URL="postgres://localhost:5432/mydb"

# Now child processes can see it
bash -c 'echo $DATABASE_URL'
# postgres://localhost:5432/mydb

# Or set and export in one step
export API_KEY="secret123"

Readonly Variables

readonly PI=3.14159
PI=3.0
# bash: PI: readonly variable

Unsetting Variables

temp="something"
unset temp
echo $temp           # (empty)

Arrays

Bash supports indexed arrays (like lists) and associative arrays (like dictionaries/maps).

Indexed Arrays

# Declare an array
fruits=("apple" "banana" "cherry" "date")

# Access elements (0-indexed)
echo ${fruits[0]}     # apple
echo ${fruits[2]}     # cherry

# All elements
echo ${fruits[@]}     # apple banana cherry date

# Number of elements
echo ${#fruits[@]}    # 4

# Add an element
fruits+=("elderberry")
echo ${#fruits[@]}    # 5

# Iterate
for fruit in "${fruits[@]}"; do
    echo "I like $fruit"
done

# Slice (elements 1 through 2)
echo ${fruits[@]:1:2}   # banana cherry

# Indices
echo ${!fruits[@]}    # 0 1 2 3 4

# Remove an element (leaves a gap!)
unset 'fruits[1]'
echo ${fruits[@]}     # apple cherry date elderberry
echo ${!fruits[@]}    # 0 2 3 4   (index 1 is gone, others unchanged)

Associative Arrays (Bash 4+)

# Must declare with -A
declare -A config

config[host]="localhost"
config[port]="5432"
config[database]="mydb"
config[user]="admin"

# Access
echo ${config[host]}         # localhost
echo ${config[port]}         # 5432

# All values
echo ${config[@]}            # localhost 5432 mydb admin (order not guaranteed)

# All keys
echo ${!config[@]}           # host port database user

# Number of elements
echo ${#config[@]}           # 4

# Iterate over key-value pairs
for key in "${!config[@]}"; do
    echo "$key = ${config[$key]}"
done

# Check if a key exists
if [[ -v config[host] ]]; then
    echo "host is set"
fi

Think About It: Why might you prefer an associative array over a series of individual variables when managing configuration values?

Special Variables

Bash provides several special variables that are essential for scripting:

Process-Related

echo "My PID: $$"
sleep 100 &
echo "Background PID: $!"
echo "Parent PID: $PPID"

Argument-Related

# In a script:
echo "Script name: $0"
echo "First arg: $1"
echo "All args: $@"
echo "Arg count: $#"

The difference between "$@" and "$*" is critical:

# "$@" preserves each argument as a separate word -- USUALLY WHAT YOU WANT
# "$*" joins all arguments into a single string separated by first char of IFS

Status-Related

ls /etc/hosts
echo $?          # 0 (success)

ls /nonexistent
echo $?          # 2 (error)

echo hello world
echo $_          # world (last argument of previous command)

Shell Configuration

echo "User $USER on $HOSTNAME in $PWD"
echo "Random number: $RANDOM"
echo "Shell has been running for $SECONDS seconds"

Hands-On: Expansion Mastery

Challenge 1: Build File Paths

# Use parameter expansion to manipulate this path:
filepath="/var/log/nginx/access.log.2.gz"

# Extract just the filename
echo ${filepath##*/}
# access.log.2.gz

# Extract just the directory
echo ${filepath%/*}
# /var/log/nginx

# Extract the extension
echo ${filepath##*.}
# gz

# Remove all extensions
echo ${filepath%%.*}
# /var/log/nginx/access

# Replace nginx with apache
echo ${filepath/nginx/apache}
# /var/log/apache/access.log.2.gz

Challenge 2: Batch Rename Using Arrays

mkdir -p /tmp/rename-lab
cd /tmp/rename-lab
touch photo_{001..005}.JPG

# Rename .JPG to .jpg using parameter expansion
for f in *.JPG; do
    mv "$f" "${f%.JPG}.jpg"
done
ls
# photo_001.jpg  photo_002.jpg  photo_003.jpg  photo_004.jpg  photo_005.jpg

# Clean up
rm -rf /tmp/rename-lab

Challenge 3: Default Values in Practice

# A script that uses default values
DB_HOST=${DB_HOST:-localhost}
DB_PORT=${DB_PORT:-5432}
DB_NAME=${DB_NAME:-myapp}

echo "Connecting to $DB_HOST:$DB_PORT/$DB_NAME"
# If no env vars set: Connecting to localhost:5432/myapp

Debug This: Quoting Gone Wrong

Someone wrote this script and it does not work with filenames containing spaces:

#!/bin/bash
# BUG: This breaks on filenames with spaces
for file in $(find /data -name "*.csv"); do
    wc -l $file
done

Problems:

1. $(find ...) is subject to word splitting
2. $file is unquoted, so it splits on spaces

Fixed version:

#!/bin/bash
# FIXED: Use find -exec or a while loop with null delimiter
find /data -name "*.csv" -print0 | while IFS= read -r -d '' file; do
    wc -l "$file"
done

Or even simpler with find -exec:

find /data -name "*.csv" -exec wc -l {} \;

What Just Happened?

+------------------------------------------------------------------+
|                     CHAPTER 18 RECAP                              |
+------------------------------------------------------------------+
|                                                                  |
|  Bash expansion order:                                           |
|  1. Brace  2. Tilde  3. Parameter  4. Command Sub               |
|  5. Arithmetic  6. Word Splitting  7. Globbing                   |
|                                                                  |
|  Quoting:                                                        |
|  - Double quotes: variables expand, no glob/split                |
|  - Single quotes: nothing expands                                |
|  - Always quote "$variable" to prevent word splitting            |
|                                                                  |
|  Parameter expansion: ${var:-default}, ${var##pattern},          |
|  ${var%pattern}, ${var/old/new}, ${#var}                         |
|                                                                  |
|  Arrays: fruits=(...), ${fruits[@]}, ${#fruits[@]}               |
|  Associative arrays: declare -A, ${map[key]}                     |
|                                                                  |
|  Special vars: $?, $$, $!, $@, $#, $0                            |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Expansion Order

Predict the output of each line before running it, then verify:

echo {1..3}_{a,b}
echo ~root/test
echo "Today: $(date +%H:%M) - PID $$"
echo '$HOME is' $HOME
echo $((2**8))

Exercise 2: Parameter Expansion Practice

Given url="https://www.example.com/path/to/page.html?query=1", use parameter expansion to extract:

• Just the protocol (https)
• Just the filename (page.html?query=1)
• The URL with example.com replaced by mysite.org

Exercise 3: Array Operations

Create an indexed array of 5 Linux distribution names. Write a loop that prints each with its index number. Then create an associative array mapping each distro to its package manager.

Exercise 4: Special Variables

Write a short script that prints: its own name, all arguments it received, the count of arguments, and the PID it is running as. Test it with various inputs.

Bonus Challenge

Write a one-liner using brace expansion that creates a directory structure for a new project with: src/, tests/, docs/, config/, and inside each a README.md file. Use only mkdir -p and touch with brace expansion.

Bash Scripting

Why This Matters

You have been typing commands one at a time. That works fine when you are doing something once. But what happens when you need to do the same thing every day? Or on fifty servers? Or in a CI/CD pipeline at 2 AM with nobody at the keyboard?

You write a script.

A Bash script is just a text file full of commands that Bash executes in sequence. But scripts can also make decisions, loop over data, accept arguments, handle errors, and call functions. A well-written script can replace hours of manual work with a single command.

This chapter takes you from writing your first script to writing robust, production-grade Bash that handles errors gracefully, parses command-line arguments, and does not break when the unexpected happens.

Try This Right Now

Create and run your first script:

cat > /tmp/hello.sh << 'SCRIPT'
#!/bin/bash
echo "Hello, $(whoami)!"
echo "Today is $(date +%A), $(date +%B) $(date +%d)"
echo "You are running Bash $BASH_VERSION"
echo "Your system has $(nproc) CPU cores"
echo "Free memory: $(free -h | awk '/Mem:/ {print $4}')"
SCRIPT
chmod +x /tmp/hello.sh
/tmp/hello.sh

You should see a personalized system summary. That is a script: commands in a file, executed as a unit.

The Shebang Line

Every script should start with a shebang (#!) that tells the system which interpreter to use:

#!/bin/bash

Common shebangs:

#!/bin/bash        # Use Bash specifically
#!/bin/sh          # Use the system's POSIX shell (might not be Bash)
#!/usr/bin/env bash  # Find bash in PATH (more portable)
#!/usr/bin/env python3  # For Python scripts

Why does this matter? Without a shebang, the script is run by whatever shell invoked it. With a shebang, it always runs with the correct interpreter, even when called from a different shell.

# Make a script executable and run it
chmod +x myscript.sh
./myscript.sh        # Uses the shebang interpreter

# Or explicitly invoke bash (shebang is ignored)
bash myscript.sh

Distro Note: On some systems (FreeBSD, certain containers), Bash is not at /bin/bash. Using #!/usr/bin/env bash is more portable because it searches $PATH.

Exit Codes

Every command returns an exit code: an integer from 0 to 255.

# 0 means success
ls /tmp
echo $?    # 0

# Non-zero means failure
ls /nonexistent
echo $?    # 2

# You can set your own exit code
exit 0     # Success
exit 1     # General error
exit 2     # Misuse of command

Conventional exit codes:

Using exit codes in scripts:

#!/bin/bash
if [ ! -f /etc/hosts ]; then
    echo "ERROR: /etc/hosts not found" >&2
    exit 1
fi
echo "OK: /etc/hosts exists"
exit 0

Conditionals

The if Statement

#!/bin/bash

if [ -f /etc/hosts ]; then
    echo "/etc/hosts exists"
elif [ -f /etc/hostname ]; then
    echo "/etc/hostname exists"
else
    echo "Neither file found"
fi

test, [ ], and [[ ]]

There are three ways to test conditions:

# These are equivalent:
test -f /etc/hosts
[ -f /etc/hosts ]

# [[ ]] is a Bash enhancement (preferred):
[[ -f /etc/hosts ]]

Why prefer [[ ]]?

• No word splitting on variables (safer)
• Supports && and || inside the brackets
• Supports pattern matching with == and regex with =~
• Does not need quoting on variables (though quoting is still good practice)

File Tests

[[ -f /path ]]    # True if file exists and is a regular file
[[ -d /path ]]    # True if directory exists
[[ -e /path ]]    # True if anything exists at that path
[[ -r /path ]]    # True if readable
[[ -w /path ]]    # True if writable
[[ -x /path ]]    # True if executable
[[ -s /path ]]    # True if file exists and is not empty
[[ -L /path ]]    # True if symbolic link

String Tests

[[ -z "$str" ]]            # True if string is empty
[[ -n "$str" ]]            # True if string is NOT empty
[[ "$a" == "$b" ]]         # True if strings are equal
[[ "$a" != "$b" ]]         # True if strings are not equal
[[ "$a" < "$b" ]]          # True if a sorts before b
[[ "$str" == *.txt ]]      # Pattern matching (glob)
[[ "$str" =~ ^[0-9]+$ ]]   # Regex matching

Integer Comparisons

[[ $a -eq $b ]]    # Equal
[[ $a -ne $b ]]    # Not equal
[[ $a -lt $b ]]    # Less than
[[ $a -le $b ]]    # Less than or equal
[[ $a -gt $b ]]    # Greater than
[[ $a -ge $b ]]    # Greater than or equal

# Or use (( )) for arithmetic comparisons (more readable):
(( a == b ))
(( a != b ))
(( a < b ))
(( a > b ))

Combining Conditions

# AND
[[ -f /etc/hosts && -r /etc/hosts ]]

# OR
[[ -f /etc/hosts || -f /etc/hostname ]]

# NOT
[[ ! -f /etc/hosts ]]

# Complex combination
if [[ -f "$config" && -r "$config" ]] && command -v jq &>/dev/null; then
    echo "Config file is readable and jq is available"
fi

Think About It: Why is [[ -f "$file" ]] safer than [ -f $file ] when $file might contain spaces or be empty?

Loops

for Loops

# Over a list
for color in red green blue; do
    echo "Color: $color"
done

# Over command output
for user in $(cut -d: -f1 /etc/passwd | head -5); do
    echo "User: $user"
done

# Over files (use glob, not ls!)
for f in /etc/*.conf; do
    echo "Config: $f"
done

# C-style for loop
for ((i=1; i<=5; i++)); do
    echo "Iteration $i"
done

# Over a range
for i in {1..10}; do
    echo "Number $i"
done

while Loops

# Basic while
count=1
while [[ $count -le 5 ]]; do
    echo "Count: $count"
    ((count++))
done

# Read a file line by line (correct way)
while IFS= read -r line; do
    echo "Line: $line"
done < /etc/hostname

# Read command output line by line
ps aux | while IFS= read -r line; do
    echo "$line"
done

# Infinite loop (useful for daemons, menus)
while true; do
    echo "Working... (Ctrl+C to stop)"
    sleep 5
done