Buffered vs Unbuffered I/O

Every write() system call crosses the user-kernel boundary. That crossing is expensive -- hundreds of nanoseconds at minimum. Buffered I/O collects small writes into a large buffer and flushes them in one syscall. This chapter shows you both layers and when to use each.

The Two Layers

Your Program
    |
    v
+------------------------------+
|  stdio (fopen, fprintf, ...) |   <-- buffered (user-space)
|  internal buffer: 4096+ bytes|
+------------------------------+
    |  fflush() or buffer full
    v
+------------------------------+
|  syscalls (open, write, ...) |   <-- unbuffered (kernel boundary)
+------------------------------+
    |
    v
  Kernel page cache / disk

The unbuffered layer (open, read, write, close) is what we covered in Chapter 28. The buffered layer (fopen, fread, fwrite, fprintf, fclose) wraps the unbuffered layer with a user-space buffer.

Buffered I/O in C: stdio

/* stdio_demo.c -- buffered I/O with fopen, fprintf, fclose */
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    FILE *fp = fopen("/tmp/buffered.txt", "w");
    if (!fp) {
        perror("fopen");
        return 1;
    }

    /* fprintf writes to an internal buffer, not directly to disk */
    fprintf(fp, "line one\n");
    fprintf(fp, "line two\n");
    fprintf(fp, "value: %d\n", 42);

    /* fclose flushes the buffer and then calls close() */
    if (fclose(fp) != 0) {
        perror("fclose");
        return 1;
    }

    /* Read it back */
    fp = fopen("/tmp/buffered.txt", "r");
    if (!fp) {
        perror("fopen");
        return 1;
    }

    char line[256];
    while (fgets(line, sizeof(line), fp)) {
        printf("read: %s", line);
    }

    fclose(fp);
    return 0;
}

$ gcc -Wall -o stdio_demo stdio_demo.c && ./stdio_demo
read: line one
read: line two
read: value: 42

fread and fwrite

For binary data or bulk transfers, use fread and fwrite instead of fprintf and fgets:

/* fread_fwrite.c -- binary I/O */
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    int data[] = {10, 20, 30, 40, 50};
    size_t count = sizeof(data) / sizeof(data[0]);

    /* Write binary data */
    FILE *fp = fopen("/tmp/binary.dat", "wb");
    if (!fp) { perror("fopen"); return 1; }

    size_t written = fwrite(data, sizeof(int), count, fp);
    printf("wrote %zu integers\n", written);
    fclose(fp);

    /* Read it back */
    int buf[5] = {0};
    fp = fopen("/tmp/binary.dat", "rb");
    if (!fp) { perror("fopen"); return 1; }

    size_t nread = fread(buf, sizeof(int), count, fp);
    printf("read %zu integers:", nread);
    for (size_t i = 0; i < nread; i++) {
        printf(" %d", buf[i]);
    }
    printf("\n");
    fclose(fp);

    return 0;
}

$ gcc -Wall -o fread_fwrite fread_fwrite.c && ./fread_fwrite
wrote 5 integers
read 5 integers: 10 20 30 40 50

Caution: fwrite of raw structs is not portable across architectures due to endianness and padding differences. For files that must be portable, serialize field by field.

Buffer Modes: Full, Line, None

stdio supports three buffering modes, set with setvbuf():

ModeConstantBehavior
Full buffering_IOFBFFlush when buffer is full
Line buffering_IOLBFFlush on newline or when full
No buffering_IONBFEvery write goes to kernel immediately

Default behavior:

  • stderr is unbuffered (_IONBF) -- errors appear immediately
  • stdout is line-buffered when connected to a terminal, full-buffered when connected to a pipe or file
  • Files opened with fopen are full-buffered
/* setvbuf_demo.c -- control buffering mode */
#include <stdio.h>
#include <unistd.h>

int main(void)
{
    FILE *fp = fopen("/tmp/setvbuf_demo.txt", "w");
    if (!fp) { perror("fopen"); return 1; }

    /* Set line buffering with a 1024-byte buffer */
    char mybuf[1024];
    if (setvbuf(fp, mybuf, _IOLBF, sizeof(mybuf)) != 0) {
        perror("setvbuf");
        fclose(fp);
        return 1;
    }

    fprintf(fp, "this flushes on newline\n");  /* flushed now */
    fprintf(fp, "no newline yet...");           /* still in buffer */
    fprintf(fp, " now!\n");                    /* flushed now */

    fclose(fp);

    /* Verify */
    fp = fopen("/tmp/setvbuf_demo.txt", "r");
    char line[256];
    while (fgets(line, sizeof(line), fp))
        printf("%s", line);
    fclose(fp);

    return 0;
}

To disable buffering entirely:

setvbuf(fp, NULL, _IONBF, 0);

Try It: Write a program that prints to stdout without a newline, then sleeps for 3 seconds, then prints a newline. Run it piped to cat vs directly on the terminal. Observe the difference in when output appears.

When to Flush: fflush

fflush(fp) forces the buffer to be written to the kernel. Common situations where you need it:

  • Before a fork() -- otherwise the child inherits the buffer and you get double output
  • Before reading from the same file you are writing to
  • Before a crash-sensitive section -- data in the buffer is lost on crash
  • Before switching between stdio and raw fd operations on the same file
/* fflush_demo.c -- explicit flush */
#include <stdio.h>
#include <unistd.h>

int main(void)
{
    printf("prompt: ");
    fflush(stdout);    /* force output before blocking on read */

    char buf[64];
    if (fgets(buf, sizeof(buf), stdin)) {
        printf("you typed: %s", buf);
    }

    return 0;
}

fflush(NULL) flushes all open output streams. Useful before fork().

Caution: fflush(stdin) is undefined behavior in the C standard, even though some implementations (like glibc on Linux) define it to discard input. Do not rely on it.

The Rust Equivalent: BufReader and BufWriter

Rust separates buffering from the file type. You wrap any reader in BufReader and any writer in BufWriter.

// buffered_io.rs -- BufWriter and BufReader
use std::fs::File;
use std::io::{BufWriter, BufReader, BufRead, Write};

fn main() -> std::io::Result<()> {
    let path = "/tmp/buffered_rs.txt";

    // Buffered writing
    {
        let file = File::create(path)?;
        let mut writer = BufWriter::new(file);

        writeln!(writer, "line one")?;
        writeln!(writer, "line two")?;
        writeln!(writer, "value: {}", 42)?;

        // BufWriter flushes on drop, but explicit flush
        // lets you catch errors
        writer.flush()?;
    }

    // Buffered reading
    {
        let file = File::open(path)?;
        let reader = BufReader::new(file);

        for line in reader.lines() {
            let line = line?;
            println!("read: {}", line);
        }
    }

    Ok(())
}
$ rustc buffered_io.rs && ./buffered_io
read: line one
read: line two
read: value: 42

Rust Note: BufWriter flushes its buffer when dropped. However, any error during that flush is silently ignored. Always call .flush() explicitly before dropping if you need to detect write failures.

The BufRead Trait

BufReader implements the BufRead trait, which gives you lines(), read_line(), and read_until():

// bufread_demo.rs
use std::io::{self, BufRead};

fn main() {
    let stdin = io::stdin();
    let handle = stdin.lock();  // locked handle implements BufRead

    println!("Type lines (Ctrl-D to stop):");
    for (i, line) in handle.lines().enumerate() {
        match line {
            Ok(text) => println!("  line {}: {}", i + 1, text),
            Err(e) => {
                eprintln!("error: {}", e);
                break;
            }
        }
    }
}

The lines() iterator strips trailing newlines and yields io::Result<String> for each line.

write! and writeln! Macros

Rust's write! and writeln! macros work on any type implementing the Write trait -- not just stdout:

// write_macro.rs
use std::io::Write;
use std::fs::File;

fn main() -> std::io::Result<()> {
    let mut f = File::create("/tmp/write_macro.txt")?;

    write!(f, "no newline")?;
    writeln!(f, " -- now with newline")?;
    writeln!(f, "pi is approximately {:.4}", std::f64::consts::PI)?;

    // Also works with Vec<u8> as an in-memory buffer
    let mut buf: Vec<u8> = Vec::new();
    writeln!(buf, "hello into a vector")?;
    println!("buf contains: {:?}", String::from_utf8(buf).unwrap());

    Ok(())
}
$ rustc write_macro.rs && ./write_macro
buf contains: "hello into a vector\n"

Do Not Mix Buffered and Unbuffered I/O

Using both write() and fprintf() on the same file descriptor leads to interleaved, corrupted output because the stdio buffer and the kernel see different states.

/* bad_mix.c -- DO NOT DO THIS */
#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(void)
{
    /* stdout is fd 1, and printf uses a buffer on fd 1 */
    printf("buffered line");        /* sits in stdio buffer */
    const char *msg = "unbuffered line\n";
    write(1, msg, strlen(msg));     /* goes directly to kernel */
    printf(" -- surprise!\n");      /* still in buffer, flushed later */

    return 0;
}

$ gcc -Wall -o bad_mix bad_mix.c && ./bad_mix | cat
unbuffered line
buffered line -- surprise!

The unbuffered write() bypasses the stdio buffer and reaches the output first. The buffered printf output appears later when the buffer flushes.

Caution: Never mix write()/read() and fprintf()/fread() on the same file descriptor. Pick one layer and stick with it.

Performance: Buffered vs Unbuffered

Let us measure the difference. Writing one million single-byte writes:

/* perf_test.c -- compare buffered vs unbuffered single-byte writes */
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <time.h>

#define N 1000000

static double now(void)
{
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return ts.tv_sec + ts.tv_nsec * 1e-9;
}

int main(void)
{
    /* Unbuffered: 1 million write() syscalls */
    int fd = open("/tmp/perf_unbuf.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    double t0 = now();
    for (int i = 0; i < N; i++) {
        write(fd, "x", 1);
    }
    double t1 = now();
    close(fd);
    printf("unbuffered: %.3f seconds (%d syscalls)\n", t1 - t0, N);

    /* Buffered: stdio batches into ~4096-byte chunks */
    FILE *fp = fopen("/tmp/perf_buf.txt", "w");
    t0 = now();
    for (int i = 0; i < N; i++) {
        fputc('x', fp);
    }
    fclose(fp);
    t1 = now();
    printf("buffered:   %.3f seconds (~%d syscalls)\n", t1 - t0, N / 4096);

    return 0;
}

$ gcc -O2 -Wall -o perf_test perf_test.c && ./perf_test
unbuffered: 1.247 seconds (1000000 syscalls)
buffered:   0.018 seconds (~244 syscalls)

The buffered version is roughly 60x faster. Each fputc copies one byte into the stdio buffer. Only when the buffer fills (~4096 bytes) does a write() syscall happen.

Unbuffered:  write("x")  write("x")  write("x")  ...  (1,000,000 syscalls)
                |            |            |
                v            v            v
             kernel        kernel       kernel

Buffered:    fputc -> [....buffer fills....] -> write(4096 bytes) -> kernel
             fputc -> [....buffer fills....] -> write(4096 bytes) -> kernel
             ...                                (~244 syscalls total)

Rust Performance Comparison

// perf_test.rs -- buffered vs unbuffered in Rust
use std::fs::File;
use std::io::{BufWriter, Write};
use std::time::Instant;

const N: usize = 1_000_000;

fn main() -> std::io::Result<()> {
    // Unbuffered: each write_all is a syscall
    {
        let mut f = File::create("/tmp/perf_unbuf_rs.txt")?;
        let t0 = Instant::now();
        for _ in 0..N {
            f.write_all(b"x")?;
        }
        let elapsed = t0.elapsed();
        println!("unbuffered: {:.3} seconds", elapsed.as_secs_f64());
    }

    // Buffered: BufWriter batches writes
    {
        let f = File::create("/tmp/perf_buf_rs.txt")?;
        let mut writer = BufWriter::new(f);
        let t0 = Instant::now();
        for _ in 0..N {
            writer.write_all(b"x")?;
        }
        writer.flush()?;
        let elapsed = t0.elapsed();
        println!("buffered:   {:.3} seconds", elapsed.as_secs_f64());
    }

    Ok(())
}

Rust Note: File::write_all does not buffer -- each call goes directly to the kernel. Always wrap File in BufWriter when doing many small writes. This is one of the most common Rust I/O performance mistakes.

Custom Buffer Sizes

The default BufWriter buffer is 8 KiB. For large sequential writes (like copying a multi-gigabyte file), a larger buffer can help:

#![allow(unused)]
fn main() {
use std::fs::File;
use std::io::BufWriter;

let f = File::create("/tmp/large_output.bin")?;
let writer = BufWriter::with_capacity(64 * 1024, f);  // 64 KiB buffer
}

In C, setvbuf does the same:

FILE *fp = fopen("/tmp/large_output.bin", "w");
char *buf = malloc(64 * 1024);
setvbuf(fp, buf, _IOFBF, 64 * 1024);
/* ... use fp ... */
fclose(fp);
free(buf);

Driver Prep: In kernel space there is no stdio. Drivers use raw copy_to_user/copy_from_user for transferring data between kernel and user buffers. Understanding why buffering matters at the user level helps you design efficient kernel interfaces.

Quick Knowledge Check

  1. Why is stderr unbuffered by default?

  2. You call printf("hello") (no newline) and then your program crashes. Does "hello" appear on the terminal? What if stdout is connected to a pipe?

  3. In Rust, what happens if BufWriter::flush() is never called and the BufWriter is simply dropped?

Common Pitfalls

  • Forgetting to flush before fork(). Both parent and child inherit the buffer contents. When both eventually flush, you get duplicate output.

  • Assuming printf output appears immediately. It does on a terminal (line-buffered), but not when piped to another process (full-buffered).

  • Using fflush(stdin). Undefined behavior in the C standard.

  • Dropping BufWriter without explicit flush(). The implicit flush on drop silently discards errors. Always flush explicitly when error handling matters.

  • Using unbuffered I/O for many small writes. The syscall overhead dominates. Always buffer.

  • Mixing buffered and unbuffered on the same fd. Output arrives in unpredictable order.

  • Not setting binary mode on Windows. On Windows, fopen without "b" translates \n to \r\n. On Linux this is not an issue, but portable code should use "wb" / "rb" for binary files.