Filesystems and the I/O Path

The inode is the file. It holds the type, permissions, owner, size, timestamps, a link count, and the map from file offsets to disk blocks. What it does not hold is a name.

A directory is just a file whose contents map names to inode numbers. So a "name" is a directory entry, and several entries may point at the same inode. That is a hard link, and it is why the inode carries a link count rather than a name.

unlink() therefore does not delete a file. It removes one directory entry and decrements the link count. The inode and its blocks are freed only when the link count reaches zero and no process still holds the file open. This is the whole explanation for the classic operations puzzle: you delete a 40 GiB log, df reports no change, and the space returns the instant you restart the process holding it open.

The kernel documentation describes the Virtual File System as "the software layer in the kernel that provides the filesystem interface to userspace programs", and the abstraction "which allows different filesystem implementations to coexist".

It is built from four object types, each with a table of function pointers a filesystem fills in:

• superblock - a mounted filesystem instance, with super_operations.
• inode - a filesystem object, with inode_operations. On disk for ext4, synthesised on the fly for procfs.
• dentry - a cached name-to-inode link. The docs are explicit that dentries "live in RAM and are never saved to disc: they exist only for performance".
• file - an open file description: a dentry pointer, a position, and file_operations.

This is why read() works identically on ext4, NFS, procfs, and a FUSE mount. The syscall reaches the VFS, which calls whatever read pointer that inode's filesystem installed.

A write() to a normal file does not reach the device. The VFS finds or allocates page-cache pages for that inode and offset range, copies your bytes in, marks the pages dirty, and returns. The call is a memory copy that happens to update kernel bookkeeping.

Reads work the same way in reverse: a hit on a cached page never touches storage, and sequential access triggers readahead so the next pages arrive before you ask.

Writeback is governed by thresholds initialised in the kernel's mm/page-writeback.c. dirty_background_ratio starts at 10, the percentage of memory at which background writeback threads begin flushing. vm_dirty_ratio starts at 20, at which point the writing process itself is made to do writeback and stalls. dirty_writeback_interval is 5 seconds and dirty_expire_interval is 30 seconds, the longest a page is allowed to stay dirty.

So a successful write() means "in RAM". Nothing more.

Note where the call returns. Everything after that point happens without the application watching.

O_DIRECT asks the kernel to skip the page cache and DMA between your buffer and the device. It exists for applications that already maintain their own cache and do not want a second, dumber one competing for RAM. Database engines are the canonical users, and the trade-offs are covered from the database side in the storage and pages lesson.

The last row is the trap. O_DIRECT bypasses the page cache, not the device's volatile write cache. Unless the write is issued with a force-unit-access flag or followed by a flush, it can still be lost on power failure.

fsync(fd) asks the kernel to push every dirty page for that file down the stack and, critically, to issue a cache-flush command to the device so the data leaves the drive's volatile buffer. It returns when the device says the data is on stable media.

Three things it does not do, each a real bug in the wild:

1. It does not persist the directory entry. Create a file, write it, fsync() it, and after a crash the file may not appear in its directory. You must also fsync() the directory file descriptor.
2. It does not cover other file descriptors. It is per open file, not per filesystem. syncfs() is the filesystem-wide version.
3. It cannot fix a lying device. If the drive acknowledges a flush it has not completed, no kernel code can detect that.

fdatasync() is the cheaper sibling: it skips metadata not required to read the data back, such as the modification timestamp, saving an inode write per call.

Replacing a file safely is four operations, and every one of them is load-bearing.

int fd = open("config.json.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0644);
write(fd, buf, len);
fsync(fd);        /* 1: the new contents are on stable media */
close(fd);

rename("config.json.tmp", "config.json");  /* 2: atomic swap */

int dir = open(".", O_RDONLY|O_DIRECTORY);
fsync(dir);       /* 3: the rename itself is now durable */
close(dir);

Drop step 1 and a crash can leave a correctly named file full of zeros, because the rename reached the journal before the data reached the media. Drop step 3 and the crash can lose the rename entirely, leaving the old file and an orphan temp file.

rename() over an existing name is atomic on POSIX filesystems: a reader sees either the old inode or the new one, never a partial file. That atomicity is what the whole pattern is built on.

A failed fsync() is not a retryable error, and for twenty years almost nobody knew.

When writeback fails, Linux historically dropped the failed pages and marked them clean rather than keeping them dirty for a retry, which otherwise lets an unplugged USB stick pin memory forever. The error is then reported to a file descriptor once and cleared. Jonathan Corbet's LWN write-up of the 2018 incident notes that Linux 4.13 improved reporting with errseq_t tracking, but a descriptor opened after the error still sees nothing.

PostgreSQL hit exactly this: Craig Ringer reported in March 2018 that a checkpoint saw EIO, retried, got success from the second fsync() because the flag had been cleared, and declared durable data that was gone. PostgreSQL now panics on fsync() failure instead of retrying.

Rebello and colleagues followed up at USENIX ATC 2020 in "Can Applications Recover from fsync Failures?", testing ext4, XFS and Btrfs against PostgreSQL, LMDB, LevelDB, SQLite and Redis, and found no application strategy sufficient to avoid data loss or corruption.

Appending one block to a file is not one write. It is at least three: allocate the block in the free-space bitmap, update the inode's size and block map, and write the data. A crash between any two leaves the filesystem inconsistent, and some inconsistencies are worse than data loss. An inode pointing at a block the bitmap thinks is free will hand your bytes to the next file that allocates it.

A journal makes the group atomic. Write the intended metadata changes to a dedicated on-disk area, flush, mark the transaction committed, and only then apply the changes in place. After a crash, recovery replays committed transactions and discards incomplete ones. That turns an fsck proportional to filesystem size into a replay proportional to journal size.

The important subtlety: the journal protects metadata structure by default. It is not a promise about your file's contents.

The kernel's ext4 documentation defines three, selected at mount time:

• data=journal - all data and metadata go through the journal. Safest, and every byte is written twice.
• data=ordered - the default. Only metadata is journalled, but ext4 "write[s] all data before committing the associated metadata to the journal". The ordering is the point: metadata never becomes visible pointing at blocks whose contents have not landed.
• data=writeback - metadata is journalled with no data ordering at all. The docs warn this "can cause incorrect data to appear in files which were written shortly before the crash": correct structure, stale or garbage contents.

The gap between ordered and journal is exactly where application-level durability lives. data=ordered guarantees the filesystem is coherent after a crash; it does not guarantee your last write() survived. That is what fsync() is for, and why databases layer their own logging on top rather than trusting the filesystem journal.

Once writeback runs, the filesystem builds bio structures, each describing a transfer between a set of pages and a device range. Since Linux 3.13 these enter blk-mq, the multiqueue block layer: per-CPU software queues that merge and sort, feeding hardware queues that map one-to-one onto NVMe submission and completion queue pairs. The old single request queue with its global lock could not saturate an SSD; this design can, which is why NVMe devices commonly run with no I/O scheduler at all.

Underneath, the abstraction leaks. Flash cannot overwrite in place: a page is programmed once and can only be reset by erasing its whole block, often megabytes. A flash translation layer therefore writes updates elsewhere and remaps, garbage-collecting stale pages in the background. The consequences are visible from above: a 4 KiB update can cost far more than 4 KiB of flash wear, and latency is occasionally dominated by garbage collection you cannot see or schedule.

• A successful write() means nothing about durability. It means "copied into RAM". Only fsync() or fdatasync() changes that.
• Never retry a failed fsync(). The second call can return success while the data is gone. Treat it as fatal, the way PostgreSQL now does.
• fsync() the directory after creating or renaming. Otherwise the file exists and its name does not.
• Free space and deleted files are different questions. df counts blocks; a deleted-but-open file still holds them. lsof +L1 finds them.
• O_DIRECT is not a durability feature and not a speed feature. It removes a cache. If you do not have a better cache of your own, you will simply be slower.
• Benchmarks that never call fsync() are measuring your RAM. Any storage comparison without an explicit sync policy is measuring the page cache.