Syscalls and the Kernel Boundary

x86 defines four protection rings; in practice operating systems use exactly two. Ring 3 is user mode, ring 0 is kernel mode, and the current ring is a field in the code segment register that only the CPU can change.

The ring is not advisory. It gates concrete capabilities in hardware:

• Privileged instructions. Writing CR3, loading the interrupt descriptor table, executing HLT or WRMSR from ring 3 raises a general protection fault.
• Memory. Every page table entry carries a user/supervisor bit. Kernel pages are simply not reachable from ring 3, enforced on every access by the same translation hardware that does paging.
• I/O. Port access and device memory are similarly gated.

So "the kernel" is not a separate program running somewhere. It is code that executes on the same core, in the same instruction stream, at a different privilege level.

A syscall is a controlled, one-way door into ring 0 at an address the kernel chose in advance.

On x86-64 the userspace side is three instructions. The number goes in rax, arguments go in rdi, rsi, rdx, r10, r8, r9, and then SYSCALL executes.

; write(1, msg, 13) by hand, no libc
  mov  rax, 1        ; __NR_write
  mov  rdi, 1        ; fd = stdout
  lea  rsi, [msg]    ; buffer
  mov  rdx, 13       ; count
  syscall            ; ring 3 -> ring 0

SYSCALL does not consult the interrupt descriptor table. It saves the return address into rcx and flags into r11, masks flags per the IA32_FMASK MSR, and loads RIP from the IA32_LSTAR MSR, which the kernel pointed at entry_SYSCALL_64 during boot. SYSRET reverses it. The syscall number is an index, checked against the table bound, into sys_call_table.

Everything between the SYSCALL and SYSRET instructions runs at ring 0 on the same core, on a different stack.

All three vector through the interrupt descriptor table, and that is where the resemblance ends. The useful split is cause and resumption.

A timer tick or a NIC packet is an interrupt: it arrives whenever, and the handler must not assume anything about what it stopped. int3 from a debugger is a trap. A page fault is a fault, which is exactly why the handler can install a translation and re-execute the very load that failed. A machine check is an abort.

A syscall is deliberate and synchronous, so it behaves like a trap even though SYSCALL skips the IDT.

Georg Sauthoff benchmarked syscall cost across several machines in "On the Costs of Syscalls". Trivial syscalls that do almost no work landed between 76 and 773 nanoseconds depending on the CPU and its mitigation state. On a Xeon Gold 6256 with mitigations disabled, getuid() cost 78 nanoseconds and close() 93. On an older Xeon E5-2643 with mitigations enabled, the same getuid() cost 464 nanoseconds.

That 4x to 5x gap is the price of Meltdown and Spectre mitigations. Kernel page-table isolation gives the kernel its own page tables, so every crossing now reloads CR3 twice and may flush TLB entries.

Brendan Gregg measured the aggregate effect after KPTI shipped: roughly 2 percent loss at 50000 syscalls per second per CPU, and a MySQL OLTP benchmark running at 75000 syscalls per second per CPU lost a measured 5 percent. The lesson is not that syscalls are slow. It is that syscall rate is a design variable.

Some syscalls return information the kernel already keeps in memory and that userspace is allowed to read. Time is the classic case, and time is queried in tight loops by every logger, tracer, and benchmark.

The virtual dynamic shared object is the kernel's answer. It maps a small, kernel-provided shared library into every process at startup. Inside it are real implementations of clock_gettime(), gettimeofday(), time(), and getcpu() that read a shared kernel data page and compute the answer with the timestamp counter, entirely in ring 3.

In Sauthoff's measurements, vDSO-served clock_gettime(CLOCK_REALTIME) cost 13 to 24 nanoseconds against 76 to 773 nanoseconds for a real syscall on the same machines. You already use it: glibc routes these calls through the vDSO automatically, which is why strace on a timing loop shows suspiciously few calls.

The vDSO trick only works for reads of kernel state. For real I/O the work must happen in the kernel, so the goal shifts from avoiding the crossing to making one crossing carry many operations.

io_uring, written by Jens Axboe and merged in Linux 5.1 in 2019, gives each ring two shared memory ring buffers mapped into both the application and the kernel: a submission queue the application writes, and a completion queue the kernel writes. Producing a request is a memory write plus a barrier, not a syscall.

One io_uring_enter() then submits an arbitrary batch and can reap completions in the same call. With submission-queue polling enabled the kernel runs a thread that drains the ring on its own, and a steady-state workload can issue I/O with effectively zero syscalls. Compare that with a read-per-request design at hundreds of nanoseconds of pure overhead each.

A monolithic kernel puts filesystems, network stacks, and device drivers inside ring 0. Calls between subsystems are ordinary function calls, so they are fast, and a driver bug can corrupt anything.

A microkernel keeps only address spaces, threads, and inter-process communication in ring 0. Filesystems and drivers become ordinary user processes, and a request that used to be a function call becomes at least two IPC round trips through the boundary you just spent five steps learning to fear. That is the whole trade: fault isolation and verifiability bought with boundary crossings.

The strongest data point for the microkernel side is seL4. Klein and colleagues, at SOSP 2009, reported the first machine-checked proof of functional correctness for a general-purpose OS kernel, over an implementation of 8700 lines of C and 600 lines of assembler. That proof is only tractable because the kernel is small, and it is small because most of the OS was pushed out of ring 0.

A namespace does not create a new kernel. It makes a global resource identifier resolve differently depending on which namespace the calling task is in. One kernel, several answers.

Per the namespaces(7) manual page, Linux currently has eight types:

• Mount, the oldest, isolating the filesystem tree.
• UTS, IPC, and Network since Linux 3.0. Network gives a namespace its own interfaces, routing tables, and socket ports.
• PID and User since Linux 3.8. PID is what makes a container's first process see itself as pid 1; User maps uid ranges, so root inside can be an unprivileged uid outside.
• Cgroup since 4.6, hiding the position in the cgroup tree.
• Time since 5.6, offsetting the monotonic and boot clocks.

A task holds one of each. clone() creates them with CLONE_NEW* flags, unshare() moves a running task into fresh ones, and setns() joins an existing one.

Namespaces control what a task can see. Control groups limit what it can consume. In cgroup v2, per the kernel's Control Group v2 documentation, there is a single unified hierarchy: every process is in exactly one cgroup, and controllers are enabled per subtree through cgroup.subtree_control.

There is no container object anywhere in the kernel. There is a task with unusual namespaces, a cgroup with limits, a root filesystem, and a seccomp filter. You can assemble one by hand:

# New PID, mount, UTS and network namespaces; pid 1 inside
$ sudo unshare --pid --mount --uts --net --fork --mount-proc \
      /bin/bash
# ps now shows two processes; hostname is independent

# Cap that shell's tree at 200 MiB and half a CPU
$ sudo mkdir /sys/fs/cgroup/demo
$ echo 209715200 | sudo tee /sys/fs/cgroup/demo/memory.max
$ echo "50000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max

cpu.max reads as 50000 microseconds of runtime per 100000 microsecond period.

• Every pointer you pass is untrusted. The kernel cannot dereference a user pointer directly; it uses copy_from_user() and friends, which validate the range and handle faults. This is also why a syscall interface is harder to write than a library one.
• Syscall count beats syscall cost. Optimising a 100 nanosecond call is pointless if you make ten million of them. Batch, or use a ring.
• strace is not free. It uses ptrace, stopping the tracee on every entry and exit. Slowdowns of an order of magnitude are normal, so never conclude anything about performance from a traced run.
• A container is a policy, not a boundary. Every namespaced process still calls into the same shared kernel, and a kernel bug is reachable from all of them. That is precisely the argument for seccomp filters and for VM-based isolation.
• Signals are delivered on the way out. Pending signals run on the return path from a syscall or interrupt, which is why a blocked call returns EINTR.