Binary formats and what the loader does

A binary is the bottom of a pipeline. To reverse-engineer it intelligently you need the layers above it.

source.c ──compile──▶ source.o (object)
                       │
source.h               │
                       ▼
                     link ──▶ a.out / app.exe (executable)
                       │
                       ▼
                   exec() ──▶ image in process memory ──▶ entry point

• Compile. Source → object file. Each object holds the compiled code for one translation unit plus unresolved references to symbols defined elsewhere.
• Link. Object files + libraries → one executable. The linker resolves cross-object references, lays out sections, and decides which libraries are bound at link time vs deferred to the loader.
• Load. The OS reads the file, maps segments into memory, applies any relocations, hands control to the entry point.

A static binary inlines all library code. A dynamic binary defers most library resolution to the loader. Almost every system binary you will reverse is dynamic — meaning the file alone does not contain everything that runs. The loader and the shared libraries it pulls in are part of the picture from the first jump.

ELF (Linux, BSD, Solaris), PE (Windows), and Mach-O (macOS, iOS) all encode the same thing: an executable image plus metadata for the loader. They differ in syntax and historical baggage, not in concept.

The concepts move between formats; only the field names and the tools change. Modern macOS binaries also wrap multiple Mach-Os in a fat binary (lipo) to ship arm64 + x86_64 in one file. Modern Linux binaries can be static, dynamic, PIE (Position Independent Executable), or stripped of debug info — all common configurations before you ever see them. Recognising the format is one file invocation away; the rest of this lesson assumes ELF as the running example, with PE/Mach-O footnotes where they diverge meaningfully.

A binary has two layouts at once:

• Section view — link-time. Many small chunks named .text, .rodata, .data, .bss, .debug_info. The linker thinks in sections; readelf -S and objdump -h show them.
• Segment view — load-time. A handful of contiguous regions described by program headers (ELF), load commands (Mach-O), or characteristics flags (PE). The loader thinks in segments and maps each one with specific permissions: R-X for code, RW- for data, R-- for read-only.

The mapping is many-to-one: .text and .rodata typically share an R-X segment; .data and .bss share an RW- segment. .debug_info is rarely loaded into memory at all.

Why this matters for RE: if you are looking for executable code, search R-X segments; for hard-coded strings, search R-- segments. The page-permission boundaries also encode the loader's policy — W^X (writable XOR executable) means the loader refuses to map a region as both writable and executable, unless the binary's program header explicitly requests it. Code that tries to JIT or self-modify has to call mprotect/VirtualProtect at runtime, which is itself a useful signal.

main() is not the binary's entry point. Several invisible steps run first:

1. Kernel side. exec() reads the file header, validates magic numbers, maps the segments into the new address space, sets up stack and heap, jumps to the binary's declared entry point.
2. Dynamic linker. For dynamic binaries, control first lands in ld-linux.so / dyld / the Windows loader. This component loads every required shared library, resolves imports (function names referenced here that live elsewhere), and patches the binary's import table with the resolved addresses.
3. C runtime init. _start → __libc_start_main → .init_array constructors → main.

Two structures dominate dynamic-import resolution on ELF systems:

• PLT (Procedure Linkage Table) — small trampolines for each imported function. First call jumps via the GOT to the dynamic linker; subsequent calls go straight to the resolved function.
• GOT (Global Offset Table) — writable table of resolved addresses. The PLT reads from it.

On Windows the equivalent is the IAT (Import Address Table). Knowing whether a call goes through the PLT/IAT or direct tells you immediately whether the callee lives in this binary or in a library that the loader brought in.

A symbol is a name attached to an address. ELF has two tables with very different lifecycles:

• .symtab — the full symbol table. Needed at link time, not needed at runtime. Strippable — strip foo removes it entirely.
• .dynsym — the dynamic symbol table. Lists exports and imports needed for runtime dynamic linking. Not strippable — the dynamic linker reads it to resolve cross-library references.

What stripping removes: names of functions and variables defined inside this binary. What it leaves: names of imported libc and library functions (in .dynsym), and often the names of exports — symbols this binary exposes for others to link against.

Practical implication: a stripped binary still tells you which libc functions it calls. printf, socket, pthread_create, OpenProcess, RegOpenKeyExA all survive stripping because they are imports. What is hidden is the binary's own function structure — that's where decompilation and pattern-matching against known compiler-generated idioms (function prologues, libc wrappers, C++ vtable layouts) does the heavy lifting in the next lesson.

Before you open a disassembler, the command-line tools tell you a great deal:

# What is this file?
file target.bin           # ELF 64-bit LSB executable, x86-64, dynamically linked

# Hard-coded strings — informative out of proportion to the effort.
strings -n 8 target.bin   # only strings of length >= 8

# ELF headers, sections, program headers.
readelf -h target.bin     # file header
readelf -S target.bin     # sections
readelf -l target.bin     # segments / program headers

# Symbols — exported, imported, defined locally.
nm target.bin             # all symbols
nm -D target.bin          # dynamic symbols only

# Disassembly without a GUI.
objdump -d target.bin     # text disassembly

The Windows equivalents: dumpbin, pefile (Python), Detect-It-Easy (multi-format ID + packer detection). On macOS: otool, nm, dyld_info.

A five-minute triage often surfaces the compiler (GCC: (Debian 12.2.0) strings), language (runtime.morestack for Go, called Result::unwrap on an Err for Rust), library versions (libcurl/8.4.0, OpenSSL 3.0), and error messages that telegraph functionality. Five minutes spent here saves hours of misdirected disassembly later.

Some binaries actively resist analysis. The first line of defense is packing: compress or encrypt the original code, replace it in the file with a small unpacker stub, and have the stub decode the real payload at runtime and jump to it.

UPX is the canonical packer — open-source and used by both legitimate compact-binary deployments and malware. Tell-tale signs of a UPX-packed file:

• Section names like UPX0, UPX1, UPX2.
• Very high entropy in the data section — close to 8 bits per byte indicates compressed or encrypted content.
• A tiny .text and a large compressed .data.

The unpacker stub executes, fills memory with the decoded program, and jumps to the Original Entry Point (OEP) — where the program would have started without packing. Static analysis of a packed binary mostly sees the stub; the real code only appears after the OEP jump. UPX is reversible with upx -d; bespoke packers are not.

Obfuscation pushes further: control-flow flattening, opaque predicates, virtualization (the binary embeds its own VM and the "code" is bytecode for that VM — Themida, VMProtect). Each layer shifts work from authoring (cheap) to analysis (expensive). Knowing which technique you are facing before committing time is itself a skill.

By the end of triage you can often guess the binary's purpose without reading a single instruction:

• Compiler fingerprint. Strings reveal GCC: (Debian 12.2.0) 12.2.0, clang version 16.0, or MSVC RTC strings. The compiler narrows what idioms to expect — function prologue shapes, stack-canary placements, exception-handling layout.
• Language. Go runtime symbols (runtime.morestack), Rust panic strings (called Result::unwrap on an Err), .NET CLR headers, Dart kernel snapshots, PyInstaller bootloader strings — each leaves fingerprints visible to strings or readelf.
• Library inventory. libcurl/8.4.0, OpenSSL 3.0, Qt 6.5, nodejs — the dependency list often telegraphs entire subsystems. Reverse-engineering a function that wraps a known library is a different (easier) task than reversing a custom implementation.
• Functional verbs. getenv("HOME"), bind(), WSAStartup, RegSetValueEx, error strings about license validation or update checks — concrete behaviors leak through error messages and imports.

A disassembler tells you how. The format + headers + strings + symbols tell you what. The most expensive RE mistakes come from skipping straight to disassembly, anchoring on a wrong assumption about the binary's purpose, and reading every instruction through that wrong lens for hours.