Disassembly and decompilation

x86-64 instructions are variable-length, from 1 byte (ret = 0xC3) to 15. Code and data can sit adjacent in the same section. Embedded jump tables, alignment padding, and switch-case dispatchers regularly look like "code" but aren't reachable as such. Decoding 1MB of .text into instructions correctly is the first hard problem.

Two classical strategies:

• Linear sweep. Start at the section's first byte, decode forward, advance by the instruction length. Simple. Fails on inline data — once you decode a misaligned byte as the start of a multi-byte instruction, every following decode is wrong until alignment by luck.
• Recursive descent. Start at known entry points (export table, _start, function symbols). Decode each instruction; for call and jmp targets, recurse; for conditional jcc, recurse both branches; stop at ret/hlt/ud2. Misses code reached only via computed jumps (switch tables, function pointers, vtables) — which is why disassemblers also run heuristics over the gaps.

Modern tools (Ghidra, Binary Ninja, IDA) combine recursive descent with linear sweep over unknown ranges and call-graph reconstruction. ARM, RISC-V, and most other architectures have fixed-width instructions, which sidesteps most of this — variable-length encoding is an x86 inheritance from the 8086 era.

Two dominant calling conventions; the platform decides which one a binary uses:

System V AMD64 (Linux, macOS, BSD)

• First 6 integer arguments: rdi, rsi, rdx, rcx, r8, r9.
• First 8 floating-point arguments: xmm0–xmm7.
• Return value: rax (and rdx for 128-bit returns).
• Caller-saved: rax, rdi, rsi, rdx, rcx, r8–r11. Callee-saved: rbx, rbp, r12–r15.
• The stack is 16-byte aligned at call boundaries.

Microsoft x64 (Windows)

• First 4 integer arguments: rcx, rdx, r8, r9. Remaining on the stack.
• First 4 floating-point arguments: xmm0–xmm3.
• Return value: rax.
• Shadow space — caller reserves 32 bytes on the stack for the callee's first 4 args (even though they came in registers). The callee can spill them there.
• Caller-saved: rax, rcx, rdx, r8–r11. Callee-saved: rbx, rbp, rdi, rsi, r12–r15.

Knowing which convention is in effect lets you read calls instantly: on Linux, mov rdi, addr before call is first argument; on Windows, it's mov rcx, addr. Misidentifying the convention turns every function call into nonsense.

Most compiled x86-64 functions follow a near-universal frame pattern:

; --- prologue ---
push rbp              ; save caller's frame pointer
mov  rbp, rsp         ; rbp now points to this frame
sub  rsp, 0x40        ; reserve 64 bytes for locals

; --- body ---
mov  [rbp-0x8], rdi   ; spill arg1 to a local slot
...

; --- epilogue ---
leave                 ; mov rsp, rbp ; pop rbp
ret                   ; return

Reading the prologue tells you immediately:

• Local size. sub rsp, 0x40 → 64 bytes of locals + spilled args.
• Calling convention being respected — register saves at the start.
• Function boundaries. The disassembler finds function starts by looking for prologue patterns the compiler emitted; in a stripped binary this is one of the few reliable signals of where one function ends and the next begins.

Frame-pointer-omitted code (-fomit-frame-pointer) skips push rbp / mov rbp, rsp and uses rsp-relative addressing for locals. This saves a register and one instruction; it also makes manual stack-trace reconstruction harder and is the default in optimized release builds. Decompilers handle either shape, but reading raw asm is slightly harder when the frame pointer is gone.

A basic block is a maximal run of straight-line code: enters at the top, exits at the bottom, no branches in or out in the middle. Every disassembler builds these as the foundation for further analysis:

┌────────────────────┐
│  block A           │
│  cmp eax, 0        │
│  je   B            │
└────────┬───────────┘
         │ false           ┌─────────────────────┐
         ▼                 │  block B            │
   ┌────────────┐          │  mov eax, 1         │
   │ block C    │ ◀────────│  jmp D              │
   │ mov eax, 0 │          └────────┬────────────┘
   │ jmp D      │                   │
   └─────┬──────┘                   │
         │                          │
         ▼                          ▼
        ┌───────────────────────────────┐
        │  block D — merge              │
        └───────────────────────────────┘

The Control Flow Graph (CFG) connects basic blocks by their possible successors. From the CFG, disassemblers derive:

• Dominators and loops — which blocks must execute before reaching a given block; where loops start and end.
• Structural reconstruction — replacing edge spaghetti with if/else, while, for, switch shapes the decompiler can emit as C.
• Reachability and dead-code detection — blocks the recursive descent never reached.

A CFG is independent of any specific decompiler. Once built, the same graph can be rendered as assembly with arrows, as a structured C-like pseudo-code, or analyzed for security properties (taint, reachability of a sink).

A decompiler lifts disassembly into a C-like representation. Internally it usually pipes through an intermediate representation (Ghidra's P-Code, Binary Ninja's BNIL, IDA's microcode), normalizes the IR, runs simplification passes (constant folding, dead-store elimination, structural recovery), and prints the result.

What it can recover, often quite well:

• Control flow — if/else/while/for shapes from the CFG.
• Function calls and their argument counts — by tracking register and stack usage at the call site.
• Basic type inference — "this is a 4-byte read followed by a 4-byte write through the same register" → int32_t.
• String constants — references into .rodata become "hello".
• Loop structure — the dominator analysis recovers natural loops.

What it cannot recover (no algorithm can):

• Local variable names. The compiler emitted offsets, not names. var_8, var_10 is the best a decompiler can do.
• Struct field names — only the offsets remain. With a known library, signature databases (FLIRT in IDA) can restore them; otherwise they are blank.
• Inlined function boundaries. Once a function is inlined, it is dissolved into the caller and is not reconstructable.
• Macro expansions, original comments, original code formatting — gone at compile time.

Decompilation is lossy and one-way. The output is a best-effort reconstruction of the same semantics, not the original source.

Four tools dominate practice; the choice is partly philosophical, partly task-shaped:

For most learners, Ghidra is the right starting point: free, multi-platform, multi-architecture, with a decompiler that for typical binaries is within reach of Hex-Rays' output. Move to IDA when you need the maturity around obscure compilers or large-team annotations; move to Binary Ninja when you want a programmable IR to build custom analyses on; move to radare2/rizin when you need everything driven from the command line.

C++ adds three concepts that surface differently in disassembly:

• Name mangling. Foo::bar(int) becomes _ZN3Foo3barEi (Itanium ABI, used by GCC/Clang) or ?bar@Foo@@QEAAHH@Z (MSVC). c++filt (or the decompiler's built-in demangler) reverses it. Mangled names survive stripping when they are exported.

• Vtables. Each polymorphic class has a virtual table — a .rodata array of function pointers. An object's first word is a pointer to its vtable; a virtual call compiles to:

  mov rax, [rcx]       ; rcx = `this`, [rcx] = vtable pointer
  call qword ptr [rax + 0x10]  ; call vtable[2]
  

  Reverse-engineering classes means recovering vtables: walk .rodata, identify arrays of function pointers, and the indices encode the class layout.

• Exception handling. GCC/Clang emit .eh_frame and .gcc_except_table — DWARF-encoded unwind information used by the C++ exception runtime. MSVC emits .pdata / .xdata. These tables describe how to unwind the stack and which catch blocks match which throw types. Heavy parsing reveals try/catch structure that is otherwise invisible.

C++ binaries are noticeably more complex than C. The trade-off — modern decompilers handle the common cases (vtables, name demangling) automatically; bespoke smart-pointer types and template-heavy code still produce dense, hard-to-read pseudocode.

Decompilation gets you close, but a list of things no decompiler can ever return to you:

• Identifier names — local variables, parameters, struct fields. Without DWARF/PDB symbols, only offsets remain.
• Macros and #includes — preprocessed away before code generation.
• Original control flow shape. A compiler is free to turn a for into a goto, unroll a loop, hoist invariants out, sink computations into branches, vectorize the body. The decompiler produces one valid C reconstruction of the semantics; there are infinitely many sources that compile to the same binary.
• Inlined functions. Once dissolved, the boundary is gone. Reconstructing it requires recognizing the original function's code shape — sometimes possible with signature databases, often not.
• Optimizations that lose information. Constant propagation, dead-code elimination, common-subexpression elimination — each pass discards source-level structure deliberately. The binary records what was computed, not how the source expressed it.

The practical reading: decompilation is a starting point. The output needs renaming (giving the var_X locals meaningful names), retyping (annotating that arg1 is struct Header *), and annotating (marking which functions are import wrappers, which are math, which are I/O). The decompiler does the mechanical work; the analyst does the recovery of intent.