Dynamic analysis and debuggers

Static analysis sees every possible code path; dynamic analysis sees the taken one, with real values flowing through it. Each loses what the other captures:

Most real RE work is hybrid: a static pass identifies interesting functions and call sites; dynamic execution provides the runtime values that close the loop. Encrypted strings appear in plaintext mid-execution. Hash functions reveal their constants. License-validation logic flows through real inputs. The static view tells you where to look; the dynamic view tells you what is actually happening at that point.

Four debuggers cover most reverse engineering:

Across all of them, the same primitive operations:

• Run / continue / interrupt — start or pause execution.
• Step — single instruction (stepi), single line (step/next if source available), or over function calls (nexti).
• Breakpoints — pause execution when reaching an address.
• Watchpoints — pause when a memory location is read or written.
• Read/write memory and registers — at any pause point.
• Scripting — gdb's Python API, lldb's commands, x64dbg's scripts, WinDbg's JavaScript engine.

The scripting layer is what makes a debugger an analysis tool rather than a step-through interface. Logging every call to a specific function with its arguments and return value is a five-line script in any of these.

Three breakpoint mechanisms, with different trade-offs:

• Software breakpoint (INT3 / 0xCC). The debugger overwrites the target byte with 0xCC (int3 — debug trap). When the CPU hits it, it raises an interrupt; the OS routes the trap to the debugger. On resume, the debugger restores the original byte, single-steps one instruction, then puts the 0xCC back. Cheap and unlimited in count, but modifies the code — visible to anti-debug code that checksums itself or scans for 0xCC.
• Hardware breakpoint (DR0–DR7 on x86). The CPU has four debug registers that trigger a trap when a specific address is executed, read, or written. Invisible to code inspection, but you only get four of them. Used by watchpoints and by stealthier breakpoints on anti-debug-aware targets.
• Page-fault breakpoint. Mark the target page as non-executable (or non-readable / non-writable for watchpoints) via the page table; the CPU faults on access; the OS notifies the debugger. Unlimited and stealthy but coarse-grained (page-sized) and slower than the other two. Used by some advanced debuggers for ranges that would exhaust hardware slots.

Knowing which mechanism you're using matters. INT3 breakpoints inside critical anti-debug code get detected. Hardware breakpoints on IsDebuggerPresent survive checksum sweeps but conflict with the next analyst who wants the same DR slot. Page-fault breakpoints survive both but slow execution noticeably.

Sometimes you don't need to pause execution — you need to log what the binary did. The tracing tools by platform:

# Linux: system calls and library calls.
strace -f ./app                  # syscalls + arguments
strace -e trace=network ./app    # only network-related calls
ltrace ./app                     # libc / library calls
ltrace -l libssl.so.3 ./app      # only calls into libssl

# Linux: kernel-level tracing.
perf trace ./app                 # newer, BPF-backed
bpftrace -e 'tracepoint:syscalls:sys_enter_open { @[comm] = count(); }'

# macOS:
dtruss ./app                     # roughly strace for dtrace
fs_usage / sc_usage              # filesystem + syscall view

# Windows:
Process Monitor                  # filesystem + registry + network + process events
API Monitor                      # hook arbitrary API calls
ETW (Event Tracing for Windows)  # the kernel-side tracing substrate

Tracing is non-invasive in the sense that it doesn't pause the process, but it reveals only what crosses the boundary you're watching — syscalls, library calls, registry reads, file I/O. Internal state changes are invisible to a pure tracer. Combine with debug pauses at interesting points to get both: the call sequence from the trace, the values at each call from the debugger.

Dynamic Binary Instrumentation (DBI) injects code into a running process at the instruction level. The major frameworks:

• Frida. JavaScript-driven; cross-platform (Linux, macOS, Windows, iOS, Android); the fastest path to scripted hooks. Used widely for mobile RE and for runtime API monitoring.
• PIN (Intel). C++ instrumentation framework; very fast; the go-to for performance and architectural analyses. Older but battle-tested.
• DynamoRIO. Open source; closer to the metal than PIN; supports many architectures.

Frida's hook pattern in 10 lines:

// Inject into a running app; hook crypto_verify and log every call.
Interceptor.attach(Module.findExportByName(null, "crypto_verify"), {
  onEnter(args) {
    this.input = args[0];
    console.log("[verify] input =", Memory.readUtf8String(args[0]));
  },
  onLeave(retval) {
    console.log("[verify] returned", retval);
    if (retval.toInt32() === 0) {
      console.log("[verify] forcing success");
      retval.replace(1);
    }
  },
});

Three things this enables that pure debugging does not:

• High-throughput logging without pausing the process — useful for protocols and games.
• Inline transformation of arguments and return values — the script can rewrite behavior, not just observe it.
• Coverage and trace collection at scale — every call to read(), every comparison of an input byte against a key byte, captured to a log for offline analysis.

Binaries that resist analysis check for a debugger and behave differently when one is attached. Common techniques and what they look for:

These checks are visible in static analysis. The standard playbook: read the binary statically to identify which anti-debug methods are in use, plan bypasses before attaching a debugger, and confirm with one dynamic run that the bypass holds before doing serious analysis.

Sometimes you don't want to run the binary on your real machine — you want to execute fragments in a controlled environment, or run code for a different architecture, or simply isolate untrusted samples. The relevant tools:

• CPU emulators. Unicorn is a CPU-only emulator extracted from QEMU's TCG; it lets you load arbitrary bytes, set up memory, run, and read registers afterwards — no OS, no syscalls. Useful for shellcode analysis and small algorithmic chunks (decrypt a buffer with the binary's own routine without launching the full program).
• OS emulators. Qiling layers Unicorn with userspace OS emulation (syscalls, libc, file system). Lets you run an ELF or PE binary on a different host, with hooks at any syscall.
• Full-system emulation. QEMU (full system) boots a complete VM. Combine with PANDA for whole-system replayable traces.
• Symbolic execution. angr runs a binary symbolically — every input is a variable; the engine explores branches and accumulates path constraints. Solves "find the input that makes this return 0 branch run" automatically. Powerful for crackmes; expensive to scale.
• Sandboxes for malware. Cuckoo, CAPE, vendor sandboxes (Hybrid Analysis, Joe Sandbox) — run the binary in a snapshotted VM, log everything, snapshot back.

The choice scales by scope: Unicorn for a function, Qiling for a small binary, QEMU for a whole system, sandbox services when you need the report and not the process.

Mature RE is a loop, not a sequence:

┌────────────────────────────────────────────────┐
│  1. Static triage (format, strings, imports)   │
│     → form hypothesis about purpose            │
└────────────────┬───────────────────────────────┘
                 ▼
┌────────────────────────────────────────────────┐
│  2. Static disassembly + decompilation         │
│     → identify interesting functions           │
└────────────────┬───────────────────────────────┘
                 ▼
┌────────────────────────────────────────────────┐
│  3. Dynamic run with breakpoints / tracing     │
│     → observe actual values, branches taken    │
└────────────────┬───────────────────────────────┘
                 ▼
┌────────────────────────────────────────────────┐
│  4. Annotate the static view with what was     │
│     learned dynamically                        │
└────────────────┬───────────────────────────────┘
                 │
                 └──── back to (2) with new questions

The key insight: every dynamic observation should feed back into the static model. Saw the value 0x52415459 at this comparison? Note it as a magic-number check in the decompiler. Saw a function called five times with the same first argument? That's a constant — rename it. Saw a string decrypted at this point? Add a comment with the plaintext.

The binary doesn't change. Your map of the binary grows after each loop. A well-annotated decompiler project, after several rounds of static + dynamic + annotation, is the real artifact of RE work — not any single insight, but a reference document that the next pass can build on without redoing the work.