Stack-based buffer overflows: a walkthrough that actually explains the exploit

Buffer overflows are the original computer security vulnerability — the Morris worm in 1988 used one — and they're still the textbook starting point for exploit development because every other technique builds on the primitives they teach. This walkthrough goes from the smallest vulnerable C program to RIP control on a 64-bit Linux binary, naming the moving parts at each step.

The vulnerable program

// vuln.c — compile with: gcc -fno-stack-protector -no-pie -z execstack -o vuln vuln.c
#include <stdio.h>
#include <string.h>

void win() {
    printf("you got it\n");
    system("/bin/sh");
}

int main(int argc, char **argv) {
    char buf[64];
    strcpy(buf, argv[1]);   // ← unbounded copy, attacker controls argv[1]
    return 0;
}

Three things to notice: (1) buf is 64 bytes on the stack, (2) strcpy copies until it hits a null byte with no length check, (3) main has a useful win() function the attacker would like to call. The compile flags disable every modern mitigation; we'll re-enable them one at a time later.

Step 1 — Find the offset to the return address

The stack frame for main looks roughly like:

[ buf (64 bytes) ][ saved RBP (8) ][ saved return addr (8) ]
^                                                          ^
RSP-72                                                    RSP+8

Send 64 + 8 = 72 bytes of "A", then 8 distinguishable bytes — when the program crashes, those 8 bytes are now in RIP. Use a cyclic pattern (pwntools' `cyclic(200)`) to find the exact offset programmatically: feed the pattern, read the value RIP got, and compute the index back.

Step 2 — Overwrite RIP with win()

On a binary with no PIE, win() lives at a fixed address — readelf -s vuln | grep win gives you the symbol value. Build the payload:

from pwn import *
elf = ELF('./vuln')
payload = b'A' * 72 + p64(elf.symbols['win'])
process('./vuln', argv=[b'./vuln', payload]).interactive()

Run it. You get a shell. Done — for the simplest case.

Step 3 — Reality intrudes (mitigations)

The flags we disabled at compile time exist for a reason. Let's turn each one back on and see what breaks.

Stack canary (-fstack-protector)

GCC inserts a random 8-byte cookie between buf and the saved RBP/return address on entry, and checks it before ret. Overflow that overwrites the canary causes __stack_chk_fail to abort the process. Bypasses: leak the canary first via a separate vulnerability (format string, partial read), then include the leaked canary in your payload at the right offset.

PIE / ASLR (-pie)

Without PIE, win() is at a fixed address. With PIE, the entire binary is mapped at a randomized base address each run. Bypass: leak the base via any output that prints a code pointer (a function pointer in a printable struct, a libc address from a leaked GOT entry, etc.), subtract the known offset, and now you know where win() is this run.

DEP / NX (-z noexecstack — the default)

The stack is no longer executable, so jumping to shellcode-on-the-stack doesn't work. Modern bypass: ROP. Find short instruction sequences ending in ret in the binary's executable code, chain them together by pushing each one's address consecutively. The ret-driven dispatch executes your gadget chain. ROPgadget or ropper enumerates available gadgets.

RELRO (-Wl,-z,relro,-z,now)

Marks the GOT read-only after dynamic linking, so you can't overwrite a GOT entry mid-execution to redirect a libc call. Affects technique selection: ROP-based libc-resolution becomes the standard approach instead of GOT-overwrite.

The full modern chain — ret2libc with leak

On a binary with PIE + canary + NX + full RELRO (the typical Linux distro default), the workflow becomes:

1. Find a leak primitive (format string, OOB read, info-disclosure bug elsewhere in the program).
2. Use the leak to recover the stack canary, the binary base, and ideally a libc address.
3. Compute system() and "/bin/sh" string addresses in the now-known libc layout.
4. Build a ROP chain: pop rdi gadget, address of "/bin/sh", system(). Pad to overwrite RIP.
5. Send the payload via the original buffer overflow primitive, with the canary at its correct offset so __stack_chk_fail doesn't trip.

Why this matters

Buffer overflows are largely defeated in modern code that compiles with all mitigations enabled — but the techniques you learn fighting them (ROP, leak primitives, gadget chains, mitigation interaction) are the foundation for browser exploitation, kernel exploitation, sandbox escapes, and every other corner of memory-corruption-based offensive security. Skipping the basics doesn't mean you understand modern exploits; it means you can paste exploit-db one-liners.

I have a 64-bit Linux binary with PIE and canary enabled. I have a stack-based buffer overflow primitive but no obvious leak. Walk me through the standard approaches to find a leak primitive, and what to look for in the binary's existing functions.