Buffer Overflows: Writing Past the Edge and Hijacking a Program

Picture a small notebook with exactly ten lines on a page, and a rule that says you must write one word per line. Now imagine you keep writing anyway, past line ten, off the bottom of the page, and straight onto the desk underneath. On a real desk the ink just makes a mess. Inside a computer, that desk is packed with other important notes, and one of those notes tells the program where to go next. Scribble over the right note and you do not just make a mess, you take the wheel.

That, in a sentence, is the buffer overflow. It is where a great deal of modern exploitation began, and understanding it explains why entire programming languages and processor features exist. The idea is simple enough to sketch on a napkin. Put more data into a container than it can hold, and the excess spills into memory it was never meant to touch. What spills, and what it overwrites, is where the danger lives.

The core idea

First, some plain vocabulary. A buffer is just a reserved chunk of memory of a fixed size. Think of it as a labelled box that a program sets aside in advance, say room for 64 characters, to hold something like a name, a filename, or a line of text a user typed in.

Here is the catch. If a program copies input into that box without first checking how long the input is, and the input turns out to be longer than 64 characters, the extra data does not politely stop at the edge of the box. It keeps writing, straight past the end of the buffer and into whatever memory happens to sit next to it.

buffer (64 bytes) | important stuff next to it |
[    your input up to 64     ]
[    your input that is WAY too long ---------->] overwrites the neighbour

In many everyday languages this is impossible, because the language quietly checks the size for you and refuses to go over. But in C and C++, two languages that sit underneath a huge amount of the world's software, including operating systems, browsers, and network services, there is no automatic bounds check. The programmer is trusted to get the sizes right. When they do not, that neighbouring memory can be something crucial.

An overflow, then, is simply the act of writing past the end of a buffer. It sounds like a tidy technical footnote. The reason it became one of the most studied vulnerabilities in history is what happens to be sitting next door.

A tour of the stack

To see why the neighbour matters, we need to look at where these buffers usually live. Most programs are built out of functions, small named units of work that call one another. When a program calls a function, it needs somewhere to keep that function's temporary working data, and it needs to remember where to return once the function finishes.

It keeps all of this on a region of memory called the stack. The stack works like a pile of trays in a canteen. Each time a function is called, a fresh tray, called a stack frame, is placed on top. When the function finishes, its tray is lifted off and the program carries on with the one underneath.

Each tray holds a few things:

[ local buffers      ]  <- the boxes for this function's data
[ saved information   ]
[ RETURN ADDRESS      ]  <- where to jump back to when this function ends

That last item is the important one. The return address is the location in memory the program should jump back to when the current function finishes, so it can resume exactly where it left off. It is the program's way of leaving itself a note that says "when you are done here, carry on from there".

Crucially, on the stack, local buffers often sit just below the return address. So if you overflow a buffer far enough, the very next thing you start writing over is the note that steers the whole program.

Why overwriting the neighbour is dangerous

Now the pieces click together. If an attacker can feed a program input that is longer than a buffer, and they can control exactly what that overflowing input contains, they can write straight over the return address with a value of their own choosing.

1. Overflow the buffer up to and over the return address.
2. Replace the return address with one the attacker chooses.
3. When the function returns, the program jumps there instead.
4. That location holds code the attacker wants to run.

Redirecting execution like this is what turns "the program crashed" into "the attacker is running their own code". The program is no longer following its own instructions. It is following the attacker's. In the classic 1990s version of the attack, the overflowing input carried the attacker's own machine code, often called shellcode because it frequently launched a command shell, and the tampered return address pointed right back into the buffer at that code.

This single mechanism is the root of countless historic remote code execution bugs, where an attacker sends carefully shaped data over a network to a vulnerable service and ends up running commands on the far end. The Morris worm of 1988, one of the first internet worms, spread partly by exploiting an overflow in a common network program. Decades of security work descend from problems that look exactly like this.

Not only the stack

The stack overflow is the classic teaching example, but it is not the only shape this problem takes. Programs also request memory on the fly from a region called the heap, used for data whose size is not known in advance. Overflowing a buffer on the heap can corrupt the bookkeeping the program uses to track those chunks of memory, and skilled attackers have long turned that corruption into control as well.

There are close relatives too. An integer overflow, where a number wraps around past its maximum value, can trick a program into allocating a buffer that is far too small, which then overflows when real data arrives. An off by one error, writing just a single byte too far, has been enough to mount real attacks. The family is large, but they all share the same DNA. A size was not checked, and memory that mattered got overwritten.

The defences that changed the game

Buffer overflows are far harder to exploit today than in the 1990s, thanks to layers of mitigation that usually work together. No single one is a silver bullet, but stacked on top of each other they raise the difficulty enormously.

• Stack canaries. A secret, random value is placed on the stack just before the return address, like a tripwire. The name comes from the canaries miners once carried to warn of gas. Before a function returns, the program checks that this value is unchanged. If an overflow has run over it on its way to the return address, the check fails and the program aborts safely instead of jumping to the attacker's target.
• Non executable memory (NX or DEP). Memory that holds data is now marked as non executable, meaning the processor flatly refuses to run instructions from it. So even if an attacker writes their shellcode into a buffer, the CPU will not execute it. Data is data, code is code, and the two are kept apart.
• Address space layout randomisation (ASLR). The locations of a program's code and data are shuffled to different, random addresses each time it runs. Since the attacker cannot know in advance where anything lives, they cannot reliably choose a return address to aim at. It is like renumbering every house on the street each morning.
• Memory safe languages. Languages like Rust, Go, and others either check every access against the buffer's real size or manage memory automatically, so the classic overflow simply cannot happen. Rewriting critical software in these languages is one of the biggest and most consequential security shifts underway right now.

The arms race

Defenders did not get the last word. When non executable memory blocked attackers from running their own injected code, they responded with return oriented programming, a clever technique that stitches together tiny snippets of the program's own existing, already executable code, called gadgets, to perform the actions they want. Because it reuses legitimate code, the non executable rule never trips.

Defenders answered again with control flow protections that check whether the program is jumping to sensible places, and the back and forth continues. The honest summary is that the bar is now far higher than it once was, but the ceiling is not infinite. This is precisely why the industry is so interested in removing the possibility of the bug altogether rather than only making it harder to exploit.

The takeaway

A buffer overflow is what happens when a program writes past the end of a fixed size buffer, corrupting neighbouring memory, and in the classic case overwriting the return address to hijack the program's control flow. It is foundational to understanding exploitation, and its relatives reach across the stack, the heap, and even simple arithmetic. Modern systems fight it with stack canaries, non executable memory, and address randomisation, and those defences have genuinely changed the economics of attacking. But the most durable fix is memory safe languages that make the mistake impossible in the first place. When you can, do not check for the overflow. Use tools that cannot overflow.