Segmentation faultfinding the line the signal will not give you

#What the signal tells you

Here is the entire output of a program that dereferences a null pointer, built -O0 -g and run:

$ ./null
Segmentation fault (core dumped)

That line is printed by your shell, not by your program. The program said nothing, because it was killed by SIGSEGV between one instruction and the next. There is no line number in that message because there is no line number in the signal. The kernel knows an address; it does not know your source.

Everything below is about getting from that to a file and a line.

#Seven bugs, three tools

Seven small programs, one bug each, GCC 11.4.0 at -O0 -g. The question in each column is the only one that matters: does this tool name the line where the bug is?

One column is right seven times out of seven. If you take one thing from this page: rebuild with -fsanitize=address -g and run it again. That is the whole technique for most segfaults, and it takes less time than reading a backtrace.

UBSan is not a weaker ASan, it is a different tool. It found the null store because dereferencing a null pointer is undefined behaviour, and it was silent on the rest because a heap overflow is not. Run both: -fsanitize=address,undefined.

#When the crash lands in libc

This is the case that makes people distrust debuggers, and they are right to. A strcpy overflows a char name[8] and writes over the next pointer that follows it in the struct. Nothing happens. The program crashes later, walking the list:

struct node { char name[8]; struct node *next; };

static void set_name(struct node *n, const char *s) {
    strcpy(n->name, s);                       /* line 7: the bug */
}
static void walk(struct node *n) {
    while (n) { printf("%s\n", n->name); n = n->next; }   /* line 10: the crash */
}

#0  0x0000fffff7e9c88c in ?? ()
    from /lib/aarch64-linux-gnu/libc.so.6
#1  0x0000fffff7e6dab0 in puts ()
    from /lib/aarch64-linux-gnu/libc.so.6

ERROR: AddressSanitizer: heap-buffer-overflow
    #1 0x... in set_name far.c:7
    #1 0x... in main far.c:13

gdb is not broken here. It is telling you the truth: the faulting instruction really is inside libc, reached through puts, and not one frame of your code is on the stack at the moment of death. The truth is just useless, because the bug ran to completion three function calls earlier and left no trace on the stack.

ASan names line 7 because it does not wait for a crash. It puts a poisoned redzone after every allocation and checks each access as it happens, so it reports the write that was wrong rather than the read that happened to notice.

#Three of them never crash

Re-read the bare-run column. The heap overflow exits 0. The use-after-free returns the value you asked for and exits 7. Both are undefined behaviour, both corrupt memory, and both produce a build that passes its tests.

char *b = malloc(16);
strcpy(b, "0123456789ABCDEFGHIJ");   /* 21 bytes into a 16-byte buffer */
free(b);
return 0;                             /* exit 0. Nothing was reported. */

This is why "we do not have a segfault" is not evidence of anything. A segfault is what happens when a memory bug is large enough or unlucky enough to leave the mapping. Smaller ones write over your own heap metadata or the next object and wait.

It is also the argument for running ASan in CI rather than only when something has already broken. The two silent bugs above are exactly the ones you will never go looking for.

#Build with -g. It is free

The same program with and without -g, and the same crash:

#0  0x... in step_a ()
#1  0x... in step_b ()
#2  0x... in step_c ()

#0  0x... in step_a (p=0x0) at lie.c:2
#1  0x... in step_b (p=0x0) at lie.c:3
#2  0x... in step_c (p=0x0) at lie.c:4

-g does not change code generation. Measured on the same source at -O2: the disassembly is byte-identical with and without it, the benchmark runs in 0.04 seconds either way, and the binary grows from 8,808 to 10,808 bytes. You are paying two kilobytes of a section that is never loaded into memory at runtime, in exchange for file names and line numbers.

Ship it, or at minimum keep the unstripped binary somewhere so you can run addr2line -e ./prog -f -C 0x... against an address from a production log. That works with no debugger installed and no core dump.

One trap with addr2line on any modern distro. Ubuntu builds with -fPIE -pie by default, so the binary is a position-independent executable and the address in your log is the load base plus the offset, not the offset. In the run above the faulting program counter was 0xaaaaaaaa0760 while nm put the function at 0x754. Feeding the runtime address straight to addr2line gets you nothing; subtract the load base first, which you can read from /proc/<pid>/maps while the process lives, or from the ASLR-disabled rerun under setarch -R.

One piece of folklore this does not support: it is often said that once you turn on optimization, the line numbers start lying. Measured at -O0, -O1 and -O2 on a four-deep call chain that gets fully inlined, gdb reported all four frames with correct file and line at every level. What degrades is variable values, which become <optimized out> from -O1. The line numbers held.

#What the sanitizer costs

The usual objection to ASan is speed. On a memory-bound loop that touches 160 MB, GCC 11.4.0 at -O2:

Roughly 10x, and that is close to the worst case: the benchmark does nothing but touch memory, which is precisely what ASan instruments. Code that computes rather than loads pays far less. ASan also raises memory use substantially, because every allocation grows a redzone and freed memory is quarantined rather than reused.

Neither number is an argument against using it. They are an argument against shipping it. Run it in tests and on the failing input; ship the plain build.

#When the tool changes the answer

One case in the corpus behaved differently under measurement, and it is worth knowing that this happens:

int main(void) { int *p; *p = 1; return *p; }

Plain build: SIGSEGV, exit 139. Under -fsanitize=address: exit 1, no report, no crash. The uninitialized pointer holds whatever the stack held, ASan changes the stack layout, and the garbage it picked up that time happened to be different. The tool did not find the bug and it also removed the symptom.

The compiler catches this one for free, and it is the reason to read warnings before reaching for a debugger:

$ gcc -Wall -c uninit.c
warning: 'p' is used uninitialized [-Wuninitialized]

A bug that changes or disappears when you observe it is a signal in itself: it usually means uninitialized memory or a data race, neither of which ASan alone is the right tool for. Uninitialized reads are what zero-initialization and -ftrivial-auto-var-init address; races are what -fsanitize=thread is for.

#Your compiler is not upstream GCC

The stack buffer overflow in the table aborts rather than segfaults, and prints something no upstream GCC build would print:

*** stack smashing detected ***: terminated
Aborted (core dumped)

That is a stack canary, and Ubuntu's GCC turns it on by default. Verified: compile a function with a fixed-size local buffer and no flags at all, and __stack_chk_fail appears in the disassembly. Add -fno-stack-protector and it is gone. Ubuntu also defines _FORTIFY_SOURCE to 2 at -O2, which is why some overflows through strcpy and memcpy are caught by glibc rather than reaching the kernel.

The consequence matters when you are comparing notes: the same source can abort on Ubuntu, segfault on a distro that does not patch these defaults, and run to completion on a third. If a crash reproduces for you and not for a colleague, compare gcc -Q --help=common output before assuming the difference is in the code.

#The order to try things in

Cheapest first, because the cheap ones win more often than their reputation suggests.

1. Read the warnings. -Wall -Wextra -g. The uninitialized pointer above is caught at compile time by a flag most builds already have on.
2. Rebuild with -fsanitize=address,undefined -g -O1 and run the same input. Seven for seven above. This is the step people skip and it is the step that works.
3. Only then reach for gdb. It is the right tool for state inspection: what was in this struct, which branch was taken, what does this pointer actually contain. It is the wrong tool for locating a memory bug that already ran.
4. Valgrind if the build cannot be changed. It needs no recompilation, which is its real advantage over ASan; it is slower and, on some platforms, needs debuginfo for the dynamic linker before it will start at all. Not measured here for that reason.
5. For a crash you cannot reproduce, keep the unstripped binary and use addr2line on the address from the log. Enable core dumps with ulimit -c unlimited and check where your system routes them; on Ubuntu /proc/sys/kernel/core_pattern pipes them to apport rather than writing a file next to the binary.

#How Prism handles it

Prism does not debug crashes; it removes one of the causes. Zero-initialization is on by default, so int *p; becomes int *p = 0; and the uninitialized-pointer case above turns from a garbage address into a deterministic null dereference. That is still a crash, but it is the same crash every time, on the line that dereferences, rather than a value that changes with the stack.

bounds-check covers the array cases at the point of access rather than at the point where the corruption is noticed, which is the same idea ASan implements at runtime.

Sanitizer flags pass straight through, so prism -fsanitize=address,undefined -g behaves exactly as cc would. Prism and ASan are not alternatives; use both.

#Further reading

Most of what this page measures has been studied properly by someone, usually a decade before the tool reached your package manager. The papers are readable and freely available.

• Serebryany, Bruening, Potapenko, Vyukov, "AddressSanitizer: A Fast Address Sanity Checker", USENIX ATC 2012. The paper behind -fsanitize=address, from the four Google engineers who wrote it. Explains the shadow-memory trick that makes a check on every load and store affordable, and why the redzone approach catches the overflow rather than the crash.
• Nethercote and Seward, "Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation", PLDI 2007. Won the Most Influential PLDI Paper award ten years later. This is why Valgrind needs no recompilation and why it is an order of magnitude slower than ASan: it is a JIT that rewrites your binary as it runs.
• Szekeres, Payer, Wei, Song, "SoK: Eternal War in Memory", IEEE Security and Privacy 2013. The map of the whole territory: every class of memory-corruption bug, every defence proposed against it in the preceding 30 years, and which defence stops which attack. If you read one thing here, read this one.
• Cowan et al., "StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks", USENIX Security 1998. The origin of the canary that printed "stack smashing detected" above, from 1998. Worth reading to see how old the mitigation on your default build actually is.
• Daroc Alden, "Fil-C: A memory-safe C implementation", LWN, October 2025. Where this is going: a Clang fork that gives every pointer an invisible capability and collects garbage concurrently, making the bugs on this page impossible rather than detectable. Roughly 4x slower, and it has compiled an entire Linux userspace.
• GCC Instrumentation Options: what each -fsanitize= value covers and which combinations conflict.
• Ubuntu toolchain compiler flags, the list of defaults that differ from upstream GCC, with the failure message each one produces.

Measured on GCC 11.4.0, gdb 12.1, Ubuntu 22.04, aarch64, at -O0 -g unless stated. Addresses and timings vary by machine. Valgrind could not be run on this host and is described rather than measured.