Debugging, Inspection and Profiling Tools
Debugging is detective work with instruments.
Learning Objectives
Use debuggers: breakpoints, stepping, watch expressions, locals and globals, memory views, call stacks and disassembly.
Analyse core dumps and crash dumps; use logging and trace points.
Combine compiler warnings, static analysis and sanitizers (ASan, UBSan, MSan, TSan, LSan) and Valgrind Memcheck — knowing their constraints.
Inspect processes and binaries: /proc, pmap/vmmap, size, nm, objdump, readelf.
Profile heap and CPU usage, including hardware counters; use Java, .NET, browser and Python memory tools; debug embedded targets with map files, watch windows and stack watermarks.
Write reproducible bug reports and follow the workflow reproduce → isolate → inspect → fix → regression test.
Plain-Language Explanation
Debugging is detective work with instruments. A debugger lets you stop a program and look inside it. Sanitizers and analysers point at bugs you did not know you had. Profilers tell you where time and memory go. Inspection tools show what the compiler and linker produced and what a running process looks like. The skill is choosing the right instrument for the question — and working methodically from a reliable reproduction to a fix that a test keeps fixed.
Precise Technical Explanation
Debugger fundamentals: breakpoints, stepping, watch expressions Break at function / line break parse / b file.c:42 b parse / b file.c:42 b 42 if i == 99 b 42 -c 'i == 99' right-click → Conditions Run / continue run / continue Step into / over / out step / next / finish s / n / finish F11 / F10 / Shift+F11 Watch memory change watchpoint set variable total print *node, p/x flags Watch / Immediate window
Hardware watchpoints (debug registers; 4 on x86) trap when a location is written — ideal for finding who corrupts a variable.
Inspecting local/global variables info locals, info args, frame N; IDE "Locals"/"Autos" windows. Optimised builds show "optimized out" — rebuild with -O0 -g or -Og when stepping.
Viewing memory and addresses x/16xb ptr (16 bytes in hex), x/4gx $rsp (4 quad-words on the stack), p &var; IDE memory windows. Combined with Module 3's maps you can tell stack, heap and code apart.
Call stack and stack frames backtrace (bt full with locals), frame 2, up/down; thread apply all bt for every thread (Module 5).
disassemble, layout asm, x/10i $pc, stepi; IDE disassembly windows interleave source when debug info is present (Module 11).
Core dumps and crash dumps Enable (ulimit -c unlimited, coredumpctl, Windows WER/procdump), load with the *same* binary and symbols (gdb./app core, WinDbg !analyze -v), then bt, info registers, inspect variables — post-mortem debugging without reproducing the crash.
Logging and trace points Structured logs with timestamps, thread IDs and correlation IDs; levels; rate limiting. Tracepoints/dynamic tracing (perf probe, eBPF/bpftrace, DTrace, ETW, LTTng) observe running systems without stopping them.
-Wall -Wextra -Wpedantic -Wshadow -Wconversion -Wformat=2 -Wnull-dereference (GCC/Clang), /W4 (MSVC); treat warnings as errors in CI.
Path-sensitive analysis without running the program: GCC -fanalyzer (demonstrated below), Clang Static Analyzer / clang-tidy, cppcheck, Coverity, PVS-Studio, CodeQL, Polyspace and MISRA checkers.
AddressSanitizer (ASan) Out-of-bounds, use-after-free, double/invalid free, use-after-return/scope, leaks (with LSan) — Module 9.
UndefinedBehaviorSanitizer (UBSan) Signed overflow, shifts, bounds, misalignment, null, invalid values — Module 9.
MemorySanitizer (MSan; tool/platform constraints) Uninitialised reads; Clang only, Linux only, requires *every* linked library (including the C++ standard library) to be instrumented.
ThreadSanitizer (TSan; tool/platform constraints) Data races (Module 14); 64-bit only; cannot be combined with ASan; sensitive to kernel ASLR settings with older toolchains, as Module 14 showed.
Valgrind Memcheck (where supported) Dynamic binary instrumentation: no recompilation, finds heap errors, leaks and uninitialised use; 20–50× slowdown; Linux (and some other Unixes). Not installed on the build machine, so no output is shown here.
Leak detection at exit, built into ASan on Linux or standalone (-fsanitize=leak) — Module 6.
GDB / LLDB concepts Both support remote debugging (gdbserver, OpenOCD/J-Link GDB servers for MCUs), scripting (Python), pretty-printers for STL types, reverse debugging (rr, GDB record) and non-stop mode.
Visual Studio debugger concepts Data breakpoints, the Diagnostic Tools window (memory snapshots, CPU usage), IntelliTrace/time-travel debugging (WinDbg TTD), Edit and Continue, mixed native/managed debugging.
Linux /proc process memory information /proc/<pid>/maps, smaps, smaps_rollup, status, statm, limits, fd/, task/ (threads), stack (kernel stack, root only).
pmap, vmmap or platform equivalents pmap -x <pid> (Linux), vmmap <pid> (macOS), VMMap (Windows Sysinternals) — summarise mappings with RSS and dirty pages (demonstrated below).
size, nm, objdump, readelf or equivalents Modules 2, 3 and 17. Windows: dumpbin; macOS: otool, nm, size.
heaptrack, Massif, memray, jemalloc/tcmalloc profilers, Visual Studio heap snapshots, Instruments Allocations — where, how much and how often memory is allocated, and what is still live.
CPU profilers and hardware counters perf record/report, perf stat (counters), Intel VTune, AMD uProf, Instruments, Visual Studio profiler, Windows Performance Analyzer; flame graphs. Counters may be unavailable in VMs and containers.
Java heap dump and GC logs -Xlog:gc*, jcmd <pid> GC.heap_dump file.hprof, Eclipse MAT dominator tree, JDK Flight Recorder + Mission Control..NET diagnostics and memory profiler concepts
dotnet-counters monitor (GC heap size, gen counts, allocation rate), dotnet-gcdump collect, dotnet-trace, PerfView, Visual Studio memory usage tool.
Browser heap snapshots DevTools Memory: heap snapshots (compare two to find growth), allocation instrumentation on timeline, detached DOM nodes; performance.memory in Chromium.
Python tracemalloc tracemalloc.start(), snapshots and compare_to to find growing allocation sites (Module 15).
Embedded debugger, watch window and map file Debug probes with SWD/JTAG, live watch windows, peripheral register views from SVD files, map files and fault registers (Module 16).
Stack watermark and RTOS trace Stack painting and uxTaskGetStackHighWaterMark; RTOS trace recorders (SystemView, Tracealyzer) show task switches, ISR timing and blocking (Module 16).
Reproducible bug reports Include: exact version/commit and build flags, platform and toolchain, minimal input or steps, expected versus actual result, full error output (sanitizer report, stack trace), frequency, and what you already ruled out.
Debugging workflow: reproduce → isolate → inspect → fix → regression test
Reproduce reliably (fix the input, seed, timing; script it).
Isolate — shrink the input, bisect commits (git bisect), disable components; sanitizers often isolate instantly.
Inspect — debugger, logs, dumps: find the root cause, not the symptom.
Fix — the cause; review similar code.
Regression test — a test that failed before and passes after (demonstrated below).
Why It Matters in Real Systems
Engineers spend more time understanding failures than writing new code. Knowing which tool answers which question turns days of guessing into minutes of measurement, and regression tests stop bugs from returning.
Key Terms
Breakpoint / watchpoint Stop at an instruction / when memory changes
Saved memory and registers of a crashed process Finding bugs by analysing code without running it
Compiler instrumentation checking behavior at run time Tool attributing allocations to call sites
CPU register counting events (cycles, misses) Visualisation of sampled stack traces
Test that guards against a fixed bug returning Binary search over commits or inputs
Memory Layout and Execution Flow
flowchart LR
R[Reproduce] --> I[Isolate]
I --> N[Inspect]
N --> F[Fix root cause]
F --> T[Regression test]
T --> CI[Run in CI with sanitizers]Syntax
Sanitizer availability depends on compiler, operating system, architecture and build configuration.
# GCC/Clang-style builds (tool support varies)
cc -Wall -Wextra -Wpedantic -g -O0 program.c -o program
cc -fsanitize=address,undefined -g program.c -o program_sanitized
# Linux / ELF tool examples
size program
nm -C program
readelf -S program
objdump -d programComplete Runnable Example
The full workflow on a small, realistic bug: a function that formats sensor IDs into a fixed buffer.
1. Reproduce — a test that fails:
2–3. Isolate and inspect — the plain build *looks* fine; AddressSanitizer pinpoints the overflow:
ASan says: a 7-byte write (memcpy of "123456" + terminator, from format_id line 11, called from main line 20) reached offset 72 of main's frame, where buf occupies offsets [64, 72) — one byte past its end. The plain build silently wrote that byte.
4–5. Fix and regression test:
/* m18_bug.c — format "S-<id>" into out; out_size is the buffer size */
#include <stdio.h>
#include <string.h>
int format_id(char *out, size_t out_size, unsigned id)
{
char digits[16];
int n = sprintf(digits, "%u", id);
if ((size_t)n + 2 > out_size) return -1; /* bug: forgets the terminator */
memcpy(out, "S-", 2);
memcpy(out + 2, digits, (size_t)n + 1);
return n + 2;
}
int main(void)
{
char buf[8];
unsigned tests[] = {7, 12345, 123456};
for (int i = 0; i < 3; ++i) {
int r = format_id(buf, sizeof buf, tests[i]);
printf("id %u -> %d %s\n", tests[i], r, r < 0 ? "(rejected)" : buf);
}
return 0;
}gcc -std=c17 -g -O0 m18_bug.c -o bug_plain
echo "--- plain build: no visible problem ---"; ./bug_plain
gcc -std=c17 -g -O0 -fsanitize=address m18_bug.c -o bug_asan
echo "--- AddressSanitizer build ---"
./bug_asan 2>&1 | grep -E '^id|ERROR|WRITE of|#[0-2] |is located|overflows this variable|SUMMARY' | sed -E 's/0x[0-9a-f]{8,}/0x…/g' || true--- plain build: no visible problem ---
id 7 -> 3 S-7
id 12345 -> 7 S-12345
id 123456 -> 8 S-123456
--- AddressSanitizer build ---
==103==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x… at pc 0x… bp 0x… sp 0x…
WRITE of size 7 at 0x… thread T0
#0 0x… in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
#1 0x… in format_id m18_bug.c:11
#2 0x… in main m18_bug.c:20
Address 0x… is located in stack of thread T0 at offset 72 in frame
#0 0x… in main m18_bug.c:16
[64, 72) 'buf' (line 17) <== Memory access at offset 72 overflows this variable
SUMMARY: AddressSanitizer: stack-buffer-overflow ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827 in __interceptor_memcpy#include <assert.h>
#include <stdio.h>
#include <string.h>
int format_id(char *out, size_t out_size, unsigned id)
{
char digits[16];
int n = snprintf(digits, sizeof digits, "%u", id);
if (n < 0 || (size_t)n + 3 > out_size) return -1; /* "S-" + digits + '\0' */
memcpy(out, "S-", 2);
memcpy(out + 2, digits, (size_t)n + 1);
return n + 2;
}
int main(void)
{
char buf[8];
assert(format_id(buf, sizeof buf, 7) == 3 && strcmp(buf, "S-7") == 0);
assert(format_id(buf, sizeof buf, 12345) == 7 && strcmp(buf, "S-12345") == 0);
assert(format_id(buf, sizeof buf, 123456) == -1); /* the case that used to overflow */
assert(format_id(buf, 3, 7) == -1); /* exact boundary: "S-7" needs 4 bytes */
assert(format_id(buf, 4, 7) == 3);
puts("all regression tests passed (also clean under -fsanitize=address)");
return 0;
}all regression tests passed (also clean under -fsanitize=address)Line-by-Line Explanation
The original check n + 2 > out_size allows n + 2 == out_size, leaving no room for the '\0' copied by the second memcpy.
The plain build overwrote the byte after buf — an unused stack slot this time, so nothing visible happened. Another compiler or flag would corrupt something else.
ASan's report gives the access size (7), the object (buf, 8 bytes, in main's frame) and both stack frames.
The fix counts the terminator (n + 3) and switches to snprintf; the tests pin down the boundary cases, including the exact one that failed.
Expected Output
The outputs are shown under each step above.
Step-by-Step Execution Trace
For id = 123456: n = 6; old check 8 > 8 is false, so it proceeds; memcpy(out + 2, digits, 7) writes bytes 2–8 of an 8-byte buffer — byte 8 is out of bounds. New check 6 + 3 > 8 is true → returns −1.
Interactive Visualization
The stack frame visualizer shows main's frame with buf and the callee frame; the byte viewer helps decode memory dumps.
Common Mistakes and Failure Modes
Debugging optimised builds without knowing variables may be optimized out.
Changing several things at once, so you never learn what fixed it.
"Fixing" the symptom (adding a check where it crashed) instead of the cause.
No regression test, so the bug returns.
Ignoring warnings and analyser reports as "false positives" without proof.
Losing core dumps because symbols for the exact build were not kept.
Static analysis finds bugs on paths you never executed — GCC 11's -fanalyzer:
The analyser followed the fail_late != 0 path, found the second free, and labelled it with its CWE number.
#include <stdlib.h>
#include <string.h>
int store(const char *text, int fail_late)
{
char *copy = malloc(strlen(text) + 1);
if (copy == NULL) return -1;
strcpy(copy, text);
if (fail_late) {
free(copy);
/* bug: falls through to the free below on this path */
}
free(copy);
return 0;
}gcc -std=c17 -fanalyzer -c m18_analyzer_src.c -o /dev/null 2>&1 | grep -E 'warning:' | sed 's/^m18_analyzer_src.c://'13:5: warning: double-‘free’ of ‘copy’ [CWE-415] [-Wanalyzer-double-free]Corrected / Safe Pattern
A minimal trace macro that records where an event happened, plus process inspection with pmap:
Output (one real run):
pmap -x lists each mapping's size, resident set and dirty pages. The 4 MiB calloc block appears as an anonymous mapping of 4100 KiB virtual size with almost nothing resident — calloc got fresh zero pages from mmap and did not need to touch them (demand paging, Module 12).
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define TRACE(fmt, ...) fprintf(stdout, "[%s:%d %s] " fmt "\n", __FILE__, __LINE__, __func__, __VA_ARGS__)
static int *load(size_t n)
{
int *p = calloc(n, sizeof *p);
TRACE("allocated %zu ints at %s", n, p ? "a heap block" : "NULL");
return p;
}
int main(void)
{
int *data = load(1 << 20); /* 4 MiB: its own mmap region */
if (data == NULL) return 1;
TRACE("pid %d ready for inspection", (int)getpid());
fflush(stdout);
char cmd[64];
snprintf(cmd, sizeof cmd, "pmap -x %d | awk 'NR==2 || /total/ || $2 >= 4096'", (int)getpid());
if (system(cmd) != 0) return 1; /* inspect ourselves while running */
free(data);
return 0;
}[m18_trace.c:10 load] allocated 1048576 ints at a heap block
[m18_trace.c:18 main] pid 29 ready for inspection
Address Kbytes RSS Dirty Mode Mapping
00007b930d3ff000 4100 4 4 rw--- [ anon ]
total kB 6880 1624 108Debugging and Diagnostic Techniques
First tool to reach for Where does it crash?
debugger / core dump backtrace; ASan if memory-related Who corrupts this variable?
hardware watchpoint; ASan LeakSanitizer, heaptrack, heap snapshots Is there a race?
ThreadSanitizer, then lock-order review Why is it slow?
sampling profiler, then hardware counters What did the compiler generate?
-S, objdump -d, Compiler Explorer What is mapped / how much RAM?
/proc/<pid>/smaps_rollup, pmap -x Which libraries are loaded?
ldd, LD_DEBUG=libs, /proc/<pid>/maps What does the firmware use?
map file, size, stack watermarks
Performance Implications
Sanitizers and Valgrind slow programs 2–50×; profilers 1–10%; debug builds several times. Use instrumented builds in CI and testing, lightweight sampling and counters in production. Logging itself costs — keep hot-path logging off or sampled.
Security and Safety Implications
Crash dumps and logs can contain secrets — restrict access and scrub them. Debug ports and verbose debug builds must not ship. Sanitizer runtimes are not hardened for production exposure. Fuzzing with sanitizers is one of the most effective security practices for native code.
Platform and Language Caveats
GDB/LLDB, /proc, perf and the sanitizers' fullest support are on Linux; Windows has its own debugger, ASan in MSVC, ETW-based profilers; macOS has LLDB, Instruments and most sanitizers except LSan.
Valgrind does not support every new CPU instruction set extension or macOS version.
Managed runtimes need their own tools (Module 15).
Real-World Application
Large projects run every commit through ASan/UBSan/TSan builds, fuzzers (OSS-Fuzz, ClusterFuzz), static analysers and regression suites; production fleets collect core dumps and sampled profiles (continuous profiling); embedded teams combine trace recorders with field crash logs saved in .noinit memory.
Best Practices Checklist
[ ] Reproduce first; script the reproduction.
[ ] Keep warnings at zero; run a static analyser.
[ ] Run sanitizer builds in CI; fuzz parsers.
[ ] Keep debug symbols for every released build.
[ ] Enable core dumps / crash reporting where appropriate.
[ ] Profile before optimising; measure after.
[ ] Add a regression test with every bug fix.
[ ] Write bug reports others can reproduce.
Exercises
E1 · Beginner. Which tool would you use first for: (a) a crash in free, (b) a slow request, (c) memory that grows overnight?
Answer: (a) AddressSanitizer (the crash site is usually not the bug site); (b) a sampling CPU profiler; (c) a heap profiler or comparison of two heap snapshots (LeakSanitizer if it is a true leak).
E2 · Intermediate. A variable limit sometimes changes to a garbage value. How do you find the culprit with a debugger?
Answer: Run under the debugger, set a hardware watchpoint on limit (watch limit) once it is initialised, continue; the debugger stops at the exact instruction that writes it, with the culprit's call stack — then run an ASan build to check for an overflow from a neighbouring object.
E3 · Advanced challenge. Write the bug report for the format_id defect using the reproducible-bug-report checklist.
Answer: Title: "format_id writes one byte past the output buffer when the result exactly fills it". Version/commit; build gcc 11.2 -O0 -g (plain) and -fsanitize=address; platform Ubuntu 22.04 x86-64. Steps: call format_id(buf, 8, 123456). Expected: −1 (does not fit) or a truncated, terminated string. Actual: returns 8 and writes 9 bytes; ASan reports stack-buffer-overflow, 7-byte write at offset 2 of buf. Frequency: always for 6-digit IDs with 8-byte buffers. Root cause: size check omits the terminator. Attach the ASan report.
Quiz
? A plain build of a program with a one-byte overflow runs without any visible error. What can you conclude?
- The program is correct. :: Undefined behavior can look harmless in one build.
+ Nothing: the overflow may silently corrupt memory; run an ASan build. :: Silent corruption is common.
- The compiler fixed the overflow. :: Compilers do not repair overflows.
- The stack grows upward. :: Unrelated.
? What did GCC's -fanalyzer report for store()?
+ A double free on the fail_late path :: It followed the path where free(copy) runs twice.
- A memory leak :: Every path frees the copy (one path twice).
- A buffer overflow in strcpy :: The allocation is strlen + 1 bytes, which is correct.
- Nothing, because the function was never called :: Static analysis does not need the code to run.
? Which sanitizers can be combined in one build? (select all that apply)
+ AddressSanitizer + UndefinedBehaviorSanitizer :: A common combination.
- AddressSanitizer + ThreadSanitizer :: They use incompatible shadow-memory layouts.
+ ThreadSanitizer + UndefinedBehaviorSanitizer :: Supported.
- MemorySanitizer + AddressSanitizer :: Incompatible.
? What makes a bug fix complete?
- The crash no longer ha
…Summary
Choose tools by question: debuggers and dumps for state, sanitizers and analysers for hidden bugs, profilers for time and memory, inspection tools for binaries and processes. Work from a reliable reproduction through isolation and inspection to a root-cause fix protected by a regression test, and write bug reports others can reproduce.
References and Further Reading
Norman Matloff & Peter Jay Salzman, *The Art of Debugging with GDB, DDD, and Eclipse*.
Brendan Gregg, *Systems Performance*, 2nd ed.; *BPF Performance Tools*.
Google Sanitizers wiki; GCC -fanalyzer documentation.
Andreas Zeller, *Why Programs Fail: A Guide to Systematic Debugging*.