https://rrampage.github.io/2025/06/13/strace-tips-for-better-debugging/

A Random Walk

  * About
  * Archive
  * Search
  * Tags
  * RSS

strace tips for better debugging

2025-06-13
Table of Contents

  * Useful options and flags
  * Print stack traces
  * Selectively tracing syscalls
  * Tampering with syscalls!

Recently, I have been building software without libc to better
understand Linux syscalls and internals better. So far, I have built
a minimal shell, terminal Snake game, pure ARM64 assembly HTTP server
and threads implementation. I have been using strace extensively
while debugging.

Useful options and flags#

I use a version of the following command:

strace -fintrCDTYyy -o strace.log -v -s128 ./binary

This looks like an alphabet soup of options! Here's what they do and
how they are useful:

  * -f: Follow child processes/threads. This is especially useful
    when dealing with spawning processes or threads as otherwise,
    strace will only trace the parent process.
  * -v: Print unabbreviated versions of environment, stat, termios
    and other structs in syscalls. I found this invaluable in
    conjunction with -s when doing assembly programming to check if
    the structs were being initialized correctly and if certain
    arguments were being sent in little/big endian format
  * -s NUM: Specify the maximum string size to print. Useful for
    large structs
  * -o: Save strace output to a log file. It is always better to do
    this to investigate the output of the original process and strace
    separately without each cluttering the other
  * -yy: Print all available information associated with file
    descriptors. This is great for expanding the file descriptor to
    either its full path in case of a file or TCP address in case of
    sockets
  * -Y: Print command names for PIDs. I found this useful when
    building the shell to check if the correct program is being
    executed
  * -t: Print current timestamp in log
  * -T: Show time spent in syscalls. Useful for some basic profiling
    although strace heavily slows down the process.
  * -r: Print a relative timestamp upon entry to each system call
  * -n: Print syscall number. Great to quickly find out syscall
    numbers on new architectures.
  * -i: Print instruction pointer at the time of syscall. Found this
    useful when debugging assembly code to check rough location of
    errors.
  * -C: Print summary of syscall count, time, errors at the end of
    regular output

Print stack traces#

The -k or --stack-trace prints the stacktrace along with the syscall.
This is useful if your program is compiled with with -g. This post is
a good read on using strace to show backtraces for a Golang program
compiled with GODEBUG.

Selectively tracing syscalls#

By default, strace traces all syscalls. We can be very selective in
tracing calls by using the -e option. It allows us to trace families
of syscalls e.g -e t=%net will trace all network related syscalls
while -e t=%mem will trace memory related syscalls.

We can also trace only syscalls which succeed using -z option or fail
using -Z option. Another useful option is -P which can be used to
only trace syscalls which access a particular path e.g strace -f -P /
usr/bin/ls sh -c ls.

Tampering with syscalls!#

strace is very powerful and can do stuff like inject faults for
particular syscalls e.g strace -e inject=%file:error=ENOENT:when=3+
ls which fails the all file related syscalls after 2 successful
invocations. We can also make syscalls return a particular value
using retval or send signal using signal or delay at beginning or end
of syscall using delay_enter and delay_exit with -e inject.

I found this useful when debugging failure cases in code for the
syscalls. The strace output will have lines like openat(AT_FDCWD, "/
lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or
directory) (INJECTED) which have INJECTED marked in them. This is
useful to differentiate genuine syscall errors from the injected
ones.

  * strace
  * linux
  * debug

Next >>
Bash - Oops I deleted my bin dir challenge (c) 2025 Raunak *

Want to become a better programmer? Join the Recurse Center!