Dynamic Tracing on OpenBSD

btrace(8) is a tool that provices dtrace(1) functionality on OpenBSD. The syntax is analogous to awk(1)

probe / filter / { action }

Examples

All system read operations

# btrace -e 'syscall:read:entry { @[comm] = count() }'
^C
@[chrome]: 14609
@[Xorg]: 2005
@[firefox]: 153
@[tailscaled]: 3
@[pflogd]: 1
@[st]: 1

Read size distribution

# btrace -e 'syscall:read:return { @readsize = hist(retval) }'
^C
@readsize:
[0]              505 |@@                                                  |
[0, 1)          9010 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2)            14 |                                                    |
[8, 16)           26 |                                                    |
[16, 32)         410 |@@                                                  |
[32, 64)         388 |@@                                                  |
[64, 128)        497 |@@                                                  |
[128, 256)         9 |                                                    |
[256, 512)        43 |                                                    |
[512, 1K)        197 |@                                                   |
[1K, 2K)         216 |@                                                   |
[2K, 4K)         220 |@                                                   |
[4K, 8K)          40 |                                                    |
[8K, 16K)       1400 |@@@@@@@@                                            |
[16K, 32K)       277 |@                                                   |

History

7.8

• Add dt(4) trace points to rwlock(9)

7.7

• Support additional interval/profile units (hz, us, ms, s)

7.6

• In btrace(8), cache ELF .symtab, .strtab entries in sorted array to improve lookup cost from O(n) to O(lg n).

7.5

• Added bt(5) and btrace(8) support for binary modulo operator ('%').

7.4

• Implemented tuples in btrace(8), allowing export of per-CPU scheduling data.
• Made it possible to store the kstack or ustack in a map in btrace(8).
• Added btrace(8) support for symbolizing utrace(2) addresses.

7.2

• Allowed btrace(8) to execute the END probe upon receiving a SIGTERM signal.
• Installed useful btrace(8) scripts in /usr/share/btrace.

7.1

• Implemented the probe variable in bt(5).
• Used unveil(2) for the possible btrace(8) script file, dt(4) and ksyms(4) nodes.
• Provided common btrace(8) scripts kprofile.bt (to save kernel stackframes and produce flamegraphs) and runqlat.bt (to measure the latency of the scheduler runqueues).

7.0

• Added btrace(8) display of time spent in userland when analyzing the kernel stack in the flame graph tool and fixed a parsing bug.
• Implemented '<' and '>' operators in btrace(8) filters.

6.9

• Added btrace(8) -n (no action) mode, which parses the program and then exits.
• Added trace points for malloc(9) and free(9), making them traceabe via dt(4) and btrace(8).

6.8

• Added support for '&' and '|' operators in btrace scripts.
• Enabled btrace(8).
• Added btrace(8) -p flag to filter all actions by PID.
• Implemented linear and power-of-two histograms in btrace(5).

Traditional Tools

Setting LD_DEBUG prints the linking operations for ld.so

ktrace(1) trace system calls

ltrace(1) trace shared library function calls