Demonstrations of biosnoop, the Linux BPF/bpftrace version.


This traces block I/O, and shows the issuing process (at least, the process
that was on-CPU at the time of queue insert) and the latency of the I/O:

# biosnoop.bt
Attaching 4 probes...
TIME(ms)     COMM             PID    LAT(ms)
611          bash             4179        10
611          cksum            4179         0
627          cksum            4179        15
641          cksum            4179        13
644          cksum            4179         3
658          cksum            4179        13
673          cksum            4179        14
686          cksum            4179        13
701          cksum            4179        14
710          cksum            4179         8
717          cksum            4179         6
728          cksum            4179        10
735          cksum            4179         6
751          cksum            4179        10
758          cksum            4179        17
783          cksum            4179        12
796          cksum            4179        25
802          cksum            4179        32
[...]

This output shows the cksum process was issuing block I/O, which were
completing with around 12 milliseconds of latency. Each block I/O event is
printed out, with a completion time as the first column, measured from
program start.


An example of some background flushing:

# biosnoop.bt
Attaching 4 probes...
TIME(ms)     COMM             PID    LAT(ms)
2966         jbd2/nvme0n1-8   615          0
2967         jbd2/nvme0n1-8   615          0
[...]


There is another version of this tool in bcc: https://github.com/iovisor/bcc
The bcc version provides more fields.
