document updated 1 year, 2 months ago, on Feb 19, 2023

OS-wide monitoring (in Linux)

Ways to do something like strace, but for every process on a server.

files being changed

Note that it's pretty easy to get the error message "upper limit on inotify watches reached", especially if you try to watch all the files on the system.

forkstat
- based on netlink, which should be available in kernels v2.0 and v2.2+
  - Note, however, that netlink is considered an unreliable communication channel, and that events may be dropped in two situations: 1) global memory exhaustion, or 2) receiver queue buffer overrun.
  - to check if netlink is enabled in your kernel, confirm that /proc/net/netlink exists
- man forkstat
- forkstat source code
- forkstat has almost no filtering options, so it may be best to pipe its output to | grep --line-buffered <pattern>
bpftrace ... sys_enter_exec
- based on eBPF, which should be available in kernels v3.18+
- bpftrace source code
auditctl
- based on auditd, which is commonly available on RHEL
- man auditctl
- man autrace
- for example, try something like auditctl -a always,entry -S execve then tail -f /var/log/audit/audit.log (NOTE — I haven't gotten this working yet)
(TODO — continue looking through this list of suggestions)

based on eBPF, which should be available in kernels v3.18+
- tcpconnect [man]
- (this list of others contains some networking tools as well)