document updated 9 months ago, on Feb 19, 2023
OS-wide monitoring (in Linux)
Ways to do something like strace, but for every process on a server.
files being changed
Note that it's pretty easy to get the error message "upper limit on inotify watches reached", especially if you try to watch all the files on the system.
processes being created
- based on netlink, which should be available in kernels v2.0 and v2.2+
- Note, however, that netlink is considered an unreliable communication channel, and that events may be dropped in two situations: 1) global memory exhaustion, or 2) receiver queue buffer overrun.
- to check if netlink is enabled in your kernel, confirm that
- forkstat source code
- forkstat has almost no filtering options, so it may be best to pipe its output to
| grep --line-buffered <pattern>
bpftrace ... sys_enter_exec
- based on auditd, which is commonly available on RHEL
- for example, try something like
auditctl -a always,entry -S execve then
tail -f /var/log/audit/audit.log (NOTE — I haven't gotten this working yet)
- (TODO — continue looking through this list of suggestions)
TCP connections being made
- based on eBPF, which should be available in kernels v3.18+
general eBPF content