More Verbose

Cause debugging info to be created:

• Look in system logs for debuging info already generated
• Change config to start writing a log file
• Change config to start outputing to stderr
• Run in a different environment where stdout/stderr can be seen
  • Don't run in daemon mode (eg. don't detach from the terminal)
  • Run cron and CGI scripts from the command-line

Ask for more debugging info:

• Add more --verbose flags
• Raise the debugging level

Layer Interfaces

Use external tools to examine data passing over "choke points"

• Network sniffing and serial logging (the most obvious/common)
  • When data isn't human-readable (eg. jpeg, https, etc...), requires a tool to generate human-readable summary info
  • Each layer of the OSI model is potentially sniffable. Start at the top and work your way down.
• Hooking APIs (especially system/kernel APIs, but any standardized/understood interface)
  • strace, truss
  • filemon, regmon, ...

Because choke points are more difficult for programmers to use than in-memory operations, the data will be more succinct and easier to analyze than "brute-force logging" (see below).

Source Code

Fetch the source code, and try to understand it.

Change the code and re-run

• Generate more debugging info to do "printf debugging"
• To experiment/confirm your theories

Brute-Force Logging

Log EVERYTHING, and spend hours/days piecing together the big picture from the little details

• System debuggers
  • Softice and kin
  • Standard debuggers attached to system emulators
    • GDB + UML or
    • certainly there are others...?
• Logic Analyzers

The goal of DRM and other applications which embed passwords in code is to force analysis down to this level.