auto-detecting a file's type

see also — en.wikipedia.org/wiki/Content_sniffing

tools available

I really like using file(1) and magic(5) to auto-detect a file's type based on its contents. Its database covers MANY different file types, and it generally seems to get things right. However, it IS still a guess, and sometimes guesses are wrong.

text vs binary

You should be aware that auto-detecting whether a file is text is a guess/heuristic, and different heuristics often disagree with each other about whether a particular file is text.

Some tools that contain text vs binary heuristics:

• Perl's -T and -B operators [algorithm description]
  • find -type f | perl -nle 'print if -T'
• file can guess at the file's MIME type, and MIME types conveniently mention when they're part of the 'text/*' type.
  • find -type f | perl -nle 'print if qx{file -bi $_} =~ m#^text/#'
• grep has a heuristic that allows it to ignore binary files [algorithm]
  • grep -rlI ^

auto-detecting character encoding within text files

see also — en.wikipedia.org/wiki/Charset_detection

Some tools that can do this:

• file
  • find -type f | perl -nle 'print if qx{file -bk $_} =~ /\bASCII text\b/'
• various Perl modules:
  • Encode::Guess and encguess — part of Perl core since v5.8.0
  • Encode::Detect
  • Encode::Detect::Detector

Perl source files

Some tools that can auto-detect if a file contains Perl source:

• file
  • find -type f | perl -nle 'print if qx{file -b $_} =~ /^perl(?! Storable)/i'
  • Note that file often misclassifies extremely short Perl files as just ASCII.
• based on file extensions and whether a shebang is present
  • Perl::Metrics::Simple->is_perl_file() [algorithm]
    • find -type f | perl -MPerl::Metrics::Simple -nle 'print if Perl::Metrics::Simple->is_perl_file($_)'
  • File::Find::Rule::Perl
• based on PPI
  • Syntax::Check
    • (This actually malfunctions for me — it has false positives, it throws errors in places that perl -c does not, across a WIDE variety of files)
    • find -type f | xargs -n 1 syncheck
    • (the above errors out A LOT, so instead try...) find -type f | perl -nle 'system "syncheck", $_'