Batch Processing with Unix Tools

Simple Log Analysis

Suppose you have a web server that appends a line to a log file every time it serves a request, e.g., using the nginx defaults (line breaks added for readability):

216.58.210.78 - - [27/Feb/2015:17:55:11 +0000] "GET /css/main.css HTTP/1.1"
200 3377 "http://foo.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.0.0 Safari/537.36"

To find the 5 most popular pages on your website, you can do:

cat /var/log/nginx/access.log |
  awk '{print $7}' |
  sort             |
  uniq -c          |
  sort -r -n       |
  head -n 5

… where:

• cat reads the log file
• awk '{print $7}' splits each line by whitespace and only outputs the 7th such field, e.g., /css/main.css
• sort makes it such that repetitions of the same URL are adjacent to each other
• uniq filters out repeated adjacent lines, with -c prepending a count for each distinct line
• sort sorts by the number at the start of each line (-n), but in reverse/descending order (-r)
• head outputs the first five lines (-n 5) and discards the rest.

Compare the chain of Unix commands to this custom Ruby program:

counts = Hash.new(0)

File.open('/var/log/nginx/access.log') do |file|
  file.each do |line|
    url = line.split[6]
    counts[url] += 1
  end
end

top5 = counts.map{|url, count| [count, url]}.sort.reverse[0...5]
top5.each{|count, url| puts "#{count} #{url}" }

Besides syntactic differences between the two approaches, also consider the amount of memory to which the job needs random access. For example, sort automatically handles larger-than-memory datasets by spilling to disk and parallelizing sorting across multiple CPU cores. – a sorting implementation that’s better than that provided by some languages.

The Unix Philosophy

Described in 1978 as follows:

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”
2. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out afterward you’ve finished using them.

To have any programs output connect to any program’s input, then all programs must use the same input/output interface. In Unix, that interface is a file descriptor – an ordered sequence of bytes. Multiple things can be represented as such, e.g., an actual file on the filesystem, stdin, stdout, a socket to another process, a socket representing a TCP connection, etc. By convention, most programs treat the sequence of bytes as ASCII text delimited by \n.

The Unix approach works best when a program uses stdin and stdout instead of worrying about particular file paths. The separation of logic from wiring makes it easy to compose small tools into bigger systems. However, there are limits to a stdin/stdout model, e.g., programs that need multiple inputs or outputs; programs that spawn sub-processes, etc.

Unix tools also make it easy to experiment and see what’s going on. Input files are typically treated as immutable, and so running commands multiple times with different command-line options doesn’t damage the input files. You can also end the pipeline at any point, e.g., pipe it into less to see if it has the expected form; output it to a file allowing you to restart later stages without re-running the entire pipeline.

References

1. Designing Data-Intensive Applications: The big ideas behind reliable, scalable, and maintainable systems. Chapter 10: Batch Processing. Kleppmann, Martin. Mar 16, 2017. ISBN: 978-1098119065 .