Learning Through Composition

A study in building modern Unix tooling


January 13, 2015



Designing versatile utilities for Unix-like systems requires attention to specific concerns and involves specific disciplines.

This talk aims to highlight the key concerns in play during the development of entrproject.org that are applicable for anyone who endeavors to develop tooling that establishes more effective paradigms for working on *BSD.


From one point of view, the process of creating utilities for BSD or Linux is not very different than writing any other kind of program. You learn the APIs that are available, follow conventions to achieve interoperability, establish a strategy for testing, and after many hours a new thing is born.

From another point of view the goal of building Unix tooling is somewhat specialized because the intent is to extend our capabilities that we can be more effective in our professional or personal endeavors. In other words, Unix tools are an investment.

Insofar as a utility or framework continues to perform under a range of demands it is a product that is a is worth your time. To this end I will first attempt to illustrate the cost of free software that does not function well under a range of use cases.

Second I will identify the key disciplines of design work, and contrast those with engineering practices.

Third, I will share some of the lessons I've learned while developing the entr utility and entrproject.org.

Finally, I will put forth a hypothesis about some of the challenges particular to our day and see if some inferences can be made about the purpose of building modern-day Unix tooling.

Architectural Land Mines

Objective: build a simple web site using basic HTML and CSS

How hard could it be?

staticsitegenerators.net lists 388 projects

staticsitegenerators.net lists 388 projects. I tried several

No doubt most of these projects work to some degree, but it is not uncommon to burn four or five hours before you encounter some hard limitation caused by one or more assumptions that the programmer made about the content of the site.

Two examples:

  1. Jekyll seemed to require files to be named by date
  2. Nanoc does not allow two files to have the same base name

These kinds of resets amount to waste. You spent time and energy for no reason. These are frameworks, not tools, but they do illustrate the human cost to design failures.

Role Based Islands

Unix-like systems are general purpose computing platforms

Command-line tools do not need to be coupled to a specific use case

Example bumping 1.1.9 to 2.0.0:
bumpversion --current-version 1.1.9 major setup.py

To me one of the remarkable aspects of good Unix tooling and, I think the BSDs in general is that there is no inherent distinction between the operating environment that users, developers and administrators use.

How might we solve this task using existing Unix tools?

What kind of mechanism might a new tool use?

There are a large number of projects on GitHub that are useful to a particular subset of people (such as those writing a web app with Ruby), but they don't solve one problem in the general way that Unix tooling traditionally has--probably because solving a more general problem requires a more carefully constructed architecture.

My first solution:

vim -c 'normal /__version__ =^Mf.^A:wq^M' setup.py

Incorporating Feedback

Engineering is a set of practices for controlling technical debt to achieve a given result

Design is the process of mapping the geography of a problem to an implementation

What disciplines might we infer from these definitions?

Software design and software engineering do not depend on super-human intelligence. Both of these are disciplines and are operating insofar as they make use of feedback to shape the emerging architecture.

Disciplines of engineering:

Disciplines of design:

Of Design & Making


1. To come or light upon; to meet; to find.
2. To discover, as by study or inquiry; to find out; to devise; to contrive or produce for the first time;

1913 Webster

We sometimes assume that systems are composed of arbitrary abstractions, so you may be surprised to hear the older uses of the word invent had as much to do with discovery as it did with making. Software designed for humans is software that tunes into the best representations.

The notion of a design that is fundamental is obscure in most systems, but evident in the BSDs. A device name (such as axe0) not only gives you an immediate view of the hardware, it associates documentation (man axe) with the hardware and the driver.

Another example is line-oriented text-based configuration files. These can be put under version control, copied, documented and compared using diff(1). Very powerful. (Be careful: XML and JSON are not ordered so they are configuration formats in the traditional sense.)

Surveying a Solution Space

Ideas are not designs

Questioning Monotheism http://input.fontbureau.com/info/

A proof of concept accounts for the geography of a problem

Texts Rasterization Exposures - An attempt to improve text rasterization algorithms using only publicly available information http://www.antigrain.com/research/font_rasterization/

I do not want to go so far as to assert that software design always require code. It does require an implementation.

In Questioning Monotheism the author(s) posed this question:

What if coding applications managed semantic vertical alignment automatically?

Then after a figure continues with:

In virtually every other form of typography, the responsibility of alignment is given to the typesetting application, not the font.

What's the difficulty here? They are speaking of a world that does not exist. In other words, an isolated idea is hypothetical.

By implementation I am not implying that you need to ship a complete solution; only that there must be a valid proof-of-concept. In Texts Rasterization Exposures, Maxim Shemanarev provided a interesting critique of the methods used in 2000s for rendering text on screen. I would like to draw your attention to just one figure he posted:

Look at it carefully, do you see something strange? Each line has a 1/10th pixel shift, so that, in the run of 30 lines it gradually (gradually!) accumulates 3 extra pixels.

He does not provide a patch for X11, but he did prototype his method and demonstrates it using the pixels on my screen.

Collaboration and Shared Experience

In 2011 a colleague and I started designing interfaces by implementing the important parts together

I learned to step through a cycle of testing more systematically

Building things with other people set my background mind to work

Now I'm going to move on and talk about some of the lessons I gained during the evolution of entrproject.org.

How many of you have dialed a colleague and asked him or her to join a VNC session while trying to troubleshoot a service disruption?

It turns out that even without a threat to data integrity or a major service disruption, sharing a keyboard and screen can be an effective way to work. Taking turns at the keyboard while developing a new software feature is a habit is called "pair programming".

In 2011 a colleague and I started designing interfaces by implementing the important parts together. One of the affects this: I became aware of how I was interacting with the command line. Instead of jumping back and forth furiously typing commands I started to learn to step through a cycle of testing more systematically. Having a copilot and being a copilot started to break old habits and work on my imagination.

Building things with other people set my background mind to work.


A utility for running arbitrary commands when files change. entr(1) was written to make rapid feedback and automated testing natural and completely ordinary.
1.0 April 2012 All major BSD platforms supported
2.1 July 2013 Mac and Linux supported with no external dependencies
3.0 December 2014 Total of 328 commits incorporating feedback from the previous 21 releases. All known corner cases resolved.


entr is not a plug-in to an IDE or an editor, it is not a build system, it is not a service that you must configure; it is a simple utility that allows you to react programmatically when files are written or updated.

It is also the first of its kind because it works.

.Dd December 13, 2014
.Nm mksocket
.Nd Fetch an unused TCP port
.Op Fl 6
utility fetches an unused TCP port and prints it to
.Xr mktep 1
.Xr bind 2
.Xr socket 2
This operation is not race-free since the socket is recycled by the
operating system when the
utility returns.

A Word About Correctness

Applications mimic snapshots and atomic operations by shuffling files

The chief aim of design work is to eliminate surprising behavior.

Shells do not propagate signals...how should this condition be handled?

ls *.js | entr -r serve.sh

Under the hood modern file systems have transactional semitics, but they expose no additional functionality to applications. Even on a local ZFS or HammerFS volume applications have to provide their own guarantees for atomicity and retention.

During the development of entr I encountered many interesting cases where the logic for handling file system events was internally consistent, but cases that left the user confused. Be very cautious when replying to an e-mail or a ticket with a mechanical explanation as if this was a substitute for supplying a smooth solution.

When you start a new project most of the challenges are unknown; as this journey unfolds you will continue to learn new things about the problem space. If you are receptive to feedback and willing for your viewpoint to be changed by what you see surprising new solutions will show up.

How many of you have a camera with an optical viewfinder and a manual focus mode?

Most SLR cameras with auto-focus have an "AF/MF" switch on the lens. This allows you to enable auto-focus or to focus manually. From a mechanical point of view this makes a lot of sense. When I am taking a photograph this choice an awkward implementation detail.

The main features I care about when taking a picture are these:

  1. Auto-focus is for getting an fast, accurate fix on a subject
  2. The focus ring for making creative adjustments

What prevents me from simply adjusting the focus at any time?

What kind of mechanism would allow me to do this?

Every Cannon SLR since 1987 has supported "full-time manual focus", and back-button focus since 1989.

Every release of the entr utility solved more edge cases that prevent users from being productive. entr-3.1 continues to advance this agenda by setting a process group in restart mode only so that users don't have to worry about doing tedious signal handling in programs that have active subprocesses.

Pitfalls Are Design Bugs

Every seemingly naive question provides valuable data

Tune documentation and base canonical examples on the tasks that seem to puzzle users

Rebuild a project if source files change, limiting output to the first 20 lines

find src/ | entr sh -c 'make | head -n 20'

The purpose of examples in a README or a man page is not to demonstrate the fringing use cases for a tool. The purpose is to provide quick orientation. Compare man chmod on BSD and Linux: the GNU manuals are useless, but the examples in BSD man pages are focused and clear.

A product can be considered to have excellent design if people other than its author can use it effectively. This is why almost any feedback has some value if you are willing to evaluate it.

Choose Tooling that Facilitates Deployment

Distribute a man page (semantic markup, not markdown-to-man)

Distribute a binary or a script that only requires the interpreter

  1. Can be copied to multiple of systems
  2. Operable from an NFS-mounted directory
  3. Easily run with sudo or ssh

Ruby Gems and similar systems provide a convenient way for developers to string in a long set of dependencies, but they provide a very unpredictable experience for the user, and are not even good Unix citizens. gem and pip do not support man page installation at all. npm does but it is not a common practice.

Package maintainers sometimes try to make up for deficiencies. Pygmentize for example does not ship with a man page, but the Debian project added a man page as a courtesy of the users.

We have already postulated that insofar as a utility continues to perform under a range of demands it is a product that is a worthy investment. This implies that the utility operates on more than my own workstation. The ideal solution to this is a single binary that does not depend on a specific shell environment.

Static linking is useful.

Own Challenges to Cross-Platform Behavior

Emulate behavior on platforms that lack decent interfaces

entr(1) is structured around kqueue(2) and BSD's libc


Portable software allows users to maintain workflows

Sometimes an attempt at supporting multiple platforms is made by sprinkling the source with compile-time or runtime conditional checks. If not used carefully this results in software that does not have a clear architecture is ultimately less portable.

By eschewing portability libraries you have the flexibility to solve cases where the features of one system do not have 1:1 feature parity with another. This is the case with kqueue(2) and inotify(7) for example.

Embedding your own translation layer also allows you to build in workarounds for bugs in other distributes. Also, bugs or missing features on the platforms you're targeting is definitely your concern.

Most importantly, software that operates smoothly across multiple platforms allows users to develop and maintain new workflows. What do I mean by "workflows"? I will demonstrate:

Demo: ~/www/localharvest.us/start

If tmux only ran on OpenBSD I would not bother to learn it well, but since it has a portable version I use it everywhere, and in ways that are tightly coupled with the project I'm working on.

Push Problems Upstream

Mac Add missing function fmemopen(3) to libc https://bugreport.apple.com/problem/viewproblem?id=14749664

NetBSD mktemp(3) mangles the pathname if not given an absolute path http://gnats.netbsd.org/47757

OpenBSD kqueue fails to register VNODE events on ext2 http://comments.gmane.org/gmane.os.openbsd.bugs/19729

Linux inotify does not send events for a binary that's running https://bugzilla.kernel.org/show_bug.cgi?id=81031

Mac Add support for NOTE_TRUNCATE to kqueue(2) https://bugreport.apple.com/problem/viewproblem?id=18294682


In a project of any significant complexity you will encounter buggy operating environments. Don't leave them that way.

Out of this list only the bug in NetBSD was fixed.

The Linux kernel was not fixed, but I was given a idea of how to hack around this bug.

OpenBSD doesn't have a bug tracker, so I can't tell if there was any conversation about this.

Mac and FreeBSD have the same problem, neither has commented on a possible workaround.

The Role of a Project Maintainer


One who, or that which, upholds; a supporter; a defender; a sustainer.

1913 Webster

A project maintainer is the primary point of contact because they are the one who ensures the entire stack works

I am the maintainer for entrproject.org, so I drive bugs upstream.

Maintainer is a term invented for the open-source software movement that is essentially a synonym of for an upholder. As such an upholder is not only the primary support contact, but the one who roots out problems up and down the stack.

An end user probably doesn't know the best way to diagnose a failure. If you built a system you know best how to articulate a failure condition to other projects.

What's Next?

reload-browser(1), a cross-platform utility for refreshing the current tab in one or more browsers

Wrong, wrong, wrong:

http://unix.stackexchange.com/questions/37258 http://stackoverflow.com/questions/5545117 http://stackoverflow.com/questions/5203419 http://stackoverflow.com/questions/4139129 http://stackoverflow.com/questions/12026953 http://apple.stackexchange.com/questions/54595 http://superuser.com/questions/744772 http://stackoverflow.com/questions/2091235 http://stackoverflow.com/questions/17119184 http://stackoverflow.com/questions/20925453

No guarantee that a correct solution is possible

entr is a general-purpose utility, but there is one task that seems to warrant a special-purpose extension: refreshing a browser window.

There are many incomplete answers on the web, and none of them operate correctly in the environments I've tested. Requirements:

As with any endeavor, there is a conversation to be started upstream

Chrome doesn't 'listen' any more to xdotool actions



Trends in computing frequently enjoin us to pivot or be uprooted

Any viable strategy for weathering change insists on a stable set of practices

The intent of assiduous design work [as applied to the making of command-line utilities] is to expand long-term capabilities

Those of us in this room probably keep a curious, but weary eye on trends in computing [in part] because shifting technologies frequently enjoin us to adapt if we do not wish to be displaced.

Any viable strategy for weathering change insists on a stable set of practices. At their best, the BSDs provide a uniquely coherent general-purpose computing platform that maintains stability not by enforcing backward-compatibility, but by searching for design decisions that will perform well over the long haul.

Hence the intent or purpose of assiduous design work [as applied to the making of command-line utilities] is to expand the short and long-term capabilities of users, system administrators and developers so that they--equipped with a stable set of core skills--have the opportunity to become craftsman in their own right.