A study in building modern Unix tooling
January 13, 2015
Designing versatile utilities for Unix-like systems requires attention to specific concerns and involves specific disciplines.
This talk aims to highlight the key concerns in play during the development of entrproject.org that are applicable for anyone who endeavors to develop tooling that establishes more effective paradigms for working on *BSD.
From one point of view, the process of creating utilities for BSD or Linux is not very different than writing any other kind of program. You learn the APIs that are available, follow conventions to achieve interoperability, establish a strategy for testing, and after many hours a new thing is born.
From another point of view the goal of building Unix tooling is somewhat specialized because the intent is to extend our capabilities that we can be more effective in our professional or personal endeavors. In other words, Unix tools are an investment.
Insofar as a utility or framework continues to perform under a range of demands it is a product that is a is worth your time. To this end I will first attempt to illustrate the cost of free software that does not function well under a range of use cases.
Second I will identify the key disciplines of design work, and contrast those with engineering practices.
Third, I will share some of the lessons I've learned while developing
entr utility and entrproject.org.
Finally, I will put forth a hypothesis about some of the challenges particular to our day and see if some inferences can be made about the purpose of building modern-day Unix tooling.
Objective: build a simple web site using basic HTML and CSS
How hard could it be?
staticsitegenerators.net lists 388 projects
staticsitegenerators.net lists 388 projects. I tried several
No doubt most of these projects work to some degree, but it is not uncommon to burn four or five hours before you encounter some hard limitation caused by one or more assumptions that the programmer made about the content of the site.
These kinds of resets amount to waste. You spent time and energy for no reason. These are frameworks, not tools, but they do illustrate the human cost to design failures.
Unix-like systems are general purpose computing platforms
Command-line tools do not need to be coupled to a specific use case
Example bumping 1.1.9 to 2.0.0:$ bumpversion --current-version 1.1.9 major setup.pyhttps://github.com/peritus/bumpversion
To me one of the remarkable aspects of good Unix tooling and, I think the BSDs in general is that there is no inherent distinction between the operating environment that users, developers and administrators use.
How might we solve this task using existing Unix tools?
What kind of mechanism might a new tool use?
There are a large number of projects on GitHub that are useful to a particular subset of people (such as those writing a web app with Ruby), but they don't solve one problem in the general way that Unix tooling traditionally has--probably because solving a more general problem requires a more carefully constructed architecture.
My first solution:
#!/bin/sh vim -c 'normal /__version__ =^Mf.^A:wq^M' setup.py
Engineering is a set of practices for controlling technical debt to achieve a given result
Design is the process of mapping the geography of a problem to an implementation
What disciplines might we infer from these definitions?
Software design and software engineering do not depend on super-human intelligence. Both of these are disciplines and are operating insofar as they make use of feedback to shape the emerging architecture.
Disciplines of engineering:
Disciplines of design:
1. To come or light upon; to meet; to find.1913 Webster
2. To discover, as by study or inquiry; to find out; to devise; to contrive or produce for the first time;
We sometimes assume that systems are composed of arbitrary abstractions, so you may be surprised to hear the older uses of the word invent had as much to do with discovery as it did with making. Software designed for humans is software that tunes into the best representations.
The notion of a design that is fundamental is obscure in most systems,
but evident in the BSDs. A device name (such as
axe0) not only gives
you an immediate view of the hardware, it associates documentation (
axe) with the hardware and the driver.
Another example is line-oriented text-based configuration files. These
can be put under version control, copied, documented and compared
diff(1). Very powerful. (Be careful: XML and JSON are not
ordered so they are configuration formats in the traditional sense.)
Ideas are not designs
Questioning Monotheism http://input.fontbureau.com/info/
A proof of concept accounts for the geography of a problem
Texts Rasterization Exposures - An attempt to improve text rasterization algorithms using only publicly available information http://www.antigrain.com/research/font_rasterization/
I do not want to go so far as to assert that software design always require code. It does require an implementation.
In Questioning Monotheism the author(s) posed this question:
What if coding applications managed semantic vertical alignment automatically?
Then after a figure continues with:
In virtually every other form of typography, the responsibility of alignment is given to the typesetting application, not the font.
What's the difficulty here? They are speaking of a world that does not exist. In other words, an isolated idea is hypothetical.
By implementation I am not implying that you need to ship a complete solution; only that there must be a valid proof-of-concept. In Texts Rasterization Exposures, Maxim Shemanarev provided a interesting critique of the methods used in 2000s for rendering text on screen. I would like to draw your attention to just one figure he posted:
Look at it carefully, do you see something strange? Each line has a 1/10th pixel shift, so that, in the run of 30 lines it gradually (gradually!) accumulates 3 extra pixels.
He does not provide a patch for X11, but he did prototype his method and demonstrates it using the pixels on my screen.
In 2011 a colleague and I started designing interfaces by implementing the important parts together
I learned to step through a cycle of testing more systematically
Building things with other people set my background mind to work
Now I'm going to move on and talk about some of the lessons I gained during the evolution of entrproject.org.
How many of you have dialed a colleague and asked him or her to join a VNC session while trying to troubleshoot a service disruption?
It turns out that even without a threat to data integrity or a major service disruption, sharing a keyboard and screen can be an effective way to work. Taking turns at the keyboard while developing a new software feature is a habit is called "pair programming".
In 2011 a colleague and I started designing interfaces by implementing the important parts together. One of the affects this: I became aware of how I was interacting with the command line. Instead of jumping back and forth furiously typing commands I started to learn to step through a cycle of testing more systematically. Having a copilot and being a copilot started to break old habits and work on my imagination.
Building things with other people set my background mind to work.
A utility for running arbitrary commands when files change.
entr(1)was written to make rapid feedback and automated testing natural and completely ordinary.
|1.0||April||2012||All major BSD platforms supported|
|2.1||July||2013||Mac and Linux supported with no external dependencies|
|3.0||December||2014||Total of 328 commits incorporating feedback from the previous 21 releases. All known corner cases resolved.|
entr is not a plug-in to an IDE or an editor, it is not a build
system, it is not a service that you must configure; it is a simple
utility that allows you to react programmatically when files are written
It is also the first of its kind because it works.
.Dd December 13, 2014 .Dt MKSOCKET 1 .Os .Sh NAME .Nm mksocket .Nd Fetch an unused TCP port .Sh SYNOPSIS .Nm .Op Fl 6 .Sh DESCRIPTION The .Nm utility fetches an unused TCP port and prints it to .Dv STDOUT .Sh SEE ALSO .Xr mktep 1 , .Xr bind 2 , .Xr socket 2 .Sh BUGS This operation is not race-free since the socket is recycled by the operating system when the .Nm utility returns.
snapshots and atomic operations by shuffling files
The chief aim of design work is to eliminate surprising behavior.
Shells do not propagate signals...how should this condition be handled?
$ ls *.js | entr -r serve.sh
Under the hood modern file systems have transactional semitics, but they expose no additional functionality to applications. Even on a local ZFS or HammerFS volume applications have to provide their own guarantees for atomicity and retention.
During the development of
entr I encountered many interesting cases
where the logic for handling file system events was internally
consistent, but cases that left the user confused. Be very cautious
when replying to an e-mail or a ticket with a mechanical explanation as
if this was a substitute for supplying a smooth solution.
When you start a new project most of the challenges are unknown; as this journey unfolds you will continue to learn new things about the problem space. If you are receptive to feedback and willing for your viewpoint to be changed by what you see surprising new solutions will show up.
How many of you have a camera with an optical viewfinder and a manual focus mode?
Most SLR cameras with auto-focus have an "AF/MF" switch on the lens. This allows you to enable auto-focus or to focus manually. From a mechanical point of view this makes a lot of sense. When I am taking a photograph this choice an awkward implementation detail.
The main features I care about when taking a picture are these:
What prevents me from simply adjusting the focus at any time?
What kind of mechanism would allow me to do this?
Every Cannon SLR since 1987 has supported "full-time manual focus", and back-button focus since 1989.
Every release of the
entr utility solved more edge cases that prevent
users from being productive.
entr-3.1 continues to advance this agenda
by setting a process group in restart mode only so that users don't have
to worry about doing tedious signal handling in programs that have
Every seemingly naive question provides valuable data
Tune documentation and base canonical examples on the tasks that seem to puzzle users
Rebuild a project if source files change, limiting output to the first 20 lineshttps://bitbucket.org/eradman/entr/src/default/README.md
$ find src/ | entr sh -c 'make | head -n 20'
The purpose of examples in a README or a man page is not to demonstrate
the fringing use cases for a tool. The purpose is to provide quick
man chmod on BSD and Linux: the GNU manuals are
useless, but the examples in BSD man pages are focused and clear.
A product can be considered to have excellent design if people other than its author can use it effectively. This is why almost any feedback has some value if you are willing to evaluate it.
Distribute a man page (semantic markup, not markdown-to-man)
Distribute a binary or a script that only requires the interpreter
Ruby Gems and similar systems provide a convenient way for developers to
string in a long set of dependencies, but they provide a very
unpredictable experience for the user, and are not even good Unix
pip do not support man page installation at all.
npm does but it is not a common practice.
Package maintainers sometimes try to make up for deficiencies. Pygmentize for example does not ship with a man page, but the Debian project added a man page as a courtesy of the users.
We have already postulated that insofar as a utility continues to perform under a range of demands it is a product that is a worthy investment. This implies that the utility operates on more than my own workstation. The ideal solution to this is a single binary that does not depend on a specific shell environment.
Static linking is useful.
Emulate behavior on platforms that lack decent interfaces
is structured around
and BSD's libc
missing/compat.h missing/fmemopen.c missing/kqueue_inotify.c missing/strlcpy.c missing/sys/event.h
Portable software allows users to maintain workflows
Sometimes an attempt at supporting multiple platforms is made by sprinkling the source with compile-time or runtime conditional checks. If not used carefully this results in software that does not have a clear architecture is ultimately less portable.
By eschewing portability libraries you have the flexibility to solve cases where the features of one system do not have 1:1 feature parity with another. This is the case with kqueue(2) and inotify(7) for example.
Embedding your own translation layer also allows you to build in workarounds for bugs in other distributes. Also, bugs or missing features on the platforms you're targeting is definitely your concern.
Most importantly, software that operates smoothly across multiple platforms allows users to develop and maintain new workflows. What do I mean by "workflows"? I will demonstrate:
tmux only ran on OpenBSD I would not bother to learn it well, but
since it has a portable version I use it everywhere, and in ways that
are tightly coupled with the project I'm working on.
Mac Add missing function fmemopen(3) to libc https://bugreport.apple.com/problem/viewproblem?id=14749664
NetBSD mktemp(3) mangles the pathname if not given an absolute path http://gnats.netbsd.org/47757
OpenBSD kqueue fails to register VNODE events on ext2 http://comments.gmane.org/gmane.os.openbsd.bugs/19729
Linux inotify does not send events for a binary that's running https://bugzilla.kernel.org/show_bug.cgi?id=81031
Mac Add support for NOTE_TRUNCATE to kqueue(2) https://bugreport.apple.com/problem/viewproblem?id=18294682
In a project of any significant complexity you will encounter buggy operating environments. Don't leave them that way.
Out of this list only the bug in NetBSD was fixed.
The Linux kernel was not fixed, but I was given a idea of how to hack around this bug.
OpenBSD doesn't have a bug tracker, so I can't tell if there was any conversation about this.
Mac and FreeBSD have the same problem, neither has commented on a possible workaround.
One who, or that which, upholds; a supporter; a defender; a sustainer.1913 Webster
A project maintainer is the primary point of contact because they are the one who ensures the entire stack works
I am the maintainer for
entrproject.org, so I drive bugs upstream.
Maintainer is a term invented for the open-source software movement that is essentially a synonym of for an upholder. As such an upholder is not only the primary support contact, but the one who roots out problems up and down the stack.
An end user probably doesn't know the best way to diagnose a failure. If you built a system you know best how to articulate a failure condition to other projects.
reload-browser(1), a cross-platform utility for refreshing the current tab in one or more browsers
Wrong, wrong, wrong:
http://unix.stackexchange.com/questions/37258 http://stackoverflow.com/questions/5545117 http://stackoverflow.com/questions/5203419 http://stackoverflow.com/questions/4139129 http://stackoverflow.com/questions/12026953 http://apple.stackexchange.com/questions/54595 http://superuser.com/questions/744772 http://stackoverflow.com/questions/2091235 http://stackoverflow.com/questions/17119184 http://stackoverflow.com/questions/20925453
No guarantee that a correct solution is possible
entr is a general-purpose utility, but there is one task that seems to
warrant a special-purpose extension: refreshing a browser window.
There are many incomplete answers on the web, and none of them operate correctly in the environments I've tested. Requirements:
As with any endeavor, there is a conversation to be started upstream
Chrome doesn't 'listen' any more to xdotool actions
Trends in computing frequently enjoin us to pivot or be uprooted
Any viable strategy for weathering change insists on a stable set of practices
The intent of assiduous design work [as applied to the making of command-line utilities] is to expand long-term capabilities
Those of us in this room probably keep a curious, but weary eye on trends in computing [in part] because shifting technologies frequently enjoin us to adapt if we do not wish to be displaced.
Any viable strategy for weathering change insists on a stable set of practices. At their best, the BSDs provide a uniquely coherent general-purpose computing platform that maintains stability not by enforcing backward-compatibility, but by searching for design decisions that will perform well over the long haul.
Hence the intent or purpose of assiduous design work [as applied to the making of command-line utilities] is to expand the short and long-term capabilities of users, system administrators and developers so that they--equipped with a stable set of core skills--have the opportunity to become craftsman in their own right.