Learning Through Composition

Abstract

NYCBUG

Designing versatile utilities for Unix-like systems requires attention to specific concerns and involves specific disciplines.

This talk aims to highlight the key concerns in play during the development of entrproject.org that are applicable for anyone who endeavors to develop tooling that establishes more effective paradigms for working on *BSD.

Preamble

From one point of view, the process of creating utilities for BSD or Linux is not very different than writing any other kind of program. You learn the APIs that are available, follow conventions to achieve interoperability, establish a strategy for testing, and after many hours a new thing is born.

From another point of view the goal of building Unix tooling is somewhat specialized because the intent is to extend our capabilities that we can be more effective in our professional or personal endeavors. In other words, Unix tools are an investment.

Insofar as a utility or framework continues to perform under a range of demands it is a product that is a is worth your time. To this end I will first attempt to illustrate the cost of free software that does not function well under a range of use cases.

Second I will identify the key disciplines of design work, and contrast those with engineering practices.

Third, I will share some of the lessons I've learned while developing the entr utility and entrproject.org.

Finally, I will put forth a hypothesis about some of the challenges particular to our day and see if some inferences can be made about the purpose of building modern-day Unix tooling.

• How many of you maintain a personal or business web site that serves up static HTML?
• How many of you are using an open-source framework to generate these pages? Why or why not?

staticsitegenerators.net lists 388 projects. I tried several

No doubt most of these projects work to some degree, but it is not uncommon to burn four or five hours before you encounter some hard limitation caused by one or more assumptions that the programmer made about the content of the site.

Two examples:

1. Jekyll seemed to require files to be named by date
2. Nanoc does not allow two files to have the same base name

These kinds of resets amount to waste. You spent time and energy for no reason. These are frameworks, not tools, but they do illustrate the human cost to design failures.

To me one of the remarkable aspects of good Unix tooling and, I think the BSDs in general is that there is no inherent distinction between the operating environment that users, developers and administrators use.

How might we solve this task using existing Unix tools?

What kind of mechanism might a new tool use?

There are a large number of projects on GitHub that are useful to a particular subset of people (such as those writing a web app with Ruby), but they don't solve one problem in the general way that Unix tooling traditionally has--probably because solving a more general problem requires a more carefully constructed architecture.

My first solution:

#!/bin/sh
vim -c 'normal /__version__ =^Mf.^A:wq^M' setup.py

Software design and software engineering do not depend on super-human intelligence. Both of these are disciplines and are operating insofar as they make use of feedback to shape the emerging architecture.

Disciplines of engineering:

• Automated testing
• Continuous integration
• Limiting work-in-progress
• Scheduling disruptive features

Disciplines of design:

• Enumerating and assessing plausible approaches to a solution
• Validated learning through a minimal prototype
• Restructuring or tuning based on the experience of early adopters
• Test-driving a system in a wide range of weather conditions

We sometimes assume that systems are composed of arbitrary abstractions, so you may be surprised to hear the older uses of the word invent had as much to do with discovery as it did with making. Software designed for humans is software that tunes into the best representations.

The notion of a design that is fundamental is obscure in most systems, but evident in the BSDs. A device name (such as axe0) not only gives you an immediate view of the hardware, it associates documentation (man axe) with the hardware and the driver.

Another example is line-oriented text-based configuration files. These can be put under version control, copied, documented and compared using diff(1). Very powerful. (Be careful: XML and JSON are not ordered so they are configuration formats in the traditional sense.)

I do not want to go so far as to assert that software design always require code. It does require an implementation.

In Questioning Monotheism the author(s) posed this question:

What if coding applications managed semantic vertical alignment automatically?

Then after a figure continues with:

In virtually every other form of typography, the responsibility of alignment is given to the typesetting application, not the font.

What's the difficulty here? They are speaking of a world that does not exist. In other words, an isolated idea is hypothetical.

By implementation I am not implying that you need to ship a complete solution; only that there must be a valid proof-of-concept. In Texts Rasterization Exposures, Maxim Shemanarev provided a interesting critique of the methods used in 2000s for rendering text on screen. I would like to draw your attention to just one figure he posted:

Look at it carefully, do you see something strange? Each line has a 1/10th pixel shift, so that, in the run of 30 lines it gradually (gradually!) accumulates 3 extra pixels.

He does not provide a patch for X11, but he did prototype his method and demonstrates it using the pixels on my screen.

Now I'm going to move on and talk about some of the lessons I gained during the evolution of entrproject.org.

How many of you have dialed a colleague and asked him or her to join a VNC session while trying to troubleshoot a service disruption?

It turns out that even without a threat to data integrity or a major service disruption, sharing a keyboard and screen can be an effective way to work. Taking turns at the keyboard while developing a new software feature is a habit is called "pair programming".

In 2011 a colleague and I started designing interfaces by implementing the important parts together. One of the affects this: I became aware of how I was interacting with the command line. Instead of jumping back and forth furiously typing commands I started to learn to step through a cycle of testing more systematically. Having a copilot and being a copilot started to break old habits and work on my imagination.

Building things with other people set my background mind to work.

entr is not a plug-in to an IDE or an editor, it is not a build system, it is not a service that you must configure; it is a simple utility that allows you to react programmatically when files are written or updated.

It is also the first of its kind because it works.

.Dd December 13, 2014
.Dt MKSOCKET 1
.Os
.Sh NAME
.Nm mksocket
.Nd Fetch an unused TCP port
.Sh SYNOPSIS
.Nm
.Op Fl 6
.Sh DESCRIPTION
The
.Nm
utility fetches an unused TCP port and prints it to
.Dv STDOUT
.Sh SEE ALSO
.Xr mktep 1
,
.Xr bind 2
,
.Xr socket 2
.Sh BUGS
This operation is not race-free since the socket is recycled by the
operating system when the
.Nm
utility returns.

$ ls *.js | entr -r serve.sh

Under the hood modern file systems have transactional semitics, but they expose no additional functionality to applications. Even on a local ZFS or HammerFS volume applications have to provide their own guarantees for atomicity and retention.

During the development of entr I encountered many interesting cases where the logic for handling file system events was internally consistent, but cases that left the user confused. Be very cautious when replying to an e-mail or a ticket with a mechanical explanation as if this was a substitute for supplying a smooth solution.

When you start a new project most of the challenges are unknown; as this journey unfolds you will continue to learn new things about the problem space. If you are receptive to feedback and willing for your viewpoint to be changed by what you see surprising new solutions will show up.

How many of you have a camera with an optical viewfinder and a manual focus mode?

Most SLR cameras with auto-focus have an "AF/MF" switch on the lens. This allows you to enable auto-focus or to focus manually. From a mechanical point of view this makes a lot of sense. When I am taking a photograph this choice an awkward implementation detail.

The main features I care about when taking a picture are these:

1. Auto-focus is for getting an fast, accurate fix on a subject
2. The focus ring for making creative adjustments

What prevents me from simply adjusting the focus at any time?

What kind of mechanism would allow me to do this?

Every Cannon SLR since 1987 has supported "full-time manual focus", and back-button focus since 1989.

Every release of the entr utility solved more edge cases that prevent users from being productive. entr-3.1 continues to advance this agenda by setting a process group in restart mode only so that users don't have to worry about doing tedious signal handling in programs that have active subprocesses.

The purpose of examples in a README or a man page is not to demonstrate the fringing use cases for a tool. The purpose is to provide quick orientation. Compare man chmod on BSD and Linux: the GNU manuals are useless, but the examples in BSD man pages are focused and clear.

A product can be considered to have excellent design if people other than its author can use it effectively. This is why almost any feedback has some value if you are willing to evaluate it.

Ruby Gems and similar systems provide a convenient way for developers to string in a long set of dependencies, but they provide a very unpredictable experience for the user, and are not even good Unix citizens. gem and pip do not support man page installation at all. npm does but it is not a common practice.

Package maintainers sometimes try to make up for deficiencies. Pygmentize for example does not ship with a man page, but the Debian project added a man page as a courtesy of the users.

We have already postulated that insofar as a utility continues to perform under a range of demands it is a product that is a worthy investment. This implies that the utility operates on more than my own workstation. The ideal solution to this is a single binary that does not depend on a specific shell environment.

Static linking is useful.

missing/compat.h
missing/fmemopen.c
missing/kqueue_inotify.c
missing/strlcpy.c
missing/sys/event.h

Sometimes an attempt at supporting multiple platforms is made by sprinkling the source with compile-time or runtime conditional checks. If not used carefully this results in software that does not have a clear architecture is ultimately less portable.

By eschewing portability libraries you have the flexibility to solve cases where the features of one system do not have 1:1 feature parity with another. This is the case with kqueue(2) and inotify(7) for example.

Embedding your own translation layer also allows you to build in workarounds for bugs in other distributes. Also, bugs or missing features on the platforms you're targeting is definitely your concern.

Most importantly, software that operates smoothly across multiple platforms allows users to develop and maintain new workflows. What do I mean by "workflows"? I will demonstrate:

Demo: ~/www/localharvest.us/start

If tmux only ran on OpenBSD I would not bother to learn it well, but since it has a portable version I use it everywhere, and in ways that are tightly coupled with the project I'm working on.

In a project of any significant complexity you will encounter buggy operating environments. Don't leave them that way.

Out of this list only the bug in NetBSD was fixed.

The Linux kernel was not fixed, but I was given a idea of how to hack around this bug.

OpenBSD doesn't have a bug tracker, so I can't tell if there was any conversation about this.

Mac and FreeBSD have the same problem, neither has commented on a possible workaround.

I am the maintainer for entrproject.org, so I drive bugs upstream.

Maintainer is a term invented for the open-source software movement that is essentially a synonym of for an upholder. As such an upholder is not only the primary support contact, but the one who roots out problems up and down the stack.

An end user probably doesn't know the best way to diagnose a failure. If you built a system you know best how to articulate a failure condition to other projects.

entr is a general-purpose utility, but there is one task that seems to warrant a special-purpose extension: refreshing a browser window.

There are many incomplete answers on the web, and none of them operate correctly in the environments I've tested. Requirements:

• Same workflow on Mac and systems running X11
• Should not steal window focus
• Must not activate a windows (causes tiling window managers to rearrange the main area)
• If possible reload should override the browser cache

As with any endeavor, there is a conversation to be started upstream

Chrome doesn't 'listen' any more to xdotool actions

http://code.google.com/p/chromium/issues/detail?id=393145

Those of us in this room probably keep a curious, but weary eye on trends in computing [in part] because shifting technologies frequently enjoin us to adapt if we do not wish to be displaced.

Any viable strategy for weathering change insists on a stable set of practices. At their best, the BSDs provide a uniquely coherent general-purpose computing platform that maintains stability not by enforcing backward-compatibility, but by searching for design decisions that will perform well over the long haul.

Hence the intent or purpose of assiduous design work [as applied to the making of command-line utilities] is to expand the short and long-term capabilities of users, system administrators and developers so that they--equipped with a stable set of core skills--have the opportunity to become craftsman in their own right.