Left-Pad and the Fragility of the Dependency Graph: How Eleven Lines of Code Broke Thousands of Builds and What It Revealed About Shared Infrastructure

The System as Its Engineers Understood It

npm (Node Package Manager) is the package registry for the JavaScript ecosystem. Developers publish reusable code packages to npm. Other developers install these packages as dependencies in their projects. npm resolves the dependency tree: if package A depends on package B, and package B depends on package C, installing A automatically installs B and C. This is transitive dependency resolution.

The npm registry hosts hundreds of thousands of packages. The JavaScript ecosystem has a cultural norm of small, focused packages. Rather than implementing a utility function directly in a project, a developer publishes the function as a package and other developers install it. This produces deep dependency trees: a typical JavaScript project might have direct dependencies on a few dozen packages and transitive dependencies on hundreds or thousands.

The left-pad package is an npm package that exports a single function. The function pads the left side of a string with a specified character until the string reaches a desired length. The entire implementation is 11 lines of code:

// left-pad package, version 1.1.3
// This is the actual code, not a reconstruction

module.exports = leftpad;

function leftpad(str, len, ch) {
  str = String(str);
  var i = -1;

  if (!ch && ch !== 0) ch = ' ';

  len = len - str.length;

  while (++i < len) {
    str = ch + str;
  }

  return str;
}

This package has been downloaded millions of times. It is a transitive dependency of thousands of other packages. Major projects, including React (Facebook’s UI framework), Babel (the JavaScript compiler), and many others, depend on packages that depend on packages that depend on left-pad.

The package is maintained by Azer Koçulu, a developer who maintains several dozen small npm packages. Koçulu also maintains a package called kik, which is an unrelated utility. Kik Interactive, the company behind the Kik messaging application, contacts Koçulu and requests that he transfer the kik package name to them, citing trademark concerns. Koçulu declines. Kik Interactive contacts npm Inc., the company that operates the npm registry. npm Inc., after a review, transfers the kik package name to Kik Interactive without Koçulu’s consent.

Koçulu is angered by this decision. He responds by unpublishing all of his packages from the npm registry. All of them. Including left-pad.

The Chain

March 22, 2016, approximately 16:00 UTC. Koçulu unpublishes all of his npm packages, including left-pad. The packages are removed from the registry. Any npm install operation that requires left-pad (directly or transitively) now fails because the package cannot be downloaded.

16:00 to 17:00 UTC. Build failures begin cascading across the JavaScript ecosystem. Continuous integration systems that run npm install as part of their build pipeline fail when left-pad cannot be resolved. The failure is not limited to projects that directly depend on left-pad. Any project whose dependency tree, at any depth, includes left-pad fails to install.

The error message is straightforward:

npm ERR! 404 'left-pad' is not in the npm registry.

This single missing package breaks thousands of builds because the dependency graph fans out. left-pad is depended on by several widely-used packages. Those packages are depended on by other packages. The tree of affected packages includes major infrastructure of the JavaScript ecosystem.

17:00 UTC. The npm team becomes aware of the issue. The scale is immediately apparent: npm’s monitoring shows a dramatic spike in 404 errors across the registry.

Approximately 17:30 UTC. npm Inc. makes an unprecedented decision. They republish left-pad, restoring the package to the registry without the original author’s consent. This is the first time npm has un-unpublished a package. The decision is made because the alternative, leaving the package unpublished, would continue to break builds across the ecosystem for an indefinite period.

Within approximately 2.5 hours of the unpublishing. The package is restored. Builds begin succeeding again. The immediate crisis is over.

The total duration of the outage is approximately 2.5 hours. The impact during those 2.5 hours is difficult to quantify precisely: thousands of CI/CD pipelines failing, developer workflows blocked, deployments halted.

The diagram shows the dependency fan-out pattern. left-pad is at the root. A small number of popular packages directly depend on it. Those packages are dependencies of a larger number of packages. Those are dependencies of an even larger number. The tree expands exponentially. A single package at the root affects every leaf. The diagram makes visible what the dependency resolution algorithm makes invisible: the structural fragility of a deep, narrow dependency tree.