Knight Capital: A Deployment, a Flag Nobody Deleted, and 45 Minutes to Lose 440 Million Dollars

The System as Its Engineers Understood It

Knight Capital Group is one of the largest market makers on the New York Stock Exchange. A market maker’s function is to provide liquidity: when someone wants to buy a stock and no seller is immediately available, the market maker sells from its own inventory. When someone wants to sell and no buyer is present, the market maker buys. Market makers profit from the spread between their buy and sell prices. The business requires automated trading systems that can process orders in milliseconds.

Knight’s automated trading system is called SMARS (Smart Market Access Routing System). SMARS receives customer orders, breaks them into smaller child orders according to configurable algorithms, and routes them to various exchanges and market centers for execution. SMARS processes millions of orders per day across hundreds of stocks. The system is the core of Knight’s business.

In preparation for August 1, 2012, Knight needs to update SMARS to participate in the NYSE’s new Retail Liquidity Program (RLP). The RLP requires new order routing logic. Knight’s engineers write the new RLP code and prepare to deploy it to the eight SMARS servers that handle NYSE-listed stocks.

The deployment uses a manual process. An engineer copies the new software to each of the eight servers individually. There is no automated deployment system. There is no deployment verification step that confirms all eight servers are running the same version. The deployment checklist exists on paper.

The new RLP code is activated by a feature flag. The flag is named, in the SEC proceedings’ description, with a name that had previously been used for a completely different function: Power Peg. Power Peg was a trading algorithm from 2003 that Knight had discontinued years earlier. The Power Peg code was never removed from the SMARS codebase. It remained in the source code, compiled into every binary, disabled by the flag that once activated it. When the engineers needed a flag name for the new RLP functionality, they reused the Power Peg flag.

The Power Peg code, written in 2003, contains a trading algorithm that is the opposite of what a market maker should do. Under certain conditions, it buys stock at the offer price and sells at the bid price, the reverse of normal market-making. In 2003, this behavior was intentional for the specific narrow use case Power Peg served. In 2012, with the Power Peg flag now being used to activate RLP code, enabling the flag on a server with the old code would activate Power Peg instead of RLP.

Nobody knows this except the code.

The Chain

July 31, 2012, evening. A Knight engineer begins deploying the new SMARS code to the eight production servers. The process is manual: copy the binary, update the configuration. The engineer deploys to seven of the eight servers successfully.

July 31, 2012, late evening. The eighth server does not receive the new code. The SEC proceedings do not detail why. The deployment checklist does not include a post-deployment verification step that compares the running binary version across all servers. The engineer believes the deployment is complete. Seven servers have the new RLP code. One server has the old code with the dormant Power Peg algorithm.

August 1, 2012, 09:30 EDT. The NYSE opens. The RLP goes live. Knight’s configuration system sends the activation signal: the Power Peg flag is enabled across all eight SMARS servers. On seven servers, this flag activates the new RLP code. On the eighth server, this flag activates the old Power Peg code from 2003.

09:30:00 to 09:31:00. The eighth server begins executing Power Peg logic. It receives parent orders and, instead of routing them through the new RLP algorithm, processes them through the Power Peg algorithm. Power Peg sends child orders to the market that buy at the offer and sell at the bid. Each such trade loses money immediately: buying high and selling low. The volume is enormous because SMARS processes thousands of orders per second.

09:31:00 to 09:32:00. Knight’s positions begin accumulating rapidly. The eighth server is taking large positions in stocks it should not be holding. Other market participants, seeing unusual order flow, adjust their behavior. Prices in affected stocks begin to move erratically.

09:32:00 to 10:15:00. For 45 minutes, the eighth server continues executing Power Peg trades. Knight’s trading desk sees the anomalous positions building but does not have a single kill switch that can halt all SMARS order routing. The team attempts to diagnose the problem while losses accumulate.

10:15 EDT. Knight halts SMARS. By this time, SMARS has executed over 4 million trades in 154 stocks, accumulating a net long position of approximately $3.5 billion and a net short position of approximately $3.15 billion. The resulting losses total approximately $440 million. This is more than Knight’s entire market capitalization.

The diagram shows the relationship between the deployment state of each server and the flag activation signal. Seven servers receive the same flag and execute new code. One server receives the same flag and executes nine-year-old code. The loss accumulation is approximately linear over the 45-minute window because the order flow is roughly constant: the eighth server processes its share of incoming orders continuously, and each order processed through Power Peg loses money.