The Chaos Test Harness

Introduction to Chaos Engineering

Chaos engineering is a discipline involving proactive fault injection into a system to observe behavior and build resilience. This approach has become crucial in ensuring the reliability and performance of distributed systems. One of the key tools in chaos engineering is Toxiproxy, a proxy framework designed specifically for simulating network and system conditions. Toxiproxy supports various toxics, including latency, bandwidth, slow_close, timeout, slicer, and limit_data, allowing for comprehensive fault injection scenarios.

Setting Up Toxiproxy with Testcontainers

To integrate Toxiproxy into a test environment, Testcontainers can be utilized. Testcontainers is an open-source Java library that provides lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container. The ToxiProxyContainer class from the Testcontainers library simplifies the setup of a Toxiproxy server within a test harness. The following Java code snippet demonstrates how to create a Toxiproxy container and a proxy using the Toxiproxy Java Client:

import org.testcontainers.containers.Network;
import org.testcontainers.containers.ToxiproxyContainer;
import eu.rekawek.toxiproxy.Proxy;
import eu.rekawek.toxiproxy.model.ToxicDirection;

public class ChaosHarness {
    private static final Network network = Network.newNetwork();
    private static final ToxiproxyContainer toxiproxy = new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.5.0")
            .withNetwork(network);

    public Proxy createProxy(String name, String targetAddress) {
        var client = new eu.rekawek.toxiproxy.ToxiproxyClient(toxiproxy.getHost(), toxiproxy.getControlPort());
        return client.createProxy(name, "0.0.0.0:8666", targetAddress);
    }

    public void injectHighLatency(Proxy proxy, int latencyMs) throws Exception {
        proxy.toxics().latency("latency-toxic", ToxicDirection.DOWNSTREAM, latencyMs).setJitter(latencyMs / 10);
    }

    public void injectTcpCut(Proxy proxy) throws Exception {
        proxy.toxics().timeout("cut-toxic", ToxicDirection.DOWNSTREAM, 0);
    }
}

This implementation provides methods for creating a proxy and injecting latency or simulating a TCP cut, which are essential for chaos testing scenarios.

Common Toxics for Fault Injection

Toxiproxy offers a variety of toxics that can be applied to simulate different fault conditions. The table below summarizes some of the common toxics used in fault injection:

Conclusion

In conclusion, Toxiproxy and Testcontainers provide a powerful combination for chaos engineering. By integrating Toxiproxy into a test harness using Testcontainers, developers can simulate a wide range of fault conditions, ensuring their distributed systems are resilient and performant under various network conditions. For further information on Toxiproxy and its applications in chaos engineering, refer to the documentation and resources provided by Shopify and the Toxiproxy community [1], [2].

Sources

[1] https://java.testcontainers.org/modules/toxiproxy/ [2] https://github.com/trekawek/toxiproxy-java