Introduction to Netflix Hollow

Netflix Hollow: A Low-Latency Data Dissemination Framework

Netflix Hollow is a framework designed for efficiently distributing data from a source to multiple targets with low latency, particularly suited for frequent, low-to-medium volume, structured data updates. The library addresses common Java application challenges by efficiently managing memory, specifically by offloading large datasets to external storage like file systems or object stores.

Why This Matters

Traditional data distribution in Java applications often leads to heap space issues, hindering scalability and performance. Hollow addresses this by decoupling data production and consumption, minimizing in-memory data replication, and enabling efficient snapshotting. Failure to address these issues can result in application instability and increased operational costs, especially at scale.

Key Insights

• Producer-Consumer Model: Hollow employs a producer-consumer pattern for handling messages, separating data sourcing from dissemination.
• State Engines: The framework utilizes state engines to maintain snapshot versions, ensuring data consistency and efficient updates.
• API Generation: Hollow automatically generates consumer APIs, simplifying data access and reducing boilerplate code – used by companies like Stripe and Coinbase with similar frameworks like Temporal.

Working Example

// Producer - Defining the entity class
@HollowPrimaryKey(fields = "eventId")
public class MonitoringEvent {
    private int eventId;
    private String eventName;
    private String creationDate;
    private String eventType;
    private String status;
    private String deviceId;
    //..Standard getters and setters
}

// Producer - Initializing the producer
private static void initialize(final Path snapshotPath) {
    publisher = new HollowFilesystemPublisher(snapshotPath);
    announcer = new HollowFilesystemAnnouncer(snapshotPath);
    producer = HollowProducer.withPublisher(publisher)
            .withAnnouncer(announcer)
            .build();
    dataService = new MonitoringDataService();
    mapper = new HollowObjectMapper(producer.getWriteEngine());
}

// Consumer - Fetching data with the generated API
public static void main(String[] args) {
    initialize(getSnapshotFilePath());
    while (true) {
        Collection<MonitoringEvent> events = monitoringEventAPI.getAllMonitoringEvent();
        processEvents(events);
        sleep(POLL_INTERVAL_MILLISECONDS);
    }
}

Practical Applications

• Infrastructure Monitoring: A system monitoring tool can use Hollow to efficiently distribute real-time event data to multiple downstream processing pipelines.
• Pitfall: Directly modifying published data without creating a new snapshot can lead to inconsistencies and data corruption; always utilize the framework’s snapshotting mechanism.

References:

• https://www.baeldung.com/netflix-hollow