GitHub Refines Layered Defenses to Reduce False Positives
These articles are AI-generated summaries. Please check the original sources for full details.
GitHub Reworks Layered Defenses after Legacy Protections Block Legitimate Traffic
GitHub engineers recently investigated user reports of unexpected “Too Many Requests” errors, tracing the issue to abuse-mitigation rules that had accidentally remained active long after the incidents that prompted them, with affected users making only a handful of normal requests. The investigation revealed that older incident rules were based on traffic patterns that were strongly associated with abuse at the time but later began matching some legitimate, logged-out requests, highlighting the importance of regularly reviewing and updating defensive controls.
Why This Matters
The technical reality of layered defenses is that they can provide robust protection against threats, but they can also lead to false positives and unnecessary blocking of legitimate traffic, resulting in unacceptable user impact, as seen in GitHub’s case, where even a small fraction of blocked requests can have significant consequences, emphasizing the need for careful management and monitoring of defensive controls to ensure they remain effective and do not outlive their usefulness.
Key Insights
- According to GitHub, only a small subset of requests that matched suspicious fingerprints were blocked, resulting in roughly 0.5-0.9% of fingerprint matches being blocked: https://www.infoq.com/news/2026/02/github-layered-def/
- Layered defenses can make attribution harder when something goes wrong, as each layer can legitimately rate-limit or block, and isolating which layer made the decision requires correlating logs across multiple systems with different schemas.
- Vercel’s published request lifecycle describes requests encountering “multiple stages” of its firewall protections, followed by an additional WAF stage for project-level policies, demonstrating the use of layered defenses in other large platforms: https://vercel.com/
Working Example
# Example of a simple rate limiter
import time
class RateLimiter:
def __init__(self, max_requests, time_window):
self.max_requests = max_requests
self.time_window = time_window
self.requests = []
def allow_request(self):
current_time = time.time()
self.requests = [request for request in self.requests if current_time - request < self.time_window]
if len(self.requests) < self.max_requests:
self.requests.append(current_time)
return True
return False
# Create a rate limiter with a maximum of 10 requests per minute
limiter = RateLimiter(10, 60)
# Test the rate limiter
for i in range(15):
if limiter.allow_request():
print(f"Request {i} allowed")
else:
print(f"Request {i} blocked")
Practical Applications
- Use Case: GitHub’s experience demonstrates the importance of regularly reviewing and updating defensive controls to ensure they remain effective and do not outlive their usefulness, highlighting the need for careful management and monitoring of layered defenses.
- Pitfall: A common anti-pattern is to introduce defensive controls without proper monitoring and maintenance, leading to unnecessary blocking of legitimate traffic and decreased system resilience, emphasizing the need for a structured approach to defense-in-depth.
References:
Continue reading
Next article
Google Looker Bugs Allow Cross-Tenant RCE, Data Exfil
Related Content
Building Real-Time Streaming Systems with Apache Kafka and Python
Apache Kafka enables distributed systems to process millions of messages per second using scalable brokers and idempotent producers.
Engineering Reliable AI Agents: Why Programmatic Tests Must Replace Prompt-Only Control Flow
Michael Tuszynski argues that reliable AI agents require programmatic tests over prompts to prevent failures like PocketOS's database loss.
Why Reference Architectures May Be Sabotaging Your Platform
Jordan warns that treating reference architectures as destinations leads to high-overhead platforms like unnecessary multi-cluster Kubernetes setups.