Denial of Service (DoS) Attack: Day 5

Continued from the previous post (Sept 12th, 2024): Denial of Service (DoS) Attack: Continued Adventure.

Edit @ 2024-09-13 14:30 PDT: See updates at the end of the post.

Day 5: CloudFlare & 800 Million Requests

We made a change to our DNS settings overnight (PDT) to route traffic through CloudFlare. This has helped a lot. While the attack is ongoing, the traffic is now being filtered through CloudFlare's network. This has significantly reduced the load on our servers and provides protection against various types of cyber mischief.

Almost 7.5TB of data has been transferred in just the past 6 hours (this particular chart only goes down to 24h, but the DNS was only switched over to CloudFlare a little over 6h ago).

On privacy

For over 10 years, Onetime Secret has taken a self-hosted approach to data privacy. This is one of the reasons we don't have a cookie notice: there are no third-party tools or services that could potentially track you. We don't even use browser-based analytics at all (not even PostHog). We've also kept our servers in the EU by default, so CloudFlare marks the first time we've had traffic potentially routed through the US. To be clear, your data has remained in the EU throughout this entire process (actually within the same hosting provider at the same data center in Germany). This will not change.

There is a trade-off here, though. We've had to route traffic through CloudFlare to mitigate the attack. There are very few solutions for dealing with attacks of this scale, so our options were limited. Even if we had found another workable solution this time, it's likely we'd face a similar decision in the future as attacks become larger and more sophisticated.

Ultimately, it came down to balancing potential privacy concerns against actual existential threats to the service. Our position has always been to keep things simple to alleviate even the perception of privacy concerns. We had to make a decision to keep the service running, which makes it slightly more challenging to explain how our infrastructure works and how we protect your data. However, in practical terms, it doesn't introduce any actual privacy issues or compromise our commitment to privacy and security.

What's Next

We'll continue to monitor the situation and work with CloudFlare to ensure we're doing everything possible to mitigate the attack. It's still possible that the attackers could change their tactics, so we'll need to stay vigilant. Having CloudFlare in place now is a significant relief.

Updates

14:30 PDT - Thank you, thank you

We're pleased to report that the attack has subsided. We're still monitoring the situation, but it looks like we're in the clear for now. We'll be working on a more detailed post-mortem to share with you all. We'll be sure to include the technical details of the attack and our response. We're thankful for your patience and understanding as we worked through this.

A special thank you to the folks that reached out with advice and support. You made a material difference in our ability to respond to this incident.

14:00 PDT - Over 200 Million Served

CloudFlare has blocked over 200 million threats since we started using their service overnight. The term threat is a bit strong for what we were facing but it gives a sense of the scale of the attack (especially considering it began 4.5 days ago). We're grateful for CloudFlare to say the least. This is not a sponsored post - we're just really happy to have our lives back and to continue running Onetime Secret.

Epilogue

In retrospect, if we had converted each threat to a burger, we'd have rivaled McDonald's total historical sales from their founding through the early 1980s. Although to be fair, we're still a long ways off from opening 1.5 fast-food restaurants every day.