Read on Twitter
Here's a non-corona thread for you about how last month I got all of http://Ghost.org kicked off Google and our entire domain no longer appeared for a single search result. Our traffic immediately tanked and things looked very bad.
It all started with illegal pornography:
It all started with illegal pornography:
A couple of years back we had open trials on http://Ghost.org to make it as easy as possible for people to try out the product. In about 3 clicks you could spin up a brand new hosted indie publication and start sharing stuff with the world. Blazing fast. Really cool.
So of course it got steadily abused, more and more, especially by people setting up spam SEO sites for link farms and people sharing torrent download links. Also, by a particularly aggressive group in the business of sharing illegal/copyrighted premium porn.
One time, I managed to trace one of the IP addresses signing up all these porn accounts to a web development / marketing agency in Malaysia -- tracked down their out of hours emergency number -- and phoned (what I assume was) the founder of the company at 2am his time.
I said please can you stop sending us all this porn. We're a 5 person nonprofit organisation just trying to survive. We're not google or facebook, and you're destroying our servers with all these pixelated penises.
He was very apologetic and for a while there was no more porn.
He was very apologetic and for a while there was no more porn.
Eventually, though, other people caught on and created more spam sites - and at the start of 2018 we finally decided to validate credit cards in order to start a free trial.
No charge on signup, and no auto-charge at the end of trial. Just validation.
Spam solved! Until 2020
No charge on signup, and no auto-charge at the end of trial. Just validation.
Spam solved! Until 2020
At the start of this year, the strategy we used to get rid of one type of spammer... attracted a new type. Fuck me.
Now there's people who do a "big hack" of cards stored in plaintext somewhere, then try to sell them on the black market before the breach is up on @haveibeenpwned
Now there's people who do a "big hack" of cards stored in plaintext somewhere, then try to sell them on the black market before the breach is up on @haveibeenpwned
But you're a hacker, and you just got 100,000 credit card details from a database somewhere which you now want to sell on. The first question you're going to get to establish market value is... "how many of them work?"
And so it turns out we inadvertently designed the ultimate credit card testing tool.
A public facing credit card form, which DOES validate a card can be charged, but DOESNT charge it, meaning it's nearly invisible to the victims:
A public facing credit card form, which DOES validate a card can be charged, but DOESNT charge it, meaning it's nearly invisible to the victims:
Doesn't show up on printed statements, doesn't show up on internet banking, doesn't show up ANYWHERE except for a temporary "pending transaction" list with some more modern providers.
Best of all! It's from a known merchant (us) with well established reputation/history!
Best of all! It's from a known merchant (us) with well established reputation/history!
So these motherfuckers start running big batches of thousands of card tests against our signup page. Spam prevention is now spam magnet.
They do it via a distributed botnet masquerading as real users filling out/submitting the form, so there's basically no way of filtering.
They do it via a distributed botnet masquerading as real users filling out/submitting the form, so there's basically no way of filtering.
Infrastructure team is trying everything but they're getting through all our honeypots, and we're losing our minds.
I'm going through our firewall logs looking at the requests and spot a couple of signatures which are easy to block. Then, all of a sudden there's a new spike
I'm going through our firewall logs looking at the requests and spot a couple of signatures which are easy to block. Then, all of a sudden there's a new spike
Thousands of requests from an IP block originating from Google as in Google Cloud Platform. Not on my watch. Block!
And it worked. For about 2 weeks our firewall diligently denied every single fucking request made by Google Web Crawler ... NOT cloud platform ...
And it worked. For about 2 weeks our firewall diligently denied every single fucking request made by Google Web Crawler ... NOT cloud platform ...
One day I searched for something we normally rank for, and we didn't. And then uh ... saw that http://Ghost.org no longer existed anywhere in Google ...
At all.
Shit.
At all.
Shit.
So yeah, terrible timing but correlation and causation are still very different things. Lessons were learned.
Fortunately rankings recovered very quickly and got reindexed after we resubmitted sitemaps, etc. Traffic recovered shortly after.
Worked with @stripe to report it all
Fortunately rankings recovered very quickly and got reindexed after we resubmitted sitemaps, etc. Traffic recovered shortly after.
Worked with @stripe to report it all
Long story short: This is why we can't have nice things.
Thank you for coming to my Ted Talk. Like and subscribe. Merch link in bio.
Thank you for coming to my Ted Talk. Like and subscribe. Merch link in bio.
