Cloudflare took down our website after trying to force us to pay $120000 within 24h
Cloudflare took down our website after trying to force us to pay $120000 within 24h
TL;DR: We've been on the Cloudflare Business plan ($250/month) for years. They suddenly contacted us and asked us to either pay them $120k up front for one year of Enterprise within 24 hours or they would take down all of our domains. While this escalated up our business we had 3 sales calls with th...
Also, interesting comment I found on HackerNews (HN):
This post was definitely demoted by HN. It stayed in the first position for less than 5 minutes and, as it quickly gathered upvotes, it jumped straight into 24th and quickly fell off the first page as it got 200 or so more points in less than an hour.
I'm 80% confident HN tried to hide this link. It's the fastest downhill I've noticed on here, and I've been lurking and commenting for longer than 10 years.
Cloudflare took down our website after trying to force us to pay 120k$ within 24h
Yikes. That sounds bad.
I'm a SysOps engineer at a fairly large online casino.
Okay all my sympathy is gone. Online casinos deserve to die.
That said, my feelings towards economic vampires aside, the way the events unfolded is concerning to say the least. Cloudflare has been racking up evil-corp points quite rapidly in recent months.
As a person who works in server hosting (not as devops or IT), I'm often privy to customer interactions. I feel like my company does a really good job at damage control - where if we fuck up, some rep gets on the phone and makes things right. We've eaten costs on behalf of our customers.
But sometimes, you just gotta tell a customer to go fuck themselves.
And those customers, those biggest complainers are often in online gambling, crypto, adult content, or racist shit.
We get DDos'd a lot from it. But I'm glad the company I work for doesn't bow down to garbage companies.
I'm honestly not surprised.
I used to hook up with a guy who was 100% convinced that he could game the system. It had something to do with break frequencies from various services and certain time windows for playing. He won sometimes, but he obviously didn't talk much about his losses. He wasn't a very happy person, and I think gambling offered an easy release.
That's my big issue with gambling. It's a business preying on addicts leaving many in financial ruin, and overall they do nothing for society at large. Here in Sweden it is regulated, but you honestly don't notice it. There are so many internet casinos vanishing and cropping up on an almost daily basis. If you turn on the radio the adverts are like 40% online casinos, 40% sex toy sites, and 20% various services, like tyre shifting, glass repairs, etc.
I just wonder how much was left out
That's fair, this is one part of the story, and it's not like screenshots can't be doctored. Any screenshot taken from the web is ridiculously easy to manipulate.
Key.
Key.
Key.
If it's providing games of skill like online poker, it's actually a very intellectually stimulating game. People have made a ton of instructional videos and many books on the poker variations.
After playing poker professionally I was able to leverage the skills of bankroll management and emotional control to become successful in investing in the stock market.
I held all of my stocks through the entire pandemic to rebound from a loss over multiple years holding tech to a $600,000 profit by buying at the bottom. If I hadn't played poker I probably wouldn't be able to stomach looking at a six digit loss in 2021. I only sold my bonds which I used to buy more stocks at a cheaper price (which was the point of the bond allocation)
I used to be in credit risk for a very large stock market company.
Calling the bottom of the market is the same as betting big and getting 21 in blackjack.
Super cool when it happens, but not skill. The number of grown men I had to hear crying because they were dollar cost averaging down to the bottom until they went broke still disturbs me.
I’m happy this worked for you, but it was not skill.
Jesus. Something shady is happening with cloudflare.
That does not inspire confidence.
Is there? The casino is on a cheap $250 a month plan they don't belong on and they broke ToS with the domains. While also costing Cloudflare money each month (as the casino admits themselves, their traffic alone is worth up to $2000 a month).
It's absolutely in the right of Cloudflare to drop a customer that's bothersome. Casinos usually are (regulations, going around country restrictions), them costing them money on top is a massive issue.
120k a year is a big slap of course, but it's probably the amount Cloudflare would want to keep them on as a customer. If they leave, so be it.
I've seen it several times before at companies I worked at. They cheaped out and went with a tiny service plan to coast by. Or even broke ToS because it would be cheaper. That usually got stopped by plans getting dropped (GitLab Bronze for example), cheap plans getting limited, or the sales team sending a 'friendly' message that we're abusing their plan and how we're going to fix it. If you don't play along at that point you're going to get the hammer dropped on you.
It also wasn't 24h as the title says, the first communication happened in April. At that point they should have started to scramble, either upgrading to a bigger tier immediately or switching providers. And it's totally normal to go to the sales team when you break the ToS of your plan or you abuse a smaller plan. They're going to discuss terms, it's not a technical issue.
Edit: And I should also say, the whole "paying for a whole year is extortion" is bullshit too. Their CFO or CEO told Cloudflare they are looking at switching providers (as they looked at Fastly). So of fucking course Cloudflare is going to demand a full year upfront. Otherwise the casino could pay for a single month and during that month they switch away to another provider. So Cloudflare would still be thousands in the red with that ex-customer after they used so much traffic the last few years.
That Cloudflare were justifiably unhappy with the situation and wanted to take action is fine.
What's not fine is how they approached that problem.
In my opinion, the right thing for Cloudflare to do would have been to have an open and honest conversation and set clear expectations and dates.
Example:
"We have recently conducted a review of your account and found your usage pattern far exceeds the expected levels for your plan. This usage is not sustainable for us, and to continue to provide you with service we must move you to plan x at a cost of y.
If no agreement is reached by [date x] your service will be suspended on [date y]."
Clear deadlines and clear expectations. Doesn't that sound a lot better than giving someone the run-around, and then childishly pulling the plug when a competitor's name is mentioned?
The first communications were intentionally misleading though. CF wasn't trying to solve a problem, they were trying to sell a service. If CF had just led with "upgrade or we nuke your site" then that's scummy, but fair. Leading these guys on about technical problems and "trust & safety" bullshit was not fair at all.
The biggest red flag is the up-front payment for a year, gives the indication that they are in actual financial trouble, meaning short in cash right now.
Fucking idiots could have been just increasing the price yearly without any resistance, it’s unlikely a big casino would care about an extra 50-100 per month.
I'm pretty heavily invested in cloudflare. This news is definitely making me reconsider that investment.
What I can say, is their stock is looking very healthy. There are a lot of people buying a lot of stock for them and the prospect over the next 3 to 5 months looks very promising. The only way they wouldn't have cash on hand as if they're spending a ridiculous amount of cash on some project that I'm not aware of, and I feel like I would be aware of it.
This is very peculiar. Definitely warrants further investigation.
As I said in another comment: The up-front payment is the only thing that makes sense for Cloudflare. You got a customer that's costing you money each month. They broke ToS. You offer them a deal still to keep the services running. And their CEO/CFO tells you they are looking at other providers like Fastly.
If Cloudflare gave them a monthly contract then the casino would simply pay for a month and switch over their services to a competitor in that time. So Cloudflare loses all the money from the past (where the casino used far too much traffic) and will barely recoup 10k (minus the running cost, so more likely 7k at the high end) for a single month. It's just not worth it.
So they offer: Stick with us for a full year at least or get fucked. Which is fair.
Exactly my thoughts
The biggest red flag is the up-front payment for a year
Another comment pointed out this was probably to prevent them from signing up for a month then using that month to bounce to another provider
I think it's far more likely there's some sales goal and or performance indicator at play here.
0F,CloudFlare don't need to subsidise an online casino with millions of subscribers, at everyone else's expense. Sure CF are a bunch of gigglefucks but this time I think they made a good decision.
I read the post and it doesn't sound abusive at all
Plus: cloudflare kept putting them in touch with the sales department. Not legal. Not technical support
It's just shit customer service, even if the customer is making a ton of money compared to your fees. Should a casino pay more for other services, too, just because they" don't need a subsidy"?
It's not the decision to ask more money, it's how they made it and in violation of their own terms of service, also extortion, so yes they are dipshits.
Found the thread on HN. Here's what (I'm guessing) a mod had to say:
It set off the flamewar detector, got flagged by users, and got downweighted by a mod.
The 'customer support of last resort' genre is common and not usually a good fit for HN [1]. If people feel this story is unusually relevant and interesting, I'm not sure I agree—long experience has taught us that one-sided articles like this nearly always leave out critical information—but I also don't mind yielding in an occasional specific case, so I've rolled back the penalties on this thread.
The issue from our point of view is not about story X or company Y—it's a systemic one: the most popular genres of submission (especially the rage-inducing ones) get massively over-represented by default, so countervailing mechanisms are needed [2] if we're to have a space for the more intellectually curious stories that the site is meant for.
Okay, that's understandable
Cracking insight - well done!
I love hacker news. The internet needs more things like this
HN thread is here and it's on the front page 7 hours old: https://news.ycombinator.com/item?id=40481808
Many mentions made that a significant part of the issue seemed to be Cloudflare IP addresses getting banned in some countries. They wanted the customer to switch to a bring-your-own-IP plan.
Also, the discussion took place over 1 month, not 24 hours.
I think the HN thread is reasonably informative and nuanced. CF didn't do great but it was somewhat a fog of war situation.
Yeah this substac just reads as we abused cloudflare then were surprised they didn't take us saying no well.
The irony here, is this is the kind of vague and obtuse fuckery online casinos and sportsbooks pull with their customers all the time.
The irony here is that the article author confirms that they break TOS of CF and he still has a Pikachu face. Reddit discussion is pretty positive that CF is right in their decision and that new provider will shut them down at some time as well.
Realistically, this is why you pay for Akamai. You don't get these shenanigans.
How the fuck were they still on a $250 dollar a month plan when they pumped through $2000 a month worth of traffic? That's shady on the companiy's part and Cloudflare shouldn't have allowed it to happen in the first place.
Each party played their part here and did shitty things. Sounds like the tech equivalent of a crackhead arguing about selling stuff to the pawn shop employee.
The $250/month plan supposedly includes unlimited traffic. If there's actually a limit where you're supposed to switch to a more expensive plan with no standardized price, maybe CF should say what the limit is?
They absolutely should have outlined a traffic limit for the $250 a month plan. That's on Cloudflare for allowing it.
That said, if you make wildly excessive use of that loophole it probably shouldn't surprise you if they do something like this. They called it "trust and safety" because it allows them to do anything they want under the guide of security.
Really, they didn't define their service clearly and wanted to fire them as a customer unless they paid up for what they felt they were owed.
"Unlimited" doesn't exist in this universe. It's always "Unlimited under fair use".
If you pay for your water park ticket and they offer unlimited free drinking water fountains, you can't pay for your ticket, call up Nestlé and bring in the water trucks.
Besides the IP poisoning from the casino, ToS violations and so on, just using this much traffic would probably be enough cause for a cancellation (or a forced plan upgrade).
I worked for Akamai for 7 years.
This is why, if your CDN infra is core to the operation of your business, you make your systems accommodate multi-CDN integration. Cutting one CDN off shouldn't be significantly difficult, and it comes in handy during contract negotiations. All the major players work this way.
Well that all reads like extortion.
"Pay us money or we will destroy your business." Pretty cut and dry extortion. The entire article was infuriating to read.
Regarding the HN shenanigans, their algorithm does some weird things.
If a new post gets too many upvotes and not enough comments, it gets demoted very quickly.
If any of the activity appears manufactured, it basically delists the post.
Very exploitable, but also prevents popular articles that don't stimulate conversation from sticking around on page 1 for too long, and makes botting upvotes do more harm than good.
HN is a libertarian hellhole full of divorced incel energy
The tl;dr seems to be this was a money losing account for Cloudflare, and they couldn't squeeze them so they weaseled out with some TOS violation to prevent losing money on what was promised to be unlimited traffic, they have better lawyers so they're not worried.
Cloudflare 100% in the wrong here, they are closing accounts for TOS violations when they are just unprofitable, I would very strongly consider how tightly to couple with them knowing how cavalier they are about squashing small businesses.
If enough of these happen though, they'll get destroyed by a class action lawsuit, and they'd deserve every bit of it
CF doesn't give a fuck about 80tb of traffic. These guys were in severe TOS violation that could affect all CF customers if CF IPs got blocked. Given 48 hours to bring their own IPs and switch to (expensive AF anywhere) enterprise account and finally shut down TWO WEEKS later after trying to weasel their way out of this instead of accepting they need to pay to play this stupid game.
We've been CF customers forever and enshitification is definitely affecting all of their services and mostly customer support, but in this instance I'm 100% on the side of CF.
I’m 100% on the side of CF.
100%?
We scheduled a call with their “Business Development” department. Turns out the meeting was with their Sales team,
...
So we scheduled another call, now with their "Trust and Safety" team. But it turns out, we were actually talking to Sales again.
This is the part that's ridiculous to me. If CloudFlare thinks they're violating TOS that's fine. If they're willing to let them continue with their business as-is as long as they pay more? That's fine. But, scheduling calls with one group and it turns out it's actually CloudFlare's sales team on the phone, that's ridiculous.
I worked for an online casino in the past. What they do is a standard in the industry. The company I worked for was a small startup and onwed hundreds of domains, mostly just to protect the brand, 98% of which redirected to the main domain, with a few serving slightly different sites for different jurisdictions (e.g. Ontario regulations require that everything happens under a .ca domain). The "blocking evasion" doesn't require CF to do anything, besides forcing the customer to block traffic from certain countries (the ones where you are suspected to evade the block). At this point - if the casino is really operating in the black or gray markets - they can just set ingress to their site outside CF for those countries only if they really wanted. I worked also for a company who was doing this to allow traffic from Russia, changing every day mirrors (and they had an IT department of maybe 20, it was a joke), and Russia was the main market for them.
If what is told in the article is true - I.e. 95% of the traffic was through the main website - then it doesn't look like they were really doing this sort of evading deliberately, considering that in that 5% you have all your alternative TLDs plus the traffic from gray/black markets. Having hundreds of domains and some small percentage of traffic from black markets is something that just happens, it's different from continuously registering new domains for providing access where the previous ones got DNS blocked (this is domain block). It doesn't seem this is what they were doing based on the article, and if they were, then CF emails didn't mention it, which is insane.
Obviously we don't know the full story, so everything has to he taken with a grain of salt.
I did a quick search through Cloudflare's TOS and did not find anything about gambling. What was the TOS violation here?
What I'm seeing is Cloudflare communicating very poorly about what actions the customer would need to take to keep their site operating, why, and what the timeline would be. "We've determined operating your casino website on Cloudflare IP addresses is an unacceptable risk to our other customers and we require that you upgrade to an Enterprise plan within two weeks or your service will be terminated" is clear, concise, and I believe entirely fair. What they did here makes me think they're an unreliable and unpredictable service provider.
Okay, yes this is an issue. But small business? This was a multinational casino site… that doesn’t scream small business to me.
Multi CDN integration is a thing. And fuck CF. Unlimited means unlimited. Stop trying to lie to your customers and change the rules.
If the IP's were an issue, then they wouldn't have offered to make the issue go away with $$$.
But, the guy admits that what they were doing with the domains was expressly permitted in the "Enterprise" class service. If it was expressly prohibited in the "Business" class service, then they set themselves up for the shakedown.
I mean more money would solve the IP issue.
Step 1: Get more money Step 2: Buy additional dedicated IPs for the casino Step 3: Profit from the profits of a casino.
250$ a month for their service seems like cloudflare was straight up losing money on the deal. Although cloudflare seemed to have given them extra time than they said before terminating service, which they didn’t have to do. That being said, I think both sides suck here.
Nah. CF initiated a contract renegotiation, and then suspended services right after being informed the customer was price leveling.
That's crappy.
They gave less than a single billing period notice for a price increase.
That's crappy.
They sent a price increase for 40x the current billings, with no corroborated cost or value.
See where I'm going here?
I agree. It's shitty for Cloudflare to just straight up destroy this company's DNS, but also it seems like the company violated the ToS. They had about two weeks to migrate to something else, but instead they just continued debating with CF. Also, this company doesn't have a secondary DNS server in case CF ever went down? That's pretty stupid on their part. Redundant systems are key, I hope they learned that lesson haha
Isn't CF advertising themselves as the solution to needing multiple DNS' with their failsafes, switchovers and load balancing?
If I need to maintain multiple anyway, what's the benefit of CF to begin with? There are a million CDNs out there I could use instead, if I still have to maintain the network architecture.
Thanks for actually reading my comment unlike the other guy
$250 a month unlimited business plan, is for a business with a Website, like a restaurant website that has a menu and takes reservations, maybe a small online store that handles a few dozen SKUs. When the website is the business they move you to an enterprise solution.
I really love cloudflare especially for my hobby projects but in this case they asked for outright Ransome. From this I learnt to keep Nameservers & domain sellers different. I am going to transfer domain away from nameserver.
Also, interesting comment I found on HN:
What is HN?
Hacker News: https://news.ycombinator.com/
Hackernews
Hackernews, unironically named to appeal tech circles, but run by venture capital fund y-combinator, mainly to promote companies they invest in.
As such it's mostly used by techbros (MBA types) and tech companies to show-off, start drama, push their PR, damage control, and occasionally post news.
It's like linkedin, in reddit format. It's all about your connections.
Sorry for confusion, I edited the post to make it clear
Pro tip: Don't waste your time over there.
Repoint your DNS, send everything to legal, delete Facebook hit the gym
Don't believe anything advertised as unlimited , cause it isn't, they always cover their asses in the fine prints in their TOS.
Right. And if you depend on them for your logic with cloudflare functions you will never be able to migrate to another CDN.
Never let a vender do anything for you beyond standardized features. That's why a "selling point" if we go with this guy we can do this... never makes sense. Because if option B can't do it also you wouldn't want to do "this", and you should probably implement it in a more old-school way.
The same thing happens in fastly with the VCL
First of all, congrats! Your business must have become pretty successful. How exactly did CF decide to “ask” you to switch to Enterprise?
Maybe...
* You violated their terms of service...
I wouldn't say Cloudflare is innocent, here, but this business handled Cloudflare the cudgel that was used to beat them. They admit to doing something with their domains that was expressly prohibited in the service they were paying for.
Then they offered to resolve it in whatever way CF deemed appropriate and CF refused to elaborate exactly which domains were the issue.
resolve it in whatever way CF deemed appropriate
CloudFlare deemed the upgrade to Enterprise service appropriate.
I'd be interested to see if / how Cloudflare will respond to this. Because at this point I'm not 100 percent sure who is in the right.
Yeah I have no sympathy for a casino and as a software developer I would never work for one, but in the other corner you have a company with too much power.
Even if it wasn't a casino, they could either be bullshitting or just be plainly incompetent. Like, idk them personally, but I wouldn't rule it out lol
I wonder how this is going to go
Badly for everyone
Yeah probably, I always get curious when a controversy involving a company starts up because I want to see what happens next
Does this mean hackernews & cloudflare are colluding together?
A simpler explanation is that users are tired of everybody with a customer support issue running to daddy HN and making a big fuss trying to get their way.
After Twitter went to shit, where else do customers have to go for customer support like this?
Admittedly, I didn't read the article, but I have seen plenty of other cases woth cloudfare or other big providers where people have only been able to set things right by kicking up a fuss on social media --- like that recent one with amazon aws.
Today's tech business model:
First you get the power, then you extort the money.
Not a huge surprise, they've got a long history of doing all kinds of scumbag shit. Nobody should be surprised when the leopard eats their face.
Sounds like any Cloudflare customer should reconsider their hosting setup . Mark Anderson has decided to strip the customers to increase the bottom line... And once the numbers are up but the customers are gone.... Will move on to the next company
Yeah pretty much all red flags from cloud flare
Can't wait for this to become a louis rossman video
Lemmy points counter at 666 as I read this post. Is this a sign?
You think we're hitting that point where line go up must wreck lives?
We hit that point centuries ago, if not millennia.
this is disgusting and knowing this, i will never pay cloudflare for anything nor recommend them to anyone ever
Good.
While I have been reading through this topic, I have been feeling worried since I was thinking about using Cloudflare to protect a site of mine for some time. This is because I found out from somewhere that they have protection against AI LLMs scraping page data from websites, which is what I'm mainly worried about since there are things and stories that I put a lot of thought and work into. And finding out about Cloudflare shutting someone else down here over what sounds like the level of traffic has me feeling I might not be able to use them and not sure what other options there are.
- what is HN? Edit: never mind, answered below: hacker news (ycombinator)
- daaamn... I hated cloudflare before for their shitty and non-adblocker-compatible (often not working at all) "I am not a bot" checks, but fuck me are those EVIL motherfuckers....
Yet more evidence that CloudFlare is inherently damaging and hostile to the Internet.
It's fine to not like CloudFlare but this isn't really "evidence" of any kind. It's a one-sided rant on sub stack.
Seriously? The article author admits they're doing illegal shit and break CF TOS and CF is inherently damaging? You ok, mate?