I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time.
Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.
load balancer / proxy, since you need a way to get the real ip
If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure the logs in their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)
*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases.
I could some day split it out into a blog and a demo site.
I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.
You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.
Block on traffic volume and request frequency if that's causing a problem.
(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years.
I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
The comments can be misleading. About 2500 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
I did not and could not downvote you as you replied to me. Only others can. I have not downvoted any comments on this thread. I appreciate all feedback including yours.
The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.
The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.
There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.
A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away.
Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do.
For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.
This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).
Oh, well that is ok for me I suppose. I add RTA/adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
VPN exits often "come from datacenters". And there are entire countries where the web can be unusable without.
If you don't have LaLiga in your country, that doesn't mean everyone is so lucky. Blanket IP range bans, in this day and age? Basically a proclamation of incompetence.
I understand your frustration. Normally a blog site would not have any or many of these measures enabled. This is more of a test site to show what could be done on any other type of site. People can pick and choose which methods to use. Some may wish to block VPNs and proxies especially on forums.
I should make an article that shows additional ways to detect VPN's, including residential.
Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.
I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.
load balancer / proxy, since you need a way to get the real ip
If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure the logs in their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
If you are unable to read this, there is an archived copy at https://archive.ph/d3236
I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.
I disabled blocking for a few hours then crawled several of my pages with their site.
Thanks. Regretful implementation*, being a normie on iOS Safari:
https://i.ibb.co/vCDH79d0/IMG-0303.png
…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)
*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases.
I could some day split it out into a blog and a demo site.
I wonder why you'd be unable to read it? :-)
Because of shitty over-zealous blocking, probably.
I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.
You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.
Block on traffic volume and request frequency if that's causing a problem.
(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years.
410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!
Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.
I mean that just might be the dying gasp of a puddle of scorched fibreglass and boiling copper where the server used to be.
Mostly harmless(c)
Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not.
People can see a few headers here [2]
[1] - https://caniuse.com/?search=sec-fetch-mode
[2] - https://nochan.net/.env
I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
and all valid traffic too, judging by these HN comments (and my own attempts to connect).
The comments can be misleading. About 2500 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
what's up with those response headers? "adult" ...
"ai" ...
response length 68
AI will not ingest or operating on anything with obscene or foul terms or at least that is what I have been led to believe.
weird misdirection excuse
edit:
>"I can not see votes or karma and do not require social validation. "
>proceeds to downvote
classic
I did not and could not downvote you as you replied to me. Only others can. I have not downvoted any comments on this thread. I appreciate all feedback including yours.
The guy is certainly kind of visibly ravaged by the more shady denizens of the global internet.
But there are some fun things to read there in any case.
The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.
The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
Blocks firefox's built in VPN.
That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.
There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.
A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away.
Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do.
For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
most (all?) of those will 100% block valid traffic too
I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.
> Obvious proxy is obvious.
> if ($http_x_forwarded_for) {....
This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).
This may block schools and libraries
Oh, well that is ok for me I suppose. I add RTA/adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
That's fair enough.
I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
VPN exits often "come from datacenters". And there are entire countries where the web can be unusable without.
If you don't have LaLiga in your country, that doesn't mean everyone is so lucky. Blanket IP range bans, in this day and age? Basically a proclamation of incompetence.
> Blanket IP range bans, in this day and age? Basically a proclamation of incompetence.
I disagree. If I'm getting problematic traffic from even a few of your IPs and they're in a DC/VPS range, I'm blocking it and moving on with my day.
But I suppose none of my clients services typically ever need to be accessible from such countries anyway.
Basically a proclamation of incompetence.
I understand your frustration. Normally a blog site would not have any or many of these measures enabled. This is more of a test site to show what could be done on any other type of site. People can pick and choose which methods to use. Some may wish to block VPNs and proxies especially on forums.
I should make an article that shows additional ways to detect VPN's, including residential.
Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.
[flagged]
[dead]