Ask HN: Has everyone just given up on blocking bots?

MattGaiser · 2025-01-24T04:58:58 1737694738

I think the challenge is solving it in a way that users can still use the product.

I’ve done a fair bit of scraping for various things. Companies that attempt to defeat scraping often ruin the user experience along the way.

As an example I am aware of, there is an airline that keeps doing all sorts of things to defeat scrapers.

Problem is, the site now constantly throws errors for regular users doing searches and people are regularly getting banned for doing too many searches.

And they still haven’t done much but made scrapers increase their retry counts.

readyplayernull · 2025-01-24T20:09:52 1737749392

It's time to level up in this arms race. Let's stop delivering html documents, use animated rendering of information that is positioned in a scene so that the user has to move elements around for it to be recognizable, like a full site captcha. It doesn't need to be overly complex for the user that can intuitively navigate even a 3D world, but will take x1000 more processing for OpenAI. Feel free to come up with your creative designs to make automation more difficult.

dTal · 2025-01-24T20:59:56 1737752396

So, Flash websites? Please no...

wruza · 2025-01-25T21:32:56 1737840776

Why flash, mp4/webm!

Btw flash websites were cool and games were amazing. You just can’t write a game like that today. You may pretend that you can, but no.

truesign · 2025-01-25T11:29:34 1737804574

I had those same problems time ago and ended up creating https://truesign.ai

Still in beta but successfully protecting millions of requests daily.

During beta I'm giving free service in exchange of user feedback. Do register if interested!

LinuxBender · 2025-01-24T11:58:25 1737719905

Has everyone just given up on blocking bots?

Not at all. I block them on my silly hobby sites with great success. [1] I've never used a CDN on my personal stuff.

[1] - https://news.ycombinator.com/item?id=42792862

johng · 2025-01-24T04:34:05 1737693245

We are considering blocking all the AI bots via Cloudflare but I wonder if there is any downside. Are we going to lose ranking in any search engines? Any downsides that I can't think of?

carlosjobim · 2025-01-25T17:50:22 1737827422

> You can maybe throw Cloudflare in front of your site, and that's it.

What's wrong with Cloudflare?

nejsjsjsbsb · 2025-01-24T10:53:09 1737715989

Think about it. A human visitor is a human using a bot (their browser) to get information on their behalf. Distinguishing between a human using a bot in the good way vs. a human using a bot in a bad way is going to be cat and mouse as nauseum.

Anything that a human can read can be used for AI training. You can maybe avoid by paywalling. Or limiting users to people you have spoken to. But maybe you can't trust em all.

There are commercial offerings to help solve this but there is a reason you need to pay.