The bots are running crazy

[Erel]

It is almost impossible now to run a non-small site without CloudFlare, protecting and caching the data for all these bots. This is a new phenomena and I wasn't surprised to see that CloudFlare failing took down large parts of the internet yesterday.

 

[Pendrush]

I'm also running website with about 400.000-450.000 unique visitors per day. Without Cloudflare, I'd probably need load balancing for the website to function.
The number of bots hitting the site's pages is unreal.

PS.
CloudFlare fall explanation: https://blog.cloudflare.com/18-november-2025-outage/
Actually is one line (82) of code in Rust, as developers love to take shortcut(s) and use unwrap():

 

[aeric]

Recently I heard some people want to rewrite Linux kernel with Rust and promote it is better in many ways compare to C or C++.

 

[Alessandro71]

Recently I heard some people want to rewrite Linux kernel with Rust and promote it is better in many ways compare to C or C++.

looks like this is not an issue of memory corruption or something a safety feature of a language can prevent.
programming errors are language-agnostic

 

[Sandman]

View attachment 168456

It is almost impossible now to run a non-small site without CloudFlare, protecting and caching the data for all these bots. This is a new phenomena and I wasn't surprised to see that CloudFlare failing took down large parts of the internet yesterday.

It doesn't seem unreasonable to only allow a small part of the forum to be public. (So newbies can see that there's activity in here.) For the full forum, one would need an account - which is simple and free to get.

That should stop a lot of bot traffic, right?

 

[Erel]

That should stop a lot of bot traffic, right?

Bot traffic is not inherently bad. I do want the data in the forum to be crawled by Google, ChatGPT and other well behaved bots.

 

[Sandman]

Bot traffic is not inherently bad. I do want the data in the forum to be crawled by Google, ChatGPT and other well behaved bots.

Ah, fair enough. I didn't think of that case.

 

[Sandman]

I saw these a couple of days ago, thought I'd post here. I don't know if this is relevant for the forum, but generally speaking it seems like a good idea.

Use the Accept Header to serve Markdown instead of HTML to LLMs

Make your website accessible to LLM agents by serving plain Markdown when they request text/plain or text/markdown. Save tokens and improve agent experience with this simple Astro and Cloudflare Workers implementation.

www.skeptrune.com

GitHub - thekevinm/HTML-to-MD-AI: Lightweight tool that converts traditional HTML pages to Markdown when being invoked by AI Agents to reduce token usage and improve readability.

Lightweight tool that converts traditional HTML pages to Markdown when being invoked by AI Agents to reduce token usage and improve readability. - thekevinm/HTML-to-MD-AI

github.com

Edit: I should have clarified that my thought was that perhaps XenForo have a plugin that serve Markdown instead of HTML if the content is being consumed by a bot. If not, perhaps they would be willing to make one, assuming there is interest from their customers.