The bots are running crazy

[Erel]

It is almost impossible now to run a non-small site without CloudFlare, protecting and caching the data for all these bots. This is a new phenomena and I wasn't surprised to see that CloudFlare failing took down large parts of the internet yesterday.

 

[Pendrush]

I'm also running website with about 400.000-450.000 unique visitors per day. Without Cloudflare, I'd probably need load balancing for the website to function.
The number of bots hitting the site's pages is unreal.

PS.
CloudFlare fall explanation: https://blog.cloudflare.com/18-november-2025-outage/
Actually is one line (82) of code in Rust, as developers love to take shortcut(s) and use unwrap():

 

[aeric]

Recently I heard some people want to rewrite Linux kernel with Rust and promote it is better in many ways compare to C or C++.

 

[Alessandro71]

Recently I heard some people want to rewrite Linux kernel with Rust and promote it is better in many ways compare to C or C++.

looks like this is not an issue of memory corruption or something a safety feature of a language can prevent.
programming errors are language-agnostic

 

[Sandman]

View attachment 168456

It is almost impossible now to run a non-small site without CloudFlare, protecting and caching the data for all these bots. This is a new phenomena and I wasn't surprised to see that CloudFlare failing took down large parts of the internet yesterday.

It doesn't seem unreasonable to only allow a small part of the forum to be public. (So newbies can see that there's activity in here.) For the full forum, one would need an account - which is simple and free to get.

That should stop a lot of bot traffic, right?

 

[Erel]

That should stop a lot of bot traffic, right?

Bot traffic is not inherently bad. I do want the data in the forum to be crawled by Google, ChatGPT and other well behaved bots.

 

[Sandman]

Bot traffic is not inherently bad. I do want the data in the forum to be crawled by Google, ChatGPT and other well behaved bots.

Ah, fair enough. I didn't think of that case.

 

[Sandman]

I saw these a couple of days ago, thought I'd post here. I don't know if this is relevant for the forum, but generally speaking it seems like a good idea.

Use the Accept Header to serve Markdown instead of HTML to LLMs

Make your website accessible to LLM agents by serving plain Markdown when they request text/plain or text/markdown. Save tokens and improve agent experience with this simple Astro and Cloudflare Workers implementation.

www.skeptrune.com

GitHub - thekevinm/HTML-to-MD-AI: Lightweight tool that converts traditional HTML pages to Markdown when being invoked by AI Agents to reduce token usage and improve readability.

Lightweight tool that converts traditional HTML pages to Markdown when being invoked by AI Agents to reduce token usage and improve readability. - thekevinm/HTML-to-MD-AI

github.com

Edit: I should have clarified that my thought was that perhaps XenForo have a plugin that serve Markdown instead of HTML if the content is being consumed by a bot. If not, perhaps they would be willing to make one, assuming there is interest from their customers.

 

[Magma]

View attachment 168456

It is almost impossible now to run a non-small site without CloudFlare, protecting and caching the data for all these bots. This is a new phenomena and I wasn't surprised to see that CloudFlare failing took down large parts of the internet yesterday.

Will be nice a tutorial for CloudFlare using... some how to, suggests.... some screenshots... exchange ideas

* for Chit Chat ofcourse not for Development Sections... (but also for those using jserver at production... some cloudflare knowledge will make the difference)

 

[Didier99]

Bot traffic is not inherently bad. I do want the data in the forum to be crawled by Google, ChatGPT and other well behaved bots.

Interestingly, ChatGPT seems to be relatively proficient in B4X.
It's not at the level of other more common languages but it was useful the couple of times I asked questions.
Just hallucinating a bit, so don't expect everything to work immediately, but useful nonetheless

Edit: I used Microsoft CoPilot today and it too was useful (my work only allows the CoPilot version tailored to our company, so that IP is not used to feed back the monster...) It had to do with Regex. The first pass did not completely work, but after refining my query, the second pass was a hit

Edit 2: one thing that ChatGPT consistently misses is the Bit functions. It uses

a And b

instead of

Bit.And( a, b )

 

[aeric]