[fpc-pascal] Effect of blocking AI crawlers in forums and wikis

[Tobias Klaus Giesen]

Hello,
I had a moderate DDOS attack on my forum too, but I am sure it was not
related to AI crawling. Too many different IP addresses.

It was very easy to solve it with Cloudflare. My forum now detects if the
visitor is a real user, but lets known search engines and AI bots thru.

The known bots registered with Cloudflare behave well and don't flood
forums.

I chose a payed Cloudflare plan (around $25 per month), but it might be
possible with the free plan too.

If not, I would be more than happy to pay for this for the FPC forum.

I think that it is very important that legitimate AI crawlers and search
engines have access.



On Sat, Sep 6, 2025 at 9:38 AM Nikolay Nikolov via fpc-pascal <
fpc-pascal at lists.freepascal.org> wrote:

>
> On 9/6/25 12:05 AM, Wayne Sherman via fpc-pascal wrote:
>
> Is it possible to make the forum and wiki data available on a mirror
> for the use of AI crawlers?
> (are search engine crawlers also blocked?)
>
> Would you pay money for the hosting and for the maintenance of such a
> mirror?
>
> You asked ChatGPT5:
> "If websites with large Pascal forums and wikis block AI data
> harvesting bots, what will be the result for LLMs missing this
> training data and what will be the effect on these LLMs for Pascal
> related queries? Provide a simple concise answer with bullet points."
>
> You should've asked:
>
> "If websites with large Pascal forums and wikis get DDoS'd by AI data
> harvesting bots..."
>
> The answer should be that, if they get DDoS'd, then nobody would be able
> to access them - neither AI bots, nor regular users. So, the outcome would
> be worse.
>
> ChatGPT said:
>
> Please don't post ChatGPT generated responses as if they have any kind of
> authority. ChatGPT comes with a fine print: "ChatGPT can make mistakes." In
> this particular case, the company that trains ChatGPT has a clear conflict
> of interest. It is only normal that it exaggerates the effect of blocking
> AI harvesting bots.
>
> Nikolay
> _______________________________________________
> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-pascal/attachments/20250906/327ecc70/attachment.htm>