[fpc-pascal] Effect of blocking AI crawlers in forums and wikis

[Marco van de Voort]

Op 5-9-2025 om 23:05 schreef Wayne Sherman via fpc-pascal:
> Is it possible to make the forum and wiki data available on a mirror
> for the use of AI crawlers?
> (are search engine crawlers also blocked?)

To my best knowledge, only misbehaving, high frequency robots are banned.

These mostly operate out of Asia with very high numbers of distinct IPs 
and very high fetch frequencies. It appears there are Chinese ISPs that 
offer packages to "accelerate" gathering content for AIs.

It is unlikely to affect good citizens like ChatGPT.