[fpc-pascal] FPC websites blocked

[Sven Barth]

Wayne Sherman via fpc-pascal <fpc-pascal at lists.freepascal.org> schrieb am
Do., 16. Apr. 2026, 17:35:

> The FPC wiki website is also blocked from archive.org.  Note the
> latest capture for this page:
>
> https://web.archive.org/web/20250909050155/https://wiki.freepascal.org/Release_engineering


That is a side effect of Anubis and could be adjusted by whitelisting the
crawler from the Archive.

Are these pages also blocked from Google search and other search
> engines?  I am unable to check that but someone with write access to
> the web server can check and monitor the status of Google's web
> crawler indexes.  See here:
> https://support.google.com/webmasters/answer/9012289?hl=en


Ordinary search engine crawlers are already whitelisted by Anubis.

Question:
> "What will the long term consequences be of blocking any of the Free
> Pascal (fpc) websites (i.e. the main website, the forum, and the wiki)
> from AI / LLM tools, archive.org, and web search engines? How will
> users and potential new users of Free Pascal be affected? Summarize in
> one paragraph."
>

It's not relevant what a Stochastic Parrot thinks about this. What is
important is the reality: if we're *not* blocking the Ai crawlers then the
Wiki will be down for *everyone*. Before we added Anubis as a protection
the wiki was often not accessible by anyone due to the high amount of
traffic added by Ai crawlers. You can see this with the forum which
regularly receives high amounts of traffic due to crawlers and then isn't
reachable until one of the admins blocks a wide range of IPs used by those
crawlers.
And no, a CDN like Cloudflare is not a solution we are willing to use
because that is it's own can of worms.

Regards,
Sven

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-pascal/attachments/20260419/b9187146/attachment.htm>