[fpc-devel] Lazarus server back online

[Marc Weustink]

Hi all,

It took a bit longer than expected, but I'm happy to inform you that the 
Lazarus services are back online.
For those interested in why it took longer, I'll explain at the end of 
the message.

Marc

On 24-12-2021 08:30, Marc Weustink wrote:
> Hi,
> 
> On Monday 27 December 9.00 CET (8.00 GMT) the Lazarus server will be 
> down for maintenance. This affects the following services:
> 
> * Lazarus website
> * Lazarus mailinglists
> * Lazarus online package manager
> * Lazarus and FreePascal forum
> 
> Thanks,
> Marc

The story....

The server was running Ubuntu 16 LTR, so it's support ended on April 
this year. An attempt at that time failed since Ubuntu decided to switch 
to systemd-resolve which resulted in a server not even able to resolve 
its own name, let alone other hosts. When Googling about it, you learn 
that it doesn't work, it's a half backed solution, not a full DNS etc. 
At that time it became clear that I wouldn't be able to solve that in an 
evening. Luckily I could try this on a cloned server (we had to rent for 
another issue) so I parked the upgrade till I could spend a full day.

Yesterday I found the "correct" solution to this which appeared to work. 
So I continued to upgrade to Ubuntu 18 and finally to Ubuntu 20 LTR. 
Everything seemed to work until I enabled the mailserver. It couldn't 
resolve any mailserver for a given domain.
What the f.. 'host -t mx freepascal.org' resolves, why can't postfix 
resolve it. Again after some Googling, postfix needs a /etc/resolf.conf. 
However one step of the DNS solution was to remove the /etc/resolf.conf 
symlink, so I tried to restore the original link to some systemd-resolve 
generated one. This one pointed to their internal resolver. Still no 
luck since I hadn't configured systemd-resolve which DNS itself should 
use. After doing so, the generated resolf.conf became empty ?????
After more Googling I found that systemd-resolve generates another conf 
where you also can link to. No clue why there have to exist another 
version, but this one works.
So this part of the server upgrade got finished around 14:00. The 
Lazarus mailing list and main website were back online.

Another wish we had was to change the database backend of the forum. It 
appeared over time that when doing a search on the database, mysql 
blocks updates, so browsing the forum becomes unresponsive. The current 
version of SMF supports different databases so we decided to go for 
PostgreSQL (I'm using them at work for years now).
Migrating MySql data to PostgreSQL seemed easy with pgLoader. The 
documentation about is was initially a bit sparse, but I could start a 
conversion with some commandline options. Unfortunately it got killed 
after 15 mins of import. After two more attempts it became clear, out of 
memory :(
Reducing the memory requirements was a build time option so I didn't 
want to go that way. Another solution was to convert only a few tables 
at a time. That required however a configuration file which has more 
options than the command line. Most of the examples I found on the web 
failed, since they lack the semi colon at the end if the configuration. 
So the parser barfs with some abracadabra, initially not giving a clue.
Fast forward, on 18:00 all but the messages table were converted. On 
22:00 the messages table was converted using 9 parts. Then I realized 
that I didn't have a php-pgsql driver installed. After doing so, I 
discovered that the Lazarus main site also showed the forum maintenance 
message ???
Those sites are running on two different virtual servers. How can the 
contents of an index.php of one site have influence on the index.php of 
another site. What did go wrong when I installed the driver ???
After an hour investigating I decided to enable the forum first and 
investigate the issue later.
Poof.... the forum results in a bunch of errors. What we didn't think of 
when switching backend, is that we use SMF (which is PostgreSQL capable) 
and TinyPortal (TP) to have the menus at the sides. And TP is full of 
MySQL only statements. Luckily there is someone who created all those 
missing functions for SMF and I created them on the database. After 
adjusting some TP php files (PostgreSQL requires a true boolean and not 
some integer <> 0), the forum started without errors. But it didn't show 
any boards. So there is still something wrong under the hood.
Meanwhile it was 24:00 and I decided to continue using MySQL and called 
it a day.
This morning I reverted my TP changes and put the MySQL database back 
online.

To be continued...


Marc