[fpc-devel] TStringList.LoadFromFile and SavetoFile - file encoding support

[Marco van de Voort]

In our previous episode, Graeme Geldenhuys said:
> This supposed solution fails horribly in practice. What if the file
> was UTF-16 encoded?

Then you can't load it into a ansistring tstringlist, since ansistring is
one char per default.

> I believe Delphi 2009 extended the .LoadFromFile(...) and
> .SaveToFile(...) methods with an optional encoding parameter.

Delphi (and 2.4 in the future) are totally separate things, since they
actually have UTF-8 support. Currently 2.2.x only supports UTF-16
(widestring) and ansistring (in the native encoding), the rest is bolted on
at best.

> Could something like this be added to TStringList etc? 

In time, when D2009 compatibility is added yes. But not for a crur

> I guess we would also need some auto encoding detection in place.

Never for a basic RTL primitive. That is something for editor programs, not
library routines, iow this must be handled at a different level as the base
RTL.

> How do other text editors managed to auto detect the file encodings - to a
> degree of accuracy? 

Start decoding the various types and count errors would be my first guess.
Maybe heuristics are involved.

> Also, if the .LoadFromFile(...) and .SaveToFile(...) methods were
> extended, then we (GUI toolkit developers) could extend File Open and File
> Save dialogs like Qt has done.

> If the auto encoding detection didn't work, the user can use the combobox
> in the file open/save dialog to specify a encoding to use. Web Browsers
> have a similar feature when displaying HTML.

Autodetection Is the job of the toolkit developer, not of the base RTL.