[fpc-devel] TEncoding.Default and default encoding for TStrings.LoadFrom*()

[Ondrej Pokorny]

Hello,

a lot of people have a problem with the TStrings.LoadFrom*() changes 
when TEncoding support was added.

Currently, the no-encoding overloads of TStrings.LoadFrom*() and 
TStrings.SaveTo*() use the TEncoding.Default, which is WIN-ANSI and not 
DefaultSystemCodePage.

Before the changes, the encoding was just ignored and the file was read 
without any codepage support (thus in fact in DefaultSystemCodePage).

A lot of people have a problem with it because in Lazarus they used to 
read UTF-8 text with TStrings.LoadFrom*(). Now the text is broken 
because TStrings.LoadFrom*() load the text in WIN-ANSI.

The Lazarus team came with some kind of solution: they changed the FPC 
internal function widestringmanager.GetStandardCodePageProc to return 
UTF-8 and thus TEncoding.Default and TEncoding.ANSI to be UTF-8. This, 
as a result, broke every ANSI code that needs real WIN-ANSI codepage 
(like the ODBC: https://bugs.freepascal.org/view.php?id=36481 ). (And it 
introduced more Delphi & other Lazarus incompatibilities just for the 
sake of keeping TStrings.LoadFrom*() legacy-Lazarus compatible.)
For me the Lazarus solution is not acceptable.

But other Lazarus team members seem not to be happy with the encoding 
breaking change in FPC TStrings.LoadFrom*() and SaveTo*() and ask for a 
solution.

---

I suggest a compromise (steps):

1.) Keep TEncoding.ANSI always WIN-ANSI and Delphi-compatible. (Don't 
change it to DefaultSystemCodePage in Lazarus.)
2.) Change TEncoding.Default value to current TEncoding.SystemEncoding. 
I.e. TEncoding.Default would correspond to DefaultSystemCodePage and 
CP_ACP. Yes, this will be Delphi-incompatible - but CP_ACP is 
Delphi-incompatible as well (!) - so the incompatibilities are 
consequent here.
3.) Delete TEncoding.SystemEncoding because it is an FPC-only construct, 
it is not needed anymore (because it will become TEncoding.Default) and 
it has not been released in any stable version.

The above steps perfectly correlate with the CP_ACP value corresponding 
to DefaultSystemEncoding. As I wrote before, the CP_ACP is not 
Delphi-compatible either. See 
https://wiki.freepascal.org/FPC_Unicode_support

It makes perfect sense to keep TEncoding consequent with CP_ACP and make 
TEncoding.Default to correspond with DefaultSystemEncoding.

It is also natural: I would expect TEncoding.Default to correspond with 
the default (Ansi)String encoding, which is DefaultSystemEncoding.

Ondrej