[fpc-pascal] Read lines into UnicodeString variable from UCS2 (UTF-16) encoded text file

[Tony Whyman]

Text files and the console should look the same from the programmer's 
point of view. The general principle is that you should be able to 
redirect stdin to a file and for there to be no difference as far the 
program is concerned when reading a file as opposed to reading console 
input.

If there is a problem here, it is that the code page of text files 
cannot be declared.  While under Linux, FPC seems to have an implicit 
assumption that the console and text files are both UTF8, under Windows 
it seems to assume UCS2 (and then transliterated to the current code 
page) for the console and (by default) UTF8 for text files. A built-in 
assumption of UCS2 for Windows text files would seem to be more consistent.

Under all OSs you should be able to set the actual code page for a text 
file and the code page that input should be transliterated to.

On 05/09/2019 10:49, Tomas Hajny wrote:
> On 2019-09-05 10:24, Tony Whyman wrote:
>> A few points:
>>
>> 1. IMHO: This is currently a Windows problem where the console buffer
>> is UCS2. Linux (and probably all other cases its UTF8 - to be
>> verified).
>  .
>  .
>
> No, the subject refers to text files, not to console. Obviously, 
> console output has its caveats, but that's something else - the 
> possibly added functionality of being able to read and write text 
> files with UTF-16 encoding using Read(Ln)/Write(Ln) does not imply 
> that you might be able to change the console to whatever codepage 
> value directly (this is not the case today either: you can perfectly 
> write UTF-8 to a text file under GO32v2 if using the fpWideString 
> manager, but the underlying communication is performed using the 
> console encoding, not the text file encoding, and translation is 
> needed on the fly).
>
> Tomas
> _______________________________________________
> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>