[fpc-pascal] Read lines into UnicodeString variable from UCS2 (UTF-16) encoded text file

[LacaK]

>
> Hi *,
>
> As promised, I discussed the idea of adding support for UTF-16 encoded 
> text files (and preferably UTF-32 as well while at it) to the RTL with 
> other core team members. Overall, I didn't come across anybody oposing 
> this idea, the only (logical) requirement is taking care of the 
> performance implications of this change, i.e. avoiding considerable 
> performance decrease in processing of 8-bit encoded files (actually, 
> this is one of reasons of my suggestion to add codepoint size 
> information to the text file record and use that instead of checking 
> individual values of the codepage variable to find out the codepoint 
> size implications every time working with the file - see below).
>
>
>> Yes, I believe that extending SetTextCodePage with supporting UTF-16
>> makes sense (with certain caveats like that calling it should be
>> performed before Rewrite in case of new files creation, or otherwise
>> the BOM mark will not be added to the beginning of the file). The
>> other question is what needs to happen within the text file record -
>> as mentioned in my other post, I'd prefer adding a new field
>> specifying the codepoint size rather than having to check for specific
>> codepage values in all code branches which would need to be created
>> for handling the difference.
>>
>> Moreover, the case of opening a file is somewhat trickier, because the
>> file may have the encoding specified within the file itself. Would we
>> add code for reading the first bytes every time Reset is called for a
>> text file not associated with another device (console) and set the
>> fields in the text file record (possibly overriding an explicit
>> setting from SetTextCodePage)? Personally, I'd do so, but others may
>> have a different opinion.
>  .
>  .
>
> After the discussion with some people from the core team, I suggest 
> the following:
>
> 1) New attribute for the codepoint size will be added to the text file 
> record and all the text file I/O needs to be checked and possibly 
> extended to with using this attribute instead of current implicit 
> expectation that the codepoint size is always 1 byte.
>
> 2) Support for UTF-16BE/LE and UTF-32BE/LE will be added to 
> SetTextCodePage, the new codepoint size attribute will be updated as 
> appropriate.
>
> 3) New function 'DetectUtfBom (var T: text): boolean' will be added. 
> This function may be called after the call to 'Reset (T: text)' to 
> check for existence of BOM at the beginning of the text file. If it is 
> found (Result=true), SetTextCodePage is invoked automatically from 
> DetectUtfBom with the codepage value corresponding to the found BOM 
> and encoding variant. If BOM is not found (Result=false), nothing 
> changes.
>
> 4) A new procedure 'SetUtfBom (var T: text; CodePage: word; BOM: 
> boolean)' will be added. This procedure may be called after the call 
> to Rewrite and allows writing BOM to the respective text file. 
> SetTextCodePage with the respective value will be called from SetUtfBom.
>
> Comments, anybody?
>
Thank you Tomas.

My comment per function names: use pattern SetText.../GetText...
So for inspiration:
- GetTextBOM or ReadBOM(var T: Text; SetCodePage:boolean=True):Word,
   (parameter SetCodePage should specify if SetTextCodePage will be 
called automatically if desired). Retunr value will be CP_NONE (no BOM) 
or CP_UTF8, CP_UTF16,...
- SetTextBOM or WriteBOM(var T: Text; CodePage: Word)
   (writtes BOM corresponding to given CodePage and will call 
SetTextCodePage). Boolean BOM parameter is IMO not needed as call to 
this function signals, that user wants write BOM else user will call 
SetTextCodePage() only

-Laco.