[fpc-pascal] Parse unicode scalar

[Nikolay Nikolov]

On 7/4/23 08:08, Nikolay Nikolov wrote:
>
> On 7/4/23 07:56, Hairy Pixels via fpc-pascal wrote:
>>
>>> On Jul 4, 2023, at 11:50 AM, Hairy Pixels <genericptr at gmail.com> wrote:
>>>
>>> You know you're right, with properly enclosed patterns you can 
>>> capture everything inside and it works. You won't know if you had 
>>> unicode in your string or not though but that depends on what's 
>>> being parsed and if you care or not (I'm doing a TOML parser).
>> Sorry I'm still curious even though it's not my current problem :)
>>
>> How can I make this program output the expected results:
>>
>>    w: widechar;
>>    a: array of widechar;
>> begin
>>     for w in 'abc🐻' do
>>       a += [w];
>>    // Outputs 7 instead of 4
>>    writeln(length(a));
>> end;
>>
>> The user doesn't know about unicode they just want to get an array of 
>> characters and not worry about all these little details. What can FPC 
>> do to solve this problem?
>
> Depends on what you need, but I suppose in this case you want to count 
> the number of extended grapheme clusters (a.k.a. "user perceived 
> characters" - how many character-like things are displayed on the 
> screen). You might be tempted to count the number of Unicode code 
> points, but that's not the same, due to the existence of combining 
> characters:
>
> https://en.wikipedia.org/wiki/Combining_character
>
> For extended grapheme clusters, there's an iterator in the 
> graphemebreakproperty unit. I implemented this for the Unicode KVM and 
> FreeVision. There it's needed for figuring out how many character 
> blocks in the console will be needed to display a certain string. For 
> the console or other GUIs that use fixed width fonts, there's also the 
> East Asian Width property as well - some characters (East Asian - 
> Chinese, Japanese, Korean) take double the space. So, to figure out 
> where to move the cursor, you need to take East Asian Width as well.

For console apps that use the Unicode KVM video unit, I've introduced 
two functions for determining the display width of a Unicode string in 
the video unit:

function ExtendedGraphemeClusterDisplayWidth(const EGC: UnicodeString): 
Integer;
{ Returns the number of display columns needed for the given extended 
grapheme cluster }

function StringDisplayWidth(const S: UnicodeString): Integer;
{ Returns the number of display columns needed for the given string }

Remember, the display width is different than the number of graphemes, 
due to East Asian double width characters.

And these work with UnicodeString, which is UTF-16, not UTF-8. But Free 
Pascal can convert between the two.

Nikolay