[fpc-pascal] Proposed tidy-up of the FPC Manual section on Character Types and the FPC Wiki

[Tony Whyman]

There has been some heated discussion on the Lazarus lists on the 
subject to character encodings etc. This has exposed several issues with 
the FPC Manual that I wanted to record.

1. The char type

The manual says: "A Char is exactly 1 byte in size, and contains one 
ASCII character. "

This was probably true when Pascal was first defined, but char is often 
now used for any on-byte character set e.g. ISO 8859-1. Replace ASCII 
with ANSI.

2. WideChar

The Manual says: "A WideChar is exactly 2 bytes in size, and contains 
one UNICODE character in UTF-16 encoding. "

This seems to be wrong as UTF-16 is not limited to code points defined 
using a single 16-bit code unit, but also permits code points comprising 
two 16-bit code units. The definition should be updated to indicate that 
a WideChar was really created for the obsolescent UCS-2 and is limited 
to a UTF-16 subset (Unicode characters that can be expressed as a single 
16-bit code unit).

Proposed replacement text: "A WideChar is exactly 2 bytes in size, and 
contains one UNICODE character in UCS-2 encoding or UTF-16 encoding 
limited to the Basic Multilingual Plane. Note that Unicode Characters 
represented by a UTF-16 code points that require two 16-bit code units 
cannot be contained in a single WideChar variable."

3. UnicodeStrings

The Manual says: "For multi-byte string types, the basic character has a 
size of at least 2."

Proposed improvement:

"Multi-byte string types are used to represent Unicode characters 
encoded as code points requiring two or four bytes".

As with UTF8String, the following caveat should also be added:

"Since a unicode character may require two or four bytes to be 
represented in the UTF-16 encoding, there are 2 points to take care of 
when using UnicodeString/WideString:

1. The character index – which retrieves a WideChar at a certain 
position – must be used with care: the expression S[i] will not 
necessarily be a valid character for a string S of type 
UnicodeString/WideString.

2. The  length of the string is not necessarily equal to the number of 
elements in the array. The standard function length cannot be used to 
get the character length of the string, it will always return the array 
length.

------------------------------------------------------

Wiki Page on "Character and string Type"

1. This needs to start with a Health Warning on the use of the word 
Unicode. Proposed Text (borrowing from Wikipedia):

"Free Pascal supports several character and string types. They range 
from single ANSI characters to unicode strings and also include pointer 
types. Differences also apply to encodings and reference counting. ANSI 
is typically used to refer to single byte character encodings - although 
FPC also uses AnsiStrings to hold Unicode UTF-8 encoded strings.

Unicode is a computing industry standard for the consistent encoding, 
representation, and handling of text expressed in most of the world's 
writing systems. Developed in conjunction with the Universal Coded 
Character Set (UCS) standard and published as The Unicode Standard, the 
latest version of Unicode contains a repertoire of 136,755 characters 
covering 139 modern and historic scripts, as well as multiple symbol sets.

Unicode can be implemented by different character encodings. The Unicode 
standard defines UTF-8, UTF-16, and UTF-32, and several other encodings 
are in use. The most commonly used encodings are UTF-8, UTF-16 and 
UCS-2, a precursor of UTF-16.

The original idea behind Unicode was to replace the typical 
256-character encodings requiring 1 byte per character with an encoding 
using 2^16 = 65,536 values requiring 2 bytes per character.The early 
2-byte encoding was usually called "Unicode", but is now called "UCS-2". 
UCS-2 differs from UTF-16 by being a constant length encoding and only 
capable of encoding characters of Basic Multilingual Plane (BMP), it is 
supported by many programs. However, "UCS-2 should now be considered 
obsolete. It no longer refers to an encoding form in either 10646 or the 
Unicode Standard.

Unfortunately, the term Unicode, in common usage, is still often used to 
refer to the UCS-2 two byte encoding and this can give rise to much 
confusion e.g. when Unicode is used when referring to the UTF-8 encoding."

2. The text on WideChar is too terse and needs to be expanded. Proposed 
text:

"A variable of type WideChar, also referred to as UnicodeChar (which 
derives from the archaic use of Unicode to mean UCS-2), is exactly 2 
bytes in size, and usually contains either:

(a) a single UCS-2 code point, or

(b) a single UTF-16 code unit.

In case (b), this is sufficient for Unicode Characters that have a 
UTF-16 code point that comprises a single 16-bit code unit i.e. 
characters in the Basic Multilingual Plane. However, all other UTF-16 
characters have a UTF-16 code point that comprises a two 16-bit code 
units. FPC provides no specific support for such characters which 
require, e.g. a WideChar pair to encoded them."

Note: that the byte order used to store a WideChar can vary between 
platforms.

2. PChar

This should be identified as a synonym for PAnsiChar in FPC, It can also 
be as a C style pointer to any AnsiString including UTF-8.

It may also be useful to add a note that in later versions of Delphi, 
PChar is a synonym for PWideChar.