[fpc-pascal] Issue with CSVDocument

[James Richters]

Thanks for the link and the work around idea.

I really don't know what settings I had for the CSV file, I just clicked Save-As and changed it to CSV... I didn't even know there was an ASCII option.  The newer versions of Excel make you jump though hoops just to see where on your hard drive you are saving the thing.. never mind what the setting are...  And I guess that's the point... no one knows anything about BOM and that there are some CSV files that have it and Some CSV files that don't and setting DetectBOM := True; should not cause a fatal crash if your user tries to use an ASCII CSV file, and the user should not get garbage if he uses a UTF-8 CSV File... the real problem is that programmers may not know this, and may not have an issue at all.. maybe they happen to set DetectBOM := True; and never think to test it with an ASCII CSV file.. why would they?  But later on some other user keeps getting crashes and no one has any idea why it's crashing... the files look the same in a text editor after all.   So I Really think that DetectBOM should also detect ASCII and no BOM..  and then DetectBOM should be set to TRUE by default... then the entire CSVDocument system always works for everyone and we never will hear about BOM issues ever again...   

I really like the way CSVDocument works, and I'll stick with it now that I have it figured out, it's just a shame that all this BOM stuff made it much more complicated than it needed to be.

Anyway, I fixed the issue as suggested like this:

Uses sysutils,CSVDocument;

Var
   LoopRow,LoopCol:DWord;
   CSVData        : TCSVDocument;
   CheckFile      : File of Byte;
   CheckByte      : Array[1..4] of Byte;

Begin
   If ParamCount>=1 Then
      Begin
         Writeln('Input From File: ',Paramstr(1));
         If FileExists(ParamStr(1)) Then
            Begin
               CSVData:=TCSVDocument.Create;
               Checkbyte[1]:=$55;
               Checkbyte[2]:=$55;
               Checkbyte[3]:=$55;
               Checkbyte[4]:=$55;
               Assign(CheckFile,ParamStr(1));
               Reset(CheckFile);
               If Not(Eof(CheckFile)) Then
                  Begin
                     Read(Checkfile,Checkbyte[1]);
                     Writeln('$'+HexStr(Checkbyte[1],2));
                     If Not(Eof(CheckFile)) Then
                        Begin
                           Read(Checkfile,Checkbyte[2]);
                           Writeln('$'+HexStr(Checkbyte[2],2));
                           If Not(Eof(CheckFile)) Then
                              Begin
                                 Read(Checkfile,Checkbyte[3]);
                                 Writeln('$'+HexStr(Checkbyte[3],2));
                                 If Not(Eof(CheckFile)) Then
                                    Begin
                                       Read(Checkfile,Checkbyte[4]);
                                       Writeln('$'+HexStr(Checkbyte[4],2));
                                    End;
                              End;
                        End;
                  End;
               If (CheckByte[1]=$EF) AND (CheckByte[2]=$BB) AND (CheckByte[3]=$BF)  Then
                  Begin
                     Writeln('UTF-8 Byte Order Mark Detected');
                     CSVData.DetectBOM := True;
                  End;
               If ((CheckByte[1]=$FE) AND (CheckByte[2]=$FF)) OR ((CheckByte[1]=$FF) AND (CheckByte[2]=$FE)) Then
                  Begin
                     Writeln('UTF-16 Byte Order Mark Detected');
                     CSVData.DetectBOM := True;
                  End;
               If ((CheckByte[1]=$00) AND (CheckByte[2]=$00) AND (CheckByte[3]=$FE) AND (CheckByte[4]=$FF)) OR ((CheckByte[1]=$FF) AND (CheckByte[2]=$FE) AND (CheckByte[3]=$00) AND (CheckByte[4]=$00)) Then
                  Begin
                     Writeln('UTF-32 Byte Order Mark Detected');
                     CSVData.DetectBOM := True;
                  End;
               Close(CheckFile);
               CSVData.LoadFromFile(ParamStr(1));
               Writeln('Read In: ',CSVData.RowCount,' Rows and ',CSVData.MaxColCount,' Columns');
               Writeln('Press Enter to Process');
               Readln;
               If CSVData.RowCount >0 Then
                  Begin
                     For LoopRow := 0 to CSVData.RowCount-1 Do
                        Begin
                           Write('| ');
                           For LoopCol := 0 to CSVData.ColCount[LoopRow]-1 Do
                              Begin
                                 Write(CSVData.Cells[LoopCol,LoopRow]+' | ');
                              End;
                           Writeln;
                        End;
                  End;
               CSVData.Free;
            End;
      End;
End.

The part of my program that checks for the BOM is bigger than the actual CSV read and write.  Please let me know if the crash issue with no BOM gets fixed so I can delete all this stuff that checks for it.

James