[fpc-devel] [patch] pscanner: differentiate between EOL & Tab characters from general Whitespace

[Michael Van Canneyt]

On Sun, 22 Aug 2010, Graeme Geldenhuys wrote:

> On 22 August 2010 13:31, Michael Van Canneyt wrote:
>>
>> I cannot accept this patch as it is.
>>
>> It breaks the parser, more specifically NextToken.
>> The parser ignores whitespace. It should also ignore the new tokens.
>
> Oh,  I'll take another look. Let me explain what I'm doing - maybe it
> makes more sense then. I'm using the tokenizer/scanner and generating
> a list of all tokens. Then search for specific areas in the code using
> the tokens (uses clauses, begin..end pares of procedures or methods),
> then insert timing code, generate the units with newly included code,
> do a compile so I can get profiling/timing information. So far this
> works just fine for me, and already found some "hot spots" in my
> DocView code.
>
> After all this, I Googled profilers to see what everything they check
> for. I searching, I stumbled accross the fpprofiler which is exactly
> what I was trying to do (don't you just hate that - after you wrote
> some code). Funny enough, the fpprofile also only uses the pascal
> scanner.
>
> I tried using the parser as you suggested in a private email
> yesterday, but that made my profiler 2x slower at processing my source
> code (I used EpikTimer to get high precision timing results). The
> scanner seems good enough for what I want. But without differentiating
> between general whitespace and EOL (required) and Tab (optional), I
> can't regenerate my source code units.
>
> Anyway, I abandoned my profiling code and started working with
> fpprofiler (it was a bit further along than my own attempt), but it
> wasn't touched for over 2 years, so didn't compile. It was also
> riddled with memory leaks (sorry Darius Blaszyk). I fixed a lot of
> things including memory leaks, removed the custom parser to rather use
> the one included in FCL etc.. But the final step is tokenizing the EOL
> character correctly - which fcl-passrc doesn't do.

Why ? 
To the compiler there is no difference between EOL and whitespace.

If you want to generate it with the exact layout, then, yes.
But for profiling I don't see the need.

> BTW:
> fcl-passrc can't parse program units either, but I'm already working
> on a patch for that.

This is a known shortcoming.

>
>
>> Other than that, I doubt the usefulness of this patch. You are on the wrong
>> track with what you attempt to do.
>
> Based on the information above, and what I'm trying to do, do you
> still think I'm on the wrong track? Otherwise, what do you suggest? I
> think all I need to do, is add the LineEnding token to the parser to
> make sure I didn't break it (sorry, I never tested the parser before I
> generated the patch).

Well, I still fail to see why you need the tab and EOL.

Michael.