[fpc-devel] [patch] pscanner: differentiate between EOL & Tab characters from general Whitespace

[Graeme Geldenhuys]

On 22 August 2010 13:31, Michael Van Canneyt wrote:
>
> I cannot accept this patch as it is.
>
> It breaks the parser, more specifically NextToken.
> The parser ignores whitespace. It should also ignore the new tokens.

Oh,  I'll take another look. Let me explain what I'm doing - maybe it
makes more sense then. I'm using the tokenizer/scanner and generating
a list of all tokens. Then search for specific areas in the code using
the tokens (uses clauses, begin..end pares of procedures or methods),
then insert timing code, generate the units with newly included code,
do a compile so I can get profiling/timing information. So far this
works just fine for me, and already found some "hot spots" in my
DocView code.

After all this, I Googled profilers to see what everything they check
for. I searching, I stumbled accross the fpprofiler which is exactly
what I was trying to do (don't you just hate that - after you wrote
some code). Funny enough, the fpprofile also only uses the pascal
scanner.

I tried using the parser as you suggested in a private email
yesterday, but that made my profiler 2x slower at processing my source
code (I used EpikTimer to get high precision timing results). The
scanner seems good enough for what I want. But without differentiating
between general whitespace and EOL (required) and Tab (optional), I
can't regenerate my source code units.

Anyway, I abandoned my profiling code and started working with
fpprofiler (it was a bit further along than my own attempt), but it
wasn't touched for over 2 years, so didn't compile. It was also
riddled with memory leaks (sorry Darius Blaszyk). I fixed a lot of
things including memory leaks, removed the custom parser to rather use
the one included in FCL etc.. But the final step is tokenizing the EOL
character correctly - which fcl-passrc doesn't do.

BTW:
fcl-passrc can't parse program units either, but I'm already working
on a patch for that.


> Other than that, I doubt the usefulness of this patch. You are on the wrong
> track with what you attempt to do.

Based on the information above, and what I'm trying to do, do you
still think I'm on the wrong track? Otherwise, what do you suggest? I
think all I need to do, is add the LineEnding token to the parser to
make sure I didn't break it (sorry, I never tested the parser before I
generated the patch).


-- 
Regards,
  - Graeme -


_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/