[fpc-devel] Macro Processing

[Joerg Schuelke]

Am Sun, 15 May 2011 00:30:38 +0200
schrieb Hans-Peter Diettrich <DrDiettrich1 at aol.com>:

> > I see this point and it is one reason for me to think very careful
> > about: "Is it possible to do it without touching the scanner?"
> > Is the rest interesting enough to make it worth a further thinking.
> > If not, trash it.  
> 
> The macro definition and expansion has to be changed.
> 
> Macro definition has to handle the formal parameter list, and this
> added code will execute only when a parameter list is present.
> 
> Macro expansion will have to deal with the actual parameter list,
> also only when such a macro is used. With proper separation of old
> and new macro handling (with/out parameters), the new code again only
> executes when new macros are used in the code.
> 

General structure in both cases.
This, an first attempt.

The explicit case:
There are a lot of pros doing it this way.

- it is not needed to touch the scanner
- the expansion is included with the same mechanism as file includes
- macro expansions which leads to compiler directives are possible
  (macro expanded file inclusion) and processed the normal way.
- it does not break the error message generation (the expansion is
  processed the normal way, but OK, not that good as with complete
  tokens stored). Maybe this is more a con, with workaround.
- the expansion process uses the sort of token we decide to use, not
  the scanner. I think this improves efficiency.

There are some cons too.
- the expander has to break the pp_tokens down to chars. This is
  ineffective, but: It may be encapsulated and sometimes later removed;
                    It does not effect the speed of the scanner in any
                    way if no macros are used.
               Maybe if the scanner someday is really working context
               sensitive (producing asm_tokens, pp_tokens, pas_tokens)
               this can be removed completely.


               sources
                |   |
                v   v
          -----------------
          |               |
          | File Manager  |->--
          |               |   |
          -----------------   |
            ^     ^    ^      |
            |     |    |      |        ---------------------
            |     |    |      |        |                   |
            |     |    |      -------->|     Scanner       |---> Token
            |     |    |               |                   |
            |     |    |               ---------------------
            |     |    |                        |
            |     |    |                        v
            |     |    |               --------------------- 
            |     |    |  finclude     |  Directive        |
            |     |    ---------------<|    Interpreter    |
            |     |                    |                   |
            |     |                    ---------------------
            |     |                             |
            |     |                             v
            |     |                    ---------------------    
            |     |       einclude     |      Macro        |    
            |     --------------------<|EXPANDER | DEFINE  |
            --------------------------<|         |         |    
                                       ---------------------    
                                           ^         |          
                                           |         |          
                                           |         |          
                                           |         v          
                                       ---------------------    
                                       |         Macro     |
                                       |  Symbol           |    
                                       |        Table      |
                                       ---------------------


The implicit case:
But I see a lot of problems doing it this way.

pros:
- some things can be done, which can not be done explicit. (But
  which??, is there really an example, where it comes to play?)
- it is easier to have error messages consistent, because the
  information is stored in the tokens you process.

cons:
- you touch the scanner much deeper, introducing new tokens.
- the tokens are really big, because of line and file positions and the
  whole stuff pascal tokens need this moment. (if you incorporate this
  information for consistent error messages)
- you have to process all tokens! There may be pp_operators in the
  input stream too. So somebody will use them too!! (But you say the
  efficiency argument may be better achieved the way restricting the
  pp_operators, I do not believe that (this time).
- You can not have (or not build ones) compiler directives inside the
  expansion!!
- You can not have includes in the expansion (Or, if you have one,
  they are expanded too. But nobody can see this!! You look at the
  include file and it is completely different worked up inside an macro
  expansion and outside included. I do not like that.

                                               ------------------    
                                               |       Macro    |
               sources                      ---| Symbol         |
                |   |                       |  |      Table     |
                v   v                       |  ------------------
          -----------------                 |          ^   ----------
          |               |           ------v------    |   | Token  |
          | File Manager  |----       | Macro     |----+-->|        |
          |               |   |       |  Expander |<-- |   |  Stack |
          -----------------   |       -------------  | |   ----------
                  ^           |                      | |           |
                  |           |  ----------------   ------------   |
                  |           |  |              |   | Macro    |   v
                  |           -->|  Scanner     |-->|  Procc   |----->
                  |              |              |   |          |
                  |              ----------------   ----------- 
                  |                    |                        
                  |                    v             
                  |              -------------------
                  | finclude     |  Directive      |
                  --------------<|    Interpreter  |
                                 |                 |
                                 -------------------


First really we should think about, is it needed to do implizit macro
expansion. I do not think about further implementation details, except
making it clear what would follow from that main decision.

-I do not see an example where it would be helpful to do it implizit.

-The kind of writing the expansion statement is not effected by this
 decision, even if you prefer writing mac_name(mac_parameterlist) you
 can do expand it explicit.

-You say restrict the pp_operators, but how? Without an concatenation
 operator you can not build up new token ones from older, but thats a
 main feature.

-Further ideas? Collection of pros and cons for both cases?

Regards
	Jörg