[fpc-devel] Console problem in Lazarus trunk

[Martin Frb]

On 25/07/2020 20:40, J. Gareth Moreton wrote:
> Hi everyone,
>
> This is likely due to something peculiar in my configuration that I 
> haven't found yet, but I can't be sure.  I'm having problems in that 
> the latest Lazarus trunk won't create a console window when I'm 
> running the compiler source in the debugger.  I tried to jury-rig it 
> to create its own using the AllocConsole() Windows API function as a 
> temporary workaround, but it failed with an "access denied" error.  
> The little I could find on that error implies that the application 
> already has a console, but it's not displayed. Anyone got any ideas?

I did a bit of testing.

Using FpDebug => fine
Using GDB:
- IDE compiled with console => fine
- IDE compiled without console => app in debugger does not open console.

I tried other Process.Opitons in
components\lazdebuggergdbmi\cmdlinedebugger.pp
function TCmdLineDebugger.CreateDebugProcess(const AOptions: String): 
Boolean;
       FDbgProcess.Options:= [poUsePipes, poNoConsole, poStdErrToOutPut, 
poNewProcessGroup];

poNewConsole instead of poNoConsole made no diff.
Adding poRunIdle => no diff

       FDbgProcess.ShowWindow := swoNone;
swoNormal made no diff.

Besides those would affect gdb, but maybe they also are seen by Windows 
when launching the target.
- One possibility is that TProcess changed, and hands something 
different to windows.
- Another option is, that the change is in how the debugged app 
communicates to windows that it wants a console.
But I do not know.....

One thing I found
components\lazdebuggergdbmi\gdbmidebugger.pp
line 3545
   // for win32, turn off a new console otherwise breaking gdb will fail
   // ignore the error on other platforms
   FSuccess := ExecuteCommand('-gdb-set new-console off', R);

Make that "on" and it works (only tested if IDE has no console).

But it is there, because at the time of creating it, it caused issues. 
So changing it would likely cause issues on some older systems.
There are countless combinations of
- Windows version/bitness
- gdb version
- fpc version
- app and/or ide has console or not
And no idea how many of them may break. Maybe its just some Win95. Maybe 
its some still supported versions.

So for now, anyone wanting to test this can change that line.
If you do, please report your experience, and all versions in your setup.