[fpc-pascal] Is such memory statistics possible?

[Max Vlasov]

On Mon, Aug 15, 2011 at 1:03 PM, Marco van de Voort <marcov at stack.nl> wrote:

> In our previous episode, Max Vlasov said:
> > about collecting some statistics.
> >
> > The idea is similar to some disk utilities that collects and sort the
> sizes
> > of directories. You know when every folder on the computer is scanned and
> > all the resulting paths are sorted by the summed size. Such utilities
> > usually help to find space on the hard drive to free.
> >
> > With memory for every memory allocation we can call the path the
> addresses
> > of procedures in stack. So when a function funci() is called from the
> > function parentfunc() and getmem is called inside func() then we should
> add
> > this size to the corresponding entries both for func() and parentfunc().
> > ...
> > Do tools(units) like this already exist? If not, is developing such unit
> > technically possible/hard with the currently available debug information?
>
> This is basically what valgrind (or fulldebugmode of fastmm) does. But they
> do this by parsing the stack on each call to the memory manager, and then
> keep track of it.
>
> Note that all these techniques can be very, very slowing. E.g. I tried to
> debug
> the CHM support with valgrind, and I terminated the valgrind process after
> 5
> hours because it was not even half way to where the bug was.
>
> Without valgrind the program reached the point in 1-2 minutes.....
>
>
Based on this discussion and the following research I made an attempt to
implement something like this for fpc/lazarus.

The final result are a couple of units and a dialog that allows to see the
results in any time inside the program. You can see the real example in the
screenshot:

  http://www.maxerist.net/downloads/procmemstat_ss.png

The download link for pascal sources:

  http://www.maxerist.net/downloads/procmemstat.zip (5k)

The module traditionally installs its memory manager replacement procs and
collects statistics. To make things faster on this step it just detects and
saves addresses on the stack that falls into code segment range of the main
module. When a request for actual dialog is made then it parses all
allocated data and resolves the addresses to symbols with GetLineInfo proc.
This step is much longer, the dialog in the screenshot took 15 seconds to
appear (on my 1.7 GHz Celeron).

Currently the monitor is only win32-compatible, but the only
platform-specific code is about getting current stack range (I took it
according this information
http://en.wikipedia.org/wiki/Win32_Thread_Information_Block) and getting
code segment range that is made with toolhelp32 snapshots and virtualquery.
Finding a way to do the same on linux will possibly make it compatible with
linux also.


The usage:
- add uProcMemMon first in the lpr
- define -dPROCMEMMON in the project options-other-custom options
- add uProcMemDlg to unit (form) from where you want to show the statistics.
- call ProcMemShowDialog from any click handler. Sure you can call the
dialog as many times as you want looking what's wrong or right with
different states of you program
- if you don't want gui then don't use uProcMemDlg, just call PMMCollectStat
that will return TStringList with objects as sizes (unsorted)

Current limitations:
- As I already said, currently it's win32-only
- I assume it's currently not thread-safe because of global variable usage
- Several first lines of the dialog are not very useful since they're either
getmem related chain or main-related procedures, those are always on stack.
- The speed of collecting is ok, but although it uses linked list so with
many allocation it can drop. Resolving to symbols takes some time by default
(GetLineInfo is not very good for thousands of queries) so in these areas
some further optimization might possible.
- No dynamic loading support. This is due to the fact that the monitor uses
the address range of the main module.
- Initially I got numbers for winners that were bigger than total allocated
memory. This was because of exception blocks so from the point view of the
parser this getmem were called at once from multiply lines of the same
function. I fixed this by ignoring function duplicate while parsing the same
memory block. It seems it worked, but maybe there are cases when some new
trick will be necessary

If someone finds time to test this approach on real projects, that would be
great. As for usefulness the time will show, this time I'd like to know that
it at least provide sane results :)

Thanks

Max Vlasov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-pascal/attachments/20110819/158b031b/attachment.html>