[Done] Must Order of loading cmem/heaptrc
lazarus / 64-FPC development

----------------------------------------------------------------------
BrunoK · 2019-07-17 11:18

FPC 3.0.6 (derivative of
//svn.freepascal.org/svn/fpc/tags/release_3_0_4a_macos_10_14).

Would someone have some idea where I should look/search in the
compiler where to patch and so to be able to have cmem/heaptrc to be
initialized BEFORE system.pp code :
  { Arguments }
  setup_arguments;

I want that because setup_arguments does use the default heap.inc
memory manager and conflicts afterward with cmem's reallocmem / getmem
/ freemem.

Maybe someone knowledgable could PM me a rough idea how to do that.

Hopefully thanks ...

----------------------------------------------------------------------
Thaddy · 2019-07-17 11:23

See the wiki. Don't mix the two!! Ever.
https://wiki.freepascal.org/heaptrc and
https://www.freepascal.org/docs-html/rtl/heaptrc/usage.html

There is a way mentioned as a note on how to select either cmem or
heaptrc:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
test;{$mode objfpc}uses // This will conditionally switch in a memory
manager// based on the presence of heaptrc {$if not
declared(UseHeapTrace)}cmem;{$ifend}begin {$if
declared(UseHeapTrace)}   writeln('Heaptrc is used.',' Heaptrc is
active? ', UseHeaptrace);{$else}   writeln('No trace of
heaptrc');{$ifend}end.
The ratio is that heaptrc is also a memory manager and you can't have
two without creating problems.

( A sick option is to rename cmem to heaptrc and rebuild, that would
insert cmem instead of heaptrc when compiled with -gh     )

----------------------------------------------------------------------
BrunoK · 2019-07-17 11:37

Quote from: Thaddy on July 17, 2019, 01:23:46 pmSee the wiki. Don't
mix the two!! Ever.
Yes, I know that bloody well, my new heaptrc WITH cmem nearly works
but bangs because HEAP.INC is USED BEFORE (in ' { Arguments }
setup_arguments;') having had the opportunity to install cmem.

Why use cmem ?

because it seems much faster in multihtreading applications.

----------------------------------------------------------------------
Thaddy · 2019-07-17 11:42

Quote from: BrunoK on July 17, 2019, 01:37:51 pmQuote from: Thaddy on
July 17, 2019, 01:23:46 pmSee the wiki. Don't mix the two!! Ever.
Yes, I know that bloody well, my new heaptrc WITH cmem nearly works
but bangs because HEAP.INC is USED BEFORE (in ' { Arguments }
setup_arguments;') having had the opportunity to install cmem.

Why use cmem ?

because it seems much faster in multihtreading applications.

See my last line above. The sick option part..
You are wrong in assuming it is faster. cmem does simply larger
allocations, which can easily cause problems in long running
applications. It looks faster, but not always and you should test.
Technically cmem is a bare and very basic OS heap based manager. The
default memory manager is more advanced than that.
With threads, you need scaling. cmem does not scale at all.

----------------------------------------------------------------------
Martin_fr · 2019-07-17 11:43

use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is
parsed

----------------------------------------------------------------------
Thaddy · 2019-07-17 11:49

Quote from: Martin_fr on July 17, 2019, 01:43:45 pmuse -Fa command
line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is
parsed
That does not work if heaptrc is selected, though. cmem will still be
loaded after heaptrc. Still the same problem about using two memory
managers.

----------------------------------------------------------------------
Martin_fr · 2019-07-17 12:38

Quote from: Thaddy on July 17, 2019, 01:49:34 pmThat does not work if
heaptrc is selected, though. cmem will still be loaded after heaptrc.
Still the same problem about using two memory managers.
He does not say he wants both at the same time. From what I read he
either wants:
- one of them (whichever he chooses) to be loaded
- a mem-mgr of his own (which is a merge of the 2) to be loaded.

So in any case just one mem-mgr.

At least that his how I read his posts

----------------------------------------------------------------------
BrunoK · 2019-07-17 12:45

Quote from: Martin_fr on July 17, 2019, 02:38:47 pm
So in any case just one mem-mgr.

At least that his how I read his posts

YES, exactly, full on target as usual !

----------------------------------------------------------------------
BrunoK · 2019-07-17 13:01

Quote from: Martin_fr on July 17, 2019, 01:43:45 pmuse -Fa command
line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is
parsed
Thanks for the tip.

Sadly, the compiler insists on having  system beeing initialized
BEFORE cmem than with heap.inc ...

Will dig further.

Quote from: Thaddy on July 17, 2019, 01:42:40 pm
See my last line above. The sick option part..
You are wrong in assuming it is faster. cmem does simply larger
allocations, which can easily cause problems in long running
applications. It looks faster, but not always and you should test.
see https://forum.lazarus.freepascal.org/index.php/topic,44590.0.html.
What took it to the top was using cmem.

I can  see it also in wp's mapviewer that leaks. A thing is sure is
that it leaks MUCH faster with cmem ...

QuoteTechnically cmem is a bare and very basic OS heap based manager.
The default memory manager is more advanced than that.
With threads, you need scaling. cmem does not scale at all.
cmem uses msvcrt in win10 and seems to be a tcmalloc allocator.
tcmalloc looks like a subtle piece of software. That heap allocator
has much evolved in the last 6 years and seems to be the mainstream
heap allocator whereas HEAP.INC doesn't seem to have evolved in the
last 10 years.
There are quite a few inconsistencies in heap.inc (those I took care
of).

As for memory utilization, the W10 task manager shows both fpc and
lazarus using less memory when build with cmem.

----------------------------------------------------------------------
Thaddy · 2019-07-17 14:17

Why not link in tcmalloc directly then?
Note msvcrt can have multiple versions depending on system and -
although I have a development build of Windows for 15+ years, I am not
aware recent  msvcrt versions have changed their malloc
implementation..
The original Google version of tcmalloc can be linked statically too.
https://github.com/gperftools/gperftools
It is also cross-platform, which msvcrt definitely is not!

Note that if Google themselves say it is the fastest they have seen,
that is most likely correct.

----------------------------------------------------------------------
Martin_fr · 2019-07-17 14:55

Quote from: BrunoK on July 17, 2019, 03:01:22 pmSadly, the compiler
insists on having  system beeing initialized BEFORE cmem than with
heap.inc ...

You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even
your unit is implicitly using system. Which means never mind how early
you load your unit, it will load system before itself.

----------------------------------------------------------------------
marcov · 2019-07-17 15:26

cmem is mostly meant for *nix, e.g. to use valgrind.

I don't think this is really tested functionality on Windows,
specially in combination with heaptrc.

----------------------------------------------------------------------
Thaddy · 2019-07-17 15:32

Quote from: marcov on July 17, 2019, 05:26:22 pmcmem is mostly meant
for *nix, e.g. to use valgrind.

I don't think this is really tested functionality on Windows,
specially in combination with heaptrc.

It has been used for many years It has the same interface on msvcrt.
It is basically a set of stdc library calls, so support by all - most
- C compilers. Another example is heapmm from KOL which uses the same
calls.
The combination, though, is out of the question. It is a matter of
either/or

----------------------------------------------------------------------
BrunoK · 2019-07-17 15:34

Quote from: Martin_fr on July 17, 2019, 04:55:26 pmYou may want to try
asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even
your unit is implicitly using system. Which means never mind how early
you load your unit, it will load system before itself.
Thanks for your help.

I think I fingered where I can try to do that.  Will try tomorrow.

Cheers, Bruno

Quote from: marcov on July 17, 2019, 05:26:22 pmI don't think this is
really tested functionality on Windows, specially in combination with
heaptrc.
I know it works because I run both FPC and Lazarus with cmem for the
last 3 months, not just 5 minutes sometimes, but for hours many days
in a week.

Of course it doesn't  work with the current version of heaptrc but it
nearly works with mine.

----------------------------------------------------------------------
Akira1364 · 2019-07-17 15:48

I don't think this is expected / supposed to work, as currently
implemented. heaptrc.pp is itself a
"memory manager unit" in the same way that cmem.pp is.

Both always replace the default memory manager with their own
implementations ("CMemoryManager" in the case of cmem, and
"TraceManager" in the case of heaptrc) and as such it is not possible
for them to co-exist.

If you want something that is still able to report memory usage the
way heaptrc does, but calls into some external malloc implementation,
the best way to go would be with a modified version of the heaptrc
unit that explicitly uses the external API for the actual allocation /
deallocation routines, which will generally return the same kind of
values allowing it to track them the same way.

As far as whether CMem specifically is generally useful on Windows
though, it definitely does massively improve the performance of multi-
threaded applications (on at least 64-bit, not sure about 32-bit.)

----------------------------------------------------------------------
PascalDragon · 2019-07-18 07:42

Quote from: Martin_fr on July 17, 2019, 04:55:26 pmQuote from: BrunoK
on July 17, 2019, 03:01:22 pmSadly, the compiler insists on having 
system beeing initialized BEFORE cmem than with heap.inc ...

You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even
your unit is implicitly using system. Which means never mind how early
you load your unit, it will load system before itself.

Correct.

If BrunoK wants to use the Windows heap manager directly for all
allocations, including those in early startup than he needs to
implement and set his heap manager directly in the System unit. This
involves at least rtl\win32\system.pp, rtl\win64\system.pp,
rtl\win\sysheap.inc and rtl\inc\heap.inc.

----------------------------------------------------------------------
BrunoK · 2019-07-18 08:52

Quote from: PascalDragon on July 18, 2019, 09:42:05 amIf BrunoK wants
to use the Windows heap manager directly for all allocations,
including those in early startup than he needs to implement and set
his heap manager directly in the System unit. This involves at least
rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and
rtl\inc\heap.inc.
The idea is to stay near current operation.
I have already added a -dCMEM compiler option in
TOption.interpret_option +                   
include(init_settings.globalswitches,cs_use_cmem); { new cs_use_cmem
in enum) +
in pmodules.pas loaddefaultunits
            { The correct heap manager should be setup BEFORE 'system'
initializes
             but it is not the case yet ~bk }
           { Units only required for main module }
           if (cs_use_cmem in current_settings.globalswitches) then
              AddUnit('cmem');

that works without any trouble, but there is still this question of
getting them in the correct order in fpc_InitializeUnits.

I'll find ...

My original post concerned only getting hints about that stuff, thats
all.

Anyway thank you all, even the usual grumpy knows all.

----------------------------------------------------------------------
marcov · 2019-07-18 09:27

But cmem is an unit, and as such implicitly requires system, which
means that system initializes before cmem?

----------------------------------------------------------------------
PascalDragon · 2019-07-19 07:37

Quote from: BrunoK on July 18, 2019, 10:52:03 amQuote from:
PascalDragon on July 18, 2019, 09:42:05 amIf BrunoK wants to use the
Windows heap manager directly for all allocations, including those in
early startup than he needs to implement and set his heap manager
directly in the System unit. This involves at least
rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and
rtl\inc\heap.inc.
The idea is to stay near current operation.
I have already added a -dCMEM compiler option in
TOption.interpret_option +                   
include(init_settings.globalswitches,cs_use_cmem); { new cs_use_cmem
in enum) +
in pmodules.pas loaddefaultunits
            { The correct heap manager should be setup BEFORE 'system'
initializes
             but it is not the case yet ~bk }
           { Units only required for main module }
           if (cs_use_cmem in current_settings.globalswitches) then
              AddUnit('cmem');

that works without any trouble, but there is still this question of
getting them in the correct order in fpc_InitializeUnits. Don't mess
with these things! The Windows RTL has a secondary entry point in
Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the
normal entry point and which might lead to allocations. It might not
do so now, but it's not guaranteed to be this way in the future as
well.

The only clean way is to integrate it directly into the System unit.

----------------------------------------------------------------------
BrunoK · 2019-07-19 09:36

Quote from: PascalDragon on July 19, 2019, 09:37:15 am
Don't mess with these things! The Windows RTL has a secondary entry
point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called
before the normal entry point and which might lead to allocations. It
might not do so now, but it's not guaranteed to be this way in the
future as well.

The only clean way is to integrate it directly into the System unit.
You are alright about maybe messing with these things. I  won't  come
complaining if it breaks and anyway it is my custom 3.0.6 version.

As an indication of what it has now, unretouched for the last 6
months, is for example :
TList = class(TFPList) refactoring. This and its descendant allocate
only one INSTANCE for each TFPList descendent. That approach works
without trouble (touch wood) both in cclasses and many List classes
(at least those I understand what they do). Apart having only one
object instance, as a by effect of that, one can inspect the list
instance during debugging and see FCount an FCapacity which are always
of interest for TFPList based instances.

That means that my TList is declared in lists.inc as
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
TListEnumerator = TFPListEnumerator;   TList = class(TFPList{
,IFPObserved})  private    procedure CopyMove (aList : TFPList);   
procedure MergeMove (aList : TFPList);    procedure DoCopy(ListA,
ListB : TFPList);    procedure DoSrcUnique(ListA, ListB : TFPList);  
procedure DoAnd(ListA, ListB : TFPList);    procedure
DoDestUnique(ListA, ListB : TFPList);    procedure DoOr(ListA, ListB :
TFPList);    procedure DoXOr(ListA, ListB : TFPList);  protected   
procedure Grow; virtual;    procedure Put(Index: Integer; Item:
Pointer);    procedure Notify(Ptr: Pointer; Action:
TListNotification); virtual;    procedure SetCount(NewCount:
Integer);  public    destructor Destroy; override;    Procedure
AddList(AList : TFPList);    function Add(Item: Pointer): Integer;   
procedure Clear; virtual;    procedure Delete(Index: Integer);   
function Extract(item: Pointer): Pointer;    function GetEnumerator:
TListEnumerator;    procedure Insert(Index: Integer; Item: Pointer); 
 procedure Assign (ListA: TFPList; AOperator: TListAssignOp=laCopy;
ListB: TFPList=nil);    function Remove(Item: Pointer): Integer;   
property Count: Integer read FCount { GetCount } write SetCount;   
property Items[Index: Integer]: Pointer read Get write Put; default; 
end;       or in  CCLasses
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   {
TFPObjectList }   TFPObjectList = class(TFPList)  private   
FFreeObjects : Boolean;  protected    function GetItem(Index:
Integer): TObject; {$ifdef CCLASSESINLINE}inline;{$endif}    procedure
SetItem(Index: Integer; AObject: TObject);  public    constructor
Create;    constructor Create(FreeObjects : Boolean);    destructor
Destroy; override;    procedure Clear;    procedure Delete(Index:
Integer);    function Extract(Item: TObject): TObject; {$ifdef
CCLASSESINLINE}inline;{$endif}    function Remove(AObject: TObject):
Integer;     function FindInstanceOf(AClass: TClass; AExact: Boolean;
AStartAt: Integer): Integer;    function First: TObject; {$ifdef
CCLASSESINLINE}inline;{$endif}    function Last: TObject; {$ifdef
CCLASSESINLINE}inline;{$endif}    procedure
Assign(Obj:TFPObjectList);    procedure
ConcatListCopy(Obj:TFPObjectList);    procedure
ForEachCall(proc2call:TObjectListCallback;arg:pointer); {$ifdef
CCLASSESINLINE}inline;{$endif}    procedure
ForEachCall(proc2call:TObjectListStaticCallback;arg:pointer); {$ifdef
CCLASSESINLINE}inline;{$endif}    property OwnsObjects: Boolean read
FFreeObjects write FFreeObjects;    property Items[Index: Integer]:
TObject read GetItem write SetItem; default;  end; No parasitic FList
co-class. This is object pascal !

I'm not much scared any more of trying things, I have fall back
backups.

Thanks for your help.

----------------------------------------------------------------------
BrunoK · 2019-08-13 14:42

Quote from: PascalDragon on July 19, 2019, 09:37:15 amDon't mess with
these things! The Windows RTL has a secondary entry point in
Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the
normal entry point and which might lead to allocations. It might not
do so now, but it's not guaranteed to be this way in the future as
well.

The only clean way is to integrate it directly into the System unit.
Quote from: marcov on July 18, 2019, 11:27:46 amBut cmem is an unit,
and as such implicitly requires system, which means that system
initializes before cmem?

Win32/64 FPC 3.0.4, the mac one with some other changes.

Done : got cmem + heaptrc loaded before system. As they are static
units they do not require any other any other units resources before
terminating heaptrc.
One bug found in heaptrc. Changed a bit Heap.inc (ReallocMem
inconsistent).

Improvements :
Added a global serial to the heaptrc records.
Hooked (Win only) TObject.NewInstance so one can get the classname of
the leaked structure.
Possibility to trail memory allocation via a personalized
setable/resetable callback.
Corrected an error related to AllocMem.

Here after :

Output of attached test program

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i 0;01609C64;
ansistring;getmem;size;100;serial;651;01609D34;ansistring;getmem;size;
100;serial;662;01609E04;ansistring;getmem;size;100;serial;673;01609D34
;ansistring;freemem;size;100;serial;664;01611C7C;unicodestring;getmem;
size;180;serial;685;01611C7C;unicodestring;freemem;size;180;serial;686
;01611C7C;unicodestring;getmem;size;180;serial;697;01611C7C;unicodestr
ing;freemem;size;180;serial;698;01609D34;ansistring;getmem;size;100;se
rial;709;01609D34;ansistring;freemem;size;100;serial;7010;01619C94;mem
ory;getmem;size;164;serial;7211;01619C94;memory;freemem;size;164;seria
l;7212;01609D34;ansistring;getmem;size;100;serial;7313;01609D34;ansist
ring;freemem;size;100;serial;7314;01609C64;ansistring;freemem;size;100
;serial;6515;01609C64;ansistring;getmem;size;100;serial;7416;01592AB4;
ansistring;getmem;size;36;serial;7517;01609C64;ansistring;freemem;size
;100;serial;7418;015A1DB4;ansistring;getmem;size;52;serial;7619;015994
1C;ansistring;getmem;size;84;serial;7720;015B1C14;memory;getmem;size;1
00012;serial;7821;015B1C14;memory;reallocmem;size;50012;serial;7822;01
5B1C14;memory;freemem;size;50012;serial;7823;015B1C04;TObject;getmem;s
ize;4;serial;7924;015B1C04;TObject;freemem;size;4;serial;7925;015B1C04
;TObject;getmem;size;4;serial;8026;01621CAC;memory;getmem;size;132;ser
ial;8127;01629CC4;memory;reallocmem;size;388;serial;8128;015F209C;TLis
t;getmem;size;20;serial;8229;015F209C;TList;freemem;size;20;serial;823
0;015F209C;TList;getmem;size;20;serial;8331;015A1E54;TComponent;getmem
;size;52;serial;8432;015F211C;TFPList;getmem;size;20;serial;8533;01592
B44;ansistring;getmem;size;36;serial;8634;015A1EF4;unicodestring;getme
m;size;52;serial;8735;015A1DB4;ansistring;freemem;size;52;serial;7636;
015F9DC4;ansistring;reallocmem;size;292;serial;7737;01592BD4;ansistrin
g;getmem;size;36;serial;8838;01592AB4;ansistring;freemem;size;36;seria
l;7539;015F9DC4;ansistring;freemem;size;292;serial;7740;015A1DB4;ansis
tring;getmem;size;52;serial;8941;0159941C;ansistring;getmem;size;84;se
rial;90Press ENTER to Quit Output of heaptrc with heap.inc
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i D:\fpc-laz-
asus\Lazarus\bk_test\HeapTrcTest\HookHeap\HookHeap\prjHookHeap.exe
prjHookHeap.exe HeapTrc START session : 13.08.2019 16:29:56Heap dump
by heaptrc unit. Memory manager type : [mmtHeap]Max trace depth : 1690
memory blocks allocated : 157152 / 17746481 memory blocks freed     :
162420 / 1758362 memory blocks used in System startup : 867 unfreed
memory blocks : -5354Call trace for block $01581CB8 size 48  serial=87
unicodestring  $004087FC  $0040856D  $00401BC1 line 89 of
prjHookHeap.lprCall trace for block $01572908 size 32  serial=86
ansistring  $004075E0  $00406B55  $00401B8F line 85 of
prjHookHeap.lprCall trace for block $015D1EE0 size 16  serial=85
TFPList > TObject  $00429C76  $00401B67 line 83 of prjHookHeap.lprCall
trace for block $01581C18 size 52  serial=84 TComponent > TPersistent
> TObject  $0042100F  $00401B56 line 82 of prjHookHeap.lprCall trace
for block $015D1E60 size 16  serial=83 TList > TFPList > TObject 
$00429C76  $00401B3A line 81 of prjHookHeap.lprCall trace for block
$01609A88 size 213  serial=81 <getmem>Call trace for block $015919C8
size 4  serial=80 TObject  $00429C76  $00401AF0 line 76 of
prjHookHeap.lprprjHookHeap.exe HeapTrc END THeaptrcObject. :
13.08.2019 16:29:56 Output of heaptrc with cmem
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i D:\fpc-laz-
asus\Lazarus\bk_test\HeapTrcTest\HookHeap\HookHeap\prjHookHeap.exe
prjHookHeap.exe HeapTrc START session : 13.08.2019 16:29:00Heap dump
by heaptrc unit. Memory manager type : [mmtCmem]Max trace depth : 1690
memory blocks allocated : 156624 / 16900081 memory blocks freed     :
156157 / 1675972 memory blocks used in System startup : 867 unfreed
memory blocks : 381Call trace for block $05E602B4 size 48  serial=87
unicodestring  $0040883C  $004085AD  $00401C01 line 89 of
prjHookHeap.lprCall trace for block $05E6020C size 32  serial=86
ansistring  $00407620  $00406B95  $00401BCF line 85 of
prjHookHeap.lprCall trace for block $05E60174 size 16  serial=85
TFPList > TObject  $00429E56  $00401BA7 line 83 of prjHookHeap.lprCall
trace for block $05E600BC size 52  serial=84 TComponent > TPersistent
> TObject  $004211EF  $00401B96 line 82 of prjHookHeap.lprCall trace
for block $05F84F5C size 16  serial=83 TList > TFPList > TObject 
$00429E56  $00401B7A line 81 of prjHookHeap.lprCall trace for block
$05F84DFC size 213  serial=81 <getmem>Call trace for block $05F84D74
size 4  serial=80 TObject  $00429E56  $00401B30 line 76 of
prjHookHeap.lprprjHookHeap.exe HeapTrc END THeaptrcObject. :
13.08.2019 16:29:00

----------------------------------------------------------------------
Thaddy · 2019-08-13 16:29

Does not compile on arm and crashes on win64 when -glh is used.

Can you add tests? I do not want to discourage you.

[edit]
on linux too.

It seems you have created something that only works for your special
case?
https://blog.codinghorror.com/the-works-on-my-machine-certification-
program/