Linker does not exclude unused procs if FPC option -g used (64-bit only!) lazarus / 1-Linux ---------------------------------------------------------------------- Hartmut · 2020-05-04 15:26 I hope I choosed a convenient subforum - if not, please let me know. Versions:  - Lazarus 2.0.6 with FPC 3.0.4 (on Linux installed by fpcupdeluxe with 32-bit cross-compile)  - OS: Linux Ubuntu 18.04 (64-bit) & Windows 7 (32-bit) Problem1 + Question1: I saw that when I compile a program on Windows, that unused procedures and functions were excluded from the executable, while when compiling the same program on Linux, unused procs were not excluded from the executable. To test this, I reduced my bigger program to this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit unit1; {$mode objfpc}{$H+} interface procedure test1;procedure test2; implementation procedure test1;   begin   writeln('This_is_a_test1');   end; procedure test2;   begin   writeln('This_is_a_test2');   end; end.  program demo1; {$mode objfpc}{$H+} uses unit1; procedure test3;   begin   writeln('This_is_a_test3');   end; begin{$IFDEF CPU32}   writeln('32-bit');{$ELSE}   writeln('64-bit');{$ENDIF}test1;end. For this test I used FPC directly without Lazarus:  - on Windows => unused procs were excluded (only 32 bit tested)  - on Linux => unused procs were not excluded (32-bit and 64-bit tested) I tested this by searching the executable for "This_is_a_test". One match = "This_is_a_test1" means, unused procs were excluded. 3 matches = "This_is_a_test1" .. "This_is_a_test3" mean, they were not excluded. After spending a lot of time I found out, that this is called "smartlinking" and that compiler options -CX and -XX are involved and that this part in fpc.cfg #ifdef cpui8086 -CX -XX #endif on Windows obviously worked, while on Linux this did not work. After adding -CX and -XX in fpc.cfg for Linux => unused procs were excluded (32-bit and 64-bit). Problem1 is now solved by this, but I want to ask: Question1: Why does "#ifdef cpui8086" work on Windows and not on Linux (on the same computer, with the same CPU)? Does anyone know a documentation about the definition of "cpui8086" (and the other possible conditionals) in fpc.cfg? Problem2 + Question2: First I thought, my problem was solved. But when I compiled my program via Lazarus, I saw:  - on Windows => unused procs were excluded (only 32 bit tested)  - on Linux (compile as 32-bit) => unused procs were excluded  - on Linux (compile as 64-bit) => unused procs were not excluded After spending again more time I found out, that Lazarus sets compiler options -g and -gl which mean:   -g  = Generate debug information (default format for target)   -gl = Use line info unit (show more info with backtraces) And when you compile a 32-bit program with -g and/or -gl => unused procs are excluded. And when you compile a 64-bit program with -g and/or -gl => unused procs are not excluded. Question2: Why are unused procs not excluded with compiler options -g and/or -gl on 64-bit, while they are excluded on 32-bit? Is this "impossible"? What must I do, that unused procs are excluded with compiler options -g and/or -gl on 64-bit too? (I'm not familiar with the linker and still a beginner on Linux). For me it is important, that if a program crashes, that I see as much debug infos as possible. Therefore I will never compile "release versions" with reduced debug infos. But I want to exclude all unused procs (on 32-bit and on 64-bit). Is that possible and how? I attached my little demo program and the used fpc.cfg. Thanks a lot in advance for your help. ---------------------------------------------------------------------- lucamar · 2020-05-04 16:38 Quote from: Hartmut on May 04, 2020, 05:26:19 pmQuestion1: Why does "#ifdef cpui8086" work on Windows and not on Linux (on the same computer, with the same CPU)? Does anyone know a documentation about the definition of "cpui8086" (and the other possible conditionals) in fpc.cfg? That section should not be pulled for anything other than the MSDOS target since it references compiling for x86 16 bit. My guess? The smartlinking options for Windows are coming from elsewhere. Conditionals documentation: Compiler defines during compilation ---------------------------------------------------------------------- TRon · 2020-05-04 16:50 Quote from: Hartmut on May 04, 2020, 05:26:19 pmWhy does "#ifdef cpui8086" work on Windows and not on Linux (on the same computer, with the same CPU)? Does anyone know a documentation about the definition of "cpui8086" (and the other possible conditionals) in fpc.cfg? I am unable to answer the first part of your question simply because afaik there is no documentation on that particular define (other then the sources of the compiler itself). For the second part: https://www.freepascal.org/docs-html/prog/progap7.html https://wiki.freepascal.org/Platform_defines Not all of them are documented (yet) or are upcoming defines, for example the z80 processor. Beside those defines, also programs (source-code) is allowed to contain defines ($DEFINE HELLO), and you can pass defines manually as an option to the compiler (-d). Especially Lazarus makes heavy use of those defines in order to make things compile as they should. If you invoke the fpc compiler from the commandline and pass -vd or -va teh compiler then you are able to see what names the compiler actually defined for compilation (or which of those are passed along from commandline or fpc configuration file. Depending on the name of the define, a single change could trigger several other defines to change (being defined or being undefined). Sorry, i have no answer to your smartlinking question. ---------------------------------------------------------------------- Hartmut · 2020-05-04 17:46 Hello lucamar again, thank you for your answer and the documentation link. You are right, "cpui8086" should be for 16-bit targets only. On Windows, I started fpc.exe with "-vc" to show the valid conditionals and "cpui8086" was not listed. So it is a miracle to me, where/why smartlinking is enabled on Windows, because if I start FPC directly (without Lazarus), where could this option come from when not from fpc.cfg? Hello TRon, thank you for the documentation links. But I don't understand what your infos about defines (-d) have to do with my linker problem only for 64-bit programs (for 32-bit it works). I started FPC on Linux with -va or -vd and got a lot of output, but don't know what to find there. The "problem" occurs (unused procs are not excluded) with this simple command line (64-bit):    /opt/lazarus_206/fpc/bin/x86_64-linux/fpc.sh demo1.pas -g If I switch to 32-bit, it's ok (unused procs are excluded):     /opt/lazarus_206/fpc/bin/x86_64-linux/fpc.sh demo1.pas -g -Pi386 So no defines (-d) are involved. ---------------------------------------------------------------------- lucamar · 2020-05-04 18:01 What happens if you explicitely do: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i /opt/lazarus_206/fpc/bin/x86_64-linux/fpc.sh -B -CX -XX -g demo1.pasand Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i /opt/lazarus_206/fpc/bin/x86_64-linux/fpc.sh -B -CX -XX -g -Pi386 demo1.pas But ... I'm not very sure but IIRC some targets don't permit smart- linking along with debugging info, so it's one or the other for them. What I don't remember is if *nix x86_64 is one of those ---------------------------------------------------------------------- TRon · 2020-05-04 18:17 Quote from: Hartmut on May 04, 2020, 07:46:03 pmHello TRon, thank you for the documentation links. You;re welcome, even though lucamar was first because i'm such a slow typer :-) QuoteBut I don't understand what your infos about defines (-d) have to do with my linker problem only for 64-bit programs (for 32-bit it works). Uhm... is my eyeside perhaps bad ? Quote from: Hartmut on May 04, 2020, 05:26:19 pmWhy does "#ifdef cpui8086" work on Windows and not on Linux (on the same computer, with the same CPU)? Does anyone know a documentation about the definition of "cpui8086" (and the other possible conditionals) in fpc.cfg? Nah... i must be imagining things ;-) QuoteI started FPC on Linux with -va or -vd and got a lot of output, but don't know what to find there. It is about the conditionals. They get initialised at the beginning of that output, and in case the commandline options or source-code affecting them is in effect, you can see where exactly that happens in the code/handling by the compiler (and or if such define influences other defines). I know it can sometimes be a long read, but if you wish to know exactly what is defined where and why, then you can find the answer there in the output. QuoteSo no defines (-d) are involved. It is more a general remark on how you can approach something like this. Do you (always) know what the cat dragged in ? :-) I for sure not (and i am to lazy to delve into each and every component/units source-code in order to see what goes around in there. @lucamar: Quote from: lucamar on May 04, 2020, 08:01:40 pmBut ... I'm not very sure but IIRC some targets don't permit smart-linking along with debugging info, so it's one or the other for them. What I don't remember is if *nix x86_64 is one of those At least the rtl should have been compiled with smartlinking turned on, else there is no no joy to begin with (unless that has changed somwhere along the line) ---------------------------------------------------------------------- lucamar · 2020-05-04 18:53 Quote from: TRon on May 04, 2020, 08:17:51 pmAt least the rtl should have been compiled with smartlinking turned on, else there is no no joy to begin with (unless that has changed somwhere along the line) Yes, but the release RTL, etc. is compiled without debuging info. That's the crux of the matter: smart-linking and also including debug info. ---------------------------------------------------------------------- marcov · 2020-05-04 19:15 Quote from: TRon on May 04, 2020, 06:50:05 pmQuote from: Hartmut on May 04, 2020, 05:26:19 pmWhy does "#ifdef cpui8086" work on Windows and not on Linux (on the same computer, with the same CPU)? Does anyone know a documentation about the definition of "cpui8086" (and the other possible conditionals) in fpc.cfg? I am unable to answer the first part of your question simply because afaik there is no documentation on that particular define (other then the sources of the compiler itself). For the second part: https://www.freepascal.org/docs-html/prog/progap7.html What is the last line of table G.2 in that link? I'll quote it for you: QuoteCPUI8086    indicates a 16-bit x86 target (i8086) ---------------------------------------------------------------------- TRon · 2020-05-04 19:33 @marcov: Uhm  Can i stil plead bad eyesight ?  My apologies to all concerned. thx for the correction. ---------------------------------------------------------------------- PascalDragon · 2020-05-05 07:20 Quote from: Hartmut on May 04, 2020, 05:26:19 pmQuestion1: Why does "#ifdef cpui8086" work on Windows and not on Linux (on the same computer, with the same CPU)? Does anyone know a documentation about the definition of "cpui8086" (and the other possible conditionals) in fpc.cfg? As others already wrote that is to check for the i8086 processor target. Inside the configuration file you can check for any define that the compiler either defines internally (e.g. the processor or the target OS) or that was passed on the command line using -d (e.g. RELEASE and DEBUG are two examples). Quote from: Hartmut on May 04, 2020, 05:26:19 pmProblem2 + Question2: First I thought, my problem was solved. But when I compiled my program via Lazarus, I saw:  - on Windows => unused procs were excluded (only 32 bit tested)  - on Linux (compile as 32-bit) => unused procs were excluded  - on Linux (compile as 64-bit) => unused procs were not excluded After spending again more time I found out, that Lazarus sets compiler options -g and -gl which mean:   -g  = Generate debug information (default format for target)   -gl = Use line info unit (show more info with backtraces) And when you compile a 32-bit program with -g and/or -gl => unused procs are excluded. And when you compile a 64-bit program with -g and/or -gl => unused procs are not excluded. Question2: Why are unused procs not excluded with compiler options -g and/or -gl on 64-bit, while they are excluded on 32-bit? Is this "impossible"? What must I do, that unused procs are excluded with compiler options -g and/or -gl on 64-bit too? (I'm not familiar with the linker and still a beginner on Linux). There are two points involved: first Windows uses (by default) the internal linker, while Linux always uses ld. Also 32-bit uses a different debug format (Stabs) than 64-bit (DWARF). Would you please test the following points: - Windows 32-bit with -glw (which will select DWARF) - Windows 32-bit with -Xe (which will use ld instead of the internal linker) - Windows 64-bit (both with and without -Xe) - Linux 32-bit with -glw - enable -CX -XX ---------------------------------------------------------------------- Hartmut · 2020-05-05 09:26 Quote from: lucamar on May 04, 2020, 08:01:40 pmWhat happens if you explicitely do: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i /opt/lazarus_206/fpc/bin/x86_64-linux/fpc.sh -B -CX -XX -g demo1.pasand Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i /opt/lazarus_206/fpc/bin/x86_64-linux/fpc.sh -B -CX -XX -g -Pi386 demo1.pasI tried both, but unfortunately it made no difference: in 32-bit unused procs are excluded, in 64-bit they are not. But now I noticed a compiler message (only in 64-bit), which I before had overlooked: "Note: DWARF debug information cannot be used with smart linking on this target, switching to static linking". @TRon: we have misunderstood each other. You wrote "For the second part:" and then talked about defines. I had thought, you mean my "Question2". Now I know, what you meant. Sorry for confusion. Quote from: PascalDragon on May 05, 2020, 09:20:46 amInside the configuration file you can check for any define that the compiler either defines internally (e.g. the processor or the target OS) or that was passed on the command line using -d (e.g. RELEASE and DEBUG are two examples). ... There are two points involved: first Windows uses (by default) the internal linker, while Linux always uses ld. Also 32-bit uses a different debug format (Stabs) than 64-bit (DWARF). Thank you PascalDragon for that valuable additional information. I did not know this. Quote from: PascalDragon on May 05, 2020, 09:20:46 amWould you please test the following points: - Windows 32-bit with -glw (which will select DWARF) - Windows 32-bit with -Xe (which will use ld instead of the internal linker) - Windows 64-bit (both with and without -Xe) - Linux 32-bit with -glw - enable -CX -XX Here are the results: Windows 32-bit with -glw (which will select DWARF) \fpc.exe -B -CX -XX -glw demo1.pas => compiler writes "Note: DWARF debug information cannot be used with smart linking on this target, switching to static linking", but unused procs are excluded! Windows 32-bit with -Xe (which will use ld instead of the internal linker) \fpc.exe -B -CX -XX -Xe demo1.pas => unused procs are not excluded Windows 64-bit (both with and without -Xe) I only have Win7 32-bit and have no 64-bit cross-compiler Linux 32-bit with -glw /fpc.sh -B -CX -XX -glw -Pi386 demo1.pas => compiler writes "Note: DWARF debug information cannot be used with smart linking on this target, switching to static linking" and unused procs are not excluded So from my understanding:  - Windows 32-bit with internal linker excludes unused procs even with DWARF  - Windows 32-bit with linker "ld" does not exclude unused procs  - Linux 32-bit with DWARF does not exclude unused procs Does that mean, that smartlinking with debug infos enabled is impossible on Linux 64-bit? ---------------------------------------------------------------------- PascalDragon · 2020-05-05 09:42 Quote from: Hartmut on May 05, 2020, 11:26:17 amSo from my understanding:  - Windows 32-bit with internal linker excludes unused procs even with DWARF  - Windows 32-bit with linker "ld" does not exclude unused procs  - Linux 32-bit with DWARF does not exclude unused procs Does that mean, that smartlinking with debug infos enabled is impossible on Linux 64-bit? Possibly, yes. I'll have to take a look myself, maybe I'll find the time for this today evening, but I don't know yet. ---------------------------------------------------------------------- Hartmut · 2020-05-05 11:21 Quote from: PascalDragon on May 05, 2020, 11:42:55 amPossibly, yes. I'll have to take a look myself, maybe I'll find the time for this today evening, but I don't know yet. Take as many time as you want. Thanks a lot for your help. ---------------------------------------------------------------------- TRon · 2020-05-07 07:24 Quote from: Hartmut on May 05, 2020, 11:26:17 am@TRon: we have misunderstood each other. You wrote "For the second part:" and then talked about defines. I had thought, you mean my "Question2". Now I know, what you meant. Sorry for confusion. I apologise in return for not having articulated that enough, causing the misunderstanding. ---------------------------------------------------------------------- Hartmut · 2020-05-16 06:27 Quote from: PascalDragon on May 05, 2020, 11:42:55 amQuote from: Hartmut on May 05, 2020, 11:26:17 amSo from my understanding:  - Windows 32-bit with internal linker excludes unused procs even with DWARF  - Windows 32-bit with linker "ld" does not exclude unused procs  - Linux 32-bit with DWARF does not exclude unused procs Does that mean, that smartlinking with debug infos enabled is impossible on Linux 64-bit? Possibly, yes. I'll have to take a look myself, maybe I'll find the time for this today evening, but I don't know yet. If you need more time, this would be ok. I only want to avoid, that you "forgot" me. In this case please allow me to send you a gentle memorization. Thanks a lot for your help. ---------------------------------------------------------------------- Thaddy · 2020-05-16 07:54 For the external linker plz examine man ld, because I suspect you can pass certain options to the linker from fpc with fpc -k-<-ld options> to keep debug info. ---------------------------------------------------------------------- Hartmut · 2020-05-16 14:14 Hello Thaddy, thanks for this suggestion. I searched "man ld" for something like enabling debug infos but found nothing. I searched "man ld" for something like smart linking but found nothing. I searched "man ld" for all command line parameters which had anything to do with "debug" or "DWARF" or "Stabs" and got this matches:  --traditional-format  --compress-debug-sections=none  --compress-debug-sections=zlib  --compress-debug-sections=zlib-gnu  --compress-debug-sections=zlib-gabi  --enable-long-section-names  --disable-long-section-names  --enable-extra-pe-debug    I did not understand most of their explanations (it's the 1st time where I have anything to do with a linker) so I tried them all for 64-bit one by one like: /fpc.sh -B -CX -XX -g -gl -k--traditional-format demo1.pas Unfortunately this did not help: The last 3 parameters leeded to an error "/usr/bin/ld: unknown Option". The first 5 parameters showed a "Note: DWARF debug information cannot be used with smart linking on this target, switching to static linking" and unused procs were not excluded (as before). Did I make something wrong? ---------------------------------------------------------------------- Thaddy · 2020-05-16 16:14 Two remarks: // linker specidic: If the options are really omitted or error, check if you are not using the gold linker as default. This works but currently only for the llvm back-end. (this is by trial and error!, be aware it can be something else..) I write this because you state you have exanined that man, so it should accept these options. The gold linker has fewer options. // fpc specific: If you want to compile as inefficient as possible, use fpc -O-. This should prevent FPC's optimizing stages from removing information before the link stage. ---------------------------------------------------------------------- Hartmut · 2020-05-16 17:09 Hello Thaddy, thanks for your help. But as often it is difficult for me to understand you. Quote from: Thaddy on May 16, 2020, 06:14:21 pm// linker specidic: If the options are really omitted or error, check if you are not using the gold linker as default. This works but currently only for the llvm back-end. (this is by trial and error!, be aware it can be something else..) I write this because you state you have exanined that man, so it should accept these options. The gold linker has fewer options. I tried to find out if my system uses the "gold linker". Above error message said "/usr/bin/ld: unknown Option". So I started    /usr/bin/ld -v and got    GNU ld (GNU Binutils for Ubuntu) 2.30 and started    /usr/bin/ld -V and got GNU ld (GNU Binutils for Ubuntu) 2.30 Supported Emulations:    elf_x86_64    elf32_x86_64    elf_i386    elf_iamcu    i386linux    elf_l1om    elf_k1om    i386pep    i386pe Does this answer your question? Quote from: Thaddy on May 16, 2020, 06:14:21 pm// fpc specific: If you want to compile as inefficient as possible, use fpc -O-. This should prevent FPC's optimizing stages from removing information before the link stage. So I started /fpc.sh -B -CX -XX -g -gl -O- demo1.pas This showed a "Note: DWARF debug information cannot be used with smart linking on this target, switching to static linking" and unused procs were not excluded (as before). So it made no difference. ---------------------------------------------------------------------- PascalDragon · 2020-05-17 11:29 Quote from: Thaddy on May 16, 2020, 06:14:21 pm// fpc specific: If you want to compile as inefficient as possible, use fpc -O-. This should prevent FPC's optimizing stages from removing information before the link stage. Hartmut does not want to keep information, he wants procedures that aren't used to be excluded. And that is currently simply not possible with DWARF on targets that use ld. ---------------------------------------------------------------------- Hartmut · 2020-05-17 14:08 Hello PascalDragon, thanks for this post. Do I understand you correctly? In reply #11 you wanted to check, if smartlinking with debug infos enabled is impossible on Linux 64-bit or not. Is this post now the definite result of your check, that it is not possible? ---------------------------------------------------------------------- PascalDragon · 2020-05-17 20:10 I've not yet found the time to look which is why I wrote that it's currently not possible. Cause the status quo is definitely that. ---------------------------------------------------------------------- Hartmut · 2020-05-18 05:55 Thank you PascalDragon for that clarification. I will wait until you found the time to check it more detailed. ---------------------------------------------------------------------- Hartmut · 2020-08-05 06:06 Hello PascalDragon, 3 month ago you friendly offered that you could find out, if smartlinking with debug infos enabled is possible or impossible on Linux 64-bit (please see reply #11). I see from time to time that you are active in this Forum and hope that you did not forget me. I would be very happy, if you could find the time to get a definitive answer for this open question on Linux 64-bit. You would help me very much with this. Thanks a lot in advance. ---------------------------------------------------------------------- process_1 · 2020-08-05 06:52 Quote from: Hartmut on August 05, 2020, 08:06:48 am3 month ago you friendly offered that you could find out, if smartlinking with debug infos enabled is possible or impossible on Linux 64-bit (please see reply #11). It is probably long time ago forgotten, 3 months is a lot of time... Is this work as should be with new FPC version? ---------------------------------------------------------------------- Hartmut · 2020-08-15 10:45 Quote from: process_1 on August 05, 2020, 08:52:36 amIs this work as should be with new FPC version? I did not plan to switch to FPC 3.2.0 in the next future *if there is no concrete need*. And I want this feature - if possible - to use with FPC 3.0.4, because most of my programs use this version. Do you have any informations, that this feature (smartlinking with debug infos enabled on Linux 64-bit) will definitely work with FPC 3.2.0 (in the release notes I found nothing about this)? ---------------------------------------------------------------------- PascalDragon · 2020-08-16 12:08 No, nothing changed there. If it didn't work with 3.0.4 then it won't work with 3.2.0 or trunk as it stands now. Only if a newer version of ld is more "intelligent" then things would improve there. ---------------------------------------------------------------------- Hartmut · 2020-08-16 13:36 Hello PascalDragon, 3 month ago you friendly offered that you could find out, if smartlinking with debug infos enabled is possible or impossible on Linux 64-bit (please see your reply #11). I see from time to time that you are active in this Forum and hope that you did not forget me. I would be very happy, if you could find the time to get a definitive answer for this open question on Linux 64-bit. You would help me very much with this. Thanks a lot in advance. If you would decide that you dont want to look after this, please let me know, I would have to respect this. Many greetinx from Germany, Hartmut. ---------------------------------------------------------------------- PascalDragon · 2020-08-17 07:23 Quote from: Hartmut on August 16, 2020, 03:36:31 pm3 month ago you friendly offered that you could find out, if smartlinking with debug infos enabled is possible or impossible on Linux 64-bit (please see your reply #11). Yes, I know, the problem is to find the time. My free time (which is also the time when I work on FPC) is quite full including many topics that are not even remotely related to software development.