Unwanted Underscores Added To External Names
lazarus / 34-Windows

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 16:50

I'm working in MSys on 64-Bit Windows 7, running "Free Pascal Compiler
version 3.0.4 [2017/10/06] for i386". I get the same behavior if I
work at the DOS prompt, issuing the same FPC commands. Here is a
library program "d.pas" that compiles with "fpc d" to produce "d.dll".

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i library
d; function answer:longint; cdecl;begin        writeln('Hello from
answer');        answer:=42;end; exports        answer name
'_answer';end.
The Dependency Walker (http://dependencywalker.com) tells me that my
exported routine is declared in the d.dll under the name "_answer", as
I expect. I have main program "m.pas" in the same directory as "d.dll"
that compiles with "fpc m" to produce executable "m.exe", which calls
"_answer".

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
m; {$linklib d} function answer:integer; cdecl; external name
'answer';  begin        writeln('Answer: ',answer);end.
This works, see below for output.

Code: Bash  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i $
./m.exeHello from answerAnswer: 42
The external declaration of "_answer" in "m.pas" does not include the
underscore. It's just "answer". The underscore is added by FPC at
compile time. If I want "m.pas" to link to a routine "answer" with no
underscore in front of it, I am unable to do so with the "cdecl". And
that's a problem for me, because the Tcl86.dll library I'm trying to
link to does not have underscores in front of its routine names, and
uses the "cdecl" calling convention.

If I switch to "stdcall" in "d.pas" and export the routine as
"answer", then switch to "stdcall" in "m.pas" and once again give the
external name as just "answer", no underscore is added by FPC when it
compiles "m.pas", and the two programs work together just fine. But
that does not help me, because the Tcl86 library is compiled with the
"cdecl" convention.

How can I stop FPC from adding an underscore to "cdecl" external
names?

----------------------------------------------------------------------
jiaxing2 · 2020-05-14 17:02

Nope. It's designed to do so. Nothing wrong. You should learn more
about calling convention. A single Google search is enough.

If the library you want to link use cdecl, use cdecl with your
declaration. Just that.

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 17:43

Dear Jiaxing2,

>  You should learn more about calling convention. A single Google
search is enough.

Ha! You sure can talk the talk Jiaxing2. Let's see if you can walk the
walk as well. I challenge you to call "answer", declared in the
following library, from an FPC main program on Windows, and using only
the "cdecl" calling convention in the main program.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i library
d; function answer:longint; cdecl;begin        writeln('Hello from
answer');        answer:=42;end; exports        answer name
'answer';end.
And no: you are not allowed to add an underscore to the export name in
"d.pas". I look forward to seeing your masterful solution, and being
impressed.

Best, Kevan

----------------------------------------------------------------------
jiaxing2 · 2020-05-14 17:55

Quote from: Kevan S. Hashemi on May 14, 2020, 07:43:12 pmDear
Jiaxing2,

>  You should learn more about calling convention. A single Google
search is enough.

Ha! You sure can talk the talk Jiaxing2. Let's see if you can walk the
walk as well. I challenge you to call "answer", declared in the
following library, from an FPC main program on Windows, and using only
the "cdecl" calling convention in the main program.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i library
d; function answer:longint; cdecl;begin        writeln('Hello from
answer');        answer:=42;end; exports        answer name
'answer';end.
And no: you are not allowed to add an underscore to the export name in
"d.pas". I look forward to seeing your masterful solution, and being
impressed.

Best, Kevan

Because export answer; is enough. No need the other part.

I have follow this guide for a long time, and have never fail.

https://macpgmr.github.io/MacXPlatform/PascalDynLibs.html

p/s: I'm on Linux and having abandoned Windows for years, I will not
install Windows again just to win a challenge of you.

----------------------------------------------------------------------
marcov · 2020-05-14 17:55

This is governed by the CPREFIX of the said target. (see
compiler/systems/i_win.pas  line 66)

These per target settings fixate normal per target conventions
(usually afaik cprefix is mainly a binary format thing. A.OUT and COFF
have _, ELF not).

However that is from the declaration side, your problem is on the
referencing side. However your reference is a bit werid, since it is
supposed to have the dll name, and not use {$linklib??}.

The default way works fine, so I think your main problem is in the way
you manually hacked together an import lib or so?

Quote
D:\Testing\2345>m
Hello from answer
Answer: 42
Code: [Select]    program m;
    
    function answer:integer; cdecl; external 'd' name 'answer';
    
    begin
            writeln('Answer: ',answer);
    end.

----------------------------------------------------------------------
jiaxing2 · 2020-05-14 18:10

Quote from: marcov on May 14, 2020, 07:55:53 pmThe default way works
fine, so I think your main problem is in the way you manually hacked
together an import lib or so?

I think so, too.

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 18:14

Jiaxing2:

QuoteI'm on Linux and having abandoned Windows for years, I will not
install Windows again just to win a challenge of you.
The problem arises only on Windows. On Linux, FPC's implementation of
"cdecl" does not add underscores to external file names.

Marcov:

Thanks for the code reference. There is this line in i_win.pas.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i           
Cprefix      : '_';
It looks like I need to change that, because it's the character that
is added to all eternal names.

----------------------------------------------------------------------
Fred vS · 2020-05-14 18:22

QuoteIt looks like I need to change that, because it's the character
that is added to all eternal names.
Or, maybe, use LoadLibrary() instead of external...

Ok, ok, I'm going out, sorry for the noise.

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 18:23

QuoteThe default way works fine, so I think your main problem is in
the way you manually hacked together an import lib or so?
No, that's not my problem. I composed some test code so that it would
be easy for you to duplicate the problem and see it for yourself. The
code I gave you has the fewest possible lines to illustrate the
problem.

The problem arises in practice because the TclTk libraries on Windows
are 'cdecl' and the routine names have no underscores in front of
them. But FPC on Windows with 'cdecl' will not allow me to declare an
external 'cdecl' routine name "Tcl_PckgProvide" because it always
converts that name to "_Tcl_PckgProvide". At that point, the linker
gives an unhelpful message, it says, "can't find d.dll". But that's
not the actual problem. The problem is that it can't find
"_Tcl_PckgProvide".

If I switch to 'stdcall', then no underscore is added by FPC and I can
link to the Tcl libraries, but of course the main program crashes when
I actually call the Tcl routine, because the calling conventions are
different.

I wanted to spare you the the whole "Tcl" business, so made you some
simple code to illustrate the problem.

Now, how can I change "Cprefix"? Do I need to compile FPC from sources
on Windows?

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 18:42

QuoteOr, maybe, use LoadLibrary() instead of external...
I just tried with LoadLibrary, using your sample code, for which I
thank you again. Even with LoadLibrary, the fact that we declare the
function as 'cdecl' is sufficient to make FPC add an underscore to the
external name when it tries to look up the routine in the dynamic
library.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
TMyMethod=function :integer; cdecl;
So the routine has to have an underscore prefix in the dynamic
library, or else there is no way to call it.

----------------------------------------------------------------------
jiaxing2 · 2020-05-14 18:43

Quote from: Kevan S. Hashemi on May 14, 2020, 08:23:29 pmQuoteThe
default way works fine, so I think your main problem is in the way you
manually hacked together an import lib or so?
No, that's not my problem. I composed some test code so that it would
be easy for you to duplicate the problem and see it for yourself. The
code I gave you has the fewest possible lines to illustrate the
problem.

The problem arises in practice because the TclTk libraries on Windows
are 'cdecl' and the routine names have no underscores in front of
them. But FPC on Windows with 'cdecl' will not allow me to declare an
external 'cdecl' routine name "Tcl_PckgProvide" because it always
converts that name to "_Tcl_PckgProvide". At that point, the linker
gives an unhelpful message, it says, "can't find d.dll". But that's
not the actual problem. The problem is that it can't find
"_Tcl_PckgProvide".

If I switch to 'stdcall', then no underscore is added by FPC and I can
link to the Tcl libraries, but of course the main program crashes when
I actually call the Tcl routine, because the calling conventions are
different.

I wanted to spare you the the whole "Tcl" business, so made you some
simple code to illustrate the problem.

So you think this is bug. Let's the FPC devs invest on it.

QuoteNow, how can I change "Cprefix"? Do I need to compile FPC from
sources on Windows?
What about compile Tcl/Tk from source? It's open source.

From what you said I think you installed them from MSYS2. Try get the
official version from their homepage.

BTW, some FPC devs here told me on Windows x64 there is only one
calling convention and regardless of how you declared them as cdecl or
stdcall it would just work. You said your program crashed so I think
it's more likely a bug of FPC.

----------------------------------------------------------------------
jiaxing2 · 2020-05-14 18:47

Quote from: Kevan S. Hashemi on May 14, 2020, 08:42:45 pmQuoteOr,
maybe, use LoadLibrary() instead of external...
I just tried with LoadLibrary, using your sample code, for which I
thank you again. Even with LoadLibrary, the fact that we declare the
function as 'cdecl' is sufficient to make FPC add an underscore to the
external name when it tries to look up the routine in the dynamic
library.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
TMyMethod=function :integer; cdecl;
So the routine has to have an underscore prefix in the dynamic
library, or else there is no way to call it.

I don't understand. An underscore is normal for cdecl. I think FPC
does everything right. Check your Tcl/Tk dll.

----------------------------------------------------------------------
Fred vS · 2020-05-14 19:11

Quote from: jiaxing2 on May 14, 2020, 08:47:14 pmQuote from: Kevan S.
Hashemi on May 14, 2020, 08:42:45 pmQuoteOr, maybe, use LoadLibrary()
instead of external...
I just tried with LoadLibrary, using your sample code, for which I
thank you again. Even with LoadLibrary, the fact that we declare the
function as 'cdecl' is sufficient to make FPC add an underscore to the
external name when it tries to look up the routine in the dynamic
library.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
TMyMethod=function :integer; cdecl;
So the routine has to have an underscore prefix in the dynamic
library, or else there is no way to call it.

I don't understand. An underscore is normal for cdecl. I think FPC
does everything right. Check your Tcl/Tk dll.

IMHO, fpc should give more liberty to developers, why add by force
that "_"?.

It is the same for so number with Unix libraries, the so number is
arbitrary deleted, so you cannot link libraries with name
'libmylib.so.2" for example.

We are not all stupid and let us the choice to do things . (even if
you find it stupid).

----------------------------------------------------------------------
marcov · 2020-05-14 19:14

Quote from: Kevan S. Hashemi on May 14, 2020, 08:23:29 pmQuoteThe
default way works fine, so I think your main problem is in the way you
manually hacked together an import lib or so?
No, that's not my problem. I composed some test code so that it would
be easy for you to duplicate the problem and see it for yourself. The
code I gave you has the fewest possible lines to illustrate the
problem.

Please do try to use the syntax that I specified:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
function answer:integer; cdecl; external 'sometcltk.dll' name
'Tcl_PckgProvide';
remove the $linklib and move away all .a's, and see what happens.

For me it worked, and I tried with Dependencies (the successor to
dependency walker) that the DLL that I tested with has no underscore.

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 19:18

QuoteSo you think this is bug. Let's the FPC devs invest on it.
No, it's not a bug. It's deliberate policy. Here is the code from
pdecvar.pas, routine name read_public_and_external:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i { Add C _
prefix }      if is_cdecl or         (          is_dll and         
(target_info.system in systems_darwin)         ) then        C_Name :=
target_info.Cprefix+C_Name;
On Windows and MacOSX, Cprefix is "_". You can call external "cdecl"
routines only if their names begin with underscores. On Linux, Cprefix
is empty, so you have complete control over the names.

Meanwhile, the "export" section of a "library" in FPC does not force
the addition of an underscore for "cdecl" routines, so we must add the
underscore ourselves on MacOSX and Windows, but not on Linux, by use
of OS-dependent compiler directives. Otherwise, the "export" and
"external" don't agree.

QuoteAn underscore is normal for cdecl.
The Linux and Windows Tcl libraries don't have underscores in front of
their names. I'm calling the Linux Tcl libraries with "cdecl" just
fine. I just examined half a dozen Linux kernel libraries with the
"nm" utility, and there are many names that have no underscores in
front of them.

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 20:03

QuotePlease do try to use the syntax that I specified:
Nice one Marcov! That works. Setting your solution in my simple
example:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
answer:integer; cdecl; external 'd.dll' name 'answer';
With this syntax, FPC ends up putting the prefix before the library
name. Here is the assembler output:

Code: ASM  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i call   
FPC_IOCHECKcall    _$dll$d$answermovw    %ax,%cx
That behavior may be a bug. I hope not, because if it's a bug, it may
be fixed later. I moved the library to another folder and used FPC's
"-k" option to pass the new path to the linker. The compiler found the
library just fine. My original declaration was:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$linklib
d}function answer:integer; cdecl; external;
An alternative is:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$linklib
d}function answer:integer; cdecl; external name 'answer';
These won't compile without an "_answer" to link to, but they produce
identical assembler code:

Code: ASM  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i call   
FPC_IOCHECKcall    _answermovw    %ax,%cx
So, I have a syntax that will work on Windows. Thank you for your
attention to my problem, and for your patience.

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 20:31

QuoteIMHO, fpc should give more liberty to developers, why add by
force that "_"?.
I agree. That would make my code cleaner, and it would bring the FPC
"export" sections into harmony with the "external" declarations, as on
Linux.

And another thing that's great about the Linux implementation: dynamic
libraries not needed at compile time. You don't have to specify the
library name or the path name. Just the routine name. It's not as if
linking to a dynamic library at compile time is any guarantee that the
library will be present at run time. Well, I suppose the link against
the dynamic library is a check that the routines exist in the library.
Maybe that's worth it.

Anyway, I am now linking to the Tcl libraries on MacOSX, Windows, and
Linux, so I'm happy.

----------------------------------------------------------------------
Fred vS · 2020-05-14 20:45

QuoteWell, I suppose the link against the dynamic library is a check
that the routines exist in the library. Maybe that's worth it.
IMHO, it is because if you use a C (gcc, clang, ...) compiler, with
C++ code, the library is needed at linking, with the header of the
library too.
But it is not the case with fpc, so I dont understand why fpc does all
that C things.

----------------------------------------------------------------------
marcov · 2020-05-14 21:34

Note:

Quote from: Kevan S. Hashemi on May 14, 2020, 10:03:54 pmCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$linklib
d}function answer:integer; cdecl; external;

The above is the linking syntax for symbols that have a unified linker
namespace like most open source unixes.  All symbols are just thrown
on one big heap and resolved.

The working syntax for windows:
Code: [Select]    function answer:integer; cdecl; external 'd.dll'
name 'answer';

is the syntax for platforms (like Windows, OS X and Solaris) that
implement linker namespaces for dynamic libraries, where each symbol
has an associated "module" (effectively: DLL/.so). That's why this
syntax has the dll name in the declaration.  The _$dll$d$answer is
probably a jump to such auto generated import library table, NOT the
final jump to the DLL itself. Probably at the _$dll$d$answer  there is
an easily (and lazy?) relocatable jmp to the real library address.

GCC/dlltool tries to hide this by adding this to the import libraries,
but FPC generates the relevant COFF/PE structures on the fly.

It could be that FPC assumes that if you use the gcc syntax on
windows, you really want to link with gcc.

----------------------------------------------------------------------
Fred vS · 2020-05-14 21:59

Quote from: Kevan S. Hashemi on May 14, 2020, 08:42:45 pmQuoteOr,
maybe, use LoadLibrary() instead of external...
I just tried with LoadLibrary, using your sample code, for which I
thank you again. Even with LoadLibrary, the fact that we declare the
function as 'cdecl' is sufficient to make FPC add an underscore to the
external name when it tries to look up the routine in the dynamic
library.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
TMyMethod=function :integer; cdecl;
So the routine has to have an underscore prefix in the dynamic
library, or else there is no way to call it.

Huh, sorry but this one I dont have the same result than you.

For me, on Windows, that code worked without problems:

Library:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i library d; 
  function answer:integer; cdecl;    begin            writeln('Hello
from d library, answer routine.');            answer:=42;    end;   
   exports answer;    end. 
Program:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   program
m;   uses dynlibs, SysUtils;     type     TMyMethod=function :integer;
cdecl;        var     MyLib: TLibHandle = dynlibs.NilHandle;   
 MyMethod: TMyMethod;    begin       //  MyLib :=
LoadLibrary(ExtractFilePath(ParamStr(0)) + '\d.dll');         [EDITED]
// On Windows, it works too if library is in same dir than exe   
 MyLib := LoadLibrary('d.dll');        if MyLib = 0 then Exit;    
 MyMethod := TMyMethod(GetProcedureAddress(MyLib, 'answer'));   
 writeln('Answer: ',MyMethod);     FreeLibrary(MyLib);      MyLib:=
DynLibs.NilHandle;    end.
Result:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i fred@fiens
~>m.exeHello from d library, answer routine.Answer: 42

(And I am very happy that my lovely LoadLibrary() is still safe.)

Fre;D

----------------------------------------------------------------------
Kevan S. Hashemi · 2020-05-14 23:06

Fred: Darn it, you're right. I just ran it myself. Mine failed because
I tried to use a forward slash instead of a backslash on Windows.
Should have send you my code so you could have pointed out my error,
rather than having you write it yourself. Well, this is great: two
solutions to my problem.

Marcov: Thank you for that explanation of the difference between
MacOSX/Windows and the Linux handling of dynamic library routines. Now
the differences between the compile behavior make more sense.

Both: I really appreciate your help. This has been a good day. I have
now proved to myself that I can indeed port all 35k lines of my Pascal
code from GNU Pascal to FPC, and have my software package run in 64
bit on all three platforms at the end. I'm celebrating.

----------------------------------------------------------------------
Fred vS · 2020-05-14 23:34

QuoteBoth:
Thanks! (yes Marco and I always do good job together).

----------------------------------------------------------------------
jiaxing2 · 2020-05-15 03:26

Quote from: Fred vS on May 14, 2020, 09:11:28 pmQuote from: jiaxing2
on May 14, 2020, 08:47:14 pmQuote from: Kevan S. Hashemi on May 14,
2020, 08:42:45 pmQuoteOr, maybe, use LoadLibrary() instead of
external...
I just tried with LoadLibrary, using your sample code, for which I
thank you again. Even with LoadLibrary, the fact that we declare the
function as 'cdecl' is sufficient to make FPC add an underscore to the
external name when it tries to look up the routine in the dynamic
library.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
TMyMethod=function :integer; cdecl;
So the routine has to have an underscore prefix in the dynamic
library, or else there is no way to call it.

I don't understand. An underscore is normal for cdecl. I think FPC
does everything right. Check your Tcl/Tk dll.

IMHO, fpc should give more liberty to developers, why add by force
that "_"?.

It is the same for so number with Unix libraries, the so number is
arbitrary deleted, so you cannot link libraries with name
'libmylib.so.2" for example.

We are not all stupid and let us the choice to do things . (even if
you find it stupid).

I just tell by my experience. I didn't said anyone stupid or even
think so. I'm just confused.

----------------------------------------------------------------------
jiaxing2 · 2020-05-15 03:30

Quote from: jiaxing2 on May 14, 2020, 08:10:35 pmQuote from: marcov on
May 14, 2020, 07:55:53 pmThe default way works fine, so I think your
main problem is in the way you manually hacked together an import lib
or so?

I think so, too.

Yeah. After all. This is the case. He used the wrong syntax from the
beginning.

p/s: Don't want to insult anyone but... it's just that.