Getting File Size from PE Header Fails with Lazarus
lazarus / 34-Windows

----------------------------------------------------------------------
msintle · 2023-10-24 19:04

So here's an odd one. Code like this works beautifully in Delphi:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i { call
passing HInstance as the Executable parameter, add Windows to uses
}function GetExeSize(Executable: THandle): Cardinal;type 
PImageDosHeader = ^TImageDosHeader;  TImageDosHeader = packed record 
 e_magic: Word;    e_ignore: packed array [0..28] of Word;    _lfanew:
Longint;  end;var  i, NumSections: Integer;  p: PAnsiChar;begin 
Result := 0;  p := Pointer(Executable);  Inc(p,
PImageDosHeader(p)^._lfanew + SizeOf(DWORD));  NumSections :=
PImageFileHeader(p)^.NumberOfSections;  Inc(p,
SizeOf(TImageFileHeader) + SizeOf(TImageOptionalHeader));  for i := 1
to NumSections do  begin    with PImageSectionHeader(p)^ do      if
PointerToRawData+SizeOfRawData > Result then        Result :=
PointerToRawData + SizeOfRawData;    Inc(p,
SizeOf(TImageSectionHeader));  end;end;
But it fails in Lazarus.

To clarify, it does still compile and run - but returns the wrong file
size (always falling short a bit).

Is Lazarus constructing the PE header differently somehow, perhaps?

----------------------------------------------------------------------
Fibonacci · 2023-10-24 19:11

Works 100% fine if reading 32bit i386.

For 64bit the code will be differet. eg TImageOptionalHeader default
is TImageOptionalHeader32, and for 64bit you need to use
TImageOptionalHeader64

BTW. Remove debug info, this is non standard and weird method to get
"file size"

----------------------------------------------------------------------
msintle · 2023-10-24 19:29

Quote from: Fibonacci on October 24, 2023, 09:11:32 pmWorks 100% fine
if reading 32bit i386.

Not for me. I am building x86 targets. Happy to upload both a console
and a GUI test app if people would like. Reproduces instantly.

Quote from: Fibonacci on October 24, 2023, 09:11:32 pmFor 64bit the
code will be differet. eg TImageOptionalHeader default is
TImageOptionalHeader32, and for 64bit you need to use
TImageOptionalHeader64

Thanks for noting that, will definitely come in handy in the future.

Quote from: Fibonacci on October 24, 2023, 09:11:32 pmBTW. Remove
debug info, this is non standard and weird method to get "file size"

How do you mean? In Delphi, it works properly even with debug info. Is
the way Lazarus is adding debug information then non-standard and
breaking the standard header based file size calculation process?

Regarding your comment that this is a weird method to get file size,
it is the only method to get file size when you are building a stub
application (such as a self-extractor which would have an archive of
variable size appended to its end, for example).

----------------------------------------------------------------------
440bx · 2023-10-25 02:11

@msintle,

For a general utility your code will _not_ work because
sizeof(TImageOptionalHeader) is likely determined at compile time
(otherwise the code you showed would not compile.)

if that code has to deal with executables other than itself then it
will only be right when the executable it is dealing with happens to
be the same bitness as the program.  If the bitness  between them
differ then the result will be wrong due to the wrong size of
TImageOptionalHeader.

On a different note, why do you calculate the file size that way
instead of simply asking the O/S for the file size ?

HTH.

----------------------------------------------------------------------
KodeZwerg · 2023-10-25 02:51

I do use something similar in my Advanced-Properties app that was
compiled with Delphi.
It is very handy to detect anything that in my app I declared as
"overlay" (or extract/strip such data)
It is also very reliable used in stubs (no matter what bitness when
using correct header translation) to detect the end of stub in a 100%
accurate way.
Many oldschool and also newschool archivers that do produce executable
files, like WinRAR, are using this way to know where streaming of
archive begin.

To OP:
I will try to port my way, pure WinAPI, to Lazarus and send source
here.

----------------------------------------------------------------------
Fibonacci · 2023-10-25 07:01

Quote from: msintle on October 24, 2023, 09:29:29 pmHow do you mean?
In Delphi, it works properly even with debug info. Is the way Lazarus
is adding debug information then non-standard and breaking the
standard header based file size calculation process?

I dont have Delphi so cant compare the outputs, but FPC is adding
debug info after EOF, overlay. If you build your exe without debug
info, your function will display correct size. In my case debug info
adds ~60 KB after PE sections (with only uses Windows).

170496 {your function} + 62931 {overlay size} = 233427 {actual file
size on the disk}

Other than overlay debug info, it also creates many actual PE
sections.

----------------------------------------------------------------------
440bx · 2023-10-25 07:43

Quote from: Fibonacci on October 25, 2023, 09:01:23 am but FPC is
adding debug info after EOF, overlay.
I'm probably misunderstanding something but, I have not seen an
instance where FPC adds debug info _after_ the executable's EOF in a
PE file. 

Do you have an example of such an occurrence and/or how to produce
such a file ?   TIA.

The following is what I've always seen (Note: details for non-debug
sections have been removed for brevity):
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     
 40.0178            - IMAGE_SECTION_HEADER(s) - Number of sections :
10      40.0178              SECTION:  1 of 10      40.0178     178
 [8]                                 Name : .text       40.01a0      
      SECTION:  2 of 10  contains: Thread Local Storage (TLS)
directory      40.01a0     1a0  [8]                                
Name : .data       40.01c8              SECTION:  3 of 10      40.01c8
    1c8  [8]                                 Name : .rdata     
 40.01f0              SECTION:  4 of 10      40.01f0     1f0  [8]    
                           Name : .bss       40.0218            
 SECTION:  5 of 10      40.0218     218  [8]                         
      Name : .CRT       40.0240              SECTION:  6 of 10
 contains: Imports directory     40.0240 +                           
            Imports address table directory      40.0240     240  [8]
                               Name : .idata       40.0268           
 SECTION:  7 of 10      40.0268     268  [8]                         
      Name : /4              1.c3f0  [12]                            
         .debug_info      40.0270     270  [4]                       
Virtual size :      9202    (37,378)      40.0274     274  [4]       
  (Relative) Virtual address :    1.6000    [Va:     41.6000] [FO:  
ea00] [      /4]      40.0278     278  [4]                     Size of
raw data :      9400    (37,888)      40.027c     27c  [4]      File
offset/pointer to raw data :      ea00    [Va:     41.6000] [FO:  
ea00] [      /4]     40.0280     280  [4]   File offset/pointer to
relocations :         0     40.0284     284  [4]   File offset/pointer
to linenumbers :         0      40.0288     288  [2]              
 Number of relocations :         0     40.028a     28a  [2]          
    Number of linenumbers :         0      40.028c     28c  [4]      
              Characteristics : 4200.0040     40.028c +              
        SCN_CNT_INITIALIZED_DATA          40     40.028c +           
             SCN_ALIGN_NOTSPECIFIED     0     40.028c +              
             SCN_MEM_DISCARDABLE    200.0000     40.028c +           
                       SCN_MEM_READ   4000.0000       40.0290        
    SECTION:  8 of 10      40.0290     290  [8]                      
         Name : /16              1.c3fc  [14]                        
             .debug_abbrev      40.0298     298  [4]                 
      Virtual size :       244    (  580)      40.029c     29c  [4]  
       (Relative) Virtual address :    2.0000    [Va:     42.0000]
[FO: 1.7e00] [     /16]      40.02a0     2a0  [4]                    
Size of raw data :       400    (1,024)      40.02a4     2a4  [4]    
 File offset/pointer to raw data :    1.7e00    [Va:     42.0000] [FO:
1.7e00] [     /16]     40.02a8     2a8  [4]   File offset/pointer to
relocations :         0     40.02ac     2ac  [4]   File offset/pointer
to linenumbers :         0      40.02b0     2b0  [2]              
 Number of relocations :         0     40.02b2     2b2  [2]          
    Number of linenumbers :         0      40.02b4     2b4  [4]      
              Characteristics : 4200.0040     40.02b4 +              
        SCN_CNT_INITIALIZED_DATA          40     40.02b4 +           
             SCN_ALIGN_NOTSPECIFIED     0     40.02b4 +              
             SCN_MEM_DISCARDABLE    200.0000     40.02b4 +           
                       SCN_MEM_READ   4000.0000       40.02b8        
    SECTION:  9 of 10      40.02b8     2b8  [8]                      
         Name : /30              1.c40a  [12]                        
             .debug_line      40.02c0     2c0  [4]                   
    Virtual size :       bd5    (3,029)      40.02c4     2c4  [4]    
     (Relative) Virtual address :    2.1000    [Va:     42.1000] [FO:
1.8200] [     /30]      40.02c8     2c8  [4]                     Size
of raw data :       c00    (3,072)      40.02cc     2cc  [4]      File
offset/pointer to raw data :    1.8200    [Va:     42.1000] [FO:
1.8200] [     /30]     40.02d0     2d0  [4]   File offset/pointer to
relocations :         0     40.02d4     2d4  [4]   File offset/pointer
to linenumbers :         0      40.02d8     2d8  [2]              
 Number of relocations :         0     40.02da     2da  [2]          
    Number of linenumbers :         0      40.02dc     2dc  [4]      
              Characteristics : 4200.0040     40.02dc +              
        SCN_CNT_INITIALIZED_DATA          40     40.02dc +           
             SCN_ALIGN_NOTSPECIFIED     0     40.02dc +              
             SCN_MEM_DISCARDABLE    200.0000     40.02dc +           
                       SCN_MEM_READ   4000.0000       40.02e0        
    SECTION: 10 of 10      40.02e0     2e0  [8]                      
         Name : /42              1.c416  [13]                        
             .debug_frame      40.02e8     2e8  [4]                  
     Virtual size :       29c    (  668)      40.02ec     2ec  [4]   
      (Relative) Virtual address :    2.2000    [Va:     42.2000] [FO:
1.8e00] [     /42]      40.02f0     2f0  [4]                     Size
of raw data :       400    (1,024)      40.02f4     2f4  [4]      File
offset/pointer to raw data :    1.8e00    [Va:     42.2000] [FO:
1.8e00] [     /42]     40.02f8     2f8  [4]   File offset/pointer to
relocations :         0     40.02fc     2fc  [4]   File offset/pointer
to linenumbers :         0      40.0300     300  [2]              
 Number of relocations :         0     40.0302     302  [2]          
    Number of linenumbers :         0      40.0304     304  [4]      
              Characteristics : 4200.0040     40.0304 +              
        SCN_CNT_INITIALIZED_DATA          40     40.0304 +           
             SCN_ALIGN_NOTSPECIFIED     0     40.0304 +              
             SCN_MEM_DISCARDABLE    200.0000     40.0304 +           
                       SCN_MEM_READ   4000.0000        40.0308       
    - IMAGE HEADER - 248 filler bytes -      40.0308     308  [ 8]  
__ __ __ __ __ __ __ __  00 00 00 00 00 00 00 00   ________........  
 40.0310     310  [16]   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00
00   ................     40.0320     320  [16]   00 00 00 00 00 00 00
00  00 00 00 00 00 00 00 00   ................      etc     etc   
 etc  
In the above, all file offsets are within the real size of the PE file
(IOW, not appended to what "used to be" its real size)

----------------------------------------------------------------------
KodeZwerg · 2023-10-25 07:46

My app does confirm what Fibonacci say.
I attached an Image that visualize it.

----------------------------------------------------------------------
Fibonacci · 2023-10-25 07:47

Quote from: 440bx on October 25, 2023, 09:43:12 amDo you have an
example of such an occurrence and/or how to produce such a file ? 
 TIA.

1. OPs function returns real image size, which is different than file
size, ergo "there is something more in the file"
2. Open binary in eg. PE-Bear and see Overlay

----------------------------------------------------------------------
440bx · 2023-10-25 07:54

@KodeZwerg & @Fibonacci

Which binary ?... I don't see a binary anywhere in this thread ... did
I miss an elephant in the room ?

----------------------------------------------------------------------
Fibonacci · 2023-10-25 08:07

Ok, if you cant/dont want to compile yourself, here you go, binary in
the attachment

Full source code
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses
Windows; { call passing HInstance as the Executable parameter, add
Windows to uses }function GetExeSize(Executable: THandle):
Cardinal;type  PImageDosHeader = ^TImageDosHeader;  TImageDosHeader =
packed record    e_magic: Word;    e_ignore: packed array [0..28] of
Word;    _lfanew: Longint;  end;var  i, NumSections: Integer;  p:
PAnsiChar;begin  Result := 0;  p := Pointer(Executable);  Inc(p,
PImageDosHeader(p)^._lfanew + SizeOf(DWORD));  NumSections :=
PImageFileHeader(p)^.NumberOfSections;  Inc(p,
SizeOf(TImageFileHeader) + SizeOf(TImageOptionalHeader));  for i := 1
to NumSections do  begin    with PImageSectionHeader(p)^ do      if
PointerToRawData+SizeOfRawData > Result then        Result :=
PointerToRawData + SizeOfRawData;    Inc(p,
SizeOf(TImageSectionHeader));  end;end; begin  writeln('image size =
', GetExeSize(GetModuleHandle(nil)));  readln;end.
(app return) image size = 58368

real file size = 98818

40450 bytes overlay data

----------------------------------------------------------------------
KodeZwerg · 2023-10-25 08:08

Quote from: 440bx on October 25, 2023, 09:54:28 am@KodeZwerg &
@Fibonacci

Which binary ?... I don't see a binary anywhere in this thread ... did
I miss an elephant in the room ?
I just opened Lazarus, a new form, pressed run and on screen image i
clicked "okay"
Afterwards I run my app that I mentioned earlier over the compiled
executable.

----------------------------------------------------------------------
440bx · 2023-10-25 08:44

Thank you both, Fibonacci and KodeZwerg, for the additional
information.  Now I see what you guys are talking about. 

Just for the record, there is no need to calculate the "real
executable size" because that is given in the PE header.  Using the
executable that Fibonacci posted:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
 40.0080            - IMAGE_NT_HEADERS -      40.0080      80  [4]   
                       Signature : 00004550    (PE00)      40.0084   
          - IMAGE_FILE_HEADER -      40.0084      84  [2]            
                Machine :  14c     40.0084 +                        
IMAGE_FILE_MACHINE_I386      40.0086      86  [2]                  
Number of sections :    b    ( 11)      40.0088      88  [4]         
           Time Date Stamp :    0      40.008c      8c  [4]          
  Pointer to symbol table : e400     40.0090      90  [4]            
      Number of symbols :  314    (788)      40.0094      94  [2]    
        Size of optional header :   e0    (224)      40.0096      96
 [2]                      Characteristics :  107     40.0096 +       
             IMAGE_FILE_RELOCS_STRIPPED      1     40.0096 +         
          IMAGE_FILE_EXECUTABLE_IMAGE      2     40.0096 +           
      IMAGE_FILE_LINE_NUMS_STRIPPED      4     40.0096 +             
         IMAGE_FILE_32BIT_MACHINE    100 Note that the "Pointer to
symbol table" is where what you guys have been calling the "overlay"
starts.  Therefore the "executable" size is 0xE400 which is 58,368.

The size of the "overlay" can be calculated by simply subtracting
0x400 0xE400 from the file size reported by the O/S. (note: corrected
wrong 0x400 size to 0xE400)

Also, if needed, the symbol table (referred to as "overlay" in this
thread) has a header with information that may be useful (depending on
what you're looking for.) e.g, in the file posted:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   address
n/a             - COFF SYMBOL TABLE -                            
 File pointer to symbol table :   e400                               
       Number of symbols :    314    (   788)                        
    File pointer to string table : 1.1b68                            
          String table size :   669a    (26,266) Again, thank you both
for providing the information I needed to make sense of what was being
talked about in this thread.

And just for the record, since the symbol table is NOT in a section,
it MUST be after the last section, therefore using the "pointer to
symbol" table as the "executable" size is reliable (barring a
malformed PE file.)

ETA:

Corrected: 0x400 to 0xE400

----------------------------------------------------------------------
msintle · 2023-10-25 12:54

Quote from: KodeZwerg on October 25, 2023, 04:51:49 amTo OP:
I will try to port my way, pure WinAPI, to Lazarus and send source
here.

Perhaps you can also paste that code here and I will see if I can make
any progress with it, if you haven't yet had time to port it. A pure
WinAPI approach does sound good indeed.

Excellent discussion and contributions from everyone, thank you very
much. To clarify - is it true (or not) that Lazarus adds debug
information to the tail end of an executable in non-standard form?
Probably asking for this behavior to be changed is moot, but yes, it
does break things and would prevent debugging of stubs in the Lazarus
IDE (not a happy proposition).

Also: It does not happen on Linux. Meaning, debug data does not
produce incorrect file sizes when reading the ELF header directly and
computing the file size in that manner.

For completeness, macOS is untested due to its app-bundle structure
and how it is innately hostile to data appended to the tail end of a
stub with things like code signing, notarization, etc.

----------------------------------------------------------------------
440bx · 2023-10-25 13:51

Quote from: msintle on October 25, 2023, 02:54:09 pmTo clarify - is it
true (or not) that Lazarus adds debug information to the tail end of
an executable in non-standard form?
It's important to realize there is more than one type of debug
information.

What is now considered debug information is DWARF or stabs.  I'll
mention specifics about DWARF because those are fresh in my mind.  In
the case of DWARF, every "part" of the DWARF debug info is in a PE
section, therefore the debug info is part of the PE file and, as a
result, that information will be included in the PE size.

If I remember correctly, stabs, which is another type of debugging
information, is also stored in a PE section.  It would be good if
someone confirmed this for stabs because it's been a while since I
dealt with it and my recollection might not be right.

The other type of debug information, is the COFF symbol table, what
has been referred to as "overlay" in this thread.  That information,
unlike DWARF, is NOT stored in PE sections therefore it can be
considered as not being part of the executable.   In the case of
Windows, there is the "Pointer to symbol table" to locate it in the
program file. 

As far as standard/non-standard... first, it isn't Lazarus that adds
debug information, it is FPC that puts it there (regardless of the
type of debug information), second, at least in the case of Windows,
the "standard" way to store the COFF symbol table is to append it to
what is PE information proper (what is in sections) and update the
"pointer to symbol table" to enable locating it.  In Windows, that's
pretty much the standard way. 

An alternative would be to create a debug directory entry (which is
what Delphi does for its debug information) but, that seems redundant
for COFF symbols given that the "pointer to symbol table" is there for
that.

HTH.

----------------------------------------------------------------------
msintle · 2023-10-25 14:03

Quote from: 440bx on October 25, 2023, 03:51:01 pmQuote from: msintle
on October 25, 2023, 02:54:09 pmTo clarify - is it true (or not) that
Lazarus adds debug information to the tail end of an executable in
non-standard form?
It's important to realize there is more than one type of debug
information.

What is now considered debug information is DWARF or stabs.  I'll
mention specifics about DWARF because those are fresh in my mind.  In
the case of DWARF, every "part" of the DWARF debug info is in a PE
section, therefore the debug info is part of the PE file and, as a
result, that information will be included in the PE size.

If I remember correctly, stabs, which is another type of debugging
information, is also stored in a PE section.  It would be good if
someone confirmed this for stabs because it's been a while since I
dealt with it and my recollection might not be right.

The other type of debug information, is the COFF symbol table, what
has been referred to as "overlay" in this thread.  That information,
unlike DWARF, is NOT stored in PE sections therefore it can be
considered as not being part of the executable.   In the case of
Windows, there is the "Pointer to symbol table" to locate it in the
program file. 

As far as standard/non-standard... first, it isn't Lazarus that adds
debug information, it is FPC that puts it there (regardless of the
type of debug information), second, at least in the case of Windows,
the "standard" way to store the COFF symbol table is to append it to
what is PE information proper (what is in sections) and update the
"pointer to symbol table" to enable locating it.  In Windows, that's
pretty much the standard way. 

An alternative would be to create a debug directory entry (which is
what Delphi does for its debug information) but, that seems redundant
for COFF symbols given that the "pointer to symbol table" is there for
that.

HTH.

That's very helpful indeed. Is there a way to obtain the size of this
COFF table from its symbol table pointer, perhaps? Such that the code
at the start of the thread may be updated to account for this extra
data and made foolproof by so doing?

----------------------------------------------------------------------
440bx · 2023-10-25 15:03

Quote from: msintle on October 25, 2023, 04:03:13 pmIs there a way to
obtain the size of this COFF table from its symbol table pointer,
perhaps?
Yes, there is. 

The "pointer to symbol table" allows you to locate the start of the
COFF symbol table.  There is a field in the optional header that
specifies the number of entries in that table.

The string table follows the COFF symbol entries, therefore, the
string table is located at "Location of COFF table" + (size of COFF
entry * number of entries.)   At that location, there is a DWORD that
gives the size of the entire string table (I believe including the
DWORD itself).  The string table is just a bunch of null terminated c
strings one after another. 

NOTE: the size of each COFF symbol entry is 18 bytes or, what is the
same sizeof(TIMAGE_SYMBOL) found in the definitions below.

you'll need some data types... (to have and use their sizes) the
following are the necessary types for the calculations I mentioned
above:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i // --------
---------------------------------------------------------------------/
/ COFF IMAGE SYMBOL related types and contants // see:////
http://www.delorie.com/djgpp/doc/coff/symtab.html//// for detailed
information about the meaning of the fields in IMAGE_SYMBOL as// well
as winnt.h  const  // these value can be found in the SymSectionNumber
to indicate the meaning  // of the SymValue field  
IMAGE_SYM_UNDEFINED      =        0;       // symbol is undefined or
common  IMAGE_SYM_ABSOLUTE       = word(-1);       // symbol is an
absolute value  IMAGE_SYM_DEBUG          = word(-2);       // symbol
is a special debug item  IMAGE_SYM_SECTION_MAX    = word($FEFF);    //
values 0xFF00-0xFFFF are special   IMAGE_SYM_SECTION_MAX_EX =
high(int16);  const  // Symbol Types - fundamental types  
IMAGE_SYM_TYPE_NULL       = $0000;         // no type 
IMAGE_SYM_TYPE_VOID       = $0001;         //  IMAGE_SYM_TYPE_CHAR   
  = $0002;         // type character  IMAGE_SYM_TYPE_SHORT      =
$0003;         // type short integer  IMAGE_SYM_TYPE_INT        =
$0004;         //  IMAGE_SYM_TYPE_LONG       = $0005;         // 
IMAGE_SYM_TYPE_FLOAT      = $0006;         //  IMAGE_SYM_TYPE_DOUBLE 
  = $0007;         //  IMAGE_SYM_TYPE_STRUCT     = $0008;         // 
IMAGE_SYM_TYPE_UNION      = $0009;         //  IMAGE_SYM_TYPE_ENUM   
  = $000A;         // enumeration  IMAGE_SYM_TYPE_MOE        = $000B;
       // member of enumeration  IMAGE_SYM_TYPE_BYTE       = $000C;  
     //  IMAGE_SYM_TYPE_WORD       = $000D;         // 
IMAGE_SYM_TYPE_UINT       = $000E;         //  IMAGE_SYM_TYPE_DWORD  
  = $000F;         //  IMAGE_SYM_TYPE_PCODE      = $8000;         //  
 // Symbol types - derived types   IMAGE_SYM_DTYPE_NULL      =     0;
       // no derived type  IMAGE_SYM_DTYPE_POINTER   =     1;        
// pointer  IMAGE_SYM_DTYPE_FUNCTION  =     2;         // function 
IMAGE_SYM_DTYPE_ARRAY     =     3;         // array  // Type entry////
The type field in the symbol table entry contains information about
the basic// and derived type for the symbol. This information is
generated by the C// compilation system only if the -g option is used.
Each symbol has exactly one// basic or fundamental type but can have
more than one derived type. The format// of the 16-bit type entry
is://// dx are two bits each, type is one nibble (4 bits)//// d6    
 d5      d4      d3      d2      d1      type////// Bits 0 through 3,
called type, indicate one of the fundamental types given in// the
Fundamental types table below://// Fundamental types//// Mnemonic    
                Value                    Type//// T_NULL             
             0       type not assigned//      1            function
argument                   T_ARG//             (used only by
compiler)////      2                    character                
 T_CHAR//      3                short integer                
T_SHORT//      4                      integer                  
T_INT//      5                 long integer                  T_LONG//
    6               floating point                 T_FLOAT//      7  
              double word                T_DOUBLE//      8           
       structure                T_STRUCT//      9                    
  union                 T_UNION//     10                  enumeration
                T_ENUM//     11        member of enumeration         
        T_MOE//     12           unsigned character                
T_UCHAR//     13               unsigned short              
 T_USHORT//     14             unsigned integer                
 T_UINT//     15                unsigned long                
T_ULONG//// Bits 4 through 15 are arranged as six 2-bit fields marked
d1 through d6.// These d fields represent levels of the derived types
given in given in the// Derived types table below://// Derived
types//// Mnemonic        Value                  Type//// DT_NON     
       0       no derived type// DT_PTR              1              
pointer// DT_FCN              2              function// DT_ARY       
     3                 array//// The following examples demonstrate
the interpretation of the symbol table// entry representing type:////
  char *func();//// Here func is the name of a function that returns a
pointer to a character.// The fundamental type of func is 2
(character), the d1 field is 2 (function),// and the d2 field is 1
(pointer).//// Therefore, the type word in the symbol table for func
contains the// hexadecimal number 0x62, which is interpreted to mean a
function that// returns a pointer to a character.//////    short
*tabptr[10][25][3];//// Here tabptr is a three-dimensional array of
pointers to short integers. The// fundamental type of tabptr is 3
(short integer); the d1, d2, and d3 fields// each contains a 3
(array), and the d4 field is 1 (pointer).//// Therefore, the type
entry in the symbol table contains the hexadecimal// number 0x7f3
indicating a three-dimensional array of pointers to short//
integers.  const   // Symbol storage classes  
IMAGE_SYM_CLASS_END_OF_FUNCTION  = byte(-1);  IMAGE_SYM_CLASS_NULL   
        =  $0;  IMAGE_SYM_CLASS_AUTOMATIC        =  $1; 
IMAGE_SYM_CLASS_EXTERNAL         =  $2;  IMAGE_SYM_CLASS_STATIC      
   =  $3;  IMAGE_SYM_CLASS_REGISTER         =  $4; 
IMAGE_SYM_CLASS_EXTERNAL_DEF     =  $5;  IMAGE_SYM_CLASS_LABEL       
   =  $6;  IMAGE_SYM_CLASS_UNDEFINED_LABEL  =  $7; 
IMAGE_SYM_CLASS_MEMBER_OF_STRUCT =  $8;  IMAGE_SYM_CLASS_ARGUMENT    
   =  $9;  IMAGE_SYM_CLASS_STRUCT_TAG       =  $A; 
IMAGE_SYM_CLASS_MEMBER_OF_UNION  =  $B;  IMAGE_SYM_CLASS_UNION_TAG   
   =  $C;  IMAGE_SYM_CLASS_TYPE_DEFINITION  =  $D; 
IMAGE_SYM_CLASS_UNDEFINED_STATIC =  $E;  IMAGE_SYM_CLASS_ENUM_TAG    
   =  $F;  IMAGE_SYM_CLASS_MEMBER_OF_ENUM   = $10; 
IMAGE_SYM_CLASS_REGISTER_PARAM   = $11;  IMAGE_SYM_CLASS_BIT_FIELD   
   = $12;   IMAGE_SYM_CLASS_FAR_EXTERNAL     = $44;  
IMAGE_SYM_CLASS_BLOCK            = $64;  IMAGE_SYM_CLASS_FUNCTION    
   = $65;  IMAGE_SYM_CLASS_END_OF_STRUCT    = $66; 
IMAGE_SYM_CLASS_FILE             = $67;   IMAGE_SYM_CLASS_SECTION    
    = $68;  IMAGE_SYM_CLASS_WEAK_EXTERNAL    = $69;  
IMAGE_SYM_CLASS_CLR_TOKEN        = $6B;  type  // if the symbol name
is inline and it is exactly 8 characters long then it  // is not null
terminated.  This buffer type provides a large enough buffer  // where
the symbol name can be copied and be null terminated.  Note that  //
this buffer may also be used to hold a symbol name that is present in
the  // string table, for this reason it cannot be made to simply hold
8 characters  // and a null terminator since most strings in the
string table are longer  // than that.   TSYM_NAME_BUFFER  =
array[0..1023] of char;   TSYMBOL_NAME_INFO = packed record    case
integer of      0 : (SymName         : packed array[0..7] of char);  
  1 : (           SymNameInline   : boolean32;  // use SymNameOffset
if false           SymNameOffset   : DWORD;      // offset to name
into string table          );  end;  type  // WARNING: this definition
depends on how FPC does bitpacking and also how  //          it
accesses bitpacked fields (this definition is tested and known  //   
     valid only for FPC v3.0.4)   TSYM_TYPE_RANGE         = -2..5;  //
the bitfields 1..6 start at index 0  TSYM_TYPE_DERIVED_RANGE =
 0..5;   TSYM_TYPE = bitpacked record    case integer of    0 : (    
    FundamentalType  : T4BITS;           DerivedType1     : T2BITS;  
      DerivedType2     : T2BITS;          DerivedType3     : T2BITS; 
       DerivedType4     : T2BITS;          DerivedType5     : T2BITS; 
        DerivedType6     : T2BITS;        );     1 : (          //
tested the array representation with value $7F3 and it _seems_ to    
    // work as expected.           DerivedType      : bitpacked
array[TSYM_TYPE_RANGE] of T2BITS;        );     2 : (         
TypeWord         : word;        );   end; type  PIMAGE_SYMBOL =
^TIMAGE_SYMBOL;                                      // 18 
TIMAGE_SYMBOL = packed record    SymNameInfo            :
TSYMBOL_NAME_INFO;    SymValue               : DWORD;   
SymSectionNumber       : word;       // usually the 1 based section
number                                         // but see IMAGE_SYM...
constants     SymType                : TSYM_TYPE;  // see
IMAGE_SYM_TYPE...  constants    SymStorageClass        : byte;      
// see IMAGE_SYM_CLASS... constants    SymNumberOfAuxSymbols  : byte; 
end; 
Lastly, it's quite likely that the above equivalent definitions are
found somewhere in the units that come with FPC but, I use my own,
that's what the above is.

HTH.

----------------------------------------------------------------------
msintle · 2023-10-25 15:11

Would it be too much to ask for a working sample?

----------------------------------------------------------------------
440bx · 2023-10-25 16:55

Quote from: msintle on October 25, 2023, 05:11:24 pmWould it be too
much to ask for a working sample?
Unfortunately, in this case, yes because I don't use the definitions
that come with Lazarus.  As a result even the smallest working sample
would take close to 3,000 lines (most of them definition of constants,
types and structures.)

here is an excerpt of my code:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    // -----
--------------------------------------------------------------------- 
// determine if there is a COFF symbol table in the PE file   if
InPeData.pe_FileHeader^.PointerToSymbolTable = 0 then  begin    //
there is no COFF symbol table, there is nothing to dump    
exit(TRUE);  end;   with CoffSymTableInfo do  begin   
CoffSymTableEntryCount := InPeData.pe_FileHeader^.NumberOfSymbols;   
if CoffSymTableEntryCount = 0 then    begin      // there is an
address for a COFF symbol table but the size indicates      // there
is no symbol table, can't dump a zero size table       // might not be
a bad idea to issue a warning about this situation       if
IsDebuggerPresent() then asm int 3 end;       exit(TRUE);    end; 
end;    // -----------------------------------------------------------
---------------  // there is a table and it is not empty, get a
pointer to it and ensure the  // entire table is readable   with
InPeData, CoffSymTableInfo do  begin    CoffSymTable := PIMAGE_SYMBOL 
                    (                       pe_Fileptr +
pe_FileHeader^.PointerToSymbolTable                      );     if
IsBadReadPtr(CoffSymTable,                    CoffSymTableEntryCount *
sizeof(CoffSymTable^)) then exit;     // determine the pointer to the
COFF SYMBOL NAMES string table     StringTable := pointer(CoffSymTable
+ CoffSymTableEntryCount);     if IsBadReadPtr(StringTable,
sizeof(StringTable^.Size))         then exit;    if
IsBadReadPtr(StringTable, StringTable^.Size)                 then
exit;  end; 
The code above shows how to calculate the location of the COFF symbol
table and the String table as well as ensuring they are both readable.

Below is the code to walk the string table, in your case this is not
necessary, I included it only to show how you could walk it if you
have to.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  // -------
--------------------------------------------------------------------- 
function CoffStringTableCount           (            const
InStringTable : PSYMBOL_STRING_TABLE           )         : DWORD;  {
used to determine the number of strings in the strings table but, this
   }  { function is _not_ currently used and it is usually removed by
the linker  } var  StringTableSize : DWORD;   StringPtr       :
pchar;  StringLength    : DWORD;  StringCount     : DWORD;  
ScannedBytes    : DWORD; begin  result      := 0;   StringCount := 0;
         // when done counting, result will have this value   with
InStringTable^ do  begin    StringTableSize := Size;     StringPtr   
  := Characters;     ScannedBytes    := pbyte(StringPtr) -
pbyte(InStringTable);     // verify that the number of scanned bytes
equals the size of the size field    // in the string table.  if that
is not the case then a change has been made    // in the record
definition of a string table which may prevent this routine    // from
working properly.     if ScannedBytes <> sizeof(Size)                
               then exit;  end;   while TRUE do  begin    if
IsBadStringPtrA(StringPtr, STRING_MAX_LENGTH)                then
exit;     StringLength    := strlen(StringPtr);    ScannedBytes    +=
StringLength + 1; // account for the null terminator     if
ScannedBytes >= StringTableSize                              then
break;     inc(StringCount);     StringPtr       += StringLength + 1; 
end;   result            := StringCount;      // return the count of
stringsend; 
I posted the data type definitions used by the code in the previous
post.

If you have any questions, just ask.

HTH.

----------------------------------------------------------------------
440bx · 2023-10-26 21:05

Attached to this post is the source code of a program that calculates
the size of its COFF symbol table.  Note that it does not use the
sections to figure it out.

Compile it with debugging information to walk through the code
(recommendation: DWARF)

HTH.

----------------------------------------------------------------------
KodeZwerg · 2023-10-27 01:13

Quote from: msintle on October 25, 2023, 02:54:09 pmQuote from:
KodeZwerg on October 25, 2023, 04:51:49 amTo OP:
I will try to port my way, pure WinAPI, to Lazarus and send source
here.

Perhaps you can also paste that code here and I will see if I can make
any progress with it, if you haven't yet had time to port it. A pure
WinAPI approach does sound good indeed.Here is my way, sorry for scary
looking, its cutted out of a really big method that analyze many
aspects of Executable data.
Tested and works with 32/64 bit files.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1; {$APPTYPE CONSOLE} uses  Windows, SysUtils,
Classes; function EndOfPE(const AFilename: string): UInt64;var  FS:
TStream;  dwSig: DWORD;  wSig: Word;  i: Integer;  HeaderDos:
IMAGE_DOS_HEADER;  HeaderFile: IMAGE_FILE_HEADER;  HeaderOptional64:
IMAGE_OPTIONAL_HEADER64;  HeaderOptional32: IMAGE_OPTIONAL_HEADER32; 
HeaderSection: array of IMAGE_SECTION_HEADER;  PHeaderSection: array
of PImageSectionHeader;  Is64Bit: Boolean;begin 
SetLastError(ERROR_SUCCESS);  Result := 0;  FS :=
TFileStream.Create(AFileName, fmOpenRead or fmShareDenyNone);  try   
FS.Read(HeaderDos, SizeOf(IMAGE_DOS_HEADER));    if (HeaderDos.e_magic
= IMAGE_DOS_SIGNATURE) then      begin        try         
FS.Seek(HeaderDos._lfanew, soFromBeginning);          FS.Read(wSig,
SizeOf(wSig));          FS.Seek(HeaderDos._lfanew, soFromBeginning); 
       FS.Read(dwSig, SizeOf(dwSig));          if (HeaderDos.e_magic
<> IMAGE_DOS_SIGNATURE) then            Exit;          if (wSig <>
IMAGE_NT_SIGNATURE) then            Exit;          FS.Read(HeaderFile,
SizeOf(IMAGE_FILE_HEADER));          if
(HeaderFile.SizeOfOptionalHeader > 0) then            begin          
  i := FS.Position;              FS.Read(wSig, SizeOf(wSig));        
    Is64Bit := (wSig = IMAGE_NT_OPTIONAL_HDR64_MAGIC);             
FS.Seek(i, soFromBeginning);              FS.Read(dwSig,
SizeOf(dwSig));              FS.Seek(i, soFromBeginning);            
if Is64Bit then                FS.Read(HeaderOptional64,
SizeOf(IMAGE_OPTIONAL_HEADER64))              else               
FS.Read(HeaderOptional32, SizeOf(IMAGE_OPTIONAL_HEADER32));          
end;          if (HeaderFile.NumberOfSections > 0) then           
begin              try                SetLength(HeaderSection,
HeaderFile.NumberOfSections);                SetLength(PHeaderSection,
HeaderFile.NumberOfSections);                for i := 1 to
HeaderFile.NumberOfSections do                  begin                
  FS.Read(HeaderSection[i - 1], SizeOf(IMAGE_SECTION_HEADER));       
           PHeaderSection[i - 1] := @HeaderSection[i - 1];           
       with PHeaderSection[i - 1]^ do                      if
(PointerToRawData + SizeOfRawData > Result) then                     
 Result := PointerToRawData + SizeOfRawData;                  end;   
         except              end;            end;        finally     
 end;      end;  finally    FS.Free;  end;end; begin 
WriteLn(EndOfPE(ParamStr(0)));  ReadLn;end.

----------------------------------------------------------------------
440bx · 2023-10-27 02:47

What I don't like about using the sections to determine the size of
the PE part of the file is that, that method does not provide any
information as to what kind of data follows the PE part.

Normally, in a properly constructed PE file, if there is a COFF symbol
table then the pointer to the symbol table will have a non-zero value
and that value _must_ be the size of the PE part.

Also, it would be fair for an antivirus to consider an exe to be
infected if there is _unreferenced_ data following the PE part. 
That's fair because the antivirus cannot simply assume that whatever
follows it is "benign". Not to mention that simply appending data to
PE part does not scale (what if there are two different blocks of data
to be appended by different utilities that know nothing of one another
?... that method doesn't cut that mustard.)

----------------------------------------------------------------------
msintle · 2023-10-28 10:40

Quote from: KodeZwerg on October 27, 2023, 03:13:05 amQuote from:
msintle on October 25, 2023, 02:54:09 pmQuote from: KodeZwerg on
October 25, 2023, 04:51:49 amTo OP:
I will try to port my way, pure WinAPI, to Lazarus and send source
here.

Perhaps you can also paste that code here and I will see if I can make
any progress with it, if you haven't yet had time to port it. A pure
WinAPI approach does sound good indeed.Here is my way, sorry for scary
looking, its cutted out of a really big method that analyze many
aspects of Executable data.
Tested and works with 32/64 bit files.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1; {$APPTYPE CONSOLE} uses  Windows, SysUtils,
Classes; function EndOfPE(const AFilename: string): UInt64;var  FS:
TStream;  dwSig: DWORD;  wSig: Word;  i: Integer;  HeaderDos:
IMAGE_DOS_HEADER;  HeaderFile: IMAGE_FILE_HEADER;  HeaderOptional64:
IMAGE_OPTIONAL_HEADER64;  HeaderOptional32: IMAGE_OPTIONAL_HEADER32; 
HeaderSection: array of IMAGE_SECTION_HEADER;  PHeaderSection: array
of PImageSectionHeader;  Is64Bit: Boolean;begin 
SetLastError(ERROR_SUCCESS);  Result := 0;  FS :=
TFileStream.Create(AFileName, fmOpenRead or fmShareDenyNone);  try   
FS.Read(HeaderDos, SizeOf(IMAGE_DOS_HEADER));    if (HeaderDos.e_magic
= IMAGE_DOS_SIGNATURE) then      begin        try         
FS.Seek(HeaderDos._lfanew, soFromBeginning);          FS.Read(wSig,
SizeOf(wSig));          FS.Seek(HeaderDos._lfanew, soFromBeginning); 
       FS.Read(dwSig, SizeOf(dwSig));          if (HeaderDos.e_magic
<> IMAGE_DOS_SIGNATURE) then            Exit;          if (wSig <>
IMAGE_NT_SIGNATURE) then            Exit;          FS.Read(HeaderFile,
SizeOf(IMAGE_FILE_HEADER));          if
(HeaderFile.SizeOfOptionalHeader > 0) then            begin          
  i := FS.Position;              FS.Read(wSig, SizeOf(wSig));        
    Is64Bit := (wSig = IMAGE_NT_OPTIONAL_HDR64_MAGIC);             
FS.Seek(i, soFromBeginning);              FS.Read(dwSig,
SizeOf(dwSig));              FS.Seek(i, soFromBeginning);            
if Is64Bit then                FS.Read(HeaderOptional64,
SizeOf(IMAGE_OPTIONAL_HEADER64))              else               
FS.Read(HeaderOptional32, SizeOf(IMAGE_OPTIONAL_HEADER32));          
end;          if (HeaderFile.NumberOfSections > 0) then           
begin              try                SetLength(HeaderSection,
HeaderFile.NumberOfSections);                SetLength(PHeaderSection,
HeaderFile.NumberOfSections);                for i := 1 to
HeaderFile.NumberOfSections do                  begin                
  FS.Read(HeaderSection[i - 1], SizeOf(IMAGE_SECTION_HEADER));       
           PHeaderSection[i - 1] := @HeaderSection[i - 1];           
       with PHeaderSection[i - 1]^ do                      if
(PointerToRawData + SizeOfRawData > Result) then                     
 Result := PointerToRawData + SizeOfRawData;                  end;   
         except              end;            end;        finally     
 end;      end;  finally    FS.Free;  end;end; begin 
WriteLn(EndOfPE(ParamStr(0)));  ReadLn;end.
Thank you, but this doesn't work.

I've tried both with GUI and console Win32 apps, the result is again
underreporting the file size by quite a margin.

----------------------------------------------------------------------
msintle · 2023-10-28 10:43

Quote from: 440bx on October 26, 2023, 11:05:06 pmAttached to this
post is the source code of a program that calculates the size of its
COFF symbol table.  Note that it does not use the sections to figure
it out.

Compile it with debugging information to walk through the code
(recommendation: DWARF)

HTH.

Thank you so much for this.

It seems like the project you attached compiles a 64 bit Windows
command line app, with which I was able to reproduce your results.

However with my 32 bit test cases, I ran into the attached issue at
compile time.

Any thoughts on how to fix the assembler code therein?

----------------------------------------------------------------------
msintle · 2023-10-28 10:50

Quote from: 440bx on October 27, 2023, 04:47:24 amWhat I don't like
about using the sections to determine the size of the PE part of the
file is that, that method does not provide any information as to what
kind of data follows the PE part.

Normally, in a properly constructed PE file, if there is a COFF symbol
table then the pointer to the symbol table will have a non-zero value
and that value _must_ be the size of the PE part.

Also, it would be fair for an antivirus to consider an exe to be
infected if there is _unreferenced_ data following the PE part. 
That's fair because the antivirus cannot simply assume that whatever
follows it is "benign". Not to mention that simply appending data to
PE part does not scale (what if there are two different blocks of data
to be appended by different utilities that know nothing of one another
?... that method doesn't cut that mustard.)

Your reservations aside, I commented out the asm code, and your code
worked 100% perfectly in both my GUI and console apps, reporting the
file size perfectly.

A massive thank you!!!

EDIT: Sadly, it still doesn't quite work

Appending data to the end of the file, we simply get the full file
size, instead of the file size for the executable itself alone.

So this is really no different than calling an ordinary file size
query in practical use?

----------------------------------------------------------------------
440bx · 2023-10-28 11:01

Quote from: msintle on October 28, 2023, 12:43:54 pmThank you so much
for this.

It seems like the project you attached compiles a 64 bit Windows
command line app, with which I was able to reproduce your results.

However with my 32 bit test cases, I ran into the attached issue at
compile time.

Any thoughts on how to fix the assembler code therein?
You're welcome.

The project I attached compiles in 32 and 64 bit without any problems
for me.

From the picture you posted, it seems you copied some of the code from
my program into your own project.  If that is what you did then you
missed copying the line  that says "{$ASMMODE        INTEL}" to let
the compiler know what assembly language it should use.  Also, you may
need to set Compiler Options -> Parsing -> Assembler style (-R) to
"Intel".  see the attached picture for 1000 words

With those two corrections your program should compile (presuming it
is free of syntax errors)

HTH.

----------------------------------------------------------------------
440bx · 2023-10-28 11:20

Quote from: msintle on October 28, 2023, 12:50:20 pmEDIT: Sadly, it
still doesn't quite work

Appending data to the end of the file, we simply get the full file
size, instead of the file size for the executable itself alone.

So this is really no different than calling an ordinary file size
query in practical use?
It's very important to understand the following: an executable is NOT
supposed to have _unreferenced_ data.  IOW, if somehow something (a
utility, a virus, a whathaveyou) manages to append data to an exe or
dll without referencing that data somewhere (in whatever place is
appropriate in the PE file - which is a reference to the data) then
that data cannot be presumed to be benign. 

To detect that some unreferenced data has been appended you will need
to iterate through the sections to determine what the file size
_should_ be, that is, the method you originally used.  That method
works in all cases but, it has the disadvantage that it does NOT tell
you if the appended data is legit or not.  If it's legit then it
should be referenced somewhere in the PE part of the file.

It is different than querying the O/S for the file size because, IF
(note the big IF) the PE file is put together correctly then the data
that has been appended should be pointed to by the "Pointer to symbol
table" field even though the data may not be a symbol table, which
makes the use of the field for that purpose a bit questionable but it
is the only way I know of to "declare" that data that is not part of
the PE data is present in the file.

If you want to be thorough then you can use your sections method AND
verify that the sections method yields the same value as that
specified in "Pointer to symbol table".  If the values differ then
that data is unreferenced and its purpose should be considered
dubious.

----------------------------------------------------------------------
440bx · 2023-10-28 14:02

Attached to this post is an "enhanced" version of the example I
previously posted.

This version calculates the PE size using the PointerToSymbolTable and
the sections method and it compares the results (which should be the
same.)

It also computes the size of the COFF symbol table and the string
table and compares the total calculated size to the total size
declared in the PE file.

It emits a warning if there is an inconsistency in sizes - which
occurs if a "utilty" simply appends data to the exe without creating a
reference to it in the PE part.

HTH.

----------------------------------------------------------------------
msintle · 2023-10-28 16:39

Quote from: 440bx on October 28, 2023, 01:01:35 pmQuote from: msintle
on October 28, 2023, 12:43:54 pmThank you so much for this.

It seems like the project you attached compiles a 64 bit Windows
command line app, with which I was able to reproduce your results.

However with my 32 bit test cases, I ran into the attached issue at
compile time.

Any thoughts on how to fix the assembler code therein?
You're welcome.

The project I attached compiles in 32 and 64 bit without any problems
for me.

From the picture you posted, it seems you copied some of the code from
my program into your own project.  If that is what you did then you
missed copying the line  that says "{$ASMMODE        INTEL}" to let
the compiler know what assembly language it should use.  Also, you may
need to set Compiler Options -> Parsing -> Assembler style (-R) to
"Intel".  see the attached picture for 1000 words

With those two corrections your program should compile (presuming it
is free of syntax errors)

HTH.

Correct, that pleased the compiler - thank you!

----------------------------------------------------------------------
msintle · 2023-10-28 16:47

Quote from: 440bx on October 28, 2023, 04:02:35 pmAttached to this
post is an "enhanced" version of the example I previously posted.

This version calculates the PE size using the PointerToSymbolTable and
the sections method and it compares the results (which should be the
same.)

It also computes the size of the COFF symbol table and the string
table and compares the total calculated size to the total size
declared in the PE file.

It emits a warning if there is an inconsistency in sizes - which
occurs if a "utilty" simply appends data to the exe without creating a
reference to it in the PE part.

HTH.

Wow, sounds like we've (well, you've) nailed it!

Please see this command line output:

Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  Directory
of c:\Users\c\Desktop\CoffSymTable2\lib\x86_64-win64 10/28/2023  07:40
PM    <DIR>          .10/28/2023  07:40 PM    <DIR>        
 ..10/28/2023  07:41 PM               495
CoffSymTable.compiled10/28/2023  07:41 PM           306,835
CoffSymTable.exe10/28/2023  07:41 PM            36,392
CoffSymTable.o10/28/2023  07:41 PM            19,220
libimp_80_kernel32.a10/28/2023  07:41 PM             8,420
libimp_80_ntdll.a10/28/2023  07:41 PM             2,052
_80_kernel32.o10/28/2023  07:41 PM            16,508
_80_kernel32.ppu10/28/2023  07:41 PM           147,600
_80_ntdll.o10/28/2023  07:41 PM            90,566 _80_ntdll.ppu      
       9 File(s)        628,088 bytes               2 Dir(s)
 213,113,245,696 bytes
free c:\Users\c\Desktop\CoffSymTable2\lib\x86_64-win64>copy /b
CoffSymTable.exe + CoffSymTable.o c.exeCoffSymTable.exeOverwrite
c.exe? (Yes/No/All): yCoffSymTable.o        1 file(s)
copied. c:\Users\c\Desktop\CoffSymTable2\lib\x86_64-win64>c          
          Calculate the size of the COFF symbol table                
                Program file size :    343227                        
         size of PE data :    240128    size of entire COFF table
(entries + strings) :    103099                         PE size using
PE sections :    240128                           COFF symbol table
size :     21348                                string table size :  
 45359      -- program executed successfully --      press
ENTER/RETURN to end this program
So PE section size + COFF symbol table size + string table size
returns the desired actual binary size.

The other calculations still return the grown file size.

Once again, many thanks!!!

Much gratitude for all your kind and patient assistance.

----------------------------------------------------------------------
440bx · 2023-10-28 17:15

Quote from: msintle on October 28, 2023, 06:47:18 pmOnce again, many
thanks!!!

Much gratitude for all your kind and patient assistance.
You're welcome.

From the output shown, it looks like you appended 36,392 bytes to the
executable before running it.  I say that because the sum "COFF symbol
table size" + "string table size" should equal "size of entire COFF
table (entries + strings)".  The only way those don't match is if data
has been appended to the file.  In this case 103099 - (21348 + 45359)
= 36392 = number of appended bytes.

----------------------------------------------------------------------
msintle · 2023-12-14 11:08

I seem to have a new problem - does the algorithm above work if strip
has been done on the file in question using the Free Pascal strip
utility after a Windows executable has already been built?

----------------------------------------------------------------------
440bx · 2023-12-14 12:53

It should work because if strip removes the COFF symbol table (which
it probably does) then it should set the value "Pointer to symbol
table" to zero.  If it does not set it to zero then that is a bug in
the strip utility.

----------------------------------------------------------------------
msintle · 2023-12-16 14:32

The file sizes do come down really nicely after the strip, but they
all seem to be affected by the same issue.

I've tried the Free Pascal strip, and some tools from the Android NDK
that were already present on my system from RAD Studio / Visual Studio
installations - three strip.exe's so far, all with the same result.

It really is such a shame that we ultimately cannot use your brilliant
work again, this time due to bugs in external tools.

More constricting is the fact that it doesn't seem to be possible to
turn off debug information generation from the command line while
using lazbuild:

https://forum.lazarus.freepascal.org/index.php/topic,65521.msg499324.h
tml#msg499324

I honestly haven't had the chance to turn back to our raw test cases
(hoping to do so once crunch time here is over), but I strongly doubt
a thorough re-examination of the issue would yield different findings
where scenarios involving removal of debug information post-build are
concerned.

At least we tried

----------------------------------------------------------------------
440bx · 2023-12-16 16:42

FWIW:

When I tried the strip utility that was included with FPC v3.0.4, the
utility would not recognize the executable as a valid file to strip. 
I didn't investigate the problem any further since I didn't think it
was worth it.

When I tried the strip utility that is distributed with Lazarus I had
no problems at all.  Everything worked as it should have (with the
same executable that failed with the FPC distributed utility.)

Note: I only tried the 64bit version of the Lazarus distributed
version of strip.exe

----------------------------------------------------------------------
msintle · 2024-01-08 00:12

I was just able to check our base test cases due to the holidays!

I actually have great news - your code works perfectly fine with
stripped executables, too.

The only value that is reported by your testing code in this scenario
is:

Code: [Select]PE size using PE sections :    <whatever the true file
size is>
Clearly I made a mistake in converting your console code to my GUI app
after that, as it failed with stripped files.

So thank you once again, and I am surely grateful I double checked
this when I had the chance.

I hope your new year is off to a great start!

----------------------------------------------------------------------
440bx · 2024-01-08 05:15

Quote from: msintle on January 08, 2024, 01:12:06 amI hope your new
year is off to a great start!
Thank you and I hope that is the case for you as well.