using bin2obj, how to create a gzip archive?
lazarus / 32-Linux

----------------------------------------------------------------------
robert rozee · 2023-05-22 17:21

hi,
    i'm using bin2obj (the utility supplied with FPC) with the -z
switch to compress then convert a binary file to a pascal array. this
uses the construct
CompStream:=TCompressionStream.Create(cldefault,Result) to perform the
compression, and according to this page
https://linux.die.net/man/1/bin2obj "zlib compression is used".

once the conversion to an array is completed, i want my program to be
able to:
1. write the array contents out to a file in /tmp,
2. use the unix gzip utility to uncompress the file,
3. use chmod to mark the file as runnable,
4. and then run it.

all works well, except that i can not for the life of me figure out
the exact require header and (particularly) footer needs to be added
to keep gzip happy! i started out working from info here:
https://unix.stackexchange.com/questions/22834/how-to-uncompress-zlib-
data-in-unix, but have now hit a brick wall.

has anyone else been down this path and know the final incantations
required?

cheers,
rob   :-)


here is what running my code produces:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ bin2obj -z -o
binary.inc -c bindata GFXterm[ *** hit compile button in IDE ***
]user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ ./project1    
 100%CRC  = 52 91 A9 E5 size = 20 42 36 00
user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ uncompress -l
GFXrun.gz         compressed        uncompressed  ratio
uncompressed_name            1062874             3555872  70.1%
GFXrunuser@DH61BE:~/pascal/tests/test 203 (bin2obj)$ uncompress
 GFXrun.gz gzip: GFXrun.gz: invalid compressed data--crc error gzip:
GFXrun.gz: invalid compressed data--length error 

and here is my source:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1; uses Sysutils; {$I binary.inc} // to create binary.inc
use:// bin2obj -z -o binary.inc -c bindata GFXterm const
CRC:int64=$e5a99152;         // generated by /bin/crc32     
FSz:int64=3555872;           // original file size function
I2C(I:int64; B:byte):char;begin  I:=I shr (B*8);  result:=chr(I and
$FF)end; var I:integer;    T:text; begin//assign(T, '/tmp/GFXrun'); 
assign(T, 'GFXrun.gz');              // work in local directory for
the moment  rewrite(T);   write(T, #$1f, #$8b, #$08, #$00,     // gzip
preamble from https://datatracker.ietf.org/doc/html/rfc1952         
 #$00, #$00, #$00, #$00);   for I:=low(bindata) to high(bindata) do 
begin    write(T,chr(bindata[I]));    if (I and $3)=0 then
write((I*100) div high(bindata):8, '%', #13)  end;  writeln(100:8,
'%');                 // compressed data written out to file  
write('CRC  = ');  for I:=0 to 3 do//for I:=3 downto 0 do  begin     
                         // crude way of adding CRC as footer   
write(T, I2C(CRC, I));    write(IntToHex(ord(I2C(CRC, I)), 2), #32) 
end;  writeln;   write('size = ');  for I:=0 to 3 do                 
  // crude way of adding original file size to footer//for I:=3 downto
0 do  begin    write(T, I2C(FSz, I));    write(IntToHex(ord(I2C(FSz,
I)), 2), #32)  end;  writeln;   close(T);                          
 // stop here as we can't uncompress it yet!!{ 
ExecuteProcess('/bin/gzip', '-d /tmp/GFXrun');               //
FAILS!!!  ExecuteProcess('/bin/chmod', '+x /tmp/GFXrun');            
 // works OK  ExecuteProcess('/tmp/GFXrun', '');                     
    // works OK  ExecuteProcess('/bin/rm', '/tmp/GFXrun') }          
       // works OKend.

----------------------------------------------------------------------
robert rozee · 2023-05-23 08:25

and, of course, in the light of day the solution is obvious: just use
gzip to do the compression!

the below works remarkably well, the only major catch is that the
default directory for the run application is /tmp.


cheers,
rob   :-)

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1; uses SysUtils; {$I binary.inc} // to create binary.inc
use:// gzip -k <binary>// bin2obj -o binary.inc -c bindata
<binary>.gz// rm <binary>.gz var TempName:string;         
 I:integer;           T:text; begin                                  
              // I:=bindata[0] end.  Randomize;  repeat   
TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5) 
until not (FileExists(TempName) or FileExists(TempName+'.gz')); 
assign(T, TempName+'.gz');  rewrite(T);                              
          // create temp file   for I:=low(bindata) to high(bindata)
do  begin    write(T,chr(bindata[I]));{ if (I and $FF)=0 then }
write((I*100) div sizeof(bindata):3, '%', #13)  end;  writeln(100:3,
'% : '+TempName);                     // .gz file created  close(T);  
writeln('uncompress image');  ExecuteProcess('/bin/gzip', ['-d',
TempName]);       // uncompress  writeln('apply chmod +x'); 
ExecuteProcess('/bin/chmod', ['+x', TempName]);      // mark runable 
writeln('run binary image');  ExecuteProcess(TempName, '');          
            // run binary  writeln('cleanup /tmp'); 
ExecuteProcess('/bin/rm', TempName);                 // delete binary 
writeln('finished OK')end. addendum1: slightly updated above code to
use GetTempFileName
addendum2: changed to random temp file name with clash detection
addendum3: removed -z parameter i'd accidentally left in comment

----------------------------------------------------------------------
marcov · 2023-05-23 09:12

Afaik gzip has an additional header before the zlib stream, with CRC32
etc. So that is not 100% the same.  In zstream you can clearly see
separate TCompressionStream and TGzipCompressionStream classes.

For a FPC only example, see the  "minigzip.pp" example in the
packages/paszlib/examples and the other files there

----------------------------------------------------------------------
robert rozee · 2023-05-23 09:58

Quote from: marcov on May 23, 2023, 11:12:44 amAfaik gzip has an
additional header before the zlib stream, with CRC32 etc. So that is
not 100% the same.  In zstream you can clearly see separate
TCompressionStream and TGzipCompressionStream classes.
aha, that makes sense! so by the looks of it, in
/usr/share/fpcsrc/3.2.2/utils/bin2obj.pp one merely needs to change
line 127 to create a .gz compatible output?

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   If
ComPressData then    begin//
 CompStream:=TCompressionStream.Create(cldefault,Result);   
CompStream:=TGZipCompressionStream.Create(cldefault,Result);   
Result:=CompStream;    end;
if it is in fact this easy, it could be a handy addition to the
bin2obj utility, controlled by a command line switch to select between
'zlib' and 'gzip' compatibility. in my present user case it doesn't
matter so much, as now that i understand what is going on i can just
use the gzip provided by linux.

next step is to find a method of binary-level patching, so that i can
create a difference file between two binaries (built against glib 2.34
and glib 'old') and create one or the other on-the-fly to be run
depending on the version of libc installed..


cheers,
rob   :-)