GZip library
lazarus / 31-LCL

----------------------------------------------------------------------
geraldholdsworth · 2021-05-28 13:19

Hi all,

I have been using the GZip library (found in ZStream) to decompress
GZip files. Most of the files I have come across only appear to have
the single block in the file, so this works (the basis for I found on
this forum):
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
Inflate(Source: String): TDIByteArray;var GZ     :
TGZFileStream; chunk  : TDIByteArray; cnt, i, buflen : Integer;const 
ChunkSize=4096; //4K chunksbegin //Initialise the
variables Result:=nil; chunk:=nil; //Open the stream try 
GZ:=TGZFileStream.Create(Source,gzOpenRead);  //This is our length
counter  buflen:=0;  //We'll be reading it in chunks 
SetLength(chunk,ChunkSize);  repeat   //Read in the next chunk 
 cnt:=GZ.Read(chunk[0],ChunkSize);   //Extend the buffer accordingly 
 SetLength(Result,buflen+cnt);   //Copy the chunk into the buffer 
 for i:=0 to cnt-1 do Result[buflen+i]:=chunk[i];   //Increase the
buffer length counter   inc(buflen,cnt);   //Until we are done  until
cnt<ChunkSize;  //Free up the
stream except end; GZ.Free;end; (TDIByteArray is defined as 'array of
Byte')

However, I have come across a file which has multiple blocks in the
file (nearly 150). The above procedure falls over in decompressing
this file (it actually only decompresses the first two). At the time I
never knew the format of GZip files, but after researching I found out
that they can be stored in blocks, each block having a header and
footer. Therefore, each block can act as a GZip file in it's own
right.

With this in mind I hatched a plan to get around this failing by
separating such files into their own separate files, each only having
a single block, decompressing each one in turn and patching them back
together. So, this appears to work on the file I found (and single
block ones too, as well as files which are not GZipped):
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
Inflate(filename: String): TDIByteArray; function L_Inflate(Source:
String): TDIByteArray; var  GZ     : TGZFileStream;  chunk  :
TDIByteArray;  cnt,  i,  buflen : Integer; const   ChunkSize=4096;
//4K chunks begin  //Initialise the variables  Result:=nil; 
chunk:=nil;  //Open the stream  try 
 GZ:=TGZFileStream.Create(Source,gzOpenRead);   //This is our length
counter   buflen:=0;   //We'll be reading it in chunks 
 SetLength(chunk,ChunkSize);   repeat    //Read in the next chunk   
cnt:=GZ.Read(chunk[0],ChunkSize);    //Extend the buffer accordingly 
 SetLength(Result,buflen+cnt);    //Copy the chunk into the buffer   
for i:=0 to cnt-1 do Result[buflen+i]:=chunk[i];    //Increase the
buffer length counter    inc(buflen,cnt);    //Until we are done 
 until cnt<ChunkSize;   //Free up the stream  except  end; 
GZ.Free; end;var F        : TFileStream; buffer, inflated :
TDIByteArray; ptr,i,old: Cardinal; blockptrs: array of Cardinal; fn  
   : String;begin buffer   :=nil; blockptrs:=nil; inflated
:=nil; Result   :=nil; //Read in the entire file try 
F:=TFileStream.Create(filename,fmOpenRead or fmShareDenyNone); 
SetLength(buffer,F.Size); 
F.Read(buffer[0],F.Size); except end; F.Free; //First, is it actually
a GZip file? if(buffer[$00]=$1F)and(buffer[$01]=$8B)and(buffer[$02]=$0
8)then begin  //Count how many blocks and make note of their
positions  for ptr:=0 to Length(buffer)-10 do 
 if(buffer[ptr]=$1F)and(buffer[ptr+1]=$8B)and(buffer[ptr+2]=$08)then 
 begin    //Make a note of the position   
SetLength(blockptrs,Length(blockptrs)+1);   
blockptrs[Length(blockptrs)-1]:=ptr;   end; end; //Separate each
block, if more than one if Length(blockptrs)>1 then begin  //Add the
file end to the end of the block pointers 
SetLength(blockptrs,Length(blockptrs)+1); 
blockptrs[Length(blockptrs)-1]:=Length(buffer);  //Set up the
container for the inflated file  SetLength(Result,0);  //Get a
temporary filename  fn:=GetTempDir+ExtractFileName(filename); 
//Iterate through the pointers  for i:=0 to Length(blockptrs)-2 do 
begin   //Create the temporary file and write the block to it   try  
F:=TFileStream.Create(fn,fmCreate);   
F.Write(buffer[blockptrs[i]],blockptrs[i+1]-blockptrs[i]);   except 
 end;   F.Free;   //Inflate the block   inflated:=L_Inflate(fn); 
 old:=Length(Result); //Previous length of the inflated file 
 //Increase the inflated file buffer to accomodate 
 SetLength(Result,Length(Result)+Length(inflated));   //Move the
inflated data across   for ptr:=0 to Length(inflated)-1 do
Result[old+ptr]:=inflated[ptr];  end;  //Delete the temporary file  if
FileExists(fn) then DeleteFile(fn); end; //If just the one block, then
don't bother splitting if Length(blockptrs)=1 then
Result:=L_Inflate(filename); //If there are no blocks, then just
return the entire file if Length(blockptrs)=0 then Result:=buffer;end;
As you can see, the original code is now a sub-function of this one.
Incidentally, don't forget:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses
ZStream,StrUtils,SysUtils,Classes(I think that's all you need)

Just thought I'd 'park' this here for anyone else experiencing the
same issue, or if anyone wants to improve on it.

Cheers,

Gerald.

----------------------------------------------------------------------
ChrisR · 2021-05-28 14:04

I believe this issue is related to
  https://bugs.freepascal.org/view.php?id=36822
Your solution may help others who encounter this.

----------------------------------------------------------------------
geraldholdsworth · 2021-05-29 11:47

It certainly does sound like it. Although I never got an exception
after the second block was inflated, but that could've been handled by
the try...except block in the original code.