FPImageException while loading Bitmap from resource stream lazarus / 39-Graphics ---------------------------------------------------------------------- Alex.Machado · 2023-09-30 04:36 Hi all, I'm trying to load a bitmap from a resource stream under Windows and the code is failing without any apparent reason. The same code would work fine in Delphi. The image is valid, the resource is linked to the EXE (Can be checked with any resource editor, like Resourcehacker). However seems to me that LCL is not able to load the bitmap header correctly. The error I get is always "FPImageException Bitmap with unknwon compression". Here is the test case that shows the problem. Is there any workaround or something that can be done to avoid this? Thanks in advance ---------------------------------------------------------------------- Dzandaa · 2023-09-30 10:09 Hi, Try with BGRABitmap instead: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,  BGRABitmap, BGRABitmapTypes;    function GetBitmap(AName: string): TBGRABitmap;var  RStream: TResourceStream;begin  AName := UpperCase(AName);  RStream := TResourceStream.Create(HInstance, AName, RT_RCDATA);  try    Result := TBGRABitmap.Create;    Result.LoadFromStream(RStream);  finally    FreeAndNil(RStream);  end;end; B-> ---------------------------------------------------------------------- wp · 2023-09-30 11:18 I studied the resource file with my hex editor and found that it in fact contains the bitmap as type RT_RCDATA which contains a full bitmap file beginning with the typical 'BM' signature which is the start of the BitmapFileHeader. When, on the other hand, I create a resourcefile with LazRes the bmp file is contained as resource type RT_BITMAP. Since here the image type is already known the first header of the bitmap structure (the BitmapFileHeader) is missing and the resource begins with the BitmapInfoHeader. Lazarus has a problem now. The class to be read is a TBitmap and makes Lazarus assume that the resource is type RT_BITMAP although the real resource type RT_RCDATA is specified in the reading command. Therefore it reads the BitmapFileHeader at the place where it expects the BitmapInfoHeader and gets incorrect data from now on... This is clearly a bug in the LCL resource access routines and should be reported (I see that you are new here, and I can report it for you). ---------------------------------------------------------------------- TRon · 2023-09-30 11:35 @wp: Have you verified with fpcres if that exposes the same behaviour ? afaik that should be used instead of lazres. nvm, it seems to do the same (wrong) thing. ---------------------------------------------------------------------- wp · 2023-09-30 12:42 No, it's not a matter of resource generation, but of resource reading. A possible solution: At the place where the BitmapInfoHeader is expected, but the BitmapFileHeader is found, the reader could check whether the first 2 bytes read are the bitmap signature 'BM'. If this is true it very probably has detected the BitmapFileHeader, must advance the stream position by the size of the BitmapFileHeader to get to the BitmapInfoHeader, and repeat from where it started reading. To be more specific: Open unit IntfGraphics (from folder lcl of the Lazarus installation)Find procedure TLazReaderDIB.InternalReadHead.Before the "begin" add a variable "Magic: Array[0..1] of ansiChar absolute BIH"After the "{$ENDIF}" closely after the "begin" insert the following highlighted linesCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  ...  Magic: array[0..1] of AnsiChar absolute BIH;begin  StreamStart := theStream.Position;  TheStream.Read(BIH.biSize,SizeOf(BIH.biSize));  {$IFDEF FPC_BIG_ENDIAN}  BIH.biSize := LEtoN(BIH.biSize);  {$ENDIF} // ------------------ wp: insert this ...  // Workaround for bitmaps in RT_RCDATA resources not being read correctly.  // These resources have the BitmapFileHeader at the place where the stream is  // here atm. The stream expects a TBitmapInfoHeader or TBitmapCoreHeader, though.  if (Magic[0] = 'B') and (Magic[1] = 'M') then  begin    TheStream.Position := StreamStart + SizeOf(TBitmapFileHeader);    TheStream.Read(BIH.biSize,SizeOf(BIH.biSize));    {$IFDEF FPC_BIG_ENDIAN}    BIH.biSize := LEtoN(BIH.biSize);    {$ENDIF}  end;// ------------------- ... until here.   if BIH.biSize = 12 Kind of "brute force"... but it seems to work. I am aware that the assumption about having the complete file structure has been made also at another place, and this makes me think that this change introduces another issue somewhere else... ---------------------------------------------------------------------- paweld · 2023-09-30 13:22 you can get around this by using TMemoryStream:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function GetBitmap(AName: string): Graphics.TBitmap;var  RStream: TResourceStream;  ms: TMemoryStream;begin  AName := UpperCase(AName);  RStream := TResourceStream.Create(HInstance, AName, RT_RCDATA);  ms := TMemoryStream.Create;  ms.CopyFrom(RStream, 0);  ms.Position := 0;  try    Result := Graphics.TBitmap.Create;    Result.LoadFromStream(ms, ms.Size);  finally    FreeAndNil(RStream);    ms.Free;  end;end; ---------------------------------------------------------------------- TRon · 2023-09-30 13:24 Quote from: wp on September 30, 2023, 02:42:58 pmNo, it's not a matter of resource generation, but of resource reading. Yeah, I can see now (took me a while to verify). Thank you for the correction. QuoteA possible solution: At the place where the BitmapInfoHeader is expected, but the BitmapFileHeader is found, the reader could check whether the first 2 bytes read are the bitmap signature 'BM'. If this is true it very probably has detected the BitmapFileHeader, must advance the stream position by the size of the BitmapFileHeader to get to the BitmapInfoHeader, and repeat from where it started reading. I see. imho LoadFromStream() should be no different than LoadFromFile() other than the one is reading from a file while the other reads the same data from a stream. QuoteKind of "brute force"... but it seems to work. I am aware that the assumption about having the complete file structure has been made also at another place, and this makes me think that this change introduces another issue somewhere else... Yes, I seem to remember that the issue was mentioned before (not able to locate it yet) and technically adding the workaround in dibreader would (also imho) not be the correct location (the dib itself does/should not contain the TBitMapFileHeader). But perhaps I am oversimplifying things right now. ---------------------------------------------------------------------- TRon · 2023-09-30 13:25 Quote from: paweld on September 30, 2023, 03:22:42 pmyou can get around this by using TMemoryStream: There are multiple workarounds possible, including yours. Fact is that it is (according to TS) not Delphi compatible which is more of an issue I guess. Though I do not understand why TS would want to store a bitmap as a DATA resource. Using a BITMAP resource would make thing much more convenient. ---------------------------------------------------------------------- TRon · 2023-09-30 14:24 hmz, it seems even worse. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TBitmap.LoadFromStream(AStream: TStream; ASize: Cardinal);var  S: THeaderStream;  Header: TBitmapFileHeader;begin  if AStream is TResourceStream then  begin    FillChar(Header, SizeOf(Header), 0); { Create a BMP header ordered as it would be on disc, noting that if the CPU  is big-endian this will be the "wrong way round" for numeric operations.      }     {$IFNDEF ENDIAN_BIG}    Header.bfType := $4d42;    Header.bfSize := SizeOf(Header) + ASize;    {$ELSE}    Header.bfType := $424d;    Header.bfSize := swap(SizeOf(Header) + ASize);    {$ENDIF}     //Header.bfOffBits := 0; //data imediately follows     S := THeaderStream.Create(AStream, @Header, SizeOf(Header));    try      inherited LoadFromStream(S, SizeOf(Header) + ASize);    finally      S.Free;    end;  end  else  inherited LoadFromStream(AStream, ASize);end; Why ? This makes it only more confusing and seems more like a hack to me. It is simply wrong to assume that every stream that is of type resource requires a BMP header. fwiw: TBitmap.LoadFromResourceXX() functions do not even seem to search RT_DATA resources and as such return a neat: "resource not found". ---------------------------------------------------------------------- wp · 2023-09-30 14:41 Bugreport: https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/40529 ---------------------------------------------------------------------- wp · 2023-09-30 14:51 I think the TBitmap.LoadFromStream looks like the place where the correct fix should be done: In pseudo-code Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TBitmap.LoadFromStream(AStream: TStream; ASize: Cardinal);...begin  if (AStream is TResourceStream) and (GetResourceType = RT_BITMAP) then  begin    ...  end else    inherited LoadFromStream(AStream, ASize);end; But how to determine the ResourceType here? [EDIT] Ah well... The ResourceStream does not begin at the bitmap itself but contains some additional header data. Therefore, above pseudo-code is too simple. ---------------------------------------------------------------------- TRon · 2023-09-30 14:53 Thank you for the bugreport wp, and I agree with your presented solution. The earlier fixes that you proposed is afaik only required because of the double header. The only option I can see for now is verifying the presence of $4d42 in the memory pointer but it is still ugly. ---------------------------------------------------------------------- Alex.Machado · 2023-09-30 20:15 Thank you all for the responses. I could possibly set the resource as a BITMAP but for now I'm more than happy to use a TMemoryStream as suggested by @paweld. I missed from the source code of the TBitmap class itself that loading directly from a TResourceStream would assume a specific type of resource... Cheers, ---------------------------------------------------------------------- TRon · 2023-09-30 20:25 @wp: The more time I have to think about this, the more I am inclined to say that it should read: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TBitmap.LoadFromStream(AStream: TStream; ASize: Cardinal);begin  inherited LoadFromStream(AStream, ASize);end; Or to be more precise: no overload at all for LoadFromStream. Perhaps at the time the workaround was introduced the resource related routines were not up to par (at least the resource compiler wasn't) so it would probably justify such a workaround back then. It is imho more logical that the LoadFromResourceXX() methods are there for a particular reason (e.g. to load a bitmap from a resource). Having said that, I do not have Delphi to be able to verify (but the documentation on Delphi's LoadFromStream also only mentions a bitmap (embarcadeo website is down for me atm)). ---------------------------------------------------------------------- wp · 2023-09-30 20:41 Quote from: TRon on September 30, 2023, 10:25:29 pm@wp: The more time I have to think about this, the more I am inclined to say that it should read: [...] Or to be more precise: no overload at all for LoadFromStream. I don't think so. Because when the stream is a resource stream the stream does not start at the BitmapFileHeader ('BM') but is offset by a few bytes and, in case of the RT_BITMAP, there is not BitmapFileHeader at all. ---------------------------------------------------------------------- TRon · 2023-09-30 21:05 Quote from: wp on September 30, 2023, 10:41:34 pmI don't think so. Because when the stream is a resource stream the stream does not start at the BitmapFileHeader ('BM') but is offset by a few bytes and, in case of the RT_BITMAP, there is not BitmapFileHeader at all. afaik that statement assumes that loadfromstream should be able to read from whatever stream and decide, based on the binary data in that stream, what type of data needs to be loaded (which seem a bit counterproductive in my eyes). Again, to make sure as I can't test myself, I don't think the Delphi implementation actually does that. If you truly want to load from a resource, then there are specific loadfromresource functions, that already know about the underlying binary data and as such can act accordingly. But perhaps I am overlooking something ? You are more versed in the matter than I am. ---------------------------------------------------------------------- wp · 2023-09-30 21:22 Here is a modified TBitmap.LoadFromStream which works with normal streams, as well as with RT_BITMAP and RT_RCDATA resource streams: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TBitmap.LoadFromStream(AStream: TStream; ASize: Cardinal);var  S: THeaderStream;  Header: TBitmapFileHeader;  P: Cardinal;  Magic: array[0..1] of AnsiChar;begin  if AStream is TResourceStream then  begin    P := AStream.Position;    AStream.Read(Magic, 2);    AStream.Position := P;    if (Magic[0] = 'B') and (Magic[1] = 'M') then      { Handle the special case of RC_RTDATA resource which contains a        complete, functional bitmap structure. }      inherited LoadfromStream(AStream, ASize)    else    begin      { Normal case of an RT_BITMAP resource lacking the BitmapFileHeader }    FillChar(Header, SizeOf(Header), 0);      { Create a BMP header ordered as it would be on disc, noting that if the CPU        is big- endian this will be the "wrong way round" for numeric operations. }    {$IFNDEF ENDIAN_BIG}      Header.bfType := $4d42;      Header.bfSize := SizeOf(Header) + ASize;      {$ELSE}      Header.bfType := $424d;    Header.bfSize := swap(SizeOf(Header) + ASize);      {$ENDIF}      //Header.bfOffBits := 0; //data imediately follows       S := THeaderStream.Create(AStream, @Header, SizeOf(Header));      try      inherited LoadFromStream(S, SizeOf(Header) + ASize);      finally      S.Free;      end;    end;  end  else    inherited LoadFromStream(AStream, ASize);end; It first checks whether the stream is a resource stream. If not, the inherited method is called. Otherwise it is a resource stream, and there are two cases: It could be RT_BITMAP or RT_RCDATA. Since the stream is at the very beginning of the bitmap structure two bytes are read from the input stream here: if they are 'BM' we have a fully functional bitmap (originating from RT_RCDATA) and the bitmap can be read as usual. If the two bytes are different we have a RT_BITMAP stream and reading must follow the original method, i.e. it introduces a THeaderStream to start reading at the TFileInfoHeader. ---------------------------------------------------------------------- wp · 2023-09-30 23:28 Quote from: TRon on September 30, 2023, 11:05:52 pmIf you truly want to load from a resource, then there are specific loadfromresource functions, that already know about the underlying binary data and as such can act accordingly. There are TBitmap.LoadFromResourceName and .LoadFromResourceID which get the raw stream data from the resource, but loading the stream into the bitmap pixel array again leads to TBitmap.LoadFromStream. ---------------------------------------------------------------------- TRon · 2023-10-01 01:10 Quote from: wp on October 01, 2023, 01:28:05 amThere are TBitmap.LoadFromResourceName and .LoadFromResourceID which get the raw stream data from the resource, but loading the stream into the bitmap pixel array again leads to TBitmap.LoadFromStream. Isn't that the heart/origin of the matter ? In these methods the code knows what to expect, and can act accordingly. Imho LoadFromStream is not the proper place to be doing that. So if any would it not be a better solution to incorporate the insertion of the header in those methods ? Or better yet introduce a loadfromresource method that does that so that both methods can invoke it (write once, use many) ? That way the LoadFromStream method can be a normal/simple implementation again. fwiw: I agree that the code you presented in your previous post does work and would be the proper way to go forward if there is currently no need for another solution. (afaik the presented code is still not 100% foolproof but it wasn't before either -> not your fault) ---------------------------------------------------------------------- wp · 2023-10-01 22:38 Quote from: TRon on October 01, 2023, 03:10:14 amQuote from: wp on October 01, 2023, 01:28:05 amThere are TBitmap.LoadFromResourceName and .LoadFromResourceID which get the raw stream data from the resource, but loading the stream into the bitmap pixel array again leads to TBitmap.LoadFromStream. Isn't that the heart/origin of the matter ? In these methods the code knows what to expect, and can act accordingly. Imho LoadFromStream is not the proper place to be doing that. So if any would it not be a better solution to incorporate the insertion of the header in those methods ? Or better yet introduce a loadfromresource method that does that so that both methods can invoke it (write once, use many) ? That way the LoadFromStream method can be a normal/simple implementation again. AFAIK that's not what the resource access routines are doing. When a RT_BITMAP resource is written it is written without the BitmapFileHeader, and when it is read, it is sent without it. I guess there are ancient reasons in the early days of Windows why this was done so, maybe saving some memory - a few bytes in those days were as precious as a few Megabytes now. Most important: The resource stream is not able to reconstruct those partial data, it does not "know" about the file structure of a bitmap or whatever data type is handled. Therefore, the correct place to fix this is within the receiving class, here: in the TBitmap. Delphi does it in the same way. It also checks the 'BM' signature in the LoadFromStream method. ---------------------------------------------------------------------- TRon · 2023-10-02 02:56 Quote from: wp on October 02, 2023, 12:38:44 amDelphi does it in the same way. It also checks the 'BM' signature in the LoadFromStream method. So you were able to verify that Delphi is able to use LoadFromStream (by that I mean only LoadFromStream) to actually construct a valid bitmap file (as stored on disk) out of a resource stream so that it actually can be loaded by TBitmap (by using the rest of TBitmaps implemented methods/functions) ? In case they did then :facepalm: and further discussion would be of no (practical) use. ---------------------------------------------------------------------- Bart · 2023-10-02 09:23 Quote from: wp on October 02, 2023, 12:38:44 amDelphi does it in the same way. It also checks the 'BM' signature in the LoadFromStream method. I hope you did not look at the sourceode of the Delphi implementation? Bart ---------------------------------------------------------------------- wp · 2023-10-02 11:10 Well, the formulation was a bit inaccurate: They do not "reconstruct" the header. There are two methods: Delphi's TBitmap.LoadFromResourceName calls LoadDIB which starts reading at the BitmapInfoHeader/BitmapCoreHeader (after the BitmapFileHeader), and TBitmap.LoadFromStream reads the BitmapFileHeader first and then continues with LoadDIB. As a consequence Delphi cannot use LoadFromStream to read RT_BITMAP resources provided by a ResourceStream - maybe this is intentional (because RT_BITMAP is "known" to provide an incomplete bitmap stream), but I tend to say that it is a bug because it makes the resourcetype parameter of the TResourceStream constructor obsolete, at least for bitmaps. Lazarus, on the other hand, uses an internal "THeaderStream" class to fake (reconstruct) the presence of a TBitmapFileHeader for the main reading done by fcl-image. Similar to Delphi's LoadDIB, the LCL provides also a TLazReaderDIB starting at the BitmapInfo/CoreHeader, as well as a descendant TLazReaderBMP reading the FileHeader first. At first sight it should be possible to adjust the TBitmap.GetReaderClass method to one of these two readers, depending on the caller. But GetReaderClass is a class method, and I don't know an easy way of telling this method whether the reader is needed for LoadFromStream or for LoadFromResourceName. And even if it could be done, it would leave us in the same situation as Delphi where RT_BITMAP resources cannot be read by LoadFromStream. ---------------------------------------------------------------------- TRon · 2023-10-03 02:06 @Bart: Don't worry too much. Most of that information that is/was presented can easily be obtained by running an executable through a debugger or disassembler. Unless you count that as "having seen the source-code". For me (but also for most if not all compilers) it is not the same as having looked at the original code. Usually a compiler allows the binary produced data to be distributed without it being copyrighted by the original author/copyright holder of the used compiler. And thank heavens for that as that would be an instant wet dream for all copyright trolls out there. Quote from: wp on October 02, 2023, 01:10:21 pmWell, the formulation was a bit inaccurate: Yeah, sorry about that but since I can only check fpc's implementation the conclusion from what I've read is that it needs to add the header to the stream in order for the rest of the code to be able to properly load the resource from the stream because the fileheader is expected to exist. It is almost impossible to write that down without a whole lot of cringe feelings about that (the requirement to add something to raw data so that the rest of the code is able to properly process it *chills running down spine*). QuoteThey do not "reconstruct" the header. Thank you very much for that detailed process of how Delphi seem to be doing things. That process makes a whole lot more sense to me as there is no need to add data to anything but just checking if something exist or not and the rest of the code acting accordingly (e.g. skip loading the fileheader but start processing the data that does exist (or is suppose to exist)) QuoteAs a consequence Delphi cannot use LoadFromStream to read RT_BITMAP resources provided by a ResourceStream - maybe this is intentional (because RT_BITMAP is "known" to provide an incomplete bitmap stream), but I tend to say that it is a bug because it makes the resourcetype parameter of the TResourceStream constructor obsolete, at least for bitmaps. I think we have a different opinion on that  For me it makes absolute sense that loadfromstream is not able to load any other arbitrary data to construct a bitmap out of it. And I know I exaggerate with the example here but before you know it people expect to feed it a stream of jpeg data and for that data to be converted into a proper bitmap automagically because, why not ? If you so must then you can use the designated loadfromresourceXXX functionality, and if that is not enough then perhaps introduce a loadfromresourcestream method that directly loads the resource data stream without the needs to use (fall back to) existing functionality ? Btw, have you noticed the implementation of function CreateBitmapFromResourceName inside unit graphics ? QuoteLazarus, on the other hand, uses an internal "THeaderStream" class to fake (reconstruct) the presence of a TBitmapFileHeader for the main reading done by fcl-image. Yeah I've noticed that Lazarus relies on the LoadFromStream method but not have looked at the readers in detail. fwiw I understand why it would have ended up in LoadFromStream as most (if not all) the other graph formats are loaded from a resource of type RT_DATA only and that is always the raw data as stored on disk. QuoteBut GetReaderClass is a class method, and I don't know an easy way of telling this method whether the reader is needed for LoadFromStream or for LoadFromResourceName. And even if it could be done, it would leave us in the same situation as Delphi where RT_BITMAP resources cannot be read by LoadFromStream. afaik as it stands now only for LoadFromStream() /but/ LoadFromResourceName() depends on LoadFromStream() so indirectly for that function as well. That is the reason this issue exist in the first place. I have not seen any (online) example in Delphi that directly loads a bitmap from a (bitmap)resource using LoadFromStream, Even embarcadero's examples (that are publicly available) uses the LoadFromResourceXXX functions in order to construct a bitmap out of a (bitmap) resource. So I personally would not regret it one bit if it wasn't possible (anymore) to load a bitmap resource using LoadFromStream() but that is a pure personal opinion on the matter. Just to avoid misunderstandings RT_DATA is ofc just raw data and can be loaded using LoadFromStream. In one of my earlier replies I stated (I had it removed after I posted) to opt for overloading/reintroducing the LoadFromResourceXXX functionality to do it more in the style Delphi seems to be doing (ans I could now more or less verify it for as far as I was able to understand from your detailed descriptions) but that does have some implications such as not being able to use LoadFromStream for loading a bitmap resource anymore and some care needs to be taken for other formats that can be loaded by TBitmap and that supports loading from a resource other than RT_DATA. I haven't had time yet to look into that into more detail (which also requires some better understanding of the reader). fwiw it is not about beating the dead horse and trying for someone (else) to revive that horse in case it isn't necessary. Your otherwise excellent solution works as expected/intended just not how I would have done it (and according to your findings Delphi doesn't seem to do so either). ---------------------------------------------------------------------- Bart · 2023-10-03 14:20 Quote from: TRon on October 03, 2023, 04:06:04 am@Bart: Don't worry too much. If wp did look at the source code, everything that is based upon his knowledge now is tainted and cannot be in LCL. And if so, all references to this must also be removed from the bugtracker. So yes, it is something to worry about. (We've had similar problems in the past.) Bart ---------------------------------------------------------------------- TRon · 2023-10-03 14:57 Quote from: Bart on October 03, 2023, 04:20:58 pmIf wp did look at the source code, everything that is based upon his knowledge now is tainted and cannot be in LCL. You have a point there (including the consequences) except that I believe that wp wasn't born yesterday. But even if he was then he is allowed to describe the process as long as he does not mention literal code (to us). ---------------------------------------------------------------------- wp · 2023-10-03 15:05 Yes, I did look at the source. But I did not publish the code, I just described the basic steps which in my eyes are rather trivial, there are only a few possibilities to do this. If you now consider this thread as being tainted you should ban every user having access to Delphi from the Lazarus development.