deBayering - Bayer to RGB conversion needed lazarus / 39-Graphics ---------------------------------------------------------------------- af0815 · 2020-02-20 13:52 I get a raw picture from a camera and i have to convert from SRGGB8 (or 16) to RGB. A (long) search give me no (native) implementation in pascal. In LazVideoCapture i found YUV to RGB. The camera works with this, but slow 1 FPS. With bayer i get a speed of 19 FPS. Tested with the v4l2 test utility Is there any implementation known of deBayering ? ---------------------------------------------------------------------- circular · 2020-02-21 13:36 I think it is not too complicated to write debayering. Here you can see the matrix of colors: https://en.wikipedia.org/wiki/Bayer_filter#/media/File:Bayer_pattern_o n_sensor_profile.svg Basically you need to interpolate the 3 color planes. First put the know values into the corresponding pixels in the final image. Interpolating between pixels is the mean of the know pixels. There are some edge cases to consider. ---------------------------------------------------------------------- marcov · 2020-02-21 14:38 debayering is going from bayer to rgb. I have some code but it is not that fast, but I lack a good test image. Do you have an example of your camera. Preferably with some object in primary colors, and a description of which is which. There are several different bayer formats, depending on which line is the first line. ---------------------------------------------------------------------- af0815 · 2020-02-21 19:43 V4L2 Test Bench says RGGB (8-Bit Bayer RGRG/GBGB) (70434752-0000-0010-8000-00aa003). Possible is also RG16 (16-Bit Bayer RGRG/GBGB). I have cameras and a running fpcVideo4L2. Cameras: DFK 33UX183 = 5.472×3.648 (20 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1 DFK 33UX226 = 4.000×3.000 (12 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1 from TheImageSource. What is a good Testimage :-) ?! I have made a picture converted to jpg and RawDate of RGGB. But i am not able to share it here in the forum because the 7z it is wit 8MB to large. https://drive.google.com/file/d/18zoyCfAWu8s2UGoqu9GgOKcMH1z4IDX8/view ?usp=sharing My Testcode to generate the raw data Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TFormVideoTmp.VideoFrameSynchronized(Sender: TObject;  Buffer: pointer; Size: integer; Error: boolean);begin   if (FScanInProgress) then exit;   FScanInProgress:=True; try   // V4L2_PIX_FMT_SRGGB8 for theImageSource possible    if Video.PixelFormat = V4L2_PIX_FMT_SRGGB8 then    begin      aMS.Clear;      aMS.Position:=0;      aMS.WriteBuffer(Buffer^,Size);      aMS.Position:=0;      aMS.SaveToFile('/tmp/test.raw');    end;     if Video.PixelFormat = V4L2_PIX_FMT_YUYV then    begin      BMP.BeginUpdate;      YUYV_to_BGRA(PLongWord(Buffer), PLongWord(BMP.RawImage.Data), BMP.Width*BMP.Height);      BMP.EndUpdate;    end;     if Video.PixelFormat = V4L2_PIX_FMT_MJPEG then    begin      aMS.Clear;    aMS.Position:=0;      aMS.WriteBuffer(Buffer^,Size);      aMS.Position:=0;      aImage.LoadFromStream(aMS,aImgReader);      aMS.Clear;      aMS.Position:=0;      aImage.SaveToStream(aMS,aImgWriter);      aMS.Position:=0;      BMP.BeginUpdate;      Move(aMS.Memory^,PByte(BMP.RawImage.Data)^,Video .Width*Video.Height*aImgWriter.Bpp);      BMP.EndUpdate;    end;  finally    FScanInProgress:=False;  end;end; ---------------------------------------------------------------------- marcov · 2020-02-22 15:12 I get an image that looks similar in about 50ms on a core i7-3770 using Delphi. (Delphi because I used some image routines from work for ease of use, but the procedure itself is compiler agnostic). The code could maybe get a bit faster, but for some real performance one would probably need SSE/AVX. Since the output image is about 60MB, it is about 1MB/ms, which is still way under move() performance, so it is not yet cache bound yet. The code originally may have come from OpenCV, considering the CV_ identifiers. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$O+}type  cvsize = packed record    width : cint;    height: cint;    end;  cvstatus=cint;  function icvBayer2BGR_8u_C1C3R( bayer0:pbbyte;bayer_st ep:cint;dst0:pbbyte;dst_step:cint;size:cvsize;code:cint):CVSTATUS; var blue : cint;    start_with_green : cint;    t0,t1 :cint;    bayer_end,    bayer,dst : pbbyte; label 123;begin    if (code= CV_BayerBG2BGR ) or (code= CV_BayerGB2BGR) then      blue:=-1    else      blue:=1;    if (code = CV_BayerGB2BGR) or (code =CV_BayerGR2BGR) then     start_with_green:=1    else     start_with_green:=0;//    int blue = code == CV_BayerBG2BGR || code == CV_BayerGB2BGR ? -1 : 1;  //  int start_with_green = code == CV_BayerGB2BGR || code == CV_BayerGR2BGR;    fillchar(dst0^,size.width*3*sizeof(dst0[0]),#0);  fillchar( (dst0 + (size.height - 1)*dst_step)^, size.width*3*sizeof(dst0[0]),#0 ); //    memset( dst0, 0, size.width*3*sizeof(dst0[0]) );//    memset( dst0 + (size.height - 1)*dst_step, 0, size.width*3*sizeof(dst0[0]) );    inc(dst0 ,dst_step + 3 + 1);    dec(size.height , 2);    dec(size.width , 2);    repeat//    for( ; size.height-- > 0; bayer0 += bayer_step, dst0 += dst_step )    bayer:=bayer0;    dst:=dst0;         bayer_end:=bayer+size.width;{      const uchar* bayer = bayer0;        uchar* dst = dst0;        const uchar* bayer_end = bayer + size.width; }         dst[-4] :=0;      dst[-3] :=0;        dst[-2] :=0;        dst[size.width*3-1] :=0;      dst[size.width*3]   :=0;        dst[size.width*3+1] :=0;         if( size.width <= 0 ) then            goto 123; // continue; // can't use contineu because of decrement operator abuse + for         if( start_with_green )<>0 then         begin            t0 := (ord(bayer[1]) + ord(bayer[bayer_step*2+1]) + 1) shr {>>} 1;          t1 := (ord(bayer[bayer_step]) + ord(bayer[bayer_step+2]) + 1) shr {>>} 1;            dst[-blue] := t0;            dst[0] := bayer[bayer_step+1];            dst[blue] := (t1);            inc(bayer);            inc(dst, 3);         end;         if( blue > 0 ) then         begin            repeat //           for( ; bayer <= bayer_end - 2; bayer += 2, dst += 6 )                 t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +                    ord(bayer[bayer_step*2+2]) + 2) shr {>>} 2;                t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +                      ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) shr {>>} 2;              dst[-1] := (t0);                dst[0] := (t1);              dst[1] := bayer[bayer_step+1];                 t0 := (ord(bayer[2]) + ord(bayer[bayer_step*2+2]) + 1) shr {>>} 1;              t1 := (ord(bayer[bayer_step+1]) + ord(bayer[bayer_step+3]) + 1) shr {>>} 1;                dst[2] := (t0);                dst[3] := bayer[bayer_step+2];                dst[4] := (t1);               inc(bayer,2);              inc(dst,6);            until bayer>(bayer_end-2);         end        else         begin//          for( ; bayer <= bayer_end - 2; bayer += 2, dst += 6 )            {\         repeat                t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +                      ord(bayer[bayer_step*2+2]) + 2) {>>} shr 2;                t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +                      ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) {>>} shr 2;              dst[1] := (t0);                dst[0] := (t1);              dst[-1] := bayer[bayer_step+1];                 t0 := (ord(bayer[2]) + ord(bayer[bayer_step*2+2]) + 1) {>>} shr 1;              t1 := (ord(bayer[bayer_step+1]) + ord(bayer[bayer_step+3]) + 1) {>>} shr 1;                dst[4] := (t0);                dst[3] := bayer[bayer_step+2];                dst[2] := (t1);                inc(bayer,2);              inc(dst,6);            until bayer>(bayer_end-2);         end;         if( bayer < bayer_end ) then        begin            t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +                  ord(bayer[bayer_step*2+2]) + 2) {>>} shr 2;            t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +                  ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) shr { >>} 2;            dst[-blue] := (t0);            dst[0] := (t1);            dst[blue] := bayer[bayer_step+1];            inc(bayer);            inc(dst,3);      end;         blue := -blue;        start_with_green := not start_with_green;//( ; size.height-- > 0; bayer0 += bayer_step, dst0 += dst_step )   123:     inc(bayer0,bayer_step);     inc(dst0,dst_step);     dec(size.height);    until (size.height<=0);     result:=CV_OK;end;  called as Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    icvBayer 2BGR_8u_C1C3R((b.getimagepointer(0,0)),b.RowPitch,pbbyte(b2.getimagepo inter(0,0)),b2.RowPitch,x,  CV_BayerBG2BGR);  where getimagepointer just gets a pointer to the first pixel of a raw image and "rowpitch" is the difference (in bytes) between the first pixels of two rows that follow eachother (@scanline[1]-@scanline[0]).  x is a TSize like record with dimensions Of course the smart thing to do (if you have at least a dual core) is to capture in one thread, and debayer in another. But it depends on if the video driver can operate zero copy (iow if you can queue your own buffers) ---------------------------------------------------------------------- af0815 · 2020-02-23 08:41 Quote from: marcov on February 22, 2020, 04:12:06 pmOf course the smart thing to do (if you have at least a dual core) is to capture in one thread, and debayer in another. But it depends on if the video driver can operate zero copy (iow if you can queue your own buffers) Thank you for sharing the code, i have now a lot to do :-) The 2 CPU solution is not so important, because i take one picture out of the stream comming from the camera, convert this picture and store it on a server. After then i have to wait a long time for the next picture. I must have the camera running because the light is complex and can change. If i make only one shoot with the camera, the automatic gain is not working correct. If the camera is working the full time, the result is nearly perfect for my need. The camera is excelent, because you can store fixed startup configuration. so you can test it life. After the tests are ok, you can fix nearly all params in the camera and can store it with the original software of the supplieren. This can be done in the next majore release of his software (i have test this with a prerelease). The Camera itselft is not expensive and the support is excellent. Working on WIndows and Linux. Windows have a well documented API (win32 and win64) and in Linux you can work with V4L2 and gstreamer. Both soloutions are working on Rasbian (RPi4, poor on RPi3B+) (Linux/arm) and Linux/64 (tested). Gstreamer can work with the GigE Versions too, is a little limited in Speed because i need a secand Ethetnet Adapter (one for camera with changed MTU)  I will test it on a RasPi 4 with 4GB :-) because this should be the targetplattform. And for me, its FPC/Lazarus only. ---------------------------------------------------------------------- georgje · 2020-02-25 09:29 My first post here. I'm looking for something like this too. Raw data, the sensor image is stored as 12 or 14 bit, Nikon anyway. Can anybody explain to me what SRGGB8 means. If I google it looks like a list of byte representing the colors in a specific sequence. But I'm missing 4 or 6 bits. How do you get to that raw data? George ---------------------------------------------------------------------- marcov · 2020-02-25 09:39 Btw, what camera do you use with RPI3/4? I'm thinking about getting one, but it seems yours is quite highres? Is it linescan? ---------------------------------------------------------------------- af0815 · 2020-02-25 10:19 Quote from: af0815 on February 21, 2020, 08:43:00 pmCameras: DFK 33UX183 = 5.472×3.648 (20 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1 DFK 33UX226 = 4.000×3.000 (12 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1 from TheImageSource. And GigE DFK 33GX178e running with a rude gstreamer implementation and with a external EthernetToUSB3 Adapter. Speed of USB3 Cams is better. Conversion from Bayer to jpg is done by gstreamer. ---------------------------------------------------------------------- af0815 · 2020-02-25 10:41 Quote from: georgje on February 25, 2020, 10:29:44 amMy first post here. I'm looking for something like this too. Raw data, the sensor image is stored as 12 or 14 bit, Nikon anyway. Can anybody explain to me what SRGGB8 means. If I google it looks like a list of byte representing the colors in a specific sequence. But I'm missing 4 or 6 bits. How do you get to that raw data? George Normally you have 8 or 16 Bits per Pixel, if not you have to look for the mapping. I have getting the data shown in the sample near the start of the thread. ---------------------------------------------------------------------- georgje · 2020-02-25 22:02 I think we're meaning different types of raw data. George ---------------------------------------------------------------------- marcov · 2020-02-25 22:05 Some sensors support 12-bit "packed", which means 3 bytes for 2 samples. But aside from that exception af0815 is right, usually pixel sizes are rounded up to whole multiples of 8 bits because of bus architectures. Anyway, step one is establishing a very precise definition of the input format, then you can start thinking about developing (or finding) a debayer routine for it. ---------------------------------------------------------------------- marcov · 2020-02-25 22:12 Quote from: af0815 on February 25, 2020, 11:19:14 amTheImageSource. I know them, but this was for private use, for which that is a bit out of the price range. Mostly I'm interested in black white cameras anyway. I know the cameras, I tested their 2MPish range, but eventually went with 2mpx AVT manta 235 B/W because of (iirc) SNR. Areascan are freaks for us anyway, we usually do linescan for larger resolutions. ---------------------------------------------------------------------- af0815 · 2020-02-26 05:18 Quote from: marcov on February 25, 2020, 11:12:03 pmQuote from: af0815 on February 25, 2020, 11:19:14 amTheImageSource. I know them, but this was for private use, for which that is a bit out of the price range. Mostly I'm interested in black white cameras anyway. A cheaper version can be the Logitech Brio on the other hand, not BW but can be mounted with a screw and is stable working. The areascan camera was choiced, because the large product is laying on a table without a movement and handled by hand. I need only one picture for documentation. ---------------------------------------------------------------------- georgje · 2020-02-26 06:53 Nikon,Canon,Sony,Leica by example use 12 or 14 bits, and the choice of compressed or not. Just a Google on "nikon raw bit depth" etc. I'm looking for a way to reach the raw data, not for debayering but to make an impression for how the sensor catches the image. Maybe we're looking at totally different cameras. George ---------------------------------------------------------------------- af0815 · 2020-02-26 07:00 Quote from: georgje on February 26, 2020, 07:53:30 amMaybe we're looking at totally different cameras. Yes, i am looking for my used camera only. Consumer products like Nikon, Canon, Sony, Leica are not for my production needs. ---------------------------------------------------------------------- marcov · 2020-02-26 10:08 Quote from: af0815 on February 26, 2020, 08:00:10 amQuote from: georgje on February 26, 2020, 07:53:30 amMaybe we're looking at totally different cameras. Yes, i am looking for my used camera only. Consumer products like Nikon, Canon, Sony, Leica are not for my production needs. (Sony afaik is the biggest sensor make in the world, specially CCD. Many if not most upscale industrial cameras have Sony sensors. They also had industrial area cameras afaik, but not very prominent. Nikon also is a pro club but more in lenses I think, most better quality "vision"  lenses that we buy are from Nikon) ---------------------------------------------------------------------- af0815 · 2020-02-26 10:52 QuoteI'm looking for a way to reach the raw data, not for debayering but to make an impression for how the sensor catches the image.This was the reason for me to different between consumer products - make internal processing of the data and put only processed data - and industrial cameras, which normal deliver different formats for processing purposes. consumer = get processed data in a common (maybe companyspecific) format (raw, jpeg, ...) and you take this on your vacaition industrial = get data in a datastream (more or less preprocessed) for processing. it my definition. Maybe i am wrong. ---------------------------------------------------------------------- georgje · 2020-02-26 11:28 In what you call consumer camera's raw is raw, the Color Filter Array. It contains the digitized values of the sensor elements. Editing is done during and/or after demosaicing. The problem is to extract that array from the raw-file and eventual decompress it. Once that is done it's becoming easy. George ---------------------------------------------------------------------- circular · 2020-02-27 11:23 You can already try displaying a grayscale image with whatever values you find. If you know the image dimensions, you can divide the size in byte by the number of pixels. If it is not round at all, multiply by 8 to get the bit depth of the pixel. ---------------------------------------------------------------------- ADMGNS · 2021-11-11 17:56 hello, if you still need it, please see attachment.. there is some code written in delphi to simulate bayerize and debayerize, but last one is not finished at all. maybe it gives you some logic or ideas.. lucks ---------------------------------------------------------------------- af0815 · 2021-11-11 20:30 For the records, i have al running soloution for me. https://github.com/afriess/GSTCamera/tree/master/pasv4l/demos