https://lupyuen.github.io/articles/fb?1 Pick another theme! NuttX RTOS for PinePhone: Framebuffer * 1 Framebuffer Demo * 2 Framebuffer Interface * 3 Render Grey Screen * 4 Render Blocks * 5 Render Circle * 6 Render Rectangle * 7 LVGL Graphics Library * 8 PinePhone Framebuffer Driver + 8.1 RAM Framebuffer + 8.2 Framebuffer Operations + 8.3 Initialise Framebuffer + 8.4 Get Video Plane + 8.5 Get Video Info + 8.6 Get Plane Info + 8.7 Update Area * 9 Mystery of the Missing Pixels * 10 Fix Missing Pixels * 11 What's Next 7 Jan 2023 Apache NuttX Framebuffer App on Pine64 PinePhone Suppose we're running Apache NuttX RTOS on Pine64 PinePhone... How will we create Graphical Apps for NuttX? (Pic above) Today we'll learn about the... * Framebuffer Interface that NuttX provides to our apps for rendering graphics * What's inside the Framebuffer Driver for PinePhone * Mystery of the Missing Framebuffer Pixels and how we solved it (unsatisfactorily) * Creating NuttX Apps with the LVGL Graphics Library NuttX Framebuffer App running on PinePhone NuttX Framebuffer App running on PinePhone 1 Framebuffer Demo Our Demo Code for today comes (mostly) from this Example App... * NuttX Framebuffer Driver Example How do we build the app? To enable the app in our NuttX Project... make menuconfig And select... Application Configuration > Examples > Framebuffer Driver Example Save the configuration and exit menuconfig. Look for this line: apps/examples/fb/fb_main.c #ifdef CONFIG_FB_OVERLAY And change it to... #ifdef NOTUSED Because our PinePhone Framebuffer Driver doesn't support overlays yet. Then build NuttX with... make Let's look at the Demo Code... 2 Framebuffer Interface What's inside the app? We begin with the Framebuffer Interface that NuttX provides to our apps for rendering graphics. To call the Framebuffer Interface, our app opens the Framebuffer Driver at /dev/fb0: fb_main.c #include #include // Open the Framebuffer Driver int fd = open("/dev/fb0", O_RDWR); // Quit if we failed to open if (fd < 0) { return; } Next we fetch the Framebuffer Characteristics, which will tell us the Screen Size (720 x 144) and Pixel Format (ARGB 8888)... // Get the Characteristics of the Framebuffer struct fb_videoinfo_s vinfo; int ret = ioctl( // Do I/O Control... fd, // File Descriptor of Framebuffer Driver FBIOGET_VIDEOINFO, // Get Characteristics (unsigned long) &vinfo // Framebuffer Characteristics ); // Quit if FBIOGET_VIDEOINFO failed if (ret < 0) { return; } (fb_videoinfo_s is defined here) Then we fetch the Plane Info, which describes the RAM Framebuffer that we'll use for drawing: fb_main.c // Get the Plane Info struct fb_planeinfo_s pinfo; ret = ioctl( // Do I/O Control... fd, // File Descriptor of Framebuffer Driver FBIOGET_PLANEINFO, // Get Plane Info (unsigned long) &pinfo // Returned Plane Info ); // Quit if FBIOGET_PLANEINFO failed if (ret < 0) { return; } (fb_planeinfo_s is defined here) To access the RAM Framebuffer, we map it to a valid address: fb_main.c // Map the Framebuffer Address void *fbmem = mmap( // Map the address of... NULL, // Hint (ignored) pinfo.fblen, // Framebuffer Size PROT_READ | PROT_WRITE, // Read and Write Access MAP_SHARED | MAP_FILE, // Map as Shared Memory fd, // File Descriptor of Framebuffer Driver 0 // Offset for Memory Mapping ); // Quit if we failed to map the Framebuffer Address if (fbmem == MAP_FAILED) { return; } This returns fbmem, a pointer to the RAM Framebuffer. Let's blast some pixels to the RAM Framebuffer... Render Grey Screen 3 Render Grey Screen What's the simplest thing we can do with our Framebuffer? Let's fill the entire Framebuffer with Grey: fb_main.c // Fill entire framebuffer with grey memset( // Fill the buffer... fbmem, // Framebuffer Address 0x80, // Value pinfo.fblen // Framebuffer Size ); (We'll explain in a while why this turns grey) After filling the Framebuffer, we refresh the display: fb_main.c // Area to be refreshed struct fb_area_s area = { .x = 0, // X Offset .y = 0, // Y Offset .w = pinfo.xres_virtual, // Width .h = pinfo.yres_virtual // Height }; // Refresh the display ioctl( // Do I/O Control... fd, // File Descriptor of Framebuffer Driver FBIO_UPDATE, // Refresh the Display (unsigned long) &area // Area to be refreshed ); (fb_area_s is defined here) If we skip this step, we'll see missing pixels in our display. (More about this below) Remember to close the Framebuffer when we're done: fb_main.c // Unmap the Framebuffer Address munmap( // Unmap the address of... fbmem, // Framebuffer Address pinfo.fblen // Framebuffer Size ); // Close the Framebuffer Driver close(fd); When we run this, PinePhone turns grey! (Pic above) To understand why, let's look inside the Framebuffer... PinePhone Framebuffer Why did PinePhone turn grey when we filled it with 0x80? Our Framebuffer has 720 x 1440 pixels. Each pixel has 32-bit ARGB 8888 format (pic above)... * Alpha (8 bits) * Red (8 bits) * Green (8 bits) * Blue (8 bits) (Alpha has no effect, since this is the Base Layer and there's nothing underneath) When we fill the Framebuffer with 0x80, we're setting Alpha (unused), Red, Green and Blue to 0x80. Which produces the grey screen. Let's do some colours... Render Blocks 4 Render Blocks This is how we render the Blue, Green and Red Blocks in the pic above: fb_main.c // Fill framebuffer with Blue, Green and Red Blocks uint32_t *fb = fbmem; // Access framebuffer as 32-bit pixels const size_t fblen = pinfo.fblen / 4; // 4 bytes per pixel // For every pixel... for (int i = 0; i < fblen; i++) { // Colors are in ARGB 8888 format if (i < fblen / 4) { // Blue for top quarter. // RGB24_BLUE is 0x0000 00FF fb[i] = RGB24_BLUE; } else if (i < fblen / 2) { // Green for next quarter. // RGB24_GREEN is 0x0000 FF00 fb[i] = RGB24_GREEN; } else { // Red for lower half. // RGB24_RED is 0x00FF 0000 fb[i] = RGB24_RED; } } // Omitted: Refresh the display with ioctl(FBIO_UPDATE) Everything is hunky dory for chunks of pixels! Let's set individual pixels by row and column... Render Circle 5 Render Circle This is how we render the Green Circle in the pic above: fb_main.c // Fill framebuffer with Green Circle uint32_t *fb = fbmem; // Access framebuffer as 32-bit pixels const size_t fblen = pinfo.fblen / 4; // 4 bytes per pixel const int width = pinfo.xres_virtual; // Framebuffer Width const int height = pinfo.yres_virtual; // Framebuffer Height // For every pixel row... for (int y = 0; y < height; y++) { // For every pixel column... for (int x = 0; x < width; x++) { // Get pixel index const int p = (y * width) + x; // Shift coordinates so that centre of screen is (0,0) const int half_width = width / 2; const int half_height = height / 2; const int x_shift = x - half_width; const int y_shift = y - half_height; // If x^2 + y^2 < radius^2, set the pixel to Green. // Colors are in ARGB 8888 format. if (x_shift*x_shift + y_shift*y_shift < half_width*half_width) { // RGB24_GREEN is 0x0000 FF00 fb[p] = RGB24_GREEN; } else { // Otherwise set to Black // RGB24_BLACK is 0x0000 0000 fb[p] = RGB24_BLACK; } } } // Omitted: Refresh the display with ioctl(FBIO_UPDATE) Yep we have full control over every single pixel! Let's wrap up our demo with some mesmerising rectangles... Render Rectangles 6 Render Rectangle When we run the NuttX Framebuffer App, we'll see a stack of Color Rectangles. (Pic above) We render each Rectangle like so: fb_main.c // Rectangle to be rendered struct fb_area_s area = { .x = 0, // X Offset .y = 0, // Y Offset .w = pinfo.xres_virtual, // Width .h = pinfo.yres_virtual // Height } // Render the rectangle draw_rect(&state, &area, color); // Omitted: Refresh the display with ioctl(FBIO_UPDATE) (draw_rect is defined here) The pic below shows the output of the Framebuffer App fb when we run it on PinePhone... NuttX Framebuffer App running on PinePhone (See the Complete Log) And we're all done with Circles and Rectangles on PinePhone! Let's talk about Graphical User Interfaces... LVGL on NuttX on PinePhone 7 LVGL Graphics Library Rendering graphics pixel by pixel sounds tedious... Is there a simpler way to render Graphical User Interfaces? Yep just call the LVGL Graphics Library! (Pic above) To build the LVGL Demo App on NuttX... make menuconfig Select these options... * Enable "Application Configuration > Examples > LVGL Demo" * Enable "Application Configuration > Graphics Support > Light and Versatile Graphics Library (LVGL)" * Under "LVGL > Graphics Settings" + Set Horizontal Resolution to 720 + Set Vertical Resolution to 1440 + Set DPI to 200 (or higher) * Under "LVGL > Color settings" + Set Color Depth to 32 Save the configuration and exit menuconfig. Rebuild NuttX... make Boot NuttX on PinePhone. At the NSH Command Prompt, enter... lvgldemo We'll see the Graphical User Interface as shown in the pic above! But it won't respond to our touch right? Yeah we haven't started on the I2C Touch Input Driver for PinePhone. Maybe someday LVGL Touchscreen Apps will run OK on PinePhone! What's inside the LVGL App? Here's how it works... * Main Function (Event Loop) of the LVGL App is here: lvgldemo.c * Main Function calls the NuttX Framebuffer Interface here: fbdev.c * LVGL Widgets are created here: lv_demo_widgets.c (See the docs for LVGL Widgets) * LVGL Version supported by NuttX is 7.3.0. (See this) Now we talk about the internals of our Framebuffer Driver... 8 PinePhone Framebuffer Driver We've seen the Framebuffer Interface for NuttX Apps... What's inside the Framebuffer Driver for PinePhone? TODO Complete Display Driver for PinePhone Complete Display Driver for PinePhone 8.1 RAM Framebuffer Inside PinePhone's Allwinner A64 SoC are the Display Engine and Timing Controller TCON0. (Pic above) Display Engine and TCON0 will blast pixels from the RAM Framebuffer to the LCD Display, over Direct Memory Access (DMA). (More about Display Engine and TCON0) Here's our RAM Framebuffer: pinephone_display.c // Frame Buffer for Display Engine // Fullscreen 720 x 1440 (4 bytes per XRGB 8888 pixel) // PANEL_WIDTH is 720 // PANEL_HEIGHT is 1440 static uint32_t g_pinephone_fb0[ // 32 bits per pixel PANEL_WIDTH * PANEL_HEIGHT // 720 x 1440 pixels ]; (Memory Protection is not turned on yet, so mmap returns the actual address of g_pinephone_fb0 to NuttX Apps for rendering) We describe PinePhone's LCD Display like so (pic below)... // Video Info for PinePhone // (Framebuffer Characteristics) // PANEL_WIDTH is 720 // PANEL_HEIGHT is 1440 static struct fb_videoinfo_s g_pinephone_video = { .fmt = FB_FMT_RGBA32, // Pixel format (XRGB 8888) .xres = PANEL_WIDTH, // Horizontal resolution in pixel columns .yres = PANEL_HEIGHT, // Vertical resolution in pixel rows .nplanes = 1, // Color planes: Base UI Channel .noverlays = 2 // Overlays: 2 Overlay UI Channels }; (fb_videoinfo_s is defined here) (We're still working on the Overlays) We tell NuttX about our RAM Framebuffer with this Plane Info... // Color Plane for Base UI Channel: // Fullscreen 720 x 1440 (4 bytes per XRGB 8888 pixel) static struct fb_planeinfo_s g_pinephone_plane = { .fbmem = &g_pinephone_fb0, // Framebuffer Address .fblen = sizeof(g_pinephone_fb0), // Framebuffer Size .stride = PANEL_WIDTH * 4, // Length of a line (4-byte pixel) .display = 0, // Display number (Unused) .bpp = 32, // Bits per pixel (XRGB 8888) .xres_virtual = PANEL_WIDTH, // Virtual Horizontal resolution .yres_virtual = PANEL_HEIGHT, // Virtual Vertical resolution .xoffset = 0, // X Offset from virtual to visible .yoffset = 0 // Y Offset from virtual to visible }; (fb_planeinfo_s is defined here) PinePhone Framebuffer 8.2 Framebuffer Operations Our Framebuffer Driver supports these Framebuffer Operations: pinephone_display.c // Vtable for Frame Buffer Operations static struct fb_vtable_s g_pinephone_vtable = { // Basic Framebuffer Operations .getvideoinfo = pinephone_getvideoinfo, .getplaneinfo = pinephone_getplaneinfo, .updatearea = pinephone_updatearea, // TODO: Framebuffer Overlay Operations .getoverlayinfo = pinephone_getoverlayinfo, .settransp = pinephone_settransp, .setchromakey = pinephone_setchromakey, .setcolor = pinephone_setcolor, .setblank = pinephone_setblank, .setarea = pinephone_setarea }; We haven't implemented the Overlays, so let's talk about the first 3 operations... * Get Video Info * Get Plane Info * Update Area But before that we need to initialise the Framebuffer and return the Video Plane... 8.3 Initialise Framebuffer At Startup, NuttX Kernel calls up_fbinitialize to initialize the Framebuffer... * "Complete Display Driver" up_fbinitialize comes from our Framebuffer Driver (LCD Driver)... * up_fbinitialize (Initialise Framebuffer) (Called by fb_register and pinephone_bringup) Then NuttX Kernel interrogates our Framebuffer Driver to discover the Video Plane... 8.4 Get Video Plane NuttX Kernel calls our Framebuffer Driver to discover the Framebuffer Operations supported by our driver. This is how we return the Framebuffer Operations: pinephone_display.c // Get the Framebuffer Object for the supported operations struct fb_vtable_s *up_fbgetvplane( int display, // Display Number should be 0 int vplane // Video Plane should be 0 ) { // Return the supported Framebuffer Operations return &g_pinephone_vtable; } (We've seen g_pinephone_vtable earlier) Now it gets interesting: NuttX Kernel and NuttX Apps will call the operations exposed by our Framebuffer Driver... Test Pattern on NuttX for PinePhone 8.5 Get Video Info Remember FBIOGET_VIDEOINFO for fetching the Framebuffer Characteristics? * "Framebuffer Interface" The first operation exposed by our Framebuffer Driver returns the Video Info that contains our Framebuffer Characteristics: pinephone_display.c // Get the Video Info for our Framebuffer // (ioctl Entrypoint: FBIOGET_VIDEOINFO) static int pinephone_getvideoinfo( struct fb_vtable_s *vtable, // Framebuffer Driver struct fb_videoinfo_s *vinfo // Returned Video Info ) { // Copy and return the Video Info memcpy(vinfo, &g_pinephone_video, sizeof(struct fb_videoinfo_s)); // Keep track of the stages during startup: // Stage 0: Initialize driver at startup // Stage 1: First call by apps // Stage 2: Subsequent calls by apps // We erase the framebuffers at stages 0 and 1. This allows the // Test Pattern to be displayed for as long as possible before erasure. static int stage = 0; if (stage < 2) { stage++; memset(g_pinephone_fb0, 0, sizeof(g_pinephone_fb0)); memset(g_pinephone_fb1, 0, sizeof(g_pinephone_fb1)); memset(g_pinephone_fb2, 0, sizeof(g_pinephone_fb2)); } return OK; } (We've seen g_pinephone_video earlier) This code looks interesting: We're trying to show the Startup Test Pattern for as long as possible. (Pic above) Normally NuttX Kernel will erase our Framebuffer at startup. But with the logic above, our Test Pattern will be visible until the first app call to our Framebuffer Driver. (Test Pattern is rendered by pinephone_display_test_pattern) (Which is called by pinephone_bringup) 8.6 Get Plane Info Earlier we've seen FBIOGET_PLANEINFO that fetches the RAM Framebuffer... * "Framebuffer Interface" This is how we return the Plane Info that describes the RAM Framebuffer: pinephone_display.c // Get the Plane Info for our Framebuffer // (ioctl Entrypoint: FBIOGET_PLANEINFO) static int pinephone_getplaneinfo( struct fb_vtable_s *vtable, // Framebuffer Driver int planeno, // Plane Number should be 0 struct fb_planeinfo_s *pinfo // Returned Plane Info ) { // Copy and return the Plane Info memcpy(pinfo, &g_pinephone_plane, sizeof(struct fb_planeinfo_s)); return OK; } (We've seen g_pinephone_plane earlier) 8.7 Update Area The final operation updates the display when there's a change to the Framebuffer: pinephone_display.c // Update the Display when there is a change to the Framebuffer // (ioctl Entrypoint: FBIO_UPDATE) static int pinephone_updatearea( struct fb_vtable_s *vtable, // Framebuffer Driver const struct fb_area_s *area // Updated area of Framebuffer ) { // Mystery Code... This operation is invoked when NuttX Apps call FBIO_UPDATE, as we've seen earlier... * "Render Grey Screen" The code inside looks totally baffling, but first let's talk about a mysterious rendering problem... Missing Pixels in PinePhone Image 9 Mystery of the Missing Pixels When we tested our Framebuffer Driver for the very first time, we discovered missing pixels in the rendered image (pic above)... * Inside the Yellow Box is supposed to be an Orange Box * Inside the Orange Box is supposed to be a Red Box * We see bits of Orange and Red Pixels (Compare with the pic below) Maybe we didn't render the pixels correctly? Or maybe the RAM Framebuffer got corrupted? When we slowed down the rendering, we see the missing pixels magically appear later in a curious pattern... * Watch the Demo on YouTube According to the video, the pixels are actually written correctly to the RAM Framebuffer. But the pixels at the lower half don't get pushed to the display until the next screen update. Maybe it's a problem with Framebuffer DMA / Display Engine / Timing Controller TCON0? Yeah there seems to be a lag between the writing of pixels to RAM Framebuffer, and the pushing of pixels to the display over DMA / Display Engine / Timing Controller TCON0. We found a workaround for the lag in rendering pixels... Fixed Missing Pixels in PinePhone Image 10 Fix Missing Pixels TODO In the previous section we saw that there was a lag pushing pixels from the RAM Framebuffer to the PinePhone Display (over DMA / Display Engine / Timing Controller TCON0). Can we overcome this lag by copying the RAM Framebuffer to itself, forcing the display to refresh? This sounds very strange, but yes it works! From pinephone_display.c: // Update the Display when there is a change to the Framebuffer // (ioctl Entrypoint: FBIO_UPDATE) static int pinephone_updatearea( struct fb_vtable_s *vtable, // Framebuffer Driver const struct fb_area_s *area // Updated area of framebuffer ) { // Access Framebuffer as bytes uint8_t *fb = (uint8_t *)g_pinephone_fb0; const size_t fbsize = sizeof(g_pinephone_fb0); // Copy the Entire Framebuffer to itself, // to fix the missing pixels. // Not sure why this works. for (int i = 0; i < fbsize; i++) { // Declare as volatile to prevent compiler optimization volatile uint8_t v = fb[i]; fb[i] = v; } return OK; } With the code above, the Red, Orange and Yellow Boxes are now rendered correctly in our NuttX Framebuffer Driver for PinePhone. (Pic above) Who calls pinephone_updatearea? After writing the pixels to the RAM Framebuffer, NuttX Apps will call ioctl(FBIO_UPDATE) to update the display. This triggers pinephone_updatearea in our NuttX Framebuffer Driver: fb_main.c // Omitted: NuttX App writes pixels to RAM Framebuffer // Update the Framebuffer #ifdef CONFIG_FB_UPDATE ret = ioctl( // I/O Command state->fd, // Framebuffer File Descriptor FBIO_UPDATE, // Update the Framebuffer (unsigned long)((uintptr_t)area) // Updated area ); #endif TODO: Can we copy the pixels for the partial screen area? Probably, needs more rigourous testing We might need to check the CPU Writeback Cache, and verify that our Framebuffer has been mapped with the right attributes. (Thanks to suarezvictor and crzwdjk for the tips!) How do other PinePhone operating systems handle this? We might need to handle TCON0 Vertical Blanking (TCON0_Vb_Int_En / TCON0_Vb_Int_Flag) and TCON0 CPU Trigger Mode Finish (TCON0_Tri_Finish_Int_En / TCON0_Tri_Finish_Int_Flag) like this... * sun4i_tcon_enable_vblank * sun4i_tcon_handler (More about sun4i_tcon_handler) p-boot Bootloader seems to handle every TCON0 CPU Trigger Mode Finish (TCON0_Tri_Finish_Int_En / TCON0_Tri_Finish_Int_Flag) by updating the Display Engine Registers. Which sounds odd... 1. Render Loop waits forever for EV_VBLANK: dtest.c 2. EV_VBLANK is triggered by display_frame_done: gui.c 3. display_frame_done is triggered by TCON0 CPU Trigger Mode Finish: display.c 4. Render Loop handles EV_VBLANK by redrawing and calling display_commit: dtest.c 5. display_commit updates the Display Engine Registers, including the Framebuffer Addresses: display.c Can we handle TCON0 CPU Trigger Mode Finish without refreshing the Display Engine Registers? 11 What's Next TODO Check out the other articles on NuttX RTOS for PinePhone... * "Apache NuttX RTOS on Arm Cortex-A53: How it might run on PinePhone" * "PinePhone boots Apache NuttX RTOS" * "NuttX RTOS for PinePhone: Fixing the Interrupts" * "NuttX RTOS for PinePhone: UART Driver" * "NuttX RTOS for PinePhone: Blinking the LEDs" * "Understanding PinePhone's Display (MIPI DSI)" * "NuttX RTOS for PinePhone: Display Driver in Zig" * "Rendering PinePhone's Display (DE and TCON0)" * "NuttX RTOS for PinePhone: Render Graphics in Zig" * "NuttX RTOS for PinePhone: MIPI Display Serial Interface" * "NuttX RTOS for PinePhone: Display Engine" * "NuttX RTOS for PinePhone: LCD Panel" Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn't have been possible without your support. * Sponsor me a coffee * My Current Project: "Apache NuttX RTOS for PinePhone" * My Other Project: "The RISC-V BL602 Book" * Check out my articles * RSS Feed Got a question, comment or suggestion? Create an Issue or submit a Pull Request here... lupyuen.github.io/src/fb.md