https://puri.sm/posts/cameras-its-complicated/ Purism * Store * Products * News & Events * About + Our story + Why Purism? + The Team * Support + Contact + Documentation + FAQ + Forums + My Account * PureOS Cameras: It's Complicated May 9, 2022 Librem 5, Software Chipsets and components / FLOSS applications / Graphics / librem 5 / Librem 5 USA / Phones * About * Latest Posts [36e3604df8] Dorota Czaplejewicz [36e3604df8] Latest posts by Dorota Czaplejewicz (see all) * Cameras: It's Complicated - May 9, 2022 * Speak to me! - October 19, 2021 * I Love You For Your Personality - December 15, 2020 Two years before I started working on cameras for the Librem 5, I thought the work would go something like this: first, write a driver, then maybe calibrate the colors, connect to the camera support infrastructure, and bam! PureOS users on the phone would then do teleconferences with Jitsi or snap selfies with Cheese, just like they can on laptops. The title of this blog post already gives away what happened. I wrote a driver and connected it up, yet teleconferences remain in the future. In fact, our version of Megapixels is the only app that can shoot photos on the Librem 5 today, and that's because it received special attention. How come? Aren't cameras pretty simple in the end? Light in, pixels out. That's pretty plug-and-play! Well actually, no. Cameras can be made of a multitude of components, and light sensors are just one piece of the puzzle. There are cameras that output raw light values, but the whole module also includes mechanical parts like a flash light, focus motor, or one for zoom, or maybe active image stabilization actuators. There are also Image Signals Processors (ISPs) which can do a multitude of tasks, like debayering, denoising, lens correction, color balance, color conversions, and more. [1090251]Librem 5 reworked camera module But that's not surprising. What's unusual is that despite the diversity of camera modules, laptop cameras generally Just Work. They are so well supported under Linux because they are connected using a protocol called... USB. Yes, laptop cameras are connected over USB! And USB cameras universally support the UVC (USB video class) protocol, which requires the webcam to provide a video stream ready to consume by applications. It's an amazing achievement to have such standardization. However, USB is not the camera protocol typically used on phones. It's not even universal on laptops! UVC is really good for basic video recording, because it provides just what typical applications need: automatic adjustment of brightness, focus, and color, and it also provides images that are already demosaiced. Those are difficult tasks, and completely out of scope for applications like the web browser or a chat client. They should be doing what they do best, rather than switch focus to image processing, and UVC helps them a lot by pushing those responsibilities to the camera hardware. If you're a phone maker, you might not be satisfied with UVC taking over those tasks. You might want to have detailed access to the ISP part of your camera module. Or you might want to expose the raw sensor instead, just like the Librem 5 does. Then you can't rely on UVC any longer. Let's take a look again at the processing steps that are really needed between when the camera receives light and an application presents a video frame. On the left, there's the UVC baseline, on the right a raw Bayer sensor.[uvc_vs_Bayer]Oh no! There's a huge gap in functionality when using a Bayer sensor! Our application can't capture frames until the gap is filled. But what should fill it? We already established that the application should not be responsible for processing. Maybe we can squeeze it back in the module? Sadly, it's not possible to add new functionality to existing hardware. What's left is the kernel... perish the thought! Adding image processing code to the kernel is a terrible idea: any bug could cause the crash of the entire system. Let the kernel remain a path to access hardware, and that only. There are no more options. Are we in a bind? Wait a moment. If you zoom in on any typical application, you will see that... it's actually not a single thing. It's made of multiple libraries combined together, each responsible for a different task. For example, your email client doesn't decode images, libpng does. And your movie player does not compile shaders needed to display the window, it uses an OpenGL library. Where's the library that provides highly processed video streams like those coming from UVC modules? It's called libcamera. Libcamera is not yet old and established like OpenGL. It's relatively young, just like running GNU/Linux on devices with custom cameras is, so it's currently not widely used. But once it becomes widespread enough, it could solve our problem of having plug-and-play camera support, regardless of the camera type. In fact, we already wrote about this in our photo processing tutorial. That's where the Librem 5 team has been directing their camera efforts now. We still have a way to go: selecting capture resolutions, forwarding controls, debayering, and auto-adjustment algorithms. We're progressing slowly, but steadily, helping the library overcome growing pains and get right its internal structure as well as external API. And once we get the functionality we need working, it will be available to others using libcamera: applications and other cameras will have them out of the box. Hopefully, by that time, libcamera will reach critical mass, and all applications would be able to use it without fuss, including teleconferencing apps. Then we'll reach our goal. Meanwhile, you can help everyone get there by contributing. If you're a student, libcamera is also regularly a mentor at Google Summer of Code. See you on the other side (of a teleconference)! Purism Products and Availability Chart Model Status Lead Time Most Secure PC Purism Librem In Stock 10 Learn More Buy Librem Mini Mini days Now Most Secure Server Librem In Stock 10 Notify MeLearn Purism Librem Servers Servers days More Buy Now USB Security Token Librem In Stock 10 Learn More Buy Purism Librem Key Key days Now Most Secure Laptop Librem In Stock 10 Learn More Buy Purism Librem 14 14 days Now Made in USA Phone Librem 5 In Stock 60 Learn More Buy Purism Librem 5 USA USA days Now Currently 52 Notify MeLearn [l5-v1-fron] Librem 5 shipping weeks More Buy Now backlogs The current product and shipping chart of Purism Librem products, updated on May 11, 2022 Recent Posts * Improved Delivery Time for Librem 5 USA: May 2022 Update * Cameras: It's Complicated * Purism and Linux 5.18 * Why I Support Purism, A Tech Company that Respects Digital Rights * Improving the Stability and Reliability with a Modular Modem in the Librem 5 Related Content * Improved Delivery Time for Librem 5 USA: May 2022 Update * Cameras: It's Complicated * Purism and Linux 5.18 * Improving the Stability and Reliability with a Modular Modem in the Librem 5 * How to Power Your CS Labs with PureOS Tags Advanced readers Battery life Benchmarks and testing Boot and BIOS Chipsets and components Communications infrastructure Crowdfunding Customer Feedback FLOSS applications Giving and contributing back Graphics Interviews Laptops librem 5 Librem 5 USA Librem 14 Librem One Linux kernel Made in USA Electronics most secure computer most secure laptop most secure pc most secure phone Newsletter and status updates Phones Power management Press Privacy Product or service launch PureOS Reverse engineering secure computing Security Social media Software freedom Supply chain Tablets Testimonials and user stories Tips and tricks User empowerment User experience design User interaction design Videos Website Wireless and networking [banner-1] --------------------------------------------------------------------- Resources * Press Room * Enterprise * Developers * Affiliates * Pages About Purism * Security * Jobs * Policies * Get involved * Warrant Canary Newsletter Want to get announcements & news digests once in a while? [ ] Subscribe You will receive an automated e-mail you will need to reply to to confirm your subscription. Safe, public social media with millions of people already active Librem Chat - End-to-end encrypted chat, VoIP, and video calling used by millions of people --------------------------------------------------------------------- Purism 3D renders are artist renderings, for illustration purposes. Images and specifications are subject to change depending on manufacturing requirements. Unless otherwise noted, contents created by the Purism team on this website are copyleft with a CC-by-SA 4.0 license.