https://jamesg.blog/2024/03/16/vinyl-record-indexing/ Skip to main content James' Coffee Blog [?] * IndieWeb * Coffee * Moments of Joy * Technical Writing * Blogroll * Book * Website Ideas * Explore * RSS Cataloguing my vinyl collection with computer vision Published on March 16, 2024 under the Computer Vision category. Every so often, a daunting thought comes to mind: I really should make a list of all of my vinyl records. I have previously recorded my collection in text files, but I always ran into the problem of context shifting (and, unrelated: I lost the file). Handling vinyls then writing information about them in text files over again is not the most comfortable process -- shifting from carefully moving large records to typing on your keyboard feels too fragile. This inspired to me to consider the following proposition: how can I make the cataloguing process easier? I decided to build a vinyl cataloguing tool powered by computer vision. The tool allows you to set up a webcam and saves every frame where a unique vinyl record is found. Those frames are then sent to ChatGPT to retrieve meta information about the album. Finally, the results are saved in a CSV file. Below is a demo of the system identifying unique vinyl records: Here is the result of the above cataloguing session: artist,album Taylor Swift,Red Taylor Swift,Lover In this blog post, I will discuss how this project works, sharing my learnings as I built it. Without further ado, let's get started! ( View source code) Identifying unique vinyl records My goal was to create an indexing system that could work without requiring any direct human input. This ruled out taking photos of every record and then processing them (i.e. with OCR, an LLM, or another method of information retrieval). I decided that being able to take a video would be most effective. I could start the video, show all my records, then have a mechanism to stop the video. I wanted a system where I could set up a camera, place each record in front of the camera, then move it back to my shelf. With this idea in mind, I started to think about how I could build it. I could use an LLM that accepts video inputs, although I was worried about records getting missed. I wanted a system where, if anything went wrong, I was able to interpret the results; if a vinyl could not be identified, I would rather have an error state than a missing record. Plus, I was not keen on the higher costs associated with having an entire video processed by an LLM, with all the redundant data that would be in the video. I could also train a computer vision model to identify vinyl record covers, then use an object tracking algorithm (i.e. ByteTrack) to retrieve all unique records. This had one major advantage: I could identify the exact location of the record then crop it for further processing. This would ensure no backgrounds were processed at later stages of the system. But, to train a model I would have to take at least a few dozen photos, annotate them, evaluate model performance, and potentially fine-tune the model on even more data to achieve the desired performance. I have done this many times, but I wanted to avoid labeling for this project. I had another idea: use a zero-shot embedding model to identify unique frames and classify them based on pre-determined labels. This is possible with CLIP-like models. CLIP is a classification and embedding model architecture with which you can calculate text and image embeddings. You can compare text and image embeddings to assign a category, or multiple categories, to an image. With a CLIP-like model, I could provide the following prompts to each frame from an incoming video feed: * Vinyl record * Something else Using a similarity calculation, I could identify if there was or was not a vinyl record in frame. something else is a good prompt to use when you are working on a classification task and want to know if none of your other labels match. For this project, I decided to use MobileCLIP, a CLIP model released by Apple in March 2024. I used MobileCLIP because it is fast, and because I had not yet used the model. Using the MobileCLIP repository instructions, I downloaded the model, then started on a script that: 1. Initialises the model. 2. Computes embeddings for three prompts: vinyl record, something else, and open palm (I'll talk about open palm later). 3. Uses OpenCV to read frames from the webcam, and; 4. For each frame, calculates the most similar embedding. The label is then saved into a deque. The deque keeps track of the labels that most closely correspond to each of the last 50 frames. If a vinyl record is identified in more than 10 of the last 50 frames, the frame is saved to a file and the embedding for the record is saved in a list. A vinyl record must be present for 10 of the last 50 frames so that the frame isn't saved when the record is still coming into view. Without this check, a record could appear in the top left corner, with most information out of frame. If this happens, it may be impossible to identify the vinyl; the full vinyl cover needs to be in view, which is enabled by waiting for 10 positive identifications of a vinyl before recording the image. Then, the deque is cleared. When a frame is saved to disk, I increment a counter on screen so that I can see the record has been successfully saved. If one or more vinyl records have been identified, there is an added check: the embedding for the current frame is compared to all of the embeddings for frames with vinyl records. Then, a cosine similarity check takes place to verify that the image is too similar to any existing record. This allows me to ensure the same record doesn't get saved multiple times. This is essential because every record needs to be post-processed: if the script records near-duplicate images that feature the same vinyl, the post-processing time -- and money, as an external service is used for post-processing, which will be discussed later -- goes up unnecessarily. Of note, the saved frames do not segment out each vinyl record. This is one drawback with this system. Whereas an object detection model can identify the location of an object (i.e. a vinyl record) in an image, a classification model like CLIP cannot. I determined this was okay because I plan to index records on a blank background, thus minimizing the extent to which background information would interfere with post-processing. Furthermore, if two records are introduced in the same frame, they may be recorded as one record. This is because the system is classifying frames, not identifying objects. Thus, it is recommended to only show one record at once. With this logic, I had a system that let me identify vinyl records and save each unique one to a file. Earlier, I mentioned open palm was one of the prompts for which I looked. This is a control prompt that is used to terminate the program. Thus, I can stop identifying records without having to touch my computer. If I hold my palm open for more than 20 frames, the program stops recording. It felt good to have a digital system that required no direct human interaction to use and turn off. The next step: identifying the album name and artist name for each record. Matching images to metadata With images of each record, I could start matching them to metadata. At first, I thought about using a reverse image search system. I tried Bing's Visual Search API, which allows you to upload an image and retrieve search results pertaining to that image. But, the system did not give me data that would not require significant -- and complex -- further processing. When I uploaded a Taylor Swift record, Bing's API returned related results, but the text associated with each result was not structured. I would need to do entity recognition, etc. to retrieve and distinguish the artist name and album name from all the other text in each result. The quality of the data made this approach unviable. I then thought about using an LLM. In my experiments with identifying books with GPT-4 with Vision, I found a high success rate. Thus, I thought I could use the same approach for vinyls. I could provide each image of a vinyl record to the GPT-4 with Vision API, then ask the model to return the name of the pictured vinyl as well as the artist that wrote it. I added a new section to my script that sends each image recorded with my previous logic to the GPT-4 with Vision API. The following prompt is used: what vinyl record is in this image? return in format: Artist: artist Album Name: name Then, I have Python code that manually extracts the two pieces of requested information: the album name and associated artist: result = response.choices[0].message.content artist = result.split("\n")[0].split(":")[1].strip() album = result.split("\n")[1].split(":")[1].strip() Further investigation is required into more robust methods of extracting the requested information. If the information cannot be extracted, an error is recorded so I know that post-processing has failed for an image. In my testing, GPT-4 with Vision was able to successfully identify my records. With that said, there may be limitations to its abilities that into which I did not run with my collection. Requests to the GPT-4 with Vision API are made concurrently to speed up processing. Then, all results are saved to a CSV file. Here is an example of the results from the CSV file: artist,album Taylor Swift,Red Taylor Swift,Lover Reflections This project demonstrates how an indexing system can be made using out-of-the-box foundation models: MobileCLIP and GPT-4 with Vision. The algorithm described above, and implemented in the source code of this project, could be used with any image embedding model and LLM. The LLM could be substituted for an OCR process with a data lookup and enrichment stage using a music API (i.e. Discogs' search API). In the ideal world, there would be a reverse image lookup API for vinyl record covers, obviating the need for an LLM or OCR entirely, but I was unable to find one. More generally, this project explores and implements the pattern of: 1. Identifying distinct frames with an object of interest, and; 2. Conducting a post-processing step (OCR with data enrichment, querying an LMM) This has broad applications in computer vision tasks in several areas of indexing where there is one primary object of interest in frame. The source code for this project is available on GitHub so you can try it for yourself. You can update the prompts to identify any object you want (i.e. books, succulents). Instructions on how to set up the project are available in the project GitHub repository. Written by human, not by AI Responses Comment on this post Respond to this post by sending a Webmention. Have a comment? Email me at readers@jamesg.blog. Go Back to the Top * Time Machine * Coffee Maps * Projects * Talks * Sitemap * Archive * Index * Privacy * Bookmarks * RSS Webmention logo indicating that you can send a webmention to this blog IndieWeb logo Microformats logo - IndieWeb Webring - [penguin]