[HN Gopher] How we used GPT-4o for image detection with 350 simi...
       ___________________________________________________________________
        
       How we used GPT-4o for image detection with 350 similar
       illustrations
        
       Author : olup
       Score  : 211 points
       Date   : 2025-01-10 21:02 UTC (4 days ago)
        
 (HTM) web link (olup-blog.pages.dev)
 (TXT) w3m dump (olup-blog.pages.dev)
        
       | olup wrote:
       | First time for me posting this kind of story - I thought it would
       | make an interesting case on solving a hard computer vision
       | problem with a crafty product engineer team.
        
         | caioariede wrote:
         | Just a small feedback... I have switched to the reader mode
         | because the font used is very challenging to read for me.
        
           | littlestymaar wrote:
           | Also, having a blog post about image detection, and not
           | showing a single picture in the whole post was quite
           | frustrating.
        
             | Oarch wrote:
             | Especially given the detailed description surely the author
             | could just generate a similar image
        
               | bl4ckneon wrote:
               | Just thinking that. Spend a few minutes trying to have
               | chatgpt generate some images with Dall-E 3. Flux would
               | probably be better to get all the specific details but ya
        
         | yannis wrote:
         | Thanks for sharing. Interesting approach. As other commenters
         | mentioned, article could do well with some hypothetical images.
         | Maybe on a follow-up blog post? Also since you mentioning your
         | Company's name you missing an opportunity for marketing by not
         | providing a link.
        
         | idkman_oops wrote:
         | Can you tell me what font is this?
        
           | martin_a wrote:
           | One of these:
           | 
           | > ui-monospace,SFMono-
           | Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier
           | New",monospace
        
           | nmeofthestate wrote:
           | The single-character-width 'fi' ligature is quite jarring in
           | a mono-spaced font.
        
       | vessenes wrote:
       | Thanks for the "bitter lesson" news from the frontlines. Curious;
       | did you experiment with 4o as the sole pipeline? And of course as
       | I think you mention, it would be interesting to know if say llama
       | 8b could do a similar job as well.
       | 
       | Congrats on shipping.
        
         | numba888 wrote:
         | they don't self-host the models, neither embedding nor last
         | step llm. taking into account low load self-hosting likely
         | would be more expensive. if so why not to use the best models.
        
       | schappim wrote:
       | I would love to see the prompt / image data sent to GPT-4o!
        
         | olup wrote:
         | On the prompt side, it's very simple, and can probably be done
         | in a variety of ways. How we did it is to prepare a prompt with
         | multiple "user" messages. The first one gives the instruction
         | 
         |  _you are given a reference and three candidates, which one of
         | the candidates do you think is a match to the reference? Only
         | output its identifier or a code when none is found_
         | 
         | Not exactly that but something along those lines.
         | 
         | Then one "user" message per car (reference + candidates) with
         | image + text indicating the type (reference or candidate) and
         | an identifier (can be as simple as the index for the
         | candidates).
        
       | gazchop wrote:
       | I hear a lot of qualitative speak but nothing quantitative.
        
       | GaggiX wrote:
       | Is there a reason to choose VGG16 over more modern models?
        
         | rmiaouh wrote:
         | Yes, there is a reason. VGG16 is a lightweight model that is
         | very cost-effective to self-host. Initially, as the article
         | mentions, we didn't have access to large, cost-efficient models
         | (like AWS Titan) capable of generating image embeddings. As a
         | result, we opted for VGG16, which is efficient, delivers good
         | performance, and can run on CPUs with just 4GB of RAM. This
         | makes it ideal for small-scale setups, such as VPS instances
         | costing around EUR10/month.
        
       | saint_yossarian wrote:
       | I mean, cool tech, but why not just print a QR code next to each
       | illustration?
        
         | nnnnico wrote:
         | just in: using gpt4o to read QRs
        
         | urbandw311er wrote:
         | This feels like one of those "NASA spent millions developing a
         | space pen, Russians took a pencil" moments.
        
           | webmaven wrote:
           | ... which ignores the hazards of pencil shavings in a zero-g
           | environment, especially the graphite, a good electrical
           | conductor.
        
             | psandor wrote:
             | and that NASA did not spend millions on this.
             | https://www.reuters.com/article/fact-check/nasa-did-not-
             | spen...
        
         | JayShower wrote:
         | Sounds like the client cared a lot about the user experience
         | being smooth (they declined the solution of presenting the user
         | with the narrowed-down choices of which car they took a picture
         | of), and I think adding a bunch of QR codes to this aesthetic
         | wall of car illustrations would not align with that goal.
        
         | cuu508 wrote:
         | Or just a human-readable label with the model and year on it.
         | Visitors would not need to mess with gadgets to read the labels
         | which would be a huge usability win.
        
         | nthingtohide wrote:
         | Or just geo tag the room itself.
        
         | olup wrote:
         | Poster here. We would have loved that, and it was one of our
         | first proposal - a QR code or some kind of marker. However, the
         | client is understandably very controlling on the aesthetics of
         | their wall as a central element of their scenography. We would
         | have pushed for it again in the last resort, but would probably
         | have lost the contract.
        
           | psandor wrote:
           | This is completely offtopic, but I would bet it was a
           | government-funded museum. A reasonable institution would have
           | worked with you to find an acceptable compromise, something
           | much easier to implement with a small sacrifice of
           | aesthetics.
           | 
           | Anyway, great work, and thank you for taking the time to
           | share it!
        
             | achierius wrote:
             | Really? I would much less expect a government museum to be
             | particular about aesthetics. Privately run
             | museums/collections/exhibitions on the other hand tend to
             | have very finicky owners -- after all, they're putting up
             | their own money to achieve their _vision_ , and so of
             | course they tend to not want to compromise on how it might
             | look.
        
       | kredd wrote:
       | A bit tangential, but I think we will see a good chunk of small
       | teams building competing products in different software business
       | segments, by just doubling on productivity and offering a cheaper
       | option due to less operational overhead (reads: paying
       | engineers). I can think of at least two businesses that can be
       | competed in costs if the team can automate a good chunk of it.
        
         | qeternity wrote:
         | > I can think of at least two businesses that can be competed
         | in costs if the team can automate a good chunk of it.
         | 
         | And which would those be?
        
           | kredd wrote:
           | We both know I didn't write it down with the hopes that I'll
           | act on the at some point in the near future, and want to
           | avoid my imaginary competitors. Even though, in reality, I
           | will ponder about it for another week or two, give up without
           | actually getting anything done, then regret for never trying
           | :)
        
             | qeternity wrote:
             | We both know I was hoping you'd tell me anyway :)
        
           | satvikpendem wrote:
           | Job applications, recruiter outreach and initial screening
           | calls. I heard of an AI interviewer via voice chat on a
           | reddit thread recently.
        
             | rad_gruchalski wrote:
             | ,,AI" talking to an ,,AI". What a time to be alive.
        
       | Imnimo wrote:
       | It's tough to judge without seeing examples of the targets and
       | the user photos, but I'm curious if this could be done with just
       | old-school SIFT. If it really is exactly the same image in the in
       | the corpus and on the wall, does a neural embedding model really
       | buy you a lot? A small number of high confidence tie points seems
       | like it'd be all you need, but it probably depends a lot on just
       | how challenging the user photos are.
        
         | Morizero wrote:
         | I find a lot of applied AI use-cases to be "same as this other
         | method, but more expensive".
        
           | Terr_ wrote:
           | Better to spend $100 in op-ex money than spend $1 in cap-ex
           | money reading a journal paper, especially if it lets you tell
           | investors "AI." :p
        
             | mattnewton wrote:
             | Your engineers cost <$1/hr and understand journal papers?
        
               | Terr_ wrote:
               | The 100-vs-1 is a ratio.
        
           | relativ575 wrote:
           | Use cases such as?
        
             | Morizero wrote:
             | I'm in an AI focused education research group, and most
             | "smart/personalized tutors" on the market have similar
             | processes and outcomes as paper flashcards.
        
           | kjkjadksj wrote:
           | That was happening even when they were still calling it
           | machine learning in the papers. Longer before that still.
           | It's the way some people reliably get papers out for better
           | or worse. Find a known phenomenon with existing published
           | methods, use the same dataset potentially using new method of
           | the day, show there's a little agreement between the old
           | "gold standard" and your method, and boom, new paper for your
           | cv on $hotnewmethod you can now land jobs with. Never mind no
           | one will cite it. That's not the point here.
        
           | miki123211 wrote:
           | It's often vastly more expensive to inference, but vastly
           | cheaper and faster to train / set up.
           | 
           | Many LLM use cases could be solved by a much smaller,
           | specialized model and/or a bunch of if statements or regexes,
           | but training the specialized model and coming up with the if
           | statements requires programmer time, an ML engineer, human
           | labelers, an eval pipeline, ml ops expertise to set up the
           | GPUs etc.
           | 
           | With an LLM, you spend 10 minutes to integrate with the
           | OpenAI API, and that's something any programmer can do, and
           | get results that are "good enough".
           | 
           | If you're extremely cash-poor, time-rich and have the right
           | expertise, making your own model makes sense. Otherwise,
           | human time is more valuable than computer time.
        
         | relativ575 wrote:
         | From TFA:
         | 
         | > LLMs and the platforms powering them are quickly becoming
         | one-stop shops for any ML-related tasks. From my perspective,
         | the real revolution is not the chat ability or the knowledge
         | embedded in these models, but rather the versatility they bring
         | in a single system.
         | 
         | Why use another piece of software if LLM is good enough?
        
           | titzer wrote:
           | Cost. Same reason you don't deliver UPS packages with B-2
           | bombers.
        
             | msp26 wrote:
             | The cost of LLM inference is cheap and will continue to
             | decrease. More traditional methods take up far more of an
             | engineer's time (which also costs money).
             | 
             | If I have a project with a low enough lifetime inputs I'm
             | not wasting my time labelling data and training a model.
             | That time could be better spent working on something else.
             | As long as the evaluation is thorough, it doesn't matter.
             | But I still like doing some labelling manually to get a
             | feel for the problem space.
        
           | comex wrote:
           | Performance. A museum visitor may not have a good internet
           | connection, so any solution that involves uploading a photo
           | to a server will probably be (much) slower than client-side
           | detection. There's a thin line between a magical experience
           | and an annoying gimmick. Making people wait for something to
           | load is a sure way to cross that line.
           | 
           | Also privacy. Do museum visitors know their camera data is
           | being sent to the United States? Is that even legal (without
           | consent) where the museum is located? Yes, visitors are
           | supposed to be pointing their phone at a wall, but I suspect
           | there will often be other people in view.
        
       | gunalx wrote:
       | Cool real life use Case. Don't think lmms usually get applied
       | reasonably where they should be and I am glad that a generic knn
       | model also was used to simplify costs and also just more
       | suitable.
        
       | wongarsu wrote:
       | Interesting approach to a a very interesting challenge, given how
       | close the images supposedly are.
       | 
       | With the limited training data they have I'm surprised they don't
       | mention any attempts at synthetic training data. Make (or buy) a
       | couple museum scenes in blender, hang one of the images there,
       | take images from a lot of angles, repeat for more scenes,
       | lighting conditions and all 350 images. Should be easy to script.
       | Then train YOLO on those images, or if that still fails use their
       | embedding approach with those training images.
        
         | brody_hamer wrote:
         | They did.
         | 
         | > " To address this limitation, we turned to data augmentation,
         | artificially creating new versions of each image by modifying
         | colors, adding noise, applying distortion, or rotating images.
         | By the end, we had generated 600 augmented images per car."
        
           | wongarsu wrote:
           | Those are pretty standard. A standard YOLO training run
           | applies more transformations than that, and there are ready-
           | made modules that do the same in keras and pytorch (for their
           | mobilenet and VGG16). I'm not sure if anyone is training any
           | serious vision algorithm without that kind of data
           | augmentation.
           | 
           | What I am talking about is that they want to recognize scenes
           | containing the images, but only have the images as training
           | data. They have a good idea what those scenes will look like.
           | Going there to take actual training pictures was evidently
           | not viable, but generating approximations of them might have
           | been.
        
       | TZubiri wrote:
       | Calling an llm and a cv model by the same name to give the
       | appearance of agi is a pet peeve of mine.
       | 
       | And someone that's not openai buying into this naming convention
       | is just unpaid propaganda
        
         | throwaway314155 wrote:
         | How would you prefer people talk about it? "Multimodal LLM"? My
         | understanding is the vision portion is indeed wired directly to
         | (and trained alongside) the language portion.
         | 
         | > give the appearance of agi
         | 
         | Can you point out where specifically they're doing this? Best I
         | can tell, they give a decent summary of the effectiveness of
         | multi-modal LLM's with support for vision, and then talk about
         | using it to solve an incredibly narrow task. The only diction I
         | could see that hints at "agi" is when they describe the
         | versatility of this approach; but how could you possibly argue
         | against that? It's objectively more versatile (if not wasteful
         | and more expensive).
        
           | TZubiri wrote:
           | I looked into the documentation and api, and it seems you are
           | right, it is genuinely part of the gpt model. Of course, we
           | cannot confirm without source code.
           | 
           | My understanding was that there was a traditional cv library
           | that was effectively producing an image to text before
           | passing it to the llm. But the more I think about it, even
           | that method would involve training for image detection to a
           | point where objects are recognized by images not by tokens.
           | 
           | So the gpt product is no longer an llm or text based.
           | 
           | Can't say much for sure at this point with closed source, we
           | will probably see competition catch up eventually and have
           | more info then. At which point openai will eventually release
           | the text2img separately and dispense with the mysticism and
           | agi pretention.
           | 
           | My guess is that this is a separate image to text model ( or
           | image+text model) and it is slapped on to the main llm code.
           | 
           | I don't think that text is just another modality, it probably
           | will always be the core.
           | 
           | I don't have a source on something as strategic and
           | subjective, I just have an finger on the pulse: their robot
           | demo that does laundry, their consistent talk about AGI,
           | their mention of power-seeking in docs, their attempt to
           | raise trillions for chip factories, transition to for profit.
           | They have a huge pressure to be THE monopoly and their risk
           | is for GPT to be a text based local maximum and for
           | intelligence not to be a sappir wolphian phenomenon.
           | 
           | P.s: early docs from 2023 refer to the img2txt submodel as
           | gpt4v, that's what we should call the submodule in my
           | opinion. (If it in fact is the same piece of tech)
        
       | JayShower wrote:
       | Alternative solution that would require less heavy lifting of ML
       | but a little more upfront programming: It sounds like the cars
       | are arranged in a grid on the wall. Maybe it would be possible to
       | narrow down which car the user took a photo of by looking at the
       | photos of the surrounding cars as well, and hardcoding into the
       | system the position of each car relative to one another? Could
       | potentially do that locally very quickly (maybe even at the level
       | of QR-code speed) versus doing an embedding + LLM.
       | 
       | Con of this approach would be that it's requires maintenance if
       | they ever decide to change the illustration positions.
        
         | armchairhacker wrote:
         | Put each painting in an artsy frame whose edges are each
         | different, colorful pattern. When the user photographs the
         | painting, they'll include all (or even most) of the frame, and
         | distinguishing the frames is easy.
        
           | arkh wrote:
           | > artsy frame
           | 
           | Embedding a QR code or simply a barcode somewhere and you're
           | done. Maybe hide it like a watermark so it does not show to
           | the naked eye and doing some Fourier transform in the app
           | won't require a network connection nor lot of processing
           | power.
        
             | ndileas wrote:
             | the article does mention that the client rejected a similar
             | approach. steganography seems like a bad choice for a
             | museum setting where you don't own the images.
        
           | jaffa2 wrote:
           | This seems the way to go... its only 350 images
        
       | babyent wrote:
       | This was a fun read. I'm not a AI expert by any means. I'm also
       | ESL. Please bear with me.
       | 
       | However the inaccuracy threshold seems fine for a museum, but in
       | enterprise operations inaccuracy can mean lost revenue or worse
       | lost trust and future business flow.
       | 
       | I'm struggling with some more advanced AI use cases in my
       | collaborative work platform. I use AI (LLMs) for things like
       | summarizations, communication, finding information using
       | embedding. However, sometimes it is completely wrong.
       | 
       | To test this I spent a few days (doing something unrelated)
       | building up a recipes database and then trying to query it for
       | things like "I want to make a quick and easy drink". I ran the
       | data through classification and other steps to get as good data
       | as I could. The results would still include fries or some other
       | food result when I'm asking for drinks.
       | 
       | So I have to ask what the heck am I doing wrong? Again, for
       | things like sending messages and reminders or coming up with
       | descriptions, and finding old messages that match some input - no
       | problem.
       | 
       | But if I have data that I'm augmenting with additional
       | information (trying to attach more information that maybe missing
       | but possible to deduce from what's available) to try and enable
       | richer workflows I'm always being bit in the butt. I feel like if
       | I can figure this out I can provide way more value.
       | 
       | Not sure if what I said makes sense.
        
         | numba888 wrote:
         | > Not sure if what I said makes sense.
         | 
         | Not sure either. But here is the lesson from this and other
         | sources. To improve the output use multistep approach. Get the
         | first answer, one or more, and pass it through the second
         | verification step(s). Like 'for this * is this *' relevant? Or
         | is it correct, does it solve the problem, etc.. Then select the
         | answer with the best scores on the filters. You see, it's very
         | similar to that in the original post. Get first candidates,
         | filter.
        
       | rldjbpin wrote:
       | reads to me like 95% of the "conventional AI" was applied to the
       | problem and then using llm in the end seems to work like a lucky
       | three-faced dice.
       | 
       | when "embeddings" are used to perform closeness test, you are
       | using a pretrained computer vision model behind the scenes. it is
       | doing the far majority of tasks of filtering out hundreds of
       | images down to a handful.
       | 
       | visual llm works on textual descriptions that seem far too close
       | for similar images. regardless, more power to the team for
       | finding something that works for them.
        
         | og_kalu wrote:
         | >visual llm works on textual descriptions
         | 
         | SOTA V-LLMs do not work on textual descriptions.
        
       | sashank_1509 wrote:
       | This has been my experience. Foundation models have completely
       | changed the game of ML. Previously, companies might have needed
       | to hire ML engineers familiar with ML training, architectures etc
       | to get mediocre results. Now companies can just hire a regular
       | software engineer familiar with foundation model API's to get
       | excellent results. In some ways it is sad, but in other ways the
       | result you get is so much better than we achieved before.
       | 
       | My example was an image segmentation model. I managed to create
       | an dataset of 100,000+ images and was training UNets and other
       | advanced models on it, always reached a good validation loss but
       | my data was simply not diverse enough and I faced a lot of issues
       | in actual deployment, where the data distribution kept changing
       | on a day to day basis. Then, I tried DINO v2 from Meta, finetuned
       | on 4 images and it solved the problem, handled all the variations
       | in lighting etc with far higher accuracy than I ever achieved. It
       | makes sense, DINO was train on 100M + images, I would never be
       | able to compete with that.
       | 
       | In this case, the company still needed my expertise, because Meta
       | just released the weights and so someone had to setup the fine-
       | tuning pipeline. But I can imagine a fine tuning API like
       | OpenAI's requiring no expertise outside of simple coding. If AI
       | results depend on scale, it naturally follows that only a few
       | well funded companies, will build AI that actually works, and
       | everyone else will just use their models. The only way this trend
       | reverses, is if compute becomes so cheap and ubiquitous, that
       | everyone can achieve the necessary scale.
        
         | bboygravity wrote:
         | Could DINO or some other model be used to identify fillable
         | form fields in webforms and/or PDF forms and/or desktop apps?
         | 
         | Or does it likely just work on real world photos and cartoons
         | and stuff?
        
           | rolisz wrote:
           | There are dedicated models for recognizing UI elements such
           | as form fields. One example is
           | https://github.com/microsoft/OmniParser
        
         | pmontra wrote:
         | > The only way this trend reverses, is if compute becomes so
         | cheap and ubiquitous, that everyone can achieve the necessary
         | scale.
         | 
         | We would still need the 100 M+ images with accurate labels.
         | That work can be performed collectively and open sourced but it
         | must be maintained etc. I don't think it will be easy.
        
           | goldemerald wrote:
           | DinoV2 is an unsupervised model. It learns both a high
           | quality global image representation and local representations
           | with no labels. It's becoming strikingly clear that
           | foundation models are the go to choice for common data types
           | of natural images, text, video, and audio. The labels are
           | effectively free, the hard part now is extracting quality
           | from massive datasets.
        
           | EGreg wrote:
           | The other way it can reverse is discovering better methods to
           | train models, or fine-tune existing ones with LoRA or
           | whatever.
           | 
           | How did Chinese companies do it, is it a fabricated claim?
           | https://slashdot.org/story/24/12/27/0420235/chinese-firm-
           | tra...
        
         | pj_mukh wrote:
         | Just 4 images?! Damn. I've had to do at least in the 100's. I
         | guess it depends on the complexity of the segmentation.
        
         | isoprophlex wrote:
         | This was exactly my experience being the ML engineer on a
         | predictive maintenance project. We detected broken traffic
         | signs in video feeds from trucks; first you segment, then you
         | classify.
         | 
         | Simply yeeting every "object of interest" into DINOv2 and
         | running any cheap classifier on that was a game changer.
        
           | ac2u wrote:
           | Could you elaborate? I thought DINO took images and outputted
           | segmented objects? Or do you mean that your first step was
           | something like a yolo model to get bounding boxes and you are
           | just using dino to segment to make the classification part
           | easier?
        
             | isoprophlex wrote:
             | We got bboxes from yolo indeed to identify "here is a
             | traffic sign", "here is a traffic light" etc. Then we
             | cropped out these objects of interest and took the DINOv2
             | embeddings of them.
             | 
             | Not using it to create segmentations (there are YOLO models
             | that do that, so if you need a segmentation you can get it
             | in one pass), no, just to get a single vector representing
             | each crop.
             | 
             | Our goal was not only to know "this is a traffic sign", but
             | also do multilabel classification like "has graffiti", "has
             | deformations", "shows decoloration" etc. If you store those
             | it becomes pretty trivial (and hella fast) to pass these
             | off to a bunch of data scientists so they can let loose all
             | the classifiers in sklearn on that. See [1] for a
             | substantially similar example.
             | 
             | [1] https://blog.roboflow.com/how-to-classify-images-with-
             | dinov2
        
               | ac2u wrote:
               | Understood. Thanks for taking the time to elaborate.
        
         | NegatioN wrote:
         | I haven't compared image models in a long while, so I don't
         | know the relevant performance metrics. But even a few years
         | ago, you would usually use a pretrained model, and then
         | finetune on your own dataset though. So those models would also
         | have "seen millions of images", and not just your 100k.
         | 
         | This change of not needing ML engineers is not so much about
         | the models, as it is about easy API access for how to finetune
         | a model, it seems to me?
         | 
         | Of course it's great that the models have advanced and become
         | better, and more robust though.
        
         | IanCal wrote:
         | Things like DINO, GroundingDINO, SAM (and whatever the latest
         | versions of those are) are incredible. I think the progress in
         | this field has been overlooked given LLMs, they're less end-
         | user friendly but they're so good compared to what I remember
         | working with.
         | 
         | I was able to turn around a segmentation and classifier demo in
         | almost no time because they gave me fast and quick segmentation
         | from a _text description_ and then I trained a YOLO model on
         | the results.
        
       | suriya-ganesh wrote:
       | This tracks with my experience. We built a complex processing
       | pipeline for an NLP classification, search and comprehension
       | task. Using vector database of Proprietary data etc.
       | 
       | We ran a benchmark of our system against an LLM call and the LLM
       | performed much better for so much cheaper, in terms of dev time,
       | complexity, and compute. Incredible time to be in working in the
       | space seeing traditional problems eaten away by new paradigms
        
       | the_duke wrote:
       | Side question: is there any good model that allows for image
       | similarity detection across a large image set, that can be
       | incrementally augmented with new images?
       | 
       | You'd somehow have to generate an embedding for each image, I
       | presume.
        
         | ResearchAtPlay wrote:
         | Yes, you could implement image similarity search using
         | embeddings: Create embeddings for the entire image set, save
         | the embeddings in a database, and add embeddings incrementally
         | as new images come in. To search for a similar image, create
         | the embedding for the image that you are looking for and
         | compute the cosine similarity between that embedding and the
         | embeddings in your database. The closer the cosine similarity
         | is to 1.0 the more similar the images.
         | 
         | For choosing a model, the article mentions the AWS Titan
         | multimodal model, but you'd have to pay for API access to
         | create the embeddings. Alternatively, self-hosting the CLIP
         | model [0] to create embeddings would avoid API costs.
         | 
         | Follow-up question: Would the embeddings from the
         | llama3.2-vision models be of higher quality (contain more
         | information) than the original CLIP model?
         | 
         | The llama vision models use CLIP under the hood, but they add a
         | projection head to align with the text model and the CLIP
         | weights are mutated during alignment training, so I assume the
         | llama vision embeddings would be of higher quality, but I don't
         | know for sure. Does anybody know?
         | 
         | (I would love to test this quality myself but Ollama does not
         | yet support creating image embeddings from the llama vision
         | models - a feature request with several upvotes has been opened
         | [1].)
         | 
         | [0] https://github.com/openai/CLIP
         | 
         | [1] https://github.com/ollama/ollama/issues/5304
        
           | jonathan-adly wrote:
           | So, there is a whole world with vision based RAG/search.
           | 
           | We have a good open-source repo here with a ColPali
           | implementation: https://github.com/tjmlabs/ColiVara
        
             | ResearchAtPlay wrote:
             | Thanks for the link to the ColPali implementation -
             | interesting! I am specifically interested in evaluation
             | benchmarks for different image embedding models.
             | 
             | I see the ColiVara-Eval repo in your link. If I understand
             | correctly, ColQwen2 is the current leader followed closely
             | by ColPali when applying those models for RAG with
             | documents.
             | 
             | But how do those models compare to each other and to the
             | llama3.2-vision embeddings when applied to, for example,
             | sentiment analysis for photos? Do benchmarks like that
             | exist?
        
               | jonathan-adly wrote:
               | The "equivalent" here would be Jina-Clip (architecture-
               | wise), not necessarily performance.
               | 
               | The ColPali paper(1) does a good job explaining why you
               | don't really want to directly use vision embeddings; and
               | how you are much better off optimizing for RAG with a
               | ColPali like setup. Basically, it is not optimized for
               | textual understanding, it works if you are searching for
               | the word bird; and images of birds. But doesn't work well
               | to pull a document where it's a paper about birds.
               | 
               | 1. https://arxiv.org/abs/2407.01449
        
               | ResearchAtPlay wrote:
               | Makes sense. My main takeaway from the ColPali paper (and
               | your comments) is that ColPali works best for document
               | RAG, whereas vision model embeddings are best used for
               | image similarity search or sentiment analysis. So to
               | answer my own question: The best model to use depends on
               | the application.
        
         | lyu07282 wrote:
         | Beyond using off-the-shelf embeddings, if you want to teach a
         | model what "similar" means exactly, that's metric learning:
         | https://paperswithcode.com/task/metric-learning
        
       | yuvalr1 wrote:
       | A completely different approach that don't require heavy AI would
       | be an app on the user phone that does this:
       | 
       | 1. Measure the distance from the wall (standard image processing)
       | 
       | 2. Use the rotations of the gyro sensors on the phone to conclude
       | which car is being looked at
       | 
       | I wonder if this could be as accurate though
        
         | mrbombastic wrote:
         | You could definitely cheat a little with something like this or
         | geofences, but that requires the photos stay in the same place
         | or the museum updates whenever they move.
        
       | lynguist wrote:
       | Huh I think this YouTube short is the same topic:
       | https://youtube.com/shorts/DA_-6296G5o?si=BLKcSP2Q1jAaca9K
       | 
       | Finding new geoglyphs from known examples.
        
       | hackerdood wrote:
       | Very neat explanation of solving these kinds of unique
       | challenges, especially given how similar the illustrations were.
       | 
       | One question I had was, knowing how difficult it was to train the
       | model with the base images, and given that the client didn't have
       | time to photograph them, did you consider flying someone out to
       | the museum for a couple of days to photograph each illustration
       | from several angles with the actual lighting throughout the day?
       | Or potentially hiring a photographer near the museum to do that?
       | It seems like a round trip ticket plus a couple nights in a hotel
       | could have saved a lot of headache, providing more images to turn
       | into synthetic training data. Even if you still had to resort to
       | using 4o as a tiebreaker, it could be that you only present two
       | candidates as the third might have a much lower similarity score
       | to the second candidate. Good write up either way.
        
       ___________________________________________________________________
       (page generated 2025-01-14 23:01 UTC)