[HN Gopher] PaliGemma 2: Powerful Vision-Language Models, Simple...
___________________________________________________________________
PaliGemma 2: Powerful Vision-Language Models, Simple Fine-Tuning
Author : meetpateltech
Score : 112 points
Date : 2024-12-05 17:46 UTC (5 hours ago)
(HTM) web link (developers.googleblog.com)
(TXT) w3m dump (developers.googleblog.com)
| minimaxir wrote:
| Hugging Face's blog post on the release is more technical:
| https://huggingface.co/blog/paligemma2
| xnx wrote:
| Even more technical detail here:
| https://arxiv.org/html/2412.03555v1
| dmvdoug wrote:
| Saw name, was expecting something to do with wreaking AI upon the
| Pali Canon (https://en.m.wikipedia.org/wiki/Pali_Canon).
| turnsout wrote:
| Does anyone know how this stacks up against other multimodal
| vision models?
| mountainriver wrote:
| They do an exceptionally poor job at evaluating it against
| competitors.
| exe34 wrote:
| does anyone know if they can output bonding box coordinates? like
| "where is the ball" -> [50, 75, 150, 175].
|
| so far cogvlm is the only one I've seen that works but it's a bit
| of a pain to run.
| xnx wrote:
| Yes: "The initial four location tokens represent the coordinate
| of the bounding box, ranging from 0 to 1023. These coordinates
| are independent of the aspect ratio, as the image is assumed to
| be resized to 1024 x 1024."
|
| https://developers.googleblog.com/en/gemma-explained-paligem...
| exe34 wrote:
| Thank you! Will have a play with that.
| __jl__ wrote:
| Gemini is surprisingly good at this. Look at example 5 here:
| https://developers.googleblog.com/en/7-examples-of-geminis-m...
|
| They also have a colab notebook with more examples linked in
| the article.
| exe34 wrote:
| I meant weights-available ones, but thank you!
| sigmar wrote:
| It is probably hard to come up with good benchmarks for VLMs like
| this, but I feel like the "Non entailment sentences" benchmark
| seems ill-suited. The examples for sentences that were non
| entailment included[1]: "There is a pile of horse manure in front
| of the horse." Which is true if you mean "in front of the
| [photosubject] from the perspective of the camera," but I think
| they marked it as non entailment because the pile is not in front
| of the horse's face(?)
|
| [1] page 20 https://arxiv.org/pdf/2412.03555
| timmg wrote:
| I recently wanted to try to get an LLM to help me organize my
| photos. (I'm someone who takes a lot of photos when I travel and
| then back them up to a hard drive -- assuming someday I'll
| organize them :)
|
| I created a prompt to try to get an LLM to do high-level
| organization:
|
| > Help categorize this photo for organization. Please output in
| JSON.
|
| > First, add a field called "type" that is one of: Wildlife,
| Architecture, Landscape, People or other. Pick the one that most
| appropriately reflects the subject of the photo. Use Other if you
| don't feel confident in your answer.
|
| > Next, if it is Wildlife, please add another field called
| "animal" that gives a general type of animal that is the focus of
| the photo. Use large, common types like Elephant, Bird, Lion,
| Fish, Antelope, etc. Do not add this field if your confidence is
| low.
|
| > If the type of animal is Bird, add a field called "bird" that
| gives the common type of bird, if you can clearly determine it.
| For example: Eagle, Hummingbird, Vulture, Crow, etc.
|
| > If it is an Architecture photo, and you can determine with good
| confidence what specific building (or city) it is a photo of,
| please add a field called "place" with that name. (Name only,
| please -- no description).
|
| I've tried with llama-vision using Ollama and it worked
| reasonably well for the top-level categories. A little less-well
| for identifying specific birds or places. And it didn't always
| generate proper JSON (and sometimes added new fields to JSON.)
|
| I also tried with Claude's API -- and it seemed to work perfectly
| (for a small sample size).
|
| It will be interesting to try with PaliGemma and see what I get.
|
| I have like 50k photos, so I don't want to pay $$$ for the Claude
| API to categorize them all. It will be cool someday (soon?) for
| an open-source DAM to have something like one of these models
| available to call locally.
| visarga wrote:
| use the json mode in ollama
| senko wrote:
| Simonw estimates it'd cost less than $10 to categorize 67k+
| photos using Amazon Nova:
| https://simonwillison.net/2024/Dec/4/amazon-nova/#gamoa
|
| I agree it'll still be cool to be able to do it all locally.
| rsolva wrote:
| The photo organizing software Ente [0] can do this, and is
| packaged into a really neat product. I have not gotten around
| to try the self hosted version yet, but it is on my list!
|
| [0] https://ente.io/ml
___________________________________________________________________
(page generated 2024-12-05 23:00 UTC)