[HN Gopher] Show HN: Nutrient insights through your grocery rece...
___________________________________________________________________
Show HN: Nutrient insights through your grocery receipts
Nutri is still in beta and the GPT-powered results are sometimes
inaccurate. The nutrient information accuracy is good to get an
overview, but there are still outliers at times. I'm looking to
improve the accuracy through food databases. Furthermore, I'd like
to add additional tips for combining / preparing food to improve
its nutritional value. For example, iron absorption is improved
through vitamin C, so combine chickpeas or leafy greens with lemon.
Or combine beans with rice to get all amino acids. On the UX side,
I'd like to integrate a QR code on the desktop version to easily
upload receipts through the phone. Furthermore, it would be great
to have analytics over weeks on nutrient improvements over time.
Nutri could also be a great accountability partner to track items
high in sugar / processed foods. What do you think?
Author : elchead
Score : 120 points
Date : 2024-01-21 11:07 UTC (11 hours ago)
(HTM) web link (nutri.adrianstobbe.com)
(TXT) w3m dump (nutri.adrianstobbe.com)
| dotcoma wrote:
| Sounds cool. Does it understand only receipts of products
| available in the US ?
| nomilk wrote:
| I just tried it with a screengrab of a past online order from a
| grocer outside the US and it seemed to work:
| https://imgur.com/a/cZ2Beo4
| Semaphor wrote:
| Works with my German receipt: https://imgur.com/a/Xhw3Acs
| barrenko wrote:
| If it's a finetuned multimodal vision transformer, it's
| probably multilingual.
| luguenth wrote:
| Super interesting project. I also started once a project to index
| food and their ingredients via gpt. The inaccuracy let me abandon
| the project. But never tried the new gpts for that.
|
| One great resource is also: https://world.openfoodfacts.org/
| Semaphor wrote:
| Not something I need, but it's a really cool idea, especially for
| people struggling with nutrition.
|
| One feature suggestion would be to accommodate specific diets
| (keto, vegan, etc.) and generally allow excluding food (e.g. I
| hate the taste of beet roots and eggplants, so I'd always exclude
| those).
|
| One thing that's not clear, is if it understands multiples, I
| uploaded a receipt [0] with 2 feta, 2 broccoli, and 3 arugula,
| but I'm not sure if it recognized that? I also don't know how
| well that can even work without some calculation check, for this
| store (REWE) the amount is always below the product, for Aldi
| it's the opposite https://i.imgur.com/YvqJCle.png
|
| [0]: https://imgur.com/a/Xhw3Acs
|
| [1]: https://i.imgur.com/YvqJCle.png
| nefrix wrote:
| Really cool. I checked also a receipt in Romanian and it worked,
| it translated the ingredients.
| senectus1 wrote:
| Sounds excellent. It would also be super useful for people with
| IBS/FODMAP like dietary issues. It could show you what
| ingredients may be causing flare ups (fructans and galacto-
| oligosaccharides and polyols)
| rob wrote:
| This is amazing, what kind of OCR are you using for the receipts?
| Or are you doing something else?
| infecto wrote:
| Not OP but gpt is multimodal and does an okay enough job for
| something like this workflow. Alternatively all the big cloud
| players and ocr/ml products to read receipts.
|
| You could stitch all of this together for this product in a day
| or two.
| rob wrote:
| Thanks! Yeah, I attached an image to ChatGPT and it worked
| great.
|
| I see a lot of examples that use receipts themselves like
| this, but one idea I had that's kind of similar would be to
| look at just the "ingredients" list on product labels and
| parse those, like these examples (under the nutrition
| labels):
|
| Example 1: https://i.imgur.com/MqpL6yh.png Example 2:
| https://i.imgur.com/3FSK0CD.png
|
| However, using things like pytesseract and Google's Cloud
| Vision API returns mixed results, sometimes missing things,
| transposing lines, etc.
|
| Any ideas on what I could do to improve being able to extract
| ingredients lists from food labels? Would I have to start
| looking into something like Vertex AI and training custom
| models?
|
| Then again, as I'm thinking out loud, I realized if these
| tools can extract all the text pretty reliably, the order and
| place doesn't really matter if you create some extractor
| that's able to just pluck out which words are actual
| "ingredients" based on some master list or something.
| infecto wrote:
| Yes I think using a flavor of the cloud providers document
| tooling is probably optimal. Gpt vision is great for
| general recognition but I found it to be hit or miss when
| you started throwing too much text at it in an image.
|
| If you can get image working via vision that's great. On
| the cloud ocr side, I know tooling like Textract is good
| enough to generally provide output as if you were reading
| left to right. So in theory the text should not be that
| transposed or fragmented and nutrition labels are standard
| enough that you can probably pull the portion you want. On
| top of that, like you allude to, LLMs are pretty good and
| figuring things out.
| DrNosferatu wrote:
| Awesome!
|
| Can you customize target intakes? Say, Food Pyramid / Food Wheel
| / etc.
|
| And vitamin B12 is crucial...
| Sai_ wrote:
| I was going to build something similar but now that you've
| already done the first part, maybe you could run with this idea -
| can you get GPT to generate small stories where characters eat
| the stuff in the grocery list and gain superpowers/are able to do
| big things?
|
| My kid loves superheroes but every meal has to be accompanied
| with a story about why that day's food item is good for him. I
| wanted to use GPT for this. Start by reading grocery bills, then
| have GPT understand the bill and come up with stories since it
| likely knows what health benefit each item in the bill confers.
| gardenhedge wrote:
| That is a strange request for the OP. I doubt they're going
| that direction.
|
| Assuming the number of meals you make is limited, you can use
| ChatGPT directly for this. Just ask it to make stories for your
| meals.
| themdonuts wrote:
| Cool idea. I've actually just taken a pic to my plate, threw it
| at gpt4 and asked it to generate a story. It worked nicely.
| infecto wrote:
| As others already said you can do this out of the box with
| ChatGPT. Take a picture and it will do a good enough job with
| what it identifies.
| Sai_ wrote:
| As a PoC, ChatGPT is fine for this. A kid would expect a more
| polished product which he himself can operate. E.g., maybe
| the UI could show images of items in the grocery bill, show
| images of superheroes he wants to include in the story, text
| to audio to narrate the story, history to see previous
| stories if there is any particular story he really likes...
| streamfunk191 wrote:
| I've been surprised by how good ChatGPT is at this, I took a
| picture of squid ink and mushroom pasta, which I made last
| night, and identified the entire meal accurately, including
| the garnish on top. If you take a second to think about this,
| it's actually incredible.
| mattkrause wrote:
| I can't think of too many other meals with that palette.
| selimthegrim wrote:
| Stories are also the only copyrightable part of recipes.
| shafyy wrote:
| This is cool. As you said, my recommendation would be to not use
| ChatGPT to get the nutrition data, but use one of the databases
| that have good APIs such as USDA's Food Data Central [1]. This is
| domain where there's structured data and it's important that the
| output is deterministic, so a LLM is not a good choice.
|
| I once started working on something similar, but the goal was to
| suggest a list of ingredients for me to buy for the week so that
| I had a good total nutritional values. I don't work on it anymore
| though.
|
| 1: https://fdc.nal.usda.gov/index.html
| sanitycheck wrote:
| I'd second this, I just gave it a try and many of the results
| in the receipt breakdown did not seem to align with the
| nutritional information available from other sources.
|
| Also, if it's going to give protein advice it should probably
| recognise meat - unless it's aimed solely at vegetarians &
| vegans. My receipt had a whole chicken, a lamb shoulder, and a
| couple of different kinds of fish - but it said I was getting 0
| protein and recommended quinoa.
| shafyy wrote:
| Haha, love its passive-aggressiveness. Lowkey pushing people
| to plant-based diets :D
| throwup238 wrote:
| This is how we eventually get Soylent Green.
| lelandfe wrote:
| "Fun tip, you have plenty of protein right at home!"
| assadk wrote:
| > I once started working on something similar, but the goal was
| to suggest a list of ingredients for me to buy for the week so
| that I had a good total nutritional values. I don't work on it
| anymore though.
|
| Cronometer has a feature that does this - it checks to see
| where you aren't hitting your daily targets for
| macro/micronutrients and then suggests foods that would allow
| you to hit those targets. Quite nifty.
| TuringNYC wrote:
| This is awesome! I wanted to do something same but was going with
| the approach that people would be incentivized to enter/map
| nutritional information for line items on their receipt...and
| then implicitly contribute that to the community (like how
| MyFitnessPal works.)
|
| Ultimately there arent that many grocery stores and there arent
| that many popular items, so you'd get an 80/20 approach.
|
| Also curious about the GPT approach, are you worried about the
| cost exploding with a GPT approach?
| danielvaughn wrote:
| The idea has merit. Aside from the demo itself, I'm more
| interested in the business opportunity.
|
| You could partner with grocery stores. The user would have an
| account, not tied to any one particular grocer. It would track
| all of your grocery purchases, and like your demo, give you
| nutrient insights as well as recipes and maybe coupons and money-
| saving tips. Lots of stuff you could do with that.
| eitally wrote:
| There's a company already pursuing this (disclosure: my current
| employer built their site/app for them), but it starts with the
| diet definition and then creates shopping lists, with callout
| links to online grocers.
|
| https://sifter.shop/
| daft_pink wrote:
| I suggest you add restaurant receipts being scanned.
| streamfunk191 wrote:
| I'd love to see a version of this based on additives and
| processed ingredients. These are big contributors to obesity and
| most Americans aren't even aware they're consuming such unhealthy
| food
| apwell23 wrote:
| just try to limit factory food you would be fine. There is no
| need for apps and stuff.
|
| people who are conscious about this dont need an app ( its not
| at all hard to tell food is bad) and ppl who don't care won't
| use the app.
|
| so there is no market for this kind of an app.
| pondemic wrote:
| Hey, how can I contact you?
| callalex wrote:
| This starts from a fundamental assumption that one person shops
| exclusively for only their own food. The vast majority of grocery
| shopping is done for a group of people (a couple, or a family).
| Those people aren't eating an exact 50% even split of every item
| purchased.
| dr_petes wrote:
| There's a "How many people do you shop for" button.
| callalex wrote:
| Yes but that assumes that you eat the exact same ratio and
| quantity as your 4 year old child, for example.
| CrazyStat wrote:
| Sometimes it feels like my 4 year old eats more than me.
| The ratios are definitely different though.
| dr_petes wrote:
| That is an example. Correct. What's the issue?
| dr_petes wrote:
| It isn't telling you how many calories you consumed. It's
| just insights into the nutrition from the food you consume.
| letsoptimizeit wrote:
| This is pretty cool! I built a prototype of something similar
| last year: starting with your family's macros and meals, it
| fetches recipes and a list of ingredients that fit. It's
| structured as a traditional optimization problem so you can, for
| example, minimize the length of your shopping list or meal prep
| time. https://planmyrecipes.com/
| Malic wrote:
| The co-op that I belong to actually emails me PDFs of receipts of
| the groceries that I buy there (to reduce waste). It'd be really
| neat if this could accept PDF receipts as an input option.
| esafak wrote:
| This fails with cash payments, e.g., at farmers' markets. If I
| were in the market, I'd prefer an app that analyzed pictures of
| my actual meals instead. Less work for me, and it represents my
| actual intake. Such an app may already exist.
| crazygringo wrote:
| It just can't be done for meal photos -- there's no way of
| knowing if the mashed potatoes have just a little milk, or two
| sticks of butter. Or if the tomato sauce is pumped full of
| sugar or not. Items that are visually identically can have
| vastly different nutritional profiles.
|
| Not to mention trying to guess what's inside of those dumplings
| or ravioli...
| esafak wrote:
| Start with the picture and maybe a verbal description,
| presenting an estimate of the ingredients and allowing
| corrections.
|
| I should stop talking because this is too much work for me
| either way. Cooking is enough work without all this!!
| golergka wrote:
| I've had similar idea with supermarket receipts about 10 years
| ago, but had trouble with implementation -- OCR APIs were very
| enterprise-oriented and unreliable. However, instead of
| nutrition, I was thinking more into automatic recipes, and re-
| ordering online, getting feedback about the stuff you bought to
| build your taste profile and recommendations.
|
| If you have good data of all the households purchases, you can do
| so much good stuff with it, the problem is getting this data
| reliably.
| virtualritz wrote:
| When I click "Get summary report" on the demo page, a "Generating
| report [...]" spinner shows up for a second and the nothing
| happens. I'm on latest Chrome on Ubuntu 23.10.
|
| Am I missing something?
| nonPlayerChar wrote:
| I don't think it would handle my January receipts well. 1: 50 lbs
| flour, 25lbs sugar 2: 12x 3oz parmesan cheese, 26x cat food
| (wet), 60lbs potatoes 3: 5x 50lbs layer pellets, 50lbs black
| oiled sunflour seeds, 50lbs whole corn, bail of straw 4: 1400lbs
| pig food 5: 20 gallons raw milk Two are from the feed store and
| one is from the dairy.
| asimpletune wrote:
| Does it work with European receipts?
| barbazoo wrote:
| See example
| alliao wrote:
| whatever you do don't sell it to insurance companies...
___________________________________________________________________
(page generated 2024-01-21 23:00 UTC)