[HN Gopher] Long Live the 'GPU Poor' - Open-Source AI Grants
___________________________________________________________________
Long Live the 'GPU Poor' - Open-Source AI Grants
Author : rajko_rad
Score : 281 points
Date : 2023-08-30 16:37 UTC (6 hours ago)
(HTM) web link (a16z.com)
(TXT) w3m dump (a16z.com)
| bodecker wrote:
| Congrats to the folks involved.
|
| Even knowing this is partly motivated by branding/marketing, it's
| great to see a16z getting more aligned with solving real pain
| points (vs crypto and churning out shallow media in recent
| years). Hope they can keep it up and hopefully more "thought
| leaders"/VCs follow suit. Best of luck.
| jacquesm wrote:
| No, they're still saving the world, only now not through crypto
| but through AI...
|
| https://a16z.com/2023/06/06/ai-will-save-the-world/
| skeeter2020 wrote:
| this just seems like a cost-effective attempt to buy
| credibility and stay relevant, but I'm a crusty, old
| curmudgeon so what do I know.
| bodecker wrote:
| Agreed, but IMO it's their job to do stuff like this (and
| prob unrealistic to expect a world where it doesn't exist).
| I'd rather see a higher percentage of marketing for things
| that have more real value vs. less
| bodecker wrote:
| Of course - there will always be investors/influencers
| pushing narratives to hype their bets. IMO everyone is better
| off when the hype is grounded in solving real pain points,
| which the AI projects I've seen seem to be closer to than
| most of the crypto projects I've seen.
|
| Certainly there are plenty of grifters in AI too (as with any
| gold rush) and many AI efforts will fizzle out. But it seems
| there is more real value being created here than in crypto,
| which is the main thing I'm excited about and hope to see
| more of
| calderwoodra wrote:
| I agree this initiative helps solve real pain points, and I'm
| not trying to defend "Crypto" with my question here but isn't
| solving a "real pain" kinda subjective? For example, many
| people/companies have the real pain/need to improve ad click-
| through-rates and reduce crypto/gas fees but maybe these aren't
| problems you care about.
| samstave wrote:
| Solving pain point == A problem we can profit from. Always
| has been.
|
| Plus A16Z has some questionable motives in general based on
| previous comments by them...
| calderwoodra wrote:
| I guess sometimes a pain can exist without a profit,
| there's only a small number of people willing to solve
| those pains without a profit incentive though (pro bono
| lawyers, doctor's without borders, etc).
| bodecker wrote:
| Agreed, "real pain" is a subjective/vague term and even
| crypto is a broad term. My intended use here is to describe
| solutions that are more grounded/integrated into reality vs.
| more grounded in a fantasy of how the world works. It is hard
| to describe well though and if you keep unrolling most
| perceptions of reality are subjective
| syntaxing wrote:
| Whoa this is honestly awesome, a lot of tools I see on the list
| are ones I've been using so it's great to see they're getting
| financial support. Really hoping Ollama joins the list soon. I
| have some fun ideas I've been wanting to test out but probably
| requires a couple hours on 8X A100s + EvolveInstruct style data
| that cost upward of $1K+. Gonna start preparing some code to see
| if I can join grant #2!
| rajko_rad wrote:
| Love it!! Good luck with the mvps!
|
| And thank you so much for the input - we are meeting the folks
| at ollama!!
| roschdal wrote:
| [flagged]
| stuaxo wrote:
| This is great, but "dual 4090s" bring this stuff down to the
| Ryzen APU level.
| naillo wrote:
| A16z is getting 'mad respect' from me for taking this line of the
| 'ai wars' and generally their attitude on all this. Really cool
| that they're going straight into 2010 era startup energy on this
| new wave. Idk I just find them really cool and inspirational.
| outside1234 wrote:
| If nothing else, it is good to see them moving off of the
| Bitcoin grift
| [deleted]
| [deleted]
| zekrioca wrote:
| I'm not sure why an application link wasn't provided. How is one
| suppose to reach them to apply?
| [deleted]
| svnt wrote:
| "We are disguising this round of seed funding as a grant,
| accomplishing all of the below:
|
| 1) brand-washing for people who don't do the math
|
| 2) avoiding angering our LPs and/or violating our fund theses and
|
| 3) getting in on the ground floor of things that might follow
| OpenAI through the non-profit-to-VC-darling door
|
| Thanks for reading our press release, feel free to try to apply
| using your personal network (somehow) because we don't care as
| we've already landed our positions."
| [deleted]
| evanwolf wrote:
| This is insurance for their already huge investments (and
| future rounds) in AI businesses. It's a pittance relative to
| everything else they have at risk. It's prudent since failure
| of small projects like these can delay work on profitable
| corporate efforts.
| notatoad wrote:
| unless i'm missing something, they're not taking equity. that's
| what makes it a grant, and not seed funding.
|
| what math should i be doing here?
| svnt wrote:
| Probably the grants have clauses that convert them to debt or
| equity in case of a transition of the company. [0]
|
| The bigger math is significance_of_contribution =
| $size_ai_grant_program/$size_ai_vc_program (or maybe
| $size_ai_grant_program - $size_ai_vc_program)
|
| [ 0 https://link.springer.com/chapter/10.1007/978-3-030-72428
| -3_... ]
| disiplus wrote:
| if you don't have a source for your claim it's as valid as
| i'm claiming that you are a stegosaurus.
| lee101 wrote:
| [dead]
| ricardo81 wrote:
| Leftfield thought, imagining a tax (vaguely like carbon credits)
| on energy/cpu cycles/modelling (just my vague idea). With the
| proceeds going to other projects to diversify the outcomes.
|
| Always seems to me our economy is being less based on money and
| more on energy and productivity, basically being rounded down to
| physics as a healthy market economy should.
| politelemon wrote:
| Very happy to see oobabooga on the list. Everyone's contributing
| in different ways, and ooba serves as a fantastic portal to try
| out various models which I appreciate. Its readme even says it
| wants to be like the automatic 1111 of text generation. For
| beginners like me who are curious about this new world, ooba is a
| great entry point.
| rajko_rad wrote:
| We agree!!
| appenz wrote:
| We 100% agree!
| wslh wrote:
| We work in source code security auditing with semi-automated
| tools using AI and other techniques.
|
| Is this approach covered in the grants?
| rajko_rad wrote:
| if the work is OSS/public and you aren't trying to start a VC-
| backed company, it could be! We usually want to see some early
| work already published, community reception and validation, but
| feel free to DM Matt or myself and we can take a look!
| wslh wrote:
| > if the work is OSS/public and you aren't trying to start a
| VC-backed company, it could be!
|
| Yes, it is a company that often applies and receives grants.
| For example, we are working on adding AI to security tools
| such as https://github.com/CoinFabrik/scout
| itissid wrote:
| Noob Q: If I have 25000$ can I buy some GPUs and support the
| effort for supporting GPU Poor? Like is there a DIYer's guide to
| do this?
|
| I don't want to donate the cash but instead set up a handful of
| GPUs(Somehow?) and let people pay at cost to use them.
| rajko_rad wrote:
| honestly would recommend using spot/rented instances on that
| budget vs. buying! A100s run around ~$2/hour, the hard part is
| getting a large allocation... training workloads are quite
| spikey anyways, usually running lots of smaller experiments
| before actually doing a larger single run...
| itissid wrote:
| Assume location is East Coast(NYC).
| capableweb wrote:
| vast.ai can help you offer this and has bunch of docs on how
| you can start hosting: https://vast.ai/docs/hosting/hosting
|
| You won't be able to limit it to a particular set of clients
| who want to use it, but you're basically supporting the "GPU
| Poor" by doing this, as I think it's mostly smaller companies,
| individuals and researchers using vast.ai, rather than huge
| companies.
| brailsafe wrote:
| No intention to attribute malice, but wouldn't this be more
| like buying a condo and renting it out to pay your mortgage
| rather than donating resources?
| rajko_rad wrote:
| we didn't buy a condo haha, are letting people choose their
| own housing :)
| redox99 wrote:
| 25K will either get you money for renting servers, or allow you
| to jerry rig yourself something like a 8x4090 machine.
|
| The servers that most people would like to run (8xA100, 8xH100)
| are over 200k, and even if you had the money they're probably
| very hard to get right now.
| naillo wrote:
| 8x4090 machines should be a lot cheaper than that.
| 1599*8=12,792. I.e. closer to 16x4090 machines. Sure labor is
| worth something but he has it in cash, and also there's
| barely a lot of people (yet) paying high prices for research
| (non ML ops) labor hours for this yet so the salary wouldn't
| be that high anyway.
|
| Just a pet peeve when ppl under exaggerate how many
| gpus/flops a dollar can buy (if you're slighly smart with
| your money).
| redox99 wrote:
| Meh, I was just ballparking. But add a threadripper,
| compatible motherboard, lots of RAM, fast storage, a few
| beast PSUs, and tax, and you're probably around 18k. So you
| ain't getting 2 of these machines for 25k.
|
| Plus you'll need to set aside some money for the power bill
| lol
| bcatanzaro wrote:
| Also possibly upgrading the wiring in your house or
| office.
| junipertea wrote:
| What about PSU to power all of this, GPU motherboards,
| SSDs, CPU to not bottleneck this and sufficient RAM? You
| could argue it's more than 8, but definitely not 16. To
| utilize those GPUs to the max you need memory bandwidth to
| even feed it enough data.
| fyloraspit wrote:
| Yeh I built a dual 3090 machine and just to have those 2
| cards run well cost a lot for all the other stuff.
| Workstation spec mobo, top tier PSU, CPU and memory with
| headroom for the system, not to mention storage and case
| and cooling. And this is just a hobbyist build. Cost
| would have easily doubled if I actually got server grade
| CPU and memory etc.
| jxf wrote:
| Q: What are the grant amounts?
| retrocryptid wrote:
| I'm still not sure what I'm supposed to be using AI for. I
| already know how to code, tried for a decade to make ACR (auto
| content recognition) work with CNNs, and as best I can tell,
| chatGPT can write a news article for you if you write it first
| and then hire someone else to edit it.
| redox99 wrote:
| I want to congratulate you based on your current grant
| recipients. They are all very impactful individuals/teams. It
| shows that you clearly did your research and put the money in the
| right places.
| rajko_rad wrote:
| thank you!! Indeed, we have been interacting in space very
| closely and doing our best to keep up and contribute where
| possible
| malux85 wrote:
| Is there a way we can apply? I am building
| https://atomictessellator.com, and since I am 100% technical
| I am not sure about commercialising or open sourcing it, I
| definitly would like to open source some of the components
| (e.g. turnkey dockerized scientific computing environments,
| the molecular force viewer, and also open up a hub for the
| environments and models) - when I was in Stanford meeting
| with some comp chemists a few months ago they got very
| excited for this.
|
| I negotiated with my day job to work 2AM to 10AM so that I
| can work on this passion project in the afternoons, some
| grant funding would really help me focus on this more.
| There's quite a bit of interesting stuff not on the website
| in catalyst discovery and molecular dynamics. Contact me if
| you'd like to chat / demo.
|
| In my ideal world I would work on this full time and totally
| open source it.
| rajko_rad wrote:
| feel free to DM on twitter or linkedin. In a nutshell, if
| you want to pursue a company now in or in the near future,
| we would likely need to consider you through our normal
| investing process for equity funding. If you are
| considering just working on it OSS, we would encourage
| taking first steps in that direct, and testing community-
| project fit validation before quite going full or near-full
| time!
| tikkun wrote:
| What are other open source LLM projects that people rely on? As a
| way to come up with ideas for future grant recipients.
|
| This is a good initiative, and an excellent initial batch. A
| great use of management fees!
|
| I'm struggling to think of additional ones because I feel like
| this covers everyone I would've suggested (particularly TheBloke,
| vLLM and oobabooga).
|
| Edit, some ideas for future grant recipients would be:
|
| * MLC LLM
|
| * ExLlama
|
| * An open source fork of text-generation-inference
|
| * AutoGPTQ
| rajko_rad wrote:
| Awesome suggestions, thank you!
| Michelangelo11 wrote:
| This is great, but I'm not seeing anything about how to apply, or
| really anything about the application and selection process at
| all. Is this by design?
| tikkun wrote:
| My guess is that they'll prefer to get nominations.
|
| Having once done a grant application process - I think their
| approach is smart.
|
| If they opened grant applications, they'd get thousands of
| (sadly mostly very low quality) applications. It's extremely
| hard to vet.
|
| It's a much safer bet to wait until projects are already
| popular in the community, already being relied on by people,
| and where those devs would love to spend more time on their
| open source projects but they're limited by money.
|
| Yes, the downside of this is that some people could make things
| happen if they were given money first. But, if I was them, I'd
| still prefer to do what they're doing and pick projects that
| are already successful and give grants to those.
|
| (This isn't a comment about your project specifically, for all
| I know your project might be a popular open source LLM project
| that lots of people already rely on, in which case they
| probably already know about your project and will probably
| reach out to you for the second round, is my guess)
| [deleted]
| zekrioca wrote:
| It seems so.
| intalentive wrote:
| TheBloke has had this sponsorship in his byline for a little
| while now.
| seism wrote:
| Hi Robin Hood. I wonder about the "continue their work without
| the pressure to generate financial returns" part. I don't see an
| expectation to work on non-commercial, or socially conscious,
| applications, just to commit to open source (what level of
| commitment also being unclear). It would actually be great if
| support programs along these lines were being offered by
| governments and public service institutions, with some serious
| governance and transparency around the process.
|
| Surely there are also some interesting econometrics on the "GPU
| poverty line" - someone has literature to share?
| evanwolf wrote:
| Yeah, this is moneyed interests looking out for their own
| investments instead of government acting on behalf of the
| public good.
| RcouF1uZ4gsC wrote:
| > Eric Hartford: fine-tuning uncensored LLMs
|
| Makes me very happy that this is being undertaken.
|
| LLMs will be a very valuable tool. Having censorship built into
| these tools makes me very nervous. Imagine if Word refused to
| allow you to type any "bad word" in the name of safety and
| ethics.
| satvikpendem wrote:
| Well, at least this time a16z is giving money to actually useful
| technologies instead of bloviating about the world changing
| potential of cryptocurrencies. I imagine they've written down
| most of their cryptocurrency investments at this point.
|
| That being said, these are some great grants for local LLM
| builders, local computation for such powerful tools is needed,
| and will continue to be needed, as industry incumbents like
| OpenAI continue to not release their models.
| [deleted]
| espadrine wrote:
| I had become honestly cynical of their actions in the past, as
| they felt hype-driven and externalities-blind. Yet I don't see
| how this grant could become rapacious. Hopefully it is an
| attempt at cleaning the brand which can work as making amends.
| zitterbewegung wrote:
| Don't VCs basically chase everything because they only rely on
| 10% or less to succeed but, that 10% needs to have at least 10x
| or more profit and then they do another round?
| artninja1988 wrote:
| Thank you for doing this. Although I'm sure it's not purely for
| altruistic reasons, open source ai is important:)
| Havoc wrote:
| Amazing. Those are really solid 1st picks. A huge chunk of the
| organic experimentation relies directly on them.
| rajko_rad wrote:
| thanks so much - we agree!
| thinktown wrote:
| What was actually being offered here? Money? How much? This seems
| like such vague VC-marketing speak.
| [deleted]
| UncleOxidant wrote:
| "We believe artificial intelligence has the power to save the
| world"
|
| I don't agree with the side that says AI is about to destroy
| everything, it seems very hyperbolic... but neither do I agree
| with this sentiment that it's going to save the world.
| andrewprock wrote:
| Agree. It's not at all obvious the world is in need of saving.
|
| That said, software of all kinds (including machine learning)
| does make the world more interesting and exciting, so happy to
| see more investment.
| hgsgm wrote:
| I don't even understand. A16Z's crypto already saved the world.
| Why is the world in trouble again?
| geodel wrote:
| It's to make it even more savier. Think if someone could
| combine LLAMA with blockchain and create Lockchain. This is
| kind of saviness we all need.
| [deleted]
| [deleted]
| minimaxir wrote:
| It's interesting to note that there's particular emphasis on
| "removing censorship from LLM outputs" given a16z's investments
| outside the AI space in some ideological social media companies.
| mirekrusin wrote:
| They should call this initiative Open AI.
| capableweb wrote:
| LibreAI for extra fun :)
| jph00 wrote:
| I'm one of the recipients of the AI grants, to support my work at
| fast.ai. I'm extremely grateful to a16z for their support. Here's
| some additional details based on questions I see in the comments:
|
| My grant covered the purchase of a Scalar server from Lambda
| Labs, which allowed me to configure a system with 8 previous-gen
| A6000 GPUs, partly also thanks to NVIDIA who has recently given
| me 3 of those cards, and Lambda Labs who offered to provide
| everything at cost.
|
| a16z didn't ask for any equity or any kind of quid pro quo, other
| than to let folks know that they provided this grant. They
| couldn't have been more helpful through the process - I didn't
| have to fill out any forms (other than sign a single one page
| agreement), the contract was clear and totally fair (even
| explicitly saying that a16z wasn't going to receive any kind of
| rights to anything), and they wired me the money for buying the
| server promptly.
| Mascobot wrote:
| Thanks for all the work Jeremy! There's a large community of
| folks that have been learning from you, the Fast.ai community
| and your work. Super excited for this grant!
| rajko_rad wrote:
| Honored to have you in the inaugural cohort @jph00!!
| jasrys wrote:
| Couldn't go to a better home. Thanks for making this field
| approachable.
| Kon-Peki wrote:
| I'm glad you guys are sharing some money around, but I really
| have a hard time thinking of a person with "Dual 4090s" as "GPU
| Poor". LOL
| ForHackernews wrote:
| It's like how you're house poor if you only have one brownstone
| in Brooklyn and a pied-a-terre in Palo Alto.
| [deleted]
| qntty wrote:
| I interpret "GPU poor" the same way as "house poor", i.e.
| you're poor (lack cash) because you've use it to purchase GPUs,
| not that you don't have enough GPUs.
| KaoruAoiShiho wrote:
| No, GPU poor comes from this definition:
| https://www.businessinsider.com/gpu-rich-vs-gpu-poor-tech-
| co...
|
| Less than 10k H100s makes you GPU poor. The author is
| presumably using it as a proxy for "possible to achieve AGI".
| capableweb wrote:
| It's all relative. Compared to the huge companies spending
| thousands per month/week/day on GPU usage, having two 4090s
| (which you could acquire for like ~$3000) could be considered
| on the lower end for sure.
| jacquesm wrote:
| Depending on where you live what you pay for power may well
| be a bigger factor than what you've paid for the hardware.
| skilled wrote:
| > We believe artificial intelligence has the power to save the
| world--and that a thriving open source ecosystem is essential to
| building this future.
|
| Thank you for understanding the fundamental flaws of the World
| and human consciousness, and thank you for spending multiple
| lifetimes as Buddhist monks (not to mention lamas) to understand
| the intricacies of the human mind from a scientific perspective.
| This is exactly what we need right now and this initiative looks
| to be the answer to all of our problems.
| [deleted]
| htrp wrote:
| ehartford and thebloke are definitely great picks advancing SOTA
| in underserved fields.
|
| Curious why there are no image ai projects receiving grants here?
| appenz wrote:
| No deeper reason. I think there just a lot in LLMs happening
| right now which skewed it towards them. We would love to do
| something in the SD ecosystem.
| rajko_rad wrote:
| Great callout @htrp... @appenz is our expert here btw !
| GaggiX wrote:
| This reminds when a random chinese user finetuned the leaked
| NovelAI Stable Diffusion model and released it online, the
| Anything V3 model, tech company like Tencent started using it in
| their products.
| waihtis wrote:
| hail e/acc and the thermodynamic god
| rajko_rad wrote:
| Hi HN, we are super excited to announce this new initiative!!
|
| Please note, this program was designed to support individuals,
| teams, and hackers who are not pursuing commercial companies.
| Nevertheless, these projects push the state of the art in open
| source AI and help provide us with a more robust and
| comprehensive understanding of the technology as it is developed.
|
| We are really proud to be contributing in this small fashion and
| grateful to the first cohort and all others contributing in this
| space!
| thewataccount wrote:
| Hello, I think this is a great project!
|
| If an individual is looking to contribute to the field with
| different training data ideas - would they need to first
| establish themselves and get your attention, or would there be
| a way to submit a proposal?
|
| For myself compute is rather expensive, I could likely afford a
| few test runs for proofs of concept but beyond that it would be
| difficult. I've got a single 4090 so I can't run llama70b
| faster then 1it/s.
| rajko_rad wrote:
| Thank you so much for your work and enthusiasm here!!
| Unfortunately, that's roughly the best way to approach it and
| how most of these have worked so far, demonstrations on
| smaller models (e.g. 7/13B), with slightly smaller datasets,
| catching the eye of the community, etc. in general it's not a
| bad approach to research too, prove out concepts at smaller
| scale before scaling up!
| thewataccount wrote:
| Totally understand! I'm sure you'd get swarmed with
| requests.
|
| Are you interested in non-LLMs as well? Stable diffusion
| for example?
| rajko_rad wrote:
| Definitely. This was kicked off in the last month or two
| when there just seemed to be a bit more happening in OSS
| LLMs, but we are prioritizing diversifying this for the
| next cohort!
| latchkey wrote:
| I'm curious, where are you getting the GPUs?
| rajko_rad wrote:
| in most cases the teams already had their own source or
| platform of choice!
| samstave wrote:
| Maybe build a FAQ for others on how to accomplish, or
| resources you can point people at?
| rajko_rad wrote:
| Interesting, hadn't thought of that, thank you! If you
| want to host end points Replicate is a great option, they
| also have a newer fine tuning api and solution! for raw
| VMs with GPUs right now it's a bit situational and you
| have to try multiple different vendors tbh, also really
| depends on the capacity you need and which machines!!
| tikkun wrote:
| I've done a version of this:
| https://news.ycombinator.com/item?id=36632397
|
| Let me know what you'd want to see added!
| kcorbitt wrote:
| A lot of people are using RunPod for experimental/small-
| scale workloads. They have good network and disk speeds
| and you can generally find availability for a latest-gen
| GPU like an L40 or 4090 if your workload can fit on a
| single GPU. One GPU is plenty for fine-tuning Llama 2 7B
| or 13B with LoRA or QLoRA. They also _sometimes_ have
| availability for multi-GPU servers like 8xA100s, but that
| 's more hit-or-miss.
|
| If you want to go even cheaper vast.ai is a popular
| option. It's a P2P marketplace for individuals to rent
| out their GPUs. You can generally get a ~20-30% discount
| vs RunPod prices by using Vast, but network speeds and
| perf are much more variable and there's always the
| possibility that the host will just shut you off without
| warning. I also wouldn't recommend using it if you're
| training with proprietary data since they can't guarantee
| the host isn't logging it, but most of the OSS fine-
| tuning community publishes their datasets anyway.
| qwertox wrote:
| Thank you!
| rajko_rad wrote:
| Our pleasure :)
___________________________________________________________________
(page generated 2023-08-30 23:01 UTC)