[HN Gopher] A new old kind of R&D lab
___________________________________________________________________
A new old kind of R&D lab
Author : jph00
Score : 176 points
Date : 2023-12-12 16:19 UTC (6 hours ago)
(HTM) web link (www.answer.ai)
(TXT) w3m dump (www.answer.ai)
| malux85 wrote:
| Is there a way of contacting these people? If you're in this
| thread could you email me please? (in my profile), I can show you
| some cool ideas and what I am building!
| nwoli wrote:
| Maybe DM one of the founders (who has DMs open to everyone)
| https://twitter.com/jeremyphoward
| eries wrote:
| good idea
| andyjohnson0 wrote:
| _" but they were also on their way to being controlled and
| understood by a tiny exclusive slither of society."_
|
| I think that should be "sliver".
| OJFord wrote:
| I was going to say they're synonymous; checked Wiktionary which
| calls it 'nonstandard' for sliver, though common in the UK
| (where I am from and live) blaming 'th fronting'.
|
| I assume that's the name for, ahem, 'that is just anovver word
| for it'.
| jph00 wrote:
| I'm Australian so I didn't know about this -- I guess I'll
| switch to "sliver" since it seems it's more broadly
| understood.
|
| Thanks for letting me know!
| OJFord wrote:
| No wuckers ;)
|
| I didn't know either, I only really checked because I was
| curious if they had a completely different etymology and
| only happened to be spelt^ and used similarly.
|
| (^or if, like spelt and spelled, the same root had just
| come to be used in two ways for the same.)
|
| So, note to self, slither is not a noun! (Except to mean
| limestone rubble apparently, but I think I can ignore
| that.)
| WendyTheWillow wrote:
| In American English, "slither" is more frequently associated
| with the movement of snakes, specifically. A snake "slithers"
| by "moving smoothly over a surface with a twisting or
| oscillating motion." [0]
|
| [0] https://www.google.com/search?q=define%3A+slither
| AnimalMuppet wrote:
| Depends on your opinion of the reptile nature of those doing
| the controlling.
| agravier wrote:
| An animal muppet would know.
| hiddencost wrote:
| So many people writing their pitches start by talking about
| scientists from the 1800s, these days. It's a pretty big red flag
| to me.
| CamperBob2 wrote:
| How so?
| WendyTheWillow wrote:
| Scope issues, mostly. I'm left unsure of what products to
| expect, so I'm less likely to follow up/check in later on. A
| _lot_ of time is spent on the analogy, too, which may mean
| there 's not much substance to say yet. It may have been
| worth waiting to announce until they had _something_ specific
| to present. I don 't know! I'm just an outsider/random
| person's perspective.
| swyx wrote:
| as someone guilty of the same - it makes your startup grander
| than it seems by tracing lineage from greater historical
| figures to yourself. of course most of these comparisons are
| overinflated... but you need to be a little ambitious to try
| to start something. if you live your life without trying to
| be a part of history you have a much lower chance of
| affecting it
| aliston wrote:
| While Faraday discovered induction, wasn't it Maxwell that
| unified electricity and magnetism? Given what answer.ai is
| attempting to do, Edison seems like a great example since he
| was both a brilliant inventor and an absolutely shrewd
| businessman.
|
| I am excited for more research in this area, since there is
| currently a huge gap between foundational model research and
| practical applications of AI.
| patcon wrote:
| Not sure why it's a flag. We have lots to learn from how
| science was done in the past, and from the actors who did
| science.
|
| Recent science is pretty objectively at a low point
| (proportionally to overall) in "breakthrough innovation"
| research. It's possible that specialization is to blame, as it
| reduces intersectionality of fields.
|
| Details here:
|
| 'Disruptive' science has declined -- and no one knows why
| https://www.nature.com/articles/d41586-022-04577-5
| andy99 wrote:
| Building on top of what exists makes sense. There's science and
| then engineering and we need more engineering. I am curious
| though how far the $10M will go and what the plan is. Building on
| top still needs some kind of training and the money won't go very
| far for anything large scale. I know they know this, I'm just
| interested to know the plan.
| jph00 wrote:
| Yeah we actually have to make money! We can't just spend.
| JohnHammersley wrote:
| Here's a short thread on the announcement, from Jeremy Howard,
| one of the founders:
| https://twitter.com/jeremyphoward/status/1734606378331951318
| jph00 wrote:
| Hi folks -- Jeremy from Answer.AI (and fast.ai!) here. Happy to
| answer any questions you have about this new thing that Eric and
| I are building.
|
| One thing I'm particularly keen to explore is working closely
| with academic groups to help support research that might help
| make AI more accessible (e.g. requiring less data or compute, or
| becoming easier to use, etc). This includes the obvious stuff
| like quantization, fine-tuning adaptors, model merging,
| distillation, etc, but also looking for new directions and ideas
| as well which might otherwise be hard to fund (since academia
| tends to like to build on established research directions.)
|
| I've opened my DMs at https://twitter.com/jeremyphoward for a
| while so feel free to ping me there, or also on Discord
| ('jeremyhoward').
| reqo wrote:
| If I understand correctly, your focus will be on smaller models
| and what can be built on top of them? How relevant do you think
| fine-tuning small models will be once AGI is widely accessible?
| Do you think smaller models can co-exit in a AGI future?
| ska wrote:
| Conversely, how relevant do you think 2023 techniques will be
| at all in a hypothetical AGI future? Doesn't the relevance of
| these questions embed some pretty strong assumptions?
| coderintherye wrote:
| Great to see this effort! Couldn't think of a better duo to
| experiment with it!
|
| >with academic groups to help support research that might help
| make AI more accessible
|
| Jeremy, are you already talking with Sky Computing Lab? I
| recall they had an interesting project about SkyPilot which
| seemed helpful
|
| https://sky.cs.berkeley.edu/ https://github.com/skypilot-
| org/skypilot
| phillipcarter wrote:
| Something I've learned as I've dived a bit into fine-tuning is
| that it's really hard. I even have what I think is good data,
| and a lot of it. But, as a non-ML engineer who just wants to
| use these tools as good APIs (and someone with the ability to
| ensure I have a coherent data pipeline), I woulnd't know how to
| proceed if I wasn't already working with an ML engineer who's
| been doing this sort of stuff for a while.
|
| Specifically, I fine-tuned gpt-3.5 and llama-2-7b on some real-
| world usage data, and I can't tell a single difference between
| the quality of these outputs compared to "base" gpt-3.5 with
| the same requests made to it. Moreover, if I attempt to remove
| a lot of the "static" sections of a prompt (after all, there's
| 2k+ lines of data it was fine-tuned on where this is all
| present), both models just go completely off the rails.
|
| I'd love to get into a world where I can fine-tune a bunch of
| different models. But it's so, so much harder than just calling
| OpenAI's API and getting really good results with that and some
| prompting work. If you're able to help crack that nut then
| there's a lot of people like me who would pay money to have
| their problems solved.
| adw wrote:
| 2k lines is (probably) not a lot of data.
| fuddle wrote:
| I'm looking forward to seeing what what projects you work on at
| Answer.ai! I'm a big fan of your fast.ai courses and all the
| work you've done in the AI industry.
| tikkun wrote:
| Looks great! What kinds of products might the lab build?
| glth wrote:
| This is a great initiative! Given what you want to achieve, I
| presume you will have a wide range of (atypical) profiles in
| your team.
| eries wrote:
| Without a doubt
| soulofmischief wrote:
| Congrats on launching! This initiative looks awesome, and I'd
| definitely love to take you up on that and chat about it for a
| bit to understand the vision and what kind of engineering
| culture you're hoping to build. Sent a friend request on
| Discord!
| gardenhedge wrote:
| "team of deep-tech generalists--the world's very best,
| regardless of where they live, what school they went to, or any
| other meaningless surface feature."
|
| How are you planning on hiring?
| behnamoh wrote:
| I like your tutorials so much! I think you're a gift to the ML
| community. How did you guys secure $10m VC while putting this
| on the webpage? I'm not criticizing or anything--just want to
| know how one can pitch an uncertain idea and yet receive
| generous funding.
|
| > We don't really know what we're doing If you've read this
| far, then I'll tell you the honest truth: we don't actually
| know what we're doing. Artificial intelligence is a vast and
| complex topic, and I'm very skeptical of anyone that claims
| they've got it all figured out. Indeed, Faraday felt the same
| way about electricity--he wasn't even sure it was going to be
| of any import:
|
| > "I am busy just now again on Electro-Magnetism and think I
| have got hold of a good thing but can't say; it may be a weed
| instead of a fish that after all my labour I may at last pull
| up." Faraday 1931 letter to R. Phillips
|
| > But it's OK to be uncertain. Eric and I believe that the best
| way to develop valuable stuff built on top of modern AI models
| is to try lots of things, see what works out, and then
| gradually improve bit by bit from there.
|
| > As Faraday said, "A man who is certain he is right is almost
| sure to be wrong." Answer.AI is an R&D lab for people who
| aren't certain they're right, but they'll work damn hard to get
| it right eventually.
|
| > This isn't really a new kind of R&D lab. Edison did it
| before, nearly 150 years ago. So I guess the best we can do is
| to say it's a new old kind of R&D lab. And if we do as well as
| GE, then I guess that'll be pretty good.
| jph00 wrote:
| That's a very fair question! The key is to find aligned
| investors. In this case, we found investors who believe in
| the fundamental opportunity, have the patience for us to
| figure out how to capture it, and believe that we're the
| right people to do it.
| Centigonal wrote:
| I'm excited by this, and I wish y'all the best!
| eries wrote:
| thank you!
| kaycebasques wrote:
| Very excited to see how the Lean Startup guy applies his own
| ideas!
| kaycebasques wrote:
| Hi Jeremy, thanks for fast.ai and Kaggle and that refreshingly
| honest & open interview on Latent Space. It sounds like you and
| Lattner are on good terms. Any plans to partner up with Modular?
| jph00 wrote:
| I love Chris and his work is amazing. I certainly hope we can
| work closely with Modular, although nothing specific in place
| yet. I was at ModCon last week and I think that Mojo might just
| be the future...
| eries wrote:
| Eric Ries here, happy to answer questions about Answer.AI or any
| of the related themes Jeremy talked about in the announcement
| post: rapid iteration, R&D, startup governance, long-term
| thinking, etc.
|
| Excited to see what comes out of this new lab. And if you're
| interested in joining the cause, please do get in touch. Both
| Jeremy and I are on this thread and generally reachable.
| patcon wrote:
| Thanks!
|
| What are your thoughts on this model of promoting breakthrough
| innovation?
|
| How to fund Breakthrough Innovations in Science (Puja Ohlhaver
| @ DeSci.Berlin) https://www.youtube.com/watch?v=guLDNMAOn24
|
| Puja has a few talks on such things, many very related and
| worth listening to imho. But most relevant: she's been working
| on a mechanism design to use quadratic funding in an existing
| hierarchy to move funding power from funders to on-the-ground
| researchers who best predict "breakthrough research" areas --
| i.e. at which intersections. This idea of "breakthrough
| innovation" is objectively measured and rewarded as "research
| that becomes highly cited, and which draws together disparate
| source citations that have never before appeared together."
|
| So the idea is that in successive funding rounds, funding power
| slowly accrues in the people who best predict where research
| innovation will appear. Even if that turns out to be *gasp*
| grad students.
|
| (I'm particularly interested to see Polis, a "wiki survey" tool
| I've been using since 2016, be used as one of the signals in
| such a system. It can help make the landscape of beliefs and
| feelings that ppl bring to the process more legible, especially
| at the collective level. Which is important, because high-
| dimensional "feeling data", when placed out-of-scope in other
| systems, are often a reason why we get trapped in local minima
| of innovation that inhibit the recombination of ideas.)
| eries wrote:
| I was going to link to Polis after I read the first part of
| your answer, but I see you've beaten me to it. And in so
| doing you've pretty much answered your own question. Thanks!
| ibestvina wrote:
| How do you look at hiring "experienced people" vs.
| "enthusiastic interns" on something like this? More generally,
| how quickly do you think the team will grow, and what the ratio
| should be between the "old" and the "young"?
| eries wrote:
| Very hard to guess how it might all shake out. I would say
| that both Jeremy and I have an almost fanatical belief in the
| power of uncredentialed outsiders. So I would guess we will
| be more looking for curious open-minded generalists more than
| any specific age or experience level. I do expect we will
| grow headcount rather slowly, but that doesn't mean we will
| launch infrequently
| wslh wrote:
| Are you open to work with other companies that are already
| working in the field? Or you are limiting participants to
| individuals?
| eries wrote:
| I expect quite a bit of partnering to make sense, though
| nothing concrete to share at this time. We explicitly
| designed this to be non-competitive with the best companies
| in the field (who have the things they do well covered).
| amrrs wrote:
| For those who aren't family familiar with Jeremy Howard.
|
| Jeremy was a cofounder and chief scientist of Kaggle (a
| competitive ML platform)
|
| Jeremy also started Fast AI with Rachel Thomas. Fastai is one of
| the best ways to learn Deep learning even today.
|
| Jeremy is a great teacher and have been a voice of debunking AI
| paranoia and closed models.
|
| Really rooting for Jeremy!
| ssheth wrote:
| He is also the cofounder of Fastmail .. a popular email
| provider with others here..
| jwuphysics wrote:
| Hi Jeremy & Eric, great to see your newest endeavor. I hope that
| Answer.AI builds on the success and impact that fast.ai has
| already enjoyed.
|
| Given new developments in hardware (by companies not named
| NVIDIA), I'm wondering if you are keen on exploring the next
| generation of model architectures and optimization procedures
| that might exploit newer hardware. In other words, will research
| directions pivot based on the hardware lottery?[1] Are you in
| conversations with companies developing these alternative chips?
|
| [1] https://arxiv.org/abs/2009.06489
| ghj wrote:
| Will you still be working on educational content on the side?
| (e.g. updating fast.ai and/or making one off lectures like
| https://www.youtube.com/watch?v=jkrNMKz9pWU)
|
| Either way thank you for all the amazing free content you've
| already put out and good luck on the new endeavor!
| moh_maya wrote:
| Some questions, if you are still around to answer, because this
| is exciting!
|
| 1) Have you read about Vannevar Bush, and what he has written,
| and his body of work?! :)
|
| 2) What sort of people are you looking for / to work with?
|
| 3) Would it need to be full-time? Are you looking to hire people
| full time (the generalists you mention), or are you comfortable
| working with people who are happy not cashing a cheque from you
| because they have jobs and other commitments / priorities, but
| still believe in what you are building and would like to invest
| significant time in supporting / driving the mission forward for
| some limited (or no) financial compensation? Because I'd like to
| check if I fit! :)
|
| (I've also spammed you on twitter with a dm, but with more
| personal details, etc.)
|
| Thank you!
| ansk wrote:
| > a new kind of AI R&D lab which creates practical end-user
| products based on foundational research breakthroughs
|
| This isn't new and if anything it's the de facto standard for
| just about every AI research lab these days. OpenAI is the
| obvious example of an AI lab with tightly coupled product and
| research roadmaps and ChatGPT is the most prominent example of a
| successful research-driven AI product. A few years ago it could
| be argued that DeepMind and (fka) FAIR were siloed off from their
| respective orgs, but these days they are littered with product
| teams and their research roadmaps reflect this influence as well.
|
| They do try to claim that what they are doing is different from
| OpenAI because they are focused on applications of AI whereas
| OpenAI is focused on building AGI, which is a laughable
| mischaracterization of OpenAI's current roadmap. I personally
| have a hard time believing that path to AGI runs through the GPT
| store.
|
| Accomplished researchers in AI can fundraise on their reputations
| alone, and Jeremy is no exception. The primary differentiator of
| any new startup in this space is the caliber of its researchers
| and engineers. But this post is really grasping at straws to
| claim that their value is from some new approach to R&D, which is
| a totally unnecessary framing.
| IceMichael wrote:
| When will the hype curve finally end...
| NtochkaNzvanova wrote:
| Is there really a shortage of companies that are trying to do
| foundational AI research, and also build the results of that
| research into end-user products? Off the top of my head, the list
| of such companies would include... you know, literally every
| large tech company. If the idea here is that they can do it at a
| much smaller scale, and more cheaply, that's great. But it's not
| clear to me from this article what the radical new approach is
| that will enable that.
|
| I wonder if putting out effectively a press release before
| actually doing the work is the right approach. If they launch a
| product or two and they flop, people will say this approach was
| doomed from the start. It would be better to create a compelling
| product in stealth, successfully launch it, then reveal how it
| was done. That would create more buy-in to the idea that such
| small R&D labs can work.
| your_friend wrote:
| Sounds like a lab I was born for.
|
| I think there's a big group of individuals out there that are
| misfits both for regular jobs and science. Entrepreneur-ish
| generalists.
___________________________________________________________________
(page generated 2023-12-12 23:00 UTC)