[HN Gopher] Ask HN: How to learn AI from first principles?
___________________________________________________________________
Ask HN: How to learn AI from first principles?
A variant of this question seems to get asked every 6 mo. but so
far, I haven't seen this question tackled directly: _If I want to
learn the concepts and fundamentals of AI from first principles,
what educational resources should I use?_ I'm not interested in
hands-on guides (eg. how to train a DNN classifier in TensorFlow)
or LLM-centric resources. So far, I've put together the following
curriculum: 1 Artificial Intelligence: A Modern Approach
(https://aima.cs.berkeley.edu/) - Great for learning the breadth of
foundational concepts, eg. local search algorithms, building up to
modern AI. 2 Probabilistic Machine Learning: An Introduction
(https://probml.github.io/pml-book/book1.html) - Going more in-
depth into ML. 3 Dive into Deep Learning (https://d2l.ai/) - Going
deep into DL, including contemporary ideas like Transformers and
Diffusion models. 4. Neural networks and Deep Learning
(http://neuralnetworksanddeeplearning.com/) could also be a great
resource but the content probably overlaps significantly with 3.
Would anybody add/update/remove anything? (Don't have to limit
recommendations to textbooks. Also open to courses, papers, etc.)
Sorry for the semi-redundant post.
Author : HardikVala
Score : 22 points
Date : 2025-01-26 05:20 UTC (17 hours ago)
| Mr-Frog wrote:
| I really enjoyed the concepts in "Artificial Intelligence, a
| Modern Approach" which really grounds a first-principles
| foundation of automated reasoning. Warning: The first 80% of the
| book doesn't have any sexy new deep learning approaches, but I
| still think it is very valuable to see the history.
| HardikVala wrote:
| +1, I'm a few chapters in and its highly instructive. Gives me
| a deeper appreciation for the modern deep learning regime.
| Also, as we enter the agent supercycle, I think many of the
| basic algorithms for search, planning, etc. will make comeback
| a in a huge way.
| InkCanon wrote:
| The question depends what you mean by first principles. Usage of
| the phrase "first principles" has sprawled into many different
| things since (I think) Musk first mentioned it as a way to learn.
| The original, philosophical meaning of first principles meant a
| fundamental truth which could be used to derive others. Much of
| the philosophising of thinkers like Aristotle or Descartes was to
| uncover these truths (eg I think, therefore I am). In physics and
| other sciences, it means calculations using established laws,
| rather than approximations or assumptions. Then it got borrowed
| into certain circles of the tech crowd with the vague meaning of
| thinking about what's important or true and ignoring the rest.
| Then it trickled down into the learning/self help world as a hack
| of some sort to learn. If we take the original meaning of first
| principles, there aren't a great deal of absolute truths in
| machine learning. It is a very empirical, approximated and
| engineering oriented endeavor. Most of the research involves
| thinking of a new approach, building it and trying it on new
| datasets.
|
| The other big question is why you want to learn it. If you want
| to learn ML in itself, than anything including the search
| algorithms (which used to be considered core to ML a long time
| ago) you mentioned is part of that. But if you want to learn ML
| to contribute to modern developments like LLMs, then search
| algorithms are virtually useless. If you aren't going to be
| engineering any ML or ML products, what you want is to gain some
| insight into it's future and the business of it. So learning
| things like transformer architecture is going to be far more
| unhelpful than say, reading about the economics of compute
| clusters.
|
| Given the empirical/engineering quality of current ML, I'd say
| building it from scratch is really good for getting the handful
| of possible first principles (the fundamental functions involved,
| data cleaning, training, etc)
| kingkongjaffa wrote:
| > Usage of the phrase "first principles" has sprawled into many
| different things since (I think) Musk first mentioned it as a
| way to learn
|
| In pop culture in 2010+ sure, but he was essentially parroting
| Feynman IIRC.
|
| "How to learn AI from first principles?"
|
| Start with
| https://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_t...
| and eventually you'll get to AI, exercise left to the reader ;)
| HardikVala wrote:
| Ya, the phrase "first principles" is vague...I meant starting
| from an axiomatic and actionable definition of AI and learning
| from there. The first chapter of AIMA does a swell job of
| enumerating different definitions of and then explicitly
| declaring which one is used and the foundational premises for
| the concepts and methods to follow. And it doesn't define AI
| then jump to neural networks, it gradually layers more atomic
| concepts, like agents (which I know, have been bastardized) and
| environments, until it gets to machine learning.
|
| > The other big question is why you want to learn it.
|
| Good question. I'm just looking for a wider context to
| understand contemporary AI. I don't know if this serves any
| practical purpose but I'm someone who likes to understand the
| "why" behind everything and starting from "first principles"
| helps uncover that.
| noduerme wrote:
| The following is _not_ a take that will get you a job or teach
| you precisely how LLMs work, because you can look that up
| yourself. However, it may inspire you and you may create
| something that has a better-than-lottery-ticket chance of being
| an improvement over the AI status quo:
|
| Without reading about how it's done now, just think about how you
| _think_ a neural network should function. It ostensibly has
| input, output, and something in the middle. Maybe its input is a
| 64x64 pixel handwritten character, and its output is a unicode
| number. In between the input pixels (a 64x64 array) and the
| output, are a bunch of neurons. Layers of neurons. That talk to
| each other and learn or un-learn (are rewarded or punished).
|
| Build that. Build a cube where one side is a pixel grid and the
| other side delivers a number. Decide how the neurons influence
| each other and how they train their weights to deliver the result
| at the other end. However you think it should go. Just raw code
| it with arrays in whatever dimensions you want and make it work;
| you can do it in Javascript or BASIC. link them however you want.
| Don't worry about performance, because you can assume that
| whatever _marginally_ works can be tested on a massive scale and
| show "impressive" results.
| HardikVala wrote:
| Interesting idea. I like it.
| Kandel72 wrote:
| Try chat gpt
| hnarayanan wrote:
| This article takes a journey from first principles:
| https://harishnarayanan.org/writing/artistic-style-transfer/
| __alexander wrote:
| I'm starting down a similar path with these two books
|
| How AI Works - https://nostarch.com/how-ai-works
|
| +
|
| Why Machines Learn: The Elegant Math Behind Modern AI -
| https://www.penguinrandomhouse.com/books/677608/why-machines...
___________________________________________________________________
(page generated 2025-01-26 23:02 UTC)