[HN Gopher] Probabilistic Artificial Intelligence
___________________________________________________________________
Probabilistic Artificial Intelligence
Author : pavanto
Score : 253 points
Date : 2025-03-10 09:50 UTC (13 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| jacob019 wrote:
| This is great. Is it available as a printed book?
| falcor84 wrote:
| From a brief search I see that it isn't (or it least not yet),
| but seeing how well-formatted the pdf is, and the fact that
| it's CC-licensed, you could print it yourself, or perhaps talk
| with them to organize a batch.
|
| Though I personally prefer to read these sorts of books
| directly from pdf, and am grateful to them for sharing it on
| arxiv.
| mnky9800n wrote:
| I wonder if one could organize an arXiv print service that
| binds and prints and ships with a unique cover and such.
|
| Also it should use LLMs and the blockchain.
|
| But this would be nice there are a number of papers and such
| that if you could submit an arXiv link to a print service I
| would probably buy a copy. I wonder why no one does it.
| woolion wrote:
| Aren't you describing Lulu but for the very niche case of
| arxiv publications that are small books but not published
| as books? I think you could do it in a weekend with their
| API.
| mnky9800n wrote:
| yes thats what i thought after the post. haha.
| ivan_ah wrote:
| If anyone is interested in trying this, here is some
| Python starter code you might find useful:
| https://github.com/minireference/lulu-api-
| client?tab=readme-...
|
| This worked four years ago when the API was still
| launched, but there might have been changes since, so no
| guarantees.
|
| Most ArXiv PDFs are probably lulu-printable out of the
| box, but to make a general solution, one would probably
| need to do some pre-processing with ghostscript (gs),
| e.g. embed all fonts and flatten images (no
| transparency).
| madcaptenor wrote:
| I wonder if they're aiming for it to be a book. Hubotter
| describes it on his web page as "notes on Probabilistic AI".
| au8er wrote:
| The "book" is accompanying studying material for the course
| Probabilistic AI at ETH Zurich. Essentially each chapter is
| the material covered in one lecture (3hrs).
|
| Source: I did the course
| madcaptenor wrote:
| I think this is where a lot of textbooks come from.
| esafak wrote:
| I do not think so. I am asking the author for confirmation.
| adrc wrote:
| There's no printed version. Btw I took this course at ETHZ
| last year (a course with this title and whose script is this
| document. Pretty nice course and pretty nice course notes,
| happy to see that the authors decided to share it outside of
| the course website now!
| abhgh wrote:
| I came across this a few days ago, and my excuse to give it a a
| serious look is that Andreas Krause has some deep and interesting
| research in Gaussian Processes and Bandits [1].
|
| [1]
| https://scholar.google.com/scholar?start=10&q=andreas+krause...
| trostaft wrote:
| It's Krause, he's one of the biggest researchers in the field.
| At least based on the other work of his I've read, he's a good
| writer too. This ought to be a worth while read.
| brador wrote:
| Interesting separation and distinction between noisy inputs,
| noisy processing and noisy chains.
| thisisauserid wrote:
| Gemini 2.0 Experimental 02-05 sees this as "only" 107K tokens.
|
| Handy if you want help breaking this down.
|
| https://aistudio.google.com
|
| 'Laplace Approximation is a "quick and dirty" way to turn a
| complex probability distribution into a simple Gaussian (bell
| curve).
|
| It works by finding the highest point (mode) and matching the
| curvature at that point.
|
| It's fast and easy, but it can be very inaccurate and
| overconfident if the true distribution doesn't look like a bell
| curve.'
| dcreater wrote:
| As a layman in this field, I have no idea the contact it
| significance of this work. Can someone better informed inform us?
| cubefox wrote:
| Apparently they don't discuss language models at all.
| cubefox wrote:
| Which is a major omission, as transformer-based language models
| are the most powerful available form of "probabilistic
| artificial intelligence". They predict a probability
| distribution over a token given a sequence of previous tokens.
|
| My guess is that most of the content in the book is several
| years old (it's apparently based on an ETH Zurich class),
| despite the PDF being compiled this year, which would explain
| why it doesn't cover the state of the art.
| antonkar wrote:
| I think we'll need a GUI for the models to democratize
| interpretability and let even gamers explore them. Basically to
| train another model, that will take the LLM and convert it into
| 3D shapes and put them in some 3D world that is understandable
| for humans.
|
| Simpler example: represent an LLM as a green field with objects,
| where humans are the only agents:
|
| You stand near a monkey, see chewing mouth nearby, go there (your
| prompt now is "monkey chews"), close by you see an arrow pointing
| at a banana, father away an arrow points at an apple, very far
| away at the horizon an arrow points at a tire (monkeys rarely
| chew tires).
|
| So things close by are more likely tokens, things far away are
| less likely, you see all of them at once (maybe you're on top of
| a hill to see farther). This way we can make a form of static
| place AI, where humans are the only agents
| ikeashark wrote:
| What
| antonkar wrote:
| What what?)
| bongodongobob wrote:
| What
| antonkar wrote:
| Another related shocking idea:
| https://news.ycombinator.com/item?id=43319726
| soulofmischief wrote:
| I had a mind-bending Salvia trip at eighteen that went sort of
| like that.
|
| My mind turned into an infinitely large department store where
| each aisle was a concurrent branch of thought, and the common
| ingredient lists above each aisle were populated with words,
| feelings and concepts related to each branch.
|
| The PA system replaced my internal monologue, which I no longer
| had, but instead I was hearing my thoughts externally as if
| they were another person's.
|
| I was able to walk through these aisles and marvel at the
| immense, fractal, interdependent web of concurrent thought my
| brain was producing in realtime.
| _rpxpx wrote:
| "When I began to navigate psychospace with LSD, I realized
| that before we were conscious, seemingly self-propelled human
| beings, many tapes and corridors had been created in our
| minds and reflexes which were not of our own making. These
| patterns and tapes laid down in our consciousness are walled
| off from each other. I see it as a vast labyrinth with high
| walls sealing off the many directives created by our personal
| history.
|
| Many of these directives are contradictory. The coexistence
| of these contradictory programs is what we call inner
| conflict. This conflict causes us to constantly check
| ourselves while we are caught in the opposition of polarity.
| Another metaphor would be like a computer with many programs
| running simultaneously. The more programs that are running,
| the slower the computer functions. This is a problem then.
| With all the programs running that are demanded of our
| consciousness in this modern world, we have problems finding
| deep integration.
|
| To complicate matters, the programs are reinforced by fear.
| Fear separates, love integrates. We find ourselves drawn to
| love and unity, but afraid to make the leap.
|
| What I found to be the genius of LSD is that it really gets
| you high, higher than the programs, higher than the walls
| that mask and blind one to the energy destroying presence of
| many contradictory but hidden programs. When LSD is used
| intentionally it enables you to see all the tracks laid down,
| to explore each one intensely. It also allows you to see the
| many parallel and redundant programs as well as the
| contradictory ones.
|
| It allows you to see the underlying unity of all opposites in
| the magic play of existence. This allows you to edit these
| programs and recreate superior programs that give you the
| insight to shake loose the restrictions and conflicts
| programmed into each one of us by our parents, our religion,
| our early education, and by society as a whole."
|
| ~ Nick Sand, 2001, Mind States conference, quoted in Casey
| Hardison's obituary
| wincy wrote:
| I feel like if all the things people believe and espouse
| about hallucinogens were true and not just the effect of
| permanently damaging your mind, with the illusion of
| wisdom, we'd be able to point at all the revolutionary
| scientific breakthroughs and discoveries made under the
| influence of hallucinogenic substances.
|
| However, everyone I've met who admits to having taken
| hallucinogens seems reduced in some way, rather than
| enhanced. Like the lights are on but someone else is home.
| antonkar wrote:
| There is the Qualia Research Institute, one of the things
| they do is using dope and making simulations of the
| experience. They basically found 2 main types of drugs:
|
| 1. Most "create more separate personalities" in you
|
| 2. One (the "toad poison") actually makes you feel like a
| giant place, the feeling is usually pleasant.
|
| So there is either "agentification" of you into more
| "agents" or "space-ification" into one giant place
| without any agency in it. I think we can make this static
| place AI and it'll be safe by definition because we're
| the only agents in it.
|
| P.S. I don't promote drugs
| bongodongobob wrote:
| It happens all the time. DNA double helix is a good
| example. You really think people are going to mention
| their drug use in white papers? I think not. Nothing to
| gain and everything to lose.
| devmor wrote:
| The record of a scientific discovery that is heavily
| criticized for plagiarism and falsehoods is probably not
| a good example, actually.
| bongodongobob wrote:
| Well here's one: tons of people do drugs. It's not even a
| question whether or not drugs have inspired discoveries.
| They obviously have.
| wat10000 wrote:
| Tons of people eat carrots. That doesn't imply carrots
| have inspired discoveries.
| soulofmischief wrote:
| That is a textbook false equivalence.
| LVL96 wrote:
| This was sort of my experience with LSD. It just broke
| me. I fell into a deep depression afterward, but the
| reason was only partly due to damaging my mind. The other
| part of it was that the LSD made me realize where my life
| was going, and how completely unfulfilled I'd end up
| being in 10-20 years. In that way, it helped me course-
| correct. I'm healthier, more honest with myself, and got
| back into college because of the experience.
|
| But it did damage my mind. I have mild to moderate
| anhedonia now. Weed hits me completely differently now
| (feels more like strong caffeine + brain fog instead of
| any pleasure). I lost my desire to write creatively.
| bongodongobob wrote:
| Unless you did a thumbprint, you're perfectly fine, no
| damage. Just get your shit together, that seems to be
| your takeaway. Sounds like it worked. Now you have to
| keep working on yourself rather than blaming a harmless
| drug for your problems.
| wizzwizz4 wrote:
| If a thumbprint _can_ do damage, why can 't a lower dose?
| What's the mechanism?
| bongodongobob wrote:
| It's similar to one aspirin vs 10000. Also, people do
| thumbprints and live normal lives after, I just threw
| that out as an extreme. LSD is very safe if you're doing
| normal doses. Most people don't take 10000 doses so I
| can't really speak to that.
| gruntbuggly wrote:
| A fair observation, but real assumptions about progress,
| what it means, and what is valuable
| _rpxpx wrote:
| There are lots of examples, you've just decided not to
| see them. Just for a start, here's an account by Ralph
| Abraham of the impact of psychedelics on mathematics.
| http://www.ralph-
| abraham.org/articles/MS%23124.Maps/maps2.pd...
| ljsprague wrote:
| I don't know if he credits the drug for the inspiration
| but the inventor of PCR was famously a user of LSD.
| antonkar wrote:
| We have our first Neo candidate)
|
| The guy who'll make the GUI for LLMs is the next
| Jobs/Gates/Musk and Nobel Prize Winner (I think it'll solve
| alignment by having millions of eyes on the internals of
| LLMs), because computers became popular only after the OS
| with a GUI appeared. I recently shared how one of its "apps"
| possibly can look:
| https://news.ycombinator.com/item?id=43319726
| neom wrote:
| If you feel like being a hippie, you can find the "rendering
| engine for reality" in here: Mandelbrot (1980) - The
| Mandelbrot Set and fractal geometry Julia (1918) - Memoire
| sur l'iteration des fonctions rationelles (Julia sets) Meyer
| (1996) - Quantum Cellular Automata (procedural complexity)
| Wolfram (1984) - Cellular automata as models of complexity
| Bak et al. (1987) - Self-organized criticality. Wolfram,
| Gorard & Crowley (2020) - "A Class of Models with the
| Potential to Represent Fundamental Physics" - Kari & Culik
| (2009) - "Universal Pattern Generation by Cellular Automata".
| Just combine the papers, ofc, that is crazy - but its fun to
| be a bit crazy sometimes. It's one of my fav thought
| experiments, just for fun. :)
| antonkar wrote:
| I used Wolfram's physics approach to "model the absolute
| future and possible ethics of it", I think it's direct
| democratic simulated multiverse, you can find the link in
| my profile, if interested
| neom wrote:
| no need to read it, no disagreement here.
| jgord wrote:
| I dont think anyone has found a good way to map higher
| dimensional space onto 4D visualizations, yet.
|
| Maybe this is why tokens and language are so useful for humans
| ? they might be the closest analog we have.
| antonkar wrote:
| Good point, I think at least some lossy "compression" into a
| GUI is possible. The guy who'll make the GUI for LLMs is the
| next Jobs/Gates/Musk and Nobel Prize Winner (I think it'll
| solve alignment by having millions of eyes on the internals
| of LLMs), because computers became popular only after the OS
| with a GUI appeared. I recently shared how one of its "apps"
| possibly can look:
| https://news.ycombinator.com/item?id=43319726
| meindnoch wrote:
| Sir, this is a Wendy's.
| fud101 wrote:
| Books suck (imho). We need a new format to teach and learn this
| deep technical stuff. Not youtube, something interactive with
| exercises and engagement.
| jgord wrote:
| yeah, I mean 3Blue1Brown has done a great job .. and maybe
| those would be even better if you could app-ify them into
| something you can interact with.
|
| Current gen of LLM programming AIs might make it less leg-work
| to make these
| whimsicalism wrote:
| 3b1b is great but if you want to do deep technical work,
| you're eventually going to have to get comfortable with text
| as a medium
| thomasahle wrote:
| > something interactive with exercises and engagement
|
| Books have exercises. It's your job to engage.
|
| This book, in particular, has 3 pages of Problems per chapter.
| The only way to learn the math is to do all of them.
| nh23423fefe wrote:
| thanks. i was worried about job security for a nanosecond.
| jcgrillo wrote:
| It's a wild world where "reading the documentation" or
| "researching a topic" has become a career superpower. I'm
| glad my education largely predated social media and cell
| phones, and that I learned to read and work problems
| independently. OTOH it often makes work a very lonely, taxing
| experience. Being a human index into documentation is a hell
| of a lot less fulfilling than working with people who also
| can read.
| vessenes wrote:
| I urge you to rethink this perspective. All research shows that
| paper increases comprehension significantly over screens and
| even over eink. Additionally hand note taking again has a
| positive impact.
| mitthrowaway2 wrote:
| Well, there's also school.
| nbeleski wrote:
| Seems similar, or at least partially overlap, with what I would
| say is the best reference on the subject, an Introduction to
| Statistical Learning from Gareth James et al [1].
|
| I wonder it this one might be a bit more accessible, although I
| guess the R/Python examples are helpful on the latter.
|
| [1] https://www.statlearning.com/
| whimsicalism wrote:
| not really, islr is a pretty basic book - this is about more
| advanced techniques to propagate probability estimates rather
| than point-wise
|
| and frankly i would not recommend islr anymore today, too dated
| keviniam wrote:
| What would you (or other informed parties) recommend?
| whimsicalism wrote:
| it's been a while since I've been a beginner so I might not
| have the best resources, but I would recommend Harvard's
| Stat 110 with Joe Blitzstein (lectures online) and then
| Machine Learning by Kevin Murphy. might be a scarier book
| to someone not confident in their math, but overall a
| better one imo
|
| for something more directly comparable to the niche ISLR
| filled, Bishop's books are generally better - although I
| can't recall their title
| esafak wrote:
| https://www.bishopbook.com/ is the new one
| jgord wrote:
| The text has some great explanatory diagrams and looks to be a
| very high quality overview of ML thru the lens of probability,
| with lots of math.
|
| I was also recently impressed by Zhaos "Mathematical Foundation
| of Reinforcement Learning", free textbook and video lectures on
| YT : https://github.com/MathFoundationRL/Book-Mathematical-
| Founda...
|
| If you dont have a lot of time, at least glance at Zhaos overview
| contents diagram, its a good conceptual map of the whole field,
| imo .. here :
|
| https://github.com/MathFoundationRL/Book-Mathematical-Founda...
|
| and maybe watch the intro video.
| vimgrinder wrote:
| The first lecture is so good. Not only from perspective of
| content, but how Zhao explain things about how to think about
| learning as a student. ty for recommendation.
| chasely wrote:
| Kevin Murphy racing to rename his Probabilistic Machine Learning
| series.
| sunami-ai wrote:
| I found Gaussian Processes with the right kernel to be very
| powerful with even just a few data points and a very small set of
| parameters. I don't know if I was using it correctly tbh, but it
| worked out great in predicting values that I could not predict so
| accurately. I used it as a predictable yet non-linear process to
| tweak the input in a computer vision task. The proof was
| literally in the pudding.
| wbakst wrote:
| > be me
|
| > open article
|
| > "holy shit it's 400 pages"
|
| > realize i already have a grasp on most of the material from
| school
|
| > "phew"
|
| > oh this stuff is cool, just like i remember...
|
| > proceed to read all 400 pages
|
| well done! :clap:
| clarkedev wrote:
| Probabilistic Robotics book by Thrun and co. offers a great
| overview of most of these concepts.
| svilen_dobrev wrote:
| stupid question: can a LLM (i.e neural network) tell me the
| probability of the answer it just spew? i.e. turn into _fuzzy_
| logic? Aaand, can it tell me how much it does believe itself?
| i.e. what 's the probability that above probability is correct?
| i.e. confidence i.e. _intuitionisticaly fuzzy_ logic?
|
| Long time ago at uni we studied these things for a while.. and
| even made a Prolog interpreter having both F+IF (probability +
| confidence) coefficients for each and every term..
| vlovich123 wrote:
| Not out of the box I think; I wouldn't trust any self-assesment
| like that. With enough compute, you could probably come up with
| a metric by doing a beam search and using an LLM to evaluate
| how many of the resultant answers were effectively the same as
| a proxy for "confidence".
| bob1029 wrote:
| I'm not 100% sure what you mean by this, but there is token
| probability available in some providers:
|
| https://cookbook.openai.com/examples/using_logprobs
| whtrbt wrote:
| Maybe stupid answer, but I've read a few older papers that used
| ensembles to identify when a prediction is out of distribution.
| Not sure what SotA approach is though.
| esafak wrote:
| Suitably modified, they can. Bayesian neural networks provide
| _uncertainty quantification_. The challenge is _calibrating_
| the predictions, and deciding whether devoting model capacity
| to uncertainty quantification would not be better spent on a
| bigger, uncertain model.
|
| https://en.wikipedia.org/wiki/Calibration_(statistics)
|
| Example: _Efficient and Effective Uncertainty Quantification
| for LLMs_ (https://openreview.net/forum?id=QKRLH57ATT)
| ramity wrote:
| The way I understand it, an LLM response is a chain of tokens
| where each is the most probable token. Maybe there exists more
| complicated candidate and selection approaches than that, but
| biggest number works for me. For the sake of simplicity, let's
| just say tokens are words. You'd have access to the probability
| of each word in the ordering of the sentence, but I'm not sure
| how that would then be used to evaluate to the probability of
| the sentence itself or its truthiness.
| nthingtohide wrote:
| https://plato.stanford.edu/entries/self-locating-beliefs/
|
| Is this helpful?
| ramity wrote:
| I could have worded my reply better, but the simplified
| explanation stands :b
___________________________________________________________________
(page generated 2025-03-10 23:00 UTC)