[HN Gopher] From Deep Learning Foundations to Stable Diffusion
___________________________________________________________________
From Deep Learning Foundations to Stable Diffusion
Author : wasimlorgat
Score : 223 points
Date : 2022-09-16 11:42 UTC (11 hours ago)
(HTM) web link (www.fast.ai)
(TXT) w3m dump (www.fast.ai)
| bilsbie wrote:
| Is there any way to motivate myself to take a class like this?
|
| I keep turning to easy distractions like twitter and Pac-Man.
| thisismyswamp wrote:
| There's no way to trick your brain into enjoying something it
| finds boring.
|
| You can only switch to something that it doesn't find boring,
| even if the final result is the same.
| dagmx wrote:
| When I teach people, the most effective thing I find is to give
| them small usable projects at the end of each little milestone.
|
| That helps them have something they can use right away, which
| helps with a few things:
|
| - at each milestone, they have a distinct goal that's reachable
|
| - they can understand concrete use cases immediately
|
| - they get a little dopamine hit if satisfaction of having
| completed something.
|
| Whenever you're doing a course that doesn't structure itself
| that way, it's good to try and break down the components and
| set yourself little tasks as you go.
| bilsbie wrote:
| Thanks. That's an idea.
| O__________O wrote:
| >> The course will be available for free online from early 2023.
|
| Anyone aware of an in-depth intro-level text-based explanation of
| Stable Diffusion that covers the whole pipeline, including
| training on an extremely limited dataset?
|
| Here's an example, but open to suggestions too:
|
| https://huggingface.co/blog/stable_diffusion
| mpaepper wrote:
| I wrote some of that as a blog entry:
| https://www.paepper.com/blog/posts/how-and-why-stable-diffus...
| nyoomboom wrote:
| This year's Deep Learning Indaba had a tutorial on diffusion
| models in Jax: https://github.com/deep-learning-indaba/indaba-
| pracs-2022/tr...
| ralusek wrote:
| > in depth course that started right from the foundations--
| implementing and GPU-optimising matrix multiplications and
| initialisations--and covered from scratch implementations of all
| the key applications of the fastai library.
|
| I haven't taken the course, but that sounds like a horrible place
| to start a course on understanding deep learning. GPU matrix
| operations are literally an implementation detail.
|
| I think the proper way to teach deep learning "from scratch"
| would be:
|
| 1.) show simple example of regression using high level library
|
| 2.) implement same regression by writing a simple neutral network
| from scratch (explain going through, multiplying weights, adding
| biases, applying activation function, calculating loss, back
| propagation).
|
| 3. use NN on more complicated problem with more parameters and a
| larger training set, so that user sees they've hit a wall in
| performance, and now implementation needs to be optimized
|
| 4. at _this_ point, say okay, our implementation of looping and
| multiplying can be done much faster with matrix multiplication on
| GPU, and even faster parallelized across GPUs on a network. If
| you 're interested in that, here is an _optional_ fork in the
| course that gets into specifics. Anything after this point will
| assume that implementation of NN calls will be using these
| techniques under the hood
|
| 5. move onto classification, q learning, GANs, transformers.
|
| 95% should have skipped step 4 and only revisited if they become
| interested in this for a specific reason. To _start_ with it is
| crazy. It 's like starting a course about flying by explaining
| how certain composites allowed us to transition from propellers
| to jets, and let's dive into how those composites are made.
| boloust wrote:
| The course basically goes through the exact steps you
| described. The main "GPU optimisation" is to basically say
| "okay, now that we know how to implement matrix multiplication,
| let's use the optimised pytorch implementation instead".
| idf00 wrote:
| > I haven't taken the course, but that sounds like a horrible
| place to start a course on understanding deep learning.
|
| Sounds like you support the course author's decision to make
| this part 2 of the course series to be taken after part 1 is
| completed!
| ralusek wrote:
| Sorry for failing to clarify:
|
| > Three years ago we pioneered Deep Learning from the
| Foundations, an in depth course that started right from the
| foundations--implementing and GPU-optimising matrix
| multiplications and initialisations
|
| They're talking about how Part 1 starts
| Yenrabbit wrote:
| 'Deep learning from the Foundations' was the previous
| version of this new course, but both built on their
| respective part 1s. So part 1 is 'practical deep learning
| for coders', very good intro to DL for getting things done
| that starts with big wins using pretrained models and then
| digs into them enough that you know how to make them
| better. Part 2 is for those wanting to go all the way to
| implementing the ideas 'from scratch', reading papers and
| diving into advanced topics.
| idf00 wrote:
| Deep Learning from the Foundations IS part 2. They are not
| talking about how part 1 starts. They are talking about how
| part 2 starts. This mirrors how their book is written, with
| the content typically in Deep Learning for Foundations
| being the second half of the book. I've taken the courses
| every year, it's always been like that.
|
| You should consider taking a look at part 1 of the course
| if you want to understand the starting curriculum:
| https://course.fast.ai/
| hansworst wrote:
| This is exactly the kind of course I've wanted to do for some
| time now. Even before stable diffusion it felt like other media
| synthesis applications like StyleGAN were what I wanted to learn,
| but most machine learning courses focus on more traditional data
| science topics.
|
| Of course you can start with a more traditional course and then
| learn something like stable diffusion afterwards, but as a newbie
| it's quite hard to figure out where to even start. A full-fledged
| course that takes you exactly where you want to go is a lot
| easier and I think it can help learners to stay motivated because
| they have a clear goal in mind. If I want to learn how to create
| cool images, I want to spend as little time as possible
| predicting housing prices in the Bay Area.
| elephanlemon wrote:
| It's a bit dated now, but the DeepLearning.ai GANs
| specialization covers topics through StyleGAN. If you have no
| experience with ML at all I would probably start with the first
| two courses of their Deep Learning specialization and then jump
| into the GANs specialization.
|
| I would also highly recommend FastAI's Deep Learning for Coders
| (and their new course that came out this year). You'll start
| immediately with some cool applications (basic image
| recognition and NLP) and then drill down from there to learn
| how they work in detail.
|
| It's set up such that you can learn as much as you want (basics
| with no depth: first chapter; basic understanding of how a
| neural network is trained with SGD: first four chapters;
| understanding of decision trees, LSTMs, and CNNs: first half;
| detailed understanding of how to build everything from scratch:
| whole book).
| mudrockbestgirl wrote:
| > If I want to learn how to create cool images, I want to spend
| as little time as possible predicting housing prices in the Bay
| Area.
|
| I think that's somewhat of a dangerous mindset to have. If you
| want to create cool images you can use pre-trained models and
| high-level APIs without needing to understand any of the
| internals.
|
| But if you want to truly _understand_ how these models work,
| you need to make effort to study the basics. Maybe not
| predicting housing prices, but learn the foundational math and
| primitives behind all of the components from the ground up (and
| the Diffusion models are a complex beast made up of many
| components). And getting an intuitive understanding of how
| models behave when you tune certain knobs takes much longer.
| Many researchers in the field have spent years developing their
| intuition of what works and what doesn 't.
|
| Both of these are fine, but I think I think we should stop
| encouraging people to be in the middle. Have courses that that
| promise "Learn Deep Learning / Transformers / Diffusion models
| in 7 days!" but then go on and teach you how to call blackbox
| APIs, giving you an illusion of knowledge and understanding
| where there is none. I don't know if this applies to this
| specific course, but there are a bunch of those out there, and
| highly recommend staying away from those. I know it's a hard
| sell in this modern instant gratification age, but if you
| actually want to understand something you need to put in some
| possibly hard work.
| jan_Inkepa wrote:
| > I don't know if this applies to this specific course, but
| there are a bunch of those out there, and highly recommend
| staying away from those.
|
| fast.ai do stuff pretty well. FWIW, I did one of their
| earlier free courses and, as a maths grad, got my fill of
| maths learning as well as my fill of practical 'doing stuff
| with ML' stuff. If I didn't have my plate full I'd probably
| pay the 500 quid or whatever to do this course now rather
| than wait for the free version.
| thingification wrote:
| https://nitter.namazso.eu/jeremyphoward/status/1568843940690.
| ..
|
| > fast.ai
|
| > Do that and your life will change
|
| Sounds like Emad Mostaque of Stability AI / stable diffusion
| thinks this course probably won't fall into "do this, no
| understanding needed" trap (I'm not contradicting anything
| you said here).
| throwaway287391 wrote:
| AFAIK Emad Mostaque is not (yet) an AI expert at all, he's
| a rich guy (former hedge fund manager) building a business
| that provides the funding for AI experts to do their thing.
| Stable diffusion itself was built by a team of academics
| [1], Emad is not a coauthor. Not to take away anything from
| what he's accomplished -- it's quite incredible -- but it
| doesn't mean he knows how to (learn to) build AI systems or
| do AI research himself.
|
| [1] https://github.com/CompVis/stable-diffusion
| emadm wrote:
| I am not a coauthor but do know a decent amount about AI
| systems from a maths and computer science degree from
| Oxford, couple decades coding and being lead architect on
| https://hai.stanford.edu/watch-caiac amongst other stuff
| :)
|
| Originally took a break from being a hedge fund manager
| to build AI lit review systems to investigate ASD
| etiology for my son along with neurotransmitter pathway
| analysis to repurpose medication (with medical oversight)
| to help ameliorate his more severe symptoms.
|
| Had 100% satisfaction from programmers with some math
| knowledge trying fast.ai and members of team active
| there, really nice take off point into a massive sector.
|
| It digs nicely into the principles and is not a surface
| level course. The stable diffusion one will need some
| good work to get through.
|
| But yeah my job now is to get billions of dollars into
| open source AI to make the world happier, happy to do my
| best and let the smart and diligent folk buidl.
| Royi wrote:
| That's interesting. How far did you get utilizing AI for
| treating ASD?
|
| Do you know other efforts in that direction?
| throwaway287391 wrote:
| Thanks for the response! Sorry for underestimating your
| background based on what little I had read/heard about
| you. I appreciate and respect what you're doing.
| boredemployee wrote:
| >> But if you want to truly understand how these models work,
| you need to make effort to study the basics.
|
| I was very confused by this in the beginning of my journey. I
| was trying to learn everything involved with ML/DL, but in
| the end everything is already implemented with APIs, and your
| boss doesnt care if you know how to implement a MLP from
| scratch or if you use Tensorflow.
|
| My (poor) analogy is: you don't need to know how a car works
| (or how to build one) in every detail to drive it. When I
| understood it, it was liberating.
| mudrockbestgirl wrote:
| I agree, and I think that's the first use case I mentioned
| above. You can use Deep Learning libraries and pre-made
| models without a deep understanding of anything and you get
| some nice results. And that's great.
|
| What's not so great is the huge number of people believing
| to understand something when they don't, i.e. the illusion
| of knowledge they're getting from some of these marketing-
| driven courses and MOOCs. I see that in job applications.
| Every resume has "Deep Learning, PyTorch, Tensorflow" on it
| now, but if you ask them _why_ something works (or why a
| variation may not work) these candidates have no idea. And
| for some jobs that 's totally fine, but for other jobs it's
| not. And the problem is when you can't tell the difference.
|
| It's kind of like putting "compilers" on your resume
| because you've managed to run gcc.
| neodypsis wrote:
| > but if you ask them why something works (or why a
| variation may not work)
|
| Interesting. Do you have an example? Is it common for
| people to practice ML problem-solving a la LeetCode
| nowadays?
| mudrockbestgirl wrote:
| It's the opposite of leetcode because it tests
| understanding, not memorization. For example, you could
| ask why position embeddings are necessary and what would
| happen without, reasoning behind certain terms in an
| objective function and what would happen without them,
| which part of an architecture the bottleneck for
| convergence is, intuitively what tuning a certain
| hyperparameter does, show them a timeseries of gradients
| for something that doesn't convergence and ask what's
| wrong with them, etc.
|
| I'm just making these up because the questions we
| previously asked were domain-specific to our
| applications, e.g. "why is this specific learning
| objective hard" or "what would you modify to help
| generalization in case X"
|
| These questions are _very_ easy to talk about for someone
| with a strong ML background. They may not always know the
| answer and often there is no right answer, but they can
| make reasonable guesses and have a thought process around
| it. Someone who just took a MOOC likely has no idea how
| to even approach the question.
| jph00 wrote:
| Sorry folks for not contributing to this thread earlier - didn't
| realise this popped up on HN while I was sleeping! I'm Jeremy,
| and I'll be running this course. Lemme know if you've got any
| questions about it, or anything related to the topic of DL,
| stable diffusion, etc.
| frozencell wrote:
| Thank you so much for doing this! Does the course mention
| sampling methods (Euler, LMS, DDIMS, etc.), what are they, how
| do you they work and relate?
| neodypsis wrote:
| This course looks interesting. My only concern is that I don't
| have real experience with NLP. Anybody can recommend resources to
| get to speed on this pre-requisite? My NLP knowledge is very
| basic.
| Version467 wrote:
| The "Practical Deep learning" course from fast.ai has a section
| on NLP that's probably a good starting point.
| neodypsis wrote:
| Thanks, I'll check it out.
___________________________________________________________________
(page generated 2022-09-16 23:01 UTC)