[HN Gopher] Reflected Diffusion Models
___________________________________________________________________
Reflected Diffusion Models
Author : tim_sw
Score : 125 points
Date : 2023-05-08 16:13 UTC (6 hours ago)
(HTM) web link (aaronlou.com)
(TXT) w3m dump (aaronlou.com)
| aantix wrote:
| We don't tolerate poorly named variable and method names in
| software development - why is it tolerated in mathematical
| notation?
| crest wrote:
| A lot of times mathematical notation use domain specific naming
| conventions for they variables and each the tradition is a
| leftover from a time when you had to write the whole line (by
| hand) for every step you wrote down and only a few name
| variables. Take a look code at the other extreme can you
| imagine writing down verbose Java code line by line using quill
| and ink? I don't even want to read such a low entropy highly
| redundant "notation" on a 34" 8k screen. The only way to
| navigate it is through load bearing IDEs.
| niemandhier wrote:
| Yes! When writing down the equations you use a form that
| docks to the right points in the mental model space of your
| audience.
|
| In the end the symbols on the paper are just a way to guide
| the reader to his own understanding of the underlying
| mathematical truth.
|
| I bet most of us do not even think about that. Rho is a
| density, n is a natural number, epsilon is small, delta a
| perturbation ...
| _dain_ wrote:
| _> Take a look code at the other extreme can you imagine
| writing down verbose Java code line by line using quill and
| ink?_
|
| Not Java, but I have done productive brainstorming sessions
| where I handwrite Python or R onto paper, to figure out some
| kind of tricky problem or algorithm. Syntactically correct
| and indented too. It's not hard to handwrite code at all.
| Plus my "comments" can have arrows and diagrams and doodles
| and anything I like.
| simion314 wrote:
| You should try it with long names and without and let's see
| what is easy to understand.
|
| e^pi _i = -1
|
| or
|
| theNatural....Constant... = theCircleConstant... _
| theSqureRootOfMinusOne = minusOn
|
| or maybe you replace One with
| theNeutralElementForMultiplication
|
| but maybe someone is too lazy and you need to give a good clear
| name for what multiplication is,
|
| Sorry, but I could not help it, I seen this kind of comments
| multiple time, some people want to jump to the middle or end of
| a book/course and skip the hard work.
| Freebytes wrote:
| What you demonstrated as the "bad example" is closer to what
| mathematicians have been doing recently, but they would use
| natLog, ratCir, and im as variable names instead inside of a
| function called gnl() and offer no comments as to what any of
| it means.
|
| From what I have seen, they should likely be putting their
| code into ChatGPT and asking it to clean it up because it
| does a better job of writing functions with properly named
| variables.
| qumpis wrote:
| This is other side of extreme. And these constants are widely
| accepted notation.
| simion314 wrote:
| >This is other side of extreme. And these constants are
| widely accepted notation.
|
| Some dude will say that "i" is also used as the vector for
| the horizontal axis, i is sometimes used in induction as
| iterator too, so clearly it should be refactored to
| sqrtOfMinusOne. Also + and * are overloaded in math, so
| maybe they should be replaced with a clear name like
| "realNumberAdition"
| thewataccount wrote:
| A lot of the times equations are meant to be general and can
| apply to a vast number of scenerios.
|
| For example x (x-bar) is used in statistics to represent the
| sample mean of a distribution. It doesn't matter what your
| distribution is, you'd likely call it x.
|
| Whenever you see an equation mentioning "n", this will very
| often be the number of items in a set/list/array, not matter
| what that set is counting.
|
| Usually these equations are very general, and they don't care
| what "n" actually is, and this way it's easy to see where you
| should plug that number into. If you look at the equation for x
| (averaging) then you'll see "n" in there, and you already know
| what it's for, and it's incredibly concise.
|
| If you're familiar with them they actually are named relatively
| well, and generally they're references that you then implement,
| they aren't meant to be the implementation themselves.
| [deleted]
| 6gvONxR4sf7o wrote:
| You often do math on the whiteboard or on paper, and it
| involves repeated references to the same things, which might
| take forever to write out at each step. Additionally, since
| it's a 2d notation, the "shape" of the formulas is informative,
| which is an aspect you lose with bigger names.
| jonahx wrote:
| Obviously, you're not a golfer...
|
| This topic is complex. For a more thoughtful take, start with:
|
| https://www.jsoftware.com/papers/tot.htm
| dheera wrote:
| It has always been tolerated in math. The more unpronouncable
| the Greek letter, the better. If you want to sound really
| badass, you start using Hebrew letters as well. The pinnacle of
| badassness is multiple capital Pi and capital Sigma with some
| set notation below instead of indexes laid out clearly like a
| for loop. Or "Einstein notation" where you stop writing the
| summation symbol because a small group of people think it is
| implied, which helps gatekeep others from the field of research
| because they can't make sense of your equations anymore. At
| that point, people give up on reviewing your paper and just
| publish the damn thing because you're from Google, Stanford, or
| wherever.
|
| If mathematicians worked like software engineers they'd label
| things like "density[x, y]" instead of "\rho_{xy}" or at least
| stop using PDFs and use HTML with a tooltip over each variable
| name like we have in pretty much every modern editor, but we
| all know they hate readability.
| [deleted]
| Kuinox wrote:
| Historically paper was costly, and there is no autocomplete on
| paper too.
| Der_Einzige wrote:
| One missing feature that could be implemented in places like the
| Automatic1111 webUI, would be a "color threshold" or even a
| "color loss" function. I should be able to specify that for a
| particular region of the image, that a particular set of colors
| should not be generated, or I should be able to bias or prefer
| certain colors on certain parts of the image.
|
| I actually wrote a jupyter notebook based off of another notebook
| that showed that color loss was possible. My jupyter notebook
| extended the idea by allowing to specify the region and the
| particular color that the color loss would operate on.
|
| I'm surprised that despite how much tooling currently exists,
| tooling for specifying colors is not really common or well known.
| I figured that controlnet would have something for this by now,
| but here we are...
| ttul wrote:
| Reflected Diffusion Models are a new class of diffusion models
| that respect general boundary constraints and correctly train for
| thresholded sampling. They bridge the gap between theoretical
| frameworks and practical implementation details, leading to
| improved performance and image quality in generative modeling
| tasks.
|
| Diffusion models are generative models that work by perturbing
| data using a stochastic process, then reversing this process to
| generate new data. However, when discretized into discrete steps
| (as in most practical implementations), the sampling process can
| negatively affect data quality. To alleviate this issue, a
| technique called thresholding is employed, resulting in better
| samples but breaking the theoretical framework.
|
| Reflected Diffusion Models use a reflected stochastic
| differential equation (rSDE) to correctly model the thresholding
| process. This allows the models to train appropriately while
| respecting boundary constraints, like images with pixel values in
| the [0, 255] range.
|
| The benefits of Reflected Diffusion Models include improved
| perceptual quality, correct handling of boundary constraints,
| better image generation with guidance, and general applicability
| to different shapes of data domains. This has the potential to
| expand the applications of diffusion models in fields such as
| image, language, and molecule generation.
| devxpy wrote:
| Is there a way to run this locally?
| toxik wrote:
| I hate to ask but is this ChatGPT?
| dr_dshiv wrote:
| Only if you don't find it useful
| refulgentis wrote:
| It is ChatGPT and it's not useful. It purposefully hides
| while it attempts to illuminate.
|
| It masquerades as summarizing the paper, but it's a random
| grab of whatever OP decided to jam into the chatbox that
| fits in the context window, (if you check, its mostly
| regurgitating the abstract).
|
| However, it presents itself as a summary as if the entire
| paper was read.
|
| I'm _really_ into AI stuff but the flippant nature its used
| at on HN has me scared. People here are generally well-
| educated, considered in their decisions, and are well
| acquainted with tech. If HN can't avoid glaring issues with
| it, who can?
|
| The top example that came to mind was someone laboriously
| sharing a really nice data-driven comment, someone asks
| source? they say ChatGPT, and acted like we were trolling
| for saying it hallucinates.
| smrtinsert wrote:
| > People here are generally well-educated This isn't a
| given (anymore?).
| visarga wrote:
| chatGPT "of the gaps", it is chatGPT only when it is not
| useful
| ttul wrote:
| As is so often the case in neural network design, preventing
| numbers from blowing up is the cause of so many necessary, but
| poorly understood math hacks. It seems these researchers have
| found a potentially better way to control the sampling process
| so that diffusion models can generate outputs that are more
| correct.
|
| I often wonder if Midjourney, behind the scenes, has already
| discovered something like this in their v5 release, which is
| substantially better at modelling things like hands.
| nmca wrote:
| This was a great investigation of a trick one might not naively
| expect to be important & a theoretically justified fix. Nice
| work!
| toxik wrote:
| This text was your classic "I'm writing a tutorial to see how
| much _I_ know", being only intelligible to the exact set of
| people who already know what it's about. A lot of important ideas
| glossed over, eg "we don't know log pt" then immediately using it
| in the loss function.
|
| I probably sound very negative but I feel that you have some
| responsibility to your readers if you publish your text.
| dimatura wrote:
| I wouldn't be so harsh. The author of the blog post is also the
| author of the paper. If I had to guess, the author decided (or
| was advised) that writing a blog post about the paper would
| increase its visibility (which turned out to be true,
| considering it's on HN). So they wrote a blog post summarizing
| the paper. But because they are really deep in the weeds with
| the technical content of the paper, might not have done a great
| job at making it more accessible to a wider audience. Doing
| that well takes some skill, and is not usually something
| graduate students get much, if any, training on. (Edit: on
| reading the blog post, I do have to agree that the presentation
| is pretty abstruse to the point that the paper is a better
| source. And yes, I am a "Phd level AI researcher").
| lbotos wrote:
| Are you a PHD level AI researcher? If not, I don't think you
| are the intended audience.
|
| I'm not, so as a reader here, I'm just happy that I didn't have
| to pay to get exposed to new challenging ideas. A lot of other
| science is sadly paywalled inside of journals.
|
| Give it a few weeks for the fast followers to dumb it down for
| us.
| toxik wrote:
| I am, yes. Most of our field's research is publicly available
| on arxiv by the way.
| yorwba wrote:
| Indeed, the paper associated with the blog post is also on
| arXiv. But how does that relate to your complaint?
| mrtranscendence wrote:
| toxik was responding to the apparent position that it's
| OK for the material to be presented in a confusing way,
| because the research has been provided freely rather than
| locked behind a paywall. It's apropos to point out that
| this isn't a great excuse.
| egocodedinsol wrote:
| they didn't submit it here - I'm not sure they bear a
| responsibility to make sure their work is accessible to HN or
| any other target audience. If you don't find an otherwise
| respectful and substantive blog post useful, perhaps best to
| leave it be.
| [deleted]
| subroutine wrote:
| This is a side note / random observation, but it seems like "An
| astronaut riding a horse" has become the Lorem ipsum / Lenna /
| Utah teapot of AI text-to-image generation.
|
| https://en.wikipedia.org/wiki/Lenna
|
| https://en.wikipedia.org/wiki/Utah_teapot
___________________________________________________________________
(page generated 2023-05-08 23:00 UTC)