[HN Gopher] Lessons from a year of AI research
___________________________________________________________________
Lessons from a year of AI research
Author : peanutcrisis
Score : 123 points
Date : 2021-11-28 11:20 UTC (3 days ago)
(HTM) web link (jetnew.notion.site)
(TXT) w3m dump (jetnew.notion.site)
| solmag wrote:
| How do you formally verify your neural networks and the like? Or
| is formal verification possible? Does this limit the areas where
| it can be applied?
| joleyj wrote:
| That site hijacks the CMD+LEFT and CMD+RIGHT Mac hotkeys for
| browser BACK/FORWARD and also has very strange behavior when
| scrolling with the keyboard. Why? Why? Why?
| dubya wrote:
| This seems really like a browser problem, and I'd love to know
| a fix. Geogebra.org hijacks CMD-` in Safari and it is
| absolutely infuriating.
| jetnew wrote:
| Hi, I'm the author of the post. I wrote it on Notion to seek
| feedback from a few friends, but seems like one friend shared
| the Notion link here :) Hope that my website would provide a
| better reading experience instead -
| https://jetnew.io/blog/2021/100-lessons/
| syats wrote:
| As well as up, down, pgup, pgdown, alt+left and alt+right on
| Chromium Linux. Stopped reading when I saw this enforcement of
| how they think their site should be experienced.
| criddell wrote:
| Do AI researchers have a commonly accepted definition for
| _intelligence_?
| solmag wrote:
| There is also no specification for "AI".
| chongli wrote:
| There isn't even an accepted definition of intelligence in
| _humans_ , let alone animals, never mind machines! We're
| pretty much at the "I know it when I see it" stage of
| definition.
| criddell wrote:
| How can that be true? If the author is doing AI research
| presumably they and others in the same field have have some
| foundational shared ideas that they build upon. If they
| didn't how would a person earning a PhD in this field ever be
| able to say if they are contributing something new to the
| study of AI? There has to be something that defines the work
| as work in AI rather than just computer science.
| solmag wrote:
| You have correctly deduced why AI research is just a paper
| factory.
| potatoman22 wrote:
| A large foundation for this field is statistical inference.
| To me, AI almost always means ML, and ML = algorithms that
| optimize themselves to make predictions.
| [deleted]
| bo1024 wrote:
| Not really, AI is more of a large collection of activities. I
| would distill it down to "problem-solving".
|
| The irony is that once you solve a problem, it's not a problem,
| so people's natural reaction is to only call things AI when
| they're unsolved. Once it's solved, "that's not AI".
| visarga wrote:
| Ability to solve novel problems with little experience. Skill
| acquisition efficiency.
|
| On the Measure of Intelligence, Francois Chollet
| https://arxiv.org/abs/1911.01547
| bo1024 wrote:
| That's not really compatible with the history of AI, where
| designing systems to solve a problem has traditionally been
| considered in scope. Seems that, like many people, Chollet
| wants to think of AI in terms of AGI (artificial general
| intelligence).
| YeGoblynQueenne wrote:
| Original post:
|
| https://jetnew.io/blog/2021/100-lessons/
| visarga wrote:
| The one that matters is missing: how do you get the best random
| seeds? /s
| oigursh wrote:
| # Sussman attains enlightenment
|
| In the days when Sussman was a novice, Minsky once came to him
| as he sat hacking at the PDP-6.
|
| "What are you doing?", asked Minsky. "I am training a randomly
| wired neural net to play Tic-Tac-Toe" Sussman replied.
|
| "Why is the net wired randomly?", asked Minsky. "I do not want
| it to have any preconceptions of how to play", Sussman said.
|
| Minsky then shut his eyes.
|
| "Why do you close your eyes?", Sussman asked his teacher. "So
| that the room will be empty."
|
| At that moment, Sussman was enlightened.
|
| http://www.catb.org/jargon/html/koans.html
| bruce343434 wrote:
| I'm not actually sure what the morale is here
| AstralStorm wrote:
| Intelligence does not start from a random state, but from a
| pretrained one. Random state is likely to roll an
| artificial overfit or if it just fits, you won't know why
| it is better than another state - or rather why your
| training method sucks from other initial states.
|
| Proper methods of bias analysis are network compression or
| random surgery, and expanding test data set. Proper methods
| of training are robust to varied initial states.
| ximeng wrote:
| If you shut your eyes, you do not see anything in the room.
| This does not mean there is nothing in the room.
|
| If you randomize your seeds, you do not deliberately bias
| your model. This does not mean your model is unbiased.
| jacquesm wrote:
| 4
|
| There you go.
| mellavora wrote:
| I have to disagree, and this close to thanksgiving I just
| don't see how you missed the best random seed
|
| pumpkin 3.14
|
| Normally I prefer apple, but during the holidays you must
| adjust.
| go_elmo wrote:
| Great one! Actual question: has anyone looked if pseudo
| randomnes is worse than full randomness?
| dotancohen wrote:
| It is pseudoworse.
| 317070 wrote:
| I once (8 years ago) did a comparison between random,
| pseudorandom and quasirandom. Quasirandom worked ever so
| slightly better than the other two in speed and final
| performance, but not enough to warrant the additional
| complexity of implementation.
| WithinReason wrote:
| 41. Remember to consider variance of results and the usage of
| random seeds.
| engineer_22 wrote:
| -> 87. Don't let yourself be too affected by the opportunity
| costs of doing research.
|
| Why not? Is there a strong payoff later? How did you learn this
| lesson?
|
| :)
|
| Good listicle!
| reliableturing wrote:
| Thanks for the write up! As someone starting a doctoral degree
| early next year, I greatly appreciate it
| mark_l_watson wrote:
| The author might find that this well thought out list will help
| when applying for work. I would suggest that they copy it over to
| GitHub in addition to their public projects.
| go_elmo wrote:
| Find peace in doing trial and error statistical black box studies
| after x years of formal systems studies. I couldnt do that for my
| part.
| hutrdvnj wrote:
| The title here and on the blog differs, I don't know why
| submiters pick other titles. I mean I wouldn't quote a newspaper
| or paper with a wrong title, would I?
| stonemetal12 wrote:
| It is one of the site's guidelines.
|
| >If the title contains a gratuitous number or number +
| adjective, we'd appreciate it if you'd crop it. E.g. translate
| "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to
| "Ys." Exception: when the number is meaningful, e.g. "The 5
| Platonic Solids."
| hutrdvnj wrote:
| Okay and why?
| yibers wrote:
| "Y"
| stonemetal12 wrote:
| PG considered them a special case of linkbait. Which is the
| only other time you are supposed to modify the title.
| hutrdvnj wrote:
| I think there are many click bait titles other than "10
| things ..." lists. Still don't understand why there's a
| special rule just for these.
|
| "You Can Now Save Money with Y New Strategy"
|
| "You Can Now Travel Abroad Without Having to..."
|
| "The Last ... You'll Ever Need"
|
| "You Won't Believe... What Y has Found"
|
| "Why You Should..."
|
| "Why You've Never Heard of This Top Travel Destination"
|
| "This is why you're losing money"
|
| Just a few examples.
| tubby12345 wrote:
| Lol this is so aspirational it could only come from an undergrad.
|
| Let me tell you that I've finally made it to the stressful part
| of the being a serious "AI" researcher, where I have a real
| project (as in difficult to achieve goals, not just "turn the
| crank" stuff) and real deadlines (deliverables on collaborators
| projects and my own conferences submissions) and the _only_ thing
| I prioritize above doing the work itself is keeping my advisor
| (and other collaborators) up to date on what I 'm doing so that
| when he reads my paper draft he's not completely lost. Everything
| like organizing papers, citations, logging infra, etc is
| meaningless when you're trying to piece together a solution. Like
| seriously somedays I barely have time to exercise and eat dinner
| with my wife (let alone organizing my bookmarks).
|
| For example I'm trying to solve a particular compilers problem
| using integer programming (note that at a high this isn't that
| high level because this is a small cottage industry) and so I
| have like 50 paper tabs open that I bounce between when
| thinking/experimenting. The way it usually goes is I'll hack, get
| stuck, go back to the papers, find something, hack, and on. And
| usually the eureka moment comes some hours later because I
| connect something.
|
| You might say that I'm a bad researcher but I know for a fact
| (external validation) that I'm not. And if you look at other
| highly productive researchers (like TT track profs at my "elite"
| school) this is indeed how they work. All of this zotero, notion,
| mlflow stuff is of the ilk of productivity porn for other flavors
| of knowledge workers (ie a mirage and/or snake oil). Let me put
| it this way: my advisor is a top 500 h-index person (the exact
| significance of that metric notwithstanding) and he doesn't have
| a bibtex of his own papers, let alone zotero for all of the
| papers he reads/comes across.
|
| The only thing that matters is code/math/etc output (whatever
| your material output is) and your abilities are also highly
| correlated with it with the casualty flowing in t opposite
| direction (make more stuff and you'll get better at making
| stuff).
|
| But I guess conversely do do some of these things when you're
| young and have the time (and I don't mean that condescendingly).
| E.g. reading outside of your area is probably the most valuable
| (from my own, admittedly a typical, experience, since I jumped
| domains many times); I very frequently can outpace even my senior
| colaborators very quickly on understanding a problem and solution
| simply because when I was younger I dabbled in ... all the things
| (physics, math, cs).
|
| The other thing that I'll say is there's something obviously
| missing from this list but only if you've really made it this
| far: collaborators and interactions with collaborators. The only
| thing that matters aside from the produce is getting people to
| make use of it. That means writing, speaking, and getting buyin
| from your collaborators. If you really truly want to be
| successful then work on your people skills as it pertains to this
| area - that means learn to speak the language of your research
| community, learn to give good (engaging, interesting, useful)
| presentations, learn to write well (including making nice
| diagrams), and learn to explain things in ways that smart but
| busy people will understand. Besides all of this being key to
| being productive it's also what feeds you (i.e. the real #1
| priority) since it gets you jobs, academic and industry.
| tinyhouse wrote:
| 101st lesson - 100 is a nice number but no one is going to read a
| blog post with 100 lessons learned. Either focus on the 5-10 most
| important lessons or consolidate and summarize.
| andreyk wrote:
| Good list! As a PhD student and therefore AI researcher for a few
| years now, a lot of this rings true. Though 100 lessons is too
| much and some of these are obvious/minor, i'd distill it down to
| the main ones.
|
| Here's my 2 cents on the topic from a thing I wrote last year
| ('Lessons Learned the Hard Way in Grad School (so far)'):
| https://www.andreykurenkov.com/writing/life/lessons-learned-...
| zwaps wrote:
| Love the timeline of failure and success (the latter of which
| is what one usually sees).
|
| I think there may be some who breeze through grad school,
| likely by being in a strong research environment beforehand, or
| by having lots of support. I mean, they have to exist?
|
| Have I genuinely met anyone like that? Nah. And, oh boy, can it
| be a struggle!
|
| During a PhD, it's very easy to put yourself into an
| increasingly hopeless situation that is mentally tough, for
| years on end, and it is really hard to even convey to anyone
| outside of academia why you would do that.
|
| It is especially hard to convey to people outside of academia
| how 5 (nowadays often 6,7, etc) years of work all come down to
| a few, often seemingly random decisions of hiring committees
| and other structures.
|
| You start grad school thinking it's something you build, bit by
| bit. At then end, it's more of an ever shifting collection of
| you, put together in the hopes it will be evaluated well. But
| then you realize, it really has little to do with objective
| quality, and much of the situation is outside of your control.
| There is no short project in grad school. It's the entire
| thing!
|
| I think the worst thing is the uncertainty about literally
| everything. Is your Professor ever telling the truth? Will you
| really get that support two years out? Will the funding
| persists? What are you even researching?
|
| I have seen people had the work of the past five years ripped
| apart during their thesis defense (or wherever), I have seen
| people make a few missteps and ruin their chances on the job
| market. I have seen people invest a lot in work that never
| makes any impact. I have seen people who thought they'd make
| Professor, and then found they have zero motivation or talent
| for research and/or teaching.
|
| You can never be sure that ain't you, until you have your PhD
| and your job.
|
| It's always on your mind. Projects aren't short anymore, you
| work years on stuff before it ever becomes a paper. You work
| every waking hour and often every sleeping hour as well. And
| you lack the knowledge and experience to assess whether what
| you are doing will really go anywhere, if the network you are
| building is right, if the people you write to will be
| interested in your work. At the very least, you never know who
| else starts with you and will compete with you on the job
| market, which is increasingly filled with people willing to
| work for little money and little job security.
|
| In the past two years, academic positions dropped 70% or more,
| at least in some fields. And we have 2-3 cohorts of applicants
| on the market due to Corona, and this will remain for years.
|
| Academic research nowadays requires nerves of steel, and I have
| seen grad school ruin the health (mental and physical) of quite
| a few people, all of them very good at what they do.
|
| So after all that doom and gloom, people need to realize that
| it is a struggle for probably everyone. But it is also
| rewarding, or it can be.
___________________________________________________________________
(page generated 2021-12-01 23:03 UTC)