[HN Gopher] What follows from empirical software research?
___________________________________________________________________
What follows from empirical software research?
Author : jimmyhmiller
Score : 30 points
Date : 2023-04-22 05:19 UTC (17 hours ago)
(HTM) web link (jimmyhmiller.github.io)
(TXT) w3m dump (jimmyhmiller.github.io)
| sampo wrote:
| > Assume for a second that a study of deep relevance to
| practitioners is replicated, and its conclusions accepted by the
| research community. It has large sample sizes, a beautiful
| research plan, a statistically sound system for controlling for
| various other explanatory factors; whatever it is that we need to
| proclaim it to be a good study.
|
| Has there ever been an empirical software study, that would have
| a beautiful research plan, sound statistical analysis, large
| sample size, and that would also have been replicated? Even one?
| politician wrote:
| Does any software engineering research take into account the
| human factors of context, interest, exhaustion, and aptitude?
| svilen_dobrev wrote:
| i doubt it. Similarly to the project trinity - functionality,
| price, time - that never included Fun ..
| Silhouette wrote:
| For personal software development projects of course there are
| other factors that matter beyond finding the theoretically
| optimal X and Y. From a management perspective in business those
| factors might also matter. You want to get the best performance
| out of your team but you're not going to do that if people keep
| quitting due to an unpleasant work environment.
|
| As far as advocacy goes though - when someone is recommending
| what _other people_ should do - I think it 's very different if
| there is relevant evidence and it doesn't back up the advocated
| position. It's even more different if there is relevant evidence
| that positively undermines the advocated position. There are
| snake oil salesmen in this industry and some of them will call
| you names if you don't follow their pet process. But if what
| they're peddling isn't backed by the evidence or even contradicts
| the evidence then they should be called out and their audience
| should probably be sceptical about anything else those same
| salesmen are selling as well. The old joke about someone finding
| it hard to believe something when their continued employment
| depends on its falsehood is as relevant as ever.
| dasil003 wrote:
| > _Assume for a second that a study of deep relevance to
| practitioners is replicated, and its conclusions accepted by the
| research community. It has large sample sizes, a beautiful
| research plan, a statistically sound system for controlling for
| various other explanatory factors_
|
| As a multi-decade practitioner and manager of software
| engineering teams, I would certainly be interested in what the
| best of the empirical research has to say. I think it's important
| to always remain open to good new ideas wherever they may come
| from--strong opinions, loosely held, as they say.
|
| That said, I don't believe that statistically significant results
| can be found that will overturn my own instincts and judgement on
| any specific project to which I am dedicated. The reason for this
| is threefold: 1) the universe of software and goals we pursue
| with is astronomically large 2) competence in software
| engineering depends on the combination of personal aptitudes and
| mindsets combined with years of practice and 3) measuring
| outcomes in software engineering across diverse projects is all
| but impossible. In other words, you can't equate tools, you can't
| equate projects, and most of all you can't equate people.
|
| At the end of the day, success in software engineering comes from
| relentless focus on the specific goals at hand. One must be
| inherently curious and have a craftsperson's mentality about
| acquiring technical skill, but never become religious about
| methodology. This requires continuous first-principles thinking
| targeted at _specifics_. At the end of the day, two expert
| practitioners could propose unorthodox and diametrically opposed
| approaches to the same problem, and they would still dramatically
| outperform a lesser skilled journeyman who attempted to follow
| every best practice.
|
| Empirical studies and the scientific method in general work
| fantastically well for uncovering the rules and inner workings of
| the natural world, but software is the creation of logical
| systems purely by human minds which is an entirely different
| challenge--there's just not enough evidence to draw on. I suspect
| results will be at least a couple orders of magnitude softer than
| sociology, and that probably won't sit well with the type of
| personality attracted to software in the first place.
| kqr wrote:
| > That said, I don't believe that statistically significant
| results can be found that will overturn my own instincts and
| judgement on any specific project to which I am dedicated. The
| reason for this is threefold: 1) the universe of software and
| goals we pursue with is astronomically large 2) competence in
| software engineering depends on the combination of personal
| aptitudes and mindsets combined with years of practice and 3)
| measuring outcomes in software engineering across diverse
| projects is all but impossible. In other words, you can't
| equate tools, you can't equate projects, and most of all you
| can't equate people.
|
| This was pretty much the word-by-word argument against using
| statistical approaches to price insurance of shipments over the
| sea back in the 1700s. Yet we all know how insurance premiums
| are calculates today, and there's a reason for it.
| dasil003 wrote:
| Word-for-word? Really? Sorry, I just don't understand your
| point.
|
| Are you saying merchants in the 1700s didn't believe
| insurance outcomes _were_ quantifiable? Or are you saying
| that software engineering output _is_ quantifiable? If the
| latter, maybe you might shed some light on how you think that
| would work, I 'm happy to be proven wrong.
| riwsky wrote:
| The author's overthinking it. He cares about productivity--it's
| just that the effect sizes are too small in these studies to
| overcome his prior beliefs.
| jimmyhmiller wrote:
| In the article I'm assuming ideal conditions. So if you think a
| large effect size is important, throw that into the ideal
| conditions. I don't think that changes anything I wrote. Maybe
| I'm missing something?
| JonChesterfield wrote:
| "Empirical software research" could mean a bunch of different
| things. This article is about studying people writing software,
| not about software used for research in empirical sciences, and
| not about research into computer science.
|
| I'm confident the answer to what follows from that is "nothing
| yet" based on various conference talks. Studying developers (or
| in a worse case students) writing software doesn't seem to be an
| effective way of working out how to write software
| better/faster/whatever.
___________________________________________________________________
(page generated 2023-04-22 23:01 UTC)