[HN Gopher] Show HN: Monte Carlo simulation for predicting proje...
___________________________________________________________________
Show HN: Monte Carlo simulation for predicting project timelines
Author : diiq
Score : 115 points
Date : 2021-06-03 10:40 UTC (12 hours ago)
(HTM) web link (quotes.vistimo.com)
(TXT) w3m dump (quotes.vistimo.com)
| orlandohill wrote:
| Joel Spolsky described this technique, calling it _Evidence Based
| Scheduling_. I 'm surprised that it didn't become more popular.
|
| https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
| commandlinefan wrote:
| > I'm surprised that it didn't become more popular
|
| It gave managers answers they didn't want to hear.
| diiq wrote:
| I think Steve McConnell was calling for statistical modelling
| even earlier than that. Definitely not a new idea!
|
| I'm working on the assumption that it hasn't caught on because
| it's a pain to do without proper tooling; but it's also true
| that many engineers believe estimation is useless, impossible,
| and a waste of time or worse. It can be hard to overcome the
| pain of having been burned by a too-tight timeline.
| FourthProtocol wrote:
| Steve McConnell wrote the Construx Estimation Tool way back
| in the '90s. It made use of Monte Carlo simulations. I still
| have a copy.
| c17r wrote:
| Came here to bring up FogBugz having this if someone else
| didn't already.
| ThouYS wrote:
| Intriguing idea
| bberenberg wrote:
| Feels like a good idea, but the hard part here is the data input.
| Do you have any plans to import from Jira or something like it?
| Ideally bi-directional sync after a given simulation is run?
| danielmarkbruce wrote:
| Out of interest, why not make this a jira app and sell it in the
| atlassian marketplace? You get low cost distribution, the tool
| would fit right into the existing product everyone is using,
| installation is easy etc, data sync is easy, you can still
| generate the nice pdfs.
| diiq wrote:
| I'm a consultant, and it happens that hardly any of my clients
| use Jira. So this is the result of scratching my itch (and the
| itch of people who already give me money) first, and dressing
| up the result for others to use.
|
| Lots of people have expressed interest in Jira integration, and
| that's definitely on my list to accomplish one way or another.
| The Atlassian marketplace is like most app stores, though -- if
| people already know your name, it's easy for them to find and
| install your app. But if you're a relative nobody, there are 50
| other estimation apps, and very little room to pitch what makes
| you unique.
| SkyMarshal wrote:
| I really like this idea and wish all project management tools
| included it by default. Fwiw Fogbugz has been using it for over a
| decade, called Evidence Based Scheduling:
|
| - https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
|
| - https://support.fogbugz.com/hc/en-us/articles/360011258994-E...
|
| - https://blog.fogbugz.com/evidence-based-scheduling
|
| There's at least one Jira app for it:
|
| - https://github.com/handshake/pystradamus
|
| And Trac also used to have EBS but I'm not sure if it still does,
| can't find it atm:
|
| - https://trac.edgewall.org/
| diiq wrote:
| Vistimo (the full version at https://www.vistimo.com) does too.
|
| It turns out the project management space is _full_ of neat
| tools with clever abilities, but it 's very hard for a team to
| switch between those tools. It's expensive and time consuming!
|
| Vistimo Quotes is an attempt to pull _just_ statistical
| estimation out on its own, so you can use it no matter what you
| use for your project management.
|
| (A note: this and EBS are pretty different in their actual math
| -- they just seem similar because the next closest thing is
| "everyone agrees on a single number per task and we add them
| up" which is so limiting that just _having_ a statistical model
| puts EBS, 50 /90, and monte-carlo into their own category.)
| BlueTemplar wrote:
| Heh, that note is why I decided that "project management" was
| just a bunch of nonsense (like any other field using maths
| without taking uncertainty into account...)
| mclouts91 wrote:
| There's a typo on the how it works page... Unkown unkows
| diiq wrote:
| Thanks, will fix.
| nwsm wrote:
| The idea is interesting, but it appears to involve using
| Vistimo's UI to break down projects and tasks. It feels
| incredibly waterfally, and also will involve re-entering mass
| amounts of project management data. I hope it integrates with
| Jira/Trello.
| alex_c wrote:
| I like this! I would consider using it for some of our projects.
|
| Biggest issue I have is with the task list, for any reasonably
| complicated project we would at least need ability to group tasks
| into features or milestones. Task lists can get pretty granular,
| but communication to stakeholders or clients does not usually
| need to be that detailed, higher level groupings are preferable
| (and also makes it easier to organize the tasks).
|
| As others mentioned, integration with Jira and such would be
| nice, but not a deal breaker.
|
| Also I think I broke my demo project, I changed one estimate and
| the projections just say "New projections are being calculated".
| There are only 4 tasks :)
| diiq wrote:
| Thanks!
|
| I totally agree, personally I want more hierarchy; but user
| testing with milestones was pretty rough. Vistimo proper
| (https://www.vistimo.com) is less slick-looking, but has
| milestones and story-mapping and bugs and enhancements and all
| kinds of hierarchical organization tools. Most users I've
| tested with have found Vistimo Quotes' approach easier to pick
| up and use on existing projects, even if it's less powerful at
| expressing the full richness of the work.
|
| Re: the long running calculations, I will take a look, but it's
| possibly just a traffic jam. There are a few more "test"
| simulations running simultaneously than usual this morning.
| pton-throw wrote:
| The UI is slick, but why should I trust this black box?
|
| The "how does it work" section doesn't even mention
|
| * modeling assumptions
|
| * training data (if any)
|
| * validation (if any)
|
| Why is this better than, say, giving the 25-75% range around my
| "best guess" assuming log-normal duration and the variance of
| prior project completion times? Can you demonstrate that it's
| better than that?
| diiq wrote:
| It's not better; it's almost exactly that, as detailed in many
| comments below. I choose slightly different thresholds -- 75%
| is not very high, you'll be wrong one time in 4 -- but that's
| the approach.
|
| If you're already doing log normal modeling of your estimates,
| and including sick days, social loafing, etc, then you don't
| need Vistimo Quotes. Just keep doing what you're doing!
| pton-throw wrote:
| > as detailed in many comments below
|
| This is something I would buy if I had confidence in the
| simulator.
|
| Why not write a blog or a brief white paper giving some hint
| of what's going on behind the scenes?
|
| Generally speaking, every bit of energy someone spends being
| cool (good UI) comes at the expense of energy spent being
| good. Maybe you've done some serious work on the simulator,
| but I can't tell from your marketing material.
| [deleted]
| virgil_disgr4ce wrote:
| I'd like to see some kind of explanation of how exactly you
| "express the various uncertainties" of the project. I'm not going
| to click the CTA until I understand what I'd actually be doing to
| use the product.
| diiq wrote:
| The video on the How It Works page
| (https://quotes.vistimo.com/how-it-works) has footage of every
| step you'll take to run an estimate.
|
| If you mean you want technical details of the math involved,
| https://blog.vistimo.com/
| weeblewobble wrote:
| There's a product called LiquidPlanner built around this idea
| thats been around for years. A friend used to work there but I
| have no other affiliation. Looks like they're still kickin'
| https://www.liquidplanner.com/
| darepublic wrote:
| Trying to get correct estimates feels like it violates some basic
| quantum principle of physics. Were venturing into Schrodinger
| territory here folks.
|
| On a more serious note, good idea and nice site. I may give this
| a try. I have been thinking along these lines myself, using
| simulations to complete different dev related tasks
| Grustaf wrote:
| Interesting idea, and very nice and simple web page! I have no
| idea if people are willing to pay for it, but good on you for
| making an attempt.
|
| A question though, isn't Monte Carlo a bit overkill? Would you
| lose much by modelling each deadline as a normal distribution?
| Then you could just calculate things analytically.
| diiq wrote:
| You're not wrong; normal distributions are OK but not _ideal_
| for estimating durations -- it can lead to weird artifacts if
| you don 't restrict yourself to just the mean.
| (https://blog.vistimo.com/post/169546687605/estimation-math)
|
| And once you're committed to using monte carlo, you're free to
| include one-off events -- sick days, vacations, social loafing,
| etc, that are really hard to model analytically, since they are
| roughly uniform in time -- the longer the project runs, the
| more days off make the project run longer.
| iainmerrick wrote:
| That blog post is very clear, nice work!
|
| As another alternative, could you convolve the probability
| distributions numerically? I'm not sure of the trade-offs
| between that approach and Monte Carlo, but if it's something
| you've considered I'd be interested to hear your thoughts.
| diiq wrote:
| Once I took a few steps down the simulation path, it seemed
| so easy and powerful that I didn't look back. There is some
| computational cost, but for me it's more than made up for
| by how easy it is to, for instance, say "I'd like to model
| sick days"; * let the government tell me how often people
| are sick (https://www.bls.gov/cps/cpsaat47.htm, see
| "Computer and # mathematical occupations", absence rate,
| due to illness or injury), * make my simulated workers take
| days off at that rate.
|
| And everything else, all the statistics etc, can stay
| exactly the same. And I can have a lot of trust that it's
| producing sane results because I can _look_ at a run, and
| say yes, that 's a plausible number of sick days.
|
| Monte carlo requires you to be clever once, setting up the
| simulation, and then you almost never have to be clever
| later when you want to introduce new twists. Any other
| statistical model you have to keep being cleverer for every
| additional complication.
| Grustaf wrote:
| I suppose you could model those one-off events with their own
| normal distributions (with 0 variance), rather than modifying
| the main distribution?
|
| But sure, Monte Carlo is more flexible, and not really harder
| to implement I suppose, since you're already on a computer!.
| breakfastduck wrote:
| I thought this would be brilliant, so I clicked 'try it out' only
| to get through and be asked to _manually write in_ the tasks. I
| have a huge backlog already in an external tool.
|
| I can't comprehend I'm expected to write them out one by one with
| no ability to import that I can see.
| diiq wrote:
| Haha, at least tell me _which_ external tool you 'd like me to
| add an import from, and I'll be happy to add it to _my_
| backlog.
| colincooke wrote:
| A simpler, if perhaps less effective, idea would be to just take
| advantage of ensembling. Get many estimates from everyone
| qualified to give them, then just fit a normal distribution to
| your final results. The average of many estimates tends to
| actually be quite accurate sometimes, with the famous example of
| visitors to a county fair guessing the weight of an ox[0]:
|
| "At a county fair held in Plymouth, 800 visitors had participated
| in a competition to guess the weight of an ox. While most
| people's estimates were too high or too low -- falling an average
| of 37 lbs. away from the true weight of 1,198 lbs. -- the median
| of everyone's guess was off by only 9 lbs., or less than 1
| percent of the true weight of the ox."
|
| Curiously there is also some "within individual" averaging that
| helps too. If you would ask the same person to estimate the
| project timeline a few times, spaced temporally, then take the
| average, it'll probably be more accurate than a single estimate.
|
| [0] https://www.npr.org/sections/13.7/2018/03/12/592868569/no-
| ma...
| shakezula wrote:
| We tried this at $OLD_JOB and what I've found is that when
| there is a larger disparity between estimates, it hints at
| information asymmetry, but that asymmetry can still exist when
| everyone has similar ideas about timelines, and so you can
| still get drastically wrong estimates when everyone thinks
| they're in agreement.
| beckingz wrote:
| Especially when people have different experience or lack
| experience around a specific aspect.
|
| Getting a data scientist to estimate how long the front end
| will take to build will not be as helpful.
| Exuma wrote:
| How was 1 std dev 37 lbs for that??? I feel like if I guessed I
| might be off by 500 or more lbs. I would have guessed "slightly
| less than a small car".
|
| 37 lbs seems astronomically accurate unless this is a county
| fair of farmers...
| SMAAART wrote:
| https://en.wikipedia.org/wiki/Prediction_market
|
| https://en.wikipedia.org/wiki/Wisdom_of_the_crowd
| Grustaf wrote:
| Exactly, I was going to suggest a prediction market, that
| would be cool. Especially if employees have to use real
| money!
| diiq wrote:
| You're right! Ensemble forecasting is great! Vistimo proper
| (https://www.vistimo.com) does both -- it gets individual
| high/low estimates from multiple people, and creates a combined
| distribution from all of them (though each estimate is a log
| normal, not normal, since it's a time estimate).
|
| But Vistimo is a full project management tool; that space is
| full and switching project management tools is expensive, so
| it's hard to get people to adopt.
|
| Vistimo Quotes is a much lighter-weight way to get many of the
| same benefits without having to talk your whole team into
| signing up.
| adam wrote:
| I'm curious if you allow the project to enter actuals after
| the fact, then you can compare them to individuals/aggregate
| estimates so you know how biased your model/individuals/the
| organization is on these types of estimation efforts. You
| could give that as feedback as an additional service: "You're
| constantly overconfident on X type tasks..."
| diiq wrote:
| Not at the moment. This is the dream, but it's
| statistically much harder than it seems!
|
| For each task you have one distribution, and one sample.
| What's the likelihood that sample was actually from a
| _different_ distribution?
|
| Instinctively it feels easy to judge "your usually closer
| to the worst-case than the best-case" but mathematically
| it's not so straightforward, especially when the estimate
| _doesn 't_ include sick days, splitting time with other
| projects, etc, but both the projection and the reality
| _do_.
| dglass wrote:
| I've used this technique before at an employer. I thought it was
| crazy at first but I came to really like it. No more story points
| and estimating the size of tickets. The monte carlo simulation
| handles the forecasting for us so the engineers can focus on
| completing tasks. The PMs focused on user stories and acceptance
| criteria. The stakeholders had a continuously updated forecast
| for when things would likely get done.
| poodler wrote:
| Nice service, lots of ways to grow. I think the ability to make
| pretty pdfs for clients is actually a strong pull.
|
| In my experience, statistical modeling of potential timelines is
| not the bottleneck to accurate quotes: it's the ability to
| actually give good guesses for "min time" and "max time" on
| critical tasks. I.e. the majority of timeline variability is due
| to just a few "tasks" that totally blow out of proportion because
| I underestimated how much work was actually involved. Idk, maybe
| that's just me.
|
| It would be nice to track my estimates vs actual time spent to
| improve the "human error" part.
| diiq wrote:
| Yep, it's down. Y'all have just about tipped over my baby redis
| instance with a few thousand project simulations all at once.
| Should be back as soon as I scale 'er up.
| poodler wrote:
| gotta love HN. no problem, thanks
| diiq wrote:
| I believe we're back up. Thanks for your patience.
| skellystudios wrote:
| You should 100% have a sample tool for people to play with on
| this homepage.
| nscalf wrote:
| Or even just a sample video. There's a good bit of friction
| before even knowing what this is.
| nwsm wrote:
| https://quotes.vistimo.com/how-it-works
| Raidion wrote:
| Yep, I'm not giving my email to something before I have a
| decent understanding of what it does. Even the "Try it out"
| redirects you instantly to a signup modal.
| tschottdorf wrote:
| I tried to use this but once I start entering tasks I don't see
| how to proceed. There's no way forward from neither there nor the
| project list page. Am I missing something?
| diiq wrote:
| Once you enter tasks, and estimate at least one, you should see
| a link to view your "Projections".
|
| (Fetching those projections uses websockets, so you may need to
| refresh if you're behind a particularly restrictive firewall.)
|
| If you're still having trouble, drop me a note at
| sam@sambleckley.com and I'd be happy to help.
| asdf_snar wrote:
| As a package this seems perfectly reasonable, but I'm skeptical
| of the use of Monte Carlo being a technical buzzword rather than
| a choice made because it was the appropriate tool for the task. I
| wasn't able to find any technical details in the "How it works"
| section, but if you assume the distribution of time to complete a
| task is uniform on the interval [worst case, best case], then at
| any time point the underlying distribution seems to be a
| composition of sums and minima of uniform distributions. These
| are available in closed form. I see in another comment the
| distributions are assumed to be log-normal -- that's fine too,
| there are good approximation to these. On the website it says
| thousands of trajectories are generated. It seems likely that one
| can straightforwardly compute the result that the Monte Carlo
| approach would give in the limit of infinitely many generated
| trajectories.
| diiq wrote:
| I promise, this is the result of real constraints! Monte carlo
| is used because of the limitations of analysis when using log-
| normal distributions. Normal distributions can be summed
| algebraically; log normal distributions cannot.
|
| https://blog.vistimo.com/post/169546687605/estimation-math
|
| Note that Monte-carlo is not used prominently anywhere on the
| site, just here because I thought HN would find that part of
| the implementation interesting.
| asdf_snar wrote:
| It is true that there is no closed form expressions for the
| sum of two independent log normal distributions. However,
| like I said, good approximations exist. Moreover, if you were
| to ask me to compute some expectation E[f(X)] where X is the
| sum of independent one-dimensional log-normals, it seems
| simplest to convolve the densities and simply compute the
| expectation by trapezoidal integration.
| diiq wrote:
| Fair enough! There's more than one way to skin a cat. This
| way was easier for me, and has additional benefits as
| described.
| asdf_snar wrote:
| I think I misled myself into thinking your selling point
| was the technology (Monte Carlo), whereas as you say
| there is little about that on the website. And your
| comments below are right: building the tool around MC
| affords you a lot of flexibility without having to be
| increasingly clever.
|
| The only thing I would worry about then are events which
| have low probability but potentially very high "weight"
| in whatever measure of risk you are using. Sometimes the
| weight can be so large that your MC estimates seem to
| converging, and then the addition of a single trajectory
| completely blows up your variance. These are related to
| the "black swan" events people sometimes talk about. I
| don't think this is a problem in your model.
|
| Thanks again for ansnwering.
| enlightens wrote:
| Very interesting! Is there a sample report / projection available
| to view? I'd like to show this to a few coworkers without having
| to spend time loading in my own sample data
| vladsanchez wrote:
| Hi Sam, your work is admirable and I'm sure lots will pay
| attention.
|
| As a side note, have you seen/heard of (sic) "Vistima" parody?
| https://youtu.be/4jVQhdlCmjc
|
| Congrats, Stay Thirsty.
| a_zaydak wrote:
| The idea seems simple enough and the math to do these kinds of
| predictions is fairly straight forward unless there is some
| secret sauce under the hood that isn't obvious. I guess the value
| is slick UI / tool that was created around it.
|
| One question that I do have about these types of tools... how
| does one evaluate how well it worked? I guess in the end as long
| as the timeline wasn't too far off then you call it a win.
|
| I think a useful tool for project planning is not necessarily
| simulating possible time-lines (or a estimated window of time for
| the project to be completed) however to intelligently identify a
| set of critical tasks within the list which would have a high
| probability of totally blowing up your time line their deadlines
| are missed. In project management / scheduling there are often
| critical path items listed. However often these critical items
| are semi-arbitrarily picked by the PM and not backed by and deep
| insights.
| diiq wrote:
| About 5 years ago, when I first started using this method, I
| gave a talk about it that went into detail about how to do it,
| and the benefits. I did a live demo showing how well it modeled
| the variance even for really simple tasks like "take off your
| shoes and then put them back on".
|
| The audience reaction was very positive, and I was pleased to
| have spread the word.
|
| No one actually put it into action, though. It turns out if you
| need better estimation tools, you don't have time to make them
| -- you're in crunch time, because your estimates were bad.
|
| The benefit is not so much slick UI -- that part is just me
| having fun. The benefit is that the work, however easy, is
| _done_.
|
| If folks "steal" this idea because it's easy to implement,
| that's great! It's not a new idea, just an underused one. I
| just want to stop being hired onto contracts where the first
| thing I have to do is explain that the deadline is impossible.
|
| I agree with you that a linear backlog is not sufficient to do
| really sophisticated planning -- Vistimo Quotes is a pared-down
| tool designed for easy adoption, because my kitchen-sink
| version requires too much buy-in for many teams.
| a_zaydak wrote:
| Sorry, I did not mean to imply that the only value that the
| creators had was making a nice UI. I truly meant it as a
| compliment. There is hard work involved in creating a product
| out of an idea. Seems like this will be a very useful
| product. Congrats to all of the people who have worked on it.
| diiq wrote:
| Haha, you're fine! It's pretty much just me behind the
| curtain.
|
| (And honestly, sometimes I worry the UI is _too_ slick --
| that it will give people expectations the tool won 't live
| up to.
|
| But I have too much fun making fancy buttons, I can't bear
| to get rid of 'em :) )
| jkaptur wrote:
| Oh wow, my first job in the industry was building something very
| much like this!
|
| It was actually quite powerful: the UI and simulation were done
| by Microsoft Project (https://www.microsoft.com/en-
| us/microsoft-365/project/projec...), which I would describe as
| "the Excel of project management" in that it was an ENORMOUS bag
| of features for building Gantt charts. The coolest part was that
| it integrated with Outlook's calendar, so if a task started on
| June 1st, was assigned to Bob, took two weeks, and Bob was on
| vacation from June 8th-22nd, it calculated the end date for the
| task as the end of June.
|
| The Monte Carlo part was run by Crystal Ball (which was a startup
| then, but has since been acquired by Oracle). It was a mature
| tool, so there was a big library of probability distributions (ht
| tps://docs.oracle.com/cd/E57185_01/CYBUG/define_assumption...),
| you could enter correlations (if X task takes a long time, Y is
| going to take a long time too), and you could analyze the output
| in all sorts of ways (how sensitive is the deadline to this task,
| how _frequently_ is this task on the critical path, etc.).
|
| The implementation was quite a hack - a hidden instance of Excel
| running Crystal Ball and lots and lots of VBA to manage
| everything. I'm not sure that it was ever used for real :(
| db48x wrote:
| Nobody can design unilateral phase detractors in just 4 hours.
| ballmerspeak wrote:
| If those buttons are appealing, I recognize them from this blog
| post by Josh Comeau
| https://www.joshwcomeau.com/animation/3d-button/
| diiq wrote:
| Haha, look closer! These buttons have a continuous curvature
| profile, rather than a rounded rectangle. It's a big part of
| what makes them feel so "70's calculator" chic :)
| awillen wrote:
| Until you have some reasonable way to quantify the likelihood
| that the CEO will see something shiny and insist on changing
| course, this just doesn't seem like it would be useful in the
| real world.
| diiq wrote:
| There's an "unknown unknowns" factor specifically for including
| guesses about new features or last-minute changes.
|
| And the great thing about being able to run estimates quickly
| is that you can tell the CEO how much their new idea will send
| the timeline off course as soon as they propose it, so they can
| decide if it's really worth the cost.
|
| I've been using this tech for years as a consulting engineer;
| this is just the dressed up version. It's been useful and
| effective for me, at least.
| poxwole wrote:
| Can anyone link some research papers on this?
___________________________________________________________________
(page generated 2021-06-03 23:01 UTC)