[HN Gopher] How to create a dashboard in Python with Jupyter Not...
___________________________________________________________________
How to create a dashboard in Python with Jupyter Notebook
Author : pplonski86
Score : 168 points
Date : 2022-06-14 08:26 UTC (14 hours ago)
(HTM) web link (mljar.com)
(TXT) w3m dump (mljar.com)
| westurner wrote:
| Surprised I hadn't heard of mplfinance. Similar code for
| econometric charts with bands and an annotation table with dated
| events would be useful and probably already exists somewhere.
| Maybe Predictive Forecasting libraries for time series already
| have something that could be factored out into a maintained
| package for such common charts? E.g. seaborn has real nice
| confidence intervals with matplotlib, too; though the matplotlib
| native color schemes are colorblind-friendly: "Perceptually
| Uniform Sequential colormaps"
| https://matplotlib.org/3.5.0/tutorials/colors/colormaps.html
|
| awesome-jupyter > Rendering/Publishing/Conversion
| https://github.com/markusschanta/awesome-jupyter#renderingpu... :
|
| > _ContainDS Dashboards - JupyterHub extension to host
| authenticated scripts or notebooks in any framework (Voila,
| Streamlit, Plotly Dash etc)_
|
| IIRC, ContainDS and Voila spawn per-dashboard and/or per-user
| Jupyter kernels with JupyterHub Spawners and Authenticators; like
| Binderhub ( https://mybinder.org/ ) but with required login and
| without repo2docker?
|
| Streamlit lists Bokeh, Jupyter Voila, Panel, and Plotly Dash as
| Alternative dashboard approaches:
| https://github.com/MarcSkovMadsen/awesome-streamlit#alternat...
|
| Says here that mljar/mercury is dual-licensed AGPL:
| https://github.com/mljar/mercury
| westurner wrote:
| From the readme I can't tell what powers the cited REST API
| functionality. From setup.py > mercury/requirements.txt it's
| DRF: Django REST Framework.
| https://github.com/mljar/mercury/blob/main/mercury/requireme...
|
| E.g. BentoML is built on FastAPI which is async (sanic) and
| built by the DRF people, but FastAPI doesn't yet have the
| plethora of packages with tests/ supported by the Django
| community and DSF Django Software Foundation.
| pplonski86 wrote:
| The REST API functionality allows to execute the notebook by
| POST request. It is powered by DRF. You can read more in the
| docs https://mercury-docs.readthedocs.io/en/latest/notebook-
| as-re...
| pplonski86 wrote:
| The Mercury is available with AGPL license (OSS version) and
| with commercial license with additional features and dedicated
| support (Pro version).
| sireat wrote:
| Mercury looks rather neat for quick dashboards.
|
| For dynamic data I build my dashboards with Flask and Plotly.
| However that is certainly a more involved process to migrate
| Jupyter notebook Python plots to JS plots.
|
| For static data one can just export html(with embedded JS) from
| Jupyter notebook that uses plotly.
|
| The main difference with quick solutions is the level of
| customizability.
| pea wrote:
| Re your HTML point, we've been hacking on a framework which
| lets you programmatically create and share data reports in
| Python which you might find helpful
| (https://github.com/datapane/datapane). It supports Plotly,
| Pandas, Altair, Folium, MPL, etc., and provides some neat
| layout components like pages, selects, columns, and dropdowns.
|
| If you need anything or want some help, feel free to make an
| issue or ping me on leo [-at-] datapane.com
| pplonski86 wrote:
| Mercury works very well with Plotly (here is a demo
| https://mercury.mljar.com/app/7). If you need to choose between
| Flask+Plotly and Mercury I would strongly recommend Mercury
| because of the speed of development.
|
| What do you mean by dynamic data? Do you need automatically
| fetch new data in time intervals? (Mercury has scheduling and
| auto-update built-in with 1 minute lowest time interval).
| kaa2102 wrote:
| Mercury is a great find for me as it solves UI/UX design &
| usability concerns for my Python automation projects for non-
| techninal users.
| oneoff786 wrote:
| My initial reaction is that there's a ton of lag on the initial
| gif. Makes me think this is going to be a cumbersome thing to
| work with. Is it representative of a typical task, whatever that
| means?
| pplonski86 wrote:
| The demo is hosted on free Heroku dyno with 0.5 GB RAM. This is
| version 1.0 and still many things can be improved in the
| Mercury framework.
| blagie wrote:
| I looked over the web site. A few questions:
|
| - Why won't the OSS version provide features like history of
| execution? It seems like those are pretty basic to adoption. I'm
| working on an open source project, and it'd be a nice tool to
| potentially integrate. The commercial license is of zero interest
| since it's, well, all open-source.
|
| - Why don't you sell the OSS version as well (e.g. for people who
| want to support you, or who want support)?
|
| - Do you provide consulting services? Are they in the $50/hour
| range (just extrapolating from $200/year for up to 4 hours
| support)
|
| - Why Django? I like Django, but it seems like using the same
| tech stack as Jupyter would make for simpler integration,
| administration, and lower TCO (plus the universe is going async)
|
| - For presentation, the speed is an issue. Regeneration takes a
| while (I'm playing with the demo on Hugging Face). It's not quite
| adequate for interactive presentation. I do really like the
| concept of making presentations in Jupyter.
|
| - And on that topic, the biggest gap I have is with interactive
| front-ends. I'd like to be able to somehow connect notebooks
| (both for dashboards and otherwise) for data munging to data
| visualization frameworks like D3. Jupyter is nice. Observable
| would be nice if it were open-source. There's a mile-high wall
| between the two.
| pplonski86 wrote:
| - What is your project? is it from data science space?
|
| - Selling OSS might be a great idea. Didn't consider it. I
| thought the Personal Pro version ($199/y) will be good for
| persons that would like to support the project or just need a
| support/guidance.
|
| - Yes, I do consulting services. I don't treat support hours in
| similar way as consulting. The support is for product that I
| love working on and I love supporting people using it.
|
| - Django - I'm pretty good at Django. What is more, I wanted to
| use Celery for processing background tasks and scheduling. This
| gives an architecture that can handle large loads. The frontend
| is in React with TypeScript.
|
| - Speed can be improved. There are many ways to better handle
| that (it is version 1.0). You can add Plotly charts to make
| presentation more interactive. The advantage of Mercury is that
| you can recompute all slides if needed.
| blagie wrote:
| I don't disclose my identity online, and I think if I posted
| the project (or a description), it would do that. If you're
| curious, I'm glad to reach out by email. It is a data science
| project, and potentially, high-profile if it goes well.
|
| On a mile-high level, Mercury seems like a perfect fit for
| what I'm doing. Conceptually, it's exactly what I was looking
| for.
|
| Diving into the code, the fit seems less great, for reasons
| which are incidental (and wouldn't be worth changing for one
| integration). I'm trying to constrain the number of
| technologies in the overall project, to make it maintainable
| and deployable, and you seem to be on a completely orthogonal
| stack.
|
| So I'll keep diving in and trying to understand it better.
| pplonski86 wrote:
| My email is piotr at mljar dot com
| maliker wrote:
| When I need to expose jupyter notebooks to non-technical users I
| put them on colab.research.google.com. It's a free, interactive,
| hosted Jupyter environment. It's not as slick as a dashboard, but
| it just requires an upload instead of maintaining a server.
| analog31 wrote:
| Ask HN: Any idea if jupyter lite is a contender for sharing
| notebooks?
| pplonski86 wrote:
| With Mercury you can:
|
| - host multiple notebooks,
|
| - you can easily hide code,
|
| - you can schedule notebooks,
|
| - you can download notebooks as PDF,
|
| - you can easily add authentication to the notebooks.
|
| It all depends on your needs and project requirements.
| eoinmurray92 wrote:
| you might want to give Kyso a try (Im a cofounder):
|
| https://about.kyso.io/
|
| We render jupyter notebooks, markown and html - we dont host
| running jupyter environments -> but we focus on making the
| experience of sharing with non-techies on your team easy. So
| can show/hide the code, get comments etc.
| anigbrowl wrote:
| This is great! I have a lot of tools built in notebooks that I'd
| like to be able to share with non-technical collaborators, and
| have been procrastinating building a UI in Flask or Django as I'm
| really not interested in front-end stuff. This is a fun and
| simple way to deploy that won't be a distraction from collection
| and analysis I actually enjoy. I look forward to using it!
| pplonski86 wrote:
| The Mercury framework was designed to make notebooks sharing
| easy. You don't need to build UI. Just define the widgets in
| the YAML header. Widgets are directly connected to variables in
| Python code. You can share notebook with non-technical users
| and hide the code (show-code: False). The user can tweak
| widgets and execute the notebook.
|
| Additional features for sharing:
|
| - user can download notebook as PDF or HTML,
|
| - you can schedule notebook execution, and send executed
| notebook as PDF attachment in email notification,
|
| - adding login view to notebooks is simple, just add `share:
| private` to YAML
| criddell wrote:
| Ever since I read about the Panic status board project [1] I've
| been looking for similar projects. This one looks pretty cool.
|
| [1]: https://panic.com/blog/the-panic-status-board/
| crisbal_ wrote:
| How does this compare to Voila from the Jupyter team? (
| https://voila.readthedocs.io/en/stable/index.html )
| pplonski86 wrote:
| Mercury has a different architecture. The Voila keeps the live
| kernel and connection to UI (using Tornado framework). To use
| Voila you need to add widgets (with ipywidgets) to the notebook
| (mix UI code with analytics code).
|
| The Mercury generates UI based on the YAML header (very simple,
| no need to mix UI with analytics code). When user tweaks
| widgets values, the whole notebook is executed with new
| parameters and converted with nbconvert (using Django +
| Celery). Mercury can serve multiple notebooks to multiple users
| on one server. It has option to export notebook to PDF or HTML.
| You can schedule the notebook execution with crontab string
| (for example `schedule: '30 8 * * 1-5'`) and add email
| notifications. You can easily add authentication to notebooks.
| It was designed to make notebook sharing fast and easy.
|
| What is more, I'm thinking about including Voila into Mercury.
| So you will be able to serve Voila apps with Mercury. The end-
| goal is to make notebooks sharing easy.
|
| Mercury github repository https://github.com/mljar/mercury
| sdoering wrote:
| This is great. Just sent the link to colleagues looking for
| an easy solution to building and sharing dashboards.
| curiousgal wrote:
| Nothing beats R's Shiny for making dashboards from my experience.
| discordance wrote:
| R/Shiny is sweet. Since stepping into the python DS world I've
| used Streamlit to roughly the same effect.
| pplonski86 wrote:
| R Shiny is fantastic solution. To be honest, the Mercury is
| heavily inspired by R Markdown.
| swyx wrote:
| there's nothing inherent about why we cant port Shiny's
| features into the Python ecosystem. what do you like most about
| Shiny that is missing? (from Mercury or otherwise)
| nicolaskruchten wrote:
| We've got a number of solutions in Python that rival Shiny
| IMO: Dash, Voila, Panel, Streamlit, new stuff like Mercury
| etc :)
|
| There's a neat overview of the landscape here
| https://www.youtube.com/watch?v=4a-Db1zhTEw
| db65edfc7996 wrote:
| Ehhhhh, I'm a Python guy, and I find all of the Python
| Shiny alternatives lacking. Shiny delivers an incredible
| amount of value per line of code. My experiments with Dash
| and Streamlit required a lot more boilerplate to get
| something running vs the R equivalent.
| pplonski86 wrote:
| I love your attitude!
| lmeyerov wrote:
| Awesome to see!
|
| We've been encouraging Graphistry users to look at Streamlit for
| the low-code side (and getting bundled into our next release for
| public+private pydata GPU dashboarding!) or Databricks
| dashboards. Then at the expense of significantly more code /
| complexity without going full Flask/Django, we generally
| recommend switching to Dash/Voila/etc for getting way more
| control. Mercury seems quite close to the Streamlit/Databricks
| side, which in our experience is right for many pydata
| dashboarding teams yet most tools don't really nail. It may even
| be on the path to one-upping streamlit on the simplicity side.
| Good job & will be tracking!
| pplonski86 wrote:
| Thanks! Mercury's goal is to make notebook sharing as easy as
| possible for creators (developers) and viewers (non technical
| users).
| urthor wrote:
| YMMV, has any technologist found a dashboard useful?
|
| I've had a very bad history with dashboards. All I've seen get
| ordered up by nontechnical leadership, not looked at, and
| involved endless fiddling with painful corporate applications.
|
| The non-technical stakeholders I've had a good relationship with,
| I vend a SQL script/Jupyter Notebook and educate. They're happy
| to up-skill.
|
| I've only had dashboards requested by users with a dysfunctional
| relationship with engineering, in companies where the data is
| irrelevant.
|
| In environments where data _matters_ , fast moving consumer
| goods, web, etc, stakeholders speak SQL/Jupyter. I've seen
| remarkably non-technological people learn SQL and query read
| replicas because they _need_ sales aggregates.
|
| In environments where data doesn't matter, dashboards are
| requested as a corporate power play.
|
| A director who doesn't care about your project asks for a
| dashboard to assert their authority, and remind you your budget
| can be cancelled.
| cirgue wrote:
| Dashboards are really useful when you're trying to distribute a
| set of metrics that large numbers of people actually need to
| know and agree on in order to do their jobs. Most metrics
| aren't like this, so there are a lot of bad dashboards, but
| they definitely have their place.
|
| Edit: A good example of this is something like incoming
| shipments for a warehouse. Management all the way down to
| warehouse workers need to know what is slated for delivery,
| what's already been delivered in the last few days, and what's
| late.
| sea-shunned wrote:
| Streamlit[0] was created specifically to create dashboards for
| ML/data science groups, and I've found it pretty useful. I've
| used it for research (model inspection and development), as
| well as teaching and it's been pretty useful for that.
|
| [0] - https://streamlit.io/
| javier_e06 wrote:
| Look at my latest gizmo...on a drone! Look at my latest data
| crunch app...on the web! Yep, smells like management all over.
| slowhand09 wrote:
| Tons of good experience with dashboards.
|
| Managing Oracle databases? Enterprise Manager Cloud Control.
|
| Managing your investments? Schwab has a good interface.
| jwithington wrote:
| I've certainly had your skepticism and these same challenges.
| Nonetheless, one study of Shopify storeowners who adopted
| dashboards saw a 4-10% increase in weekly revenue post
| adoption:
| https://pubsonline.informs.org/doi/abs/10.1287/mksc.2022.135...
|
| But to your point there's often a principal agent problem in
| larger organizations. A decision-maker wants to appear "data-
| driven" so asks for a dashboard because it costs nothing for
| them to ask--someone else does the work. And when the dashboard
| shows something different than expected they dismiss the
| concern or find a reason that the dashboard is "wrong."
| pplonski86 wrote:
| You are lucky! There are many people that don't want to touch
| any code. Setting up a Python environment locally might be a
| big pain. That's a huge blocker for notebooks sharing.
|
| I treat dashboard as visualization part. What is more,
| visualized data is used to trigger further actions - for
| example, send alert email. Yes, I found them useful.
| kulikalov wrote:
| You can share notebooks using hosted solutions
| pplonski86 wrote:
| Yes, that's the solution but only if you share with people
| that understand Python and if you are allowed to upload
| code + data to the cloud.
| syntaxfree wrote:
| Docker?
| quinoablast wrote:
| You're definitely right that metrics are often used as post-hoc
| justification of decisions already made.
|
| An example of actual dashboard culture is AWS's operational
| metrics meetings. The construction of the dashboards themselves
| is scrutinized just as much as any latency or availability
| metric (may God have mercy on your soul if use a pie chart or
| display the median of metrics without justification). And it's
| actually useful for surfacing issues and knowledge sharing. The
| frequent cadence can be a big cause if stress though.
|
| https://aws.amazon.com/blogs/opensource/the-wheel/
| kulikalov wrote:
| Non-techies wielding SQL is a big advantage for sure, but a
| moderately complex dataset sends SQL aggregations complexity
| through the roof. While modern data warehouses can handle such
| queries, I can hardly imagine non-technical person being able
| to write it.
|
| That's where dashboards come in handy.
| urthor wrote:
| Denormalized views? Every dashboard is an exercise in
| denormalization anyway.
|
| Certainly I would never expect people to be experts. SQL
| Group By is nontrivial.
| kulikalov wrote:
| Still limited to 2-dimensional data. Data warehouses
| support structs and arrays, but it's not easy for a human
| to read.
| enos-ape wrote:
| > I've only had dashboards requested by users with a
| dysfunctional relationship with engineering, in companies where
| the data is irrelevant.
|
| Sry, but this completly ... not true! Maybe in your subjective
| business environment, but definitely not for the whole
| community of python users.
|
| We are placed in a scientific environment, teamworking with
| highly dimensional data. Here there is not a single person who
| has time to tinker around with another ones code nor data
| structure.
|
| This will be a great benefit for regular report sessions - THX
| to the author!
| nkrisc wrote:
| From the non-technical side: it isn't necessarily that
| dashboards are not useful, it's just that most people who want
| them don't actually know what data is useful.
| pid-1 wrote:
| Yes I've seen many good uses of dashboards. That said, I
| totally get where are you coming from, been there as well.
| Mehdi2277 wrote:
| I quite like monitoring dashboards and use them frequently.
| Mainly for examining system health and debugging operational
| issues.
|
| My previous team had some business metric charts/dashboards
| related to system we worked on that we reviewed weekly. Most
| weeks review was it looks normal and was pretty fast review.
| Occasionally we'd see an unexpected deviation in some top level
| metric and that could trigger follow up work. Ideally all
| interesting deviations an alert would catch, but alerts are
| never complete.
|
| Most of dashboards I've cared for my team/department was
| primary user of. I have not made dashboards much for other
| stakeholders.
| analog31 wrote:
| Whenever I see a "dashboard" I think of something like this:
|
| https://www.amazon.com/Fisher-Price-DYW53-Rollin-Strollin-Da...
|
| It's a kids toy that lets them pretend they're driving the car.
| cptaj wrote:
| I've found them very useful at impressing clients and getting
| contracts XD
| [deleted]
| monkeybutton wrote:
| People that run businesses and care about data don't want
| dashboards, they want spreadsheets. Lately I've been having
| some success with creating "user friendly" data views in big
| query and giving out access to them with connected Google
| sheets. If anyone wants to make a nice graph, pivot table or
| add their formulas to the workbooks they're free to do so using
| a tool they're already familiar with. The the data is backed by
| the database and can update on its own so we're not constantly
| sending out new versions of exported files.
| pplonski86 wrote:
| I would say it depends on data type and size. There are data
| sizes that would not fit in spreadsheets and need to be
| presented in a different way.
|
| BTW, I love spreadsheets. Spreadsheets + Python is very
| powerful connection :)
| pid-1 wrote:
| Excel can get data from APIs nowadays (and even use modern
| auth!).
|
| I've been reasonably successful at creating a data
| environment for my company in which all the data is exposed
| through REST APIs.
|
| The existing tooling for APIs in general is top notch. Also,
| you get a lot of architectural flexibility.
| keithalewis wrote:
| "I find your ideas interesting and would like to subscribe to
| your newsletter." Sadly, your common sense observations will
| get belittled by the growing crowd of Liliputians who seem to
| enjoy celebrating their ignorance.
| urthor wrote:
| I don't mean this as a criticism of the link author. Nobody
| wants to use a toilet plunger. You still want a good one.
|
| Merely an observation about the dashboard industry.
| apwheele wrote:
| Ditto, although for my applications they are not live (typical
| business processes lag by a month or more). So most of my
| stakeholders are happy that I just send a monthly PDF with
| updates.
|
| The endless tinkering is still requested with the static
| reports, but that is much easier to deal with than serving up
| dashboards _and_ tinkering with that.
| blagie wrote:
| Yes, I've found dashboards useful.
|
| Generalizations or problematic, since so much of this depends
| on industry, company, and role.
|
| I'm in an industry (education) where at-a-glance dashboards
| have a proven track record of dramatically improve learning
| outcomes. It turns out that:
|
| (1) Teachers, while believing they know their students,
| actually don't. Teachers also regularly fail to grasp what
| students do and don't understand.
|
| (2) Allowing teachers to monitor their students, both
| individually and classroom wide, leads to much better outcomes.
|
| In the context of a classroom, teachers are busy, non-technical
| (and sometimes incompetent), so an at-a-glance view is
| critical. For an example of real-time dashboarding done well,
| see Learning Catalytics. It grew out of a highly-respected
| physics education research project (Peer Instruction, by Eric
| Mazur), and allows teachers to monitor student understanding
| during lecture. Showing where misconceptions exist allows for
| deeper dives and deeper conversation. There are many systems
| like it, and ample evidence of improved student growth when
| they're used.
|
| That's very specific, I know, but there are similar examples in
| other industries I've worked in.
| urthor wrote:
| Real time monitoring during a live task is certainly a use
| case I've never thought of.
|
| A very good point.
|
| I remember a piece on programming education research awhile
| ago. Said education outcomes improved _considerably_ if you
| constantly bombard students with multiple choice questions.
|
| Administering that through a dashboard would be essential.
| falcor84 wrote:
| > education outcomes improved considerably if you
| constantly bombard students with multiple choice questions
|
| I would love to read more about that, if you happen to have
| a link to share.
| urthor wrote:
| https://journals.plos.org/ploscompbiol/article?id=10.1371
| /jo...
| falcor84 wrote:
| Much obliged.
|
| I see the relevant part seems to be the following, in the
| "Tip 4" section:
|
| """ The key to making demonstrations more effective is to
| make learners predict the outcome of the demonstration
| before performing it. Crucially, their prediction should
| be in some way recorded or public, e.g., by a show of
| hands, by holding up cue cards marked with A, B, C, or D,
| or by talking to their neighbour. We speculate that the
| sting of being publicly wrong leads learners to pay more
| attention and to reflect on what they are learning;
| regardless of whether this hypothesis is true,
| instructors should be careful not to punish or criticise
| students who predicted wrongly but rather to use those
| incorrect predictions as a spur to further exploration
| and explanation. """
| blagie wrote:
| That part has been replicated (see e.g. vicarious
| learning, from Derek Muller as well as Miki Chi, studies
| on Khan Academy in-video questions, etc.).
|
| However, it's not the most relevant part. The most
| relevant part is simple use of active learning. The
| classic paper here is Hake (Interactive-Engagement vs.
| Traditional Methods: A Six-Thousand-Student Survey of
| Mechanics Test Data for Introductory Physics Courses).
| However, it's been replicated discipline-by-discipline
| (lots of papers in PNAS). Best summary is ICAP by Miki
| Chi. Miki didn't invent it, and ICAP isn't the most
| rigorous, but she's pretty good at these summaries.
|
| Having students do anything beyond listening leads to
| huge improvements in student learning, basically no
| matter what it is. Some things lead to bigger
| improvements than others, of course. Even clicking a
| "next" button occasionally shows a (very modest) gain.
| For good interactive engagement, doubling of learning
| gains is a good lower-bound over passive listening.
| mistrial9 wrote:
| honestly this is a nightmare'ish point of view for anything
| but hard science at senior level. The emotional environment
| of being monitored constantly is toxic. Control-oriented
| people gravitate towards constant monitoring in similar ways
| as the Executive and their Dashboard are described above. How
| can thinking and feeling humans delegate interaction in a
| learning environment to machinery ?
| blagie wrote:
| That can be true, but it's not true in this case. Eric
| Mazur's classroom is sort of a visitor sport -- people come
| from around the world to observe. I'd encourage you to do
| the same. It's very human.
|
| As originally implemented, the technique was pretty simple:
|
| - You give a traditional lecture
|
| - Every five minutes or so, students are asked a question
| to see if they understand concepts
|
| - The system peers students who answered differently to
| discuss. Students can change their answers at the end.
|
| - The instructor and class see aggregated statistics (e.g.
| 90% of people answered 27, 5% answered 54, and 5% gave
| other answers)
|
| - Based on the feedback, the instructor can do a deeper
| dive into places where there are common misconceptions
|
| Today's class is very different. Mazur juggles all sorts of
| peer learning models, 360 reviews, etc. in ways which are
| superhuman. But that's the original model.
|
| Most of the uses of formative assessment lead to classrooms
| feeling more personal, more personalized, and more human.
|
| There are control-oriented systems which scare the heck out
| of me, like GoGuardian and Securely, but this isn't it.
| There is a complex ethics question of how to build systems
| which don't dehumanize students.
___________________________________________________________________
(page generated 2022-06-14 23:01 UTC)