[HN Gopher] JupyterLab 4.0
___________________________________________________________________
JupyterLab 4.0
Author : jonbaer
Score : 190 points
Date : 2023-06-13 19:39 UTC (3 hours ago)
(HTM) web link (blog.jupyter.org)
(TXT) w3m dump (blog.jupyter.org)
| impulser_ wrote:
| Is there a similar notebook application like
| https://livebook.dev/ for Python?
|
| I like Jupyter, but after trying out Livebook with Elixir I wish
| there was something similar in Python.
|
| Smart cells and Toggling parts of code on/off is extremely useful
| features in a Notebook app.
| folli wrote:
| Can somebody give me a quick rundown on the difference between
| JupyterLab and JupyterNotebook? Whose the target audience for
| each one?
| modriano wrote:
| The Jupyter Notebook interface/server just opens singular
| notebooks (i.e. one notebook per browser tab).
|
| Jupyter Lab is an IDE where you can open notebooks, files,
| terminals, etc all in one interface. Additionally, you can
| easily adjust the layout of open files (want notebooks side by
| side? Click one notebook's tab and drag it to one side of the
| screen. Want a terminal on the bottom of the screen? Open a
| terminal and drag its tab to the bottom of the screen. Want 3+
| notebooks side by side? Click and drag. Etc).
|
| I basically only use the Jupyter Lab interface when I'm working
| with notebooks (sorry about using the term notebooks so much,
| it's a synonym for a .ipynb file as well as the name of a
| server mode Jupyter offers).
| nashashmi wrote:
| [flagged]
| anigbrowl wrote:
| Please stop doing this. If you have personal experience to
| share that is valuable. If you are just cutting and pasting
| from an AI, anyone on HN can do that. It is no more helpful
| than copying a list of results from a search engine. Your HN
| contributions should be written by you, not a computer.
| nashashmi wrote:
| That definitely sounds sensible. I had the same question
| and figured I ask AI. It gave a pretty good answer. So I
| shared it.
|
| I didn't think it would get a backlash. But agreed, the
| answers are too verbose and polite for an HN comment and do
| not fit with the culture here.
| mikepurvis wrote:
| I keep experimenting with Jupyter in the context of
| telemetry/fault analysis and then hitting a wall with it where:
|
| - I get an analysis that I like, but there isn't a good way to
| share it with others, so I end up just taking screenshots.
|
| - There isn't a good way to take the same analysis and plug new
| data into it, other than to copy-paste the entire notebook.
|
| - The process to "promote" fragments of a notebook into being
| reusable functions seemed very high-friction: basically you're
| rewriting it as a normal Python package and then adding that to
| Jupyter's environment.
|
| - There aren't good boundaries between Jupyter's own Python
| environment, and that of your notebooks-- if you have a
| dependency which conflicts with one of Jupyter's dependencies,
| then good luck.
|
| Some or all of these may be wrong or out of date-- Jupyter
| definitely passes the "oooh nifty" smell test, and I'd love to
| figure out how to make it usable for these longer-term workflows
| rather than just banging out one-off toys.
| smith-kyle wrote:
| What do you use to collect telemetry data?
| izzymiller wrote:
| I'm always hesitant to self-promote in an hn comment (actually
| I have never done it before!), but your problems w/ Jupyter are
| just too closely mapped to what Hex (https://hex.tech/) solves
| to not plug it here!
|
| _- I get an analysis that I like, but there isn 't a good way
| to share it with others, so I end up just taking screenshots._
|
| You can publish any Hex notebook with literally just a few
| clicks, and anyone you share it with can access it, or edit it,
| or fork it, without installing anything-- or you can even make
| it public. You can easily turn a notebook into an "app" or
| interactive report if you want, hiding/showing certain cells or
| choosing cells to show only code/only output. You can just
| share the raw notebook though too.
|
| _- There isn 't a good way to take the same analysis and plug
| new data into it, other than to copy-paste the entire
| notebook._
|
| Super easy to duplicate a Hex project and hit a different table
| or data source, or you can use input parameters (like
| ipywidgets) to make one notebook parameterized and work on a
| bunch of different data sources.
|
| _- The process to "promote" fragments of a notebook into being
| reusable functions seemed very high-friction: basically you're
| rewriting it as a normal Python package and then adding that to
| Jupyter's environment._
|
| You can promote any part of a project to a "Component" (docs:
| https://learn.hex.tech/docs/develop-logic/components) that you
| can import into other projects. They can be data sources,
| function definitions, anything. If you make upstream changes to
| the component, you can sync them down into projects that import
| it.
|
| _- There aren 't good boundaries between Jupyter's own Python
| environment, and that of your notebooks-- if you have a
| dependency which conflicts with one of Jupyter's dependencies,
| then good luck._
|
| Hex has a ton of default packages in its already installed
| standard library, and all the dependencies are ironed out-- if
| you have packages you want to use that aren't there, you can
| pip install them, pull them in from a private github repo, or
| ask us to add them to the base image. You can also run Hex
| projects using a custom-provided docker image if you have super
| custom needs.
|
| You should *definitely* check it out if you have these pain
| points. Here's an example of a pretty complicated public Hex
| project: https://app.hex.tech/hex-
| public/app/9b882bc1-ead3-4f0b-87d1-...
|
| And here's a simpler one I just made the other day on a cool
| Silk Road dataset https://app.hex.tech/hex-
| public/app/cdc1b8fe-144b-4a74-a5ef-.... There's a bunch more
| examples at https://hex.tech/use-cases. Happy to answer any
| questions!
| sails wrote:
| I was going to post hex as an ideal option, I definitely do
| not miss dealing with jupyter and all the Python env related
| headaches, hex solves all of these and then the rest of the
| more UX related issues with notebooks. Definitely recommend
| as the best compromise for notebook based Python data
| analytics tasks
| Twirrim wrote:
| > - I get an analysis that I like, but there isn't a good way
| to share it with others, so I end up just taking screenshots.
|
| Jupyter supports printing out as a PDF, an all-in-one-webpage
| and all sorts of other formats. I've used this and the support
| for Markdown to create a number of documents that I've
| presented up the leadership chain, or to other engineers across
| the org "Here's the data, here's the narrative, and here's the
| code that shows you how I got to it so you can reproduce". Can
| even share the Jupyter notebook itself.
| fluidcruft wrote:
| Do you have a way to store data directly into the notebook? I
| do a bunch of device testing and I have master notebooks
| setup to analyze and condense raw data. I use papermill to
| evaluate the master notebook to generate a report. But I want
| to also store/attach intermediates (say a named numpy array)
| into the notebook for further analysis.
| fn-mote wrote:
| > I want to also store/attach intermediates (say a named
| numpy array) into the notebook for further analysis.
|
| Is the issue that you do not want to save the data and
| report into a folder and distribute that? That is, you want
| an entirely self-contained notebook? Or is there something
| else going on here?
|
| I'm sure that's possible but it seems kind of wrong to put
| your binary "data" in with your analysis and presentation-
| making code.
| snicker7 wrote:
| Have you considered using Pluto notebooks instead? They are
| designed to be reproducible (including version references to
| transitive dependencies). They are also reactive, so it is
| trivial to change the dataset.
| spywaregorilla wrote:
| > The process to "promote" fragments of a notebook into being
| reusable functions seemed very high-friction: basically you're
| rewriting it as a normal Python package and then adding that to
| Jupyter's environment.
|
| Don't get in that situation to begin with. Pop an `%load_ext`
| `%autoreload 2` at the top and just write functions in an
| imported .py file from the get go.
| fluidcruft wrote:
| One thing that I keep running into myself is I want to include
| data in a notebook as a sort of report or record of an
| analysis. I really like papermill for creating notebooks that
| execute and then store results. But you can only include text
| output or plots. If I wanted to store a numpy array within the
| notebook for inspection or input into.a next step, there
| doesn't seem to be a way. I understand it could be difficult to
| figure out how to make the all of that work. But I don't know.
| Maybe pickling or something should work?
| kzrdude wrote:
| Ploomber does this kind of proper jupyter notebook pipeline.
| I've used it, but not stuck with it (yet)
| proto-n wrote:
| At least for the last point I use conda with the
| nb_conda/nb_conda_kernels packages, which makes you able to
| switch kernels to different environments.
| ptero wrote:
| For a while I viewed Jupyter as a toy that is neither here nor
| there (sitting between the chairs of development and
| explanation, briefing or visualization and not doing either job
| great). But about 2 years ago when I changed jobs into a
| "Jupyter heavy" environment I was forced to learn it and have
| grown to really like it.
|
| I primarily use Jupyter for prototyping: trying ideas, plotting
| results and sharing notebooks for others to improve on (or
| punch holes in). Once a piece clearly shows promise I move it
| into a module. While this usually means a significant rewrite I
| actually see it as a benefit: I can be messy in original
| prototyping plus a rewrite after experimentation often leads to
| a better code with a small time investment.
|
| Beyond the minimal self-discipline of actually moving code into
| modules I still have two three-character friction points: vim
| and git. Pointers on addressing those appreciated!
|
| I would _love, love, love_ a tight vim and Jupyter integration
| to be able to switch, easily and frequently, between editing a
| set of cells in vim and in Jupyter with solid sync between
| them. I am perfectly OK with vim ignoring the output. And I
| would love to have a git mode that only checks in the changes
| that cause actual differences in the python code; not
| timestamps or the output.
| geysersam wrote:
| This is a plugin that makes browser text area elements into
| nvim buffers.
|
| https://github.com/glacambre/firenvim
|
| Haven't tested it in combination with Jupyter but I imagine
| it should work
| leksak wrote:
| jupytext will solve the git problems for you. Pair with a
| percent script, ignore the ipynb files.
| kzrdude wrote:
| jupytext is great. It even allows you to use only .py files
| directly as notebooks, but I recommend "pair with ipynb"
| and version controlling the .py file. The ipynb acts like a
| cache of the cell outputs between invocations of
| jupyterlab, which is handy too.
| slt2021 wrote:
| I always wanted a Jupyter like environment, but the one
| that would natively support and output .py files
|
| jupyter notebook as IDE, but with .py files instead of
| .ipynb
| bjornasm wrote:
| >- There isn't a good way to take the same analysis and plug
| new data into it, other than to copy-paste the entire notebook.
|
| Cant you just set the parameters in the first cell?
|
| >- I get an analysis that I like, but there isn't a good way to
| share it with others, so I end up just taking screenshots.
|
| Export to md, pdf or html?
| Equiet wrote:
| We looked into many of these issues with Deepnote (YC S19)
| [https://deepnote.com/]. What we found is that these are not
| necessarily problems of the underlying medium (a notebook), but
| more of the specific implementation (Jupyter). We've seen a lot
| of progress in the Jupyter ecosystem, but unfortunately almost
| none in the areas you mentioned.
| singhrac wrote:
| I manage a Jupyter instance where we do ~75% of all of our
| work, and before that used Jupyter daily as a quant. In
| response to your comments + sibling's:
|
| Sharing: there hasn't been a good solution, so you have to hack
| something together. For us (small team) symlinking a
| "shared_notebooks" network path works fine (but looking forward
| to the RTC mentioned in the post). It's very important to
| restart your notebook before sharing.
|
| New data: I've never really run into this in a form that
| "restart + run all cells" didn't really work. Are you trying to
| keep older versions?
|
| Reusability: I highly recommend installing a personal Python
| package in development mode to your kernel (i.e. pip install -e
| .). Then just move functions from the notebook to the package
| and reload.
|
| Boundaries: As others have said, I would just make a "user"
| kernel right away and never touch the Jupyter environment (as
| TLJH does by default).
|
| I am kind of hopeful that Armin Ronacher's rye fixes Python
| environment hell and we end up with better solutions for many
| of these things, but there are definitely some issues with
| version management in Jupyter.
| bityard wrote:
| > There aren't good boundaries between Jupyter's own Python
| environment, and that of your notebooks-- if you have a
| dependency which conflicts with one of Jupyter's dependencies,
| then good luck.
|
| It's cumbersome, and I'm not totally sure it's the correct way,
| but I remember getting around this by creating a virtualenv for
| my projects and then using that virtualenv's python as
| Jupyter's "kernel".
| modriano wrote:
| Yeah, that's essentially the right way (I use conda envs, but
| venvs are probably fine too). Have one env for running the
| Jupyter Lab/Notebook server (install jupyter, jupyter
| widgets, nbextensions, etc into this env) and then create
| other envs for your projects and register that kernel with
| jupyter [0]. You can also just install all of the jupyter ide
| cruft in each env if you want, but dependencies can get
| annoying.
|
| [0] https://stackoverflow.com/questions/39604271/conda-
| environme....
| garbagecoder wrote:
| I just do a venv inside the venv so I can target several
| Python versions. I also use Jupiter for C#, Clojure, and
| Julia. I wish the Clojure integration was as good as the
| others I listed.
|
| I'm to the point now where if anything other than venv/pip is
| required I won't use it. Unfortunately there are many things
| that insist on conda.
| anvuong wrote:
| This is the way. I use miniconda to create env for Jupyter
| and install only Jupyter and its dependency on that, then I
| just config it to point to all the other Python envs I use
| for my different projects. This is only one-time setup and it
| is absolutely worth it given how messy the ecosystem is.
| awinter-py wrote:
| had a ton of trouble getting jupyterlab to target an
| arbitrary virtualenv
| kzrdude wrote:
| Export to HTML is usually a decent report. If you want to
| refine it a bit, you can export and show only outputs, or (with
| some code) only certain tagged outputs.
|
| To plug in new data, I experiment with papermill.
|
| There are solutions that can put jupyter and the kernel in
| different environments. It's even not too hard to setup, but it
| astounds me it's not a common way to set it up. It's natural to
| have kernel + the analysis's deps in its own environment.
| nigma wrote:
| I'm using Jupyter Book (https://jupyterbook.org/) to build all
| kinds of daily reports and it works really well. Individual
| notebooks can also be converted to HTML or pdf with nbconvert
| using command line or "download as" menu item.
| cadr wrote:
| +1 for Jupyter Book. I had to fight with it a bit for some
| things around LaTeX/PDF generation, but was very pleased with
| the result.
| riekus wrote:
| Still looking for a nice setup with JS. Tried some notebook
| promises with danfoJS but couldn't find something ergonomic
| rollinDyno wrote:
| I have been using VSCode notebooks with .ipynb file extensions,
| this gives me many advantages as I am able to configure things
| I'm not able in JupyterLab. I also have access to a very rich
| ecosystem of plugins. If there is anyone aware of VSCode as a
| solution but keeps using JupyterLab, could they explain why?
| specproc wrote:
| I've not been back to full-fledged Jupyter since getting in to
| VSCode.
|
| Most of my analytical work now is done in .py files, broken up
| into blocks with `#%%`. Real notebooks feel really clunky since
| adopting the approach.
| haberman wrote:
| Does `#%%` have special meaning to the tooling? Are you able
| to view the output as a pseudo-notebook?
| fumeux_fume wrote:
| For me it's a cleaner UI for experimentation and when you run a
| cell it doesn't jump to weird places depending on the output.
| I've been using jupyter for so long that I find it very
| familiar and easy to use. With that said, VSCode has improved
| so much for notebooks that I'll usually reach for it over
| jupyter.
| petemir wrote:
| I would love to use vs code for notebooks, but I just cannot
| get the interactive console to work as I would like. Right know
| I don't exactly recall the problem, but it had to do with the
| keyboard shortcuts and running the piece of code in the
| interactive console. For some reason other shortcuts took
| precedence, even if I disabled them, so I couldn't get my code
| to reliably run on the console.
| babl-yc wrote:
| VSCode Jupyter notebooks + Github Copilot is my favorite way to
| interact with notebooks. The autocomplete is super helpful for
| assisting with discovery of matplotlib or numpy operations.
| anvuong wrote:
| Maybe because the Jupyter is setup on the computing server,
| where everyone needs to log in to do their works? This was the
| case for my last 4 companies.
| arijun wrote:
| I use VSCode (with the remote extension) for the situation
| you're describing, and find it works extremely well. There
| are a couple little pain points, but none related to the
| remote part of the equation.
|
| Big positives are how it integrates with the rest of the IDE
| so go to definition, debug cell, and data explorer just work.
|
| Some negatives are a possibly onerous setup if not already
| using VSCode as your IDE (to get some of the IDE-like stuff
| to work), and how there isn't exact parity on hot keys so
| muscle memory fails you occasionally.
| rollinDyno wrote:
| I see. If I faced the same constraint I would try to find
| whether VSCode can access files remotely, which is very
| likely.
| arijun wrote:
| The VSCode remote access is IMO better than just accessing
| files remotely; it's s a large part of why I use it instead
| of Pycharm.
|
| It splits the editor into a UI that is run locally, and a
| server that does the heavy lifting on the remote machine.
| Conceptually it's very similar to Jupiter, where you have a
| user facing front end with the UI run on JavaScript and
| rendered by your browser, and a python kernel backend, and
| the two communicate over pipes that can be run over the
| internet.
|
| What it effectively means for VSCode is that you get a more
| seamless experience than I experienced with Pycharm remote
| development.
| garbagecoder wrote:
| Only because I can use it anywhere that way, including on
| machines that I am not allowed to install things on. But if it
| works in vscode.dev then I am out of excuses, because you're
| correct, it's much much better in vscode.
| bramathon wrote:
| I have tried VSCode many times, but I find the performance of
| the notebook UI to be terrible. It seems to be popular, so
| maybe it's just me.
|
| In any case, jupyterlab + jupyter-lsp gets most of the benefits
| for me.
| wenc wrote:
| Would be curious how long ago you tried this and what your
| setup is? The lowest spec machine I have is a Windows 10
| machine with an i7 and 8 GB RAM and it is super responsive on
| the latest version of VS Code.
| wenc wrote:
| I too use VS Code as my Jupyter platform (running remotely on a
| powerful EC2 instance with 32 CPUs and 256GB RAM -- my own
| desktop is a 7 year old Intel core i7 with 8GB RAM). VS Code's
| Remote extension is amazing, works over any SSH host and
| seamlessly blends local and remote. It's also fast since the UI
| is local while the filesystem and execution is remote.
|
| The experience is a lot better than JupyterLab (which I am
| forced to use from time to time on SageMaker). The VS Code UI
| is cleaner plus I get a full language server which means I can
| rename variables and refactor fearlessly.
|
| I also get full access to VS Code plugins.
| digdugdirk wrote:
| Really interesting setup. What kind of monthly expense does
| this run?
|
| And on a separate but related note, does it change the way
| you think about how you spend your time coding? (Assuming the
| costs do ramp up with usage such that time literally does
| equal money?)
| kachnuv_ocasek wrote:
| I'm curious about your setup as I might have to do something
| similar soon due to my machine's performance constraints. I
| understand you can connect to a Jupyter server remotely, but
| how do you sync your code? Do you have your Git repo cloned
| on the remote host and just run Git commands over SSH? Or
| does VS Code have some kind of integration for remote file
| systems with version control?
| jonnycomputer wrote:
| Anyone have experience with JetBrain's Data Spell product?
| spywaregorilla wrote:
| Anyone got experience with the real time collaboration and have
| feedback?
| sireat wrote:
| I've been using Jupyter Notebooks for 7+ years but keep failing
| to find a use case for JupyterLab.
|
| JupyterLab feels like a clunky web based IDE. I check it every
| year or so and go back to Notebooks.
|
| I used to and still run a Littlest Jupyter Hub:
| https://tljh.jupyter.org/en/latest/ for my org.
|
| I keep thinking whether migrating to full blown JupyterLab is
| worth the pain.
|
| With the improvements that Visual Studio Code has made in ipynb
| support there is even less reason these days.
|
| The biggest thing keeping me on VS Code of course is full blown
| Copilot support. Whenever I have to fall back to Colab I feel
| 2-3x less productive.
|
| My workflow is:
|
| * Notebook for exploration/fiddling around 90% of the time is
| spent here - keeping state open is so convenient
|
| * Extract/export code to regular .py for production
| kzrdude wrote:
| Anyone know about compatibility with existing notebooks?
| Presumably stellar, right?
|
| For now, I need jupytext (which is a great extension), so then I
| stay with v3.6 until jupytext is resolved.
|
| For this announcement - or in the changelog - it would be great
| to know why it is a major version bump - what is the major
| compatibility change?
| sva_ wrote:
| I would think of JupyterLab as more of an IDE, and the major
| version bump need not affect their notebook format.
|
| I think if they made breaking changes to the notebook format,
| they'd mention this right away.
| smith-kyle wrote:
| See https://github.com/jupyter/nbformat
| abdullahkhalids wrote:
| This is only a change in the interface. I don't think there is
| any underlying change in the notebook format. Your old
| notebooks are completely safe.
|
| P.S. Juptext is a pretty cool extension
| https://pypi.org/project/jupytext/
| smith-kyle wrote:
| nbformat is reference implementation of the notebook format:
| https://github.com/jupyter/nbformat No recent major releases.
|
| jupytext is great, you can also review your notebooks as
| `.ipynb` files using GitNotebooks [https://gitnotebooks.com].
| (I'm the solo dev)
| fancymcpoopoo wrote:
| I heard this is good for data analysis. But hasn't enough data
| been analyzed already?
| ModernMech wrote:
| The analyses are just more data to analyze.
| anigbrowl wrote:
| Comedy is not your forte.
| abdullahkhalids wrote:
| I see no mention of an improved debugger. IMHO, the atrocious
| debugger in JupyterLab is one of the primary reasons why it is
| difficult to write good code in this environment.
|
| Even a simple improvement like remembering the sizes of various
| subpanels in the debugger sidebar will make me feel like I am not
| pulling teeth when I use it.
|
| And don't get me started on inspecting the value of variables. If
| you are looking for more than an object within an object, you
| might as well go back to print statements.
| esafak wrote:
| Also it has a poor multi-user experience.
| mkl wrote:
| Do you know how it compares to using Visual Studio Code's
| debugger with notebooks? I'm wanting to find a notebook
| debugger recommendation for my students.
| aidos wrote:
| Is this in the context of students learning to code, or
| something else where they need to use the notebooks to
| supplement their study?
| Xeamek wrote:
| Have you tried pyCharm? I believe even community (free)
| edition should have support You are looking for
| mkl wrote:
| No I haven't, but it looks like the community edition can't
| do notebooks:
| https://www.jetbrains.com/help/pycharm/jupyter-notebook-
| supp...
| Xeamek wrote:
| Ah, my bad then.
|
| If it's for student, you could maybe get educational
| license, that is completely free for all their products,
| but I imagine it could be quite a hassle
| lozenge wrote:
| The debugging engine is the same. They both use debugpy 'via'
| the Jupyter wire connection.
| animal_spirits wrote:
| The VSCode debugger works amazing with notebooks. I find it
| super useful all the time when I develop. Just install the
| Jupyter extensions and it should work
| sdenton4 wrote:
| Eh, print/log debugging works fine. Especially in an
| interactive environment: you've got direct access to the
| variables and objects, and can easily inspect them directly.
|
| At some point I felt like I was a bad dev for not using a
| debugger, but at this point I think I'm more versatile since
| I'm less dependent on finicky tooling to figure out what some
| code is doing... Every language has it's own debugger to learn,
| but logging (and good strategies for logging) works about the
| same everywhere.
___________________________________________________________________
(page generated 2023-06-13 23:00 UTC)