[HN Gopher] Tqdm (Python)
___________________________________________________________________
Tqdm (Python)
Author : manjana
Score : 298 points
Date : 2021-12-16 19:10 UTC (3 hours ago)
(HTM) web link (tqdm.github.io)
(TXT) w3m dump (tqdm.github.io)
| perrygeo wrote:
| Check out Will Chrichton's talk "Type-Driven API Design in Rust"
| where he live-codes a tqdm-like progress bar for Rust iterators.
| Solid presentation and eye-opening how straightforward it was to
| extend the core language through traits.
| MapleWalnut wrote:
| I wish there was something this good for Golang.
| dud3z wrote:
| While not having all the tqdm features, i find mpb to be quite
| good actually.
|
| https://github.com/vbauerster/mpb
| emehex wrote:
| chime [1] is another python package that fans of tqdm might like.
|
| [1] https://github.com/MaxHalford/chime/
| BiteCode_dev wrote:
| I really love the wave of new tools that are gaining traction in
| the Python world: tqdm, rich (and soon textual), fastpi, typer,
| pydantic, shiv, toga, doit, diskcache...
|
| With the better error messages in 3.10 and 11, plus the focus on
| speed, it's a fantastic era for the language and it's a ton of
| fun.
|
| I didn't expect to find back the feeling of "too-good-to-be-true"
| I had when starting with 2.4, and yet.
|
| In fact, being a dev is kinda awesome right now, no matter where
| you look. JS and PHP are getting more ergonomics, you get a load
| of new low level languages to sharpen your hardware, Java is
| modern now, C# runs on Unix, the Rust and Go communities are busy
| with shipping fantastic tools (ripgrep, fdfind, docker, cue,
| etc), windows has a decent terminal and WSL, my mother is
| actually using Linux and Apple came up with the M1. IDEs and
| browsers are incredible, and they do eat a lot of space, but I
| have a 32 Go RAM + 1TO SSD laptop that's as slim as a sheet of
| paper.
|
| Not to mention there is a lot of money to be made.
|
| I know it's trendy right now to say everything is bad in IT, but
| I disagree, overall, we have it SO good.
| axg11 wrote:
| Your positivity and enthusiasm is infectious. Thanks for being
| an optimist.
| karmasimida wrote:
| tqdm is like around for 5 years or something now
| rbdixon wrote:
| doit [0] is a superb toolkit for building task-oriented tools
| with dependencies. It isn't too complex to get started and
| gives you a lot to work with. I've used it for an internal tool
| used by everyone in the company for 5+ years and it has never
| given me a headache.
|
| [0]: https://pydoit.org
| a-dub wrote:
| or if you prefer rust, there's just.
| https://github.com/casey/just/
|
| better yet, use both and _just doit_
| mayli wrote:
| The pythontic replacement for makefiles?
| BiteCode_dev wrote:
| And surprisingly underrated.
|
| I mean, it's declarative, works on Windows, easy things are
| (very) easy, and because you can mix and match bash and
| python actions, hard things are suspiciously easy too.
|
| Given how complicated the alternatives are (maeven, ninja,
| make, gulp...), you'd think it would have taken the world for
| years.
|
| Yet I've only started to see people in the Python core dev
| team use it this year. It's only getting traction now.
| genidoi wrote:
| "hard things are suspiciously easy too."
|
| Care to share an example?
| BiteCode_dev wrote:
| Here is a task that groups all static files from a web
| project into a directory.
|
| It has to make sure a bunch of directories exist, run a
| node js command to build some files from the proper dir,
| then run a python command to regroup them + all static
| files from all django apps into one dir. Simple.
|
| But then I had a problem I had to hack around. This
| required me to change an entry to a generated TOML file
| on the fly at every build.
|
| doit just lets me add a 5 lines python function that does
| whatever I want, and insert it between my bash tasks, and
| I'm done. def
| task_bundle_static_files(): def
| update_manifest(): conf =
| Path("var/static/manifest.toml") data =
| toml.loads(conf.read_text())
| data["./src/main.js"] = data["index.html"]
| conf.write_text(toml.dumps(data)) return
| { "actions": [ "mkdir
| -p ./var/static/", "rm -fr
| ./var/static/* ", "cd frontend/; npm
| run build", "python manage.py
| collectstatic --noinput",
| update_manifest, "cp -r
| ./var/static/* ", ] }
| systemvoltage wrote:
| Despite of all the hate Python gets (pkg system), it's still my
| go to language. Most fun. Most rewarding. Getting things done
| fast.
|
| I can go home with Python at 5pm. And have a good time with my
| family.
| BiteCode_dev wrote:
| Right?
|
| Doing advent of code in Python is almost cheating. It feels
| like playing a video game.
| smilekzs wrote:
| For me at least, Poetry [1] automates the process almost on
| par with npm, cargo, etc.
|
| [1]: https://python-poetry.org/
| evilduck wrote:
| I really like Poetry for my own uses but as a semi-
| infrequent Python user my struggle is dealing with other
| people's Python repositories that aren't using it.
|
| I can never remember all the differences and subtleties
| between virtualenv, pipenv, venv, pyenv-virtualenv, workon,
| conda, and so on when I encounter them in a random git
| repo.
| tedivm wrote:
| I hate python, I just hate everything else more.
|
| Jokes aside it's a great language. I would love to see a
| better package management ecosystem for it, as that is also
| my biggest issue. Python also does something no other
| mainstream scripted language does- it allows you to install
| extensions to the language right in the same package manager,
| as it can compile libraries from source when wheels are
| unavailable- this makes it a much harder challenge. At the
| same time I'm really happy that PyPI is a non-profit
| organization and won't have to go through the issues that
| something like NPM did.
| systemvoltage wrote:
| I love Python (flexibility, ecosystem, datamodel). I love
| Java (verbosity, explicitness, robustness). Now, kill me. I
| don't let others define what I should and shouldn't think
| about a tool, in this case a programming language. I make
| up my own mind based on my own experience with the tools.
| Remember, they're just tools. Not the end game.
| trutannus wrote:
| TQDM isn't all that new, it's one of the first things I started
| to play with when learning to write code. That said, it is an
| amazing tool.
| dheera wrote:
| Yeah, I've been using it for a while, but with the barrage of
| memes and politics on the internet nobody finds out about the
| good stuff until years later.
| BiteCode_dev wrote:
| tqdm is indeed 8 years old, but it's been popping up
| everywhere for only a couple of years now. Hence the "gaining
| traction".
| underdeserver wrote:
| I used an early version of this in 2009.
| yairchu wrote:
| Ive used tqdm 20 years ago
| BiteCode_dev wrote:
| I've used tqdm 30 years ago!
|
| Although the first release on pypi was in 2013:
| https://pypi.org/project/tqdm/#history
| genidoi wrote:
| New features get added and these posts are a nice reminder to
| check the docs on a tool I literally never need to do that
| for because it's so habitual to wrap any long-running
| iterable inside of a `tqdm()`.
|
| TIL tqdm now has a dead simple telegram hook
| https://tqdm.github.io/docs/contrib.telegram/
| iooi wrote:
| tqdm has been around for ages now, it's not exactly new.
| BiteCode_dev wrote:
| tqdm is indeed 8 years old, but it's been popping up
| everywhere for only a couple of years now. Hence the "gaining
| traction".
| waterproof wrote:
| You said "wave of new tools that are gaining traction"
| maybe you meant to say "wave of tools newly gaining
| traction".
| BiteCode_dev wrote:
| A new wave "of tools gaining traction"
| dogline wrote:
| I had to look these up (just me doing Google searches):
|
| - tqdm: progress bars (https://tqdm.github.io/)
|
| - rich: text formatting (https://github.com/willmcgugan/rich)
|
| - textual: TUI, using rich
| (https://github.com/willmcgugan/textual)
|
| - fastpi: Rest APIs (https://fastapi.tiangolo.com/)
|
| - typer: CLI Library, uses Click (https://typer.tiangolo.com/)
|
| - pydantic: Custom data types (https://pydantic-
| docs.helpmanual.io/)
|
| - shiv: Create Python zipapps
| (https://shiv.readthedocs.io/en/latest/)
|
| - toga: GUI Toolkit (https://toga.readthedocs.io/en/latest/)
|
| - doit: Task runner (https://pydoit.org/)
|
| - diskcache: A disk cache
| (https://github.com/grantjenks/python-diskcache/)
| debo_ wrote:
| I just had the pleasant experience of importing tqdm into a
| script, checking HN, and seeing tqdm here :)
| djstein wrote:
| this looks like a fantastic tool! would love to use this in some
| Python CLI's to help users understand the progress
| jasonpeacock wrote:
| Might as well toss out Plumbum:
| https://plumbum.readthedocs.io/en/latest/
|
| No more bash! No more subprocess! Write shell-like code with the
| convenience of Python!
|
| Seriously, it's a great module. A bit of a learning curve, but
| then it feels natural.
| mrbonner wrote:
| wow, didn't know this existed. I've been using ipython as
| shell-ish replacement for not-so-serious-thing. I need to take
| a detail look at this from my initial glance over. thanks
| seertaak wrote:
| Plumbum is great but I actually use the bang notation in
| Jupyter even more.
|
| Actually, for me Jupyter (I use lab but I'm sure NB is
| amazing too) is the tool that boosts my productivity the
| most. (And omg vim mode.)
| andrewshadura wrote:
| Also sh.
| emehex wrote:
| `from tqdm import tqdm; [i for i in tqdm(range(10))]`
|
| ...bit of a learning curve?
| seertaak wrote:
| I think what OP is referring to is how fiddly it is to run a
| subprocess and capture the output. By the way, also super
| annoying in Java - you have to faff about with a thread (and
| you can't even really do that in python, which as much as I
| love the language, is quite pathetic) that drains stdout
| before it gets jettisoned by the OS, and in both cases long
| story short is that it's easy to write something that works
| when the output is short, but getting it to work on long
| output is an exercise in frustration. One neatly solved by
| plumbum, and so I'm definitely with OP it's use makes code
| measurably less shit.
| mrfusion wrote:
| How would it know how long a loop is ahead of time? Is this
| solving the halting problem?
| wedn3sday wrote:
| For things with a fixed length, it looks it up ahead of time
| (i.e. things that return things when you call len() on them).
| For other things such as generators, you can supply a total=N
| value when you create the progress bar. If you dont know the
| total ahead of time, it doesnt give you a percent completion,
| but still tells you thinks like number of iterations per second
| and number of iterations finished.
| geofft wrote:
| The halting problem is easy to solve most of the time. For
| instance, you can easily tell whether the following two
| programs halt: while True: pass
| while False: pass
|
| What's impossible is solving the halting problem 100% of the
| time with 100% accuracy. Most people don't need to do that.
| Solving the halting problem most of the time and then saying "I
| don't know, that's too hard" the rest of the time is of immense
| practical value, and many practical systems (the kernel's eBPF
| verifier, the thing in your browser that detects stuck pages,
| etc.) do exactly that.
|
| In this particular case, tqdm solves the halting problem in
| cases where it's easy:
| https://tqdm.github.io/docs/tqdm/#__init__
|
| > _total: int or float, optional_
|
| > _The number of expected iterations. If unspecified,
| len(iterable) is used if possible. If float( "inf") or as a
| last resort, only basic progress statistics are displayed (no
| ETA, no progressbar)._
| nostoc wrote:
| Have you ever used a for loop? You don't know how long it'll
| take, but you know how many iterations, and at any given point
| you can know how long past iterations have taken. Extrapolation
| is not hard, even if not always accurate.
| cowsandmilk wrote:
| There are plenty of for loops in which you don't know how
| many iterations you will have. With a generator, it can even
| be infinite.
| formerly_proven wrote:
| This is not how for loops work in Python, which does not have
| ranged loops.
|
| The example works because range objects implement __len__
| which is where tqdm will get it's total/count information
| from.
| ejdyksen wrote:
| If you like tqdm, it's worth checking out pqdm, a parallelized
| version. If you have embarrassingly parallel work to process in a
| script, it makes it dead simple to parallelize _and_ monitor the
| progress of something. Highly recommend:
|
| https://github.com/niedakh/pqdm
| minimaxir wrote:
| tqdm is one of the very few Python packages that makes it into
| every script I write. It's a very high ROI for managing and
| tracking simple loops.
|
| My only complaint is the smoothing parameter; by default it
| predicts the estimated time remaining based on the most recent
| updates so it can fluctuate wildly; smoothing=0 predicts based on
| the total runtime which makes more sense given law of large
| numbers.
| akdor1154 wrote:
| The older I get, and the more i come across these issues, the
| more sympathy i have for the Windows file copy progress
| dialog...
| geenew wrote:
| Yet the estimate of when you'll become fully sympathetic to
| the Windows file copy dialog changes wildly by the day, right
| :)
| bigdict wrote:
| Yes, assuming i.i.d. samples. If the first batch is "warmup
| samples", this goes out the window.
| minimaxir wrote:
| Even if it's not i.i.d, a small amount of samples with much
| higher/lower times will just average out as noise.
| winstonewert wrote:
| well, it seems to me that its quite likely that speed varies
| across the process, so the more recent updates are probably a
| more useful default, so if your first part of the process is
| slower/faster this won't permanently mess up the estimates.
| ya3r wrote:
| tqdm is really great. (Also great name BTW) It has a lot of
| features like cli, jupter notebook widget, flexibility, etc.
|
| I have been using it for 4-5 years now.
| amelius wrote:
| I think the author doesn't fully appreciate how insanely
| difficult it is to write accurate progress meters for anything
| slightly more difficult than a single for-loop.
| teddyh wrote:
| See also the shell command pv(1).
| drexlspivey wrote:
| It has no dependencies which is pretty cool
| avelis wrote:
| I always wondered how a post like this a year ago didn't go
| anywhere but this post makes the front page. Love the Python
| package. https://news.ycombinator.com/item?id=23160774
| dwater wrote:
| tqdm & click make a nice CLI pairing for taking a simple idea and
| turning into a reusable tool that you can feel good about sharing
| in minutes.
| Spivak wrote:
| I have such mixed feelings about Click; I've reached the point
| of using it so much that I now hate it. Which I think probably
| means that it's great but only as as stopgap until you learn
| how to bend argparse to your will. It's just too magic, too
| much global state, and too much spooky action at a distance.
| skrtskrt wrote:
| I actually don't think Click is too much global state and
| spookiness, at least in relation to a lot of popular Python
| libraries.
|
| I do wish I could hook into it to test better, the only thing
| you can really do right now is to have it print stuff out and
| assert the output string. It's not really necessary to just
| build a CLI with click, but I want to build a library that
| integrates with it and testing the integration is a PITA.
|
| I want to write a config-loading library for CLI apps like
| Golang's Viper lib, but for Python
| globular-toast wrote:
| I went the opposite way. Used to do everything with argparse,
| then discovered click and never looked back. While argparse
| lets you do anything, click forces you to build a good CLI.
| formerly_proven wrote:
| Click inherits the eternal argparse limitation of "you
| can't stick --verbose everywhere", doesn't it?
| pydry wrote:
| I had the same feeling. I wish there were an equivalent that
| tried to minimize the magic and global state a bit while
| still letting you make decent CLIs with a tiny amount of
| code.
| geenew wrote:
| I wasn't aware of Click, it looks promising.
|
| For reference, it's a backronym for 'Command Line Interface
| Creation Kit': https://click.palletsprojects.com/en/8.0.x/
| nxpnsv wrote:
| Click is really good, typer is great...
| abdusco wrote:
| I love Typer, it's my go-to tool for building CLIs, but I'm
| worried about its future. It's developed by a single
| developer, there are too many issues left unattended, no
| development for months, lacking a good API to extend it or
| to interact with Click.
| [deleted]
| rripken wrote:
| I use tqdm (with argparse) in a pyinstaller packaged exe.
| 5-stars - Its great! I call the exe from Java to do some ML and
| forward the tqdm progress and status messages to a Swing
| progress bar. It makes the user experience seamless. Depending
| on the task and the user's settings the tool usually takes
| about 7-8 seconds(including the PyInstaller extract) but it can
| also take up to a minute. When its 7-8 seconds the progress
| messages fly by and the tool feels snappy. When its 50-60
| seconds the users are very grateful for the progress bar.
| Meanwhile I can develop and test the tool from the command-line
| and see progress info and when I want to run the ML code in
| Jupyter notebooks the progress bars can still be made to work.
| atum47 wrote:
| Love tqdm, I used to use it on all my python scripts.
| curiousgal wrote:
| Fun fact, tqdm has an official (pre-alpha) C++ port!
|
| https://github.com/tqdm/tqdm.cpp
| 1-6 wrote:
| tqdm also works on Colab (Jupyter Notebooks) yay
| minimaxir wrote:
| To clarify, the tqdm.notebook progress bar (which is much
| prettier) works on Colab. If you want to make the script more
| agnostic, you can import tqdm.auto.
|
| A fun quirk is that tqdm.notebook didn't work with Colab's dark
| mode so the text was readable; this was very recently fixed.
| pak wrote:
| Big fan of tqdm in python, so I ported it to ruby 5 years ago:
|
| https://github.com/powerpak/tqdm-ruby
|
| (shameless plug and an invitation for pull requests)
___________________________________________________________________
(page generated 2021-12-16 23:00 UTC)