[HN Gopher] Measuring memory usage in Python: it's tricky
___________________________________________________________________
Measuring memory usage in Python: it's tricky
Author : itamarst
Score : 74 points
Date : 2021-07-05 14:52 UTC (8 hours ago)
(HTM) web link (pythonspeed.com)
(TXT) w3m dump (pythonspeed.com)
| rhn_mk1 wrote:
| > With this particular measurement, we're using 3083MB, or 3.08GB
|
| Don't mix up SI and binary prefixes, kids! The listing shows 3083
| MiB, which is either 3.01 GiB, or 3.23 GB. There's no mainstream
| interpretation giving 3.08 of something.
| kungfufrog wrote:
| Is MiB/GiB binary? And, GB/MB SI? Or the other way around?
| teddyh wrote:
| Yes, yes, and no.
| tialaramex wrote:
| It's easy to remember. The SI prefixes are always decimal,
| that's why people call it "the decimal system". Kilopascals,
| megajoules, centimetres, always powers of ten. So a megabuck
| is 10^6 bucks, 1 million bucks, and a gigabyte is 10^9 bytes,
| 1 billion bytes.
|
| The weird power-of-two prefixes get the special weird symbol
| and name, because you wouldn't want those anywhere except
| computing.
| tbabej wrote:
| For folks looking for a good modern profiling tool for Python
| codebases: my current favourite is Scalene [0], which allows
| profiling CPU/GPU and memory usage in a performant way. The
| author gave a talk about it on last PyCon US [1].
|
| [0]: https://github.com/plasma-umass/scalene [1]:
| https://www.youtube.com/watch?v=nrQPqy3YY5A
| itamarst wrote:
| Scalene is pretty neat, yes. The memory profiling, however, is
| more useful when you're trying to find memory leaks. This is a
| common scenario in web apps, say.
|
| If your goal is to find peak memory usage, which is what you
| care about in data processing batch processes, Fil is probably
| a better choice (https://pythonspeed.com/fil).
| notorandit wrote:
| With interpreters it's always difficult to track resource usage.
| It should have full blown instrumentation w/ kernel support. I am
| not sure python has it.
| dehrmann wrote:
| I recently had an adventure with Python subprocesses and memory.
| This isn't just a Python thing, but it's more of an issue because
| Python lacks proper multitreading: processes don't necessarily
| return memory to the OS after free(), and Python doesn't
| necessarily free() unused memory after a GC run. If you're using
| a process pool where some work can be memory-intensive, this can
| massively increase total memory use as the job bounces between
| subprocesses.
| ababaiem wrote:
| Good timing! I'm currently battling memory issues with a project
| where I'm straightening full resolution panoramas. I often reach
| close to my machine's limit of 16GB.
|
| I think this might help me immensely. Thank you OP.
| itamarst wrote:
| One random and possibly incorrect memory, from back when I was
| using scikit-image: switching from order=3 to order=2
| interpolation on images can reduce CPU usage and... I think
| memory usage? significantly, for only minor (but real)
| reduction in quality of output.
| [deleted]
| pikrzyszto wrote:
| Remember that a lot of software not only relies on memory that
| the developer explicitly allocated but also on buffers and
| various caches, disk cache being an example. It goes a long way -
| kernel might _refuse_ dropping disk cache when it considers it 's
| too hot (ex. db machine.)
|
| The question "how much free memory do I have" or "how much memory
| does this program use" is a very hard one, and you can't just
| read the "MemAvailable" column from /proc/meminfo or RSS/PSS/Virt
| from ps/pmap.
___________________________________________________________________
(page generated 2021-07-05 23:01 UTC)