Post ASdlLI0LqaSneo88FU by ali1234@mastodon.social
(DIR) More posts by ali1234@mastodon.social
(DIR) Post #ASc7HMI5mLqUT32UZE by penguin42@mastodon.org.uk
2023-02-12T20:02:09Z
0 likes, 0 repeats
Got very confused over some threading behaviour until someone pointed out Python doesn't really do threads and it's really processes - no wonder it's not sharing anything! (and OpenCL gets a little freaked when 32 clients suddenly startup in parallel)
(DIR) Post #AScE6IPAaMA0uS1P7o by edavies@functional.cafe
2023-02-12T21:18:33Z
0 likes, 0 repeats
@penguin42 Either our terminologies or our understandings of Python threading are not aligned.threading.Thread threads within one operating-system process share the same memory, file descriptors, etc. There is (or was until very recently, I'm not quite clear on this) a global interpreter lock which prevents two threads within the same OS process from being interpreted at the same time so multiple Python threads won't make use of multiple cores But in no way does that make them more like processes than threads; quite the opposite actually.
(DIR) Post #ASdMHRLwePmwdcIYCW by penguin42@mastodon.org.uk
2023-02-13T10:24:57Z
0 likes, 0 repeats
@edavies I didn't have any understanding of Python threading! I was getting confused why the Python script I was changing wasn't remembering variable values in classes I was changing; and only then discovered it was really spawning multiple independent python processes.
(DIR) Post #ASdMNGTC1j5OcdSw7M by edavies@functional.cafe
2023-02-13T10:25:59Z
0 likes, 0 repeats
@penguin42 Ah, right, so nothing to do with Python threading?
(DIR) Post #ASdMRgFW7ax7jZAsqG by penguin42@mastodon.org.uk
2023-02-13T10:26:49Z
0 likes, 0 repeats
@edavies Well, I'd thought the script used threads in python - which is what confused me!
(DIR) Post #ASdMxc5N8ZJq5TejvE by edavies@functional.cafe
2023-02-13T10:32:33Z
0 likes, 0 repeats
@penguin42 Python threading is actually quite nice for some use cases. I use it a lot to implement a sort of actor model - lots of objects with a single internal thread to avoid concurrent access to their internal state with a Queue for input messages from other objects.Well worth a look if you're ever dealing with such things. Modules threading and queue.
(DIR) Post #ASdfmf1HmoJV8BEfiq by ali1234@mastodon.social
2023-02-13T13:41:51Z
0 likes, 0 repeats
@edavies @penguin42 The script in question uses multiprocessing to implement massively parallel calculation. The confusion is because multiprocessing tries to act like a thread library but really it doesn't use threads at all because of the GIL performance penalty.
(DIR) Post #ASdfmfzYAkWG95Qnei by penguin42@mastodon.org.uk
2023-02-13T14:03:29Z
0 likes, 0 repeats
@ali1234 @edavies Yeh it's just my Python nodiness took me ages to work out what was going on!
(DIR) Post #ASdlLI0LqaSneo88FU by ali1234@mastodon.social
2023-02-13T15:05:45Z
0 likes, 0 repeats
@penguin42 @edavies The confusion isn't helped by me naming the option `--threads`.There's also another gotcha where you can only have one CUDA context per process and the library remembers which thread you created it from and won't let you use it from any others.