Skip to main content

Concurrency and Stabilty. More Zen for Python.

I was planning on writing something about the current conversations on the Python-3000 list about concurrency, which just doesn't stop being brought up and never gets resolved to any point that anyone is happy with. In an unrelated action, I went to check on my older blog, which I thought of resurrecting for some non-development articles, and I found a draft from a previous time the topic came up. I read it and was surprised to find that it proposes pretty much exactly what is being put on the table over this past weekend, with reference to walling off multiple interpreters in a single process and controlling messages past between them. The same technique scales for multiple cores, multiple processors, or multiple machines.

I've decided to take the easy way out and just post the original draft with minor editing. I enjoy how spot on I ended up being with what is currently becomming an acceptable solution, it might seem. The original was written nearly a year ago. Does this mean my predictions are worth something? Decide for yourself!

Concurrency is a hot topic on the Python mailing lists lately. There is a strong push to get some kind of native concurrency into Python, as the 3.0 branch is a great opportunity to do things that we can't otherwise do, as they would break old code. If we don't get something in now, particularly, something that can scale to hundreds of thousands of tasks and take advantage of multiple processors, we may not get a chance at what could be the best improvement of the language, until the next major version 4.0.

A large part of the problems stems from how horrible an idea threads really are, as they are typically implemented. Threads, as the basic level, are just multiple pre-emptive tasks running with access to the same memory space. Doing this can be a boost for performance, but is hell to control properly. The threads must be syncronized to access their shared resources without clobbering each other. This can be done, but it is very error prone and very difficult to debug.

The solutions seem to lean toward two ends of the spectrum: cooperative tasks and processes. With cooperative tasks, each concurrent unit runs until it says "OK, I'll let someone else run now", and so there is no explicit syncronization needed, because nothing happens without you knowing it, idealy.

Processes, on the other hand, are basically threads without shared memory. These are the same way the concept of processes are implemted at an OS level. Some, including Guido van Russom, even think that is how we should go: multiple system processes communicating via pipes and sockets.

What I propose is a process implementation with python itself. This would offer a lightweight process execution, where each process would consist of a thread of execution and an "object space". Each thread would only be able to access objects within this space. Communication would occure through channels between processes, which can be used like generators (gen.send(10), for example). I've created a basic implemenation, available on my webserver, here.

With my basic demo, you can create object spaces, each with their own global and local dictionaries. There is no real protection, but it shows how it would work and offers an idea of how it would be used if we had real protected object spaces. You can run a function in a space with the run method, which takes a resonse function first, which is called when the the function returns and will be passed the called functions return value. If a function is already executing, the request is queued until its turn.


I need to look around and find that demo/prototype code. I barely remember writing it, but I remember being pleased with the results. I'll look around and resurrect it. Perhaps it will serve as an interesting proof of concept for a possible solution to some of our concurrency problems. I wish I could put more work into a solution now, but at the moment I have little practicle use for concurrency of this type. Maybe in a while I can find justification to spend time on it.

Comments

Anonymous said…
Lightweight processes and good communication primitives the way Erlang does it are really great. Something similar for Python would be a real benefit.
Paul Boddie said…
See also the parallel/pprocess module for similar thinking:

http://www.python.org/pypi/parallel
Fazal Majid said…
I don't buy this "we can't do threads right, therefore they must be useless or evil" mentality. If you want to manage pools of Oracle connections, for instance, threading is your only viable option.

That said, having higher-level IPC facilities available than socket, spread, mmap or the like would probably help. A transparently multi-process implementation of Queue.Queue for instance.
Paddy3118 said…
'Parallel' processing should be built on top of OS processes not threads, with the OS supplying base level memory protection and inter-process communication. Python libraries should abstract away the OS provided services and possibly provide higher level functionality such as transparently running 'parallel' processes across a network of machines with different OS's.
The accent should be on ease of development and maintenance rather than raw speed of each process as parallel programming has proved difficult, and now it seems that multi-core machines are the way to get more bang-per-buck Python needs to evolve.
- Paddy.
Fredrik said…
"each process would consist of a thread of execution and an "object space""

That's the "obviously correct" solution, of course. Just post the code already!
Mystilleef said…
Yeap, looks just like how Erlang solved the concurrency problem. Doesn't Stackless Python do something similar too?

Popular posts from this blog

On Pruning Your Passions

We live in a hobby-rich world. There is no shortage of pastimes to grow a passion for. There is a shortage of one thing: time to indulge those passions. If you're someone who pours your heart into that one thing that makes your life worthwhile, that's a great deal. But, what if you've got no shortage of interests that draw your attention and you realize you will never have the time for all of them?

If I look at all the things I'd love to do with my life as a rose bush I'm tending, I realize that careful pruning is essential for the best outcome. This is a hard lesson to learn, because it can mean cutting beautiful flowers and watching the petals fall to the ground to wither. It has to be done.

I have a full time job that takes a lot of my mental energy. I have a wife and a son and family time is very important in my house. I try to read more, and I want to keep up with new developments in my career, and I'm trying to make time for simple, intentional relaxing t…

The Insidiousness of The Slow Solution

In software development, slow solutions can be worse than no progress at all. I'll even say its usually worse and if you find yourself making slow progress on a problem, consider stopping while you're a head.

Its easy to see why fast progress is better: either you solve the problem or you prove a proposed solution wrong and find a better one. Even a total standstill in pushing forward on a task or a bug or a request can force you to seek out new information or a second opinion.

Slow solutions, on the other hand, is kind of sneaky. Its insidious. Slow solution is related the Sunk Cost Fallacy, but maybe worse. Slow solutions have you constantly dripping more of your time, energy, and hope into a path that's still unproven, constantly digging a hole. Slow solutions are deceptive, because they still do offer real progress. It is hard to justify abandoning it or trying another route, because it is "working", technically.

We tend to romanticize the late night hacking…

Finding "One Game A Month"

I was really excited about the One Game A Month challenge as soon as I heard about it.
For about two years I've struggled in fits and starts to make my way into game development. This hasn't been productive in any of the ways I hoped when I started. Its really difficult to be fairly experienced as a developer, which I believe I am in my day job as a web developer, while struggling really hard at an area in which your experience just doesn't exist.
Its like being a pilot who doesn't know how to drive.

But this challenge provided a new breath to this little hobby of mine. It gave me a scaffolding to experiment, to learn, to reflect on finished projects. I had spent far too much time on game projects that stretched on far past their exciting phases, bogged down by bad decisions and regret.
And it has worked.
I have a lot to learn. I have a lot of experience to gain through trial and error and mistake and discovery. I have a lot of fun to be had making more small games t…