Skip to main content

Python, Concurrency, and My Two Cents Today

This is not the first and it will not be the last that I write about the state of concurrency in Python, comment about some debate going on in the community, and outline what I think we need to solve any apparent problems and capitalize on what a lot of us think is the future of software development. Anyone following the Python blogs is bound to have caught wind of the Guido-Eckle debate about what Python 3.0 has become, compared to what it could have been. This was followed immediately by an open letter from Juergen of SnapLogic about the GIL. I feel sure this has all happened again and that all parties involved are just playing some recorded macros.

The most compelling case we have right now against the arguments to remove the GIL are two. Firstly, it was already done! A branch of Python removed the GIL many years ago and actually found a two core/cpu system would run the same code slower, due to all the locking involved to protect mutable structures. So, while people continually say that the GIL needs to be removed, gets in their way, and generally is a wart on Python, we need to remind them that its been done and it was a bad idea. The GIL is not being kept in as a product of laziness.

Secondly, threading is not the definitive answer to concurrency needs! This is a really important one, because one of the areas that I always see the Python community strive at is finding the right way to replace the popular way. Threads are very popular in a lot of circles, but there is a huge consensus that they are simply a misrepresented de facto with little in the way of justifying the use they see. The Java world, in particular, seems to think that throwing threads at a problem can solve it. What we have to realize is how many problems are caused by threading and if they outweigh the benefits. A lot of us can't see those benefits through all that cost, so we've looked in other places for concurrency, and we've found it. In some ways, the GIL acts as a deterrent to force us into finding a better way. I, for one, am all for keeping it around just for that reason.

Where can I really go with this? Not far. I could ramble and rant about processes being better concurrency primitives than threads, but I don't feel this is the time or the place. But, please, can we stop asking for the GIL to be removed? No on is going to listen to that plea. The issue is going to come up again, and that is absolutely a promise. We're going to see this again and again, until we have something solid, standard, and powerful enough to distract the thread lovers from the GIL issues. I don't know what that solution is, but we need to figure it out soon. Guido is right, of course, and this is a library issue, not a language issue. However, we can't deny what value this library issue has for the language, and a little encouragement or name dropping on his part might do well to push a good answer to the forefront. Eventually, something needs to get to the point that we can bring it into the standard library and say "This is how you do concurrency in Python."

Who wants to answer the great question? Step up.


Technorati Tags: , ,

Comments

Carlo said…
Parallel Python looks promising, as it supports both parallelism over cpus as well over clusters, and its underlying implementation uses processes. I think it just needs some better documentation and a little bit of promotion.
Jesse said…
@Calvin - you and I are in complete agreement. That's why I posted what I did asking for desired tests and benchmarks.

Removing the GIL to have "real threads" may be an option for some, but what you said is exactly the problem - threads may not be the answer.

I for one want to explore the alternatives and put up code examples/information and work towards getting something of a "concurrency" library in the standard library.

If you have suggestions for either proof-of-concepts, tests, etc I'd like to hear them. Name-dropping from Guido may help spur things, but as Guido himself has already said: the community itself needs to propose, prove out, benchmark and implement something reasonable for this.
Richard Jones said…
What is this "concurrency" you speak of? Is it doing some image processing over multiple files spread across as many CPUs as possible? Is it producing a weather forecast from a bunch of model data using all available resources? Is it a GUI application that needs to download some data in the background without affecting the responsiveness of the GUI? Or is it a game which needs to run AI logic concurrently with the rest of the program, again with as little impact on the game as possible? Does it involve share context of separated context? And finally, is it multi-platform?

Popular posts from this blog

Respect and Code Reviews

Code Reviews in a development team only function best, or possible at all, when everyone approaches them with respect. That’s something I’ve usually taken for granted because I’ve had the opportunity to work with amazing developers who shine not just in their technical skills but in their interpersonal skills on a team. That isn’t always the case, so I’m going to put into words something that often exists just in assumptions.
You have to respect your code. This is first only because the nature and intent of code reviews are to safeguard the quality of your code, so even having code reviews demonstrates a baseline of respect for that code. But, maybe not everyone on the team has the same level of respect or entered a team with existing review traditions that they aren’t acquainted with.
There can be culture shock when you enter a team that’s really heavy on code reviews, but also if you enter a team or interact with a colleague who doesn’t share that level of respect for the process or…

CARDIAC: The Cardboard Computer

I am just so excited about this.


CARDIAC. The Cardboard Computer. How cool is that? This piece of history is amazing and better than that: it is extremely accessible. This fantastic design was built in 1969 by David Hagelbarger at Bell Labs to explain what computers were to those who would otherwise have no exposure to them. Miraculously, the CARDIAC (CARDboard Interactive Aid to Computation) was able to actually function as a slow and rudimentary computer. 
One of the most fascinating aspects of this gem is that at the time of its publication the scope it was able to demonstrate was actually useful in explaining what a computer was. Could you imagine trying to explain computers today with anything close to the CARDIAC?

It had 100 memory locations and only ten instructions. The memory held signed 3-digit numbers (-999 through 999) and instructions could be encoded such that the first digit was the instruction and the second two digits were the address of memory to operate on. The only re…

How To Care If BSD, MIT, or GPL Licenses Are Used

The two recent posts about some individuals' choice of GPL versus others' preference for BSD and MIT style licensing has caused a lot of debate and response. I've seen everything as an interesting combination of very important topics being taken far too seriously and far too personally. All involved need to take a few steps back.

For the uninitiated and as a clarifier for the initiated, we're dealing with (basically) three categories of licensing when someone releases software (and/or its code):
Closed Source. Easiest to explain, because you just get nothing.GPL. If you get the software, you get the source code, you get to change it, and anything you combine it with must be under the same terms.MIT and BSD. If you get the software, you might get the source code, you get to change it, and you have no obligations about anything else you combine it with.The situation gets stickier when we look at those combinations and the transitions between them.

Use GPL code with Closed S…