Sunday, December 31, 2006

Flow Testing - Initial Ideas

I'm toying with the idea of a new way to write tests. This is a cross between state and interaction based testing, and also tries to make setting up the test environments easier. I call it "Flow Testing" as it tests how the various states of the test target flow from one to the next.

The idea is several-fold. We begin on the idea that we are testing names, because everything in Python starts with a name. You will test different things about names, such as type, what attributes or other properties apply to certain conditions, etc.

Flow testing is very tree-oriented. For every name, you define the state you expect to begin, and branch into the state you expect for various properties. For example, you may have a state for the name 'foo' and a branch state for 'foo.bar'. These are branches within one state, while wthere are other branches across states called Interaction Branches. These states define the interactions that lead to them and the states expected.

The example below defines a test for a simple module that should have a simple function. It defines that the name foo should be a module and that the state of its 'bar' attribute should be a callable. It then defines the state of a call to bar, and details it under more specific conditions of the interaction (specific arguments given).

import flowtest

import foo


class test_foo(flowtest.State):
targetType = flowtest.module
targetHasAttributes = ('bar',)

class test_bar(flowtest.State):
targetType = flowtest.callable

class testReturn(flowtest.State):
targetType = basestring

class testReturn(flowtest.State):
flowtest.callArguments()
flowtest.callArguments('hi')

targetContains = u"hello"

class testReturn(flowtest.State):
flowtest.callArguments('bye')

targetType = basestring
targetContains = u"goodbye"


This is just an idea, and does not work or exist yet. Any comments? If it seems solid, I'd like to start working on it.

Saturday, December 23, 2006

Holidays

I am on vacation. It is very, very nice.

Even before this, things have been slow. On the edge of worryingly slow, but still not slow enough that I've had to fret over anything. I've been enjoying some relaxation and time with my growing baby boy and my beatiful wife.

We've taken the holiday with her family and our friends in sunny North Carolina, where I have been enjoying warm, sunny weather to start my winter. I actually miss the cold.

Although not much has happened externally, a lot has happened internally. I am using the down time to organize myself, my plans, and prepare movements forward for proper exploration of some plans I've had on the back burner for a while now.

More of my time has been able to be spent on more recreational activities, as well, and that is nice. Reading has been lost to me for some time, and I've picked up a few books I've enjoyed. I plan to start another blog, which will be focused on reviews of literature developers, gamers, and other geeks will enjoy, but not necessarily literature about those topics. I want to bring other people in on this one, so if anyone is interested in reviewing good books, new or old, contact me (ironfroggy (ta) gmail (tod) com).

Speaking of the holidays and recreation: I got a Wii! I've been loving it. I have the WiiSports that came with the system, and the new Zelda, but I actually play the WiiSports more, so far. It is so fun. I've been taking the system around to friends' houses and we all have sore arms from so much bowling, the funest game for multiple people and one controller (and maybe my favorite one besides that). Anyway, when I get some good online games, I can't wait to play against some others in the community.

My Mii number is 3456 8115 5527 0431.

So far I can't bowl over a 175, the words "fowl ball" have lost all meaning, and I can sleep easier under the delusion that a higher score is better, even in golf. Zelda is slow going to start, but I'm enjoying it. Kind of obvious how it wasn't started as a Wii game, however. Most of the time the wiimote is just used to make a little fairy (with, so far, no mention in the game what-so-ever) fly around. And I can fish.

I will possibly move away from blogspot soon. I need more flexability.

Friday, December 08, 2006

ExtSession 0.3

I released ExtSession 0.3 today. Added error reporting through the result objects, iteration over remote objects, cleaned up much of the code, and did a some bugfixing.

Also included a tutorial on extsession usage, so hopefully some of the people that had difficulty can read that and understand how to use it.

Go and get it.

EDIT: Fixed a broken link! What a way to mess up a release, no matter how small.

Wednesday, December 06, 2006

Announce: ExtSession

As a support package for some other things, I've written a little module called ExtSession. The basic idea is to have an easy way to control a python session running in another process. You can simply instansiate the ExtSession class, and call the execute() method. The results are asyncronous-like, but will be improved later. Very likely support will be added to get Deferreds. For now, you can simply poll or wait. The results of each execute() call (best done one line at a time) are seperated and returned individually of output from other commands, making it very easy to work with over a session's lifetime.

Following is a stripped pydoc pull of the docstrings. I hope someone finds this as useful as I think I will.

NAME
extsession

FILE
/home/calvin/extsession/extsession.py

DESCRIPTION
ExtSession Module
For executing code in an external interpreter.

There are many cases where operations are either intensive or blocking by
waiting, and in both situations the extsession module can be used to control
a python session in a seperate process for the purposes of executing code
in parellel without the dangers associated with threads or the complexities
in dealing with asyncronous frameworks.

The core of extsession is the ExtSession class. See its help for details.

Also, look at the exteval and extexec functions for quick, blocking code
execution in a fresh interpreter. (note: these functions do not execute in
parellel, only in a seperate process, but blocking the original)

CLASSES
__builtin__.object
ExtEval
ExtResults
ExtSession
Session

class ExtEval(__builtin__.object)
| Evaluates expressions in a new process with an asyncronous result.
|
| The result attribute evaluates to the result of the expression, or access
| raises a ValueError, if the result is not ready.
|
| Methods defined here:
|
| __init__(self, expr)
|
| ----------------------------------------------------------------------
| Properties defined here:
|
| result
| = _get_result(self)
| Get the result only if its ready.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __dict__ =
| dictionary for instance variables (if defined)
|
| __weakref__ =
| list of weak references to the object (if defined)

class ExtResults(__builtin__.object)
| Represents results from an operation in another process.
|
| Methods defined here:
|
| __init__(self, stdout, stderr, done)
|
| read(self, size=None)
| Read data from stdout of process from operation.
| Raises ValueError if the operation is not complete.
|
| wait(self)
| Return only when the operation is complete.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __dict__ =
| dictionary for instance variables (if defined)
|
| __weakref__ =
| list of weak references to the object (if defined)

class ExtSession(__builtin__.object)
| Controls a python session in another process.
|
| Methods defined here:
|
| __getitem__(self, name)
|
| __init__(self)
|
| execute(self, source)
| Execute arbitrary python source.
| Returns an ExtResults object to access the results.
|
| exit(self, code=0)
|
| readcodes(self)
|
| readline(self)
|
| writeline(self, line)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __dict__ =
| dictionary for instance variables (if defined)
|
| __weakref__ =
| list of weak references to the object (if defined)

class Session(__builtin__.object)
| Manages the session in this process. Used by ExtSession in spawned
| interpreters. Can also be used for a very light sandbox in the same
| process.
|
| Methods defined here:
|
| __init__(self)
|
| execute(self, source, stdout_fn=None, stderr_fn=None, done_fn=None)
| Executes code in a semi-controlled environment and redirects output
| to given stdout and stderr filenames, or random temp locations. Writes
| a code to the file at done_fn when finished. A code of 'DONE' is
| expected. When a code appears in the done file, the stdout and stderr
| files are ready for reading and are complete.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __dict__ =
| dictionary for instance variables (if defined)
|
| __weakref__ =
| list of weak references to the object (if defined)

FUNCTIONS
exteval(expression)
Creates a new process running a new python interpreter, evaluates the
given expression, and returns the result. The result must be basic types,
but may expand in the future to any pickle-capable type.

extexec(source)
Creates a new process, executes the source in a new python interpreter,
and returns a tuple of the stdout and stderr captured.

sleep(...)
sleep(seconds)

Delay execution for a given number of seconds. The argument may be
a floating point number for subsecond precision.

Tuesday, December 05, 2006

Statement Functions

At a small suggestion in #python, I wrote up a simple module that allows the use of many python statements in places requiring statements. This post serves as the announcement and documentation. You can find the release here.

The pattern is the statement's keyword appended with a single underscore, so the first, of course, is print_. The example writes 'some+text' to an IOString for a URL query string. This mostly follows what it seems the print function will be in py3k.

print_("some", "text", outfile=query_iostring, sep="+", end="")

An obvious second choice was to wrap if statements. They take a condition value, and expect a truth value or callback an an optional else value or callback. Values and callbacks are named if_true, cb_true, if_false, and cb_false.

if_(raw_input("Continue?")=="Y", cb_true=play_game, cb_false=quit)

Of course, often your else might be an error case, so raising an exception could be useful.

raise_(TypeError("Wrong datatype!"))

When you do have exceptions, you often want to handle them and we can even do this in expressions now. Maybe we just want to report the error. The try_ function takes keywords for except_, else_, and finally_ callbacks. All take exception type, exception, and traceback arguments, with else_ and _finally also taking first arguments as the result of the attempted callable.

try_(some_func, except_=lambda e,et,tb:print_(et, ':', e))

Errors also can come from failed assertions.

assert_(lambda: True == False)

Loops are also supported. for_ is mostly like map, but it also takes an else_ argument which is called with the last value from the iterator. while_ takes a condition callable, either a for_each callable or an iterator. The condition is given the result of the last loop.

for_("abc", print_, lambda i: print_("done"))

while_(lambda last:last<99, xrange(100), lambda_ i: print_("done"))

Finally, some simple ones:

pass_(10, foo="a") # returns ((10,), {'foo': 'a'})

exec_("a = b", globals(), locals())

I don't recommend using this a whole lot, but in the cases its useful, enjoy.

Wednesday, November 29, 2006

Calling [adultswim] Fans

Adult Swim fan? Thought to myself "would be nice to converse with fellow fans during the block". Especially during Sunday nights. Interested? Join ##adultswim over at irc.freenode.net, the next time you're watching [as].

PS - The double #'s is due to an oft-critisized policy at freenode for "non-official" channels to be moved to such a name, signifying it as an "about channel", and allocating the single # channels as "official".

Sunday, November 19, 2006

Bad Blogging Weekend?

Is it just me or does it seem like a bad weekend for blogging? I don't want to say I expect huge traffic, but I'm definately seeing a huge decrease in hits since friday night. Massive, even. Is a two-console launch weekend too much drain on the blogosphere to make much blogging worth it?

Saturday, November 18, 2006

Forums, Feeds, and Foundations for Answers

The web forum. Mailing lists. Newsgroups. They all boil down to the same thing: digital forums of discussion. We have relied on these mediums to carry the Internet from prototype to fad to craze to necessary base of our culture. The discussions we enable through the web (and its friends) drive everything we do in the digital world. How can we pay tribute to this with an upgrade?

We have migrated from discussing to just broadcasting our opinions. Maybe we'll tune in to some other opinions on similar topics. Maybe some of them will have read ours, who knows. We are deviating from the formula that has driven us to everything we think models the technology we desire. We need to move back to real discussions, bringing what we've learned from feeds, blogging, and content subscriptions.

Forums, beloved as they are, have lost their way. Unfortunately the number of Internet users has risen so high that no forum can carry nearly the percentage of knowledgeable users they once could. This leaves all forums at a loss for intelligent information and leads to a loss for anyone using them. Either we must spend eternity locating the perfect forum for each exact question, or we cross post dozens of forums and monitor them all for updates. This simply does not cut it.

Everyone has at least one feed these days, and what we feed the world with is still pretty limited. We write little posts, usually barely enough to call an article. We can post anything. We can feed the Internet our pictures, voices, or our minds. We can send out our questions, formatted for consumption, and allow everyone subscribing to our feed or receiving the questions through aggregation and search engines to get a full range of questions from the curious and in need. Instead of browsing to a forum page, we can find those in need through the same feed readers we already enjoy.

For the responses to the questions, the same feeds can be used to broadcast responses with meta-data attaching them to the right questions from the current feeds. The questions can even be posed with information on HTTP requests the responder can make to inform the original poster of a feed with a response.

No, this isn't even a draft of any idea for a technical spec. It is just a few ideas, and maybe even just enough for a micro-format. It would not be a lot to put together, would be less to pose the questions like this, and most of the work would be in informing those asking of the answers, but not much work.

Tuesday, November 14, 2006

Help JMRI

I don't even use the software from the JMRI project, but I had to donate a little to help their potentially important case. The violators are claimer that by being free (as in beer), all copyright is waived and thus any GPL restrictions are not even the project's own to claim. Wow, that is an evil claim.

Do what you can, even ajust five or ten bucks.

Self Promotion

I am trying to improve and promote my media (reviews and such) and arts (writing and painting) blogs, so here is a bit of shameless self promotion.

Spilt Mind has some ramblings about my attempts at rejuvenating my creative side.

Mental Outlash just got a review to the amazing Stranger Than Fiction.

Interested? Check them out. Otherwise, sorry to bother you. I want to keep these minimum. I should probably append some cross references to the end of the next post I write here.

Asyncronous Database Records

Through persistance systems, notably Divmod's Axiom project, i have been experimenting with the idea of asyncronous request of items which may or may not exist for some time. The idea is an abstraction of a terrible first idea for "persistant deferreds," which my very suggestion of lead to horrible responses over in #twisted, but well deserved, I now believe.

The concept is similar but perhaps simplified for the limitations and complications involved. Operations may return an "asyncronous item," which in my implementation is done by an Item implementing the IAsyncronousOperation ("operation" may be replaced with "Item") interface. This is akin to returning a deferred. The item allows the caller to control the response to the availability of the item, but in a way that can survive server crashes and reboots, and is otherwise a persistant record, and not an emphemeral object.

Borrowing additionally from Twisted, the asyncronous results can support both positive and negative handlers, set for managing the result as success or error. The creation of these handlers constitutes an additional asyncronous result, which can be used to chain handlers, akin to the callback chains of Twisted. In the event that the requested item is ready, which is either immediately or in the future, the appropriate handler is called and the asyncronous result handlers are cleaned up.

I will release the code soon, when the rest of the unittests are complete and an example usecase can demonstrate the usefulness. I have gotten some negative reaction from this one, and I really hope it can be attributed to misconceptions of my intent and failure to consider the right usecases. Hopefully I can remedy this.

Wednesday, November 08, 2006

Design for Testing

Good testing influences your design. Too many developers (and managers) are stuck in the waterfall mentality of design, code, test, and deploy. Forces across the industry are pushing to move the testing phase to the beginning of this line, and many just don't understand how that works. Testing should be the first consideration, and thus is influential to all other aspects of the development process.

The problem with testing as a secondary consideration is the design and architecture of the software never lends itself to proper testing when you don't plan way ahead. The consideration of testing can drive your design to be easier to test, but also can encourage generally good programming practices and well-made designs in the architecture. We can use a persisted class as a good example, because this is a use-case where testing is very important, but we have to consider the burden of a full database tied into the class we are testing.

When we develop our database item class before any testing is considered, we create it fairly in a straight-forward manner. After everything is coded, we decide to do some testing but we have a couple problems to face. The first thing to concern us is that our class inherits some ItemSchema super-class, and instances of it must exist in context to some database, which creates a large dependency on the test and thus leads to the test being unreliable. Secondly, we have many functions not easily testable (perhaps they can only be confirmed by locals within the function, which we can not access). We need to redesign everything to solve these issues, but we could have avoided this by using a more testable design in the first place.

To solve the first problem, we have to consider what we are actually testing. We are not testing the persistence framework our application utilizes, but just one ItemSchema sub-class we had to write. Obviously, separation is key. We only care about the functionality we wrote into the type, and we can extract all this into a mixin class, which our original ItemSchema can inherit. However, in our tests, a special TestItem class may also inherit it and perform the testing we need, without bringing a database into the picture.

Tackling the second problem of individual functions, the solutions can vary. If any internal data is important, perhaps it is too internal, and the code generating it could be extracted into its own method, which we can test independently. If our method does not return anything, perhaps this is something it could return and thus we could test for it. However, don't return things you do not need in production, as this bloats the interface and inevitably some code will come to depend on this contract that was only intended as a testing mechanism. It can even be acceptable to wrap a returning method with a public API non-returning version, simply to push a more stable API (it is easier to add returns than remove them) to the consumers of the API.

We have to stop looking at the testing as a second, or third class citizen in the steps we take in development. Our design, development time-lines, and architecture should all be done with testing and quality assurance first in mind. We must focus on how to ensure the stability and accuracy of our code before we can ever trust it, and if we can not trust it, all time developing it is a waste. It is this fact that offsets all arguments by opponents of proper testing (yes, they exist) who are afraid of the time wasted by testing. Wasted time is illusionary, as we only see the time it takes but we don't see the time it save us. However, isn't that the point?

Sunday, October 15, 2006

Spilt Mind

I won't be able to use it much, but if I have the need to write something and get some creativity flowing (it helps improve your coding mind, as well), I will be posting those writings or art pieces to a new blog I created, Spilt Mind. So easy to make a new blog, and I keep getting distraught over my lack of writing and painting over the last year or two. This is a much needed outlet, however much it gets used.

Uprooting and Replanting

It looks my time doing contract work is coming to an end. I will soon be officially accepting a full-time position with a relocation to the Houston, TX area. I'll have much more to blog about as we accelerate development timetables and things begin to roll along.

I'm tired and exhausted from spending days with this on my mind, so I'm not writing much. I just wanted to ask, is anyone from the area? I see there is no User Group near Houston, at least not listed at the Wiki. Does anyone know of one or would be interested in starting one?

Any tips for long (1700 mile) moves?

Saturday, October 14, 2006

Identity Comparison vs Comparing Identities

An explaination of the virtues of foo is not None over foo != None lead to an explaination of identity comparison with the is operator. A fellow equated this to, roughly, id(a.x) == id(b.x), which I told him was roughly correct but probably not actually correct. It only took a little bit of through to see how uncorrect it was.

The following code creates a simple class with one property descriptor (read-only). It solves the requirements that with to instances of this class, a and b, id(a.x)==id(b.x) can be true while a.x is not b.x! How does this happen?

class foo(object):
x = property(lambda s: id(s))
a = foo()
b = foo()
assert id(a.x) == id(b.x)
assert a.x is not b.x

How on earth does this code prove what it does? a.x and b.x are created on the fly, passed to the id function, and then destroyed with no references left. Because they are both created in the right order with the id(a.x)==id(b.x) expression, they just happen to get the same memory addresses, which in CPython is used for the id function's result. This leads to misleading results, so don't rely on them in such ways. Identification is what it is, and you shouldn't try to break it down.

Thursday, October 12, 2006

RuPy, Snakes and Rubies, Night and Day

What is going on?

All around me I see people relating Python and Ruby. The communities seem somehow intertwined with their users, usages, and publicity. It seems like you can never find a blog that talks about one without the other, or a new frontier being tested by one of the "new" languages and not being explored by its counterpart. Where do people see this distorted similarity between two languages that are as different as night and day?

To Rubyist out there, don't take offense. I have far more experience with Python, than with Ruby, so I might sound biased. Please, take this as an honestly unbiased opinion. I'm just talking from the middle in this case.

Python stands out from most of the other languages around. Largely it is understood as "the language without brackets", but it is not entirely alone there. Ruby almost appears like a language extremist. Their core types have fifty million methods each, because if you expect it to be there, it should be there. Python is important, because it explores the methods a language can encourage good coding practices onto developers. Ruby is vital, because it explores interesting constructs like continuations and compact code forms utilizing their quirky syntaxes. Both might have their place, but they are far different places. Python is unassuming. It will take nothing for granted, such as not taking for granted that you wanted a method call, not a call to a function in some method. Ruby is assuming. It understands the difference between objects and types, just based on the casing of the name.

Fuzzyman states a similarity between the languages, but can this be taken as anything more than common usage? We can see an assumption of similarity across both communities, but nothing really ties them together anymore than any languages are connected by virtue of all being languages.

I've been known at times to seem like a Python zealot and anti-PHP/Perl/Ruby, but in this case I'm setting that aside and I want to state very clearly: these two languages are not similar. They are very different beasts, with more differences than anything else. The illusion of a relation between Ruby and Python only harms both of them. We have people asking for continuations in Python constantly, when they are a great idea in Ruby, but would not likely work well in this very different environment. Rubyist get attacked by a barrage of Rails killing Python projects, trying to take their steam.

Shouldn't we go our separate ways?

Wednesday, October 11, 2006

Balence, Tranquility, and SOAP

I am behind schedule with my work. I attribute a good bit of this to my vices as a developer, and just as much of the problem to my good attributes. I place an equal portion of the blame on SOAP.

Striving for the goal of Beautiful Code, we can find ourselves lost on the way to actually writing something that gets the job done. Throwing away perfectly working code, because an alternative way to achieve the same results is more elegant isn't something that we might see as a bad idea. If the code is more l33t now, it will give us less trouble tomorrow when we need to port it to my toaster. We'll use anything to justify the overworking for code beauty.

Is it always worth it? How beautiful is enough and when are we just wasting our time (and money)?

There are terms thrown around like "elegant" and "pythonic" to measure the quality of code with no attention to the code actually reaching the goal it sets out to perform. The code may work, but that doesn't make the code good code. Without a sense of time, scale, and the big picture, the search for good code can overshadow any good developer's work towards working code.

However, as any issue as a flip side, those developers getting lost are doing so in the name of a good fight: the first against bad code. We might get lost and never complete our code, or complete it late, but we do so with the complete belief that it was worth it. The code took several weeks longer to develop, but just look at how beautiful it is. Without the struggle for good code, our working, bad code would eventually overshadow us just as much and consume our time with maintenance, refactoring, and the mother of all frustrations in coding: trying to read your own work.

I am wrapping up some SOAP-heavy work and the path to completing it has been a testimony to the struggle of balance in code. Recent refactorings of the actual SOAP response processing ended with a good chunk of bad code. I don't like the way I'm doing lots of things, or the fact that it doesn't parse corner cases the service I'm using doesn't even use. The code is not the beautiful code I would like to call my own, but the code is working code and does everything it needs to do. I had to bite my own hand to keep the refactoring to a minimal and focus solely on the aspects of functional goals, ignoring aesthetics.

Be careful on the road to good code. Somewhere along the way, you can easily get lost and never reach the point of having actual, working product. Sure, the code will be incomplete, but it will be a fragment of beauty. Learn the value of a completed mediocre code set over the eternal development of more beautiful code, which does exactly the same thing.

Monday, October 09, 2006

You Go, Gamer!

I would consider myself a gamer. The video game industry has been a huge influence on my life from childhood to parenthood. My father was a gamer, who even owned an arcade as a younger man, before I was born. The last machines, a unremembered pinball unit and a standup Qix were the first things walking in the door of his place. I even had a two-screen, six-player TMNT Turtles in Time arcade machine, which made me the coolest kid among my friends. Yes, I am a gamer.

Today I denounce my fellow gamers, and I call out to those of you with any backbone to stand with me in decrying those among us who, basically, just make us look bad.

The subject of debate is a recent survey making statements about the percentage of females among the online gaming community. The surprising results state that 64% of online gamers are female, and the response to this news is just as surprising: outright denial. Male gamers seem completely impossible to accept that they are not the majority of gamers, while they have been simultaniously complaining about the lack of female gamers for years.

Apparently, someone forgot to include a requirement that they be "real" gamers.

Over at the popular gamer blog and online comic, My Extra Life, the story was given with a note of disbelief and the commentors followed suit. Reading through the comments, you can see such Male Stereotype Improving comments as the following samples:
  • The survey sounds fishy, unless they included "free games that are in the browser or games like second life." It still sounds fishy.
  • "No. I am a minority darn it!
    I will not play with 64% of other women!

    Maybe its a typo, maybe they meant 64% of gamers are kids.."

  • "women under 16 years of age sound like dudes"
  • "I agree.. 90% of them are playing bejeweled right now."
  • "they’re not the kinds of games correlated with the “gamer” title"
Neopets is probably one of the older and most popular online games around, and any real "gamer" will correct you if you say its a real game. It is only a kids toy. Or, a chick thing, if you find someone really intelligent.

I had my say over there, so I'll quote myself now: "I am disgusted by my fellow gamers’ responses at this announcement. I have heard nothing but denial, attacks on the survey, insults towards are female gamers, and downright insulting behavior coming from my demographic. We have to find all sorts of reasons why the survey is inaccurate, or counted things that “aren’t real games”, because we aren’t man enough to admit to not being on top of something we long thought was our own. Get a life and just admit the truth. It doesn’t matter if you play Halo or Scrabble, and its sexist and degrading to assume thats the only kind of game those women would play. Degrading to other male gamers, who have to be lumped in with the deplorable comments being made around here.

Of course more women would play online games! Lots of women love to game, but its not like they can play among other gamers face-to-face. When’s the last time you treated your buddy’s girlfriend as an equal in a game of Halo? No, she goes on as a handicap. No one takes her seriously. So, what is so to do to satisfy the thirst for games? Play online, where no one even realizes she has breasts. That’s why 64% of online gamers are women: because you don’t know and you hunt them down in Counter-Strike just as ruthlessly as anyone else."

One of my best friends is more of a gamer than I can claim to be. I just can't justify spending as much on games as he does, and my buddy has three times as many kids as me. His wife can't hold her own against the worst of us on the old Halo nights (which I miss so much), but she could kick my butt in a game of online poker or a few rounds of guitar hero.

Besides all the response making me ashamed of my fellow male gamers and wishing I could avoid the association until this blows over, I found the comment about Second Life to be pretty confusing. I understand why you don't consider a Scrabble game on Yahoo to be a "real game", even though I don't agree, but Second Life is one the greatest acheivements in gaming history. What gives, traveller18?

Ironically, when I sought out some stock photos for this post, over at stock.xchng, I turned up both of those photos of girls gaming, but not a single one of a male gamer.

Sunday, October 08, 2006

Movable IDLE: Keep it Free, Fuzzyman!


Several weeks ago, I contacted Michael Foord (aka, Fuzzyman) about the idea of a simpler, slimmed version of his Movable Python distribution for those interested in the language, and often not computer literate enough to put up with all the different components you need for any decent development environment. This is exactly what we need to convince those people still hanging on to BASIC.

I don't know if its from my suggestion, although I got a personal e-mail from Fuzzy on the announcement, but its here and its free and its exactly what I need for the many interested people I am trying to bring Python to.

Google Reader Upgrade Dissappoints


Yet again, I find myself wishing I could say I like what Google has done with something, only to be forced into admitting: they disappoint me. Well, add one more to the list of things Google can't get right, even with an army of PhD holders and more money than you can shake a redwood forest at: Google Reader has gone from simple gold to contorted crap. The original version was a great excersize in simplicity in design that let me jump in, read, and get on with my life. The "upgrade" is a mess of a noisy interface for me to get lost in as my browser slows to a crawl with far more JavaScript than a simple reader needs, and even the occassional forgetting of everything I haven't read yet.

I was a little late in the Blogscene, which is a relative statement given that most of the world doesn't know what a blog is, despite the fact that most bloggers think otherwise. I started with my trusty KDE's Akregator, which is admirably usable, and then looked for a web-solution to use better from multiple boxes and a laptop, right around the time the first Google Reader was released. I jumped on board, and I loved it. Right off the bat you have your entire reading queue, waiting for you to read through one post at a time. Read, hit the j button, read, j, read, j, j (I skip things, a lot), j, j, j, read, j, read. If I had a backlog, I could just pick labels or subscriptions to read first and be on my way until I had time to read less important things. The key is it was simple. Most of the time, I only ever used a single button: the j. It was fast, showing just what I was reading and a few things coming up on the list. Thanks to (possibly accidental) details of the implementation, I could scrollwheel over the reading list and pre-load hundreds of articles, so that I could read them offline in my web-based reader! Again, best of all, it was simple. It did all this and it was clean, and simple. God, it was simple.

The new Reader is a beast. There is a busy tree of labels and subscriptions listed on the left, repeating my feeds for every label they are in. The unread counts are always inaccurate. It tries to show me everything I've read so far on the page, which adds up quickly. I can't mark anything as read without everything before it getting marked too, which means no holding things to read later. The javascript slows the page down and even locks up FireFox for a few moments when loading the next posts. it is not simple.

Google, use your many brains. I don't know how you could have messed this up so badly. It brings up an interesting question: is it OK to compete with yourself? They say that the new Reader meets the middle-ground between what everyone wants, but does that mean it doesnt actually fit what any one person wants? Re-release the original as Google Quick Reader or something.

I'm sorry. This post was badly written. Call it a rant. I just miss my reader.

Friday, October 06, 2006

Stuff of Interest - Week of Sept 17 - 23, 2006

Wow! It has been almost two weeks since my last post, and I was doing so well. Unfortunately, i got quite sick and then had an unexpected trip out of state (read: I forgot about it until the day before!) and now feel ill, yet again. But, determination brings me back. I was planning to post this on the 23rd of September, but the 6th of October is close enough. Regular posting will continue starting tomorrow.




This is the first in my weekly post of interesting links around the web. These are articles, websites, services, photos, and anything that else that I want to bring up on my blog, but can't use an entire post for. Being on any aggregation sites makes you think harder on each post and puts some weight on you against those little posts, at least in my mind.

Hopefully, you'll enjoy whatever I post here. Maybe you'll find a useful site, or learn something you wouldn't otherwise.

Really Smart Stuff

Ross Jekel, over on the Python 3000 mailing list, supported the existance of the GIL (Global Interpreter Lock) in a very clear paragraph). This came from a thread (yet again) about the possible removal of the GIL for Python 3.0, which will almost definately not happen. Instead, the most weight seems to be toward improving the effectiveness and productivity of multi-process or multi-interpreter-in-one-process communication to offset call between interpreter boundries. Awesome stuff if it happens!
After some initial surprise when I learned about it, I'm now okay with a GIL or even single threaded python (with async I/O if necessary). In my opinion threaded programs with one big shared data space (like CPython's) are fundamentally untestable and unverifiable, and the GIL was the best solution to reduce risk in that area. I am happy the GIL exists because it forces me to come up designs for programs and systems that are easier to write, more predictable both in terms of correctness and performance, and easier to maintain and scale. I think there would be significant backlash in the Python development community the first time an intermittent race condition or a deadlock occurs in the CPython interpretor after years of relying on it as a predictable, reliable platform.

Really Cool Stuff

In my effort to professionalize my blog, snazzy it up a little, and a general interest in having cool things to look at, I found stock.xchang, a free stock photo exchange website. There are great photos available there, and when I get my camera working again, I will definately be contributing to the collection.

Sometimes you just gotta say "Wow." I had a little trouble with the prototype applet running sluggish and buggy, but watching the demo video is just amazing. The software will extract 2D shapes into 3D models and allow you to extrude, cut, reshape, and just do some amazing things with an interface so simple that a kid could, and has, use it. Makes me want a touchscreen all the more, so I can have extra fun playing with this.

Really Quick Stuff

JavaScript Scope (and this) Explained in Detail
ParenScript - Lisp to JavaScript translator

Being Helpful by Not Answering Questions

As some of my readers (I have readers?) may know, I am a frequent of #python over at Freenode. A great place. One of the most supportive IRC channels I have ever been a member of. Over the years I have been a frequent member of this channel, I have received an awful lot of help. It is where I went when I first decided to learn Python, and the kind folks there did great things to guide me along. I learned and I stayed, because I still need some good minds to knock ideas around with, and figure things out. I also stayed because the best way I can repay the help I received is to return it to others who seek just that.

I want to think my help is appreciated. I happen to know it is. There is an increasing number of regulars, learning their way through, who explicitly seek me for help, send me entire projects to look over, and generally befriend me in response to the advice I give them. I try not to think highly of myself, but I do believe I am valuable to that channel and that many others would agree.

A few, however, seem to hate my guts. A growing minority of users are continually harassing me over my methods of giving advice. They have a problem with how I talk to people that ask simple questions, even with those people not being them and happily taking my advice over the complaints of these few difficult IRC'ers. What they seem to have a problem with is my tendency to answer questions with questions, investigate why someone thinks they want to do what they ask how to do, and suggesting other ways to reach their goals that may be better than what they came seeking.

This is not a technique of myself alone. Python has a strong community of developers with strong opinions. It is not unusual for people to ask about threads and be told that Twisted, separated processes, or Stackless is better. If someone asks how to set a variable with a name in some string, they aren't told about globals() and locals(), but to use a dictionary instead, and usually will be given a small talk about how all variables exist in dictionary, including the globals and locals, so there is no overhead in this and its a perfectly good thing to do.

People aren't given a gun to shoot themselves with. They are given advice not in answering their question directly, but delving into the source of the question and solving the problems that lead to their asking a question, although sometimes misconceived.

Is it wrong to assume you know someone shouldn't do what they ask how to do, and tell them something else instead? Does anyone have the right to insult and verbally abuse those who practice such techniques of helping others?

Does anyone have thoughts on this? Lending a hand is important, so we should be doing it right.

Thursday, September 21, 2006

User Expectations for Free Services

"Google, you have no right to track my searches!"

How many people have been saying that, especially since the whole thing with AOL "leaking" those users' search records? "Leak" nothing, they released them carefully as an opportunity for research, and we did nothing but attack them. They could have sold them, names and personal billing information included, without ever telling us a thing. What they did was perfectly within their bounds, just read your agreement with them.

Google seems to almost get more cruft about the whole thing than AOL. Maybe people just can't get surprised when they think they see something a company they never liked doing something they disagree with. People want to love Google, so they will get defensive about them almost to the point of taking it personally. How do you draw a line to the rights of users, when they do not even pay for the services being considered? Can we have any rights without being customers? If I asked you a question, do I have some right to demand that you take that question to the grave?

We are throwing an ever increasing amount of information we consider our own into free services across thousands of miles over unsecured channels, and we expect some kind of privacy and set of rights about how that information is dealt with. We have more than search history to deal with, because we're starting to replace desktop office apps with websites to entrust with our unpublished novels and family finance records. We obviously have a large trust in the services, but why do you expect this? Do we even have a right to expect that trust, or are we just delusional?

I do not care if Google tracks every site I browse to, analyzes it, locates patterns, adjusts my searches accordingly, shows me adds based on my browsing habits, or anything else they might be doing with the information. They are an ad driven company. They make money on advertisement clicks and I won't click on something I don't want to buy, so they have to do everything with getting advertisements in my face that I actually want to click, and that means it must be something I'm actually interested in (why else would I click?), so I'm getting as much out of it as they are. Google makes a quarter, and I finally found that great flatscreen TV deal that saves me a grand. Who is the winner here, and all I had to do was drop my pretension notions and enjoy a free service for its hidden cost: a little anonymous (outside Google) exposure and a few pixels of screen on the result pages. Wow, I am so winning in this relationship.

Amazon has been tracking your use of their site for years and doing amazing things with the way they cater to individual users, and there has been little fuss about it. No one complains, because they are too busy watching the movies Amazon suggested they buy. We have rejected the advertising the drives the economy for decades, but when it starts to actually be things we want to buy, do we even see an advertisement, or just a cool new album by your favorite artist? Advertising is meant to convince you to buy some product, and the best way to convince you is to find you already wanting to buy what they have to sell. You can't convince someone who doesn't want or need a product very well, you can just annoy them. That means getting rid of useless advertising is great for the consumer, and I welcome my personally tailored advertising.

Now, if only I could figure out why these amazing, personally specific advertisements were trying to sell outdoor equipment on my programming blog.

Wednesday, September 20, 2006

Thank God for Standards! Or, Thank Someone Else?


Millions of dollars went into the development of the Digital Video Disc format, its licenses, encryption schemes, patents, bushiness deals to ensure its success, and all the work that went into getting rid of those troublesome plastic cases with yard after yard of magnetic tape. We can all be thankful for that!


We have gone through many forms of video, and we can take a lesson from the history of recorded audio. In 1878, Edison invented the phonograph, which we usually think of today as a cylindrical record. The lateral-disc records were technically invented in 1888, as a direct adaptation from Edison's design, but remained covered by patents and used as novelty talking toy technology for another thirty years, until the patents ran out in 1918, forty years after Edison's original invention. It would be another 56 years before the introduction of the cassette tape in Germany. The trend ended there, and we became frustrated with cassette limitations, seeing the introduction of the Compact Disc (CD) in 1982, just 18 years later. It would be only 15 more
years before the first MP3 player hit the market, although the technology of compressed digital audio had been seeing wide use in other circles for at least half a decade.

We have seen similar transition from reels, to beta, to VHS, to DVD, and now to both HD-DVD and Blu-Ray DVD. When I say to both, I really mean it! Warner Brothers is developing a triple-formatted disc, which has a DVD on one side, HD-DVD on the other, and Blu-Ray on a two-way mirror-like surface over the HD-DVD layer, allowing a single disc to work in traditional DVD players, HD-DVD, and Blu-Ray players.

Seriously, folks, what point is there to all these competing formats, when the end result is to just use all of them? Before you know it, everyone will license this WB patent and all the players we buy will support both HD-DVD and Blu-Ray, and we won't even know which format will be used because our movies and players will be playing split-personality the whole way.

The madness probably won't end until we move away from physical distribution methods, and even then there will be encoding, compression, and encryption wars, but at least our smart players and laptops will be able to just download any drivers they need automatically. Yeah, there still won't be a point to the varying formats, but it will be a little bit easier to ignore that there are competing formats, as we always will inevitably do.

Doesn't that tell anyone something?

I can't believe I can use the term "Traditional DVD", already.

Tuesday, September 19, 2006

Concurrency and Stabilty. More Zen for Python.

I was planning on writing something about the current conversations on the Python-3000 list about concurrency, which just doesn't stop being brought up and never gets resolved to any point that anyone is happy with. In an unrelated action, I went to check on my older blog, which I thought of resurrecting for some non-development articles, and I found a draft from a previous time the topic came up. I read it and was surprised to find that it proposes pretty much exactly what is being put on the table over this past weekend, with reference to walling off multiple interpreters in a single process and controlling messages past between them. The same technique scales for multiple cores, multiple processors, or multiple machines.

I've decided to take the easy way out and just post the original draft with minor editing. I enjoy how spot on I ended up being with what is currently becomming an acceptable solution, it might seem. The original was written nearly a year ago. Does this mean my predictions are worth something? Decide for yourself!

Concurrency is a hot topic on the Python mailing lists lately. There is a strong push to get some kind of native concurrency into Python, as the 3.0 branch is a great opportunity to do things that we can't otherwise do, as they would break old code. If we don't get something in now, particularly, something that can scale to hundreds of thousands of tasks and take advantage of multiple processors, we may not get a chance at what could be the best improvement of the language, until the next major version 4.0.

A large part of the problems stems from how horrible an idea threads really are, as they are typically implemented. Threads, as the basic level, are just multiple pre-emptive tasks running with access to the same memory space. Doing this can be a boost for performance, but is hell to control properly. The threads must be syncronized to access their shared resources without clobbering each other. This can be done, but it is very error prone and very difficult to debug.

The solutions seem to lean toward two ends of the spectrum: cooperative tasks and processes. With cooperative tasks, each concurrent unit runs until it says "OK, I'll let someone else run now", and so there is no explicit syncronization needed, because nothing happens without you knowing it, idealy.

Processes, on the other hand, are basically threads without shared memory. These are the same way the concept of processes are implemted at an OS level. Some, including Guido van Russom, even think that is how we should go: multiple system processes communicating via pipes and sockets.

What I propose is a process implementation with python itself. This would offer a lightweight process execution, where each process would consist of a thread of execution and an "object space". Each thread would only be able to access objects within this space. Communication would occure through channels between processes, which can be used like generators (gen.send(10), for example). I've created a basic implemenation, available on my webserver, here.

With my basic demo, you can create object spaces, each with their own global and local dictionaries. There is no real protection, but it shows how it would work and offers an idea of how it would be used if we had real protected object spaces. You can run a function in a space with the run method, which takes a resonse function first, which is called when the the function returns and will be passed the called functions return value. If a function is already executing, the request is queued until its turn.


I need to look around and find that demo/prototype code. I barely remember writing it, but I remember being pleased with the results. I'll look around and resurrect it. Perhaps it will serve as an interesting proof of concept for a possible solution to some of our concurrency problems. I wish I could put more work into a solution now, but at the moment I have little practicle use for concurrency of this type. Maybe in a while I can find justification to spend time on it.

Monday, September 18, 2006

Responding to Mencia About "Outdated" Technology

I wasn't planning to write this, but a quick segment on Mind of Mencia made me want to say something about his piece on tonight's show. It was a rerun, of course, but I don't get to watch the show regularly, although I am a fan.

Carlos Mencia blasted some technologies he deemed as outdated and berated anyone who still uses them. I normally enjoy anything he does, having been a fan since before his currently popular show. However, I just had to say something how much I disagree with him on all but one of these technologies.

  • Corded Phones
    Yeah, I can't go without a cordless phone in a large-ish two story home, but sometimes you can't beat a five dollar phone that you can't loose between the couch cushions.
  • Nintendo 64
    Retro gaming. I don't have to say more.

  • Fax Machine
    I hate fax machines, but the gas company won't let me e-mail them forms because, like it or not, its still easier than e-mail to get from deadtree to deadtree across long distances in a few minutes.
  • Pager
    I've actually considered getting a pager instead of a cell phone. There are times you must be reachable, but must not be interupted. A device that takes away the desire to answer it, because you can't, can be a limitation from more than technology (its limitations original source) to a limitation for convinience. Technology can be a burden, and sometimes we must dial back to reclaim our lives.
  • Cassette Tape
    The one thing I'll agree on. I probably have a few tapes left laying around, but if you dont replace or migrate the content soon, they'll degrade and be lost anyway, so its time to leave them behind.
Technology moves forward at ever-quickening paces, but if we turn our back on it, we loose more than just brick-sized car phones.

Sunday, September 17, 2006

Developing is the new Programming

I've been saying this increasingly in debates and discussions over in #python, where I frequent. It has to be something considered before, by more intelligable people than myself, but I don't remember hearing such a statement from anyone else.


The focus on the software world has drifted over the years from a focus on programming to a focus on developing. The difference is import and subtle. We can see this in the trend of popular software related books, the evolution of practices and languages, and changing patterns in the industry markets.

Books, Blogs, and Bantering

The landscape has changed on the kind of information pushed across the board to techie types. Where as reading the official specification of the C language was once a good software book, the best of today have no mention or dependance on any particular language. The emphasis is on books on development as can be applied broadly and generally, such as the excellent Prefactoring.

This can also be seen on the blogscene and what was made available online the most vigourously in the past. Resources are less and less often references to the boring syntaxes and APIs of programming languages and their libraries. More and more often, resources found talk about testing practices, organizational details, mindsets, and the best coffee to start your day coding.

Everything 37 Signals has to say is usually worth putting some thought into absorbing, even though they use a language I dislike for various reasons. They are the source of Ruby's recent spike in popularity, yet it is rare to see them mentioning anything about on Signal vs Noise. Instead, they opt for a kind of content that sometimes has nothing to do with development at all, yet can be applied directly to every line of code written in any language.

Is this an artifact of my personal interest and information source drifting, or a wider trend of focus across the board?

Backposted to meet my personal post-a-day deadline (but only by 34 minutes!)

Saturday, September 16, 2006

3 1/2 out of 10 Languages You Should Learn Right Now

Yes, I do believe a good developer has to know more than a handful of languages. Ten sure is an arbitrary number to pick, but whatever!


Read the list, I don't want to recap it.

Of the ten languages listed, I don't find three and a half of them compelling choices.
  • C#
  • Python
  • JavaScript
  • AJAX (this is the half)
Some people might find that an odd list. If you know a little about any of the languages, you might find it even odder. If you know a lot about the languages, you'll have no trouble agreeing, I feel confident.

C# is largely what Java should have been. Python is, of course, a fantastic language for most purposes. JavaScript is essential for web work, and web work is essential for getting the rent paid, these days. AJAX, much the same.

C# and JavaScript get some crap from the OSS communities. People look down their noses at C#, having such a distaste for all things wearing the brand of Gates. JavaScript can be an ugly, incompatable-with-anything mess, but it can be elegant and powerful, as well. Both languages can play very nicely with Python, via good web services and toolkits.

Python gets some cruft from the communities who normally work with C# and JavaScript. C# developers are often not open source fans, and they don't know what these weird hippies are talking about, because Visual Basic is just fine. JavaScript users often work exclusively in the web industry, where languages like PHP reign supreme. Python and PHP do not clash, so the JavaScript world hasn't had a great past with respect to Python. A lot of this is changing with the work being done at Mozilla getting Python support into the platform, adopting Python features for JavaScript 2.0, and PyPy being able to compile a python interpreter to JavaScript (crazy!).

And, quickly, why the other 6 1/2 languages just don't make the cut, or under what circumstances you might justify learning PHP, for example.
  • PHP
    So many of us hate it, but we work with it day in and day out. Sometimes you hate what you do, but you the bills it pays. Do many people enjoy PHP? No, but many of those people get paid for it.
  • Perl
    Very little perl these days is used off the web, and if you don't have the freedom to use something fun like Python, then PHP is even a more fitting choice than Perl.
  • Ruby and Ruby on Rails
  • Ruby on Rails isn't really a language, unless you buy into the idea that Ruby allows for use as a DSL engine. Ruby was around for a long time with no fan real fanbase, until Rails stole the scene and we are largely just waiting for the tidal wave to subside. Ruby isn't a terrible language, but I find it odd and there is little weight in a language that is effective carried by a single product.
  • Java
    The promise this language once showed has waned just in time to see Sun almost get the picture before its flame is extinguished. The niche Java fit is filled nicely with languages like Python and Ruby.
    There is a long held stubborn view in the Java world that all things must be Java. Look at their toolkits written entirely in Java, and even prefering low-level work done in Java, rather than wrap existing libraries. Much of this is in the name of portability, but in the end it just means that I have to wait longer for Eclipse to load than it took my machine to boot up in the first place.
  • VB.Net
    I have not used the .Net versions of Visual Basic, but its ancestor is not a fun thing to live with. Even given its improvements (so I hear), it looses a lot of its weight anymore.
    These statements should be taken with salt, because I haven't worked with VB.Net, but here they are anyway. Being a .Net language, it shouldn't matter what language you use, so long as it utilizes the CLR. That means you could even use C# or Python to write the code for Excel and Access, which was one day the only reason anyone had to touch Visual Basic. So with the advent of IronPython, and the myriad of languages that can take VB's place in any given project, where is the need for this language anymore?
As Johnny Storm says, "Flame on!"

PS - That was a punny reference to flamewars. ... get it? ahem. I'm leaving.

The Internet Flashed Me

IGN has a case of split personality. On the one hand, I don't have a new enough version of Flash installed for Firefox on Linux to play their videos.
On the other hand, if I want to watch some advertising, they wouldn't dare to stop me from doing what I wanna do!
Why do so many sites think I don't have a new enough version? Is newer Flash better? No, but the company pays for new versions of Flash Studio anyway, and it exports by default to a newer version, so why bother to change it? Thats like a couple mouse clicks or something! Jeez.

Friday, September 15, 2006

But, didn't PHP break the Web in the first place?

Rasmus Lerdorf is an opinionated man.

However, so am I, so I have some things to say about these particular opinions. Lerdorf is claiming the web is broken. I do not disagree. Lerdorf is claiming PHP is the cure, and I couldn't disagree more if he had written that statement on a shard of tin and jammed it in my stomach.

That is quite a strong disagreement.

I mean, didn't PHP break the web in the first place?

Right off the bat, I should note that I do believe PHP can be used well. Any language (almost) can be used properly enough that it can be a decent environment to use, so long as you follow strict rules. PHP is a great example of a language that promotes ignoring any rules. Following a good set of policies, one can develop well structured and elegant applications with PHP, but the fact of the matter is that the language does very little to promote anything in the way of good use of itself.

PHP might not promote bad coding, but it simply does so little in the way of promoting good code that it might as well provide facilities for plaintext passwords in query strings built in at the global level. There are too many aspiring developers finding their way to PHP, being drawn by the crowd, rather than the quality of the language. There is a critical mass of bad information about all the wrong ways we can do things in PHP, and none of them tell you its the wrong way. Are they evil? No, they just don't know any better either.

PHP is not evil. PHP broke the web with nothing but good intentions. PHP still broke the web, and only with a massively backwards-compatibility breaking (and that means no options to enable it back!) revisions to the language would anything be remedied. Even in that case, either everyone will migrate to other languages or the language would fork, because the only way to fix PHP is to become like Perl 6 and not exist at all.

I am beginning to sound like a language bigot.

Why Johnny Can Code

This is in response to the article Why Johnny Can't Code , over at Salon Tech.

I have heard several people already agreeing with the arguments against this article, so I know I am not alone. Although I completely agree that it is a very good thing for kids to have a quick and easy way to program on their computers, should they have the curiosity, I do not believe the author made very compelling points on the use of BASIC, or anything resembling it. Particularly, I think the author has far too high a reverence for BASIC and fails completely to see the damage the language can do to an aspiring developer, which I won't go into in this article. Conversely he seems to find languages that could fill the gap he says, namely Python, as somehow wrong for a job, which is entirely an incorrect idea. This is pointing from multiple vantages as being written by an unenlightened developer.

Yes, a fundamental understanding of how software works can be a good starting point for a life of technology as a hobby, career, or even just what burns the weather forecast into your toast. Love it or use it, technology with a background is more meaningful. In the past, this was intrinsically bound with BASIC, but that doesn't and should not hold true today. We have moved on out of more than disdain for our old friend and enemy. We have learned better.

Very quickly does the author of the original article mention and pass over Python and Perl as alternatives that don't mean some non-mentioned requirements for the job that BASIC filled in the past. Mentioning of putting his son to C++ after finishing with BASIC make me think that he is not a fan of this class of languages. There is some missing opportunity here, obviously. If he had brought his son to Python, rather than buying an old Commodore 64 out of frustration for finding a good BASIC interpreter for a modern OS, his son could have moved directly to more complex programs in a "basic" language, instead of learning to write the same "basic" programs in a complex language. Python is a fantastic educational language, but shines equally well when scaled to professional usage, which eliminates the bridges needed to cross from one language to the next through the levels of one's programming education.

Some shred of truth found, however, is the problems with bringing a language to the forefront of education in the was BASIC once was. There are too many languages that would argue they are superior. I will not deny that I think none of them old a flame to Python, especially in this context. However, I know I will be debated on this point. The fact remains that BASIC was a bad language, which served a purpose well, but has seen its day and is obsolete for a good reason. We have better ways of doing things, and that doesn't have to mean more complex languages, simply better languages.

David Brin , thanks for wrapping up Foundation , but look into the gaps in your logic about educational programming please, for your son's sake.

Friday, September 08, 2006

IronPython and a New Era

As many people have written about, IronPython 1.0 was recently released. There has been a lot said about it, and most interesting has been the response from many groups with little or no connection to the Python community. The fact that the language runs on .Net is more important than most Pythoners realize yet, and there is a predicted large number of future users of Python coming in from this new area of exploration. Many developers using C# or Visual Basic will be glad to move to a fun language like Python for at least part of their projects, and the integration with the CLR makes it flawless to mix with the rest of the components. It all reminds me of a line from Victor Vernado, the black albino comedian: its all the fun of dating a black man, without the disappointing look from your father. IronPython opens the door for a lot people who were unable to move into the community, due to personal or corporate barriers to leave the safety of the Microsoft Umbrella. IronPython is a bridge between two very strong and thriving camps of software development. It will be interesting to see who and what crosses this bridge and what new connections between these two arenas will rise.

I'm personally interested in IronPython for tapping some new ground in the commercial realm, but I won't speak of it much yet. I am also keen on seeing how well IronPython will work with and be succesful in areas like game development with the new XNA technologies, Mono on non-Windows machines, PyPy related compiling functionality, and Twisted on .Net platforms.

Its surely one hell of a Brave New World.

Tuesday, September 05, 2006

Give it a REST

I'm wrapping up a REST layer to the service backend I've been developing for my still-unnamed-employer (find out when we launch, real soon now!). I had never developed a service under the "REST" acronym before, so my boss gave me a crash course, I read some things, I thought I got it. REST, a buzzword in its own right, is like stapling smoke to water when you try to define it. That isn't because its vague, its because most of the people who talk about it don't know what they're talking about.

Maybe I'm one of them and I shouldn't be posting this.

REST is Not:
  • HTTP
  • XML
  • RPC
  • A protocol, format, or even much of a specification
Rest is:
  • An idea(l)
  • Agnostic on just about every specification associated with it
Requests on a REST Service are:
  • Atomic. No request relies on any other made before or after it.
  • Self Authenticating. Every request must include any credentials. See point 1.
  • Self Describing. This is most commonly XML, and sometimes people think it must be, but it can be anything. We use JSON.
Some of the most interesting things I've learned working with a REST service are the things that do not fit the model well. No model fits every need perfectly, and REST doesn't escape that fact, I'm afraid.

In particular, you are not always transfering a state. There is a distinct difference between state transfer and a request to perform some operation upon a state. Unfortunately, any ways around some of the problems posed are directly rejected by the rules of REST.

For example, say you want to provide as a service a simple counter. You expose PUT on /counter/foobar to register a new counter, and then GET on /counter/foobar will provide the current level of the counter. Following the rules of REST, how do you provide an interface to safely increment such a counter? We can not perform a GET and a PUT, because it violates that each request be self contained, and it will break when any other client of the service is incrementing at the same time. We need a single operation to alter the state, without performing a state transfer.

The best thing you can do is use POST on a resource, and transfer a request to increment. It seems to violate the tenents of REST that the resource you POST will not actually reside at some permenant location, as they are throw-away requests. You either have to live with a not-exactly-REST interface (but, isn't that it works the important thing?) or actually keep requests around for some time. Maybe put them at some location, where they can be checked for review of their status.

I don't know if this is helpful to anyone else writing REST services, but the information around isn't always accurate, so why should I worry if I am?

Friday, June 30, 2006

New Job, Fun Projects, and Amazon S3

I haven't posted in a while, but things have been going on. I thought I'd post about some of the more interesting aspects. I've recently began a fairly regular contracting deal with an interesting company, and I'll have to be a little vauge on some aspects, because of NDAs and such.

During one of my usual nights of aiding the Pythoners of #python on irc.freenode.net, I was discussing a project someone was trying to complete and the long debate about various routes that could be taken led to me being contracted for the job, which I've had fun with. I've been contracted to build a Fuse package, which uses Amazon's S3 as its storage mechanism. It is a fun system to work with, because of its simplicity and challenging limitations. For example, all operations are redundant, but non-atomic, because the same data could be changed at the same time, and its unpredictable how it would propogate across their redundant network. Mostly this hasn't been an issue, because you have almost the same trust in file locks on a local system anyway, and the only issues have been how to ensure integrity within directory entries and file node chains.

This aspect of the work is to be released under the GPL upon completion, and hopefully I can factory out some things that will be useful for other uses of the S3, which I've developed for the use in this project. I'll try to factor out modules for the following features:
  • Define classes that represent a type of data stored in an S3 entry
  • Easily define meta-attributes, with coercers and defaults
  • Unique IDs generated for entries
  • "Sub-Entry" concept, where one entry is owned by another
  • Caching of data both in disk and over memcache, with an open API to implement other cache-types, like local-memory caches, or even other web services.
  • Node entries, which can span data across multiple entries for more efficient (and cost effective) reads and writes that do not involve the entire data buffer.
  • Test facilities for both BitBucket (a Python S3 access package I use) and Python-MemCached, which I use for offline testing. Both mirror all the functionalty (read: most) of the related projects, so they can be tested against without actual network use.
My work with this project has led to the beginning of a long-term working relationship with the company, which I am very excited about. I can't talk about the specifics of the work I will be doing, until the company launches in a few months. As soon as that happens, I'll be blogging extensively about some of the aspects I can devolge, and of any additional software that might be released freely (I don't know if there will be any).

If you are interested, look forward to the S3 packages I'll wrapping up this weekend. Hopefully, someone will find them useful.

Friday, June 16, 2006

Pandora Pop-Up White Bars Bug the Crap Out of Me!

Pandora Internet Radio - Find New Music, Listen to Free Web Radio

So, they added this weird white-bar to top top of pop-up options, and I wrote them to complain because I find it very ugly and unintuitive. I would expect a titlebar-like control as such to be used to move a widget or window, and the X at its left to close the widget just enforces that thought, even tho the bar seems to serve no purpose. Pandora's people tell me it was for some very soon to come updates and that I would see why soon.

So, this morning I see they updated the interface again, and there are auto-appearing controls to vote on songs instead of clicking for the menu on each song. That means I will almost never ever see that popup. Is that what they were talking about? Am I missing something?

technorati tags:, ,

Blogged with Flock

Thursday, June 15, 2006

I've Joined the Flock

Flock — The web browser for you and your friends

So this Firefox-based browser is pretty cool. It integrates del.icio.us, Flickr, and blogging. It seems to be strongly related to Yahoo!

I'll probably use this for all my casual browsing, keeping Firefox around for development, only.

technorati tags:, , ,

Blogged with Flock

Wednesday, May 31, 2006

XML versus Binary Document Format Debates

Responding to Sean McGrath on his recent post about XML-vs-binary document formats:

Look, its just like I was trying to say in #python@irc.freenode.net the other day: There is nothing wrong with a format being binary. There is no virtue to be found in every byte of a file being interpreted as a textual character (or part of one) that represents your real data. "There ain't such thing as plain text," says Joel Spolsky! There isn't any difference in interpresting the binary as text than just interpretting it directly as your data.

You can easily have XML formats as undocumented and inconsistant between versions as any binary format, but you get the added benefit of extra processing overhead, bloated filesizes, and limitations on structure and performance (try keeping efficient on-disk indexes into an XML file up-to-date).

I do believe text-based formats and XML has its place, but these places are limited. I would have much perfered an opening and standardization of a relational-based format the way Word documents worked internally before Microsoft was bullied into an XML-based format.

I'm wondering if I actually know Sean and didn't realize it, because his points are exactly those I was arguing against in #python the other day, and so I wonder if someone I was arguing was McGrath by a different name. Knowing your strangers is a great gem of the internet-age, isn't it?

Thursday, May 25, 2006

DeferArgs 0.4 (Busy day!)

Continuing discussions in #twisted about the benefits and complaints of my deferargs drove me to add more things and release pretty often today. This is version 0.4. New is tests and ability to move the functionality to the callsite, so you can do this:

def printArgs(*args, **kwargs):
print args
print kwargs
deferargs(printArgs)(10, defer.succeed(20))


Also, and I don't know how useful this will be, you can not define specially handled argument types, such as lists that might contain deferreds. These are optional, and not enabled by default. Lists are the only special type handled so far. Use it as follows:

@deferargs([list])
def printList(l):
print l
printList([1,2, defer.succeed(3)])


I might add dictionaries, sets, and tuples to the next release.

DeferArgs 0.3 by Example

DeferArgs 0.3, just a few hours after 0.2, is now released here. The only update is support for attemp/catch-all blocks without any @catch(type) handlers. In response to some people in #twisted@irc.freenode.net, here are examples that show what can be done more clearly.

To write any function that can accept deferred arguments nicely:

@deferargs
def printArgs(*args, **kwargs):
print args
print kwargs


To add psuedo-exception handlers and a psuedo-finally block to a @deferargs decorated function:

@deferargs
def printArgs(*args, **kwargs):
print args
print kwargs
print "This one is important! ", kwargs['important']
@catch(KeyError)
def onKeyError(e):
print "There was no important kwarg!"
@catch()
def onAnyError(e):
print "Crap!"
@cleanup()
def _(r):
print "Every thing is done."


There is also a model for a psuedo-try block, which basically decorates just calls the function immediately when you define it and its catch/cleanup handlers:

@attempt
def this():
assert False
@cleanup
def _(r)
print "And thats it."

DeferArgs supports asyncronous try/except/finally constructs

A new version of DeferArgs is available at the cheeseshop.

New in 0.2 are the attempt, catch, and cleanup decorators. These mirror the functionality of try, except, and finally. Any deferargs decorated function can be followed immediately by any number of error handlers decorated as @catch(ErrorType) or @catch() to catch all, with the same semantics as except clauses if they were after a try block that surrounded the code in the function. The catches can all be followed with a cleanup decorated function, which will be called when everything else is done, errors or not.

There is also a convinience decorator called attempt, which automatically calls the function once a @catch() decorated handler is defined (signifying that all error handlers for it are prepared). Of course, calling it only really creates the deferred that fires when its arguments are ready, so you dont have to expect it to run for real untl after the attemp/catch/cleanup is completely defined.

As always, comments are welcome!

Wednesday, May 24, 2006

DeferArgs lets you write syncronous looking functions that really aren't!

So a little side project for my own uses provided a simple decorator that lets me write a function I can pass deferreds and regular arguments to, and have the function return a deferred that fires when all of its deferred arguments are ready and the function has processed them. Some example usage:


@deferargs
def printArgs(*args, **kwargs):
print "Positional Arguments: ", ", ".join(args)
print "Keyword Arguments: ", ", ".join("%r=%s"%(k,v) for (k,v) in kwargs.items())

printArgs("foobar", baz=someNetworkRequest())


Really basic, but it can prove useful for a large portion of Twisted code you might write. I'm planning to add some semi-evil way to do something that looks a lot like a try/except/finally block but is actually (obviously) not, and works with any errbacks from deferreds within the try-like block of code. The reaction has been interesting. I've had some people stand up for the idea, which is similar to
defgen
, and others who think it is a bad idiom that is dangerous to encourage.

Using this kind of abstraction over asyncronous code, you do have to be careful to remember what is asyncronous and take consideration that your code won't run until all deferred arguments are ready, even if some parts could be run with only some of the arguments. In those cases, however, you should just break up the function, and I'd like to note that you can make the same mistakes using deferreds and the like directly, so I don't really see it as an issue.

If you want to do any error handling for the moment, you need to attach errbacks to the deferred from the function call. I want to work in my semi-evil error handling soon, because my goal here is to hide the fact that there are deferreds as much as possible, but for now this is just fine, and I've already had use of it myself.

You can get it from the Cheeseshop, so check it out there now and place any comments about it here.

Filed in:

It's a Boy!


This is a little late, because I've just been so busy, but my first child, my son, Caelan Mathew Spealman, was born on May 13, 2006 at 11:45, just fifteen minutes before Mother's Day. It was the most amazing expirience in my life, to go from a couple to a family as my son was brought into this world, and honestly, this is just so cool! I can't wait to teach him to program. I might be showing how much geek I am by saying this, but my one-and-a-half week old son already has his own computer.

Wednesday, May 03, 2006

Early Morning

Woke up at 6 AM, made a pot of coffee (with my patent pending technique of adding sugar and spices directly into the grounds before brewing), and caught up on my 600+ unread messages in GMail. Really made me want keyboard shortcuts for labelling, but the only Greasemonkey userscript I found didn't seem to work. It was supposed to let me hit the 'l' key and type in an autocompleted label name, but I did it the new old fashioned way in the end. Anyone know of such a userscript that does, in fact, work?

Wednesday, April 26, 2006

I Can Work in Synergy

Just wanted to quickly link to the Synergy project, which I'm now using. Wow, it is so cool. Anyone who has multiple systems at their desk or sits a laptop beside their desktop's monitor should really take a look at this. Dual screen is one level of cool, but being able to move my mouse across multiple screens running multiple operating systems is a blast, and productive too. Copy and paste and keyboard and mouse can all be shared. Now, I'm very interested in if windows could be dragged between the screens with this one day...

Python AST Manipulation for Transparent Defering of Calls?

A lot of work has been going on with the Python AST and being able to manipulate it for more runtime uses. This is a generalized suggestion for something we could do in the Twisted community to utilize this.

Given a simple function like this:



def processPage(url):
d = getPage(url)
d.addCallback(cb_processPage)
d.addErrback(eb_processPage)
def cb_processPage(page):
print page
def eb_processPage(error):
print "Could not load page. Error: ", error

We write like that, but what we really mean, and just need to express in a more difficult manner, is:


def processPage(url):
try:
print getPage(url)
except LoadError, e:
print "Could not load page. Error: ", e

What I want to know, is can we take the second example and process the AST branch to produce the first example? Lets step through and see how it would work. First of all, we need to know what is deferred. A simple way would be to check every function return and determine if it is a deferred, but there may be more efficent methods we can discover later. For any possibly deferred call, the expression it is a part of can be refactored out into its own function, as can the exception handler code. When a deferred is detected, the callbacks can be attached and if the operation is not deferred, the callbacks can be used directly. This might even bring about something slightly new: optionally deferred operations. Instead of using defer.succeed and such, returning a deferred could cause code to handle the deferred properly, which would otherwise act in a normal syncronous manner. Depending on just how the AST stuff progresses, maybe the returning of the deferred could trigger the inspection of the calling function to inject the deferred handling code, so only functions that ever get a deferred will need processing.
I write here about programming, how to program better, things I think are neat and are related to programming. I might write other things at my personal website.

I am happily employed by the excellent Caktus Group, located in beautiful and friendly Carrboro, NC, where I work with Python, Django, and Javascript.

Blog Archive