Sunday, January 29, 2006

Information Overload, Needles, and Haystacks: Part One

So this is a little more subjective than most posts I want to make here, but it still fits better than on either of my other blogs. Do I have too many blogs? That will have to do, until I get around to utilizing more proper and tag-supporting blog software. Anyway...

The internet is a amazing. The myriad of information is just fascinating, and would be overwhelming, if it were possible for any one person to truely grasp just how much data there really is out there. Not only can we not possibly grasp it, but there is no way to really utilize it. We make baby steps, every day, to obtaining more and more of that information. Or, rather, we make steps in obtaining more specialized and exact pieces over a wider selection of that information. The ammount of information we can take in has pretty much maxed, I think. All we can do now, is utilize that limit by being more intelligently selective of what is available for us to take in.

The Needle is something you want, and maybe you know it, or maybe you have no idea. The Haystack is, of course, the internet. How do you find the Needle in the Haystack, especially if you don't know you're looking for it? We need to understand what form the Needly may take, because it could be a lot of things. Is it an informational piece of text, a multimedia entertainment stream, or maybe a small tool you could utilize for your own uses? It could be any number of things, but maybe the most important part of understand what it is, is understanding that you might not know. Convergence of all the types of Needles is a necessity to finding what you want, because so often, we really don't know what it is we are looking for, even vaguely.

Next part will dive into how we bring all these different types together, and more on just how small those needles can get.

pyx Introducing NevowPavel

I just found NevowPavel, Donovan Prestons year-long-so-far project that really takes everything Nevow has been reaching for to another level. JotLive has always been a pretty cool thing, but I honestly found it more gimmicky than anything else, and couldn't see myself finding it all that useful. Pavel, however, is jaw-dropping.

Go and watch the webcast, as it is really the best way to see what this is capable of. I would love to play with this sometime soon, and maybe try my hand at creating some object types to play with in Pavel. I'm already formulating ideas to pitch to Donovan, including container types (which I imagine might be anticipated already) and possibly configurable handlers for incoming objects passed through portals within certain containers. Just think of the usefullness of a "Reviews Pending" space that had a portal configured to toss everything coming in into a container that listed everything for review. What about one-to-many portals? I don't know where they would go when followed from the one, but maybe they would only be one way. How about multi-part text objects? A lot of this is probably already in his own mind, but I can't wait to see where this progresses.

Recursion Bites with Complex Nevow Pages

There is a recently discovered issue with Nevow's Athena LivePages. LiveFragment nesting fails if the nesting gets too deep, due to repeated cloning of the contexts, including the full chain of parents back to the root of the document. This was hitting the call depth limits in some tests idnar was working on. I found this to be a little worrysome, as my designs for the current project included some relatively deep nesting of LiveFragments; at least, as deep as the tests that found the error. Eventually, contexts are to be removed from nevow entirely, as I understand it, but this is far down the road. A temporary solution was needed, besides just not using so much nesting. I decided that instead of redesigning the system I was building, I would fix the bug, and I have. I posted Trac Ticket #602 along with a patch that fixes it.

Recursion is a very useful software construct, but sometimes it can bite you in the end when you don't even expect it to. There is usually a way around it, but I do wish there was a more explicit way in some language to express a construct meant to avoid recursion. All I had to do was traverse up the parent attributes and create a list of all the parents, in order. I took each of these, and if the clone method was the same as our own, so we know for sure how it works, I just manually clone it and make it the parent of the clone of its original child. This way, no recursion is ever needed between parents being cloned, unless a different context class is being used, which also redefines the clone method. If that becomes a problem, it should be fixable as easily.

This may seem a little hackish, but its a common enough issue that I am really surprised there has never been a really good recursion alternative in any language. I'm not sure what form this would take, but in places it would be nice to say "this works like recursion, but without recursing".

You are 100% accountable for your source code

You are 100% accountable for your source code: "It is just amazing how the mainstream software development industry is behind the times in comparison to other more advanced disciplines; the below fragment by Mary Poppendieck summarizes what I am writing about."

Caught this through the feed of MSDN blogs and it hits a good point. Specifically, every line counts and every line is one more line of source you need to keep correct in relation to every change made anywhere else in the code. Now, this has some other points to consider, such as private constructs being a little less dependant on other code, but on the whole it is a pretty true and simple statement. Keep your source code lean, and don't add what you don't need, because every line you write is another line you need to worry about when you make any change anywhere. Of course, this is well known in most senses, and nearly all techniques of intelligent code design revolves around keeping all these increasing lines under control and protected from one another. However, no matter how good we think we can deal with it, we can never forget the simplest way to keep the code correct: less code.

Saturday, January 28, 2006

Nevow: Post-Render Injection of LiveFragments

In part of my current project, I've tried to make things very spiffy and use that nice AJAX stuff we all hear so much about. I do this through the wonderful Nevow, which makes much of the work a breeze. Some of the things I wanted to do, however, it isn't there on yet, so I had some more work on. Here is a story.

Nevow has a concept of a LiveFragment, which is a piece of a dynamic webpage that can be placed into a LivePage and attached to a counter class in JavaScript. The Python and JavaScript classes on either side of the pool are able to call methods between one-another, to facilitate anything you want that can be done in either language. All of this works through the transport system of the LivePage to communicate back and forth. Some recent changes allow easier post-render initialization of new LiveFragments, but it isn't perfect yet.

This is how is basically works, and some of this may change soon, as Nevow is a work in progress.

First, we need to take our LiveFragment on the server-side and convert it to something we can pass over to the browser in a remote call. We'll just render the fragment into XML, which makes sense. This is called flattening and is done when Nevow fragments are rendered anyway. We do this after creating and setting up the new LiveFragment. In this context, we are adding the new fragment as a child fragment of the current one.

This assumes you already know how to use LiveFragments, and write the jsClass to go along with it.


f = MyLiveFragment()
f.setFragmentParent(self)
html = flat.flatten(tags.span(xmlns="http://www.w3.org/1999/xhtml")[flat.flatten(f)])
d = self.callRemote('addChild', html)
d.addErrback(lambda m: logmsg("Error adding child: %s" % m))


Now, we need to handle this on the client-side with a JavaScript function implementing this addChild method of our jsClass.


MyModule.MyClass.method(
'addChild',
function (self, html) {
posting_list = Nevow.Athena.NodeByAttribute(self.node, 'class', 'list_node');
posts = Array();
var runtime = Divmod.Runtime.theRuntime;

for (var p=0; p<posting_list.childNodes.length; p++) {
posts.push(posting_list.childNodes[p]);
}
runtime.setNodeContent(posting_list, post_html);
for (var p=0; p<posts.length; p++) {
posting_list.appendChild(posts[p]);
}
});


Now, this is the meat of the deal, so I'll go over it in a piece by piece. The first thing we do is find the node we're going to add a child to. In this case, for some extra excitement, we're going to insert the new child at the beginning, instead of just tacking it on the end. It turns out this isn't the easiest thing in the world, but it isn't difficult by a long shot. We create a new array, children and iterate over the nodes, adding them each to this array. At this point, we use Divmod.Runtime.theRuntime.setNodeContent() to replace all the children of the node with just our new one. Now, we iterate over the list we made and append them all, in order, back to the node. This inserts our new node before any of the others. When we insert the new node with setNodeContent, it is parsed internally and this process requires it is XHTML 1.0 and that the namespace is declared as such. However, the node you flatten is actually prepended first with a script element, in this case, which initializes the component on in the browser, when it all gets added to the document. This will cause some errors parsing, so we wrap the whole thing up in its own span tag and give it the right namespace. Keep this in mind when working out any DOM logic, that you're LiveFragment's root node will be the childnode of this wrapping span. Of course, you can use any type of element to wrap it together, as long as its one top-level node of the XHTML 1.0 namespace.

To summerize, we flatten our LiveFragment. We wrap the script node and the live fragment up in something of the XHTML 1.0 namespace. This is passed in a remote call to a javascript function. The javascript generates a DOM from this and inserts it on the client-side, and the script node initializes the fragment for us. After this, the LiveFragment and its widget can make remote calls back and forth as normal.

This technique is very useful in creating dynamic pages. In my case, I'm using it to allow new postings to a page to appear immediately, and for those postings to be immediately editable through an embedded LiveForm. It can be used for a lot of things, and helps seperate your logic and design.

I don't know if this will be helpful to anyone, but hopefully so. Please let me know of any errors.

Thursday, January 26, 2006

Teaching a Friend: Can Twisted be a good First Impression?

A friend of mine has expressed an interest in learning to program, to dive into the dream of creating the next great video game. I've decided to tutor him, and teach him some things, as we make a small game for his first jump into code. I've made a decision to go about this differently than most.

I'm teaching him using Twisted.

Twisted is an asycronous programming framework, and while it works great for games, some would question my use of it on a newbie. I actually think that the fresh mind might absorb the concepts easier than an otherwise tainted programmer's mind. We'll see, I suppose.

Sunday, January 22, 2006

New Programming-Centric Blog

I've decided to start a second blog, where I will post only about programming-related topics, and I will leave my other blog (http://ironfroggy.blogspot.com/) for any of my non-programming posts. I think this will help me in the long run. Too often, when I want to post something, I wonder if its too geeky or too non-geeky, so obviously there is an issue there.

So, I'm trying to make a career move. I'm tired of programming being a hobby; I want it to be a career. Right now, I don't even have a career. I only have a job. This choice comes at an interesting time with a reason that both gives me more reason for this Big Move, and more to worry over about doing so. My wife is pregnant, and our first child is due in May. When Caelan is born, I want to be here with my wife and my son! Of course, that means I will be worrying about supporting a child, now. Although I can make more money doing this, the stability of it is not as secure as a normal job. I've got my first contract right now, but I can't go full-time with this yet. I'm working out the details of a possible medium-term second project, and if I get this I may make the plunge.

So, in this first job and the one I'm pitching right now, I am using the nifty stuff from Divmod. Some great stuff, and wrapping my head around it is (kinda) fun! I'll be posting some "just so you know, without all the trouble I had" as I go.
I write here about programming, how to program better, things I think are neat and are related to programming. I might write other things at my personal website.

I am happily employed by the excellent Caktus Group, located in beautiful and friendly Carrboro, NC, where I work with Python, Django, and Javascript.

Blog Archive