Developing Upwards

I couldn't think of a better title that fit my "How To ..." pattern. The point is, I wanted to make a response to the 37signals post that I found a little harsh. Sure, if you were able to build your company up without investors, that's a great thing! It doesn't make it a terrible thing to get a boost in the early stages or give you a license to insult people trying to pay the bills and put children through college. Making great products is something a lot of us aspire to. Frankly, that simply isn't all of us and there really are good developers out there who are still only in it for the money. I don't know that is the case with Mint.com, but neither does anyone over at 37signals. Belittling them for taking a quick-cash option assumes a lot that may just be completely wrong about the intentions. Now with a chunk of change, maybe the founders are planning to jump ship in a couple years and self-fund their real dreams. On the matter of start-up investment it...

I want to say a few things for my own benefit. Maybe that's the only thing I do here. As always, I hope something I have is useful to someone else. In this case, if you're in any position to deal with a big surge on a small site, you might get something useful, or at least enjoy, what I have to from my experience getting a bump from some guy named Mike Arrington with a little blog called TechCrunch. This is about reaction and what was the right and wrong way to react to the impact of a weeks traffic in a couple hours. Should natural means have brought our typical traffic to these levels (time will bring this) the means to handle it on a day to day basis would have been put in place. The sudden increase began to timeout our FastCGI processes and this was alerted to me quickly. I confirmed this and my first response was to initiate a restart cycle, restarting each process in turn, which did nothing to help. I brought up a new instance on EC2 and prepared to roll it out as a new p...

When I complained about the problem, I promptly outlined some ideas about solving it, vaguely. Now, I want to narrow that outline into systems I actually use. I do most of my work with Django, some hobby time is spent with App Engine and Twisted, and I enjoy Amazon Web Services, so I'm thinking from these perspectives when I approach this. Parts one and two were broad, but some of this might only apply to fewer of you. Either ignore those or adapt to whatever you use. Django's cache layer sucks. Simply stated and simply true. Any time I decide I can cache something, I should ask myself if I could have built it before I even had the request in the first place. Doing that with the template caches simply isn't possible. It should be possible and it should be the first path you take, instead of forcing us to go out of our way to do the better thing. Anything I might want to cache, I also might want to be sure I'm not doing in more place than once, and forcing them inline i...

Distributed version control is a good thing. I've started wondering, abstractly, removing the middle word of that phrase. In other words, how are we being affected by "distributed control" and how will the landscape of free software politics change as it becomes more predominate and we all become more comfortable with it? Even centralized version control began the distribution of control. At least, it made it easier for more than one person to control the changes of a codebase. In old days of e-mailing patches around, it was pretty much a requirement that a single person be responsible for the merging of patches into any single codebase (or any section of that codebase). Source control allowed multiple developers to commit changes and began to put less burden and less power in any one person's hands. Anything that makes the submission of new code easier is going to thin that power even more. When anyone can come along and submit changes to change functionality or ad...

I did my best to outline the problem in Part 1. Now I have to stand up and propose some kind of solution. Otherwise, I'm just complaining and contributing nothing of real value. Our frameworks make certain things easier. They don't provide tools to help us with other things. For some other set of activities, they may actually prohibit. The problem here is a combination. Django makes it easy to query your database and wrap functionality up into re-usable template tags. While I'm thankful for that, I am also realizing that ease of one thing can prohibit another. When one path is made easier it creates the perception of greater difficulty in other paths. I think this is why, when our web frameworks give us all these tools to response to a web request, we completely lack in everything we could do aside from that request. How can we make it easier to work outside the web request? We need some idea of what working outside the web request means. We also need to define these in ter...

Something is bothering me this past week. I've been taking some stabs at reducing the maximum render time of a site, when the caches are all empty. I cache certain components and queries, but when the caches are primed the render time is under 500ms and I think that's pretty good. That worst case senario, however, is just not acceptable. Worse than a couple seconds. That isn't time that should be taken. I dug in and found a really bad pattern. It isn't hard to make a page faster, but the default is to be as slow as possible. We have to understand this pattern. I am looking at this in relation to Django, but I have a feeling there are similar patterns other places. The common tagging application is a good example. It makes it really easy to tag objects, count them, query by them, and build those clever little clouds. You're given lots of new wrappers for all the common tag-related queries you'd need to do. This may be a source of the problem. We've gotten...

As the web becomes more "2.0" we're collapsing collections of pages into fewer and even single, more dynamic pages. Instead of ten pages of event listings, we might have page that loads further items dynamically as you scroll. The state that was once static to a page is now loose and can alter in the lifetime of a page, which grows longer every day. Parameters of the page state have always sat in either the path, in URLs like http://myblog.us/post/how-are-you or the querystring in cases like http://www.coolstore.com/view.html?product=4356 . Neither approach works when those parameters are changing for the life of the page, and where a single URL needs to be able to represent these multiple parameter values at any time. In most uses so far, the bullet is simply bitten. The user can browse to your site and click around, and if they bookmark or send the link to a friend, they'll always come back to the front page, because the state of the page is no longer held in that ...

I had some down time today to relax, and in true obsessive fashion I spent it coding for the hell of it. I got something in my head and whipped up a demo of the idea. Do you ever need to let someone select a range of things? Maybe they need to pick which and how many items to show in a search result or which letters of names they want to see from an address book? I wanted to allow selection of both "what" and "how much" in one click. click for demo The range being selected from can be anything: numbers, letters, weeks of the year, etc. Users can click among that like a list of a page numbers, like they would expect. I think this would work well in situations where you don't need the entry to be exact, although it can be used for precise entries. Multiple quick selections would also be easy here, maybe quickly changing the range you're viewing in an analytics app. I'd also like to look at adding a "zoom" feature, so that one selection fills the ...

Youtube, what the hell is up with your upload APIs? Here we are, hacking along and being all "Hey, we got accounts syncing and thumbnails popping up and videos getting attached to blog posts and its nifty-pie, oh yeah." when we make the move to be even more web 2.0y and add full authentication with youtube accounts and integrate the full upload cycle into the media selection. We take some little video upload tests and we're happy about that. Oh, that was a nice milestone to hit in FogBugz , I tell you what. The week passes and suddenly I'm asked why all the video uploads keep failing. All the usual things are checked, but the upload tokens validate and the headers are correct. All our tests are within the upload limits, too. These aren't friendly "Hey, I really hate to tell you this, but the video you just uploaded didn't go so well. Can you try it again in a bit? Thanks"-errors, either. These are "Fuck off. I just reset your HTTP connection, bi...

While ignoring completely that I was promised access to the FogBugz OnDeman 7.0 Beta program and just forgotten somehow, I'm going to come say that one day with the official release and I'm more excited than ever to be a FogCreek customer. Yes, I am still a card carrying free software nut-bag . I'm absolutely sure certain individuals will get grated at me, again, for not using Trac like a good geek. Call me a fanboy, but boy-oh-boy is this a sweet release. It is a shining example of knocking a release out of the park and impressing everyone (who could be impressed at all, and thus disregarding those who will never be impressed by a commercial, for-profit bug tracker, ever, no matter what, not in a million years). The experience is absolutely slick. Faster, brighter, shinier. Packing new features, improved features, and bug fixing in a new package is a great way to make the functional improvements stand out. Even if we have a good design, any product should take a note fro...

Or, the working title: How To Look Down At The Tower of Turtles We're a recursive bunch. We're more repetitive than . There is no shortage of writing that its caches all the way down or that we're repeating the mainframe/dumb terminal era. I have an argument that our entire profession is hinged on repeating ourselves. Repetition is in the DNA of what we do. Software is the ultimate commodity, approaching zero-cost production. Solve one problem and the solution is applied to a thousand problems. Generalize and solve a million. Everything we do is repeated and is about repeating things. At the core, we're just moving little bits around and we repeat that action over and over, with very slight alterations. We abstract the repetition, and then we repeat the abstraction so much we need to abstract that. We could continue to make individual observations, like the mammoth stack of caches every bit goes through or the abstractions we build up over and over on top of our languag...

How To Teach Software Development Introduction Developers Quality Control Motivation Execution Businesses Students Schools What's the Point? Some opinions, while held, are held softly. I believe the understanding is the opinion changes nothing and you aren't doing anything about it, so giving a damn is pointless. You may call it apathy, but I call it misunderstanding the nature of information. Information spreads from those who have it to those who do not and those in agreement grease the wheels of that distribution. Of course, there are good developers who don't care if there are bad developers. I'm not convinced they're still reading, at this point. If they are, then the reasons we can make a difference should help convince you to care about that difference in the first place. The more widely held the beliefs that we can and should do something to improve the quality of this industry, the more likely anything will happen. You might not be lecturing in the class...

How To Teach Software Development Introduction Developers Quality Control Motivation Execution Businesses Students Schools Doctors and lawyers in the United States have the American Medical Association and the American Bar Association, respectively, and surely have analog organizations in other countries. As representatives of their professions, they work collectively with their colleagues at a reasonable and useful goal: quality control. It is in the interest of doctors that their med students are not idiots. It is in the interest of lawyers that their opponents are not (more) unethical. There are other, perhaps less admirable uses of these controls (too many doctors would lower all their salaries). There is no such quality control in our industry. We have individuals at the lowest end of the ability spectrum and at the highest peaks of skill, and, we have the teaches of many years of expensive higher educations on some of our shoulders and only the passion of self-teaching for oth...

Yes, this is a screenshot of a screenshot. Stick with me, but I really do has a point to this! I have to admit, publicly, that I clicked the button. The one in the screenshot. The one that isn't a button, just a PNG image. I should be glad it wasn't a pop-up! I realized my mistake at the moment I was clicking on it, but it happened to fast to stop. I had to sit and think for a moment. Why did I do that? It drove me to write this pretty immediately and do a couple mock ups for solutions. I never want to let my users loose information or control over it. That is, we don't want them to OK a message away and neglect to actually read it and we don't want them to click "send" before they're really, really ready to confess their never ending love to Glenn Beck. Those are two distinct safety nets. Information the user missing for being click-happy and actual actions within the application they might have wanted to avoid. Any reversible actions, like closing a dial...

The two recent posts about some individuals' choice of GPL versus others' preference for BSD and MIT style licensing has caused a lot of debate and response . I've seen everything as an interesting combination of very important topics being taken far too seriously and far too personally. All involved need to take a few steps back. For the uninitiated and as a clarifier for the initiated, we're dealing with (basically) three categories of licensing when someone releases software (and/or its code): Closed Source. Easiest to explain, because you just get nothing. GPL. If you get the software, you get the source code, you get to change it, and anything you combine it with must be under the same terms. MIT and BSD. If you get the software, you might get the source code, you get to change it, and you have no obligations about anything else you combine it with. The situation gets stickier when we look at those combinations and the transitions between them. Use GPL code with ...

After a couple days trouble with the Youtube Data API and the provided Python wrappers around it, I thought it would be good to collect my findings on what works and doesn't and to fill in the gaps that I see in the docs. I really hope this series will be useful to some others in my position. Some Doors Are Locked and Some Doors Are Ajar A lot of the APIs use requires no authentication, not even a developer key. This makes a lot of the most common, read-only integrations a snap. However, I think this makes it more difficult to adjust when the need to authenticate for other integration comes along. This did some damage to my schedule, so I'm going to help others avoid the problem. Public operations are simple. Youtube gives us resources in the form of feeds and images and other things at API locations, like http://i.ytimg.com/vi/ FedVhnHYn-Y /0.jpg to get the first thumbnail of a video. Just plug in your video ID and go. When you get into the realm of authenticated requests, yo...

In the spirit of the old name of this blog, Ranting Techno Rave, this is a rant about a personal experience. This happened in the line of duty, so it is on topic. Has anyone else dealt with this kind of thing? Tell me about it. This title is purposefully "provoking" and if you're the one I'm talking about, you know who you are . This might even apply to you if you're someone else with the same kind of behavior. Maybe you know or have to work with someone that exhibits the particular personality traits I've had to deal with. In whatever way this applies to you now or in the future, beware as much if you are this type of coder as if you have to deal with one of them. The lone ranger was a terrible cowboy. Assertive personalities are important. They point out mistakes, instead of allowing problems through inaction. There is an issue of tact , as a line one needs to watch as they walk the road of the assertive. Code review requires assertion as you tell someone, ...

How To Teach Software Development Introduction Developers Quality Control Motivation Execution Businesses Students Schools Education is broken. Education about software development is even more broken. It is a sad observation of the industry from my eyes. I come to see good developers from what should be great educations as survivors, more than anything. Do they get a headstart from their education or do they overcome it? This is the first part in a series on software education. I want to open a discussion here. Please comment if you have thoughts. Blog about it, yourself. Write about how you disagree with me. Write more if you don't. We have a troubled industry. We care enough to do something about it. We hark on the bad developers the way people used to point at freak shows, but we only hurt ourselves but not improving the situation. We have to deal with their bad code. We are the twenty percent and we can't talk to the eighty percent, by definition, so we need to impro...

Software Development in Really Big Steps How To Work a Sigmoid How To Work a Sigmoid - Part Two The last time I wrote about the curvature of project estimations, I was just speculating. Since then, I've discovered that FogBugz does track estimation over time, with a daily estimation record, and offers a graph of the 0, 50, and 100 percent estimates over time. I've been watching this develop for a small time, working more with tracked estimates, and I think some expansion on my thoughts is ready. You can see my own estimation graph here and it demonstrates exactly what I predicted. I suspect a more complex plotting of points would emerge with the length of the project, but I have a few curiosities about how this would expand over time. The basic prediction of a generally unchanging estimation from the start, an increase in the estimation's growth in the middle, and ending with a calming and final flattening on the systems best guesses, as you slow down how many cases you fi...

We're all "making software," but what's that mean? There is no shortage of resources on writing code. Debates rage on about this library and that, emacs versus vi, or nix versus windows versus osx. How much of it matters? We're arguing what car dealership gives us the best deal, automatic versus manual transmissions, and shades of colors to promote the best feelings when you see that shiny new car. Great, you've got the nice car (we all do), now you've got to drive the damn thing and keep it maintained for its lifetime. Who is paying attention here? We spend thousands of hours discussion how to write software and millions of dollars helping us do it, but most of us have no clue how to keep that code around and get it in the hands of users. I won't make this a post about "The Cloud", but I will say its largely successful, because it solves a problem most developers either ignore or are never properly exposed to. I won't blame PHP, but it f...

UPDATE: Fixed 'Response' to 'Respond' in title. Sorry about that. We all have to do it, so I might as well take my turn. First impression: no surprise here. There are expectations in two forms here. We can expect certain things to come of this and we can expect certain things to disappoint us about this. There is a third, external expectation that techies will divide into a camp of people who think its Rilly, Rilly Important and a camp who thinks you're all wasting your time . I mean, gosh, its almost like this is exactly like any other topic we split down some arbitrary middle about. Get over it. I Expect To Like: Cheaper netbooks Installing Chrome OS on old hardware I Expect To Dislike: Feeling like I have an OS that won't let me install anything but a browser Not being able to install Android Apps Not being able to run real Chrome on Android Having no way to persist the state of a Javascript VM, so that I can close applications or save memory on long run...

Some suggestions to improve a content voting system sparked some thoughts about the idea and I wanted to write them down to record my thoughts. The initial move was to remove down voting. No one uses it and negatives are, well, negative. So we'll drop "vote down" and replace "vote up" with "like", because what is more friendly than liking something? You know, its like you're in first grade and the article is that cute girl eating paste. At the same time we were discussing sorting. Everything is chronological, but people might want to see popular things. Is it popular because people vote up on it or because lots of people read it? Of course, lots of places weight these today (like Reddit), so that was discussed. Third, given the relatively higher traffic we're seeing on video content (duh, Youtube generation), adding a second row of video thumbs to the front page makes sense. I also rolled the idea in my head of adding a little randomness into t...

Today was a very troubling and frustrating day for both myself and one of my best clients. This is my declaration of ownership for the my own failure to make today not happen. The short story is right after declaring the "make the site more stable" milestone complete and shipping out an invoice, the site spent its most unstable day ever being frantically put on stilts and duct taped to the wall by myself. For the long version, read on. I had already spent roughly a week and a half working on an impromptu milestone in the project to increase the reliability and stability of the site, as well as beinggreenlit to apply hours to better build, test, and deployment processes. This is a good thing and it still stands as such. Now, the site wasn't fragile before, but a couple incidences understandably gave concern about long term quality. We had a few instances of corrupt MySQL logs, ran out of space on ourEBS volume, and embarrassingly I've had occasion to deploy code and f...

I am running Windows 7 via VirtualBox, and I skipped Vista completely, so some of my comments might also apply to Vista and thus be outdated. Too bad. You can probably expect a few other short pieces as I find something I like and something I don't. So, we see Internet Explorer here trying to help you out and tell you the download seems safe. Of course, it also lets you report that the download is, in fact, unsafe! This will no doubt be fed back into their SmartScreen Filter service, and when enough users report something, future users will be warned on downloading whatever bit of malware it might be. What a great way to protect your users. Now, the only obvious place to report the download is right here, in the download dialog box, which disappears as soon as the download completes and you can open or run it and actually discover anything threatening about it to warn others about. My Windows 7 review will eventually be the composite of many small pieces. I'll build up a score ...

An update to my previous post : I so called it .

This is all about the strange and confusing state of win we see repeatedly today, where a brand or concept gains such control and mindshare that no one even recognizes them as a thing anymore. Few people think of Q-tips as a brand versus just being the name of a thing. Pepsi Co executives probably grind their teeth thinking about movie goers ordering "just a coke" when Pepsi products are prominently and solely for sale. Most Internet Explorer users go beyond not understanding what IE is, they don't even understand what a browser is! Today, I want to talk about something newer and more specific, and less sure. The direction is visible that Mercurial is being given steps (it is important to phrase it this way, as I'll explain) to not matter, and that is precisely why they will win. CVS still matters, which is precisely why it has lost so utterly in the imaginary battle for geek mindshare. If you are using CVS, it is important to remember that along the way, because it a...

This is just a quicky tutorial for anyone else hitting the problem I had: installing Google Gears on 64-bit Linux. Google has not released support for this officially, but are apparently working on it and while there have been no official beta releases to try it out, someone posted a 64-bit build on a forum post . This will give you an XPI, the packaging format used by Firefox extensions, but it won't work right away. Firefox needs some special instructions to make a link actually install something and the forum post doesn't include it. There are no obvious ways to tell Firefox, "Install this XPI at this URI," so what are we to do? It turns out that it will initiate an installation if you select the XPI from the "Open File..." option in the File menu. So, download it locally and then open it in this manner, and you'll restart Firefox with a working Google Gears extension. Enjoy. I've got to get back to work now. So little time to post these days!

The infamous Zune 30GB failures were traced to a leapyear issue, and apparently they use some code we can see in the Freescale codebase. Take a look at the following sample of code, which determines the year from the day number (counting from January 1, 1980). I don't know about you, but the infinite loop is immediately obvious.  On a leapyear, the main loop continues when days = 366  and the incrementing is never reached, because days > 366  fails. Am I naive to think that even a casual code review would have caught this in a moment? year = ORIGINYEAR; /* = 1980 */ while (days > 365) { if (IsLeapYear(year)) { if (days > 366) { days -= 366; year += 1; } } else { days -= 365; year += 1; } } UPDATE: Fixed formatting issues. It looked fine when I posted it, honest!