Cloud Computing Chronicles: Stirring to Life

So I've been working on a software development project for a couple months now. Sporadically, but working on it nonetheless. I look at the changesets in my source control system (Subversion), and the dates go back to 9/4/2008.

The idea for the project started germinating with Google.  I have been developing software for more than a few years now, but the technologies I'm into these days aren't necessarily aligned with what Google is doing.  I use Google's products as part of my everyday life, of course.  I started using its search engine early in its lifecycle, and I couldn't help but notice the clever software that Google continually pumps out (I'm writing this post with Google Docs). But software development is a mammoth, fragmented industry.  For the last several years, I've been working with Microsoft technologies. Google... hasn't.

My path intersected with Google's, though, when one of my (day job) projects needed some basic mapping functionality.  At first we looked at some proprietary GIS (geographic information system) software, but decided to look at Google Maps as well.  Google's price was more attractive (essentially zero vs. big bucks), the interface was easier, and the feature set did all we needed.  The choice was not hard.

More intriguing was the level of community participation in Google Maps.  People have created everything from maps of historical journeys to step-by-step maps of red light districts.  I wondered if this was the way things were in the Google developer community.  Legions of citizens across the world, leveraging Google to unleash their imaginations.

I took a superficial look into the other APIs that Google offers -- data, YouTube, docs, etc. -- and thought it would be cool to create an application that could lead me into this world.  When I want to learn something in development, the only way to inject it into my bloodstream is to build an application with it.  Reading about a technology or writing sample code can only take me so far. I was ready to go, so I put the idea on the pile of other projects I'd like to do. My ratio of projects I'd like to do and projects I have time for is somewhere around 100:1.  So not much happened.

Engine roarin'

The concept gained some momentum with the release of Google App Engine.  Here came a service where Google would host your application right in its data centers.  What better place to work on Google's technologies?  Sounded like fun. The drawbacks to GAE, of course, are well-documented by commentators of all stripe.  Namely, you can only develop with the Python language, and there's a little matter of a non-relational database at the heart of the system.

But hey.  I was up for a challenge.  Development is both my vocation and my hobby.  The projects I do for my work are mainly out of my control -- they're what clients are willing to pay for.  But on my own time, the field is green. Building an application with GAE would put me in contact with all sorts of worlds I only knew superficially -- Python, Linux, Google APIs.  So I started working on it, as I found stray hours here and there.  I'll say more about my technical struggles learnings in other posts.

Wizard of Ozzie

Another piece fell into place when I attended Microsoft's Professional Developer Conference in late October, 2008.  One of the big technologies Microsoft was pushing was Azure, the cloud computing platform.  Ray Ozzie, Microsoft's Chief Software Architect, gave props to Amazon for blazing the trail of how cloud computing for the masses should work.  I'm sure he's right.  Amazon's EC2 (Elastic Cloud Computing) was the first behemoth out of the gate. From where I was sitting, though, the initiative looked a lot like Google App Engine. 

So now there was another angle: Microsoft was coming into the cloud computing game full-bore.  I could make my project into a narrative -- how the Microsoft and Google approaches stack up to each other, as I build an application.  But I'm also interested in seeing what the developer experience is like for other comparable offerings between Microsoft and Google.  For instance, Microsoft has Live Maps, which is remarkably similar to Google Maps.  Microsoft answers YouTube with MSN Video.  Microsoft is moving Office online, to compete in the space Google Docs occupies.

If you ask either company which has the better offering, you could probably predict the answer.  And that's generally who you have to rely on for in-depth answers about the details of technology. I would contend that the main difficulty of modern software development is not technical.  It is deciding from a seemingly infinite number of choices.  There are a finite number of problems in computer science, but they are generally complex problems.  And there are a lot of people/organizations who have thought up solutions.  If you're a developer, the stakes are high when you sit down to create a project and you choose a technology.  You can almost never predict where a project will take you.  You may end up finding out that the technology you chose is a dead end -- that it will not solve the problem.  By that time, you've probably lost a great deal of time and money.  So.  Doing this kind of exploratory development might help my technology bets be more accurate.

Underpromise and Overdeliver

Despite having made my living as a writer at various points in my life, and despite the fact I enjoy sharing experiences with other developers, I have yet to find the enthusiasm to write a technical blog.  I've had some fits and starts, as you can see for yourself by reading previous posts on this blog. Maybe the material from this project will give me the "hook" to get into it.  Or maybe this will be my last post.  Guess we'll just have to see.

But thanks for reading.

Red - Unit Testing with Python

So.  Armed with my new favorite IDE -- PyDev, the first thing I'm starting to look for is a strategy for unit testing Python.  Since I'm starting afresh, I'm planning to write tests to lay out my objectives, then learn how to create the functionality for the tests.

I'm not trying to be blinded too much by the way I usually work, but my expectation is that I will be able to find a unit testing framework that can be integrated into PyDev (or Eclipse, as the case may be).  Then I can write tests that will live "alongside" my classes and I can run the tests programmatically.

We'll see.

Green - Developing in Python

I selected PyDev (http://pydev.sourceforge.net/) as the tool to start out with.  There seems to be huge enthusiasm for plain-text editors in the Python community.  No doubt, it's fun to improvise on the command line.  I liken the feeling to going back to nature -- grabbing hold of the rough edges and learning to love them.  But I've grown too addicted to more full-featured editors.

PyDev is an open-source Eclipse (http://www.eclipse.org) plug-in with a pretty healthy feature set -- code completion, syntax highlighting, integrated debugging, refactoring, unit testing, third-party plug-ins, and on and on.  The main criticisms I see about PyDev are about the complexity/overhead of Eclipse and the fact that you need to run it on a pretty beefy machine.  After years of Visual Studio, I'm used to omnibus IDEs.  And the machines I use have some horsepower.

I saw that PyDev was recently purchased by Aptana.  I don't know enough about the Python ecosystem to know if I should be concerned that PyDev will go a commercial route.  But I suppose on the scale of worries, that one is pretty small.

So off I go.

Red - Developing in Python

For various reasons, I've decided to undertake a project using Python as the primary language.  I'm going in fairly cold, with plenty of experience in other areas, but none in Python.  My first thought was to develop with IronPython, since I'm steeped in Microsoft developer technologies.  (IronPython is a Microsoft Common Language Runtime implementation of Python).

However, I found myself falling back too much on the things I know, and one of the goals of my project is to branch out into things I don't know.  No use in doing it, if I'm using a new language to do the same things I'm doing now. So, I'm going whole hog.  There are plenty of development tools for Python, so I looked at some overviews and comparisons:

http://spyced.blogspot.com/2008/05/ide-update.html
http://wiki.python.org/moin/IntegratedDevelopmentEnvironments

…etc.

My Own Personal Red, Green, Refactor

So, if you peruse the sporadic entries up until this one, you might notice that they run long.  That's my tendency -- to write long, with exhaustive research and explanations.  It's also what keeps me from blogging very often.  I'll have ideas, but the prospect of climbing Everest dissuades me from undertaking it.  Before I commit the time to explaining the development travails I went thorugh,  I'm working on the next idea, and I've left my previous troubles behind.

I thought of an idea that might work for this site.  I dig the agile practice of test-driven development, where you start out with an idea of what to do -- write a test, knowing that it's going to fail (red), then work until that test passes (green).  That first pass most likely will not be the best solution, so you refactor until you've hit the best solution (given your constraints).

I'm going to try to use a derivation of that method here.  I will tee up whatever development issue I'm facing in an initial blog post (call it red).  I will then talk about resources I tap into and experiments I attempt in trying to solve that issue.  The first idea that works is the green post.  And then onto the process of refactoring. 

Most likely, the information will not be the definitive answer for the problems I run into.  However, the content will be actual issues encountered during actual development.  It will not be a contrived example to discuss a new feature.  And I won't bother posting about the easy stuff, or the stuff that's a simple google search away.

I will try to keep a consistent set of headlines, so that if you find a topic that interests you, you can search the blog and get an idea of how the problem arose, and how I tackled it.

The Sublime Beauty of the Apple Aesthetic

That title may sound like I'm writing another dewy-eyed tribute to the genius of Apple.  I'm not.  This is more a story of an old friend returning to help out in a pinch.

I bought my first Macintosh in 1990 -- a Macintosh Classic with a 9 inch monitor and 4MB of memory.  Those were the days when the Mac OS had a clear advantage on anything Microsoft was producing.  Even the biggest Microsoft fans would agree that Windows was an abomination before version 3.1, and that version didn't hit the market until a few years after I bought that Mac Classic.

Things change, of course.  The story of what's happened in the industry since that time has been told in many different ways, by many different people.  Personally, over the years, I've worked with Macintoshes some, but they've clearly faded in importance for me.  Making my living as a  consultant specializing in Microsoft technologies doesn't often push me in the direction of Apple technologies.

I've never lost a fondness for Apple's industrial design, though.  Making things look as cool as they do must be harder than it seems.  I always thought that some PC maker could have a viable business by simply copying Apple's designs, and manufacturing the machines for the PC market.  But whenever anyone tries that, the result is egg-suckingly lame.

When Apple switched over the Intel chipset, and running Windows on Apple hardware became a viable option, I knew it was just a matter of time before I bought my next Macintosh.  I bought a MacBook Pro and I used it almost exclusively for running Windows.  I tried out some of the built-in Apple applications, and I played around with the new (for me) operating system. (The Macintosh OS is on version 10.5 now.  The last version I used seriously was somewhere around 9.) The experience was fine, but I had a lot more use for my machine as a Windows box.  In fact, I might be one of the few people on the planet who looked for a skin for my MacBook Pro that replaced the glowing Apple logo with a Windows logo.  (Only because I tended to get a hard time from my Windows-loving cohorts).

Recently, though, something brought me back into the Mac OS, with force.  I'm currently starting a personal project that involves delving into innumerable technologies that I have not used before. I'll be chronicling the details later, but let's just say that the leap is huge, and my plate is more than full. The amount of new information to process is astronomical.  I decided to start podcasting, to go along with the project. Setting up a process to begin podcasting felt like a big hurdle I had to clear before I could even start.

My goals were fairly simple at the outset:

   1. I didn't want my podcast to sound like I was recording in a bus terminal's bathroom stall (which many podcasts do)
   2. I wasn't interested in building a home studio, or becoming an audio engineer.  I wanted just enough to make it work.

I thought to accomplish those goals, I would have to buy a decent microphone, research the available software packages, weigh the advantages of each, go through intense machine configuration, learn the vagaries of the software package, and on and on.  This wasn't just my imagination run wild.  It's the normal course of events for addressing a technology-related problem.

I started off by taking a flyer on the microphone.  I went onto Amazon and got a microphone that seemed at least suitable for the serious hobbyist. Maybe it was better, but I had no objective basis of comparison.  I just bought it. http://www.amazon.com/gp/product/B000H4HS72

A little later, I started my exploration of the podcasting options the same I way I begin just about everything these days -- with a Google search.  I noticed something interesting on my first page of results.  The program "GarageBand" appeared in several places.  I had heard some vague buzz about that program as a great way to record music.  This was the first I heard that it had the ability to be used for podcasting.  I was fairly certain that the program had been installed with my Macintosh, but I wasn't sure.  There is a suite of programs that come with the operating system, but there are also some programs that you have to pay for.

My first pleasant surprise was that I had GarageBand, and I did not have to pay for it.  Then I noticed that by simply launching the program, I was given a clear path to make a podcast.  How much simpler could it be?  There is a button that says "Create a New Podcast Episode" on the main screen.  There is even a link for video tutorials right next to it.  Watching the tutorial video had me productive about five minutes later.

I felt certain that there would be some work to configure the program to operate with the microphone.  Other than plugging in the USB cable, there was not.  The interface was clean. Having done some video editing in the past, the paradigm was utterly easy for me to pick up.

Now, there are a lot of issues I have with Apple.  Their pricing for commodity components borders on predatory.  Their smug superiority about operating system security is setting them up for a huge fall some day.  Their cavalier attitude wholesale changes for each OS upgrade means that you have no choice to but keep buying software and hardware at each revision.  But, damn.  My experience with GarageBand was one of the best I've had with software.  And I've had a lot of experience with software.

This is the kind of thing consumer software should aspire to.  Someone has a problem, and they don't want to be burdened with a learning curve or massive upfront work, before they even start solving their problem.  The program hides the complexity, and the user gets to work.  I mean, look: I know there's a whole lot more to broadcasting than what I see in GarageBand.  There's a whole scientific discipline to audio engineering that people have spent their lives working with and advancing.  Who knows?  Maybe there will come a day when my own needs or interests outstrip what I can accomplish with GarageBand, and I have to look for something more powerful.  But I doubt it.  As of now, GarageBand gives me a very simple solution to a complex problem and produces great results.

I think, in a larger context, that is why Apple is successful.  Software designers tend to get hung up on feature wars. They think that if their software does not do everything (even if the use cases are obscure), they are at a competitive disadvantage.  Apple seems to take the approach of asking how you can solve a problem without having to become a software expert.  I mean, when the iPod came out, the market for MP3 players felt like it had peaked years before.  And it had, for the tech savvy.  Apple made it approachable for normal human beings.

So there you go.  I'll keep this experience in mind.  Software isn't finished when every feature is implemented.  It's done when the benefit it provides is much greater than the effort it takes to learn how to use it.

Let it Bleed - On the Edge with Silverlight 1.1 Alpha - Part 1

So I found an excuse to get my hands dirty with Silverlight 1.1.  I'm involved in campus recruiting for my company, and each semester we give an employer presentation that involves a talk about the company, then a short demo of some current Microsoft technology.  Generally, I give the spiel about the company, then organize some other folks to show off the technology.  I decided to flip it this semester, so that I could devise some way to show off Silverlight.

Facebook felt like a good place to start.  For the Fall semester, we had given a presentation about using the Facebook API, and it had resonated.  As far as I can tell, Facebook adoption among college students runs about 100%.  It permeates every aspect of their lives, so talking about how to work with the Facebook API has built-in relevance. 

Then I wanted to think of something that exercised a broad range of the Silverlight feature set.  Graphics, audio, video, animation, the works. 

It was also a project that would give me a chance to learn in an enjoyable way.  That is -- dive right in, try things, see if they work.  If they don't work, dig into resources to find out why. That's one of the reasons you'll find that developers work on projects on their own time.  When a client is footing the bill, you can't be quite so free. 

However, there were a number of times during the project where "enjoyable" wasn't the adjective I would have selected.  The fact that Silverlight is alpha software meant that there were hiccups, blind alleys and mysterious behavior.  Even more brutally, the documentation and online resources are not terribly mature.  It's funny -- in the space of seven years or so, the development process has changed completely.  It used to be that you learned by instinct, brute force, or hazy documentation.  Now, if you need to find out how to solve a particular problem, chances are, someone in the world has faced a similar problem and posted a solution. The answer is a Google search away. That's not necessarily the case with Silverlight at this point in its lifecycle.

But I appreciate the experience now. It was enjoyable It was like a throwback to the wilder days.

So.  When I started, here was my initial thought of what would make up the Silverlight demo:

• A mechanism to sign in for Facebook
• A mechanism to harvest a user's friend list
• A XAML-based map of the United States, to indicate where friends are located
• Animation
• Video

From there, who knows?  I wanted to identify opportunities as I went. I will take you through some of the highs and lows of my experience.