Powers Unfiltered

Whenever I’m tempted to write about technical computing topics, I lie down until the urge goes away — because someone smarter will write something better than I would have anyway.  Want proof?  Read the article by Robert W. Anderson and Daniel Ciruli in the latest Dr. Dobb’s Journal, “Scaling SOA with Distributed Computing.”  While lots of people can mutter “SOA” and “Grid” in the same sentence, these guys de-mystify both concepts and lay out what architects and developers need to know.  The article just came out today, and it’s great.  Go read it.  Now.

Digipede is sponsoring Silicon Valley Code Camp this weekend (October 7-8).  There are many good sessions, including (so far) three by Team Digipede.  Our Evangelista Kim Greenlee has a session on concurrency, and another one on debugging in VS2005.  Our Director of Products Dan Ciruli is talking about using .NET behind Excel Services. Go see ’em!

OK, here’s where I get to tell the real story about the new release the Digipede Network that I could not fit into the somewhat restrictive form of a press release earlier this month.  For the facts, you can go see “what’s new.”  But for the STORY, well, read on.  First, let’s go deep into the history of Digipede, and into what the Digipede Network is all about.  Hang on — this will be fun.

When we started Digipede back in 2003, we could see the market for distributed computing getting ready to take off, and we could see where we would fit into that market.  Before we started writing code, I went around and interviewed current and former CEOs and CTOs of successful and not-so-successful software companies in distributed computing.  We read, we went to conferences, we shopped the competition.  We went to see customers and prospective customers with real distributed computing needs.  And everywhere we looked we saw — layer after layer of needless complexity.  Customers were clearly being frozen out of the market by what they perceived as an insurmountable threshhold of technical and commercial complexity.

We set three criteria for ourselves and our product very early in the process:

1. The Digipede Network must provide dramatic improvements in application scalability and performance.  This is what grid computing is all about.  Sure, it’s also about asset utilization and virtualization and automation and provisioning and federation and flexible policies and IT agility and a side of fries, but we had to start somewhere.  The Digipede Network v1.0 would be all about improved application scalability and performance.
2. The Digipede Network must be radically easier to buy, install, learn, and use than any other grid solution.  I realize this sounds like marketing language.  Before it ever became marketing language, though, it was a battle cry in our office.  From the moment we started specifying our product until the moment we released it, we kept focussed on simplifying grid computing.  Before this was marketing language, this was hundreds of individual design, development, licensing, and pricing decisions.  Simple is better.
3. The Digipede Network must be the slam-dunk choice for anyone wanting a grid computing solution on the Microsoft platform.  We saw lots of grid work being done on the Linux platform (based in part on grid computing’s academic roots, and in part on IBM’s early recognition of grid’s potential), and very little commercially interesting grid activity on Windows.  An ecosystem of grid startups had formed around IBM; we declared ourselves charter members of the Microsoft grid ecosystem, and set out to make that mean something.

With those principles in hand, we went to work.  Version 1.0 was the product of more than two years of hard-core startup work — long days, long nights, passionate arguments, inadequate resources, testing and more testing, benchmarking, documenting and more documenting, beta feedback, head scratching, and more testing.

Released June 28, 2005, the Digipede Network Version 1.0 soon won critical acclaim and a critical mass of customers, We felt like the market had validated the path we were on — so we went back to work.

Version 1.2 (don’t ask where 1.1 went) made us compatible with Microsoft .NET 2.0; we also added some cool features like easier distribution of .NET objects, integration with Visual Studio 2005, job dependency, and better — you guessed it — scalability and performance.

And then we did something else — while Version 1.2 was out there doing well, we decided to release our SDK to all developers, in a special Digipede Network Developer Edition, for free.  After all, we’d just spent all that time on some pretty developer-friendly features, and we wanted to see what clever things developers could do with it.

And developers just went nuts.

They started to try out the Digipede Network as a way to increase the scalability and performance of their applications.  Word got around (I still frankly don’t entirely understand how), and developers from Bangor to Bangalore started coming at us with every scale-out problem under the sun.  Can we use the Digipede Network to process millions of large image files?  Can we scale out tax return processing?  Can we put a bioinformatic search algorithm behind a Web site and maintain quality of service?  Can we price fixed income assets faster?  Can we predict storm damage more accurately by increasing the number of scenarios analyzed?  Can we embed the Digipede Network inside our genetic algorithms for finding new asset trading opportunities?  Can we create visual If we have SharePoint, and we publish compute-intensive spreadsheets using Excel Services, can the Digipede Network scale out the calculations in the Excel user-defined functions?  And to their surprise and delight, the answer usually came up — yes.

And so we came to a decision for Version 1.3 — time to double down.  Time to focus even more on developers.  We took our already-great APIs and opened them up further, documented them better, wrote up more examples, baked in finer-grained control of jobs and tasks, and took the suggestions of some of our smartest developer-customers to make the grid computing system of developers’ dreams.  (Face it — developers dream about some weird stuff.)  All for the simple reason — the more applications are adapted to the grid, the more everybody needs the grid.

So that’s it — Version 1.3 is about developers, developers, developers.

And it couldn’t come at a better time.  Microsoft announced general availability of Windows Compute Cluster Server 2003 (CCS) last month (earth’s best way to deploy and administer many high-performance Windows servers together), and we’ve got earth’s best way to adapt .NET and COM applications to a grid of Windows machines (running CCS and every other Windows OS since 2000).  It’s never been a better time to develop scalable applications on the Windows platform.  So what are you waiting for — come and get it!

Digipede Evangelista Kim Greenlee will present at the Bay Area .NET User Group on Thursday, September 28.  The show starts at 6:30, at 1 Market Street (the “Landmark Building”) in San Francisco.  (Yes, that’s Microsoft’s office — go to the second floor, and score some free pizza.)  It’s an easy BART ride from wherever you  are (in the Bay Area), and it’s well worth the trip.  She’s talking about concurrency, a topic near and dear to the hearts of architects and developers everywhere.  Multi-threading, grid computing, clusters, grid objects — if you’re thinking about scalability and concurrency, this talk’s for you.  Details and registration — go here.

Kudos to Dan Ciruli (our Director of Products, and also Product Manager for the Digipede Network, and also Bug Hunter, Documentation Czar, Demo Application Coder, Stand-in Sales Engineer, frequent Webcast Demo Guy and probably half a dozen other things I’m leaving out).  He’s been working late into the night on the release of Version 1.3 of the Digipede Network (as has everyone at Digipede with the possible exception of me).  And then he got sick (he’s got some moderately nasty cold he picked up from travel and late nights).  And now he’s got to sort out what might have to go into a 1.3.1, what can wait for 1.4, what’s 2.0 going to look like…  Basically, he’s pegged.

But while he’s pegged, and sick, he still just can’t stand it — he’s got to think of yet another way to show the world how insanely great the Digipede Network is.  So he’s crashed on his couch at home trying to kill this damn cold, and instead of (while?) watching poker reruns, he’s figuring out how to capture video of our Mandelbrot demonstration so that he can post it on YouTube and link it to his blog.  Basically, the demo is a drag race between a single machine and a testbed of five machines running the Digipede Network.  That demo is now accessible to anyone who has a minute and nine seconds to spare (go ahead, you’ve got 1:09 to spare — go look).

How cool is that?

I don’t know where I first heard the expression “pioneers get the arrows, settlers get the land,” but it applies in a great many new businesses.  My hat is off to Sun for taking the roll of “giant pioneer” in the area of utility computing, with their SunGrid $1-per-CPU-hour offering that’s been in the headlines for months.  It certainly appears that they are taking their share of arrows (see, for example, today’s CNET article by Stephen Shankland).  “Sun is revamping Sun Grid, which has attracted more hype than paying customers.”  Ouch.  And apparently, they’re losing one or more high-level executives over the continuing apparent non-success of SunGrid.

The offering has been an enigma to me from the outset.  Great hardware (both servers and networking) from a great hardware company, which is certainly important.  Apparently good reliability in terms of keeping the service up and running, which is certainly important.  The missing link?  Applications!  (Here we go again.) 

I was at GridWorld 2006 last week, and there were Sun folks all over the place with signs up for their “Gridathon” where they were showing folks how to adapt applications to their grid, and the excitement was, well, let’s just say “under control.”  I didn’t see a lot of interest, and I think it’s about tackling the wrong problem first.  It appears to me that there’s a long hard hill to climb to get applications onto SunGrid, and until that problem is fixed, few will care if the price is a buck or a penny per CPU-hour, even if the racks are full of nice hardware.

Now a utility that used the simplest method available for adapting a wide array of applications to the grid — THAT would be something.  Any of you hardware / network / datacenter guys want to talk?

OK, I don’t hype many of Digipede’s new customer announcements here, but go read this.  I’ll wait.

We are very excited to be working with award-winning visual effects studio Digital Dimension.  These guys get it, and are a lot of fun to work with.  Multiple rendering programs, multiple locations, multiple simultaneous projects — and a very keen sense of how to manage complex workflow.  Indeed, the key driver for this implementation is not raw computing power — a pile of hardware and a bunch of application-specific batch schedulers can provide that — but the IT agility provided by giving a clearer view into their entire production pipeline.

I’ve said it before, but I’ll say it again — small and medium businesses have a much clearer idea of how IT relates to their bottom line than most big enterprises.  With the right tools, Digital Dimension can turn CPU-hours into money.  We make it easier to manage their production pipeline, they make money.  We let them take on more projects (and/or bigger ones), they make money.  We help them adjust to changing customer priorities without disruption, they make money.  I don’t need to do their ROI analysis for them — they know it already.  It’s in their DNA.
Watch for more about this one.

I always appreciate (crave?) recognition of Digipede’s accomplishments and role in the market.  I was very pleased and honored today to receive the GridToday Editor’s Choice award for “Best Price / Performance Middleware Solution for a Grid Implementation.”  Here I am with Tom Tabor, publisher of GridToday; thanks, Mitch, for taking the picture.

GridWorld shows promise as a conference.  Attendence is not huge, but enthusiastic.  Vendors are few, but good.  Sessions are mixed — some quite good, some pretty bland.  More later.

I gave a presentation earlier today, which actually went pretty well.  I packed four “mini case studies” into 45 minutes.  I had good attendence, good questions, and lots of interest afterward — so that counts as a success. 

A few months back, when I wrote an abstract for GridWorld 2006 about “Grid Computing in Small and Medium Businesses,” well, suffice it to say that we had a few discussions before the presentation was accepted.  After all, everyone seems to KNOW that there’s no such thing as grid computing in small and medium businesses — everyone, that is, except Digipede and its many SMB customers.

In any case, a bit of gentle pursuasion got the abstract accepted, and I was given the plumb agenda position of — Dead Last, on the last afternoon.  Fine, whatever.

A few weeks back, however, the organizers called with an offer — would I like to trade Dead Last, last afternoon, for 10 AM Tuesday?  What?  Present while people might actually still be attending the conference?  Sure!  I went for it, and yes indeed, my talk is now 10 AM, Tuesday September 12.  Be there!

Now, my mom, dad, grandparents, and various other wise influences all taught me — “when it sounds too goood to be true, it probably is.”  And I don’t know why no alarm bells went off before I accepted this “too good to be true” offer.  But upon further review of the agenda, I’m not the only one with a great time slot.  Indeed, Papa Grid Ian Foster is speaking at the exact same time.  I suppose I can take some solice in the fact that everyone who knows jack about grid computing has already heard Ian speak (probably many times), but the fact remains — I’m up against pretty much the Number One name in grid computing, and I’m speaking on what many consider a pretty damn obscure topic.  Plus — I’ll have to miss Ian’s presentation!  Oh well — reading between the lines, it looks like Dan’s got it covered.

In any case, I’m looking forward to a good event — many grid luminaries, many good presentations, and Digipede will have some new announcements (watch this space).  All I can offer anyone who wants to skip Ian’s presentation is some real-world stories, a description of our recent work for Digipede’s largest non-finance customer, and no-holds-barred Q&A.  See you in DC!

Robert W. Anderson started his blog, currently known as Expert Texture, one year ago today.  He celebrates at with a divide-by-zero joke.

As Digipede’s oldest and best-read blogger, he’s an inspiration to us all.  Happy Birthday, E.T.