One of the most painful and joyful events in the life of a software company is the release of new software.  While this seems like an obvious statement, let me just say to all my friends who are NOT in the software business — you have no idea.

Many customers, prospective customers, and industry observers shrug and even smirk at a press release that “merely” announces the release of a new version of an existing product.  (Smirk away — here’s ours.)  But it’s gratifying to receive congratulations from those who actually understand this process (thanks, friends at Microsoft!).

So now that the apparently-endless cycle of build and test is over, and the last (known) snafu has been fixed (how the %^$&$! did we put an uninstallable version of our SDK out on our community site?), we can take a deep breath, step back, and discuss what this release means — to our customers.  Because, as our press release says, this is a software release entirely driven by our customers.

From the beginning, we set out to make the Digipede Network “radically easier to buy, install, learn, and use” than any other distributed computing platform.  Reviewers say we’ve done that, and customers tell us they can come up to speed quickly with our software.  Ah, but once a customer comes up to speed quickly, that customer gets ideas!  “Why does Digipede use all the cores on each compute resource? Can we reserve one or more for other uses?”  “When I try to delete thousands of jobs at once, weird things happen — are you guys just idiots or what?” “I thought you guys were supposed to be Microsoft-savvy; why can’t I host a .NET 4 application on your software?”  “When we run millions of jobs with lots of really short tasks, the Digipede database gets really big — can you fix that?”  You get the idea.

Well, to be honest, we never tested that “queue thousands of jobs while thousands of other jobs are running and then just delete the thousands that are queued” case, so yeah, weird things happened.  Should be better now.  And yeah, .NET 4 is a reasonable expectation from us — works fine now.   Yes, it’s true that there are ways to make the Digipede database grow — and while we’ve always had tools for managing that, those tools are simpler and more useful now.

That multi-core thing turned out to be the most popular one though, and it’s been one of my pet issues for a while, so let’s talk about that in more detail.  I’ve spoken at conferences, written articles, made videos, and given interview for years saying basically this:

• Mainstream developers know single-threaded object-oriented coding techniques, which take advantage of a single core.
• Meanwhile, chip makers are developing CPUs with more and more cores.
• The Digipede SDK is the simplest way for a mainstream developer to WRITE WHAT THEY KNOW (i.e., single-threaded object-oriented code) and EXECUTE that code on multiple cores on a chip, multiple chips in a box, and multiple boxes on a grid, all using the same programming paradigm.

And this has been great for us and for our customers — up to a point.  For purely compute-intensive applications, this approach scales linearly in cores and machines up to hundreds and even thousands of multi-core compute resources.  But many complex applications have a lot of I/O requirements as well, and just loading up (for example) an 8-core server (most likely, a dual quad-core box) with 8 cores worth of computation can actually slow down execution as processes wait for I/O.

So in the most recent release, we took a very simple brute-force approach to fixing this issue – we now allow users to “reserve” one or more cores per compute resource through a simple option in Digipede Control.  Early users report excellent results, with 6 or 7 cores computing away while the remaining one or two handle all other chores (including I/O).  Equally important, this approach is robust to additional increases in the number of cores per chip (which is forecast to reach several dozen within just a few years).

If you want to take the new version for a spin, ask for a free evaluation copy here.

Now, how about what’s NOT in our press release?  Well, you won’t find the word “cloud” in there…

Is it just me, or has the cloud meme really jumped the shark?  Look.  I used cloud computing before it was called that, and I’ll use it after that name has wandered off into the scrapheap of forgotten marketing buzzwords.  If a cloud is Google and a cloud is a cluster in a datacenter somewhere the user can’t see it, then a cloud is everything and nothing.  If a cloud is Amazon or GoGrid, then sure, our customers can deploy the Digipede Network there, or they can deploy it on their own infrastructure (then, if they want to, they can tell their bosses they’ve built a “private cloud” for all I care!).

The market knows Digipede as a provider of distributed computing software for the Windows platform, and as a provider of  high-productivity distributed computing tools for .NET developers.  That’s our role in the cloud and on the ground and everywhere in between.

The video of that conversation is now on Channel9; you can watch it here.

We’re offering developers who watch that video (and even those who don’t) a free copy of the Digipede Network Developer Edition — go to this page to get yours today.

Many thanks to  William and the rest of the Microsoft Evangelists for giving us this opportunity to get the word out about how Digipede and Microsoft work together to make software run faster and scale bigger!

Penny Crosman provided some good news for banks, hedge funds, and other money managers in her article today in Wall Street & Technology — good news for financial developers and IT professionals who need to access more processing power without complex application re-engineering. 

You can read the article for yourself — there are good quotes from AVM CTO Paul Algreen, a longtime Digipede customer – but from my perspective, the gist is this:

• CPUs are getting faster these days almost exclusively through putting more cores on a chip. 
• Hence, when you buy a fancy new server, performance only improves for applications that take advantage of multi-core architectures. 
• Yet most applications are single-threaded, leaving all but one core doing, umm, nothing. 
• AVM noticed this problem more than two years ago, and started using the Digipede Network to address it. 
• They’ve adapted compute-intensive legacy applications to run on a grid of multi-core boxes without expensive re-engineering, seeing huge performance gains. 
• Thanks to the intuitive programming model offered by the Digipede Framework SDK, AVM has added more and more applications to the grid since then, and they haven’t looked back.

This is quite typical of the experience many Digipede customers have had — that for most applications in financial services, multi-core and grid computing can be handled most effectively as two cases of the same general distributed computing problem.  

And yes, I’m going to plug our now-famous four-minute video on this topic again — you can watch it here.  Then you can request a free evaluation copy of the Digipede Network, and try it out on your own compute-intensive applications.  Because Intel and AMD aren’t waiting for the world to re-tool a few million enterprise developers; they’re banging out chips with more and more cores with every new generation. 

But with the right tools, you can take advantage of all that power — and that’s a welcome dose of good news for Wall Street!

We did a press release today  about the recent sale we made to the US Navy.  You can read that here, so let’s go a little deeper. 

The Navy has access to huge volumes of very accurate geodetic data — information that tells the location and elevation of every point on earth.  (I never knew much about this area until this year, but a lot of decent public information is available.  You can look at this Wikipedia article if you’re interested, and dig around from there.)  Processing geodetic data is a very compute-intensive process.  Combining that information with 2-dimensional image data is even more complex. 

One application for this data is “terrain generation,” a process of converting raw data into a format useful for visualizing terrain in flight simulations.  A group at Lockheed Martin develops specialized software for this purpose; we’ve been working with their TOPSCENE group for some time. 

That group approached us for assistance in increasing the speed of their terrain generation process.  They’ve been great to work with — they’ve put the Digipede Network to work on their problem, and have achieved great results.  Their own press release  about the Digipede-enabled version of their software documents a 20x speedup in processing for our mutual customer at the US Navy.   

We’re excited about this application, for a variety of reasons.

First, Lockheed Martin is the largest independent software vendor (ISV) we’ve worked with, and they’ve validated our premise that ISVs would find the Digipede Network (and particularly the Digipede Framework SDK) the best choice for grid-enabling complex applications.  This is an important point for us.  ISVs can work with other vendors, or build their own application-specific distributed computing solution, and Lockheed Martin certainly has the resources to pursue either path — but they chose Digipede, and have achieved great results. 

Second, we see many, many more applications in processing geodetic data.  We’ve already made other sales in that area to government agencies outside of defense (no announcements yet, but stay tuned!), and we see increasing interest from commercial customers in this area as well.   

Finally, as many financial customers struggle with market issues (some of our clients from 2007 no longer exist in recognizable form), it’s important that Digipede diversify and demonstrate growth in other markets.  Customers like the US Navy certainly help with this important goal.  While government purchases can be  slow, we’ve been quietly working this area for years, and it’s great to see results we can discuss. 

The new head of Red Hat, Jim Whitehurst, says “The clouds will all run Linux.”

Really? What’s next? The head of Boeing telling us that all transportation will be via 787s? Or maybe a statement from OPEC saying that cars will all burn gasoline?

Get serious. “The Clouds” will run Linux and UNIX and Windows and OS X and whatever else paying customers want.

“The Clouds” are already running more than just Linux, and if cloud computing is going to grow beyond today’s super-early-adopter proof-of-concept market, it’s going to get more diverse, not less.

Every time there’s a new IT buzzword (oops, I mean “revolution”), some market-oblivious engineer or attention-deficit analyst declares that finally legacy computing is dead, a new paradigm is here, there’s One Right Way to do everything now, the open-source rapture is at hand, you’re free from your chains, yada yada yada.

But a guy like Jim Whitehurst should know better.  Yeah Jim, “the clouds” are going to kill Microsoft. Yeah Jim, “the clouds” mean Oracle is finished. Yeah Jim, “the clouds” will all run Linux, and Slashdot will replace all other news outlets.

I’ll have lots more constructive things to say about cloud computing soon here, but for now let me just say: the cloud computing offerings I’ve seen so far look a lot like computing. There’s hardware infrastructure, there are operating systems, there are development tools, there are applications, APIs, and user interfaces. There are administrative tools, management consoles, and buckets of kludgy tricks to make anything actually work the way you want it to work. Different vendors expose different parts of all this to their users in different ways as they struggle to differentiate. But anyone who believes that “the clouds” will “all” standardize on a single OS (or database, or programming language, or much of anything else) is just blowing smoke.

In a single sentence containing at least three significant understatements, Ms. Gage writes:

Microsoft still gets criticized sometimes for being slow to the Internet or hard to do business with, but Lewin has won praise over the years for his courtesy, efficiency and ability to connect outsiders to the right people inside Microsoft, which is not an easy task.

Whew.  Let’s parse that.

“Microsoft still gets criticized sometimes for being . . . hard to do business with . . .”  There is no question that doing business with ANY huge company is hard.  Building a close relationship with Microsoft (or any tech giant) is not for the faint of heart.  Microsoft presents some special challenges that I could go on about at length (oh wait, I’ve done that multiple times…), but let’s just stipulate that some of these criticisms are justified while some are not.

“…but Lewin has won praise over the years for his courtesy, efficiency and ability to connect outsiders to the right people inside Microsoft…”   Bingo.  Dan’l Lewin has done more to expose the helpful side of Microsoft to startups, entrepreneurs, and VCs than anyone would have thought possible just a few years ago.  His Emerging Business Team is the API for startups that want hooks into Microsoft.  Digipede has received numerous tangible and intangible benefits from working with the EBT; the group brings the attitude that they can’t wait to help interesting startups, and it’s Dan’l who sets the tone and agenda for that critical group.

“…which is not an easy task.”  No kidding.  I’m back to my API analogy.  If you would rather to try to reverse-engineer the Microsoft org chart from the outside, good luck — but a single call to the EBT can get you to the right person within Microsoft faster than any other method I know.

The Bay Area is teaming with “experts” who would have us believe that Microsoft has become irrelevant.  In my experience, entrepreneurs ignore Microsoft at their peril.  Far better to understand what they’re doing and why than to pretend they aren’t there.  Dan’l and his team are great resources for entrepreneurs who want to understand and work with Microsoft.  So — good job, Ms. Gage, for profiling Microsoft’s local champion of innovation.  Well worth reading.

In any case, Derrick Harris of Grid Today has done a great job of connecting the dots for us in his excellent article today.  Read the whole article, because he offers good insight on how important this announcement really is, but here’s his analysis of how it affects Digipede:

Whatever emerges from Velocity also should be good news to Microsoft’s technology partners — in particular Digipede, which has been delivering distributed computing to .NET apps and now might get the add-on technology it needs to compete with the big boys. Digipede has received no shortage of praise from customers and commentators alike about its relatively inexpensive and very user-friendly solution, but one of the drawbacks has been its limitation in terms of what types of jobs the Digipede Network can handle, namely CPU-intensive jobs benefitting from parallel processing. If Microsoft and Digipede can make Velocity and the Digipede Network function as a unit and keep the price down, Digipede could find itself selling to a whole new, real-time-data-loving audience. That this integration will occur is pure speculation on my part, but it seems to make sense on the surface.

I have no comment on specifics at the moment, but let’s just say — Derrick, you nailed it.

His article is in two parts, and this first part provides a good overview of the complex problem he’s facing — he confronts issues in financial modeling, data sources, genetic models and grid computing. As a result, Part One does not dig too deeply into coding details. But it’s worth a read — you’ll understand the architectural decisions he’s facing, and how he’s planning to address them. Plus, from what I’ve heard about Part Two (which will be out in June), there’s plenty of detail (and code) coming.

This is the second time in two months that we’ve seen influential financial modelers implement their public examples using the Digipede Network (see also Matt Davey’s recent Dr. Dobb’s article).

This is consistent with what we’re seeing from customers. While there are many grid offerings in the market, there seems to be a growing consensus that if you use .NET, there are significant advantages to working with a grid solution built on .NET. Or conversely, there’s no point trying to fit a square peg into a round hole — i.e., there’s no point trying to graft a .NET application onto a grid built for other technologies when a better option exists.

This is the “application centric” view — grids should follow applications, making it easier for developers to adapt applications to a grid, even if that means limiting the options for running those applications to a particular set of resources (in Digipede’s case, Windows machines running .NET).

The other view is “infrastructure centric” — that OS should not matter, that a grid should allow applications to be deployed across all resources, even if that means restricting the application technologies and development patterns allowed for such deployment.

Digipede has been unapologetically in the “application centric” camp for five years now, but what do others think? Has Derek made a wise choice by trading off ease of development for deployment limited to a single OS? We think so, but let’s hear from you!

For those zero of you who don’t read GridToday until I point you there, his recent interview of me is here.

More later from High Performance on Wall Street.

We’ll be releasing Version 2.0 of the Digipede Network, and meeting with many analysts, press, customers, prospects, partners, and bartenders.  Since some of that could be pretty time consuming, we’ll also be in town September 18-19 for Microsoft meetings and more of the above.

If you’ll be there, let me know and we’ll find a way to meet up.