The world is indeed getting smaller — and quicker, and better connected. And you might as well talk about what you’re doing, because smart well-connected people will figure everything out immediately anyway.

Case in point: Marc Adler’s recent post about Velocity, in which he says:

. . . It is no secret that the most prevelant use of object caches on Wall Street is with Grid Computing. How will Velocity interface with Compute Cluster? How about Digipede (if I know John Powers, he probably has support for Velocity already)?

Well, Marc, if you’re going to post on Sunday mornings, you may not get instant confirmation of your clever guesses, but now it’s Monday afternoon, so I can confirm — yes, we have a working PoC in the lab at Digipede, proving (to ourselves at least) that the Digipede Network and Velocity work great together. And yes, it provides significant performance improvements for certain types of activities important to Wall Street folk.

If anyone is still wondering why we chose .NET as the basis for our grid computing software, this is just the latest example — Microsoft just keeps giving us great free stuff on which to build. The Velocity CTP came out last week; this week, we have working code that provides real benefits.

Rob just posted his initial comments; there will be more.  And we’ll have feedback for the Velocity team.  Watch this space (and Rob’s, and Dan’s…).

Tags: .NET, Digipede, grid-computing, marc adler, Microsoft

I can’t believe I still haven’t had a chance to write about Velocity, Microsoft’s recently announced in-memory cache.  I think  this is just further proof that I have an endless backlog of topics about which I should be writing.

In any case, Derrick Harris of Grid Today has done a great job of connecting the dots for us in his excellent article today.  Read the whole article, because he offers good insight on how important this announcement really is, but here’s his analysis of how it affects Digipede:

Whatever emerges from Velocity also should be good news to Microsoft’s technology partners — in particular Digipede, which has been delivering distributed computing to .NET apps and now might get the add-on technology it needs to compete with the big boys. Digipede has received no shortage of praise from customers and commentators alike about its relatively inexpensive and very user-friendly solution, but one of the drawbacks has been its limitation in terms of what types of jobs the Digipede Network can handle, namely CPU-intensive jobs benefitting from parallel processing. If Microsoft and Digipede can make Velocity and the Digipede Network function as a unit and keep the price down, Digipede could find itself selling to a whole new, real-time-data-loving audience. That this integration will occur is pure speculation on my part, but it seems to make sense on the surface.

I have no comment on specifics at the moment, but let’s just say — Derrick, you nailed it.

Digipede CTO Robert Anderson is blogging about a recent experiment we’ve conducted in our lab, assessing what it would take to get the Digipede Agent running on Mono. (For those of you who don’t know, Mono is a cross-platform implementation of .NET, developed as an open-source project led by Miguel de Icaza, and sponsored by Novell.)

And as he reports, thanks to improvements in both Mono and the Digipede Network, the answer is — not much. We’ve got a working prototype of a Digipede Agent running under Mono on Linux that runs a Digipede job.

Digipede on Linux? Has the world turned upside down?

Hardly.

Since the beginning, Digipede has been focussed on adding value to the Microsoft platform. And customers know that. Customers also understand that Microsoft is getting better and better at making sure its products interoperate with others, even on other platforms, and Microsoft’s partners have to facilitate that. We get questions from customers pretty frequently about Mono, and lately those questions have gotten more specific, so it seems prudent to investigate any technical blockers from time to time.

So let me re-state what Robert said, and what I said above — this is an initial assessment, a technical experiment only, not a shipping product.  Rob’s post (and mine) are not a product announcement — this is a blatant “trial balloon.”  We want to hear what the market thinks of Digipede on Mono.

Why might this be interesting? Let’s back up a step and take a look at enterprise grid and HPC deployments.

Most enterprise customers have what is often called a “mixed” IT environment. That’s a euphemism for an unplanned and chaotic assortment of technologies that have piled up over the years into some type of barely-managed infrastructure. In almost every enterprise, Windows runs on most or all of the desktops. In almost every enterprise, there is some mixture of Windows and Linux servers, with maybe some Solaris and/or other UNIX flavor(s) thrown in. In almost every enterprise with an HPC infrastructure, most or all HPC nodes run Linux.

This is just reality — Windows is miles ahead in 2008 desktop market share, and Linux is miles ahead in 2008 HPC market share. Do I wish it were different? Sure — if Microsoft had a bigger share of the HPC market (and we’ve been working diligently to help make that happen), we’d have an even bigger market into which we could sell our software. And that will happen, I have no doubt. We tell all our customers “Windows HPC Server is the best option for adding power to a Digipede grid,” and that’s the truth. Go buy some now.

But the fact remains, there’s a lot of existing infrastructure — desktops, 32-bit Windows servers, Linux servers and cluster nodes, Solaris servers, and more — that enterprises are not going to throw away.  All this infrastructure represents potential grid computing power.  The Digipede Network has always run on heterogeneous Windows networks — with Agents running on 32- and 64-bit Windows desktops, 32- and 64-bit Windows servers, and cluster nodes running Windows HPC Server (formerly Compute Cluster Server). Our reluctance to include boxes that don’t run Windows has always been mostly about applications — it’s still relatively rare to find applications that are actually deployed across multiple operating systems simultaneously.

But as Mono gets better and better, we hear from enterprise customers and prospects who are getting more interested in it. They like the idea of being able to use more of their existing infrastructure more efficiently. They want to take advantage of Digipede’s great developer experience to deploy more applications — with minimal changes to that infrastructure.

So let’s get back to Robert’s closing question: “Now that we can do it, what should we do with it?” What do you think? Is the market crying out for a multi-OS .NET grid? Or is what we’re hearing just idle curiosity?  Let’s hear from all sides.

Tags: Digipede, grid-computing, HPC, HPC Server, Linux, Microsoft, Mono, Solaris

Releasing software is hard.

Sure, the individual steps like specifying, developing, testing, documenting, and planning support for new software features are difficult enough — but the discipline of knowing when to STOP adding features, and to focus instead on finishing a complete, polished, release-ready product is tougher than it sounds to those outside the industry.

In any software organization worthy of the name, there are more good ideas than can possibly be put into any specific product release. There are also just a stunning number of bad ideas competing for inclusion in shipping products (I am notorious within Digipede for proposing needlessly specific bad ideas. Mercifully, my partners of 20 years have honed their skills in talking me out of the worst of them.)

We decided early on at Digipede that our feature set would be guided by three principles: Performance, simplicity, and a focus on adding value to the Microsoft platform. Over the past five years, these principles have helped us make decisions on what to include (and as importantly, what to exclude) from our software.

Last month, we reached general availability of the latest release of the Digipede Network, Version 2.1. You can see what’s new in this release on our Web site, but now that our customers have had an opportunity to upgrade, let’s look at a few of the specific new features to see how we did in sticking to those principles:

Job concurrency: The improved Digipede Agent software can manage different applications running simultaneously on multiple cores of a single compute node, maximizing utilization of compute nodes on the grid. Users can set Job Concurrency values to allow the Digipede Agents to work on multiple jobs simultaneously: designate which applications can safely run with other applications, which applications can run side-by-side with themselves, and which applications are not compatible for concurrent jobs.

Performance! This one is just amazing. As new machines ship with more and more cores inside, I am continually baffled at the lack of attention from ALL the major vendors out there about how to take advantage of those cores. Sure, Intel talks about compilers and Microsoft talks about Parallel Extensions and so on — but in shipping products in 2008, there’s just incredibly little help for users and developers who want to take advantage of multi-core processors. What we shipped in Version 2.0 last September is still miles ahead of other software options in terms of both development patterns and execution modes for multi-core processing. With Version 2.1, we’ve extended that lead significantly — if you want to take advantage of dual-CPU quad-core servers and desktops TODAY, you need to take a look at how the Digipede Network handles concurrency. Watch the 4-minute video that shows how, then get an evaluation copy of the software and try it yourself!

Management APIs: New management APIs give developers programmatic ability to create, modify, and delete resource pools. (Available in Professional Edition only)

Performance (specifically, scalability), and Simplicity (of grid management). A browser-based UI for grid management is great — for small grids. As our customers deploy larger and larger grids, they need both the browser-based UI of Digipede Control and a wider range of tools for the programmatic manipulation of grid resources. It is vastly simpler to take advantage of thousands of grid nodes through simple extensions to our management API.

Risk-free sharing: “Pool Rank” permits risk-free sharing of resources: you can add your servers to the enterprise grid and ensure that they always work on your jobs first. That means that by joining the grid, you can only improve your application performance. You can donate your cycles when you are not using them without worrying that your application performance will degrade, because you are always guaranteed that your machines will work for you.

Performance and Simplicity. We’ve also referred to this feature as “Selfish sharing.” We hear from other grid vendors about how users “must” get over the practice of “server hugging.” We try not to be so arrogant; we’ve never found that scolding our customers is good business practice. If customers want to preserve unconditional priority on their own servers, we say “good for them.” So we’ve built a straighforward way to preserve absolute priority for the resource owner, even when they offer to share surplus resources. From what our customers tell us, we think this approach encourages efficient resource sharing far more than lecturing ever would.

First Grid Computing Solution Certified for Windows Server 2008: We followed the long and winding road of the Early Adopter program to become the first grid solution to obtain this important certification, so that customers can be confident that our software works not only with the Microsoft products they use today, but with all the latest improvements Microsoft is bringing to market now.

Performance, Simplicity, and Microsoft focus. By aligning with Microsoft’s technology and strategy, we help our customers create a truly dynamic IT infrastructure. Server 2008 brings many benefits in performance and manageability, and we’re confident that our customers will be upgrading quickly (more quickly than, say, to Vista); we want to be sure they can use our latest capabilities on Microsoft’s best OS platform.

Let me be candid here; these benefits do not come free to ISVs. I have considerable anxiety over extending yet further the number of versions of Microsoft products we support — for example, while I think Server 2008 is great, and Visual Studio 2008 is great, and the new SQL Server 2008 will be great, staying current means we’ll have to start enforcing our requirements by turning away requests for support of Windows 2000 and SQL Server 2000. The combinatorics for testing on multiple OS versions, .NET versions, SQL Server versions, IIS versions, and upgrade paths for our own software versions get out of hand quickly. I’ll have more on this issue another day. For now I’ll just say I’m happy with our decision to stay current — mostly.

Automatic Failover Package and Integration with NLB: Failover has long been a feature of the Digipede Network Professional Edition but with the optional Automatic Failover Package, organizations can now have complete out-of-the-box integration with Windows Server 2008 load balancing, giving “hands-free” failover to mission-critical applications.

Performance, Simplicity, and Microsoft focus — yes, even this advanced capability was guided by our goal of simplicity. While automatic failover is often considered a complex requirement, we made some basic decisions to keep it as simple as possible. First, we made automatic failover it’s own SKU, so customers without the need for high-availability configuration don’t even have to think about it. Second, we did away with a lot of the manual scripting that often slows implementation of failover solutions — you can have it running very quickly. Finally, we left as much as possible to popular existing technologies — SQL clustering and NLB — so the implementation steps will be as familiar as possible.

Reports Package: Assembles critical information about the use and optimization of the grid, with easy-to-understand charts and graphs, flagging of critical information, and drill-down capability, giving enterprises fully integrated optimization of grid performance, with tracking of who contributes to and who benefits from grid resources.

Performance, Simplicity, and Microsoft focus — In larger systems, simple and informative visual tools are essential for wringing the most performance possible from a grid. Users and administrators become far more productive in their routine monitoring functions and troubleshooting activities with this new package, which plugs directly into Digipede Control (our admin UI). And by building on SQL Reporting, we’ve created a framework for future extensions.

Overall, I’m pleased with the extent to which we have driven the improvement of our product by staying focused on the three principles described above. To be a little less self-congratulatory, I wish we had stopped adding features at least two months earlier and brought most of these capabilities to market sooner, rather than piling quite so much into a single release (and there’s certainly more than I’ve had a chance to discuss here).  Perhaps another day, I’ll have a chance to discuss some of the things we (purposely) left out!  Now that V2.1 is in the market (and getting rave reviews from our customers), I’m eager to see what great new applications our imaginative customers create and deploy on our latest platform.

Tags: Digipede, grid-computing, Microsoft, software, SQL Server, Visual-Studio, Windows Server

Derek Furguson of Bear Stearns (now JPMorgan Chase) has a good article in .NET Developer Journal about how to apply genetic algorithms and grid computing to the problem of market timing in stock trading. I was pleased to see that he chose to implement his algorithms using the Digipede Network.

His article is in two parts, and this first part provides a good overview of the complex problem he’s facing — he confronts issues in financial modeling, data sources, genetic models and grid computing. As a result, Part One does not dig too deeply into coding details. But it’s worth a read — you’ll understand the architectural decisions he’s facing, and how he’s planning to address them. Plus, from what I’ve heard about Part Two (which will be out in June), there’s plenty of detail (and code) coming.

This is the second time in two months that we’ve seen influential financial modelers implement their public examples using the Digipede Network (see also Matt Davey’s recent Dr. Dobb’s article).

This is consistent with what we’re seeing from customers. While there are many grid offerings in the market, there seems to be a growing consensus that if you use .NET, there are significant advantages to working with a grid solution built on .NET. Or conversely, there’s no point trying to fit a square peg into a round hole — i.e., there’s no point trying to graft a .NET application onto a grid built for other technologies when a better option exists.

This is the “application centric” view — grids should follow applications, making it easier for developers to adapt applications to a grid, even if that means limiting the options for running those applications to a particular set of resources (in Digipede’s case, Windows machines running .NET).

The other view is “infrastructure centric” — that OS should not matter, that a grid should allow applications to be deployed across all resources, even if that means restricting the application technologies and development patterns allowed for such deployment.

Digipede has been unapologetically in the “application centric” camp for five years now, but what do others think? Has Derek made a wise choice by trading off ease of development for deployment limited to a single OS? We think so, but let’s hear from you!

Tags: .NET, Bear Stearns, Derek Furguson, Digipede, dotnet, genetic algorithms, grid-computing, JPMorgan Chase, market timing, Matt Davey

In his most recent Dr. Dobb’s article, Matt Davey has some good commentary on Parallel Extensions to the .NET Framework, including PLINQ — as well as some very practical ideas about how developers can work with LINQ and the Digipede Network to build high-performance grid applications in .NET today.

Matt illustrates his ideas with some sample code that can “price trades in parallel on a Digipede grid,” with the results returned directly to his client LINQ application — simple, elegant, and very useful.

Matt has many more ideas for financial developers — his blog, Tales from a Trading Desk, is a must-read.

Never one to leave well enough alone, my colleague Dan Ciruli’s recent commentary on Matt’s workmentions Digipede’s concurrency patterns for taking advantage of multi-core processors. We’ll have lots more on that shortly.

Tags: .NET, Digipede, HPC, LINQ, Matt Davey, Parallel Extensions, PLINQ

Dan Ciruli, Nathan Trueblood and I will be at the Windows Server 2008 Launch in Los Angeles tomorrow.

Wear our sticker at this event — you could win an XBox 360!

Digipede has a station in the “Partner Pavillion” in the LA Convention Center, at which we will be demonstrating the only grid computing solution Certified for Windows Server 2008.

We’ll also be handing out the highly coveted Laptop Fashion Statement of 2008 — an oval sticker featuring the Digipede logo. A randomly selected attendee spotted wearing this sticker (on your laptop or anywhere else!) will win an XBox 360, so stop by our station and wear your sticker proudly!

We’ll be showing cool grid and multi-core demos like this one, too.  If you’ll be there, contact me, and we’ll meet up.

Tags: Certified-for-Windows-Server-2008, Digipede, grid-computing, Launch, Microsoft, Server-2008, XBox

It’s no secret that multi-core computing is the future; Intel and AMD have told us they can’t make a single core go much faster, but they can pack more and more cores onto a single chip.

Current operating systems, compilers, and frameworks do little to assist the application architect or developer in taking advantage of this radical change in hardware. 

Digipede’s grid computing software is remarkably good at distributing application workloads — not just across multiple machines on a grid, but also across multiple cores on a chip.   Here’s a video I made this weekend that shows how we use the same exact technology to distribute calculations first across multiple cores on a single server, then across a larger grid.  Please have a look — I managed to keep it under four minutes!

    http://www.digipede.net/downloads/digipede_multicore_grid_demo.html

As a complement to this video, you may also want to check out Dan Ciruli’s earlier demonstration that shows the code changes required to grid-enable an application — just 20 lines of code!

   http://www.digipede.net/products/whitepaper.html  (scroll down to the “Videos” section)

I’ll have lots more about this topic in the coming few weeks. 

Dan and I will be at another one-day conference in New York on Monday, February 11 — this one is Web Services / SOA on Wall Street, at the Roosevelt Hotel.  We’ll be in booth 211.

I’ll be interested to see what the crowd is like.  While the broader IT world remains somewhat divided on how widely applicable service-oriented architectures really are, we’ve seen many of our financial services clients moving quite rapidly toward SOA.

We’ve been helping these clients build scalable services using grids based on the Digipede Network.  And, as Rob and Dan said so well in their now-classic Dr. Dobb’s article on grid computing and SOA, ” You can’t build a scalable SOA on top of services that don’t scale.”

We’ll have some nifty demos (as always), and some new customer stories (ditto), and some controversial opinions to contribute to the conversation (no surprise there, either).  If you’ll be there, contact me and we’ll find a way to meet up.

Tags: Digipede, grid-computing, scalable, SOA, Web-services

Functional programming and grid computing are a good match for some types of computing problems. 

Digipede Director of Products Dan Ciruli has been experimenting with F# and the Digipede Network.  He has posted a code sample showing how to distribute F# calculations across a grid using the Digipede Framework SDK, along with some comments describing his experiences.  He’s looking for comments — check it out!

Tags: code-sample, Dan-Ciruli, Digipede, F#, functional-programming, grid-computing