Derek Furguson of Bear Stearns (now JPMorgan Chase) has a good article in .NET Developer Journal about how to apply genetic algorithms and grid computing to the problem of market timing in stock trading. I was pleased to see that he chose to implement his algorithms using the Digipede Network.

His article is in two parts, and this first part provides a good overview of the complex problem he’s facing — he confronts issues in financial modeling, data sources, genetic models and grid computing. As a result, Part One does not dig too deeply into coding details. But it’s worth a read — you’ll understand the architectural decisions he’s facing, and how he’s planning to address them. Plus, from what I’ve heard about Part Two (which will be out in June), there’s plenty of detail (and code) coming.

This is the second time in two months that we’ve seen influential financial modelers implement their public examples using the Digipede Network (see also Matt Davey’s recent Dr. Dobb’s article).

This is consistent with what we’re seeing from customers. While there are many grid offerings in the market, there seems to be a growing consensus that if you use .NET, there are significant advantages to working with a grid solution built on .NET. Or conversely, there’s no point trying to fit a square peg into a round hole — i.e., there’s no point trying to graft a .NET application onto a grid built for other technologies when a better option exists.

This is the “application centric” view — grids should follow applications, making it easier for developers to adapt applications to a grid, even if that means limiting the options for running those applications to a particular set of resources (in Digipede’s case, Windows machines running .NET).

The other view is “infrastructure centric” — that OS should not matter, that a grid should allow applications to be deployed across all resources, even if that means restricting the application technologies and development patterns allowed for such deployment.

Digipede has been unapologetically in the “application centric” camp for five years now, but what do others think? Has Derek made a wise choice by trading off ease of development for deployment limited to a single OS? We think so, but let’s hear from you!

Tags: .NET, Bear Stearns, Derek Furguson, Digipede, dotnet, genetic algorithms, grid-computing, JPMorgan Chase, market timing, Matt Davey

Or more accurately — Dan Ciruli is on DotNetRocks this week. 

If you want to know how to scale out Web services using .NET (and other grid-related topics), drop everything and listen to this week’s DotNetRocks program.  Well worth it.

Tags: Dan-Ciruli, Digipede, dotnet, dotnetrocks, grid-computing, Web-services

Marc at Magmasystems relates his findings from a recent trip to London, where he says:

Got together with some ex-colleagues, who were marvelling at the Lodnon consulting market.
The hot areas are Grid Computing, with the prevelant stack being DataSynapse and Tangasol. Also demand is picking up for WPF, with Morgan Stanley leading the way. The daily rates for qualified individuals are about 1000 pounds per day, which at the current exchange rates, is about $2000.

And

It will be interesting to see if Microsoft’s Compute Cluster Server and Digipede can make any inroads intot his market. There seems to be a very strong bias against using .NET for a grid infrastructure, something which I hope to see turned around in 2007.

It will be interesting to see, indeed.

The “strong bias” Marc reports is real — in some places.  But the financial services market is large, and surprisingly diverse.  Most of the bias we encounter seems to melt away when customers experience real benefits. 

.NET penetration is large and growing in financial services companies, and .NET workloads are (quite) difficult to adapt to a grid based primarily on Linux and Java.  We don’t have to win the hearts and minds of every Linux-centric grid user to make a big impact in this market. 

In our experience, the bias Marc describes is strongest in IT, which has been taught for years that grid computing means Linux and UNIX almost by definition.  But the developer community is different, and often more in touch with the scalability requirements of specific applications.  These are the hearts and minds Digipede and Microsoft are winning — because adapting applications to the grid needs to be easier, and that’s our strength.  Developers who use Microsoft Visual Studio to develop their applications (.NET, COM, or anything else) find the Digipede Framework SDK provides the most natural approach available for adapting their applications to a grid.

And it’s free, as part of the Digipede Network Developer Edition.  Check it out, .NET developers — it might just be your ticket to 1000 pounds a day!   Here you go. 

Digipede and Microsoft are also working together to win over the IT guys.  With the new Windows Server 2003 Compute Cluster Edition (CCE), Microsoft has made the deployment and administration of many servers as easy as one (and dropped the price for compute-grid deployments by about 80% too — you need to check this out).   There is no question that for grid computing deployments in financial services, CCE represents the most cost-effective way to add computing power to a Digipede-based grid.

So - the change Marc is hoping for in 2007 is exactly what we’re working to make happen!

Tags: .NET, DataSynapse, Digipede, dotnet, financial-services, grid-computing, Magmasystems, Microsoft

Digipede is sponsoring Silicon Valley Code Camp this weekend (October 7-8).  There are many good sessions, including (so far) three by Team Digipede.  Our Evangelista Kim Greenlee has a session on concurrency, and another one on debugging in VS2005.  Our Director of Products Dan Ciruli is talking about using .NET behind Excel Services. Go see ‘em!

Tags: .NET, Code-Camp, concurrency, Dan-Ciruil, Digipede, dotnet, Excel-Services, Kim-Greenlee, Silicon-Valley-Code-Camp

OK, here’s where I get to tell the real story about the new release the Digipede Network that I could not fit into the somewhat restrictive form of a press release earlier this month.  For the facts, you can go see “what’s new.”  But for the STORY, well, read on.  First, let’s go deep into the history of Digipede, and into what the Digipede Network is all about.  Hang on — this will be fun.

When we started Digipede back in 2003, we could see the market for distributed computing getting ready to take off, and we could see where we would fit into that market.  Before we started writing code, I went around and interviewed current and former CEOs and CTOs of successful and not-so-successful software companies in distributed computing.  We read, we went to conferences, we shopped the competition.  We went to see customers and prospective customers with real distributed computing needs.  And everywhere we looked we saw — layer after layer of needless complexity.  Customers were clearly being frozen out of the market by what they perceived as an insurmountable threshhold of technical and commercial complexity.

We set three criteria for ourselves and our product very early in the process:

1. The Digipede Network must provide dramatic improvements in application scalability and performance.  This is what grid computing is all about.  Sure, it’s also about asset utilization and virtualization and automation and provisioning and federation and flexible policies and IT agility and a side of fries, but we had to start somewhere.  The Digipede Network v1.0 would be all about improved application scalability and performance.
2. The Digipede Network must be radically easier to buy, install, learn, and use than any other grid solution.  I realize this sounds like marketing language.  Before it ever became marketing language, though, it was a battle cry in our office.  From the moment we started specifying our product until the moment we released it, we kept focussed on simplifying grid computing.  Before this was marketing language, this was hundreds of individual design, development, licensing, and pricing decisions.  Simple is better.
3. The Digipede Network must be the slam-dunk choice for anyone wanting a grid computing solution on the Microsoft platform.  We saw lots of grid work being done on the Linux platform (based in part on grid computing’s academic roots, and in part on IBM’s early recognition of grid’s potential), and very little commercially interesting grid activity on Windows.  An ecosystem of grid startups had formed around IBM; we declared ourselves charter members of the Microsoft grid ecosystem, and set out to make that mean something.

With those principles in hand, we went to work.  Version 1.0 was the product of more than two years of hard-core startup work — long days, long nights, passionate arguments, inadequate resources, testing and more testing, benchmarking, documenting and more documenting, beta feedback, head scratching, and more testing.

Released June 28, 2005, the Digipede Network Version 1.0 soon won critical acclaim and a critical mass of customers, We felt like the market had validated the path we were on — so we went back to work.

Version 1.2 (don’t ask where 1.1 went) made us compatible with Microsoft .NET 2.0; we also added some cool features like easier distribution of .NET objects, integration with Visual Studio 2005, job dependency, and better — you guessed it — scalability and performance.

And then we did something else — while Version 1.2 was out there doing well, we decided to release our SDK to all developers, in a special Digipede Network Developer Edition, for free.  After all, we’d just spent all that time on some pretty developer-friendly features, and we wanted to see what clever things developers could do with it.

And developers just went nuts.

They started to try out the Digipede Network as a way to increase the scalability and performance of their applications.  Word got around (I still frankly don’t entirely understand how), and developers from Bangor to Bangalore started coming at us with every scale-out problem under the sun.  Can we use the Digipede Network to process millions of large image files?  Can we scale out tax return processing?  Can we put a bioinformatic search algorithm behind a Web site and maintain quality of service?  Can we price fixed income assets faster?  Can we predict storm damage more accurately by increasing the number of scenarios analyzed?  Can we embed the Digipede Network inside our genetic algorithms for finding new asset trading opportunities?  Can we create visual If we have SharePoint, and we publish compute-intensive spreadsheets using Excel Services, can the Digipede Network scale out the calculations in the Excel user-defined functions?  And to their surprise and delight, the answer usually came up — yes.

And so we came to a decision for Version 1.3 — time to double down.  Time to focus even more on developers.  We took our already-great APIs and opened them up further, documented them better, wrote up more examples, baked in finer-grained control of jobs and tasks, and took the suggestions of some of our smartest developer-customers to make the grid computing system of developers’ dreams.  (Face it — developers dream about some weird stuff.)  All for the simple reason — the more applications are adapted to the grid, the more everybody needs the grid.

So that’s it — Version 1.3 is about developers, developers, developers.

And it couldn’t come at a better time.  Microsoft announced general availability of Windows Compute Cluster Server 2003 (CCS) last month (earth’s best way to deploy and administer many high-performance Windows servers together), and we’ve got earth’s best way to adapt .NET and COM applications to a grid of Windows machines (running CCS and every other Windows OS since 2000).  It’s never been a better time to develop scalable applications on the Windows platform.  So what are you waiting for — come and get it!

Tags: .NET, ccs, Compute Cluster Server, developer, developers, Digipede, Digipede-Network, distributed-computing, dotnet, grid-computing, Microsoft, scalability, Windows

Digipede Evangelista Kim Greenlee will present at the Bay Area .NET User Group on Thursday, September 28.  The show starts at 6:30, at 1 Market Street (the “Landmark Building”) in San Francisco.  (Yes, that’s Microsoft’s office — go to the second floor, and score some free pizza.)  It’s an easy BART ride from wherever you  are (in the Bay Area), and it’s well worth the trip.  She’s talking about concurrency, a topic near and dear to the hearts of architects and developers everywhere.  Multi-threading, grid computing, clusters, grid objects — if you’re thinking about scalability and concurrency, this talk’s for you.  Details and registration — go here.

Tags: .NET, Bay-Area-.NET-User-Group, concurrency, Digipede, dotnet, grid-computing, grid-objects, Kim-Greenlee, scalability

Don Canning describes “how a startup becomes an industry success,” providing his own take on our recent demonstration of the Digipede Network running on Windows Compute Cluster Server 2003 (CCS) at the Securities Industry Association (SIA) Technology Management Conference. 

When Don says:

Microsoft as a team is working closely with Digipede Technologies to enable seamless programmatic integration between .NET applications and cluster resources,

He’s not kidding — many, many thanks to the Microsoft Financial Services Group, the CCS team, the Office 2007 team, and the .NET team for their enthusiastic support! 

(Dan Ciruli also has some additional information on additional integration we’ve demonstrated since that event, in his post on “Putting a grid behind Excel 2007.”)

Don is also correct in pointing out the numerous compelling advantages this Microsoft / Digipede solution has over Linux grid solutions; I’ll have more on that in subsequent posts.

Thanks to Don and the whole Emerging Business Team for continuing to open doors, at Microsoft and beyond, to help this startup become an industry success.

Tags: .NET, ccs, Compute Cluster Server, dotnet, EBT, Emerging-Business-Team, Entrepreneurship, finance, grid, grid-computing, ISV, Microsoft, startup

In his blog “Tales from a Trading Desk,” Matt Davey poses some interesting speculation:

Will any investment bank has the guts to deploy Mono on its servers to run .NET trading software? I’m not sure. If the bank has already deployed Microsoft .NET to all its workstations why wouldn’t it just use Microsoft .NET on the server unless it explicitly wants to use a non-Windows OS which is unusual since the only banks who really deploy .NET servers are the Microsoft shop banks. However, if just one large bank moving into using Mono, the ripple effect could happen very quickly. Investment banks like to follow the leader.

We certainly follow Mono with interest; it’s a neat project, and we’re as curious as Matt about significant commercial adoption.  But the reasons to adopt it for server-side .NET grid projects are dimishing day by day.  With the release of Microsoft’s Windows Compute Cluster Server 2003 (CCS), the OS is becoming more and more capable for heavy-duty grid computing projects.

The combination of CCS, Windows desktops, any other available Windows servers, and the Digipede Network (yes, yes, I’m the president of Digipede) gives an investment bank the ability to put together a .NET grid that will blow away Linux-based grids, enabling .NET developers to create grid-enabled applications in the same familiar tools and programming paradigm they already use today.   Despite Mono’s improving capabilities, the process of creating a heterogeneous grid of Linux and Windows machines is not nearly as smooth.
I agree with Matt’s last point — the first bank that takes THIS step could cause a very significant ripple effect!

Tags: .NET, ccs, cluster, Compute Cluster Server, Compute-Cluster, Digipede, dotnet, finance, grid, investment-bank, Microsoft, Mono

I’m at the Securities Industry Association (SIA) Technology Management Conference at the Hilton in midtown Manhattan this week.  I took the redeye on JetBlue last night to make it to a meeting that’s been canceled, I’ve been ineptly “helping” Dan and Nathan set up the Digipede booth (#4506), and have not slept in exactly 24 hours, but I’m STILL in a great mood because I LOVE NEW YORK! 

Born and raised in White Plains, NY (about 30 miles north of my current location), I came into the City many times as a kid.  It still excites me today.  I’m a fully naturalized Californian by now (marrying a California girl can have that effect), but there’s an energy here that does not compare to anywhere else I’ve been. 

We’re looking to make a bit of a splash here this week.  You can see our press release for some of what we’ll be showing, but if you’re in town, come see us in Booth 4506, or in the Microsoft booth (2211), or the HP booth (3106), or in the Microsoft / HP “break room” where we’ll be demonstrating the first .NET-based grid-enabled financial applications running on the new Microsoft Windows Compute Cluster Server 2003 (CCS). 

As Dan pointed out in a recent post, CCS had it’s RTM announcement and official unveiling at TechEd last week.  Kyril Faenov and his HPC team emphasized many important features and benefits of this exciting new product before the Microsoft faithful.  This week, we’ll take CCS into a less friendly environment, where Linux and UNIX still occupy much of the market.  We’ll emphasize many of the same points the HPC team did — plus a few more of our own.  We’ll show how to make CCS a key part of a larger grid, incorporating the rest of your Windows desktop and server resources into an unbeatable high-performance, high-throughput computing system — as financial services companies need.

But more on that later; I’ll also comment on the other developments I see here over the next few days.

Tags: .NET, ccs, cluster, Compute Cluster Server, Digipede, dotnet, finance, grid, HP, Microsoft, SIA