Congratulations to Mike Gunderloy on the one thousandth issue of The Daily Grind. There is no better place for your daily dose of .NET developer news. Great work, Mike — Keep it up!
Tags: .NET, developer, Larkware, Mike-Gunderloy, the-daily-grind
26
20:39
October '06
25
23:27
October '06
Emil Sit has an excellent post on his recent experience with SunGrid, charitably titled “Observations on SunGrid Customer Care.” He begins with:
I haven’t used the SunGrid this week. In fact, no one has: there was a four day outage from last Saturday morning through this morning.
His post is quite illuminating about Sun, customer care, individuals involved in support services, communication philosophy, and related issues. His conclusion:
As an idea, the SunGrid is a fast and easy way to get parallelism and performance flexibly. But Sun has to continue to improve the user interface (e.g., beyond the clever hack for job monitoring suggested to me by a Sun engineer) and relability of their infrastructure. Unless they do, people without CPU grants are going to start looking at alternatives like using Amazon’s hosted EC2 or running their own DigiPede.
Emil is, sadly, more than right. Sun has to improve a lot of things, not just the UI, and not just reliability, because most people ARE ALREADY looking at (and using) alternatives (and — thanks for mentioning us!). The problem isn’t UI — the problem is getting way, way ahead of the market, technology, and Sun’s own skill set.
Let me confess that I spent 20 years in or near the electric utility industry, and I know A LOT about utilities. SunGrid is not a utility. The “utility computing” analogy bothers me in general, and I don’t have nearly the time or energy to break it down fully, but the standards, reliability, infrastructure, training, and commitment are not there (anywhere in the IT industry, including Sun) to run a utility.
I almost wrote “not there YET” in that last sentence, but I remain unconvinced that a utility model will ever (or should ever) apply very well to computing. Let’s be clear — I think outsourced computing in several forms can be viable, and that there are things you may want to compute on someone else’s computing infrastructure. That doesn’t make it a utility. And that’s probably OK.
The electric utility industry is boring. Yes, I still have many fine friends there, and they are not, mostly, boring, but the industry as a whole does best when it changes slowly or not at all — and with good reason. Appliances built in 1910 still run just fine when plugged into a socket — a socket whose voltage and frequency standards have not changed in a century. Inventions since Edison, Tesla and Westinghouse have been largely incremental — in generation, transmission, and distribution. Ways to USE electricity have changed radically over time, but even these are constrained (dramatically) by unchanging standards for voltage, frequency, and a handful of other key parameters.
The tech industry has very few of the constraints (or benefits) of standards like these. (Again, I don’t have time to elaborate fully (stay tuned for future posts), but if anyone wants to debate me on whether a Web Services standard compares to the 60-Hertz standard, I warn you — you’re going to lose.) The tech industry delights in overthrowing standards, and as a result has made some pretty phenomenal progress — but as a result, use and generation of computing power have remained very closely coupled.
Is this good or bad? The jury is out (more on this another day). But is it fact? Yes. The tech culture is not the utility culture (again, watch out if you want to debate me on this). So even as we all root for SunGrid (even me), the deck is stacked against them.
Tags: Digipede, Electricity, grid-computing, Sun, SunGrid, utility, utility-computing
21
3:14
October '06
I have to admit, I was pulling for the Mets in the NLCS — but now that it’s the Cardinals and Tigers, some long-unaccessed memories of the 1968 World Series come flooding back. (OK, OK, I’m old.) I was eight years old and just entering my formative years as a baseball fan; I can’t remember a single World Series event of any kind before that, but I remember this one well.
My grandparents had just moved in with us at our house in New York, and my grandfather was a fanatical baseball fan. He had pitched a little semi-pro ball when he was 17 or 18, and had been a sportswriter on several newspapers in the Midwest after that — he knew baseball. My grandfather was a National League fan, and I inherited that from him; we were rooting for the Cardinals as Bob Gibson and Lou Brock and Curt Flood and the rest took on Denny McLain, Mickey Lolich, Al Kaline and company. I sat and watched transfixed as the ‘68 Series went seven games, with more twists and turns of plot and momentum than I can remember — but the Tigers won that one.
Like Robert Anderson, I’ll be rooting for the Cardinals this time — I’m a LaRussa fan from his tenure in Oakland as well — but mostly, I hope the series is as entertaining as the last time these two teams met.
Tags: 1968, Baseball, Cardinals, Tigers, World-Series
16
16:36
October '06
Greg Nawrocki has another excellent article today, featured in GridToday (under the unlikely title Building the Perfect ‘Grid Sandwich’). He reviews various issues related to grid adoption: Hardware? Check. Network? Check. Middleware? Many choices. Eventually he orbits back to his favorite topic and mine — applications.
But what about the applications? I’ve mentioned this before, and the idea is not originally mine, but the majority of financial calculations are still done using spreadsheets with large overhead, interpreted languages. I think this is a pretty good place to start.
OK, Greg, you’re on the money yet again, and it’s time for a bit more depth on spreadsheets. There have been a number of efforts to grid-enable spreadsheets in the past; for the most part, they’ve been “demonstration projects” that bolt a spreadsheet onto the front of an existing grid system, and the uptake has been pretty disappointing.
But this political correctness of the grid community in never naming names is starting to get in the way of real communication. Let’s be real — the right word isn’t “spreadsheets,” it’s “Excel.” And it’s not “large overhead, interpreted langauages.” It’s “VBA or .NET.” When we talk about applications for the grid, one of the key insights we need to have is that ”applications” are made by application companies, and they’re made to run on a certain platform. In the space Greg identifies, financial modeling in spreadsheets, there is only one player — Microsoft Excel. So using financial calculations in spreadsheets as “a good place to start” requires solving the big, complex problem of grid-enabling Microsoft Excel spreadsheets on Windows — not the huge and probably intractible problem of abstracting out another layer to grid enable “spreadsheets” on diverse operating systems.
So — what’s the right way to grid-enable financial models in Excel? That turns out to be a function of how the Excel spreadsheet is structured, but by focussing on .NET on Windows (not “applications” on “operating systems”) it’s pretty straightforward, and Digipede customers are now using Excel on a grid quite routinely. Whitepapers and videos are available on this topic, so I won’t go into more detail here.
The great thing about Excel is that it’s so universally used, especially within the financial services community, that innovation (from within Microsoft and beyond) continues at a blistering pace. Any Next Big Thing for Excel has the potential to be very lucrative, because of the huge base of wealthy and demanding users. (Make a demanding Firefox user happy, and you have bragging rights; make a demanding Excel user happy, and you have money.)
So one Next Big Thing for Excel is Excel Services. Microsoft has its own way of describing this new Sharepoint-based product, but I just tell people it’s the way work done by your smartest analyst can be published securely for your whole organization to use. This product aligns Excel (wittingly or not) with the SOA movement — Excel Services can be accessed through a browser, or programmatically through a Web Services interface, and hence can be incorporated far more easily into larger applications. Grid-enable THAT and you’ve really got something.
Oh wait — we did. Dan Ciruli’s excellent series of posts describes what he did to put the Digipede Network behind Excel Services, so all the work by “your smartest analyst” can scale when you expose it to “your whole organization.” The series goes:
Gettin it on with Excel Services
What Is Excel Services 2007, and What Is a User Defined Function and Why Should I Care?
Adapting a Spreadsheet to Excel Services
Why Beta Software Is Hard To Use, and How A Huge Company Still Listens to the Little Guy (OK, that one’s a bit of a detour, but a worthwhile part of the story); and
All’s Well that Ends Well, and Why Put a Grid behind Your Spreadsheet in the First Place
I’ll wrap up by saying that we think Greg’s focus on applications is dead on, and his example is dead on as well. Just as spreadsheets (VisiCalc and Lotus 1-2-3) were the killer application for the personal computer, spreadsheets (Excel and Excel Services) may become the killer app for the grid as well.
Tags: Digipede, Excel, Excel-Services, Greg-Nawrocki, grid, grid-computing, Grid-Meter, GridToday, spreadsheet, spreadsheets
05
20:42
October '06
Whenever I’m tempted to write about technical computing topics, I lie down until the urge goes away — because someone smarter will write something better than I would have anyway. Want proof? Read the article by Robert W. Anderson and Daniel Ciruli in the latest Dr. Dobb’s Journal, “Scaling SOA with Distributed Computing.” While lots of people can mutter “SOA” and “Grid” in the same sentence, these guys de-mystify both concepts and lay out what architects and developers need to know. The article just came out today, and it’s great. Go read it. Now.
Tags: architect, distributed-computing, Dr.-Dobbs, grid, grid-computing, scalability, SOA
03
3:30
October '06
Digipede is sponsoring Silicon Valley Code Camp this weekend (October 7-8). There are many good sessions, including (so far) three by Team Digipede. Our Evangelista Kim Greenlee has a session on concurrency, and another one on debugging in VS2005. Our Director of Products Dan Ciruli is talking about using .NET behind Excel Services. Go see ‘em!
Tags: .NET, Code-Camp, concurrency, Dan-Ciruil, Digipede, dotnet, Excel-Services, Kim-Greenlee, Silicon-Valley-Code-Camp
02
20:23
October '06
OK, here’s where I get to tell the real story about the new release the Digipede Network that I could not fit into the somewhat restrictive form of a press release earlier this month. For the facts, you can go see “what’s new.” But for the STORY, well, read on. First, let’s go deep into the history of Digipede, and into what the Digipede Network is all about. Hang on — this will be fun.
When we started Digipede back in 2003, we could see the market for distributed computing getting ready to take off, and we could see where we would fit into that market. Before we started writing code, I went around and interviewed current and former CEOs and CTOs of successful and not-so-successful software companies in distributed computing. We read, we went to conferences, we shopped the competition. We went to see customers and prospective customers with real distributed computing needs. And everywhere we looked we saw — layer after layer of needless complexity. Customers were clearly being frozen out of the market by what they perceived as an insurmountable threshhold of technical and commercial complexity.
We set three criteria for ourselves and our product very early in the process:
- The Digipede Network must provide dramatic improvements in application scalability and performance. This is what grid computing is all about. Sure, it’s also about asset utilization and virtualization and automation and provisioning and federation and flexible policies and IT agility and a side of fries, but we had to start somewhere. The Digipede Network v1.0 would be all about improved application scalability and performance.
- The Digipede Network must be radically easier to buy, install, learn, and use than any other grid solution. I realize this sounds like marketing language. Before it ever became marketing language, though, it was a battle cry in our office. From the moment we started specifying our product until the moment we released it, we kept focussed on simplifying grid computing. Before this was marketing language, this was hundreds of individual design, development, licensing, and pricing decisions. Simple is better.
- The Digipede Network must be the slam-dunk choice for anyone wanting a grid computing solution on the Microsoft platform. We saw lots of grid work being done on the Linux platform (based in part on grid computing’s academic roots, and in part on IBM’s early recognition of grid’s potential), and very little commercially interesting grid activity on Windows. An ecosystem of grid startups had formed around IBM; we declared ourselves charter members of the Microsoft grid ecosystem, and set out to make that mean something.
With those principles in hand, we went to work. Version 1.0 was the product of more than two years of hard-core startup work — long days, long nights, passionate arguments, inadequate resources, testing and more testing, benchmarking, documenting and more documenting, beta feedback, head scratching, and more testing.
Released June 28, 2005, the Digipede Network Version 1.0 soon won critical acclaim and a critical mass of customers, We felt like the market had validated the path we were on — so we went back to work.
Version 1.2 (don’t ask where 1.1 went) made us compatible with Microsoft .NET 2.0; we also added some cool features like easier distribution of .NET objects, integration with Visual Studio 2005, job dependency, and better — you guessed it — scalability and performance.
And then we did something else — while Version 1.2 was out there doing well, we decided to release our SDK to all developers, in a special Digipede Network Developer Edition, for free. After all, we’d just spent all that time on some pretty developer-friendly features, and we wanted to see what clever things developers could do with it.
And developers just went nuts.
They started to try out the Digipede Network as a way to increase the scalability and performance of their applications. Word got around (I still frankly don’t entirely understand how), and developers from Bangor to Bangalore started coming at us with every scale-out problem under the sun. Can we use the Digipede Network to process millions of large image files? Can we scale out tax return processing? Can we put a bioinformatic search algorithm behind a Web site and maintain quality of service? Can we price fixed income assets faster? Can we predict storm damage more accurately by increasing the number of scenarios analyzed? Can we embed the Digipede Network inside our genetic algorithms for finding new asset trading opportunities? Can we create visual If we have SharePoint, and we publish compute-intensive spreadsheets using Excel Services, can the Digipede Network scale out the calculations in the Excel user-defined functions? And to their surprise and delight, the answer usually came up — yes.
And so we came to a decision for Version 1.3 — time to double down. Time to focus even more on developers. We took our already-great APIs and opened them up further, documented them better, wrote up more examples, baked in finer-grained control of jobs and tasks, and took the suggestions of some of our smartest developer-customers to make the grid computing system of developers’ dreams. (Face it — developers dream about some weird stuff.) All for the simple reason — the more applications are adapted to the grid, the more everybody needs the grid.
So that’s it — Version 1.3 is about developers, developers, developers.
And it couldn’t come at a better time. Microsoft announced general availability of Windows Compute Cluster Server 2003 (CCS) last month (earth’s best way to deploy and administer many high-performance Windows servers together), and we’ve got earth’s best way to adapt .NET and COM applications to a grid of Windows machines (running CCS and every other Windows OS since 2000). It’s never been a better time to develop scalable applications on the Windows platform. So what are you waiting for — come and get it!
Tags: .NET, ccs, Compute Cluster Server, developer, developers, Digipede, Digipede-Network, distributed-computing, dotnet, grid-computing, Microsoft, scalability, Windows