Powers Unfiltered

As promised, here’s the latest from my experiences with Velocity Micro.  When last we left my poor sick PC, it was on the operating table at Velocity Micro’s labs in Richmond, VA.  After a two-week illness here at home, when a house call did not do the trick, it went back to its birthplace for major surgery.  

After my comment on the Velocity Micro blog, and my long entry here, Velocity CEO Randy Copeland intervened and said it was time to build me a new replacement PC.  The Velocity team sprang into action, built me a replacement, and tried to bring it to life as a near-replica of my original system, so I would not lose too much data or time.  And here our saga enters a new chapter.

Anyone who has attempted to migrate from an old PC to a new PC knows this does not work as well as it should.  Yes, there are tools, some from Microsoft and some not, and a quick Google will tell you of the mixed experience users have with these tools.  Even the migration from HEALTHY old PC to new PC is uncertain; the migration from SICK old PC to new PC is fraught with peril. 

Guy Five at Velocity Micro was put in charge of the operation, and things began well enough.  As previously reported, Velocity places a high priority on customer communication (even BEFORE the CEO intervenes), and Guy Five was no excecption.  So I learned relatively early in the process that Windows was badly currupted on my old machine, and that transferring applications and settings would be essentially impossible (although they thought probably they could at least move my email settings, which would be a start).

So — build and test a new machine, copy my files onto it, ship it, and it’s up to me to restore the applications and settings.  OK.  Not what I was hoping for, but at least this shouldn’t take long, right?

Ten days.  Randy’s CEO intervention came on 2/11/2007, and my new machine arrived 2/21/2007, with my data and without my applications, accounts, or settings.  Every step of the way I got daily emails and occassional phone calls documenting progress and setbacks. 

• Everything looks good. 
• Uh oh, the D drive has bad sectors and is causing crashes — better replace it. 
• Everything looks good, if all goes well you’ll have it by Saturday (2/17). 
• Yup, still looks good, it will go out Friday, you’ll have it Saturday. 
• Hmmm, ran into some trouble with the TV tuner software, caused some crashes, we want this thing stable, better keep it running stress tests over the weekend. 
• Oh, yah, kind of a long weekend, but you’ll have it Tuesday. 
• Oh, didn’t actually go out until Tuesday, you’ll have it Wednesday.

And indeed, it did arrive on Wednesday, and it worked right out of the box (no, the email settings weren’t restored, but I can do that along with the rest). 

So from time of first crash to the time I got a working system back under my desk — thirty-four days.  And it will be longer yet before it’s back to the personalized state that makes it “my” PC — I still have a few applications to install, settings to tinker with, and so on.

But it seems stable, it’s fast as can be, and it works, and it’s backed by a company that clearly is trying to serve its customers. 

So would I buy another Velocity Micro machine?  Frankly, I’m glad I don’t have to decide that today.  On the plus side,

• Their PCs are fast and appear to be intelligently designed
• Their staff is professional, courteous, and intelligent
• When things go wrong, they tell you, and admit their mistakes
• Their CEO is willing to engage openly with customers (although frankly I’m disappointed that my comment on Randy’s post on Velocity’s blog was not published)

On the minus side

• I had hardware trouble after just two months
• They could not fix it with a house call
• Their shipping department whiffed on promises at least twice, resulting in frustrating and needless delays
• Getting a replacement system took longer than expected, even after raising a fuss
• Based on my experience, I have to question their QA — I believe that their team eventually identified problems with the graphics card, TV tuner card, original motherboard, RAM sticks, and D drive (not to mention Windows getting corrupted).  But all that stuff shipped to me after “burn in” of my original system.  Was it working when it left their shop the first time?  Possibly.  Did it all die at once at my house?  Who knows?

So it depends what happens now.  If I have a long and happy relationship with my computer from here out, yes, I’d probably still go back.  Even after more than a month of trouble, these guys have managed to stay mostly on my “good side.”  Why?  Because in more than a dozen interactions with their staff, I was never once treated like an idiot.  I know this shouldn’t differentiate them from other vendors, but sadly, it does.  If you’ve ever had trouble with a Dell (and I have, and we have at work as well), you’ll know what I’m talking about.  Nobody asked me to try rebooting, nor tried to walk me through a fifty-step process unrelated to my problem.  Everyone was clearly trying to identify and solve my problem, in a way that would work for me.  That matters. 

So hats off to Randy and his crew for screaming hardware and smart, supportive staff.  Iron out a few processes that I would attribute to growing pains, and you’ll have a happy customer and a winning company.

 

 The New York Times reminded everyone this week what many of us already know — that pretty much all new improvements in processing speed will come from more cores.  In this article by John Markoff, Intel prepares a demonstration of a single chip with 80 cores.  (We’ve been hearing about this for many months, but apparently now it’s ready to show off.) 

The article repeats Intel’s claim that this is the first “Teraflop chip,” says that it will make custom video creation a snap, and generally paints a bright picture.

Only much later in the article do we reach some words of caution.

The shift toward systems with hundreds or even thousands of computing cores is both an opportunity and a potential crisis, computer scientists said, because no one has proved how to program such chips for many applications.

“If we can figure out how to program thousands of cores on a chip, the future looks rosy,” said David A. Patterson, a University of California, Berkeley computer scientist who is a co-author of one of the standard textbooks on microprocessor design. “If we can’t figure it out, then things look dark.”

Mr. Patterson is one of a group of Berkeley computer scientists who recently issued a challenge to the chip industry, demanding that companies like Intel begin designing processors with thousands of cores per chip.

In a white paper published last December, the scientists said that without a software breakthrough to take advantage of hundreds of cores, the industry, which is now pursuing a more incremental approach of increasing the number of cores on a computer chip, is likely to hit a wall of diminishing returns — where adding more cores does not offer a significant increase in performance.

This analysis is correct in my view, and is a major reason why we started Digipede almost four years ago.  Intel and AMD will no longer bail us out with higher clock speeds — if we want software to run faster and scale better, we just have to learn to deal with more processors (more cores on more chips in more boxes).

And let’s be clear — I’m not claiming Digipede is “the answer” to this software “crisis,” because lots of answers are needed; so far, no one multi-processing approach will address all application loads, and it is doubtful any single approach ever will.  But for a fairly broad class of CPU-intensive processes, we’ve taken our shot and put it out there for developers (it’s free — go now — I’ll wait). 

Right now, our software is best suited for combining single- and dual-core chips in single- or dual-CPU boxes (i.e., making grids out of lots of boxes), because that’s where the sweet spot is for commodity hardware today.  But as the market evolves, you’ll see us bring out more features to help you make use of more cores on more CPUs in each box, too.

Digipede Evangelista Kim Greenlee writes and speaks to developer organizations about concurrent development.  See this recent post for example.  The age of concurrency is here — let’s embrace it together!

Marc at Magmasystems relates his findings from a recent trip to London, where he says:

Got together with some ex-colleagues, who were marvelling at the Lodnon consulting market.
The hot areas are Grid Computing, with the prevelant stack being DataSynapse and Tangasol. Also demand is picking up for WPF, with Morgan Stanley leading the way. The daily rates for qualified individuals are about 1000 pounds per day, which at the current exchange rates, is about $2000.

And

It will be interesting to see if Microsoft’s Compute Cluster Server and Digipede can make any inroads intot his market. There seems to be a very strong bias against using .NET for a grid infrastructure, something which I hope to see turned around in 2007.

It will be interesting to see, indeed.

The “strong bias” Marc reports is real — in some places.  But the financial services market is large, and surprisingly diverse.  Most of the bias we encounter seems to melt away when customers experience real benefits. 

.NET penetration is large and growing in financial services companies, and .NET workloads are (quite) difficult to adapt to a grid based primarily on Linux and Java.  We don’t have to win the hearts and minds of every Linux-centric grid user to make a big impact in this market. 

In our experience, the bias Marc describes is strongest in IT, which has been taught for years that grid computing means Linux and UNIX almost by definition.  But the developer community is different, and often more in touch with the scalability requirements of specific applications.  These are the hearts and minds Digipede and Microsoft are winning — because adapting applications to the grid needs to be easier, and that’s our strength.  Developers who use Microsoft Visual Studio to develop their applications (.NET, COM, or anything else) find the Digipede Framework SDK provides the most natural approach available for adapting their applications to a grid.

And it’s free, as part of the Digipede Network Developer Edition.  Check it out, .NET developers — it might just be your ticket to 1000 pounds a day!   Here you go. 

Digipede and Microsoft are also working together to win over the IT guys.  With the new Windows Server 2003 Compute Cluster Edition (CCE), Microsoft has made the deployment and administration of many servers as easy as one (and dropped the price for compute-grid deployments by about 80% too — you need to check this out).   There is no question that for grid computing deployments in financial services, CCE represents the most cost-effective way to add computing power to a Digipede-based grid.

So – the change Marc is hoping for in 2007 is exactly what we’re working to make happen!

 

I’ve planned a series of posts on the opportunities and perils of growth, with non-Digipede examples.  Here goes.

The first example comes from, of all places, my new home PC.

Last summer, my reliable old Gateway began to suffer from the same fate that eventually afflicts every PC — it was getting old, slow, and cranky.  I limped along with it until late fall, but enough was enough.  Five years of use and all the associated waxy buildup had brought it to the end of its useful life as the Top Dog computer in the house.  Oh, it’s still useful (I’m typing this on it now, more on that later), but it was time for a hot new PC.  I shopped around, decided not to go boring / mainstream, and settled on an “upper-middle” performance PC from boutique vendor Velocity Micro.  A little pricier than a Dell / HP / Gateway / whatever, but they got good reviews, looked cool, and claimed engineering, attention to detail, and support far above those of mortal huge companies.  And their blog, to which CEO Randy Copeland contributes, proclaims in the heading:  “Velocity Micro — Obsessed with building the perfect PC experience.”  I decided to find out.

The buying experience was quite good.  I configured my machine online, placed the order, and began receiving timely updates on progress — order received, released to production, built, shipped.  The steps took a few days longer than forecast, but that can happen, and at least communication was good.  My machine arrived, set up and ready to rock — and it rocked.  Fast, beautiful, quieter than expected, NOT overloaded with stupid trial bloatware (are you listening, Dell and HP??), very nicely built.  Ahhh.

I went through the (non-trivial) migration process to put all relevant files and applications on the new machine.  Then I reformated the old Gateway hard disk, did a clean install of Windows XP, and sent it off to a corner to serve up music files to the house network.  The Velocity Micro box took its place under the desk in the den as Top Dog computer in the house, where it performed flawlessly — for about 60 days. 

Then, quite suddenly, it suffered some type of misfortune (most likely some sort of hardware failure, more on that below) that caused frequent application and OS crashes.  Frequent as in every hour, then every half-hour, then every few minutes.

OK, that can happen.

My first call to tech support was excellent — smart guy, can-do attitude, did not treat me like an idiot — and we worked out a plan.  I got a memory test utility, which I ran overnight, and sure enough, found many, many memory errors.  A couple of quick experiments moving RAM around made it seem like it wasn’t the memory sticks but the motherboard.

OK.  That can happen.

My second call to tech support was excellent — I reached the same guy (we’ll call him Guy One), and he agreed that it was most likely the motherboard, but said he’d have both a new motherboard AND new RAM sent out right away, just in case.  When the parts came, I could schedule a tech to come out to my house and fix the whole thing (I can swap RAM, but I’m not going to mess with replacing a motherboard.)

Wow, things are going great.  I would NEVER get support like this from Dell or HP or Gateway (yes, I know this from personal experience).  No wonder Velocity Micro is experiencing “triple-digit” growth (to quote CEO Randy Copeland on the company’s blog).

But — and here comes the tie-in to my earlier “growth” comments — the parts were never sent.  I called three days later, got a different guy, who found that despite Guy One’s best intentions, “Shipping” had never bothered to send my parts. 

OK, that can happen.

But it shouldn’t.  This is where the growth-induced strain in Velocity’s internal systems and culture began to show.  Guy Two is also smart, has a can-do attitude, and did not treat me like an idiot, and he was clearly upset that his company had failed to deliver for a customer – but at this point the customer (me) had lost interest in the difference between “Guy Two” and “Shipping.”  The customer wants Velocity to function as one unit and to deliver — instead, I was learning about how somebody I could NOT speak to was screwing up the life of the guy I COULD speak to.  Quite possibly, to handle triple-digit growth in orders, Velocity’s processes and systems for coordinating customer support and shipping have changed; I imagine that both functions have grown, new people are working both new systems, and some person, process, or system had slipped, and my parts were just sitting around in Richmond as a result.  Guy Two went off to kick some ass, but my good will was rapidly being used up.

So pretty soon my parts arrived, and the process of scheduling a tech was amazingly quick and smooth (thanks again, Guy Two), and the tech came out the next day (a Saturday). 

Wow!  Things are going great again!

But wait.  The tech swapped the motherboard, and everything still crashed left right and sideways.  Not exactly the same way as before, but close enough.  He spent over three and half hours here, reached the end of his troubleshooting skills (and far more than the end of the hours alotted to the problem), and he threw in the towel. 

OK, that can happen. 

It’s notoriously difficult to troubleshoot hardware from 3000 miles away, and it’s possible that this was not a problem with the motherboard, or RAM, or that the new parts were defective too, or, well, you get the idea.  But it’s also possible that the tech (not a Velocity employee) was not quite the right guy for this type of problem, or did not bring all the diagnostic equipment / software / skills / whatever to my house on a Saturday afternoon for a full-blown troubleshooting session.  I don’t know — I’m just the customer, I don’t build or repair computers for a living.  As the customer, this was another disappointing interaction with Velocity, because I had to spend Saturday afternoon at home while my problem was not getting fixed.

And on Saturday night, Velocity support is closed, and I had to leave on a business trip on Sunday.  No main computer for the Powers family this week. 

Guy Three at Velocity called while I’m away.  Velocity decided it was time for the machine to come home to Richmond for factory troubleshooting and repair.  Sadly, I agreed — there’s no point sending out random parts so that non-Velocity techs can turn my desk into a test bench.  On my return, I called to make arrangements for shipping.  Guy Four said they’d email me a UPS shipping label, and that the process for this happens mid-day each day, so I should get it by email the next day.  I said “OK, but if I don’t receive it I’ll be shipping it anyway and billing Velocity,” and he agreed.

The machine was backed up relatively recently before the crashes began, which is not the same as saying it’s current.  I at least wanted my email identities and recent documents and so on backed up before I shipped the beast away.  I spent hours and hours and frustrating hours between crashes and spontaneous re-boots trying to get a decent backup of those files, and only partially succeeded.  I pulled the old Gateway out of music-server status, re-installed Office, and re-commissioned it as the main household computer.  Email files are out of sync, a bunch of other settings could not be recovered from the Velocity Micro box — life sucks, but goes on.

Next day, no shipping label.

I decided to wait one more day because it’s a hassle to go to the UPS store and figure out all the options, so I packed it up and got it all ready. 

Next day, no shipping label.

I grumbled down to UPS to ship it myself, and notified Velocity of this, and faxed the receipt to the person they designated. 

Two days later (as the PC was arriving in Richmond), sure enough, I got a shipping label via email from UPS.  (I’m inclined to believe Guy Two’s assessment of where some of the problems may be — are you listening Shipping?  How about you, Randy?)

I emailed this information back to Velocity so at least they don’t have to pay UPS twice.

My computer arrived in Richmond as scheduled, and my machine now sits on a bench there on life support; today is its sixth day in intensive care.  Guy Two (I’m back to him now) calls daily to explain that it’s still failing, and we discuss various theories and chat about the difficulties of troubleshooting hardware, and how a hardware failure on Part A can cause Part B to fail, and when you replace Part A, Part B can cause the NEW Part A to fail, and so on.  I like Guy Two, he’s smart and connects well with customers and no doubt has a tough hardware problem on his hands — maybe next time I’m in Richmond, I’ll go get a beer with him.  Or not.

It was after one of these status calls that I saw Randy Copeland’s February 5, 2007 blog post, called “Watching the PC Industry.”  In this post he takes a few shots at his competitors, points to Velocity’s own “triple digit growth,” decries industry trends that he feels do not serve the customer’s interests, and generally pounds the table insisting that Velocity has it right while everyone else has it wrong.  I was particulary struck by the assertion that:

A simple formula of premium components, fair pricing, and my dedication to the ultimate support experience have made our company a national contender.

On another day, I might have considered this type of post a great example of the sort of entrepreneurial optimism and assertive attitude I often admire (and sometimes project).  But given the circumstances, it struck me as self-congratulatory and out of touch.  The premium components failed, the pricing can only be considered “fair” if the machine lives up to its billing (which it has not), and the “ultimate support experience” is, well, documented above. 

Listen Randy — my $3000 machine has done no useful work in over three weeks.  I know the status of my machine pretty much every day, which is good (seriously), but not good enough.  This is the Main Computer for the Powers household, and it’s been out of commission since January 18.  I’ve twice had multi-day delays in the repair process attributable to snafus in basic systems and procedures.  While your team is filled with smart, hard-working, technically competent people who understand the importance of customer communication, they are struggling with your rapid growth to deliver on their promises.  And no one has offered a date on which I can expect to get my machine back (or a new machine with my old drives in it, for example). 

Velocity can still recover.  I still remember the attractive box, the apparent build quality, the great performance, the positive references from other customers, and the positive interactions I’ve had with individuals on the Velocity team.  I am favorably impressed by the professionalism, intelligence, and customer focus of everyone I’ve spoken to at Velocity.  But none of that means anything if I don’t have a working computer.  I’m happy for your triple digit growth, your industry awards, and your obsession — but you have not delivered anything close to the “perfect PC experience” for me. 

So the next few days are critical — if I get a reliable high-performance PC that I can use for years to come, then the memory of the past few weeks will quickly fade.  If the slip-ups continue, and I remain unable to use the PC that Velocity sold me, well, our relationship will end badly.

As for lessons at Digipede, our own triple-digit growth company is hard at work reviewing QA and customer support systems and processes.  Have we slipped up?  Yes indeed.  And we recently had a customer call us on it.  (And now, all is well with that customer.)  But we also all (and I mean all) see the opportunities and the peril, and are working steadily to maintain the highest levels of quality and service as we grow.  The good news is we have been here before — the whole Digipede management team has experienced very rapid growth before, and we’re ready for it.  We are committed (I use that term advisedly) to getting it right for our customers, and in creating the “perfect grid computing experience” for them.  And if we don’t, let me hear about it!

I have been fortunate to be part of rapidly growing companies throughout my career, and it’s happening again.   

Growth is fun.  With growth comes opportunity — often, opportunity to show what you can really do. 

Growth is perilous.  With growth comes new expectations, new responsibilities, public exposure of warts, new processes (as “winging it” fails to cut it anymore), and a general raising of the stakes.

Digipede is growing, at a pace that is generating plenty of fun, opportunity, and peril.

I have in mind a series of posts about entepreneurship and growth, but I’ll never get it all done.  But I’ve seen a few examples lately outside of my own company, and I’m trying to learn some lessons, so I’m at least going to start.  Let’s see how far we get.

I know, I usually write about partnering with Microsoft — but it’s time to give our other 800-lb gorilla partner some kudos.  More and more folks at HP are apparently convinced they’re going to sell a LOT more Windows CCS clusters by working with Digipede — and we agree.

First, we have an upcoming event in Bellevue, WA on Monday.  HP is hosting a briefing on its Windows-based HPC solutions for partners, customers, and prospects in HP’s Bellevue office (14475 NE 24th Street) on Monday, January 29, from 9:00 AM to 3:00 PM.  I’m told they’ll buy lunch.  Dan Ciruli (Digipede Director of Products) and I will be there to demonstrate the Digipede Network running on Windows Compute Cluster Server 2003 (CCS).  I believe several Microsoft folk will also be there (it’s quite near the Redmond campus). 

Should be interesting.  HP’s got some really impressive hardware (including some fancy new blades, which are selling like hotcakes), and they’ll be showing that off.  Dan wrote a much more complete post about the event, so I’ll leave it there.

Next, Dan and I continue our HP Western US Tour with a trip to Houston for HP’s HPC Summit — another HP partner and sales fest, at which we hope to find more partners eager to integrate, re-sell, and consult about the Digipede Network.  The HP people in Houston have been really great to us, and I’m looking forward to seeing them again.  (Actually, I’ve had great experience with people in Houston all through my professional life, with the exception of a handful of arrogant SOBs at Enron…)

Finally, HP has also invited us to participate in an upcoming special edition of their “Transforming Your Enterprise” publication, focused entirely on Windows CCS.  More good exposure for us.  Thanks for looking out for us, HP!