Releasing software is hard.

Sure, the individual steps like specifying, developing, testing, documenting, and planning support for new software features are difficult enough — but the discipline of knowing when to STOP adding features, and to focus instead on finishing a complete, polished, release-ready product is tougher than it sounds to those outside the industry.

In any software organization worthy of the name, there are more good ideas than can possibly be put into any specific product release. There are also just a stunning number of bad ideas competing for inclusion in shipping products (I am notorious within Digipede for proposing needlessly specific bad ideas. Mercifully, my partners of 20 years have honed their skills in talking me out of the worst of them.)

We decided early on at Digipede that our feature set would be guided by three principles: Performance, simplicity, and a focus on adding value to the Microsoft platform. Over the past five years, these principles have helped us make decisions on what to include (and as importantly, what to exclude) from our software.

Last month, we reached general availability of the latest release of the Digipede Network, Version 2.1. You can see what’s new in this release on our Web site, but now that our customers have had an opportunity to upgrade, let’s look at a few of the specific new features to see how we did in sticking to those principles:

Job concurrency: The improved Digipede Agent software can manage different applications running simultaneously on multiple cores of a single compute node, maximizing utilization of compute nodes on the grid. Users can set Job Concurrency values to allow the Digipede Agents to work on multiple jobs simultaneously: designate which applications can safely run with other applications, which applications can run side-by-side with themselves, and which applications are not compatible for concurrent jobs.

Performance! This one is just amazing. As new machines ship with more and more cores inside, I am continually baffled at the lack of attention from ALL the major vendors out there about how to take advantage of those cores. Sure, Intel talks about compilers and Microsoft talks about Parallel Extensions and so on — but in shipping products in 2008, there’s just incredibly little help for users and developers who want to take advantage of multi-core processors. What we shipped in Version 2.0 last September is still miles ahead of other software options in terms of both development patterns and execution modes for multi-core processing. With Version 2.1, we’ve extended that lead significantly — if you want to take advantage of dual-CPU quad-core servers and desktops TODAY, you need to take a look at how the Digipede Network handles concurrency. Watch the 4-minute video that shows how, then get an evaluation copy of the software and try it yourself!

Management APIs: New management APIs give developers programmatic ability to create, modify, and delete resource pools. (Available in Professional Edition only)

Performance (specifically, scalability), and Simplicity (of grid management). A browser-based UI for grid management is great — for small grids. As our customers deploy larger and larger grids, they need both the browser-based UI of Digipede Control and a wider range of tools for the programmatic manipulation of grid resources. It is vastly simpler to take advantage of thousands of grid nodes through simple extensions to our management API.

Risk-free sharing: “Pool Rank” permits risk-free sharing of resources: you can add your servers to the enterprise grid and ensure that they always work on your jobs first. That means that by joining the grid, you can only improve your application performance. You can donate your cycles when you are not using them without worrying that your application performance will degrade, because you are always guaranteed that your machines will work for you.

Performance and Simplicity. We’ve also referred to this feature as “Selfish sharing.” We hear from other grid vendors about how users “must” get over the practice of “server hugging.” We try not to be so arrogant; we’ve never found that scolding our customers is good business practice. If customers want to preserve unconditional priority on their own servers, we say “good for them.” So we’ve built a straighforward way to preserve absolute priority for the resource owner, even when they offer to share surplus resources. From what our customers tell us, we think this approach encourages efficient resource sharing far more than lecturing ever would.

First Grid Computing Solution Certified for Windows Server 2008: We followed the long and winding road of the Early Adopter program to become the first grid solution to obtain this important certification, so that customers can be confident that our software works not only with the Microsoft products they use today, but with all the latest improvements Microsoft is bringing to market now.

Performance, Simplicity, and Microsoft focus. By aligning with Microsoft’s technology and strategy, we help our customers create a truly dynamic IT infrastructure. Server 2008 brings many benefits in performance and manageability, and we’re confident that our customers will be upgrading quickly (more quickly than, say, to Vista); we want to be sure they can use our latest capabilities on Microsoft’s best OS platform.

Let me be candid here; these benefits do not come free to ISVs. I have considerable anxiety over extending yet further the number of versions of Microsoft products we support — for example, while I think Server 2008 is great, and Visual Studio 2008 is great, and the new SQL Server 2008 will be great, staying current means we’ll have to start enforcing our requirements by turning away requests for support of Windows 2000 and SQL Server 2000. The combinatorics for testing on multiple OS versions, .NET versions, SQL Server versions, IIS versions, and upgrade paths for our own software versions get out of hand quickly. I’ll have more on this issue another day. For now I’ll just say I’m happy with our decision to stay current — mostly.

Automatic Failover Package and Integration with NLB: Failover has long been a feature of the Digipede Network Professional Edition but with the optional Automatic Failover Package, organizations can now have complete out-of-the-box integration with Windows Server 2008 load balancing, giving “hands-free” failover to mission-critical applications.

Performance, Simplicity, and Microsoft focus — yes, even this advanced capability was guided by our goal of simplicity. While automatic failover is often considered a complex requirement, we made some basic decisions to keep it as simple as possible. First, we made automatic failover it’s own SKU, so customers without the need for high-availability configuration don’t even have to think about it. Second, we did away with a lot of the manual scripting that often slows implementation of failover solutions — you can have it running very quickly. Finally, we left as much as possible to popular existing technologies — SQL clustering and NLB — so the implementation steps will be as familiar as possible.

Reports Package: Assembles critical information about the use and optimization of the grid, with easy-to-understand charts and graphs, flagging of critical information, and drill-down capability, giving enterprises fully integrated optimization of grid performance, with tracking of who contributes to and who benefits from grid resources.

Performance, Simplicity, and Microsoft focus — In larger systems, simple and informative visual tools are essential for wringing the most performance possible from a grid. Users and administrators become far more productive in their routine monitoring functions and troubleshooting activities with this new package, which plugs directly into Digipede Control (our admin UI). And by building on SQL Reporting, we’ve created a framework for future extensions.

Overall, I’m pleased with the extent to which we have driven the improvement of our product by staying focused on the three principles described above. To be a little less self-congratulatory, I wish we had stopped adding features at least two months earlier and brought most of these capabilities to market sooner, rather than piling quite so much into a single release (and there’s certainly more than I’ve had a chance to discuss here).  Perhaps another day, I’ll have a chance to discuss some of the things we (purposely) left out!  Now that V2.1 is in the market (and getting rave reviews from our customers), I’m eager to see what great new applications our imaginative customers create and deploy on our latest platform.

Tags: Digipede, grid-computing, Microsoft, software, SQL Server, Visual-Studio, Windows Server

For the folks following our Adventures in Partnerland, here’s an update.

About two weeks ago, I wrote a post here about how the persistent technical difficulties with the Microsoft Partner Program Web site (partners.microsoft.com) had spilled over from every-day inconvenience to genuine negative impact on our relations with our customers. I howled, because I think Microsoft (and the Partner Program in particular) can do better — the people in that group are really great (I know dozens of them, and can’t think of one that hasn’t left a positive impression), yet the systems have been substandard for years.

My post apparently set off a frenzy of activity. I called out executives by name, and judging by the little traffic tracer I use (Statcounter rocks by the way) that got somebody’s attention in Redmond, and that somebody sent around a very-widely-read email, and Powersunfiltered had its all-time highest traffic day — almost all from Microsoft (and it’s PR agency — hello Waggener Edstrom readers and welcome!).

And then comments came in from other partners who were having the same (or similar) experiences — but not a word from Microsoft. After four days I got a two-line email (entirely unofficial, from outside the Partner group) from a friend at Microsoft who said that my post “got some attention here,” and he had heard that “a fix” for the Partner Web site was in the works — that was the extent of Microsoft’s response.

Not long thereafter, something was indeed fixed, our references were approved, and I renewed our Gold Certified Partner status without further incident. (And — thank you, sincerely, to those who fixed whatever was broken.)

Microsoft’s public reaction, as far as I can tell, was none whatsoever.  No comments here, no comments on other blogs reporting the same problem (like this one), no postings of their own that I could find.  The Partner group’s silence on persistent problems  faced by multiple partners seems like a missed opportunity, and callers from within Microsoft have told me the same. They (and I) can’t tell what the Partner group is thinking:

• a. Some malcontent overreacted, but to quiet things down we stayed up late, made a patch, and rebooted some servers, so now everything is ok; or
• b. There’s good reason to look at how the Partner Program systems and processes are supporting the needs of Microsoft and its partners, and to put some brainpower into reworking those systems; or
• c. Something else entirely.

As for what’s next, well, if the Partner Program wants to engage, I’m game — but in the meantime, I’m moving on.  Our problem is fixed (thank you again), an opportunity for constructive public dialog appears to have passed (if I misplayed this, so be it), and I have work to do — and more positive things to write about.

Tags: Digipede, Microsoft, partner, statcounter

All my regular readers know I use the Microsoft partner Web sites fairly often, and that I experience frustrating time-sucking issues almost as often. Believe me, I spare my readership the great majority of my experiences with the unending nightmare that is the Microsoft partner online world, and when I do raise these issues I’m trying to be constructive — I really want to see Microsoft improve the online experience for its partners and customers. So I sometimes do a lot of self-editing and softening of language and hedging . . .
But I’m done caring who I offend — this is just ridiculous. The emperor has no clothes.

One thing I get to do every year is renew our status as a Microsoft Gold Certified Partner. Fine. The program requirements are difficult enough without technical hassles. Previous years have been a joy — one time, partners.microsoft.com was sufficiently screwed up that I managed to pay our $1500+ fee twice (yes, I eventually got a refund) — and this year appears to be worse.

So far, using the forms on the Microsoft Partner Program Web site, following absolutely basic and routine steps that are completely necessary to renew our status, I’ve encountered timeouts a few dozen times in the past two weeks. Fine. If Microsoft wants to provide a poor online experience for its partners, we can assess for ourselves whether the value is worth the pain.

But those few dozen timeouts don’t even include the repeated timeouts encountered by Digipede’s best customers, who have agreed to serve as references for us.

That’s right — my best customers get to sit in front of multiple timeouts waiting for the Microsoft Partner Program Web site, following Microsoft’s own instructions in an autogenerated email, just so they can provide Microsoft verification that we’ve done work for them. And yes, I get to explain to my best customers that they should be patient with Microsoft in order to do Digipede a favor. And yes, this conversation is every bit as delightful as it sounds.

My experience is not “bad luck,” nor is it unique. I’ve tried to submit information at different times of day, and on different days of the week; my customers have tried the same. And yes, I have talked to technical support, and reproduced the problem for them (”yeah, we’ve been having some of those problems lately…”). Is this just viewed as an acceptable cost of doing business by other Microsoft partners? By the rest of Microsoft?
Hey Debra Chrapaty — do you really expect to deliver online “cloud computing” services to hundreds of millions of customers when you can’t serve 640,000 partners any better than this? Do you want the help of those 640,000 partners in delivering online services to your customers? What level of confidence do you think your best partners and best customers have in your ability to deliver responsive online applications when our primary online experience with Microsoft goes through partners.microsoft.com?

Hey Kevin Turner — is this the “operational excellence” we keep hearing so much about?

Hey Allison Watson — how can your team be so great in person (and they are) while your systems are so consistently flawed (and they are)? Never in all my interactions with Microsoft do I encounter as much eye-rolling and well-practiced apologies as when I talk with your team about your systems.

(Woohoo! Invoking Allison’s name must have been the key — I just hit “OK” for the 8th or 9th time on the dialog box that’s been sitting there mocking me as I type this — these are LONG timeouts –, and FINALLY I’ve assigned ONE additional customer reference successfully. Hang on — gotta go tell another of my best customers to hit retry while the site may (briefly) be working!)

And these minor trials and frustrations are NOTHING compared to the slog we’ve been going through to get Certified for Server 2008 — but that’s another story for another day.

UPDATE:  In the past two hours, I’ve had about 70 hits on this post from within Microsoft.  In the past two hours, I’ve had one of my customers give up trying to provide a reference through the Microsoft Partner Program Web site because he does not run Internet Explorer (not sure what he uses, but I just tested with Firefox 2, and sure enough, there are some features that apparently do not work).  In the past two hours, a different one of my customers tried again to approve his reference; he received timeouts again, and now wants to know how to fax in his reference instead.

Tags: Allison-Watson, Debra-Chrapaty, Digipede, Kevin-Turner, Microsoft, Partner-Program, timeout

I post a lot about our experiences as a partner and customer of Microsoft, and anybody who cares to go back and count knows that most of my posts are positive.  For whatever reason, my complaints / rants tend to get more attention, so I want to re-iterate – our experience with Microsoft is on balance largely positive, and we remain an enthusiastic Microsoft partner.

With that out of the way – LiveMeeting has to go.  It’s great software, and good, reliable service – with no understanding of business whatsoever.  The treatment our company  has received from the Microsoft Office LiveMeeting organization (still referred to within Microsoft by its old name, Placeware), has ranged from comical to appalling.  Today, we reached appalling.

In 2005, we started experimenting with LiveMeeting as a way of demonstrating our software to prospective clients.  There were pluses and minuses, but on the whole it proved to be an effective tool.  For a while, we mooched off a friend’s account at Microsoft, but by spring of 2006 we settled into a pattern of bi-weekly Webcasts, and got our own account.  My partner Dan Ciruli set up that account; the only form of payment they offered for this type of account at the time was to put the monthly charge on his credit card, so he filled out the necessary forms and received confirmation from Microsoft.  So far, so good.

Then, Microsoft never billed him.  Now I don’t know about you, but I don’t spend a lot of time reviewing my credit card statement for transactions that aren’t there.  And apparently, neither does Dan.  So he did not notice that Microsoft was NOT billing us for this service.  And neither, apparently, did Microsoft.

Then somebody from Microsoft popped up in February 2007 and said to Dan HEY YOU HAVEN’T PAID US and Dan said “so take the money already!”  He completed a second credit card form, and was told (again) that everything was fine.  Then the exact same person from Microsoft popped up in March 2007 and said the exact same thing, as though he’d never had the previous exchange, and Dan forwarded all the exact same information, and got the same response – oh, sorry, everything is fine now.  It became a running joke in the office, wondering how they made any money.

And then in April, somebody new at Microsoft woke up and said HEY YOU GUYS OWE US A WHOLE TON OF MONEY.  Always a great day when that happens.  So Dan and I researched it and sure enough, we had been getting free LiveMeeting service for almost a year.  So I called the friendly and helpful person at Microsoft (all names withheld on this one) and asked if we could just pay by check (a) for the outstanding balance, and (b) monthly thereafter.  She agreed, and said she’d send (a) an invoice for the outstanding balance, and (b) a monthly invoice thereafter.  And everything was fine again.

Then, Microsoft never billed us.  Now, I don’t know about you, but I don’t spend a lot of time searching the mail for bills that aren’t there.  You put a valid invoice in my hand, I pay it promptly.  You don’t, oh well.  I pay a lot of invoices for other parts of Microsoft – most departments / divisions / business units / subsidiaries / whatever at Microsoft are pretty competent at getting a valid invoice to my desk, and I pay ‘em right quick.  But the LiveMeeting group apparently has not mastered this Business 101 concept. 

And of course, there’s only one way these things can end – we found our LiveMeeting account deactivated seven minutes before our scheduled Webcast this morning.  With attendees invited everywhere from here to India.  No warning emails, no onscreen notice that the account will be suspended in the future, no reason given on the Web page that says we’re dead, no “pay to reactivate” option, just — dead.  I don’t get real upset in the office very often, but I hit the roof. 

I called the toll-free number helpfully posted on the deactivation notice “for immediate service,” which presents a robotic phone tree with four options none of which are “TURN ME BACK ON NOW.”  I punch 4 to “talk with an attendant.”  I am transferred to Microsoft tech support, and am told I need to “talk to Placeware.”  I’m steaming now.  Four minutes and counting.  I call back and punch 2, “sales,” because in my experience humans usually answer when you talk to sales.  I get a human who is sympathetic, but says I need to call tech support; I get a new number to try.  Three minutes to go.  I call the new number, and find out the name of my account representative, and am transferred to him, but it turns out to be someone who sits near him, and eventually I reach my account representative.  Nope, I am assured that there is no way no how no human on earth who can turn my account back on in two minutes; my partner sends a note canceling the Webcast to all participants.  We look like idiots, and I am seriously pissed.

I stay on the line with my account manager, who asks many questions about who I had been working with previously on our account and billing issues (we find a couple of names, one gone and one still there).  I vent some more at him, and he’s sorry, and we’re done. 

The urge to open a WebEx account is now quite strong, but we suddenly realize we’ve got ANOTHER LiveMeeting with an important prospective customer later today.  Time for a deep breath.

I call the person who had previously promised to send me an invoice by mail, but never did.  I explain the whole situation again – she was “just talking with” my account manager.  I ask “did he mention that I ripped him a new one?”  She laughs nervously.  Between further apologies and digging into records, we eventually just put the whole bill on MY credit card, and she promises (really) to send me monthly invoices from here on.  Within an hour, the account is turned back on, and we’re good to go.

So – if customer service is “falling on the grenade,” these folks get a B+.  (To get an A, I had to be back up and running for the FIRST meeting we had scheduled.)  For processes and procedures – F.

Now we’ll see if I ever get a monthly bill – and yes, this is one I’ll be watching for.  And the Webex experiement begins in parallel next week.

Tags: LiveMeeting, Microsoft, Placeware

OK, I’ve posted about our positive experience at a recent Microsoft conference, and I meant every word of it — it was a great event, and we had great treatment by all the Microsoft folks in attendence.  Now I want to address an issue about how policies attributed to Microsoft Legal have the (possibly unintended) consequence of reducing value for partners and customers.  (This is FAR beyond the control of individual event coordinators at Microsoft, who deserve both praise for their performance and sympathy for having to deal with misguided Microsoft corporate policies).

Here’s how the partner lead generation at a recent Microsoft event was organized (others are similar). 

• All the partners were given a badge scanner, with which to scan the bar codes on attendee badges when an attendee comes to the partner booth. 
• Any attendee who gets his or her badge scanned by at least half the partners at the event is entered into a drawing for some cool Microsoft thing (an XBox or a Zune or whatever). 
• Wonderful.  This helps to drive traffic to the partner booths.  cool.  Partners love traffic.
• But then Microsoft keeps all the attendee information (which it already has), including who was scanned by each partner, and does not share this information with the partner.   

What?

Yes, that’s right — the partner gets none of the contact information for the attendees scanned by the partner, in the partner’s booth.  Because this would somehow constitute inappropriate sharing of secret information provided by the user to Microsoft and Microsoft only.

Yes, Microsoft uses partner presentations and booths as an added draw for propective attendees when promoting their events.  Yes, Microsoft gets partners to pay a good chunk of the cost of this event (my back-of-the-envelope says maybe $60-$70K for this recent event).  But no, attendees may only (conveniently) share their information with Microsoft, not partners.  
  
Microsoft product teams (and others) go through detailed and valuable customer profiling and use case modeling exercises, doing their best to look at Microsoft’s products from the viewpoint of the user.  Let’s try this for the event process as well — here, I’ll start.  From the viewpoint of the conference attendee, here’s what’s happening. 

• I register for a conference.  Through this registration process, I give the conference organizer my contact information.
• I get a badge with a bar code on it.  I know this code allows me to be identified by a scanner.
• I visit a sponsor booth.  I see something that interests me.  I ask for more information.
• The sponsor scans my badge, and when I get back to my office after the conference, I have an email with the information I requested, and I am able to follow up as I see fit.

This use case has been mastered by conference organizers with far less experience and technical sophistication than Microsoft.  But Microsoft Legal has convinced itself that Microsoft events are different — that Microsoft is barred from sharing ANY attendee registration information with partners, even when the attendee specifically requests it from the partner.  No, the attendee must IN WRITING request FROM MICROSOFT that the partner be allowed to send any followup information.  No, the request by the attendee to the partner to scan the attendee’s badge and send information is insufficient.  
 
So here’s the surreal attendee use case we encounter instead at some Microsoft events:

• I register for the conference.  Through this registration process, I give Microsoft my contact information.
• I get a badge with a bar code on it.  I know this code allows me to be identified by a scanner.
• I visit a Microsoft partner booth.  I see something that interests me.  I ask for more information.
• The partner asks for my card.  I say I’ve run out (hey, I’m a developer, on the average day I give out zero cards, I probably forgot them in my office).
• I ask the partner to scan my badge instead.
• The partner hesitates, saying, “um, this scanner goes back to Microsoft and I can’t get the information off it.”
• I am confused.  This partner must be mistaken.  I say “no no, I gave Microsoft my information already, you don’t need to scan me for them.”
• The partner, now trying to avoid getting into an argument with a potential customer, must further explain:  “Actually, if I scan you here, you could win a prize from Microsoft.”
• I say “OK, scan away, I love prizes.”  I still don’t understand that the partner won’t get the information — I’ve been to 100 other conferences, and at each one, I get scanned and the firm doing the scanning is the one that sees my contact info.  (But the prize is also the escape clause for Microsoft Legal — no, Mr. Partner, the attendee might not want to hear from you, he or she might just want a Zune, so it’s inappropriate to share that contact information.  No, Mr. Partner, you can’t get a separate scanner to collect information for yourself — the scanned information must be linked back to the database we’ve collected and can’t share.  Catch 22.)
• The partner scans me, and as I turn to go, says awkwardly “err, could you just write your contact information on this notepad here?”
• Now I’m just irritated.  Maybe I write my contact info, maybe I don’t, but all convenience and familiarity has gone out of the interaction.
• And I get back to the office and maybe I have the follow-up information I requested and maybe I don’t — the transfer of information via notepad scribbling is slower and more error-prone than the scanning process.  

Hey Microsoft — Partners drive sales.  Partners drive 96% of your sales.  I read that on a poster in Redmond.  I heard that from executive after executive at the Worldwide Partner Conference.  And we drive sales by talking to customers and prospective customers.  You should be jumping at every chance you have to facilitate this process. 

If I scan the badge of an event attendee, that attendee has the reasonable expectation that he or she is providing contact information to me (I’ve heard the arguments to the contrary, and do not find them plausible). 

I understand that Microsoft Legal wants to protect customer privacy, but in my opinion they’ve overdone it in this case.  There are better ways (e.g., a partner code of conduct, an event-specific agreeement with partners, a clear statement to attendees in advance about contact with partners, or even a simple note about the rules of a particular giveaway at the conference — use your legal imaginations).  This can be fixed — let’s fix it.

Comments from Microsoft, partners, and customers welcome. 

Tags: conference, Event, Microsoft, Partnering, Partnering with Microsoft, partners

I use a password manager and form-filler-outer called RoboForm.  It is by far the best solution I’ve found to creating and remembering multiple non-trivial passwords.  (Like much of the reliable technology I use, this was recommended to me by Robert Anderson, who is almost as good at finding technology that works as he is at building it.)  RoboForm is simple to learn, works flawlessly, and is published by people at Siber Systems who clearly care passionately about quality, customer service, and security.  I have two licenses — one for my laptop (my “work computer”) and one for my home desktop.

As my patient readers know, my home desktop recently melted down (see other recent posts), and was replaced by a new one.  Like other software, RoboForm licenses need to be “activated,” and I’ve used up my license activations.  I verified this by trying to activate a license on my new machine — nothing doing.  So I went to the RoboForm support page, explained the issue in plain English, and got a response from a human by email within 40 minutes saying “we’ve added another activation, it should work now.”  And it did.

I did not have to “prove” that my computer had died, I did not need to wait on hold, I just told them my problem and they solved it.

Go and buy their software now.  You  will be very glad you did.

I’ve planned a series of posts on the opportunities and perils of growth, with non-Digipede examples.  Here goes.

The first example comes from, of all places, my new home PC.

Last summer, my reliable old Gateway began to suffer from the same fate that eventually afflicts every PC — it was getting old, slow, and cranky.  I limped along with it until late fall, but enough was enough.  Five years of use and all the associated waxy buildup had brought it to the end of its useful life as the Top Dog computer in the house.  Oh, it’s still useful (I’m typing this on it now, more on that later), but it was time for a hot new PC.  I shopped around, decided not to go boring / mainstream, and settled on an “upper-middle” performance PC from boutique vendor Velocity Micro.  A little pricier than a Dell / HP / Gateway / whatever, but they got good reviews, looked cool, and claimed engineering, attention to detail, and support far above those of mortal huge companies.  And their blog, to which CEO Randy Copeland contributes, proclaims in the heading:  “Velocity Micro — Obsessed with building the perfect PC experience.”  I decided to find out.

The buying experience was quite good.  I configured my machine online, placed the order, and began receiving timely updates on progress — order received, released to production, built, shipped.  The steps took a few days longer than forecast, but that can happen, and at least communication was good.  My machine arrived, set up and ready to rock — and it rocked.  Fast, beautiful, quieter than expected, NOT overloaded with stupid trial bloatware (are you listening, Dell and HP??), very nicely built.  Ahhh.

I went through the (non-trivial) migration process to put all relevant files and applications on the new machine.  Then I reformated the old Gateway hard disk, did a clean install of Windows XP, and sent it off to a corner to serve up music files to the house network.  The Velocity Micro box took its place under the desk in the den as Top Dog computer in the house, where it performed flawlessly — for about 60 days. 

Then, quite suddenly, it suffered some type of misfortune (most likely some sort of hardware failure, more on that below) that caused frequent application and OS crashes.  Frequent as in every hour, then every half-hour, then every few minutes.

OK, that can happen.

My first call to tech support was excellent — smart guy, can-do attitude, did not treat me like an idiot — and we worked out a plan.  I got a memory test utility, which I ran overnight, and sure enough, found many, many memory errors.  A couple of quick experiments moving RAM around made it seem like it wasn’t the memory sticks but the motherboard.

OK.  That can happen.

My second call to tech support was excellent — I reached the same guy (we’ll call him Guy One), and he agreed that it was most likely the motherboard, but said he’d have both a new motherboard AND new RAM sent out right away, just in case.  When the parts came, I could schedule a tech to come out to my house and fix the whole thing (I can swap RAM, but I’m not going to mess with replacing a motherboard.)

Wow, things are going great.  I would NEVER get support like this from Dell or HP or Gateway (yes, I know this from personal experience).  No wonder Velocity Micro is experiencing “triple-digit” growth (to quote CEO Randy Copeland on the company’s blog).

But — and here comes the tie-in to my earlier “growth” comments – the parts were never sent.  I called three days later, got a different guy, who found that despite Guy One’s best intentions, “Shipping” had never bothered to send my parts. 

OK, that can happen.

But it shouldn’t.  This is where the growth-induced strain in Velocity’s internal systems and culture began to show.  Guy Two is also smart, has a can-do attitude, and did not treat me like an idiot, and he was clearly upset that his company had failed to deliver for a customer – but at this point the customer (me) had lost interest in the difference between “Guy Two” and “Shipping.”  The customer wants Velocity to function as one unit and to deliver — instead, I was learning about how somebody I could NOT speak to was screwing up the life of the guy I COULD speak to.  Quite possibly, to handle triple-digit growth in orders, Velocity’s processes and systems for coordinating customer support and shipping have changed; I imagine that both functions have grown, new people are working both new systems, and some person, process, or system had slipped, and my parts were just sitting around in Richmond as a result.  Guy Two went off to kick some ass, but my good will was rapidly being used up.

So pretty soon my parts arrived, and the process of scheduling a tech was amazingly quick and smooth (thanks again, Guy Two), and the tech came out the next day (a Saturday). 

Wow!  Things are going great again!

But wait.  The tech swapped the motherboard, and everything still crashed left right and sideways.  Not exactly the same way as before, but close enough.  He spent over three and half hours here, reached the end of his troubleshooting skills (and far more than the end of the hours alotted to the problem), and he threw in the towel. 

OK, that can happen. 

It’s notoriously difficult to troubleshoot hardware from 3000 miles away, and it’s possible that this was not a problem with the motherboard, or RAM, or that the new parts were defective too, or, well, you get the idea.  But it’s also possible that the tech (not a Velocity employee) was not quite the right guy for this type of problem, or did not bring all the diagnostic equipment / software / skills / whatever to my house on a Saturday afternoon for a full-blown troubleshooting session.  I don’t know — I’m just the customer, I don’t build or repair computers for a living.  As the customer, this was another disappointing interaction with Velocity, because I had to spend Saturday afternoon at home while my problem was not getting fixed.

And on Saturday night, Velocity support is closed, and I had to leave on a business trip on Sunday.  No main computer for the Powers family this week. 

Guy Three at Velocity called while I’m away.  Velocity decided it was time for the machine to come home to Richmond for factory troubleshooting and repair.  Sadly, I agreed — there’s no point sending out random parts so that non-Velocity techs can turn my desk into a test bench.  On my return, I called to make arrangements for shipping.  Guy Four said they’d email me a UPS shipping label, and that the process for this happens mid-day each day, so I should get it by email the next day.  I said “OK, but if I don’t receive it I’ll be shipping it anyway and billing Velocity,” and he agreed.

The machine was backed up relatively recently before the crashes began, which is not the same as saying it’s current.  I at least wanted my email identities and recent documents and so on backed up before I shipped the beast away.  I spent hours and hours and frustrating hours between crashes and spontaneous re-boots trying to get a decent backup of those files, and only partially succeeded.  I pulled the old Gateway out of music-server status, re-installed Office, and re-commissioned it as the main household computer.  Email files are out of sync, a bunch of other settings could not be recovered from the Velocity Micro box — life sucks, but goes on.

Next day, no shipping label.

I decided to wait one more day because it’s a hassle to go to the UPS store and figure out all the options, so I packed it up and got it all ready. 

Next day, no shipping label.

I grumbled down to UPS to ship it myself, and notified Velocity of this, and faxed the receipt to the person they designated. 

Two days later (as the PC was arriving in Richmond), sure enough, I got a shipping label via email from UPS.  (I’m inclined to believe Guy Two’s assessment of where some of the problems may be — are you listening Shipping?  How about you, Randy?)

I emailed this information back to Velocity so at least they don’t have to pay UPS twice.

My computer arrived in Richmond as scheduled, and my machine now sits on a bench there on life support; today is its sixth day in intensive care.  Guy Two (I’m back to him now) calls daily to explain that it’s still failing, and we discuss various theories and chat about the difficulties of troubleshooting hardware, and how a hardware failure on Part A can cause Part B to fail, and when you replace Part A, Part B can cause the NEW Part A to fail, and so on.  I like Guy Two, he’s smart and connects well with customers and no doubt has a tough hardware problem on his hands — maybe next time I’m in Richmond, I’ll go get a beer with him.  Or not.

It was after one of these status calls that I saw Randy Copeland’s February 5, 2007 blog post, called “Watching the PC Industry.”  In this post he takes a few shots at his competitors, points to Velocity’s own “triple digit growth,” decries industry trends that he feels do not serve the customer’s interests, and generally pounds the table insisting that Velocity has it right while everyone else has it wrong.  I was particulary struck by the assertion that:

A simple formula of premium components, fair pricing, and my dedication to the ultimate support experience have made our company a national contender.

On another day, I might have considered this type of post a great example of the sort of entrepreneurial optimism and assertive attitude I often admire (and sometimes project).  But given the circumstances, it struck me as self-congratulatory and out of touch.  The premium components failed, the pricing can only be considered “fair” if the machine lives up to its billing (which it has not), and the “ultimate support experience” is, well, documented above. 

Listen Randy — my $3000 machine has done no useful work in over three weeks.  I know the status of my machine pretty much every day, which is good (seriously), but not good enough.  This is the Main Computer for the Powers household, and it’s been out of commission since January 18.  I’ve twice had multi-day delays in the repair process attributable to snafus in basic systems and procedures.  While your team is filled with smart, hard-working, technically competent people who understand the importance of customer communication, they are struggling with your rapid growth to deliver on their promises.  And no one has offered a date on which I can expect to get my machine back (or a new machine with my old drives in it, for example). 

Velocity can still recover.  I still remember the attractive box, the apparent build quality, the great performance, the positive references from other customers, and the positive interactions I’ve had with individuals on the Velocity team.  I am favorably impressed by the professionalism, intelligence, and customer focus of everyone I’ve spoken to at Velocity.  But none of that means anything if I don’t have a working computer.  I’m happy for your triple digit growth, your industry awards, and your obsession — but you have not delivered anything close to the “perfect PC experience” for me. 

So the next few days are critical — if I get a reliable high-performance PC that I can use for years to come, then the memory of the past few weeks will quickly fade.  If the slip-ups continue, and I remain unable to use the PC that Velocity sold me, well, our relationship will end badly.

As for lessons at Digipede, our own triple-digit growth company is hard at work reviewing QA and customer support systems and processes.  Have we slipped up?  Yes indeed.  And we recently had a customer call us on it.  (And now, all is well with that customer.)  But we also all (and I mean all) see the opportunities and the peril, and are working steadily to maintain the highest levels of quality and service as we grow.  The good news is we have been here before — the whole Digipede management team has experienced very rapid growth before, and we’re ready for it.  We are committed (I use that term advisedly) to getting it right for our customers, and in creating the “perfect grid computing experience” for them.  And if we don’t, let me hear about it!

Tags: Digipede, Entrepreneurship, grid-computing, Growth, QA, randy-copeland, support, velocity-micro