Metrics, the hard way, part three
The most important product metric: show users value in ten minutes or less
Finally, part three
The most important product metric is time-to-value: A user engages with your software and experiences value in less than ten minutes. Your product solves her problem.
Why ten minutes? She commits so little time in return for accomplishment that those ten minutes feature in the day’s recap to her partner; she plants seeds with her boss and sends a ‘get ready’ email to the project manager. Longer, and attention wanes. She stumbles through complex tasks. An unwanted persona intrudes–she needs help to finish the job.
During these first ten minutes, your customer forms an opinion about your product and your company and decides how easy it will be to work with you. You won’t win a deal in these ten minutes, but you may very well lose one.
In part one, I introduced four metrics to measure product team performance. Together they tell how quickly you deliver value to customers.
Product lead time measures how long it takes to uncover, validate, and write requirements into your backlog.
Design time measures how long it takes to build prototypes or mockups you can use to get actionable feedback.
Design fail rate measures how often designs require significant changes or are rejected entirely and feeds back into the design time metric.
Voice of the customer measures how often and with how many customers you interview to gather feedback.
In part two, I defined each metric and created example Objectives with one or two Key Results. With metrics and a framework to track their progress, let’s talk about time-to-value as a measurable outcome.
This ten-minutes-or-less axiom works like a core principle. It is how you know those four metrics achieve an outcome. It focuses your design and development work on simple, usable functions and gives the team a benchmark. Did the work we just finished add or subtract time? Short product design and delivery lead times, positive in-product usage analytics, and user experience feedback teaches us how much value the user experiences.
Now, answer the simple question: How to reduce the time and increase the value?
Here comes the Sun
Sun was a hardware company that built software products. The software strategy was, I think, to showcase the Solaris Operating System and the Java platform. Sun executed the strategy poorly.
During my first week at Sun, I was given a Sparc Ultra 10 workstation, a handful of DVDs, and instructions to “install the software and play around.” I sat grim-faced in a windowless office, sliding glass door cracked to let the heat out, and stared at this.
A formidable stack of software sat on my desk next to an equally formidable stack of printed documentation. The docs explained the exact sequence of clicks and clacks to install the software. Upon completion, they proclaimed, I would be ready to work!
A colleague poked his head through the crack in the door. “If the install doesn’t work,” he said, “don’t bother trying the uninstall/reinstall procedure. The uninstaller leaves a bunch of artifacts on the machine that you have to clean up by hand. It never works. Just start over by reinstalling the OS.”
I blinked in alarm.
Dutifully, I found the Solaris DVD and started the installation. Hours passed. The utility of foosball tables washed over me like a revelation. Finally, a home screen appeared.
I found the Sun Java Enterprise System DVDs. These contained dozens of software products. The documentation insisted I needed them all, the better to experience Java Enterprise System’s full value. I managed exactly one of those products, Sun Access Manager. Could I install just that one? “Sorry,” said the documentation, “but that would be too easy. You must install them all.”
Ambitious. Naive. “It’s not even lunchtime yet. I’ll have this knocked out by the end of the day.”
I inserted the first DVD, and the computer’s fans whirred to life, rising to a jet-engine whine as the installer screen rendered checkbox after checkbox. I moused the cursor to hover briefly over the Install> button and clicked. Game on.
The Sun sets
Fluorescent lights dangled awkwardly overhead, threatening to burn out my eyeballs. The installer pinwheeled endlessly, alerting me to ready the Solaris DVD. Again. Ten days since the first mouse click, and still no working software.
On one of those blurred-together days, CEO Jonathan Schwartz held an all-hands where he read an email from a customer who’d dutifully left a dozen Dell servers on his loading dock while waiting for Sun to answer his calls, respond to his emails, and ship him his servers. All he wanted to do was pay, said the email. They never arrived. Sun was doomed.
I considered quitting.
Six simple steps
Who’s going to spend four thousand eight hundred minutes installing our software and still not know what it does?
Trapped in the installation nightmare known as Java Enterprise System, or “jez”, the Access Manager team gathered to ponder open revolt.
Our main competitor, Netegrity’s SiteMinder product, nuked our chances at every customer by being faster and easier at, well, everything.
What if we could be faster and easier? I threw out an improbable goal–up and running in ten minutes. So simple that even your hapless product manager could do it. The engineering manager and architect shared a conspicuous chuckle. The rest of the engineers stared at me, a suspected escapee from the Santa Clara campus’ notorious Clock Tower (aka, the Agnews Insane Asylum.)
“Come to think of it,” someone suggested, “we could create a web app to install this much faster.”
The room flooded with ideas. I scribbled notes. First, a name: the Configurator; then package the configurator as a .war file; bundle needed components into a zip file and automatically install them; ship with preconfigured defaults; customize the configuration in six steps.
The team scattered to execute an act of treason, our next release.
Over the next few weeks, we sketched designs for the Configurator, engineers built the web app, and the architect created a default configuration file. We filled our zip file with the necessary software: a Tomcat servlet container, configuration repository, embedded user directory, a sample app, and an agent to protect it.
We tested each step, decided whether we needed more capabilities, and calculated the time each would add or save. Finally, the whole package was ready. The “jez” team remained ignorant of our treachery.
I copied the zip file to my laptop. Again, I hovered my cursor reluctantly and then decompressed the file. Magic happened.
Ingenious scripts installed and configured software, deployed sample data, and started services. My CPU chugged sympathetically, and the fan offered moral support. Five minutes passed when a message appeared saying the installation and default configurations were done. I opened my browser and pasted the web app’s URL, and marveled. Behold! The Configurator!
I typed passwords in the boxes and clicked without hesitation. Success!
The user could follow six steps to create a configuration. Whether taking the defaults or creating a new configuration, the user could complete the basic task–protect the sample web application with authentication and authorization–in ten minutes or less.
Time-to-value obsession
Thus began my time-to-value obsession.
The “jez” team was very, very mad. They threatened capital punishment, tattled to executives, and wrote derogatory memos. We didn’t care. [We only finally escaped the vortex by open-sourcing the whole product, which is a different story for another day.]
We transformed an impossibly complex ultra-endurance event into a jog around the block. We impressed users with elegance and simplicity, made a difficult thing look easy, and produced a competitive product without adding a single feature.
Pay special attention to that last point. Without an excellent getting-started experience, add features at your peril.
Since this hard-won lesson, I insist my products show clear value to the user in under ten minutes. You should too.