Building product teams, part two: The product manager programming test
How will your candidate the job?
Building product teams, part two
Hiring is a lot like finding a new roommate. You invite a small pool of hopefuls, talk to each for perhaps thirty minutes, and take their answers on faith. You huddle with roommates to determine whether you can live with that person.
The process is fraught.
Things you cannot verify. Do they pay rent and other bills on time? Will they do the dishes? Will they steal your sweet potatoes and eat all your ice cream? Does he (and this is exclusively a ‘he’ thing) pee haphazardly in a shared bathroom? Will he clean and fix his bicycle in the living room? Will he clean the shared bathroom as well as his bicycle?
The roommate-finding process presents just enough information for someone to say, “Yeah, they seem cool.” That’s it. And then YOU HAVE TO LIVE WITH THAT PERSON. Like, in the same house.
I once interviewed a potential roommate to occupy a very cheap room in an apartment in San Francisco’s Mission District. She seemed so cool, like a unicorn. She was an artist and musician but with a full-time job and steady income! She claimed to be neat, loved cooking, and often spent time with her girlfriend’s family in Marin. (The perfect roommate is one who’s never around and fun when they are.)
She offered a reference to back up her story.
We offered her the room. Remember, so cool!
Her reference never answered the phone. Her “full-time job” was busking in BART stations. On her steady income of coins and small bills, she could barely make ends meet. She paid late every month. She ate boiled cabbage. She spent hours in our lone bathroom, leaving wet floors and soaked towels behind. Every dirty dish languished in the sink.
A nightmare roommate.
So it goes with hiring
Statistically, we spend more time with employees and coworkers than with friends and family, turning hiring into a decision to live with a stranger for, say, the next five years. Do we learn enough about the people we hire before we hire them? Suppose they answered all your questions brilliantly. Can they actually do the job? Can you run the startup or corporate gauntlet and not come to despise one another? How do you find the answers to these questions?
Engineering teams have it easy. To know how well someone writes code, give them a programming test. All at once, you learn whether they develop good code but also whether they’ll follow your team's coding conventions, architecture patterns, and quality standards. The exercise affords the opportunity to assess personality and team fit.
No such test exists for product managers.
Like software developers, product managers come with well-defined skill sets. So there ought to be a way to evaluate talent and skill objectively. The industry anguishes over the enigma that is product management. What do they do? How do they do it?
You can test for this. I call it the “product manager programming test.”
The test reveals different capabilities and attributes than other, more common interview tactics such as having the candidate research and present a case study, pitch their current product, or puzzle through inane problem-solving questions like how many doorknobs you’d need in New York City or which marbles belong in which jars. I’m not saying don’t ask these questions, but don’t expect the answers to tell you much more than how well your candidate solves riddles. You are trying to find out how well they will do THIS job working with THIS team.
Design your test
This needn’t be overly complicated. Start by identifying the skills you most need: product management, industry or domain knowledge, agile process, information architecture, or user experience design. Hiring objective determined, simply match the test to the objective.
In this case, I was searching for product management skills. I needed a team to build a new product in a new market category. Early hires stocked the company with domain expertise, but I was the only product person. I needed help building the product function. I could teach hardened PMs domain skills.
I devised a test to measure a candidate’s ability to perform core PM skills–managing a backlog and running the ceremonies that drive the development lifecycle. I wanted to see specifically how a candidate would run ceremonies like team inceptions, iteration planning, standups, retros, and iteration reviews; use tools for brainstorming and ideas; and manage the backlog.
I selected Jira, Whimsical, and Miro for the experiment’s tools.
Setup your candidate for success
Don’t surprise the candidate. I am dismayed by hiring managers springing “gotcha” style questions on candidates. High-functioning companies don’t surprise employees–they set clear objectives and empower their people to achieve them.
Create the conditions for success by telling your candidates ahead of time about the experiment, the goals, the tools you’ll use, and how the interview session will flow. Schedule sixty minutes and tell the candidate to limit the exercise to forty-five minutes, leaving time for discussions and questions. However far the candidate progresses, stop the clock at forty-five minutes. Bonus points to candidates who finish in the allotted time.
Give candidates access to your tools. Let them play around for a few days ahead of time. Bonus points for candidates who show up with a groomed backlog, a sketched-out storyboard, or a UX mockup. Even if it’s wrong, not related to the product, and you end up changing everything during the interview, it tells you something about their initiative, willingness to take risks, the impulse to experiment, familiarity with the tools (or the ability to adapt to new tools), and general excitement about the interview.
State your plan as plainly as possible.
“I’m looking forward to meeting you on Thursday. The exercise is a mock iteration planning meeting. The goal is to have work-ready stories that the developers could execute. We will use three tools, Jira, Whimsical, and Miro. Familiarize yourself with these tools. I’ll set up a mock backlog in Jira with an epic and a few stories. Don’t worry about the technical content of the epic and stories. [Editor’s note: technical content matters if you are testing for domain expertise.] We will groom and prioritize the backlog during the session. The stories are to do with the user experience, so we may need to use Whimsical and Miro for storyboarding, UX mockups, and anything else you feel is necessary to give developers the clarity they need to work effectively. I’ve invited the team’s tech lead, one developer, and our tech writer to join the session, so it should feel like a real planning meeting.”
Setup a Jira backlog
Make up a fake backlog. Your candidate likely knows little about your product’s inner workings, so build the exercise around something more abstract, like the product’s getting started experience. The candidate doesn’t have to know anything specific about the product to plan this type of work.
I create an epic with three to five stories.
It doesn’t really matter what the epic is about or what the stories describe. Write the stories in priority order and see whether the candidate reviews and adjusts those priorities. The point is to observe how the candidate interacts with the other team members, uses the tool, and conducts the ceremony.
Create a Whimsical file
Create a space for your candidate to do brainstorming and creative work. I like Whimsical because it is simple, easy to use, and free. The candidate can sign up independently and play with it, even once you grant them access to your interview workspace.
The product’s getting started experience often requires storyboards and wireframes. Wireframing tasks don’t take too much of your allotted time; you can create a simple wireframe like this in under five minutes.
The candidate doesn’t have to use this tool. Maybe their process takes them in another direction, and that’s alright. Maybe the team takes them in the direction of using the tool, creating artifacts and diagrams attached to stories. I don’t give any specific guidance here; observing the candidate’s process is more important. Have the participating team members give feedback about their needs and see how the candidate reacts.
Create a Miro board
Miro and Whimsical serve similar purposes. Multiple tools solving similar problems challenge can disrupt a team’s smooth-flowing process. How does the candidate approach this? Does she use different tools in a specialized way, using one for storyboards and another for UX mockups? Does she point out the proliferation of overlapping tools? Does she suggest another tool that streamlines the process and retires two tools?
Run the ceremony
Start the meeting with the briefest intro–assume everyone has read your prep note and knows what is happening; this should take no more than the first minute of the session. I usually start with a prompt to the candidate, “Ok, get us started. What would you do first?”
And then let them run with it.
Good candidates will summarize the objectives and confirm their understanding or ask for clarifications. They’ll engage the team members to fill in details and find points of collaboration.
I usually guide the interviewing teams to prompt the candidate to create new stories or cause them to reprioritize the existing backlog. And to push the candidate to make requirements and work as clear as possible. Can it be independently implemented? Is it broken down sufficiently? How will the team know when the work is done? Ask the evaluating team to include these criteria when scoring the candidate–more on this below.
See what the candidate does with your backlog.
Gather feedback from the team
Do a retrospective with the interviewing team no more than 24 hours after the interview.
I use Google Forms to gather and analyze feedback. Other feedback-gathering tools exist, but I’ve found Forms is fast and easy to set up and lets you capture and collate qualitative and quantitative data. Your HR or recruiting team may ask you to use their application tracking system, and that’s ok. Your Forms data translate to other formats easily.
Below I’ve mocked up a simple form mimicking what I use.
Have each team member give an overall rating on a 1-5 scale. Supplement the rating with qualitative feedback: did well, coachable, did poorly. Ask for specific examples for later pattern matching and tie-braking.
Next, evaluate how the candidate used your tools on a scale of 1-5. Do they recognize the overlap and potential for friction? Did they suggest an alternative? My general rule is that any new tool should replace at least two existing ones. Be curious about how candidates approach this problem.
This test measures individual capabilities and team fit simultaneously. Interview participants have an opportunity to weigh in on a critical hire, and you’ll observe the future team dynamic firsthand. Scale the Team Fit 1-5 and ask for two or three specific examples of what worked, what didn’t work, and how to improve.
Rate on a scale of 1-5 how they ran the ceremony, adapted to your process, or suggested changes for the better. Were stories sufficiently clear? Can the work be independently implemented? Was the work broken down and prioritized sufficiently? Are there clear definition-of-done criteria? If you need detailed quantitative data, scale each of your questions independently.
Learn something
Several serious hiring blunders made the product manager programming test necessary. I experimented with different approaches until landing on, in retrospect, the simplest and most obvious solution. Ask your candidate to do the job. Design the test to discover which candidates possess the skills your team most needs: product management skills, domain expertise, design, or information architecture.
Need domain expertise? Create a mock backlog with stories deep in the technology and have the candidate give details about why something should be built and how it should work. Need information architecture? Ask the candidate to map out your current (publicly available) documentation and marketing materials and create a backlog that improves the customer’s learning journey.
Learn something from the candidates during the interview process.
The programming test won’t always work, and the approach may not fit your company’s culture or hiring practices. But I’d argue the approach introduces objectivity and weeds bias out of the hiring process. You’ll build balanced, diverse teams with well-matched skills.
Try it. Make your product, company, and the industry stronger and more resilient.
~~~~~~~~~~~~~
I’d love your feedback and comments on this one.
What have you tried, and what’s worked for you?
What kinds of tools and techniques resulted in your best hires?
What didn’t work?
How could I improve the test?
