Building an AI Product: Roles and Responsibilities

Machine learning is all the rage today, and lots of people are interested in a machine learning career. When I get a chance to chat with people that “want to do machine learning stuff”, I usually ask what roles they’re interested in, and they usually mumble something like “Data Scientist”, in a very unconvincing way. 🙂

The good news is that in addition to data scientists, lots of other roles contribute to the process of building AI products. To highlight these roles, let’s walk through the process for building an AI product, and see who we need along the way.

According to Yael Gavin, a former Facebook PM (who by the way has a great series of posts on machine learning for PMs), the process for building an AI product (or ML model) is as follows:

1. Ideation: This is when the product team comes together to align on their goals, customer problems they want to solve in order to achieve those goals, and metrics to measure success. While this stage is primarily driven by the product manager, engineering lead, and UX designer, additional contributors include the product analyst, who helps the team to think deeply about metrics, and the data scientist who helps the team understand how machine learning capabilities can help solve the customer’s problem.

2. Data Prep: Here the product team decides how to collect the data necessary for training their ML models, and computing the metrics needed to measure their success. This stage is driven by engineering, with help from the product analyst, who makes sure that relevant metrics can be calculated, and the data scientist, who makes sure that required ML model inputs (also known as features) can be produced from the data.

3. Prototype & Testing: This is the heart of the model building process. Here the data scientist trains the ML models and checks to make sure that the predictions or classifications from the trained models are within acceptable levels of accuracy. This highly iterative process is driven by the data scientist, with some help from the product analyst and engineering. The product analyst tracks model accuracy metrics, while engineering helps the data scientist understand the data, and delivers additional data where necessary.

4. Productization: At this stage the model is doing what it’s supposed to, and is ready for primetime. The product manager, UX designer and engineering lead the process of hosting the model in a scalable manner, and building the delightful user experience that leverages the machine learning model to actually solve the customer’s problem. Additional contributors to this stage include the product analyst, who tracks progression of metrics towards pre-determined targets, and the data scientist who monitors how the ML models perform in production, and makes improvements in an iterative manner.

As you can see, traditional product roles like product managers, software engineers, and so on contribute heavily to the process of building AI products. The reason for this is that your customers don’t care about fancy machine learning models; they care about delightful user experiences that allow them shop, learn, travel, connect and create more, but faster and cheaper.

The proven model for delivering these delightful experiences is still the goal-oriented, cross-functional, and autonomous product team. However, this team has a new member: the data scientist. What this means for other members of the team is twofold. First, educate yourself on the basics of machine learning. You don’t need to become an ML expert, but the UX designer for example should know as much about machine learning processes as he does about software engineering. Second, develop the ability to identify customer problems that were unsolvable in the past, but could now be solved with machine learning. For example, you should be able to tell if a customer problem sounds like a prediction problem, or a classification problem, so you can brainstorm with your data scientist on what is possible solution wise.


Tricks of the Product Management Trade

Every profession has tactics used by experienced practitioners to be more efficient and productive. Product management is no exception, and in the past few years I have picked up a few tricks of the PM trade that I believe are worth sharing. Three of them are as follows:

1. N.I.H.I.T.O: I learned this concept from the product management classes offered by Pragmatic Marketing. NIHITO is an acronym for Nothing Important Happens In The Office. What this means is that as a product manager, you need to get out of the building and frequently engage with customers to identify new problems to solve, and new opportunities to go after.
Product demos, customer visits, and stakeholder management rarely happen from your desk (unless you’re using video conferencing technology). This is why there tends to be an inverse relationship between the number of hours you spend at your desk per week, and your overall productivity as a PM.

2. Date for A Date: At the very early stages of a project, you usually don’t know when it will be delivered as there’s still too many unknowns. Unfortunately, this doesn’t stop your stakeholders from  constantly asking you for a delivery date! They’re not trying to be mean or anything, they just need to make their own plans which happen to be dependent on yours. Instead of telling them that you don’t know, share the date by which you will know, and spend the time between now and then identifying and eliminating the key risks of your project. Once this is done, you can hopefully share a still imperfect, but better informed estimate of when your project will land. This will go a long way to increase your credibility and professional reputation.

3. Meetings Before The Meeting: Your role as a PM might involve selling a complex idea to multiple stakeholders in a high-stakes meeting. If this meeting is the first time your stakeholders are seeing your idea, you will most likely fail. The reason for this is that this setting is actually the wrong one for having the detailed conversations needed to get buy in from your stakeholders.
The way around this problem is that before the big meeting, meet with key stakeholders individually to share your idea and give them the opportunity to raise their concerns. It’s a lot more work of course, but it dramatically improves your chances of success.

I hope you find my tricks helpful. Also, if you have tricks of your own please share! I’m always looking for my next career/life hack. 🙂


3 Books I Read in 2017: Worth Their Weight in Gold

I decided to take a page from my buddy Rohan Rajiv’s playbook, and reflect on the books I read this year that really resonated with me. They are:

1. Inspired: How To Create Products Customers Love by Marty Cagan: This book is about nine years old, but it still totally nails the contemporary issues that product managers (PMs) face. What should be the nature of the relationship between product management and marketing teams? What about product management and engineering? How can PMs really understand customers, assess new opportunities, or recruit development partners? It’s all addressed in this book from a very practical perspective. There’s a new edition out that came out this month which I’m definitely looking forward to reading.

2. Daring Greatly: How The Courage To Be Vulnerable Transforms the Way We Live, Love, Parent and Lead by Brene Brown: Since I watched her insanely popular TED video, I’ve become a huge fan of Brene Brown and her work around vulnerability. This books dives into how to use vulnerability in key aspects of our daily lives, just like the title says :). A key learning for me was that having the courage to be vulnerable is not a sign of weakness; it is a sign of strength. Also, a popular but counterproductive way for us to deal with our shame is to look for and judge people doing worse than us in that area of our lives (parenting is a great example). Finally, when we take the time to reflect on the reasons why we feel shame, we usually realize that things aren’t as bad as we think. Great book!

3. Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland and JJ Sutherland: Most product teams today build their products using some variation of scrum, which makes this book quite relevant as  it addresses how to get product development right using scrum. The book contains lots of good war stories about the scrum-adoption journey of very different organizations (I really liked the FBI Sentinel story), but the key insight for me was that great product teams are autonomous, cross-functional, empowered and goal-oriented.

That’s it! Cheers to a New Year, and to continuous learning!

Two Years In Product Management: What I’ve Learned

About this time last year I did a blog post on what I’d learnt after spending a year as a product manager. It turns out that I picked up some additional things in my second year of product management that I believe are also worth sharing. Learning is really supposed to be a lifelong process so this shouldn’t be too surprising right? Okay here’s what I learned:

1. A Product Manager’s Greatest Contribution To their Product Team is Customer Insights: While product managers can wear many hats (UX, project management, analytics, e.t.c), I believe that the greatest contribution that a product manager can bring to their team is customer insights, or knowledge of their product’s buyers and users.
There are two reasons for this. First, customer insights are essential as most great products start with great customer insights. Without a deep understanding of the customer, great engineering, UX and analytical skills will not result in a successful product. Therefore, product managers have to make sure that their product teams never lack this important ingredient. The second reason why a product manager’s greatest contribution to their team is customer insights is that within the product team, the product manager is ultimately responsible for customer insights, just like the software engineer is responsible for delivering amazing code, and the UX designer is responsible for delivering compelling and delightful product experiences.

2. You Cannot Outsource Your Product Strategy To Your Most Important Customer(s): We all know how important it is to listen to customers. What may not be as obvious is that it is possible to listen too much to customers. This happens when product managers develop their product strategy and roadmaps based on customer input alone. This makes sense at some level right? If you want to build what customers want then just go ahead and ask them! What makes this approach risky is that in most cases, the customers that product managers get input from may not adequately represent their product’s current and future customer base. Customers who usually talk to us are tiny segments like the really big customers, the really sophisticated ones, or those who really love or really hate our product. In order to create a holistic product strategy, we need to consume and distill additional sources of customer insight such as actual product usage data, competitive analyses, and – where applicable – feedback from customer support and sales.

3. Data-Driven Product Development is Hard: A few months ago I did a blog post on data-driven product development: a systematic way for ensuring that product investments actually deliver the right outcomes to customers and the business. What I’ve learned from helping other product teams become more data driven, and from the data-driven journey of my own product team, is that this stuff is really hard to do. Few things suck as much as working on a new, “game changing” feature for months, and then discovering that the feature did not impact any of your key business metrics after release. The very real temptation then is to go back to the old days of celebrating the release of a feature without checking the impact of that feature on your business. This will definitely guarantee lots of good feelings all around within your product team, but it will not guarantee the future of your product, your team, or your current job. 🙂

That’s what I’ve learned this past year. What have you learned in your own role?


I Want To Use Data To Improve My Product. Where Do I Start?

Most technology professionals will probably agree that data-driven business and product decisions deliver better outcomes than simply going with your gut, or worse: with the HiPPO. However, an important challenge remains for product teams trying to become data-driven: where do they start?

Below is a series of steps for navigating the path to data-driven product management. These steps could be applied to an entire product, or to a feature existing within a larger product. Also, while it is focused more on technology products, I believe parts of it could be applied outside the technology industry as well. Here goes:

1. Does your product work?: An awesome app that crashes every single time it’s opened is a worthless app. Therefore, the first set of data and metrics to be collected and analyzed should revolve around product health and operational intelligence. This will help you answer questions like how many hours in a day your product is available, what your crash rate is, and what your product’s performance is like during peak times.
Examples of publicly available product health and operational intelligence tools include Splunk and the elastic stack, or ELK.

2. Is it popular?: Once you get going with analyzing your product’s health, the next step is product and user analytics. This means collecting the data and developing the metrics needed to figure out how many people are using your product, and whether that number is trending up, or down. You also want to analyze your user base to identify your product’s user segments, and to begin understanding which segments are valuable, and what the valuable segments care about so you can tailor your product roadmap and marketing programs accordingly. This is not easy stuff, but getting it right pays huge dividends.
Publicly available tools for enabling product and user analytics include Localytics, Mixpanel, and Amplitude.

3. Are product updates changing key metrics?: Every update made to your product that results in no change in any of the metrics that you care about is probably a worthless update. On the other hand, increases or decreases in your business and product metrics may be as a result of external factors such as the weather or U.S elections. 🙂 One way to improve your certainty around what effects your updates are having on your key metrics is by running experiments. In very simple terms, this involves releasing product updates to a subset of your product’s user base, and then comparing the subsequent behavior of the users who  received the update with those who didn’t receive any product updates.
Publicly available tools for enabling experimentation include Optimizely and Unbounce.

4. Is your product Intelligent?: Artificial intelligence (AI) is all the rage today, as it promises to unleash a new wave of tech-enabled productivity on consumers and businesses. However, a critical ingredient for enabling AI scenarios is data. For example, the reason why Amazon can tell you that people who bought item X also bought item Y is because it has huge amounts of transaction data on both items. For your product to be able to support such intelligent scenarios, you need to collect and organize the relevant data sets, and then employ the right algorithms.
The process of enabling AI scenarios within a product usually involves custom development by software engineers. However, publicly available tools that can facilitate this process include Amazon ML and Azure ML.

Image by

A Year in Product Management: What I’ve Learned

A few weeks ago, I celebrated my first year as a product manager. Much to my disappointment, I didn’t get a whole lot of congratulatory messages. Only LinkedIn remembered (thanks LinkedIn!).

Self reflection is something that I truly value, so I decided to take some time to think about what I’ve learned about the product management discipline during the past year. I came up with three key findings:

1. Wearing Multiple Hats is the Name of the Game
Before I became a product manager, my thoughts about the role can be summed up in this beautiful Venn diagram from the pm heels blog:


Now that I have a year’s worth of product management experience, I’ve modified this diagram to become something like this:


2. It is Important to Always be Working on the Most Important Things

From my product management Venn diagram, you can see that there is an infinite number of tasks that a product manager can be involved in. Yes those tasks can have UX, business, or  technology components. However, they can also involve random stuff like getting pizza for engineers who are working late (I haven’t had to do this yet but I happily will).

This brings us to the problem of prioritization. Since no one can do everything, product managers have to constantly check to ensure that whatever they’re working on has a higher chance of improving their product’s success metrics compared to other things that they could be doing. In case you’re wondering, there most definitely are some cases where the most important thing a product manager can do is get pizza for engineers who are working late. 🙂

3. Relationships Matter

I once heard someone describe product management as “the grease and the glue of product development”. It’s up to us to rally our stakeholders around a common goal, and to ensure overall progress towards the achievement of that goal. This means that we have to constantly influence our peers in design, development, and marketing. As none of these people work for us, it really comes down to influencing without authority.

What I’ve learned in the past year (and from folks like Robert Caldini) is that people who like and respect you are more likely to listen to and understand your ideas. In the same vein, you are more likely to listen to and understand the ideas of people that you like and respect. This insight has led me to always try to make relationship investments up front, before they were needed; by then it was usually too late.