Design sprints are a powerful tool for rapid software development, and now it’s an even better tool in building Generative AI (“GenAI”) products. In this article I’ll share my experience as a PM using a version of the Design Sprint method to quickly test GenAI solutions, discuss tradeoffs and provide a step-by-step guide get your team on their way.
Reacting to Generative AI Product Disruption
Time was not on my side. As a Product Manager on the Google Assistant in Home, we had a few months to prove out Generative AI concepts given shifting priorities. Thanks to our fearless directors,* we paused our existing roadmap and got to work fast prototyping with large language models (“LLMs”).
While the demos around new kids features were impressive and by far the most complex voice AI experience I had ever interacted with, the team was left wondering, what exactly should we focus on? There’s too much opportunity, unclear priorities, and just not enough time.
At Google, ‘Design Sprints’ are just part of our routine — we need to create forward progress when our roadmap and goals are ambiguous, and as product leaders, we need to bring the team along for the ride. I love Design Sprints, first encountering them during a workshop with Jake Knapp at Twitter earlier in my career. And ever since, I’ve been running these workshops to take the ambiguous and finding opportunities to rapidly advance. In fact, our design sprint at Lyft led to Rides for Others and Family Accounts.
GenAI Considerations
Despite this repertoire, nothing could prepare me for the shift in product management from LLMs with the birth of ChatGPT. As tech leaders it’s paramount to understand what GenAI is good for and the limitations.
My First GenAI Sprint
Based on our prototypes, we generated data and thought through potential problems we wanted to solve and considered the constraints. With the holidays fast approaching we needed to move very quickly to get a scalable demo in front of executives. Here’s the process:
- Problem Definition: We used prototypes to understand problems suitable for LLM integration (e.g., novel voice experiences).
- Ideation: We sketched solutions, prioritizing those that capitalized on GenAI’s strengths.
- Rapid Prototyping: We employed prompt engineering to create testable demos, evaluating user experience as a primary success metric.
At a 3 day huddle we discussed several areas that could be funded, sketched solutions and began working on a short term roadmap. The result was really neat as we took select queries Assistant couldn’t handle, applied LLMs, creating a novel experience.
GenAI Sprint — How to Run Fast
Since this experience, I’ve tweaked the approach and have run it as part of Google.org’s AI Incubator with organizations who are less familiar with technical design sprints.
The key difference from a standard design sprint is that (a) it’s compressed (b) meant to be iterative and (c) allows testing with a real GenAI prototype for feedback. GenAI offers endless possibilities, and deciding between them can be paralyzing. Let’s talk about each stage and the approach.
Pre-work: You’ll want to bring in data to help make decisions. For example, if you have select problem areas, focus in on some of the user behavior and metrics. Then consider what GenAI is good at and not good at, this allows you to quickly rule out focus areas for the sprint. Also, ensure a technical lead is briefed and can talk through approach limitations, including scalability.
Ideate: Invite experts and hold a quick panel with targeted questions to educate the group on the problem space. Move into groups and sketch out solutions. Finish the day voting and asynchronously getting the decision maker to weigh in on where they want to invest.
Prototype: Recap the previous day and begin storyboarding as a smaller group. Have someone familiar with prompt engineering to roughly demo a possible experience. Use tools such as AI Studio or ChatGPT Developer Platform. Here’s how I might approach it —
- Make a decision on the user experience you want to test
- Write it in plain language as a series of steps with the specific instructions and data that might be needed. See tips.
- Prompt engineering! As model sizes increase you can pretty much take any raw set of instructions and see what the LLM will produce.
Learn: Arrange customer interviews and go through your scrappy prototype. Create hypotheses before and ensure you’re able to get some directional guidance on those issues. Follow standard Design Sprint methodology for testing. Finally, decide next steps and goals.
Scaling Your GenAI Prototype
So, what’s next? If you decide GenAI is a useful technology for your product you should begin the process of collecting the data that’s necessary for a minimally viable product (“MVP”), write out example queries and responses with experts, and begin the process of fine tuning. Consider:
- Strengths and Weaknesses: GenAI excels at creative tasks but can struggle with accuracy. Prompt engineering and data analysis are crucial to understanding its potential and limits in your context.
- Deployment Challenges: Start with smaller, well-defined projects. Collaborate with engineers to anticipate technical constraints.
Data Design & Considerations
It’s critical to have experts review and provide feedback given hallucination and how sensitive your customers may be to the end result. For example, if you were to work on a legal analysis tool I might suggest constraining it to a subset, getting the experience right and ensuring your prompt design limits responses to only this domain.
Consider creating a very high quality set of examples, let’s call it the “platinum data” set. Vet it very carefully and ensure it’s highly representative of the problem area you’re hoping to tackle. Now, feed it into a very large LLM, asking it to ‘expand’ and create more examples and variations which we’ll call the “golden data” set. This could be used to then run on a smaller (and more cost effective) production stage LLM. You’re essentially leveraging a larger brain to help you tell a smaller but cheaper brain to follow the golden data set examples.
Create a methodology of scoring, and rigorously track through evaluation methodology. For example, if you expect the model to return back an answer that contains critical elements, have a human evaluation of those results and track over time to feed back into the model for fine-tuning and adjusting data sets as needed.
Embracing GenAI to Learn
Finally, embrace the rapidly evolving space: GenAI tooling and models are shifting every month, I would recommend getting an MVP out there, learning and iterating. Because LLM are constantly evolving, I also recommend tracking quality, thinking about model drift, continual fine-tuning (using your data set and evaluations). Good Luck!
Resources
*Hat tip to my fallen colleagues, we advanced the voice experience by a decade in those last few months.