As we're still in the early days of building applications with foundation models, it's normal to make mistakes. This article covers some of the most common pitfalls I've seen, both from public case studies and from my personal experience.[reference:126]
Pitfall 1: Treating Everything as a Nail
Every time there's a new technology, senior engineers everywhere sigh: "Not everything is a nail." Generative AI isn't an exception. Its seemingly limitless capabilities only exacerbate the tendency to use it for everything.[reference:127]
A team pitched me the idea of using generative AI to optimize energy consumption. They fed a household's list of energy-intensive activities and hourly electricity prices into an LLM, then asked it to create a schedule to minimize energy costs. Their experiments showed this could reduce a household's electricity bill by 30%.
I asked: "How does it compare to simply scheduling the most energy-intensive activities when electricity is cheapest?" They said they would try it later. They never followed up, and abandoned the app soon after.[reference:128]
I've seen this scenario over and over again. A big company wants to use generative AI to detect anomalies in network traffic. Another wants to predict upcoming customer call volume. A hospital wants to detect whether a patient is malnourished (really not recommended).[reference:129]
Pitfall 2: Forgetting That Product Is Harder Than AI
Many teams dismiss gen AI as a valid solution because they tried it out and their users hated it. However, other teams successfully used gen AI for similar use cases. In both cases I investigated, the issue wasn't with AI, but with product.[reference:130]
Example: Meeting Summaries
A friend works on an application that summarizes meeting transcripts. Initially, her team focused on getting the right summary length. Would users prefer 3-sentence summaries or 5-sentence summaries? However, it turned out that users didn't care about the actual summary. They only wanted action items specific to them from each meeting.[reference:131]
Example: Skill Fit Assessment
When developing a chatbot for skill fit assessment, a team discovered that users didn't want correct responses. They wanted helpful responses. If a bot responds "You're a terrible fit," that might be correct but not helpful. Users want tips on gaps and how to close them.[reference:132]
Example: Intuit's Tax Chatbot
Intuit built a chatbot to help users answer tax questions. Initially, they got lukewarm feedback. After investigation, they found out that users actually hated typing. Facing a blank chatbot, users didn't know what it could do. So Intuit added suggested questions for users to click on, reducing friction and building trust.[reference:133]
Pitfall 3: Premature Optimization
Teams agonize over what vector database to use when a simple term-based retrieval solution works.[reference:134] They jump into fancy frameworks and finetuning before trying simpler approaches.
Incorporating external tools too early can cause problems. Tool developers can make mistakes. For example, I often find typos in default prompts when reviewing a framework's codebase. If the framework updates its prompt without telling you, your application's behaviors might change and you might not know why.[reference:135]
Pitfall 4: Underestimating the Journey from Demo to Product
It took one team 1 month to achieve 80% of the experience they wanted, and an additional 4 months to surpass 95%. The initial success made them grossly underestimate how challenging it is to improve the product, especially around hallucinations.[reference:136]
A startup that develops AI sales assistants for ecommerce told me that it's hard for tonal requests in the system prompt to be perfectly obeyed, and hard to create a specific set of unit tests because the combination of queries is basically infinite.[reference:137]
As the paper UltraChat shared: "the journey from 0 to 60 is easy, whereas progressing from 60 to 100 becomes exceedingly challenging."[reference:138]
Pitfall 5: Relying Entirely on AI Judges
To automatically evaluate AI applications, many teams opt for the AI-as-a-judge approach. A common pitfall is forgoing human evaluation to rely entirely on AI judges.[reference:139]
While AI judges can be very useful, they aren't deterministic. The quality of a judge depends on the underlying model, the prompt, and the use case. AI judges must be evaluated and iterated over time, just like all other AI applications.[reference:140]
The teams with the best products all have human evaluation to supplement automated evaluation. Every day, they have human experts evaluate a subset of outputs, anywhere from 30 to 1000 examples.[reference:141]
Pitfall 6: Crowdsourcing Strategy Without Direction
Unable to come up with a strategy for what use cases to focus on, many tech executives crowdsourced ideas from the whole company. "We hire smart people. Let them tell us what to do." They then try to implement these ideas one by one.[reference:142]
And that's how we ended up with a million text-to-SQL models, a million Slack bots, and a billion code plugins.[reference:143]
Individuals might be biased toward problems that immediately affect their day-to-day work instead of problems that might bring the highest returns on investment. Without an overall strategy, it's easy to get sidetracked.[reference:144]
Key Takeaways
- Gen AI isn't a one-size-fits-all solution to all problems. Many problems don't even need AI.[reference:145]
- For many AI products, AI is the easy part, product is the hard part.[reference:146]
- Fancy frameworks and finetuning shouldn't be your first course of action.[reference:147]
- Initial success can be misleading. Going from demo-ready to production-ready can take much longer.[reference:148]