Building a successful product with generative AI is notoriously difficult. The path from a promising demo to a reliable, production-ready application is littered with pitfalls that catch even experienced teams off guard. The core challenge, as many are discovering, isn't the AI itself, but the product and strategy around it.
This problem often starts at the very beginning. Teams get excited by the possibilities and try to use generative AI for everything, including problems that are better solved with simpler technology. For example, one group proposed using a large language model to optimize a household's energy consumption, promising a 30% reduction in electricity bills. But a far simpler solution, like scheduling high-energy tasks during the cheapest hours, would likely be just as effective. The team later abandoned the project. This is a classic case of seeing a new, exciting tool and treating every challenge as a nail, when a simple screwdriver would do. Companies have wasted resources trying to use gen AI for tasks like anomaly detection or customer call volume prediction, where traditional algorithms are cheaper and more reliable.
AI is the Easy Part, Product is the Hard Part
Once you've identified a suitable problem, the real work begins. Many teams find that the technical aspects of the AI are straightforward, but the user experience is a major hurdle. Generative AI is changing how we interact with software, and we're still figuring out the best ways to do it. In fact, failing a product launch is often about the product, not the technology.
Consider the counter-intuitive lessons learned by real teams. A friend was building an app to summarize meeting transcripts. Her team spent weeks debating whether summaries should be three or five sentences. In the end, users didn't care about the summary at all. They only wanted a list of action items specific to them. Another team building a chatbot for skill assessment discovered that users didn't want the most correct, blunt response (e.g., "You're a terrible fit"). They wanted a helpful response that explained the gaps and how to close them. Intuit encountered a different UX friction with a tax chatbot. Users hated it until they realized people didn't know what to type. By adding a few suggested questions for users to click on, Intuit drastically reduced friction and improved user trust. The takeaway is clear: differentiation in AI products comes from the product experience, not the model itself.
Beware of Shiny New Tech
Everyone is using similar foundation models, so the AI is not your differentiator. Yet, teams still get distracted by the newest frameworks and vector databases. Don't agonize over which vector database to use for months when a simple term-based retrieval solution would work perfectly fine. Adopting complex new tools too early can backfire, especially when those tools are unproven. Framework developers can make mistakes, and their updates can change your application's behavior without warning.
The 80/20 Trap and Production Complexity
This is perhaps one of the first painful lessons anyone building an AI product learns: it's easy to build a demo, but hard to build a product. The initial success of a prototype can be grossly misleading. One team reported that it took one month to achieve 80% of the experience they wanted, but an additional four months to surpass 95%. The journey from a good demo to a great product is exponentially harder.
Challenges pile up quickly. Hard-to-control tone requests, a near-infinite combination of user queries making testing difficult, and the constant battle with hallucinations are just the start. Teams also must contend with infrastructural issues from API providers, like 10% of calls timing out, and a host of legal and security concerns around copyrights, data privacy, and training data lineage. This is why a well-known paper, UltraChat, aptly shared that "the journey from 0 to 60 is easy, whereas progressing from 60 to 100 becomes exceedingly challenging."
Don't Abandon Human Evaluation
In a rush to automate everything, many teams try to evaluate their AI applications with AI judges, or LLM-as-a-judge. While useful, relying on this alone is a mistake. AI judges are not deterministic. Their quality depends heavily on the underlying model and prompt, and if not properly developed, they can give dangerously misleading evaluations. The teams with the best products supplement automation with human evaluation. Every day, experts review a subset of outputs, which helps correlate human and AI scores and gain a better understanding of user behavior. Manual inspection of data remains one of the highest-value activities in machine learning. As one expert noted, staring at data for just 15 minutes can save hours of headaches.
Strategy Over Random Ideas
Finally, many enterprises have failed by crowdsourcing AI use cases from the entire company. This is how we ended up with a million text-to-SQL models and Slack bots. While it's good to listen to employees, individuals are naturally biased toward the problems affecting their day-to-day work, not the ones that offer the highest ROI. Without a coherent, big-picture strategy, companies end up with a series of small, low-impact applications and then conclude that generative AI has no ROI.
The generative AI landscape is still young, and mistakes are a normal part of the process. The key is to learn from them quickly. The teams that succeed will be the ones that focus on solving real problems with the right tools, obsess over user experience, and remain cautiously optimistic about the road ahead.