The Four Signs of Success for an AI Enterprise

Agentic artificial intelligence (AI) has become somewhat of a buzzword in recent months. But Intercom’s Co-founder and Chief Strategy Officer Des Traynor wants to dispel the myth that an AI customer solution “is easy and you can build it yourself.”

Intercom’s solution is Fin, an AI customer service agent that it rolled out in March 2023; its latest version, Fin 3, launched this year. With an average success rate of 65% for customer query resolutions, Fin is designed to be deployed and trained by Intercom’s clients to eventually deliver “the mythical 100%.” 

When it comes to AI rollout, expectations for customer service companies are high. A December 2024 survey by research firm Gartner found that the majority of respondents thought customer service organizations had the primary responsibility to “identify new AI opportunities”, “roadmap the evolution of AI initiatives” and “drive the adoption of AI initiatives”, as shown in the graph below. 

In the latest edition of OPTO Sessions, Traynor discusses how the advent of AI changed the software development process, the process of improving AI agents, and his four criteria for successful AI-driven companies. 

Factories vs Labs

In the past, Traynor explains, product development for software-as-a-service (SaaS) firms involved a “fairly predictable routine, which was: research the problem, understand the users, design a solution to the problem, build the solution, go through user testing, and ship and iterate” — a process he compares to the operation of a factory floor.

Now, Traynor says, development has gone from a “linear process with high certainty” to a more experimental “lab mindset”: “AI has introduced this extreme layer of probabilism where you don’t really know if something’s going to work. If it does work once, will it work every time?”

This new reality means developing successful AI involves “experiments, uncertainty, empirical evidence, consistent evaluation, always trying new things.” To ensure Fin works, and works well, the company takes a rigorous scientific approach. “Most of our AI group are scientists,” Traynor explains. 

The result? A complex, rigorously evaluated model. “There are 27 or so different components of Fin. The number changes all the time.”

When it comes to reaching peak effectiveness, however, the secret may not just be in training the AI model, but in training the humans who use it. 

“I think, given sufficient knowledge, Fin is already smart enough to deliver the mythical 100% [of ticket resolutions]. It’s not more intelligence that we need to make these things perfect: it’s largely a function of actually understanding the back office, what’s actually going on inside the product.”

This is where training data comes into play. “You need to teach Fin certain procedures,” Traynor explains. “You need to keep it up to date with all your product updates.” Additionally, the agent is designed to learn from the tickets it cannot resolve, as well as those it can. “Every time Finn hands [a ticket] over to a human, it doesn’t just throw it over a wall and look away. It follows that conversation to work out what happened in reality with the human.” 

As a result, the percentage of tickets that have to be handed over to human agents falls progressively. “Our goal with Fin is that the human support team should be answering things for the first time and the last time. One human writes the canonical response, Fin ingests that, and then that query never gets handed over again.”

Eventually, Traynor sees the agent reaching a point where it can train itself with access to a given company’s product updates, policy and press releases. “Could Fin possibly read the source code of a product to infer how it works? Then, the business would never have to tell us, [Fin] would just know because it could read.”

Ultimately, this would represent a key breakthrough in Fin’s effectiveness. “The degree to which Finn can interrogate internal business processes and machine code, and actually draw on new knowledge that wasn’t previously codified within a business, would massively increase the amount of resolutions it could deliver.”

As customers learn how to interact with AI, Traynor sees the effectiveness of solutions like Fin getting a further boost. “The biggest evolution we’ve been slowly noticing is that once people realize that the best way to engage with a bot is actually to treat it like it’s a human … you’ll get a way better outcome.” 

The Four Qualities of AI Success

There is a huge amount of AI products on the market, and it can be difficult to spot potential winners in the increasingly crowded space. As an investor himself, Traynor has identified four key criteria he believes point to success in developing an AI product. 

First, “it’s really important that the product is used by customers”. A lot of AI solutions are what Traynor calls “shelf-ware”, meaning the product “was bought, it’s sitting on a shelf, no one’s touched it for a year.”

Second, the product should solve a real business problem, as opposed to being a “cool tool” that is fun to play with, but doesn’t actually improve clients’ operations. 

Third, “there needs to be deeply differentiated AI … there has to be a real gnarly problem at the core.” By confronting a specific, difficult problem with a complex, custom-made solution, companies also have more assurance that their product cannot be easily replicated by competitors. Especially in the AI ecosystem, Traynor cites convergence as an underrated risk: “At some abstract level, everyone’s building the same thing.”

Lastly, the potential for profitability is key to long-term sustainability. “There just has to be some evidence that the margin can turn positive,” Traynor underlines. “If you’re burning the entire haystack to find the single needle and people don’t even really care that much about the needle, that’s not an efficient business.” 

In many ways, building a successful product means continually returning to the initial problem you mean to resolve. “Every time you’re releasing a product and you have like a red blinking light or you pop an error message, you should really assume that’s the beginning of a support conversation.” By keeping this in focus, SaaS firms can stay ahead, Traynor says. 

“Proactive support will become the future battleground of this space.”