To scale agentic AI, Notion tore down its tech stack and started fresh
Many organizations would be hesitant to overhaul their tech stack and start from scratch. Not Notion. For the 3.0 version of its productivity software (released in September), the company didn’t hesitate to rebuild from the ground up; they recognized that it was necessary, in fact, to support agentic AI at enterprise scale. Whereas traditional AI-powered workflows involve explicit, step-by-step instructions based on few-shot learning, AI agents powered by advanced reasoning models are thoughtful about tool definition, can identify and comprehend what tools they have at their disposal and plan next steps. “Rather than trying to retrofit into what we were building, we wanted to play to the strengths of reasoning models,” Sarah Sachs, Notion’s head of AI modeling, told VentureBeat. “We've rebuilt a new architecture because workflows are different from agents.”
Re-orchestrating so models can work autonomously
Notion has been adopted by 94% of Forbes AI 50 companies, has 100 million total users and counts among its customers OpenAI, Cursor, Figma, Ramp and Vercel. In a rapidly evolving AI landscape, the company identified the need to move beyond simpler, task-based workflows to goal-oriented reasoning systems that allow agents to autonomously select, orchestrate, and execute tools across connected environments.
Very quickly, reasoning models have become “far better” at learning to use tools and follow chain-of-thought (CoT) instructions, Sachs noted. This allows them to be “far more independent” and make multiple decisions within one agentic workflow. “We rebuilt our AI system to play to that," she said. From an engineering perspective, this meant replacing rigid prompt-based flows with a unified orchestration model, Sachs explained. This core model is supported by modular sub-agents that search Notion and the web, query and add to databases and edit content. Each agent uses tools contextually; for instance, they can decide whether to search Notion itself, or another platform like Slack. The model will perform successive searches until the relevant information is found. It can then, for instance, convert notes into proposals, create follow-up messages, track tasks, and spot and make updates in knowledge bases. In Notion 2.0, the team focused on having AI perform specific tasks, which required them to “think exhaustively” about how to prompt the model, Sachs noted. However, with version 3.0, users can assign tasks to agents, and agents can actually take action and perform multiple tasks concurrently. “We reorchestrated it to be self-selecting on the tools, rather than few-shotting, which is explicitly prompting how to go through all these different scenarios,” Sachs explained. The aim is to ensure everything interfaces with AI and that “anything you can do, your Notion agent can do.”
Bifurcating to isolate hallucinations
Notion’s philosophy of “better, faster, cheaper,” drives a continuous iteration cycle that balances latency and accuracy through fine-tuned vector embeddings and elastic search optimization. Sachs’ team employs a rigorous evaluation framework that combines deterministic tests, vernacular optimization, human-annotated data and LLMs-as-a-judge, with model-based scoring identifying discrepancies and inaccuracies. “By bifurcating the evaluation, we're able to identify where the problems come from, and that helps us isolate unnecessary hallucinations,” Sachs explained. Further, making the architecture itself simpler means it’s easier to make changes as models and techniques evolve. “We optimize latency and parallel thinking as much as possible,” which leads to “way better accuracy,” Sachs noted. Models are grounded in data from the web and the Notion connected workspace. Ultimately, Sachs reported, the investment in rebuilding its architecture has already provided Notion returns in terms of capability and faster rate of change. She added, “We are fully open to rebuilding it again, when the next breakthrough happens, if we have to.”
Understanding contextual latency
When building and fine-tuning models, it’s important to understand that latency is subjective: AI must provide the most relevant information, not necessarily the most, at the cost of speed. “You'd be surprised at the different ways customers are willing to wait for things and not wait for things,” Sachs said. It makes for an interesting experiment: How slow can you go before people abandon the model? With pure navigational search, for instance, users may not be as patient; they want answers near-immediately. “If you ask, ‘What's two plus two,’ you don't want to wait for your agent to be searching everywhere in Slack and JIRA,” Sachs pointed out. But the longer the time it's given, the more exhaustive a reasoning agent can be. For instance, Notion can perform 20 minutes of autonomous work across hundreds of websites, files and other materials. In these instances, users are more willing to wait, Sachs explained; they allow the model to execute in the background while they attend to other tasks. “It's a product question,” said Sachs. “How do we set user expectations from the UI? How do we ascertain user expectations on latency?”
Notion is its biggest user
Notion understands the importance of using its own product — in fact, its employees are among its biggest power users. Sachs explained that teams have active sandboxes that generate training and evaluation data, as well as a “really active” thumbs-up-thumbs-down user feedback loop. Users aren’t shy about saying what they think should be improved or features they’d like to see. Sachs emphasized that when a user thumbs down an interaction, they are explicitly giving permission to a human annotator to analyze that interaction in a way that de-anonymizes them as much as possible. “We are using our own tool as a company all day, every day, and so we get really fast feedback loops,” said Sachs. “We’re really dogfooding our own product.” That said, it’s their own product they’re building, Sachs noted, so they understand that they may have goggles on when it comes to quality and functionality. To balance this out, Notion has trusted "very AI-savvy" design partners who are granted early access to new capabilities and provide important feedback. Sachs emphasized that this is just as important as internal prototyping. “We're all about experimenting in the open, I think you get much richer feedback,” said Sachs. “Because at the end of the day, if we just look at how Notion uses Notion, we're not really giving the best experience to our customers.” Just as importantly, continuous internal testing allows teams to evaluate progressions and make sure models aren't regressing (when accuracy and performance degrades over time). "Everything you're doing stays faithful," Sachs explained. "You know that your latency is within bounds."
Many companies make the mistake of focusing too intensely on retroactively-focused evans; this makes it difficult for them to understand how or where they're improving, Sachs pointed out. Notion considers evals as a "litmus test" of development and forward-looking progression and evals of observability and regression proofing. “I think a big mistake a lot of companies make is conflating the two,” said Sachs. “We use them for both purposes; we think about them really differently.”
Takeaways from Notion's journey
For enterprises, Notion can serve as a blueprint for how to responsibly and dynamically operationalize agentic AI in a connected, permissioned enterprise workspace. Sach’s takeaways for other tech leaders:
Don’t be afraid to rebuild when foundational capabilities change; Notion fully re-engineered its architecture to align with reasoning-based models.
Treat latency as contextual: Optimize per use case, rather than universally.
Ground all outputs in trustworthy, curated enterprise data to ensure accuracy and trust. She advised: “Be willing to make the hard decisions. Be willing to sit at the top of the frontier, so to speak, on what you're developing to build the best product you can for your customers.”