S08E03 · How Notion Runs Its Organization in the Age of AI

0:02 I have designers writing code. I have engineers designing. We have PMs. In hindsight, that was a complete waste of time because nobody wanted a button that generated AI. We'll look at the thumbs down and that helps us understand where the gaps are. And then we try and use it ourselves. Bonjour tout le monde, je suis Dimitri Baeli de Tech.Rocks et je vais continuer en anglais. Hello everybody, welcome to the Tech.Rocks podcast. Today, we are preparing, continuing to prepare the Tech.Rocks summit for December and we have the luck to have Shir from Notion. Notion will have a booth at the summit and we have some time to discuss together. So, Shir Yehoshua is an AI engineering lead at Notion, and I'd like to know a bit more about you. We'll have some discussion about leadership, the adventures of Notion currently.

1:05 Welcome. Hi, thanks. Thanks for having me. Great to be here. So tell us a bit about you at Notion. We'll have some discussion about leadership just after. Sure. Yeah. So I joined Notion just over four years ago, was there to start our AI team a few months into joining and have been trying to build AI and product features at Notion. since then. It's been a wild and crazy journey, but really fun. Four years, it's eternity for AI. Do you remember four years ago or not much? Yeah, well, so I first joined before AI Wave hit to work on platform and API. And it's actually been really cool to see all of that work come back as more and more agents can write code. Our APIs also are more important. But the early days, I remember deeply seeing LLMs saying, oh my goodness, these are super powerful, but like, what in the world are we going to do with this technology?

2:10 And a lot of trial and error, build something, see if it's useful, learn that it's not useful, shut it down, build something else, see that there is usage there, reinvest more. And so a zigzag of a journey, but lots of trial. Let's talk a bit about leadership. Do you have your own definition of who and what is a leader in the tech industry? I think a leader is whatever the team needs from you. Sometimes it's a very servant leader where you're just unblocking. Sometimes the team needs a very crisp, concrete vision. Sometimes the team doesn't need a leader. They have a bunch of leaders within there, so you just need to get out of the way. And so I think ultimately the best leader, especially a technical leader, is one that can understand. enough of what's going on to become the thing that the team needs. Quite a lot of work to do currently. Yes. What type of leader would you define for you?

3:11 I very much try and ascribe to that same philosophy of become a chameleon, try and contort myself into what's most necessary, fill in the gaps when there are gaps, if it's a capability I don't have. So for example, I'm not somebody that knows how to train models. So I'll hire somebody to train the models. I will dive into the technical details when I need to, dive into the product details when I need to, and get out of the way when I need to. It's good. Filling the gaps. I define it as a super sub myself. I'd be always available as a super sub in the team, but it's a good definition. Filling the gaps. What's your job? I'm filling the gaps. Do you have an example or recent one where you filled the gap? Yeah, so in a few different occasions, you know, we'll be lacking maybe a product manager resource where you don't have somebody available full time. And so I'll lean in, work with the team, come up with the product roadmap.

4:11 And so did that recently as we were trying to figure out what parts of our AI agents capabilities were most important. There's also times where we're making really big decisions between a couple of different foundational investments. And so working with a team to help be. A tiebreaker if they need on that. It's good for a team to know that you're there when they are missing things. It's a good thing. Are you learning to code or coding again? What's your relationship with coding? Yeah, so I actually, when I joined Notion, all of my experience before Notion was in C++, and Notion is a TypeScript shop. So not super relevant, my C++ knowledge. And when I first joined, I... I actually did a whole bunch of kind of side quests to learn TypeScript, but I never quite got good enough. I never quite had the need to code a lot.

5:12 But now I don't need to know TypeScript. I just need to know some engineering fundamentals, which I have from the C++ days. And so now I'm generating way more code than I would have otherwise. And, you know, sometimes fixing a feature, I'm building a lot, like writing a lot of scripts to just automate things for myself. All the bad patterns of not getting a manager coding on production are forgotten. So we'll discover if it's an issue or not. Oh, yeah. And the biggest thing out there on coding is that I'm also using agents to read and understand the code base, which is something that I didn't used to do before. But now I can just ask, hey, how does this thing work? And use that to gain an intuition, especially when I'm trying to. That's the magic currently. And we'll speak a bit about that because probably in Notion, AI did few changes. We all speak about AI. AI is a bit changing the world. It would be interesting to have a kind of a summary of how AI hammered or helped Notion to evolve the last four years.

6:21 Do you have a kind of a story to tell around that? Yeah, so AI, the most obvious way it helped Notion is... It was a boon for our business. I think we've seen our growth accelerate as we build more and more with AI and as people use Notion more and more as their place to store context and place to run agentic workflows. The business likes it. In terms of how we work internally, there's a few different types of changes I've seen. One is just straight automation. We have regular triage processes, feedback gathering processes, customer calls, etc. All of that is automated now, and I have access to more of that information. I don't have to go through a bunch of different kind of convoluted docs to figure out what a customer thinks or what feedback is on a feature. I can just ask AI and all the information has been cleaned up and automated for me. So that's on the like kind of internal communication and exchange of information has just been made much more efficient.

7:24 But then I think the thing that I didn't expect that was kind of a surprise, although in hindsight should not have been, is how much the functions are getting blurred. The functions? The functions between designers, PM, software. I have designers writing code. I have engineers designing. We have PMs. It's awesome to see and I am experiencing your team is just your starting point now. It's just the problem that you're starting with, but how you solve it, the org boundaries are disappearing. And I'm seeing that even across different... different kinds of engineers. An interesting example is you have your backend engineers who could really benefit from a tool to be able to debug something or figure something out. And historically, they're backend engineers. They're not frontend engineers, so they wouldn't have built the UI. But now there's nothing stopping them. And so you're also seeing people extend past their org and past their team, past their official skill set to actually just solve the problem.

8:33 And so it's been a huge unlock as well. Yeah. So mountains of data, mountains of information. Everybody's swapping roles. How do you organize and plan work within that mess? Sorry for the word. No, it is definitely messy at times. We take kind of an extreme position, which is that we don't plan. That's a dream for me. I would call it continuously figuring out what to do. And so it's a very iterative cycle where as we get towards the end of one feature, one product, one ship, maybe a few weeks before we start to think about, okay, what is the next thing we want to do? And that is the perfect moment to figure it out because that's also when you're getting the most feedback on the product feature itself. So we've moved to this world where it's like this continuous process of as somebody frees up, you kind of place them somewhere. somewhere else and you can manage that process better because you no longer have as specialized needs for each feature.

9:39 You could have a backend engineer do front-end work, a front-end engineer do backend work. It's not ideal, but it's freeing and opens up the set of possibilities. I would frame it as more value-oriented than contributions or scope chaining one after the other. So it's key. the waterfall system. How do you manage to not start everything at once? Because if it's an iterative discovery, there's a risk to do everything at the same time. FOMO being at its best. How do you... Avoid a bit of that or do you just jump in it and try everything and fail everything? I don't know. Yeah. So, I mean, I think it's important that when you try something and you don't follow through with it, that you actually properly cancel it and properly shut it down. And so try and hold this principle of, yes, we're experimenting all the time.

10:40 We're trying new things all the time. But we're not trying everything all the time all at once. We're being focused in where we're making that investment. And then if the investment is not working, shut it down. So it's fail fast. Exactly, exactly. Which I learned before Notion, I was at Waymo, learned deeply from Waymo. How to fail? No, sorry. Fail, yeah. Because if you're trying to build something that has never been built before, which is what we're doing at Notion now too, you have to try and you're not going to know if it's going to work until it works. Learning to fail, and in case it doesn't fail. Yes, absolutely. It's interesting. I'm not sure how in France we are used to do that. We love planning. I don't know if there are some cultural... differences here. Do you have some French people in San Francisco? And do you see some behaviors being different around planning, for example?

11:40 I don't know that I see planning differences specifically, although one thing that I really appreciate, we have one French person on a team, and one thing I really appreciate about him is he makes sure that we all do team lunch together every day. Which is a very important part of the culture. It actually breeds a lot of team trust as well. In terms of planning, I think, I don't know that it's, I mean, I'm not French, so I can't say, but I think it's also various company to company. It's not that we're not planning at all. It's that we're not planning the like quarterly, yearly, everybody gets together, spend two, three weeks coming up with a document. But you're still constantly thinking about what you're going to do. Let me try to rephrase about a bit the planning way. I would expect that you're not planning solutions, but you're more planning problems to solve. Am I right? Yeah, we're charting out which area to explore. The reason I always hesitate to call it planning is because you might chart out an area to explore, spend two hours exploring it and realize that there's nothing there.

12:48 And so you don't want to plan to work on it, right? If you know very quickly that there's nothing there. How many topics are open at the same time? I'm not going to answer your question directly, but the way that I think about it is that each person should not be working on more than one thing or more than one area at a time. And ideally, you also have your sort of uncharted territories. Well, you'll scope them out faster, understand if there is something there is not something there, if you have multiple people exploring it at the same time. And so I think of it as not like how many things can we do at once, but how few things can we do and like, you know, explore them thoroughly. And did the speed of coding has changed the way of delivering value for Notion? Yeah. Because there's people saying that speed of coding hasn't changed the way projects are delivered because issues are outside now. Did you succeed to find a way to produce more with the speed of coding currently?

13:50 Yeah, so I think the biggest difference in how we work with agentic coding is the process of figuring out what it is that you're trying to build. Basically, the strength of a prototype. So these days, almost everything that we build starts with a prototype. So for example, even with we are rebuilding the data model for how we store version history, it was a big, meaty project, but it's so important because now that you have agents running wild in a workspace, you need attribution for every single thing the agent did and which human controlled that agent. And so the data model needs to change. In order to explore what the new data model should be, instead of like going into a room thinking hard for two weeks, the engineers actually prototyped how hard it would be to consume a couple different versions of the data model. How hard would it be to build this diff view, to build that diff view, to show and surface this attribution?

14:52 And that helped us pave the path for what the right solution was. And so that is the way in which we are building is working. It's not that that code was shipped. But that could help us make a decision. Yeah, it's a dream come true about prototyping, throwing, and spending a lot of money in trying things and dropping. But the way of dropping things is really interesting. Do you have a story of forgetting to drop something that is failing? Absolutely. We have many. Although we're trying to get better at this over time. The one that I learned the hardest lesson from was in the early days of AI. We have a button feature where you can set up an automation. You click this button, it generates this page, it creates a database, whatever. A few different options there. And at the time, I had the strongest conviction that we should just add an AI step to that button. Click the button, then AI will do something. And I had these like big dreams that this would, you know, revolutionize everything.

15:56 This was like early 2023. So LLMs were still pretty bad. And we built it out fairly quickly. But you clicked the button and AI had so much trouble just deciding where to put the information. Sometimes it would put it on top of the button, below the button in a new page. And we spent maybe a few weeks just trying to like get the LLM to always put the text below the button. In hindsight, that was a complete waste of time because nobody wanted a button that generated AI. We forgot to value. that actually having AI run on a button click was even a good idea in the first place. And we got lost in the sauce of trying to have it put the text in the right place. I have plenty of images of AI buttons everywhere in many apps currently where I'm not clicking in it. Exactly. It reminds me of the clippy feature in Microsoft. This one, we should bring it back. Maybe now it will work.

16:56 Maybe. Yeah, and LLMs are better now, so it probably wouldn't have that. Yeah, but do you want some help? No? Thanks. I'm okay with that. And do you have some tricks now to be triggered yourself about a project that is lasting too long? Do you have some way to be reminded? Yeah, I think we have this internal motto of meet reality. As often as possible, we want to meet reality. And so that means find customers, design partners who are willing to trial something out before it's fully ready and, quote, meet reality with them. And so most I did this actually this morning. I asked the team, hey, can we meet reality with this feature? And they were like, no, no, it's going to be ready in two weeks. I'm like, well, what if you learn something today by talking to a customer that changes what you want to do in the next two weeks? So that's the principle. It reminds me of the Joel sloppy test. You know Joel on Softer? Maybe you're too young. What is the test?

17:57 It's 12 questions about working. with software? Do your candidates write code during interviews? Do you have a bug database? Do you have planning? Are you fixing bugs before writing new code? And the last one is, do you do all the way testing? So do you test with some random users? And it's Joel's software. It's my god in software engineering. He was an architect at Excel at the time. And he was saying there's 12 questions. And if you don't say yes to at least 10 of those, you have serious problems. So it's still relevant. To test, to meet reality, it reminds me that I reread this one, Joel on software, sloppy test. It's a sloppy test, 12 questions, and it might give some ideas. But it brings back to the users, and it's interesting to see prototypes helps to go back to the users.

18:59 Do you see change in the user needs? Because AI is changing our way to work, but probably the way users... Use the product. For me, it's frightening, but how it is for Notion point of view about the usage. So one fascinating thing that I've observed in how our users use our AI products is that we used to have much more of a blank canvas problem where you show a user a text box. They You have no idea what to type into it, but you kind of have to really teach them over time. As ChatGPT and a bunch of all of the AI products started to become more popular. New type of users. Yeah, it's a new expectation from users. And we saw the examples people were trying to do move from search queries into action-taking queries. And at the beginning, we couldn't support all the things that users were asking us to do. But now we can basically support all of them.

19:59 But just... the expertise level of the users and their like the audacity of what they would ask an AI to do has definitely changed over time. Changing and maybe continuing to change. How do you discover those needs? A couple of different sources. Talking to customers is by far the easiest way to do it. We try and get as much feedback as we can, but we also see what we look at our logs. Customers will give us feedback, you know, thumbs up, thumbs down on how a response was. We'll look at the thumbs down. That helps us understand where the gaps are. And then we try and use it ourselves. So what is it that I tried to do yesterday that didn't work? What capability are we missing to get there? And iterate from there as well. So all sources. Great. I'm really impatient to see which features we won't have for the summit in six months, because I guess you have a plenty of list of tests you'd like to try and they will all fail. Maybe one will survive. I don't know. It was really a pleasure to have you on board.

21:02 Do you have some questions about the French tech community of leaders? Yeah, I'm curious. What is it that keeps you all up at night? We're in this like AI echo chamber here. Is it the same over there? The same FOMO experience? FOMO is probably, I don't know, maybe we should have a new word for that. It's not even FOMO, it's forced of missing out. Every new model getting out and... Currently, the budget of consuming AI for coding, are we spending enough money? That was two months ago. I guess it's quite aligned. It goes so fast now that the same problems are getting. It was maybe two months later, a year ago, but now it's two days later. We have the same problems. I think the main difference might be around sovereignty, the data, the protection, the technologies, trying to have European technologies and things like that. So it's a tricky topic to discuss with a US company.

22:05 So you'll be welcome. There might be some questions about where the data is. How do you ensure the data is stored in Europe and things like that? Those questions are getting a bit more political currently, and it will be a bit of a discussion in the summit about politics. We're out of tech and we'll have some topics. We can't be apolitical anymore. Being silent is political, is becoming political. So I don't know if it's a topic in the US, but here in France, it's getting a bit of a pressure to say you're making a choice anyhow. So think about it. It's not neutral. Neutrality was comfortable, but now we are losing a bit about that. Well, I mean, we have our customers globally, and so we're very familiar with data residency needs and data protection needs and trying to the best we can. And just give customers the tool to choose their own adventure, put the data where they feel most safe.

23:06 Thanks a lot. It was a pleasure to have you on board for 30 minutes. Yes, thank you so much for having me. See you soon, maybe in Paris, if you want to come to the summit. And at least Notion will have a booth and we'll receive plenty of questions. And I hope we'll have some new features to try out. Perfect. Coming soon. Thanks a lot. Bye-bye.