[ { "start": 51.0, "text": "Tobi: Welcome to the Alphalist podcast. I'm your host Tobi. And today I'm super proud and honored to welcome Peter Gostev. And Peter Gostev is the Head of AI at Moonpig. Um, and the ones you don't who don't know Moonpig, it's a company that sells gift cards, right? Or cards, like greeting cards.", "end": 73.0 }, { "start": 73.0, "text": "Peter: Greeting cards and gifts. Yeah.", "end": 74.0 }, { "start": 74.0, "text": "Tobi: It is worth like over 300 million pounds as far as I know. So it's a British company. Is that", "end": 79.0 }, { "start": 79.0, "text": "Peter: Yeah, yeah, it's it's it's more now. Yeah. It's a it's about", "end": 83.0 }, { "start": 83.0, "text": "Tobi: It's even more now.", "end": 83.0 }, { "start": 83.0, "text": "Peter: Yeah. No, we're we're doing quite well. Yeah.", "end": 87.0 }, { "start": 87.0, "text": "Tobi: You're doing quite well. And and you have a bit of an unusual role, which is like new in many companies and many companies are thinking about like centralizing AI and yeah, I I really love talking about it. I mean, I I always have to be careful like not to mix in too much AI because uh like many people also tend to be be annoyed by the AI hype right now, but I think um uh your role is quite uncommon still, um and and uh like um I'm also um at Task Goup, my my my company just uh I've just hired a head of AI, um or I've just hired that sounds so so so top down. No, we just hired a head of AI and I'm I'm super pumped uh to explore this. Um and it's really I can imagine like a super explorative job. Beforehand, you already did AI strategy uh at Nat West Group, um and like beforehand like diverse strategy and analytics gigs, um and um you studied at at Harvard Business School, right? Um and yeah, you're you're you're a regular guest on many AI podcasts. Um and you are you have a big reach on LinkedIn as well, like 60k followers. So that's super exciting and you you're posting a lot of uh like hype firing things, but also like sometimes you you clarify things that what I really really like and enjoy. So thanks for being my guest here.", "end": 179.0 }, { "start": 179.0, "text": "Peter: Cool. Thanks for inviting me. Excited to be here.", "end": 182.0 }, { "start": 182.0, "text": "Tobi: I'd love to talk about your perspective on AI in general, um and where we are heading. Um but also like how to really drive to really change cultural um aspects in in modern companies. Um and also as a learning for CTOs explicitly, how you feel about like how how how to influence the process and where we should all head. Um that would be like really as a takeaway value from you really super important. Um but maybe before we start with that, like let's let's go a bit deeper on on your um nerd journey. Like basically we start with the first memory of of a computer that really hooked you.", "end": 229.0 }, { "start": 229.0, "text": "Peter: So, I my my first computer at home uh was Pentium 2, I remember. And the good thing about that was that our parents, so it was me and my brother playing with it and our parents had no idea what was going on. And I feel like that was golden age of for for me and that was like a sweet spot that it was not too complicated, so it it was not for really, really technical people, so you could still make it work. And if you want to play some computer games or if you've got some dodgy CD uh from back of a market that you want to try and get going, it doesn't work. That was a really good learning to to just a good motivation uh to start learning how how computers worked. Um yeah, so and I do wonder if if younger people now, if they start with an iPhone and a Mac and it works super well and you just I don't know, swipe and click buttons, whether you're actually not motivated to to learn like a bit more depth in terms of how computers work. Uh so yeah, I feel like that at least gave me some good grounding um in that motivation to play computer games when you're 10 years old, I think is a good grounding for for anyone starting in in technology.", "end": 306.0 }, { "start": 306.0, "text": "Tobi: So your gateway drug clearly was video games. Um but what was like, was there any moment when you like shifted towards like, this is what I want to work with, this is what uh like how I can influence the the the machine doing more than just, I don't know, shooting or whatever in games, like", "end": 328.0 }, { "start": 328.0, "text": "Peter: Yeah, I also there was never one specific point and I remember when I was working on early projects in consulting or in banking, you can kind of see that pretty much every single exciting project that I ever did was something to do with data and machine learning and it kind of went up the the that intelligence curve where started off with, I don't know, maybe basic analytics and then when I was at that West in the data science team there, we were doing some really interesting cool stuff. Um and um towards towards my uh time the end of my time there, it was generative AI wave was coming up and that was kind of pushed me towards here. But I would say over time, it just it really appealed to me that idea of how much we can do with data, uh with machine learning, and now with generative AI. Um yeah, and I think there's never really been more exciting time in my career versus now and every time we can we can just do more and more and more, uh which is just such a such a cool position to be in.", "end": 399.0 }, { "start": 399.0, "text": "Tobi: So now is always the most exciting time, I guess then, right? Like at least if you're in AI, then like it's it's really Yeah. Now and here and now, right? Um Yeah. And what's what's your perspective on like let's say the hype cycle and where we stand and uh and how it will look like in in a year or so?", "end": 421.0 }, { "start": 421.0, "text": "Peter: Yeah, it's a very like whenever I think about where we are with with hype, it's a very kind of mixed picture. There are some things that really frustrate me where people just hype things up that uh are maybe like useful but not actually like that important in particular. And or we are way too early or um there's a lot of complexity that they still need to get through. But on the other hand, I also feel like people really underestimating where we are and or how much more there is to go. And what what gives me really reason to say this is that if we look at the amount of investments that are going in in terms of GPUs, um and all all of these big clusters, we are really just at the beginning of these investments starting to come through. Um all of the most important models so far have really been trained kind of on old technologies. So for example, H100 GPUs were really the workhorse of all of the current models. Um so like the biggest uh cluster is uh Groc has been trained on. This is just H100 GPUs or three model H100 GPUs. This is a model that came uh this is the GPU that came out before Chat JPT. So it was I think it was something like summer 2022. So clearly they haven't even like they couldn't have possibly designed it specifically for LLMs in mind. So um and the amount of compute that we're getting soon, um in the next year, even even within this year and beyond that is just kind of mind-boggling. It's just going to multiply several times. And this is not this is not speculation in the sense that, oh, I don't know, maybe things will happen. No, they have paid money for this. This is already like Nvidia recorded the revenue is and these data centers are being built. Like this is happening. And it's hard to predict like maybe things will flop and you know, GBT 5 is nothing burger and like I don't know, Gc 4, which is should come out soon is also not interesting. Like it's possible, but it's also, I think quite likely that we'll just even just because of the compute, the amount of compute we're going to get, we we actually underestimating how much more there is to come. Um so that kind of gives me pause and in this kind of this duality of of the hype cycle. I think some people over hype some maybe not important stuff, but maybe also don't realize how much more there is still to come.", "end": 576.0 }, { "start": 576.0, "text": "Tobi: Do you have an example for like, um stuff that is not as important but people over hype it?", "end": 581.0 }, { "start": 581.0, "text": "Peter: Yeah, I think uh maybe maybe it is important, but for example, the whole agentic wave right now, um I think people really underestimate how difficult it would be to uh to how how difficult it would be to build a fully agentic system. Um right now, we're looking at some some projects internally for us which which we're still working on and we're looking at what would it take for us to build a more broader agentic system that will do whole spectrum of tasks and essentially we would end up uh probably quadrupling or more the size of the project just because of the number of integrations we need to build, the number of controls we need to build, all of the different um blockers that we'll have um and and so on. So, I think whenever you see some big claims about agents going to do everything and we're going to automate all of the work, once you actually start doing it and planning the project, you just realize I I could build it but it will take me a year uh or two years and probably it will be completely out of date in terms of the technology that we're using by that time.", "end": 655.0 }, { "start": 655.0, "text": "Tobi: and sometimes it's also not worth it because what you're getting from the agent really is like, I don't know how often you have to execute it, like depends on the the the amount of executions and the time you spend doing it manually, um like it it yeah, you you better sometimes end up not automating it because automating like maintaining automation and agents, like even if it's just like a few prompts that you basically have to script out ultimately and connect, um like like maintaining this is also some work, right?", "end": 694.0 }, { "start": 694.0, "text": "Peter: Yeah, yeah, and I think um the definitions of agents really vary a lot by different people. So for for me, we've built a bunch of workflows that we have automated where you kind of stitch prompts together and have various data floats. I would not define this as an agent because it's quite precisely defined in terms of what what happens uh when. Uh so for me it's just kind of automated workflow. Uh which definitely, I think that's a good use case. Yeah, for you you're right in some cases it's still not worth it because yeah, if you do it once a week, it doesn't really matter. But for some processes that would be quite quite valuable and we've definitely had some processes inside Mobeck where that we've automated completely so that humans don't touch them. Um but they're not that many of them that we could just pick up and automate. Um yeah, and then once you go in that more fully agentic sphere where you really give much more control and and give the agent ability to make judgments and all of that, then uh while the potential the number of potential use cases really broadens, your ability to actually build it well really goes down and the reliability is not as high and the the amount of software and integrations and all of that that you need to build is actually really high. So that that's the complexity and I think there will be a tipping point when actually maybe it's the connections are there, maybe the models are way, way smarter, so you don't need to babysit them at every single step, then it might just work. And then then maybe all of the hype that we're seeing now with agents and becomes real, but um as of today, I think it's really hard to build good good full agents. And I think Andre Carpati I I I watched a talk by him recently and he he said um he he had a similar point of view having worked in uh Thomas driving and how long that took to get from that kind of working demos to actual deployment. That took over a decade. Um and yeah, he was saying we should see it as a decade of agents, which I think is yeah, much closer to reality.", "end": 831.0 }, { "start": 831.0, "text": "Tobi: Yeah, I tend to like for me it like is is like constantly up up and down like how uh the my my perception uh of of of how much I can or how much, for example, software can be replaced uh or like rewritten or like auto written by agents. Like it it constantly changes. Um and I like I I like if I do something bigger, then at a certain point, I hit like a disappointing moment and stop doing it. I also don't have to do it in my daily work, but recently I, for example, started using uh Claude, um or like the command line interface, Code. And I must say that that is really it is really it really moved far from my perspective. And it it really is at a at a very good point where I can, I don't know, I I I think like teaching your juniors how to use it, teaching like everyone how to use it, um it kind of makes sense from my perspective, even though you won't use it every day and you don't like you maybe also hit a wall at a certain point. What's your perspective on that? Like, uh explicitly with that with that uh software engineering perspective.", "end": 909.0 }, { "start": 909.0, "text": "Peter: Yeah, that that's something that is is actually really tricky because the way um and I I agree with your observation as well. I have a similar experience where when I when I try things and there's sometimes they work, then I get tired of it and maybe don't use it for a bit. Um and the tricky thing here is that it's a skill in itself to use these tools and what you need to pay attention to, what your workflow is is quite different to regular software engineering. Um so what is tricky is to get an organization, we've got, I think something like 150 software engineers. It's really hard to basically tell them, actually stop doing what you're doing, how you're doing it, and actually do it this way. Because I can't really even describe what this way means necessarily, because people work on front end, back end, and you know, with different languages, with different circumstances. So for me to really help them do that well, I have to be super engineer myself and you know, I I'm not and I doubt many people like can can be in a position to really explain to everyone. So I think this is a big blocker that because it requires the the change of flow and workflow of how people actually build and how they interact with these systems, it makes a lot a lot harder for um for people to adopt these tools. And like you said with cloud code, like it is very good, but it's also not really how people build software, right? I think it's quite uncommon for people to be like, yeah, let let's just go into terminal and just going to build like that.", "end": 1011.0 }, { "start": 1011.0, "text": "Tobi: And type in something.", "end": 1012.0 }, { "start": 1012.0, "text": "Peter: Yeah, so that's really uncommon. So, and then the challenge is that I can change the strategy of the organization on how we build software and say, actually, everyone should be using cloud code and that's how you do it, and I write a perfect guide and we train everyone, then I don't know, Corsa releases something else or open I release something else.", "end": 1031.0 }, { "start": 1031.0, "text": "Tobi: Yeah, in a week, right? I mean, I I was also excited by Corsa like four weeks ago and", "end": 1035.0 }, { "start": 1035.0, "text": "Peter: Yeah. So this is really tricky and I think this is a problem that we there's not any foolproof way how we just adopt this technology and even yeah, what tools do we use, what do we teach people, um and I think right now there's just a massive gulf between those who really take time to adopt the technology and those who don't. And those who do, they they develop the intuition for it, what what's important, all those little tricks, different workflows, um you change your setup and it could be the way you write code and the way you document things and the way you just I don't know, set up the file structure, the way I don't know, even silly things like the way you name files. Right? It might be obvious to you, but if the AI system cannot find the file because it's some generic name or like overlaps with many other files, like it will just be much worse. Um so even even trivial examples like that, um people who really take time to adopt these tools are way, way more productive and and better at that. But it's not something I can just tell engineers or could be in other professions as well, just like can you just do that? Because it's almost like a mindset and and a skill and and something you just have to keep working at and it changes all the time. So it's even if you learn it, you still need to relearn it probably a few months later.", "end": 1120.0 }, { "start": 1120.0, "text": "Tobi: It reminds me a bit like I'm I'm I don't know, uh developer at heart and every once in a while I I try to use a certain technology and it reminds me a bit of the the times when like all the new front end technologies came out. Um and there were like many people really um or like a new tool every day that like people adopted and people were like basically shouting out for and often like influenced by big tech. Um hey, this is the new framework. This is like uh I don't know, you now have to reuse react or you you now have to use use Angular two or whatever. And then like then like almost like every few weeks there was like a new build tool and a new way to manage dependencies. And um with AI it's a bit similar like uh like every time I open up my computer I could try like 10 different new new things. Um and also I can like build a lot of stuff in a in a no time. Um how do you think or is there what's what's from your perspective if I want to change people's behavior, um then I what's what's what's a good or like I don't know if I want to change people's behavior, but how how do you get in between? Like I think the the idea of cloud code, for example, is is a good one because many developers are on the command line and really changing your real work behavior like adopting a different editor, stuff like that, um like with VS code or um Cursor uh as like a replacement then for VS code. It's a it's a big step, right? But um just using the command line tool, to me, it felt super natural. Like I I had a moment like recently, like a I don't know, a few months back when I thought like, hey, I use Z H like a like a shell for for MacOS. And I wanted to have like something which is really AI powered and like close to Z H and um and that's in a way it's it's cloth, right? Um and and that like is it maybe like a niche where like AI stepped in that that is natural for a few developers. But but yeah, how does that look like? How does that flow?", "end": 1251.0 }, { "start": 1251.0, "text": "Peter: Yeah, it's interesting. I think you you might be right that maybe there is like a natural ergonomic like product market fit because it seems to be really taking off. And I know um there were some reported numbers for Anthropic revenue that they really that they doubled the annual um recurring revenue in like six months or something like that or quadrupled it some kind of crazy number. And um that could be because of cloud code and and just so for coming out. Um and and I think it's actually also important combination that um I I haven't used it too much myself, but the Gemini um also released the the CLI tool and some of the tasks that I've seen were pretty terrible. Um and part of it was that it's probably just the model is not quite well aligned to to it's not agentic enough to do to do well enough. So it looks like the the model is playing a big part of it. But but yeah, in terms of your your point of adoption, um I don't have a a magic answer there and um I I hope that showing people multiple ways how they can get value and maybe having some aha moments and just maximizing that could work. Um but it also could also be naive that maybe they'll see it and it looks like a cool demo and they just go back to what they were doing. So I don't really know. So far what I've seen is that people seem to just have mindset and personality type almost regardless of their job, uh job title, um where they just pick these things up much better and they just use them better. And there is this kind of point that maybe it will go away, maybe things will just kind of get magically better and and things will just work. But at this point in time, you do need to have intuition for how these models work. And um what what are the subtle things that you need to do? Um for example, one thing that just I'm so used to it, so I even forget to mention this to people that you need to, for example, if you're using chat GPT, you need to start a new chat if you're trying to solve like a different problem. But the number of times that I I get tripped up when I speak to people and they just have like they have different conversations in the same chat. And I'm like, why would you do that? Like clearly is the wrong way to do that. But it's but to be honest, it's also not unreasonable, right? If if you're going to speak, I don't know, to your friend, you're not like saying, okay, we're like stopping this conversation, let's have this conversation.", "end": 1420.0 }, { "start": 1420.0, "text": "Tobi: let's move to another one.", "end": 1421.0 }, { "start": 1421.0, "text": "Peter: yeah, it's change context.", "end": 1422.0 }, { "start": 1422.0, "text": "Tobi: Yeah,", "end": 1422.0 }, { "start": 1422.0, "text": "Peter: Yeah, but I think just context management is just so unnatural for people. So um and it's one small example, but it just changes it completely. And I think for developers is similar. If um if they don't realize that they actually should start new chats and they should be careful like what's in the context, they just they have completely different experience of of using it. Um yeah, that's just one small example, but there's just so many subtle things that you have to learn um to to use these tools well that yeah, it's just I also feel for people because it's not people's job to to to learn this necessarily. I mean, they they probably should for their own good to to invest time in this, but it's also kind of an unreasonable thing. Could you just like spend hours and hours learning about things.", "end": 1473.0 }, { "start": 1473.0, "text": "Tobi: And you need an extra day, right? You need an extra day to to to do that and and then there are so many um let's say blind or or dead roads, dead end roads here, um where you discover after a while, okay, no, I use this SAS tool now to do XY Z um and I just realized it doesn't really help me. And just figuring that out, um is is is like a huge time investment and also money investment, right? Like you spend money on almost every tool you you buy or to start using and then you you stop using. So I think um AI heavy companies in the SAS world, um people that provide tools for uh AI gold diggers, they see like huge churn, right? Like they see like it's it's far different different pattern and behavior. Like people sign up, um use it for a while. After three months, discover that they're still paying $20 a month for this or that and then they stop paying or they they realize, oh, I'm not using this or that anymore and they they stop paying. So really getting getting to a like high adoption is super hard, especially because there's no silver bullet. How do you manage that in your day-to-day? Like is that your responsibility also push like people to adopt AI in the company or is it mostly like around like building AI related products, uh generating cards using AI or like uh is that is that your your thing as well?", "end": 1567.0 }, { "start": 1567.0, "text": "Peter: Yeah, so for for me, I split my my work uh within Mobe in three different categories. One is definitely tool adoption. So just making better use of what we've got already. So it could be things just like use JPT better and I don't know, how we create custom JPTs or how we create this workflow. Um that that's definitely a part of it, but also we've got a lot of creatives within within our business. So extending researching and understanding what's going on with image generation or video generation, that's that's also big big part of it. We've got big opportunities there. Um then the second part are pretty sure things such as for example, automating a workflow that we're pretty sure we know how to do, but just we we need to actually do the work and do the engineering to to do that. Um so that's the second part. And then the third one is more experimental, um maybe harder things that that quite often closer to the customer facing features. And how we how we build them out. Um so the idea here is that I want to have a portfolio of different things that I I'm covering, partially because just to make sure that we've got value delivered at all points. So there's not any like there's no value for six months and let's just wait and until something happens. So we we don't want that. Um but part of it, we also don't want to be just let let's just adopt some tools and automate things here and there. We want to also have bigger projects that hopefully if they deliver, then they become quite big and impactful for the organization.", "end": 1671.0 }, { "start": 1671.0, "text": "Tobi: And are you like a lone wolf in the organization or do you have a like an AI team around you? Like AI engineers that help on the product side or how does that work?", "end": 1681.0 }, { "start": 1681.0, "text": "Peter: Yeah, so the way we've set it up is that we've got two separate teams that could quite often in different organizations are combined. So we've got a data science team that is separate, they're still train machine learning models and there's quite a lot of work there to do on like recommendations and search and things like that. Um and then there's the second part which is my team, uh which is the more applied AI engineering and I've got several engineers there and quite often their background would typically be software engineers. Um who and that's also the difference with the data science team who are typically data science, machine learning people and then um they they would kind of extend their capabilities for and for for my team, it's typically software engineers who learn how to do AI. Um and the reason for that is that quite often on any given project, most of the work is actually non-AI work. When we want to implement a new customer facing feature or automate some workflow or I don't know, create some kind of agent or something like that, actually most of it is integrations, scaffolding, deployment, um UI, all of the things like that. And then the the AI part of it is normally maybe 10, 20% of of the actual work. Um but doesn't mean to say it's easy and kind of could could be ignored. And we quite often actually start our projects by doing prototypes that we make sure that the AI part actually works um before we embark on any other uh harder software engineering tasks. And then once we're happy and we've got stakeholders who are interested and engaged, then we can move on and and build the real thing.", "end": 1793.0 }, { "start": 1793.0, "text": "Tobi: Mm, mm. And I you like a lone wolf in the organization or do you have a like an AI team around you? Like AI engineers that help on the product side or how does that work? How does that flow?", "end": 1810.0 }, { "start": 1810.0, "text": "Peter: It's actually a mixture. So we've got the definitely, we've always got projects that we're delivering ourselves directly. There's plenty of those that just and the reason for that is sometimes there's not the capacity in that particular organization to to build uh what we want to build. Um or uh there's quite often it's not just the capacity but also like the right capacity. So maybe like maybe there is one person who could could build it but actually we we um we couldn't get their time so it's much better if if we build it and we kind of have to learn the particular domain. Um or there there are other cases when there's just no engineering at all. And actually, uh in those areas. So for example, most organizations would have no real engineering in something like customer service, although that that's maybe more mixed, but finance, HR, um commercial teams, maybe even marketing teams, they pretty much would not have any engineering support. But actually there are areas where these models can do really quite impactful work. So actually us partnering with them is quite often the only way to do it. There are other cases when we've got teams uh where we would mock something up together like an early prototype and then they're quite happy to get on with it and and build it themselves. Um so that's that I really like these kinds of projects because they're quite high leverage uh for for me that it could be a few days of work or maybe a few weeks of kind of calendar time, let's say, over over a period of time where we work on some prototype and then the team goes off and does it themselves, that that's great because that's something they want to do. It's not like us kind of forcing them to do some stuff. That that's actually a real problem.", "end": 1926.0 }, { "start": 1926.0, "text": "Tobi: And that's what they will continue doing, right? I think that's the most important thing. Like, I mean, like teaching people like how to code with Claude or whatever you want to do it, right? Um is is one thing but then the the constant adoption and um really like ideally also being able to get feedback every once in a while how how adoption goes is is important. How how do you manage that? Like do you have AI ambassadors in each part of the organization as well that like then like tell you every once in a while how things are going or is it just like you and your team and then like the events that you organize and then the the calendar time that you spend with people, like how how does that look like?", "end": 1971.0 }, { "start": 1971.0, "text": "Peter: Yeah, so we we're small enough that we I think maybe we could have more formal structure with like ambassadors and and things like that. For now, we still manage it through, yeah, we've got direct collaboration with several teams. Um and then on we we've got direct collaboration with several teams. Um and then on on which is more kind of a portfolio of work that that we that we uh have and and progress on it directly. But also, if there are some some extra projects we want to pick up on, yeah, then then great, we can work on us together. I always try and say yes to people if we even if we can spend a bit of time to to make some progress. And quite often, I would say, especially the early POC part, quite often there there is quite a lot of impact in just even doing the simplest, easiest thing. So we we had one example where uh we had a use case um where we just needed to like take in some data and and process it and put it in another format. Um and the originally the engineering team planned to do quite complicated projects uh where they would have like various structured rules about how they would process data and that would have just taken weeks of work. And actually I just did a version where I've made two LLM calls to just process the data and and output it in a structured format. And that took me like a one evening of work. And then next day I had a I had a version of it ready, just a working prototype that they could test. And they just realized, actually, you know, that is far simpler and let us just do that. And then it ended up being not the whole engineering team for like three weeks uh for kind of semi-good solution and it was one engineer for like a few days and yeah, they they just did that. So I think there's quite a lot of leverage in just showing the the art of the possible. And quite often, and also maybe it's worth dwelling on what I mean by prototypes. Sometimes we will build a piece of software and it could be just me kind of vibe coding a piece of software just to to kind of show how it could work together. But it could also be uh just a prompt in chat GPT where we would just have a I don't know, custom GPT or just a prompt or could be prompt on the playground which would just show how that could be done. And quite often that's actually enough. We don't need to build some complex agents or anything like that and that that will be sufficient. Um and that could make a lot of difference to the team just realizing, actually, yeah, our road map was actually not the best and let's just do it this way and it's much simpler and and we'll have more impact.", "end": 2145.0 }, { "start": 2145.0, "text": "Tobi: Yeah, I mean the the the the sole power of a custom GPT is already very impressive, right? Like just realizing that like how does it like why do you do it? Like why would you build it? For what purpose? And then like constantly using it and enriching it and then like creating like a few automations that maybe are fired by that, that's for me like an enormous um time saver in so many situations. And it is in a way, it is an agent, but yeah, very very simple one, right?", "end": 2179.0 }, { "start": 2179.0, "text": "Peter: Yeah, yeah, and it's I would say the magic there is when you can get a subject matter expert who and it could be, I mean it sounds dramatic, but it might literally be like I know how I need to write my, I don't know, a Slack message which is like a weekly update on a project. Like that could could be the subject matter expert. It's just a person who really knows what they need to do that is really, really hard for someone outside of that who doesn't run this process to come in and tell tell them, oh, this is how you should do this Slack update or something. But if someone knows how to do that and they could just work out how to use, yeah, custom GPT or whatever they're using, then they could just unlock a lot of value. Um and yeah, that's I think there's a lot of power in in getting these kinds of people really engaged who are not necessarily technical people at all. Um and who who just know what they need to do and just showing them the way a little bit of that actually it is possible. You can just throw in, I don't know, PDFs of your presentations and spreadsheets or whatever it might be, just into all three and get it to output in a consistent format. Like you could just do that and that that could be a big unlock. And I think coming back to the agent's point, I think not not that it's mutually exclusive, but I think sometimes when we talk about, yeah, it's agents going to do everything. Yeah, but we we haven't done even that bit of of actually just just helping people to use it to their degree.", "end": 2282.0 }, { "start": 2282.0, "text": "Tobi: Yeah, exactly. And then not not again, it's not mutually exclusive. You can kind of skip that bit to build those agents, but actually you still need that expertise. Like, for example, if we were to build an agent which would, I don't know, collect all the data and to to do the slack update, you still need the expertise to say what it's meant to be, like what's important in the slack update. So it's not just a bunch of AI slope that is like vaguely looking correct. Um that's like that that uh that's not the right way to do it. So, yeah, there's there's yeah, there's definitely a lot of opportunity in that direction.", "end": 2319.0 }, { "start": 2319.0, "text": "Tobi: so to have a very narrow use case and then uh tell an LLM to do this or that with the data you throw in, right? Like, I don't know, rate all the CVs that I throw in for me, right? Uh like this is the criteria that is important for me. Works quite well. Uh like stuff like that, like the little use case and I I can imagine that um this is also the reason why like I I think many consultancies are having problems with really like finding big gigs in AI because it's gradual, like many things are just like gradual improvements that uh basically save you a bit of time each day and make you more efficient, while not making you a superhuman straight away. Um or not, I don't know, replacing a workforce of 50 people straight away, um and uh like just to to come to the extreme McKinsey cases, let's say, uh and and and this makes it makes it hard to to to to to earn money with it in a way.", "end": 2378.0 }, { "start": 2378.0, "text": "Peter: Yeah, it's interesting. Yeah, and I don't want to underplay the the opportunities that that are in the kind of in the bigger, more automated bucket. I think that definitely there. Um so I think we could with enough effort probably build a lot more software that's LLM powered that that could do that. Um but it's hard. Like that in terms of my portfolio, I could maybe do a couple of projects like these a year that we just and it's not necessarily that, okay, if I had 10 more engineers, I would do, I don't know, five of them. It's also like head space and testing time and just making sure that we get it right time. Uh so it's not necessarily just scalable in that kind of unlimited way. But if that these kinds of use cases also keep keeping them in mind, there's definitely a massive amount of value that's just still left on the table that we still don't like people don't really use those tools well enough. Um and that's why I'm also I had this initial skepticism about tools like uh co-pilot like tools. And I I couldn't quite articulate why exactly, but the reason I I'm personally skeptical that I think I kind of crystallized that is that point of that for certain things to work, if they're not very superficial, such as superficial example could be like, oh, just help me write this email better, which is like, okay, but whatever, who who cares? Like it it's not only adds arguably doesn't doesn't actually help. Um but if I want to say, I need to process these invoices and I need to output them in this way and and I don't know, help me do this process. That is not a generic co-pilot question. You actually need to embed quite a lot of expertise that is very, very hyper local to your use case because I don't know, someone else set up a spreadsheet in this way. So you actually have to craft your context, you have to craft your prompt to make sure that it does does actually work. And that's why I think whenever I haven't used the Microsoft tools that much, but kind of judging by rumors around adoption is probably true that people just don't really get enough value. And for me, that's probably the reason where it's you you don't sit down and kind of craft precise context, you don't craft precise prompts, you don't embed your expertise in there to say, okay, this is how the the flow should work. But if it's just generic, help me write the email better, it's like next to useless, um that that would be my my assessment.", "end": 2550.0 }, { "start": 2550.0, "text": "Tobi: yeah, you also recently wrote a post on uh like uh written content from like uh generating written content with with with LLMs which was more not not so um you were you weren't pushing people to to do so, right?", "end": 2565.0 }, { "start": 2565.0, "text": "Peter: Yeah, yeah, my my basic point there was that I think people slide into that idea that because it looks good, um because maybe, I don't know, it just sounds coherent that it's it's better to just write with LLMs. But for me, it's not it's not the best use of of AI in that in that case. So quite often in my work, what what I would do is, if I need to write something and you know, I I write recreationally and also inside inside the organization. Um if I need to write something that I don't know maybe enough about, I would use LLMs to to research like crazy. Like I would if sometimes I was I was doing some work and um collaborating a bit of a paper and the amount of all three and uh all three Pro I was using is insane. I had like three three, four, five, 10 windows in parallel doing different bits of research, double checking, running same queries multiple times to just make sure I got the data correct, all of that. And then when it came to synthesizing that data and actually putting it together into something to share with people, I would still write it myself. Not not because I don't know, LLMs are better at writing or something, but that's the point here is that that is my thinking that I need to put down on paper. Like if if people are interested in um LLMs saying kind of providing their answer, they've also got chat GPT. They could also do that. So the the if the point is that they're asking you, then you should answer. And um and the reason why you still want to ask a human is that, you know, there's a throat to choke, you know, if someone if you're saying some rubbish, then you can the person can come back to you and say, why are you telling me this rubbish? Like that was wrong. So I think that there are things like that. There's the points around that the human would have way more context that goes way beyond any chat conversations. They would know some past experiences and strategy and wider wider information. So it's not necessarily that the person is smarter, but they would just be have more awareness of what else is going on. And yeah, when when people just go to LLM and say just write me this and it's just there's it's just kind of what's the point? You're just wasting everyone's time. Yeah, and I think my and this like um there there obviously many scenarios when it's like a bit more reasonable to use it. Like for example, I use it sometimes to to write more directly if it's much more utilitarian kind of writing where I'm just got I've got this, I don't know, reports and I just need to output a table with this information. Like that that's fine. You know, I've got no problem with that. It's just like regurgitating information which is useless sometimes. But yeah, if you need to express your opinion, if you need to say what's in your head and put it down on paper, I think it's really a bad idea to use LLMs and it's um it's kind of going exactly the point of like why why write at all.", "end": 2765.0 }, { "start": 2765.0, "text": "Tobi: So um yeah, I also had some recent moments when like someone sent me like a huge doc with I don't know, IT compliance rules or whatever where I thought like, hey, can you give me the prompt please? Exactly. Like to see the prompt. Give me give me the prompt. Let let's rather version the prompt instead of this document because this will change over time but uh I'd like to see the prompt for it because I want to see what what your real thoughts are and what your real um what the real workflow should be. Um in your uh like day-to-day like or in your in in your setup at Moonpig like do do you have anything that's like really everyone is using like um I don't know, do you give everyone chat GPT or or Claude or whatever as like a a gift? Do do people have to pitch for it? Like can everyone use whatever they want? Um and do you have some underlying tools like I don't know, N8N or something like that that like really everyone can use and also people enjoy using that you would recommend to every organization out there to be to be adopted?", "end": 2828.0 }, { "start": 2828.0, "text": "Peter: Yeah, the the hardest part is the adoption. Um so and and that's why I'm a little bit hesitant just giving access to a bunch of different tools because we we can definitely do it. It's not a problem. It's not even a cost problem, but then we when we end up with tool here, tool there, it's really hard to make sure that people understand how to use it and and so on. So and it's kind of coming back to to our conversation earlier about yeah, people actually need to know and get a feel for it and and so on. So the tools that we're using for general population is chat GPT. And the way we've approached it is was was I think kind of interesting that we basically did no marketing internally and no like forcing people to use it and no forcing people to adopt it. And the reason for that is that I I kind of wanted to avoid a scenario where I just tell people to just like go use it and beating them over the head and saying why you're not using chat GPT and so on. So we just kind of left it open. If you want to have a license, you'll get it. We're not going to ask you like many questions. Um and we basically waited for people to come to us and um so we've got something like in in kind of head office, I think we've got about 600 people, something like that, maybe maybe 700 people. And um we had after the first slack message, which is something like, hey, we've got chat GPT, do you want access? We probably had about like 130 sign ups like within a week, which I I think is pretty good going uh with like no no extra follow ups. Um then now we are close to I think about 400 again, I not a single time I send like a message to to I don't know, everyone saying, why haven't you signed up yet or anything like that. So it just happens naturally and I think people like, oh, I actually need to need it to to to do my job. Um yeah, so I I think that's I'm I'm kind of semi happy with how that's gone. I think there's still a lot left on the table in terms of the use of it. Um and it it is difficult because they're definitely like when you run these sessions, engagement sessions and how to sessions, you kind of get the same 20% of people show up to these things who are clearly more engaged and I definitely want them to be engaged. That's great. But yeah, it's a bit harder when you go a bit further those people who signed up like a year after we've got the license, you know, like are they like are they using it properly? I don't know. It's it's hard hard to kind of crack that and and know that for sure.", "end": 2998.0 }, { "start": 2998.0, "text": "Tobi: Maybe you should run surveys or something.", "end": 3000.0 }, { "start": 3000.0, "text": "Peter: Yeah, no, we've tried and then, you know, people tell you things. It's I you know, I I don't know how you find it. Whenever I see these kinds of surveys, I don't know, by consultancies or or uh or or things like that where you kind of get these results or 40% of executives things AI is a good idea or something, like something like that or I don't know, 80% of companies use AI or something like that. It's like this there's something really fundamental is missing there that I could if I wanted to write a like a Moonpig promotional marketing memo, we could say, um yeah, 80% of the organization is using AI, you know, weekly basis and you know, we've got these seven use cases which are automation workflows and actually we've built a bunch of agents like all of that. But still in terms of the potential use, if I if I was to sit down and go through every single process in the organization and or even imagine all the things that we could do, I don't know, we're probably, I don't know, to guess 5, 10% in in terms of like what we could be doing in terms of just the the processes. Just the number of I think anytime I speak to a team, any team and it could be from, I don't know, legal to product to I don't know, marketing, like we come up with like five, seven things that they could just do tomorrow that they're not doing yet. Um there's just so much. Um so yeah, maybe maybe 10% is even generous. Um so so yeah, that's why it's yeah, we've got I think decent adoption. Um we've I think we've done a a good enough, well, not good enough, I think it's okay job, but yeah, if um yeah, if if I had the magic wand, yeah, we'd like we just sit everyone down, look at their processes for like a couple of weeks, stop everything else and just like learn to do this properly and we'll be probably like 10x the usage straight away.", "end": 3722.0 }, { "start": 3722.0, "text": "Tobi: maybe you should build like a boot camp or something. Um and", "end": 3726.0 }, { "start": 3726.0, "text": "Peter: Yeah, I don't know. I'm open to ideas. You know, I'm definitely not saying that that's all it's all been resolved. Um but you know, it was interesting. I was uh in a different context. I was at um House of Commons once in the UK where we there was a event um where they were trying to understand how to educate and increase the use of AI in the UK. And they had many different areas of discussion, but one of them was just educating the public. And it was it was actually really hard to say what what's the recommendation should be to like UK government because I I it feels really uncomfortable to say that we should have some kind of national chat GPT training program. It just sounds crazy to to do this because again, it come comes back to the same point like we're discussing like what do we actually teach people? Like do you teach them how to write prompts, but then for what model because there's like new model come out comes out every few months and then actually it's a bit different the way you approach it. So um even like national level, I have no idea like what the UK government should be doing or any any government in terms of like adoption. I don't know. I think there was it UAE or one of the governments in the Middle East where they just what gave everyone license in their in their country. Which is, I don't know, be interesting.", "end": 3812.0 }, { "start": 3812.0, "text": "Tobi: Yeah, I also write that.", "end": 3813.0 }, { "start": 3813.0, "text": "Peter: Yeah, I don't know. Maybe that will work. I don't know. That that would be interesting to see. But uh although I I don't know how that structure works, but one thing that I would definitely just hope happens at some point is that we're going to stop giving just access to these um sloppy, cheap, bad models in like WhatsApp or something like that and people get exposed to really bad models when they interact with AI and then they think AI is rubbish because you know, they ask them and it's some 7B model saying something stupid, which I would also agree is rubbish. Uh and people don't know. I I wonder what's the population awareness about model like O3 or Gemini 1.5 or Cloud 4. I don't know, like I don't know what do you think? Do 5% of the population know about these models? Like I it's or tried it at all? I don't know, feel feels like probably no, right?", "end": 3868.0 }, { "start": 3868.0, "text": "Tobi: Yeah, potentially no. Um yeah, yeah, it's a hard problem. Um talking about the future a bit more, um what is the from your perspective, what's the what's what's your your your vision for for for devs and and and CTOs or or like very technical people in the in the future? Like will the workload quadruple? Will it like I don't know, will like will we be challenged like by like a lot of different people entering this space or what what do you think is is happening? Will it become more competitive? Will there be like a big depression of human labor? What what do you think?", "end": 3910.0 }, { "start": 3910.0, "text": "Peter: Yeah, it's it's impossible to predict. Uh yeah, I'll probably look back at what I'm going to say now six months later, a year later and it'll be the most absurd answer. But I think there are some things in terms of if if we look at like software development, I think there are some things that will not change uh probably that much in terms of things like what do you want to build, uh what's important to build, how do you architect it, design it um in the kind of bigger sense. I think these sorts of questions still need to be handled by humans. Um and and even more so because once it becomes trivial to just generate code, it really becomes it's pretty much the only job becomes like what code do I actually want because yeah, if you have the magic wishes, then like you actually need to have the wishes. And I'll I'll give you one example. I was testing Codex for for Open AI, the their kind of cloud version where you just give it task and it does it. And I did seven tickets in one evening, which was amazing. It was such a cool experience to do this. But a part from maybe an odd ticket here and there, I actually haven't done, you know, I haven't done seven tickets every evening because I don't have seven tickets every every evening. Like I actually haven't got the ideas to like what what can I do well with it to to like really move it forward. I could come up with like random tickets, but they're still things that I need to do, I need to speak to users, I need to look at the results, I need to think about like is this going in the right direction? I need to like debug some stuff that is not like debuggable with AI. So there's still there's still like these more important fundamental questions that I cannot see how they could change um with with AI, even if we've got like AGI, I think you still need people who who've got the intent to build it. Um I think that that that's one part of it. More mechanically, I think we will start shifting away from syntax and just the the general kind of hands-on coding more and more. Not that it will go away completely and for certain use cases it will probably not go away at all. Um like it was interesting to hear, for example, like Open AI researchers talk about how they train GPT for and a half, they released the podcast, um like a few months ago and they were just describing what kind of issues they were facing. And they didn't talk about this explicitly, but it was just trying to read between the lines like did they use LLMs for any of that? And I think the answer was like no, it was 0% benefit to use LLMs for that. And I think there will still be cutting edge things that it's just there's no way LLMs can like sort things out. But cutting edge is very, very tiny. So most of software is like here's a bunch of boxes and forms and we've just got some basic integrations. Like that's most of software.", "end": 4091.0 }, { "start": 4091.0, "text": "Tobi: Yeah. Yeah, yeah, and there there there are a lot of of that like week-long uh state machine building, etc. Um might no longer take weeks, right?", "end": 4104.0 }, { "start": 4104.0, "text": "Peter: Yeah.", "end": 4104.0 }, { "start": 4104.0, "text": "Tobi: Even though maybe I don't know, you need the like you you're not sure what what to do with that mental capacity that freeze up, but um like it might not take weeks anymore, which is positive.", "end": 4115.0 }, { "start": 4115.0, "text": "Peter: Yeah, it's Yeah. because complexity also, I mean made it get to those weeks, right? It was easier years ago. Yeah. Yeah, that's true. And yeah, I think in terms of capacity, like it it's very hard to predict like what labor market will do because there are many moving parts with that, but if we just take like my team or my own time, there is in no way that we have started working less. Like this is this is a crazy idea that it's like, oh, because we use AI, we're like, oh, we're like sold these tickets so I guess we're taking afternoon off. Like that's literally does not happen at all. So what we'll do instead is just move on to the next thing and we want to build more stuff and that's like what motivates us and yeah, that's just I think just the level of ambition that that you have is much higher. Um I don't know how it manifests in the job market. It could be that we just have smaller teams that much more ambitious and can just do a lot more and they just stay small and never grow. So that will be negative for the job market. But it could also be that each developer is much more productive, so actually it's worth having a lot more developers. So I could kind of see both of these extremes, but it's yeah, it's hard to know. But one thing that's for sure that I I don't know whether you had more interactions with like younger people who are starting to code now and go to university and things like that. My my experience was very limited so I don't want to make some broad sweeping statements, but I'm a little bit worried that they don't seem to be like that switched on and that on board with like really properly fully embracing LLMs for like coding and building and stuff, not just using AI but also AI applications. My impression, I don't know, my general assumption was that, you know, young people always the ones just taking up the new technologies just run with it. Yeah. And it may be true as like I don't know, as their their just help and assistant, but I I'm a little bit worried that I don't know, maybe younger people are maybe it's our universities haven't like quite adopted this, but I was speaking to someone at at university and doing like technical degree and they just like did no AI uh education. Did they had no projects that actually explicitly build with AI tools or actually building something that uses AI. I was like seem to be only those with like agency to do projects on the side who do really well. And I do worry and I I have no like statistical basis for this, but I do worry that like an average person who goes through university just has like no basis to actually succeed and and actually be one of those people who, I don't know, resolves seven tickets in like a few hours. Um but which which like that part kind of scares me more than oh they're just not going to be like junior developer jobs. Um so yeah, I don't know, I hope I'm wrong on that but that that worries me a bit.", "end": 4610.0 }, { "start": 4610.0, "text": "Tobi: I was hoping that it's the other way around, yeah, as you said, but maybe maybe we should then build an AI university, right? Like an LLM university starting today. Peter and Tobi. Um that would would be a good outcome.", "end": 4625.0 }, { "start": 4625.0, "text": "Peter: Yeah. No, I and I think the key question is like what do you teach people? I I think the only antidote because things change so much and the only antidote I know to this is just you have to constantly follow this stuff, which I feel like is a crazy requirement to give to people. Like I I do it because I really enjoy it and that's my job, but I don't know, if I was an accountant or like a developer in I don't know, with no non-AI company, like is this a reasonable thing to ask? Like I probably not.", "end": 4656.0 }, { "start": 4656.0, "text": "Tobi: Yeah. Yeah, yeah, it's hard, it's a hard ask, yeah. Um and but yeah, lean back, enjoy the ride and learn a lot. Yes. That's the request, right? Um so, um Peter, um we slowly have to come to the end. Um and um while you were talking, like I was I was slowly uh or silently um texting Codex, um tasking Codex on a few tickets. Um and uh the the task was actually to build a time machine. And uh it it yeah, it builds some Python code. Like I I never tried it out, but I I tried it out on my computer now. Um and um let's see if we have the chance to travel back in time when you were working for Lloyd's Banking Group in I think 2010 as a graduate, right? Um you were like sitting there, building some dashboards, some analytics stuff, SQL, heavy lifting, and you now have the chance to whisper like for 10 seconds um something into your younger self's ears. What what would it be?", "end": 4727.0 }, { "start": 4727.0, "text": "Peter: I would say that do the hardest technical thing like early in your career while you still can because it's much harder to get paid high amounts of money to do hard technical things badly late in your career. Um and I think when you're early in your career, like making bad software is acceptable, but later it becomes harder and harder and the the just the mechanics of your world set up is different. So yeah, early on do do the hard things while you can.", "end": 4768.0 }, { "start": 4768.0, "text": "Tobi: So for the ones who still do hard things, do them now. Thanks a lot, Peter. Um very very exciting podcast, very exciting episode. Um thanks a lot for the chat and hope to see you soon in real life, maybe at a certain point when you return from 37 degrees France where you are right now. Enjoy your vacation.", "end": 4789.0 }, { "start": 4789.0, "text": "Peter: Cool, brilliant. Thanks a lot. Really enjoyed our conversation.", "end": 3948.792 } ]