[ { "start": 0.14, "end": 2.66, "text": "Hello friends, this is the AlphaList podcast." }, { "start": 3.04, "end": 4.08, "text": "I am your host Tobi." }, { "start": 4.4, "end": 11.5, "text": "The goal of the AlphaList podcast is to empower CTOs with the info and insight they need to make the best decisions for their company." }, { "start": 11.84, "end": 17.76, "text": "We do this by hosting top thought leaders and picking their brains for insights into technical leadership and tech trends." }, { "start": 18.16, "end": 24.06, "text": "If you believe in the power of accumulated knowledge to accelerate growth, make sure to subscribe to this podcast." }, { "start": 24.4, "end": 33.6, "text": "Plus, if you're an experienced CTO, you will laugh the discussion happening in our Slack space where over six hundred CTOs are sharing insights or visit one of our events." }, { "start": 33.88, "end": 36.2, "text": "Just go to alphalus.com to apply." }, { "start": 51.3, "end": 52.94, "text": "Welcome to the Alphalus podcast." }, { "start": 53.24, "end": 67.12, "text": "I'm your host Tobi and my guest today is a Dutch hacker who managed to put an entire data warehouse inside a single Python process and Half the few cloud builds along the way." }, { "start": 67.6, "end": 71.02, "text": "Welcome, DuckDB creator, Mark Rosveld." }, { "start": 71.34, "end": 71.52, "text": "Thank" }, { "start": 71.58, "end": 72.42, "text": "you very much for having me." }, { "start": 74.12, "end": 75.64, "text": "So yeah, thanks for being here." }, { "start": 76.02, "end": 77.5, "text": "And I have to be super honest." }, { "start": 79.12, "end": 84.54, "text": "I'm a new user to DuckDB and just realized how powerful it is like a few days ago." }, { "start": 86.48, "end": 92.44, "text": "With my acquisition company, SAS Group, we actually acquired one company that has like tons of data." }, { "start": 93.96, "end": 102.0, "text": "Like, I don't know, a few billion of requests per day that they managed to crunch on very large old hardware." }, { "start": 102.84, "end": 104.6, "text": "And they are right now moving to the cloud." }, { "start": 104.64, "end": 108.6, "text": "And I said, yeah, go for BigQuery, go for BigQuery, go for it." }, { "start": 110.02, "end": 117.16, "text": "And the guy who's investigating that all and tries to move it to Greenfield, a few days ago mentioned Dr." }, { "start": 117.44, "end": 121.62, "text": "B. Um, and I just, I just realized, hey, I have a podcast recording." }, { "start": 121.66, "end": 121.74, "text": "Right" }, { "start": 121.8, "end": 123.92, "text": "on time, right on time," }, { "start": 124.8, "end": 125.76, "text": "right on time, right on time." }, { "start": 126.12, "end": 144.1, "text": "And, and, and he actually, he actually mentioned that, um, they basically crunched like a few days of data just on a, on a single machine, on a, on a MacBook, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh," }, { "start": 144.14, "end": 145.32, "text": "uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh," }, { "start": 146.04, "end": 146.74, "text": "uh, uh, uh, uh, uh, uh." }, { "start": 146.78, "end": 147.12, "text": "Um, that's amazing." }, { "start": 147.16, "end": 147.22, "text": "Yeah." }, { "start": 147.26, "end": 155.7, "text": "So maybe, maybe, maybe we start a little earlier before, before we dive deeper into duct B and how you, how you came to that idea and, and build it." }, { "start": 155.96, "end": 164.26, "text": "Um, maybe starting with you, Mark, um, rewind a little, like, um, what, what's your, what's your nerd path?" }, { "start": 164.36, "end": 166.16, "text": "Like, why did you get into computing?" }, { "start": 166.22, "end": 167.78, "text": "Why are you fascinated by it?" }, { "start": 167.98, "end": 169.22, "text": "And, and why do you do what you do?" }, { "start": 170.28, "end": 170.9, "text": "Of course, of course." }, { "start": 170.96, "end": 172.46, "text": "I mean, I can go all the way back." }, { "start": 173.8, "end": 176.62, "text": "So I got into computing kind of by accident." }, { "start": 176.76, "end": 183.26, "text": "I got into CS by accident, mostly because I was really into video games growing up, which I guess is a path that" }, { "start": 183.5, "end": 183.5, "text": "a" }, { "start": 184.86, "end": 185.6, "text": "bunch of people share." }, { "start": 186.66, "end": 195.94, "text": "And I actually, I went into, like I left high school and I had to pick something to do as one does, like pick something to study." }, { "start": 196.6, "end": 203.5, "text": "And I saw a study program at the university that was called like game programming or game technology." }, { "start": 204.46, "end": 213.12, "text": "And that spoke to me because I had been doing computer science also at, in my high school, they had a computer science class." }, { "start": 214.28, "end": 218.96, "text": "And I was doing there and I had made some video games there as well and I was doing like other computer science things." }, { "start": 219.42, "end": 220.48, "text": "So I thought it was very cool." }, { "start": 221.08, "end": 242.9, "text": "And then when I arrived at the program, I realized they had tricked me because they called it game programming, but it was really a computer science degree with like a little sort of asterisk that a lot of the practical assignments were related to like making video games instead of like the regular computer science curriculum in which I mean, I haven't followed this, but I guess you make like more programs sort of thing." }, { "start": 243.28, "end": 248.18, "text": "But it had a lot of the same curriculum, including database systems and things like that." }, { "start": 249.24, "end": 255.52, "text": "So then my bachelor in that in like computer science with a sort of annotation of programming video games." }, { "start": 256.14, "end": 266.36, "text": "And then when I came time to do my master's degree, I realized that actually, you know, what I liked so much about the bachelor was not specifically video games, although I liked that as well." }, { "start": 266.48, "end": 267.86, "text": "It was actually just programming." }, { "start": 267.94, "end": 271.06, "text": "Like I just liked computers and technology and stuff like that." }, { "start": 271.6, "end": 275.92, "text": "So I figured I'd just do a regular masters in computer science." }, { "start": 276.62, "end": 281.2, "text": "And as part of my master's, I followed a database course." }, { "start": 281.4, "end": 286.16, "text": "And what then became my supervisor for my master's?" }, { "start": 287.1, "end": 288.46, "text": "basically gave a guest lecture." }, { "start": 288.56, "end": 291.32, "text": "So I was doing my master's at the Utsers University." }, { "start": 292.06, "end": 306.58, "text": "And during my lectures that I was taking as part of that, Honus Mulaisen He came and gave a presentation about the system he was working on all the time at the research lab, the database architecture group at the CWI, and what they were working on." }, { "start": 307.16, "end": 308.22, "text": "And I thought that was really cool." }, { "start": 308.4, "end": 310.86, "text": "And I ended up going there and working together with him." }, { "start": 311.74, "end": 314.02, "text": "And then I did my master's there." }, { "start": 314.24, "end": 316.88, "text": "I did my PhD there, like I just kind of stuck around." }, { "start": 317.66, "end": 321.82, "text": "And during my PhD, together with Hanna's, we together built DuckDB." }, { "start": 322.18, "end": 324.46, "text": "Like that's kind of the story of... how that ended up." }, { "start": 324.58, "end": 332.72, "text": "I did like a bunch of work on the other database system they had there and yeah that's the path of how we ended up building the system together." }, { "start": 332.8, "end": 337.2, "text": "That was I think around two thousand eighteen, two thousand nineteen I want to say." }, { "start": 337.94, "end": 340.64, "text": "and then that kind of just has been steadily growing." }, { "start": 341.26, "end": 347.32, "text": "It started out fully as sort of like a research project but with the intention that people would start using it." }, { "start": 347.78, "end": 356.66, "text": "They did start using it and then In the end, we ended up building a company around it, spinning out, and now we're here, and there's tons of people using it, and more and more each day." }, { "start": 357.02, "end": 359.2, "text": "It's a very cool trajectory you've had." }, { "start": 359.96, "end": 360.86, "text": "That's super interesting." }, { "start": 360.96, "end": 372.24, "text": "I mean, from wanting to, I don't know, build story-heavy games to now building a database system." }, { "start": 372.74, "end": 375.0, "text": "that's a bit boring." }, { "start": 375.96, "end": 381.84, "text": "Well, we try to make Duck to be fun to use, which I think is actually kind of part of why we're successful." }, { "start": 382.4, "end": 386.36, "text": "So a lot of database systems are, I think, not fun to use." }, { "start": 386.42, "end": 390.84, "text": "And one of the things that we do try to do is to make it fun to use Duck to be." }, { "start": 391.1, "end": 394.7, "text": "And that's like, it goes down to some of the little things as well." }, { "start": 394.8, "end": 399.28, "text": "Like if you started up on the CLI, you got like some syntax highlighting and like we try to make." }, { "start": 400.2, "end": 402.7, "text": "data development fun and easy kind of." }, { "start": 402.78, "end": 404.28, "text": "So it's part of our model." }, { "start": 404.58, "end": 405.98, "text": "We try to make it not boring." }, { "start": 406.3, "end": 407.6, "text": "And for me, it's certainly not boring." }, { "start": 407.66, "end": 410.24, "text": "I mean, I got really sucked into the database world." }, { "start": 410.46, "end": 414.62, "text": "It's an amazing world, amazing smart people solving really hard problems." }, { "start": 414.72, "end": 416.56, "text": "I'm in love with it." }, { "start": 416.7, "end": 419.92, "text": "Obviously, I've been into it for a while." }, { "start": 420.7, "end": 425.34, "text": "So, how often did you go out for a beer with Monty or others then?" }, { "start": 426.28, "end": 445.8, "text": "I don't think I've met the creator of MariahDB, but at every sort of database conference, database meetup, there's like a crew of people, we all hang out together, and it's maybe not well known outside of the database community, but within the database community, it's certainly well known that database people are a bunch of alcoholics, so there's like... every time." }, { "start": 446.58, "end": 448.88, "text": "I think it may have something to do with like where they're based." }, { "start": 448.96, "end": 453.44, "text": "So database people have a heavy skew towards Germany and Netherlands." }, { "start": 453.56, "end": 455.94, "text": "And that like, it kind of leans into it." }, { "start": 456.1, "end": 457.44, "text": "So absolutely." }, { "start": 458.04, "end": 460.3, "text": "Can you tell us a bit more about the database community?" }, { "start": 460.44, "end": 461.78, "text": "Like, how is it?" }, { "start": 461.86, "end": 472.16, "text": "Like, I just know the creator of Mariety B, who also accidentally created MySQL back in the days, who's like, well known, but who else is in that scene?" }, { "start": 472.2, "end": 474.18, "text": "And how did that develop?" }, { "start": 475.68, "end": 479.6, "text": "I think the database world, the world I come from is the research world." }, { "start": 480.9, "end": 491.04, "text": "I think it's very interesting because it's like in a typical research world, there is generally not that much like money to go around, not that much industry interest, right?" }, { "start": 491.42, "end": 495.8, "text": "Because a lot of what research does is kind of orthogonal to, or like." }, { "start": 495.94, "end": 498.98, "text": "it's not orthogonal, it's like kind of more of an early stage." }, { "start": 499.64, "end": 503.94, "text": "It's not like directly in contact with like solving problems that users are facing." }, { "start": 504.52, "end": 514.179, "text": "And because the database research world, because it's so applied, there is a lot of connections between industry and research." }, { "start": 514.38, "end": 531.88, "text": "There's a very tight-knit community where there's database researchers, industry, and there's a lot of... cross-contamination there, if you will, where a lot of the database researchers, they go work in industry for a while, or maybe they leave altogether to work in industry." }, { "start": 532.02, "end": 540.44, "text": "There's database systems that come out of academia that become big systems, and also the other way around." }, { "start": 540.98, "end": 545.02, "text": "The big systems in industry, they're very heavily researched in academia again." }, { "start": 545.5, "end": 551.76, "text": "So there's a lot of sort of collaboration between the two, like the industry and academia." }, { "start": 552.34, "end": 553.54, "text": "And it's really nice to see." }, { "start": 553.7, "end": 556.18, "text": "and it's really, yeah, really cool." }, { "start": 556.66, "end": 567.82, "text": "And I think it's one of the reasons, like one of the things that drew me to databases is how it's like so fiercely practical while still being like at the research level, like very, very interesting, right?" }, { "start": 569.16, "end": 573.4, "text": "The real version of it is that at conferences you hang out and drink beers" }, { "start": 575.0, "end": 575.16, "text": "with" }, { "start": 575.94, "end": 578.08, "text": "everyone who's in that scene, I guess." }, { "start": 578.74, "end": 579.28, "text": "Absolutely." }, { "start": 579.38, "end": 579.78, "text": "Absolutely." }, { "start": 579.84, "end": 579.96, "text": "Yeah." }, { "start": 580.58, "end": 582.28, "text": "We call it networking, by the way." }, { "start": 582.8, "end": 584.38, "text": "It's not just hanging out and drinking beers." }, { "start": 584.46, "end": 585.12, "text": "It's networking." }, { "start": 586.2, "end": 587.64, "text": "But yes, that's what it comes down to." }, { "start": 587.74, "end": 587.84, "text": "Yeah." }, { "start": 587.92, "end": 590.24, "text": "The database community, they are a bunch of alcoholics." }, { "start": 590.46, "end": 593.14, "text": "So it's always a good time." }, { "start": 593.54, "end": 598.48, "text": "And it's definitely like it's both academia and like research like side by side at these events." }, { "start": 599.16, "end": 599.38, "text": "Cool." }, { "start": 601.52, "end": 619.4, "text": "What I really admire if I look at your story is the fact that you like really bootstrapped out of science in a way like you started off as a scientific project and then spun it out of university and then like really built a company out of that." }, { "start": 619.6, "end": 636.3, "text": "and you don't have like huge funding but you're only like thirty people and you're disrupting a market which is like heavily VC backed or used to be heavily VC backed and very cloud dominated by big players like Google is active there." }, { "start": 637.0, "end": 641.58, "text": "like more like, let's say, older technology, like Spark, et cetera." }, { "start": 642.34, "end": 647.62, "text": "And you kind of went into that and you disrupted that without a dollar, without a penny, I would say." }, { "start": 648.58, "end": 650.62, "text": "Well, we had our research salaries, of course." }, { "start": 651.08, "end": 652.5, "text": "Yeah, sure, sure, sure." }, { "start": 652.56, "end": 655.04, "text": "But you managed to do that." }, { "start": 655.68, "end": 663.02, "text": "Can you tell us a bit more about your story and the challenges that you faced on the way when bootstrapping the company straight out of university?" }, { "start": 664.36, "end": 667.62, "text": "So I would actually lie if I said it was very difficult." }, { "start": 667.68, "end": 669.38, "text": "I think we got super lucky." }, { "start": 670.86, "end": 672.0, "text": "The pieces fell into place." }, { "start": 672.2, "end": 682.88, "text": "If you want to develop a database system, I think there's a lot of work that needs to go into it to be able to even run your first query, to get your first user on board." }, { "start": 683.06, "end": 684.52, "text": "There's a lot of stuff that needs to happen." }, { "start": 685.1, "end": 690.56, "text": "I think research for the lab where we were at was a perfect place." }, { "start": 690.9, "end": 697.78, "text": "to do that initial work, partly because there was a bit of like, what we like to call the management vacuum." }, { "start": 698.32, "end": 713.74, "text": "So in some research labs, there's a sort of very strict oversight of people where they are basically like, you have to write like two papers a year and like you have to get like results like this month and the next month you have to get results, a lot of like management going around." }, { "start": 714.34, "end": 721.48, "text": "The situation we were at was that there was not much of that low-level management, and we could do whatever we wanted." }, { "start": 722.72, "end": 733.22, "text": "We were in a nice position where I was doing my PhD, but I had already written up all my papers and finished my thesis, but I still had some time left on my contract, so no one really cared." }, { "start": 734.48, "end": 737.74, "text": "Honest had just gotten tenure, and at that point, you can do whatever you want, as you know." }, { "start": 739.62, "end": 752.8, "text": "We were both in this position where we could just spend a few years working on something kind of crazy that may not have immediate sort of research benefits while still working as researchers." }, { "start": 752.88, "end": 754.24, "text": "So I think that worked extremely well." }, { "start": 754.86, "end": 757.78, "text": "And it tied in very well with actually a lot of the research I was doing." }, { "start": 757.88, "end": 767.36, "text": "So before what I worked on, before I made Duck to Be, me and Honest Together, we worked on uh, basically database systems for data analytics." }, { "start": 767.94, "end": 785.08, "text": "And there was another system that was developed at the CBI called Mineteby, and we tried to basically get researchers like data scientists, data analysts, data engineers to use that system and try to figure out like, okay, what is the nicest way of interacting with the database system?" }, { "start": 785.12, "end": 786.36, "text": "And we tried different approaches." }, { "start": 786.42, "end": 789.06, "text": "We tried like user defined functions within the system." }, { "start": 789.12, "end": 790.44, "text": "We tried like different integrations." }, { "start": 791.1, "end": 802.66, "text": "And kind of all we landed on was exact like kind of the precursor to DuckDB, which we called Mineti-B lights based on SQLite, which was this embedded analytical database system." }, { "start": 802.86, "end": 807.3, "text": "And that's something that a few like of our users, like we got, we got actually some traction." }, { "start": 807.64, "end": 808.9, "text": "A few people really loved it." }, { "start": 809.1, "end": 810.72, "text": "They said it solved like tons of their problems." }, { "start": 812.46, "end": 819.34, "text": "And, but we ran into like some fundamental limitations of like taking an existing database system that was not built for that." }, { "start": 819.96, "end": 829.0, "text": "and putting it, using it as an in-process database system, which is why we eventually, after several years of trying it, decided to build something custom for this scenario." }, { "start": 829.96, "end": 831.36, "text": "And that worked out very well." }, { "start": 831.4, "end": 846.72, "text": "And we basically, like, as we were developing, continuously got users and users until we got to the point where people started putting it in production and they were like, We want to put this in production, but we need to have like some contracts to like get support, things like that, right?" }, { "start": 847.24, "end": 852.0, "text": "And then we talked, we went to our university actually first and we're like, hey, can we do a support contract?" }, { "start": 852.1, "end": 854.74, "text": "And the university was like, yeah, we only take grants, sorry." }, { "start": 855.92, "end": 863.84, "text": "So we actually convinced some companies, which is the craziest part, to do grants with like, like basically just giving money for nothing, right?" }, { "start": 863.9, "end": 864.76, "text": "Like a contract." }, { "start": 864.8, "end": 868.86, "text": "that was like, here university have money in exchange you get nothing." }, { "start": 868.94, "end": 872.24, "text": "And then we had like a little sort of Wink wink nudge nudge of like." }, { "start": 872.28, "end": 874.7, "text": "yeah, we'll help you and we obviously did right?" }, { "start": 875.5, "end": 878.1, "text": "But it's like some other companies weren't so cool with it." }, { "start": 878.3, "end": 880.64, "text": "That's why we kind of ended up spinning out." }, { "start": 880.7, "end": 887.92, "text": "because like when people start telling you like we want to give you money and You can't take it like at some point like something has to give" }, { "start": 888.96, "end": 897.58, "text": "and from from your today's perspective I think you did something that like many bootstrapers actually advise you to write." }, { "start": 897.62, "end": 908.9, "text": "like you start doing consulting somewhere and then with like, twenty hours you do consulting and the rest of your time, you spend on your real product that you want to develop." }, { "start": 910.98, "end": 922.76, "text": "Like from your today's perspective, how do you think those financial constraints and this constrained development influenced the product?" }, { "start": 922.86, "end": 925.08, "text": "Like, is it for good or for worse?" }, { "start": 925.18, "end": 926.32, "text": "Like, how do you see that today?" }, { "start": 926.8, "end": 933.1, "text": "I think our scope initially when building DuckDB was actually much smaller than what it ended up becoming." }, { "start": 933.64, "end": 942.6, "text": "So I think that we started off with this idea of like, okay, we can make a improv analytical database system that will solve some of the problems that data analysts face." }, { "start": 942.66, "end": 948.2, "text": "Basically, like we set out to make DuckDB's Python package that was kind of the core idea." }, { "start": 948.86, "end": 955.38, "text": "And then it has kind of like continuously grown from that and like people continuously surprises with use cases." }, { "start": 955.48, "end": 956.54, "text": "I think that's kind of something that's." }, { "start": 956.92, "end": 974.14, "text": "um what was surprising to us initially as well was that hey we can take this like python package that does like very fast analytics like and we can actually just like use that as the back end for our whole data stack or like like replace large parts with with that." }, { "start": 974.26, "end": 975.46, "text": "So it's it's." }, { "start": 975.6, "end": 981.34, "text": "I think we started out with a small scope and people just kept using it for more, right?" }, { "start": 981.4, "end": 984.2, "text": "Like it was for us also a surprise in many ways." }, { "start": 985.72, "end": 987.84, "text": "So you did like practice cavalry in a way, right?" }, { "start": 987.96, "end": 994.02, "text": "You started early, had early early users who told you how they used the product and then you added more features, etc." }, { "start": 995.78, "end": 1004.22, "text": "Do you have to be super careful about which things you add to the system and which things you rather leave out?" }, { "start": 1004.78, "end": 1008.22, "text": "Like how do you manage, how does your governance look like?" }, { "start": 1009.06, "end": 1010.06, "text": "Yeah, absolutely." }, { "start": 1010.3, "end": 1025.0, "text": "So our governments, the way the project is run is that we have a foundation and the foundation is like a nonprofit that owns the IP and the trademarks of all the code, it owns the deputy name, the domain, all that stuff." }, { "start": 1025.76, "end": 1032.0, "text": "And our company, it basically works and contributes all that back to the foundation." }, { "start": 1032.46, "end": 1035.819, "text": "So the project is like kind of safeguarded in that foundation." }, { "start": 1037.3, "end": 1049.6, "text": "As for the code-wise of that, we try to keep the core of Duck to be lean, and there's quite some core set of principles that we have of what should and should not go into the core system." }, { "start": 1051.0, "end": 1069.48, "text": "And we kind of try to mimic like other successful projects like Python, which fun story, by the way, Python comes from the same research institute that we come from, also from the Siba at Amsterdam, where the core is kind of this lean thing, and there are extensions and plugins that you can install to do other things." }, { "start": 1070.68, "end": 1082.26, "text": "Precisely because there is this sort of like, we realize there's this sort of like large landscape of applications for the system that are quite, like, divergent in some ways." }, { "start": 1082.32, "end": 1089.38, "text": "Like, they go to completely different directions and territories, and they don't necessarily all need to have the same sort of code footprint." }, { "start": 1089.98, "end": 1093.28, "text": "In fact, for some deployments, having like a small code footprint is very important." }, { "start": 1093.34, "end": 1095.26, "text": "So we have our WebAssembly deployment." }, { "start": 1095.36, "end": 1096.8, "text": "Maybe you've seen it, where Dr." }, { "start": 1097.0, "end": 1098.06, "text": "B can run in your browser." }, { "start": 1099.88, "end": 1106.02, "text": "It's very critical that that is as small as possible because we send the whole database system to your browsers like a JavaScript package, right?" }, { "start": 1106.46, "end": 1107.52, "text": "So it needs to be small." }, { "start": 1109.82, "end": 1114.2, "text": "But yeah, so we are very strict about what goes into the core." }, { "start": 1115.12, "end": 1117.34, "text": "And then we try to encourage people to build extensions." }, { "start": 1117.38, "end": 1119.8, "text": "And we also have a framework for pluggable extensions." }, { "start": 1120.0, "end": 1123.98, "text": "We have a community extensions repository where people can make their own extensions." }, { "start": 1124.48, "end": 1127.72, "text": "We have our own that we call the core extensions that we maintain as a team." }, { "start": 1128.66, "end": 1133.28, "text": "And basically anything that we think shouldn't go into the core system goes into those extensions." }, { "start": 1133.9, "end": 1139.92, "text": "So yet there's no commercial software that you sell right like you don't sell software you don't sell any modules." }, { "start": 1140.36, "end": 1141.0, "text": "it's all open." }, { "start": 1141.1, "end": 1143.24, "text": "it's all for free and you just a support." }, { "start": 1144.08, "end": 1144.78, "text": "Absolutely so." }, { "start": 1145.22, "end": 1155.8, "text": "we we originally when we started off like our company we thought of like different ways to monetize it and one of the things we did think about was okay we can make commercial extensions." }, { "start": 1157.02, "end": 1159.1, "text": "and we have talked to some customers about it." }, { "start": 1159.54, "end": 1165.86, "text": "But it turns out most of our customers actually don't want things to be commercial, like to be private." }, { "start": 1167.1, "end": 1171.56, "text": "Because if it's open source and other people start using it, we have to maintain it." }, { "start": 1172.16, "end": 1175.68, "text": "Whereas if it's private, we don't." }, { "start": 1176.82, "end": 1181.4, "text": "If they stop paying us, for example, then we will just leave their private extensions." }, { "start": 1181.44, "end": 1183.18, "text": "They can keep using them, obviously." }, { "start": 1184.16, "end": 1187.68, "text": "they will basically have to keep on contracting with us in order for us to maintain it." }, { "start": 1188.1, "end": 1193.54, "text": "Whereas if it's in the open source, we have some obligation to keep it up to date and keep maintaining it." }, { "start": 1194.92, "end": 1206.86, "text": "I think it's something that we learn is that there is a lot of both the people that are customers that pay us to develop the software and for us and the project." }, { "start": 1208.6, "end": 1211.7, "text": "Making everything open source has worked very well because of that." }, { "start": 1211.82, "end": 1216.42, "text": "like they want it to be open because then it gets to the open it becomes maintained." }, { "start": 1216.74, "end": 1219.7, "text": "other people do like testing of it like it becomes more stable." }, { "start": 1219.74, "end": 1221.98, "text": "that way you get a lot of the benefits of open source." }, { "start": 1222.52, "end": 1227.5, "text": "and we want it to be open because anything that we make if it goes into the open source it grows." }, { "start": 1227.68, "end": 1228.84, "text": "project and from like." }, { "start": 1230.32, "end": 1233.86, "text": "Obviously we want to grow the project just because We're excited about it, right?" }, { "start": 1233.9, "end": 1242.1, "text": "We like the project, but also from a business point of view, we want to grow the project because the project is like kind of the entry point for customers." }, { "start": 1242.18, "end": 1244.56, "text": "Like people start using Duck to be for free." }, { "start": 1244.94, "end": 1247.8, "text": "They get the open source version and then they talk to us, right?" }, { "start": 1247.86, "end": 1250.14, "text": "So the more people use Duck to be the more." }, { "start": 1250.58, "end": 1258.08, "text": "people that want to talk to us, that want to contract with us, that want to support contracts, that's kind of like the way we see that sort of the pipeline." }, { "start": 1258.12, "end": 1262.24, "text": "And that's why we want to, among others, grow the project as much as possible." }, { "start": 1263.3, "end": 1279.96, "text": "To me, many of the things you say and the success you have with your database kind of rings the bell that there are crazy war stories where someone approached you to, I don't know, buy your whole company or back you with a trillion dollars or something like that." }, { "start": 1280.46, "end": 1291.04, "text": "is there anything where you where you thought like whoa this this is too crazy like xyz wrote me and like we have to turn down the offer but it sounds so nice." }, { "start": 1292.1, "end": 1298.72, "text": "So it's interesting because we had a long rounds when we started the company of talking to venture capitalist and." }, { "start": 1300.46, "end": 1306.68, "text": "We initially thought we were going to take venture capitalist funding, but we ended up not taking it after a lot of consideration." }, { "start": 1306.74, "end": 1310.34, "text": "We talked to many venture capitalists, like all of the big ones." }, { "start": 1312.16, "end": 1327.62, "text": "And I think the reason was precisely because we thought that taking venture capital, it would lead to this sort of this moment where we would have to choose between like paying them back in a way and the project." }, { "start": 1327.7, "end": 1328.92, "text": "and we kind of wanted to avoid that." }, { "start": 1329.0, "end": 1334.18, "text": "We want to make sure that we can always keep the projects like in a good healthy state." }, { "start": 1335.6, "end": 1341.02, "text": "So but we actually there's one thing that we have done that we think is very interesting." }, { "start": 1342.4, "end": 1349.12, "text": "There is another company which is our kind of like partnership slash partially a daughter company called Mother Duck." }, { "start": 1349.22, "end": 1350.06, "text": "Maybe you've heard of them." }, { "start": 1351.32, "end": 1352.88, "text": "And they have taken VC money." }, { "start": 1354.78, "end": 1364.3, "text": "And so that was actually one of the ideas that came from one of the professors at our research institute that was advising us was in the past made companies." }, { "start": 1365.78, "end": 1368.94, "text": "In the past has made several startups and has experience in this domain." }, { "start": 1370.0, "end": 1386.14, "text": "And he thought, okay, What if you make a company that is like not VC funded that maintains the core system, but there are you like kind of work together with other companies that built like sort of like we call it like a verticals out of the." }, { "start": 1386.62, "end": 1387.26, "text": "out of the core." }, { "start": 1387.86, "end": 1391.86, "text": "And the idea is that duct B is very, very like general purpose." }, { "start": 1391.9, "end": 1393.86, "text": "You can use it for many different verticals, right?" }, { "start": 1394.64, "end": 1400.82, "text": "You can obviously build a data warehouse with it, but maybe you can also embed it in like, like a car or something, right?" }, { "start": 1400.86, "end": 1403.0, "text": "Like it's different verticals you can go into." }, { "start": 1403.74, "end": 1406.52, "text": "And the idea is that we maintain the core product." }, { "start": 1407.02, "end": 1413.0, "text": "And then there's other companies that we work together with that pay us and partially also in shares." }, { "start": 1413.6, "end": 1415.3, "text": "to develop that vertical." }, { "start": 1415.48, "end": 1418.84, "text": "And then we work very closely together with them to make sure it's a success." }, { "start": 1420.02, "end": 1429.38, "text": "And then that means the core sort of product, the project itself is isolated, because we as a company have not taken VC money, we're kind of independent." }, { "start": 1429.94, "end": 1434.24, "text": "But there's still this sort of hyperscaler stuff that can happen on the fringes." }, { "start": 1434.36, "end": 1448.56, "text": "So MotherDuck, they make a service product around DuckTB, like DuckTB as a service, where they do the whole like, okay, let's build a products around people that want to use DuckTV but don't want to necessarily manage stuff themselves, right?" }, { "start": 1448.6, "end": 1455.38, "text": "Like sort of the big query of DuckTV, if you will, where you just hand someone your credit card and they will take care of the complexity for you." }, { "start": 1456.94, "end": 1458.56, "text": "And they're building that and that's great." }, { "start": 1459.06, "end": 1460.48, "text": "They are like excellent at that." }, { "start": 1460.54, "end": 1468.34, "text": "They're great at the like, they do all the hard stuff of like, okay, we need to get like the Kubernetes deployments, right?" }, { "start": 1468.38, "end": 1470.12, "text": "They need to like build a service out there." }, { "start": 1470.24, "end": 1473.72, "text": "They need to do all the sort of like, oh, We're actually managing your data." }, { "start": 1473.78, "end": 1480.42, "text": "So we have to get like all the, all the nice like certificates of like, oh, we're like compliant to X, Y and Z and stuff." }, { "start": 1480.86, "end": 1482.46, "text": "They do that and they do that very well." }, { "start": 1482.54, "end": 1484.74, "text": "And then we work together and we maintain the core." }, { "start": 1484.84, "end": 1485.06, "text": "So it's," }, { "start": 1485.84, "end": 1487.82, "text": "you can do the cool stuff and the nutshell." }, { "start": 1488.74, "end": 1494.2, "text": "I, for me, I think that is the cool stuff, but everyone thinks different things are the cool stuff." }, { "start": 1494.56, "end": 1497.84, "text": "So I'm sure they also have a great like." }, { "start": 1498.48, "end": 1499.9, "text": "the stuff they're doing is also very cool." }, { "start": 1500.34, "end": 1501.84, "text": "So I don't want to like diminish." }, { "start": 1502.22, "end": 1503.34, "text": "We can focus on the core, yes." }, { "start": 1504.48, "end": 1504.88, "text": "Quite cool." }, { "start": 1505.24, "end": 1508.1, "text": "But yeah, coming a bit more to the core, actually." }, { "start": 1508.56, "end": 1513.66, "text": "Your concept is basically SQL Lite for data analysts, right?" }, { "start": 1513.84, "end": 1514.84, "text": "Yeah." }, { "start": 1514.88, "end": 1518.5, "text": "In a nutshell, and slightly inspired by that potentially as well." }, { "start": 1519.04, "end": 1525.22, "text": "And then your use case is typically like slow running queries for analytics." }, { "start": 1526.1, "end": 1531.34, "text": "that instead of, I don't know, spinning up a Spark cluster, Yeah." }, { "start": 1533.06, "end": 1540.52, "text": "Your customers typically have or most users out there have too little data to spin up a Spark cluster in reality." }, { "start": 1541.28, "end": 1552.54, "text": "And then you basically just offer a single core or multi-core single machine software that people can just use on the command line on their like" }, { "start": 1553.4, "end": 1553.9, "text": "for machine" }, { "start": 1553.96, "end": 1564.92, "text": "with a few gigabytes, a few hundred gigabytes of RAM potentially and a very, very, very efficient and fast CPU to just analyze, I don't know, a petabyte or what?" }, { "start": 1565.66, "end": 1567.88, "text": "Yeah, something small, right?" }, { "start": 1571.46, "end": 1584.0, "text": "Yeah, so I think one of the sort of realizations that I think it has been made is that like computers have been getting Very very good." }, { "start": 1584.44, "end": 1588.32, "text": "Like you your lap like the laptop I'm talking to you on right now." }, { "start": 1588.42, "end": 1592.24, "text": "It has like sixteen cores Sixty four gigs of RAM, right?" }, { "start": 1592.28, "end": 1594.0, "text": "It has like a two terabyte disk." }, { "start": 1594.1, "end": 1596.08, "text": "like these are big numbers." }, { "start": 1596.78, "end": 1602.32, "text": "and if you look back to where this whole like the the spark thing which comes from MapReduce, right?" }, { "start": 1602.36, "end": 1603.64, "text": "Like the Hadoop all that like" }, { "start": 1604.42, "end": 1604.86, "text": "distributed" }, { "start": 1604.94, "end": 1613.28, "text": "computing like When that really started was like the early two thousands and back then you didn't have sixty four gigs of memory." }, { "start": 1613.36, "end": 1617.8, "text": "You have maybe sixty four megabytes or maybe a bit more, but not that much more." }, { "start": 1618.28, "end": 1628.06, "text": "So the realization was that computers have basically gotten better, much faster than most datasets have grown." }, { "start": 1628.76, "end": 1632.78, "text": "And the reason is also because most datasets are kind of constrained in size." }, { "start": 1633.22, "end": 1638.94, "text": "Like, if you think about the amount of people on the planet, right, it's growing, but not that fast." }, { "start": 1639.02, "end": 1641.08, "text": "It's kind of like the growth has slowed down, right?" }, { "start": 1642.58, "end": 1648.34, "text": "And a lot of interesting data, at least, comes in part from humans, right?" }, { "start": 1648.4, "end": 1650.76, "text": "Like humans make most interesting datasets." }, { "start": 1651.18, "end": 1654.38, "text": "And that could be customers or like people typing on a keyboard." }, { "start": 1654.44, "end": 1656.92, "text": "And there's only so many people that can type on so many keyboards." }, { "start": 1657.36, "end": 1666.92, "text": "Obviously, if you add in like machine generated data, you can go infinite, like you can get as much data as you want, but often that's lower value, less interesting data." }, { "start": 1667.38, "end": 1674.66, "text": "So when you look at only the high value interesting data, the amount of data has been growing like this, right?" }, { "start": 1674.72, "end": 1675.7, "text": "Like it's been going like this." }, { "start": 1676.44, "end": 1681.12, "text": "But the speed of computing has been going like this, like very, very steeply." }, { "start": 1681.6, "end": 1690.34, "text": "So things that maybe you needed a cluster of computers for in you don't need, you can do on your laptop in two thousand twenty five right like." }, { "start": 1690.38, "end": 1698.64, "text": "there's this sort of like this intersection points which we call the data singularity where ninety nine percent of all data problems can be solved on the same computer." }, { "start": 1698.74, "end": 1700.66, "text": "and that has been like ever increasing." }, { "start": 1702.2, "end": 1716.24, "text": "and so there's this sort of like this increase and I think that's also in part because of computers and also because of software like a lot of software for analysis has been kind of suboptimal and not really state-of-the-art." }, { "start": 1716.32, "end": 1747.38, "text": "So there's very cool algorithms that have been implemented in the database research world and have found their way into basically the commercial high-performance offerings, like Snowflake has an excellent query engine, it's very fast, BigQuery, excellent query engine very fast, but the off-the-shelf I run my data thing that is open source that you can run yourself has always been Spark and Spark is just not very fast on a single node, right?" }, { "start": 1747.44, "end": 1750.74, "text": "Like the whole selling point of Spark is that it can scale up." }, { "start": 1752.62, "end": 1760.9, "text": "And kind of like where DuckDB now sits is that we can process up to like several terabytes of data on a single computer." }, { "start": 1761.36, "end": 1768.96, "text": "And it turns out that's More than most people have in terms of data and especially more than most people have in terms of hot data, right?" }, { "start": 1769.0, "end": 1773.96, "text": "They're like the actual the working sets we call it which is the data They're actually looking at right" }, { "start": 1774.2, "end": 1777.92, "text": "which is something like a day of data or something that you crunch exactly" }, { "start": 1778.28, "end": 1778.78, "text": "Exactly." }, { "start": 1778.92, "end": 1780.78, "text": "Yeah, because you may have a lot of historic data." }, { "start": 1781.0, "end": 1783.78, "text": "Yeah, but if all your queries involve only like the last week." }, { "start": 1783.9, "end": 1787.0, "text": "then that's the sort of data set size that matters." }, { "start": 1787.82, "end": 1795.18, "text": "Like you may have a petabyte of historic data because it goes back thirty years, but then your last week may only be like a few terabytes." }, { "start": 1795.24, "end": 1797.24, "text": "And then again, it goes back to fitting on a single machine." }, { "start": 1799.76, "end": 1800.48, "text": "Yeah, super interesting." }, { "start": 1800.54, "end": 1805.56, "text": "And are you like, I don't know, constrained it on data formats?" }, { "start": 1807.38, "end": 1810.92, "text": "You mentioned you use Parquet, for example, to store your data typically." }, { "start": 1811.2, "end": 1813.3, "text": "So that's like the standard storage engine, I guess." }, { "start": 1814.52, "end": 1818.46, "text": "But you're still able to also analyze large CSVs, et cetera, right?" }, { "start": 1818.78, "end": 1818.9, "text": "For" }, { "start": 1818.98, "end": 1819.42, "text": "sure, yeah." }, { "start": 1819.84, "end": 1821.58, "text": "Using your plug-in infrastructure, I guess." }, { "start": 1822.44, "end": 1823.12, "text": "Absolutely, yeah." }, { "start": 1823.24, "end": 1825.06, "text": "We have our own set of data formats." }, { "start": 1826.14, "end": 1831.96, "text": "I think one of the core things we're trying to do is to make analytics as easy as possible." }, { "start": 1832.04, "end": 1836.0, "text": "And part of that, of course, means working with whatever datasets you have, right?" }, { "start": 1836.04, "end": 1843.46, "text": "Like as a user, you can't often control what data you need to read because someone else gives you a CSV file." }, { "start": 1843.56, "end": 1845.36, "text": "Someone else gives you a JSON file, right?" }, { "start": 1845.84, "end": 1849.02, "text": "So we are not like purists in that regard." }, { "start": 1849.08, "end": 1851.26, "text": "Like we're happy to read a CSV file for you as well." }, { "start": 1852.26, "end": 1856.28, "text": "And I think that kind of comes down to also like you mentioned before the SQLite for analytics." }, { "start": 1858.1, "end": 1864.6, "text": "I think there is a bit, like we try to be the SQLite for analytics in the sense that we're easy to use, easy to deploy and things like that." }, { "start": 1865.0, "end": 1876.82, "text": "But there's I think one big difference between analytical workloads and transactional workloads, which what SQLite is optimized for, which is that analytical workloads are generally much more varied, right?" }, { "start": 1876.9, "end": 1877.04, "text": "Like" }, { "start": 1877.46, "end": 1877.56, "text": "you" }, { "start": 1877.96, "end": 1880.42, "text": "get a bunch of data from somewhere like." }, { "start": 1880.82, "end": 1889.22, "text": "a government website like someone else decided to write like get or some CSV files or like dump a bunch of parquet files to S three or like." }, { "start": 1890.04, "end": 1895.5, "text": "there's a lot more variety in what you're reading and we try to accommodate for all of that as well." }, { "start": 1896.78, "end": 1901.22, "text": "And your data storage engine is in a way like immutable." }, { "start": 1901.34, "end": 1901.66, "text": "so you." }, { "start": 1902.54, "end": 1907.66, "text": "like you only add new things, you only add schema changes through additional data, right?" }, { "start": 1907.74, "end": 1910.74, "text": "Like you don't change the data in like itself." }, { "start": 1912.36, "end": 1917.94, "text": "Yeah, so I think what you're talking about is the lake house technology that we have this duck Lake." }, { "start": 1919.04, "end": 1922.8, "text": "And indeed for duck Lake, we have different ways of storing data." }, { "start": 1923.36, "end": 1924.76, "text": "One of them is our own file format." }, { "start": 1925.26, "end": 1930.76, "text": "And the other like the other one is duck Lake and we can of course also write data to CSV files, JSON files, et cetera." }, { "start": 1932.0, "end": 1938.92, "text": "but one of our more recent additions is the Lakehouse Technology Duck Lake and how that works is effectively." }, { "start": 1939.54, "end": 1950.74, "text": "you have in as three buckets or like a blob store right some file system where you want to store data and that file system is constrained in the sense that you can generally only write new files or delete files." }, { "start": 1951.64, "end": 1990.3, "text": "That's a very common sort of use case nowadays and Effectively, what we have done is we have built a database system like a classical database system that you're used to if you have run, say, Postgres, where you can create tables, insert data, run deletes updates over that abstraction by having a bunch of parquet files sitting there together with a metadata store that can sit in another database system, like for example, Pulsegres, or also our own storage format that I just mentioned, because we also have our own storage format, which is similar to SQLite in a way." }, { "start": 1990.96, "end": 1993.08, "text": "You have that own format to keep it simple or?" }, { "start": 1994.46, "end": 1999.52, "text": "We have our own format because there are just like our own storage format." }, { "start": 1999.6, "end": 2002.06, "text": "It's a single file storage format similar to SQLite." }, { "start": 2003.38, "end": 2007.96, "text": "However, it's optimized for analytical use cases, so data stored in a columnar way." }, { "start": 2008.52, "end": 2013.4, "text": "there is data compression, and we support basically the multi-table sort of thing." }, { "start": 2013.88, "end": 2020.5, "text": "So when we started off creating DuckDB, there was already Parquet that already existed, although it was." }, { "start": 2021.12, "end": 2023.34, "text": "way less popular than it is now." }, { "start": 2023.46, "end": 2023.94, "text": "I think that" }, { "start": 2024.0, "end": 2025.54, "text": "was created by Twitter, right?" }, { "start": 2026.18, "end": 2028.46, "text": "As far as I remember, like by some Twitter engineer." }, { "start": 2029.68, "end": 2031.76, "text": "I'm not sure." }, { "start": 2032.36, "end": 2033.96, "text": "Somewhere in that Hadoop ecosystem." }, { "start": 2034.0, "end": 2034.22, "text": "Yeah," }, { "start": 2035.3, "end": 2036.7, "text": "it comes from the Hadoop ecosystem." }, { "start": 2036.78, "end": 2038.84, "text": "I'm not sure which companies exactly." }, { "start": 2039.34, "end": 2048.239, "text": "I think the original sort of work on Parquet comes from Dremel, like from Google, they had like Dremel and then like that turned into like an open format." }, { "start": 2049.98, "end": 2053.54, "text": "Parquet has been there for a while, but it was not very popular for a long time." }, { "start": 2054.159, "end": 2061.739, "text": "I think we together with others kind of popularized it by accepting it as a standard and making it easy to work with." }, { "start": 2062.659, "end": 2063.78, "text": "But that already existed." }, { "start": 2063.94, "end": 2072.92, "text": "The thing is with Parquet is that it is this sort of immutable write data once and then never read it again sort of thing." }, { "start": 2074.199, "end": 2076.1, "text": "our own format, it's a bit different." }, { "start": 2076.679, "end": 2082.239, "text": "So per K, what we call that is a table format, because it stores a single table." }, { "start": 2082.699, "end": 2090.48, "text": "And our format is a database format, which means it can store multiple tables, but also things like views or functions or types, right?" }, { "start": 2090.78, "end": 2093.739, "text": "Schemas, like basically a full database in a single file." }, { "start": 2095.139, "end": 2096.12, "text": "Okay, understood." }, { "start": 2096.76, "end": 2103.8, "text": "And generally, like your typical users are like, is basically everyone, right?" }, { "start": 2103.86, "end": 2113.44, "text": "Like almost everyone out there, like apart from Google, who potentially have more hot data to crunch it with your solution." }, { "start": 2115.02, "end": 2119.56, "text": "But typical customers, I guess, have like less than one terabyte of data." }, { "start": 2120.66, "end": 2126.78, "text": "And typical queries scan like one megabytes or something like that, right?" }, { "start": 2129.7, "end": 2145.54, "text": "And how do you feel that like The world out there that most people also are facing, the reality out there is the story that people are told is kind of bloated." }, { "start": 2145.6, "end": 2155.18, "text": "I mean, basically Google makes you believe that you just need BigQuery and everyone has a BigQuery use case." }, { "start": 2156.52, "end": 2158.64, "text": "How many people really have a BigQuery use case?" }, { "start": 2158.82, "end": 2159.62, "text": "from your perspective?" }, { "start": 2160.0, "end": 2162.54, "text": "I think it is extremely few." }, { "start": 2164.66, "end": 2166.78, "text": "Google themselves have published these numbers, by the way." }, { "start": 2166.84, "end": 2167.96, "text": "This is not my data." }, { "start": 2169.2, "end": 2176.08, "text": "But I think something like, ninety-nine percent of queries run on BigQuery operate on less than three hundred gigabytes of data." }, { "start": 2181.46, "end": 2188.26, "text": "Once you start going into what workloads actually need BigQuery, from their own workloads, right?" }, { "start": 2188.32, "end": 2195.02, "text": "So that's already skewed towards people that have like gotten BigQuery because they feel the needed from their own workloads." }, { "start": 2195.14, "end": 2197.94, "text": "It's already like marginal to fractional amounts." }, { "start": 2198.54, "end": 2211.66, "text": "And one thing we see a lot of people doing is they may not migrate fully to DuckTB because they may still have that one query that needs BigQuery, but they can still save tons of money by using DuckTB for everything else, right?" }, { "start": 2211.72, "end": 2222.08, "text": "Like, just because you have BigQuery doesn't mean you need all your analysts that are just running some, like, some analytics over like a few gigabytes of parquet files to also use BigQuery for that." }, { "start": 2222.6, "end": 2225.44, "text": "So we have seen a lot of these sort of incremental things." }, { "start": 2226.06, "end": 2232.66, "text": "where people use DuckDB for, like, ninety-nine percent of their workloads to save tons of money." }, { "start": 2232.8, "end": 2236.36, "text": "Because, again, like, your analysts already have a laptop." }, { "start": 2236.46, "end": 2238.32, "text": "It's free real estate, right?" }, { "start": 2238.36, "end": 2240.76, "text": "Like, you have a laptop right in front of you." }, { "start": 2241.2, "end": 2244.5, "text": "Why not just run the query there instead of going and paying Google to run it?" }, { "start": 2244.96, "end": 2245.08, "text": "Yeah." }, { "start": 2245.58, "end": 2270.54, "text": "In fact, I heard a fun story that at, I think, SoundCloud, there's a query printed, like a BigQuery printed and uh, like sticking to a wall, a frame to a wall, uh, that costed, I think, six hundred K or something like that on BigQuery or was, was at least meant to cost that, that much, uh, much money because someone accidentally forgot the limit." }, { "start": 2271.74, "end": 2271.8, "text": "Yep." }, { "start": 2272.06, "end": 2272.54, "text": "Classic." }, { "start": 2273.08, "end": 2273.14, "text": "Yeah." }, { "start": 2273.2, "end": 2276.98, "text": "And then they'll be happy to run it for you, obviously, because they're charging you." }, { "start": 2277.6, "end": 2278.6, "text": "They're making money, right?" }, { "start": 2278.66, "end": 2282.62, "text": "Like it's the incentives are, but it's, uh, misaligned, maybe, let's say." }, { "start": 2284.44, "end": 2303.22, "text": "And is it in fact the case that your stuff just runs faster because it's on a single machine and the overhead of distribution is so big that it actually slows many use cases down instead of accelerating it?" }, { "start": 2304.52, "end": 2305.98, "text": "So there's different components." }, { "start": 2306.24, "end": 2323.54, "text": "So I think when you compare DuckDB to BigQuery and Snowflake, which are like state-of-the-art amazing engines in their own rights, I think that's definitely the case, that running things on a single process, a single computer, it just removes all of the sort of communication overhead." }, { "start": 2323.78, "end": 2325.62, "text": "You can get a lot of speed up over that." }, { "start": 2327.34, "end": 2337.22, "text": "But when you compare against, for example, Spark, a big difference is also that DuckDB does use these state-of-the-art query processing technologies that make it much faster just innately as well." }, { "start": 2338.12, "end": 2339.86, "text": "So there's two sides of this." }, { "start": 2339.96, "end": 2343.7, "text": "When you compare it to BigQuery and Snowflake, absolutely less communication overhead." }, { "start": 2344.44, "end": 2355.3, "text": "We don't have to spin up a Kubernetes instance to get a worker to coordinate and communicate and send over data." }, { "start": 2356.36, "end": 2359.92, "text": "If it's all there, it's going to be much faster, oftentimes instant." }, { "start": 2360.36, "end": 2365.86, "text": "Sometimes the whole bottleneck is this coordination, especially if your datasets are small." }, { "start": 2366.46, "end": 2368.64, "text": "So absolutely, it saves a lot of time." }, { "start": 2370.1, "end": 2370.36, "text": "Okay." }, { "start": 2371.18, "end": 2372.04, "text": "And where are the limits?" }, { "start": 2372.62, "end": 2373.26, "text": "of your solution?" }, { "start": 2375.12, "end": 2378.0, "text": "When do you touch too much of this guy?" }, { "start": 2379.48, "end": 2381.68, "text": "When do you need to actually use BigQuery or Spark?" }, { "start": 2381.78, "end": 2383.18, "text": "Yeah, I mean, that's a great question." }, { "start": 2383.92, "end": 2388.68, "text": "I think these use cases absolutely still exist, but they're very rare." }, { "start": 2389.02, "end": 2395.44, "text": "So it's hard to put an exact number on it, but it's generally up in the multi-terabyte range." }, { "start": 2395.48, "end": 2404.48, "text": "So if you go to AWS or your favorite cloud hosting provider, They make single mode boxes very, very big nowadays." }, { "start": 2404.54, "end": 2409.24, "text": "So you have like, you can rent a computer and it has like several terabytes of RAM." }, { "start": 2409.4, "end": 2412.12, "text": "It has like a four hundred cores, right?" }, { "start": 2412.18, "end": 2421.54, "text": "Like it has a ton of CPU power and that can crunch through like multi-terabytes workloads, no problem." }, { "start": 2422.92, "end": 2426.78, "text": "Once you get to like the petabyte range of working data, right?" }, { "start": 2426.86, "end": 2429.96, "text": "Like we talked about this before, your working set is like a petabyte." }, { "start": 2430.36, "end": 2435.6, "text": "Like maybe you want to actually run a query over all your historic data and you're gathering multiple terabytes a month." }, { "start": 2437.6, "end": 2443.76, "text": "Then you may get into that sort of range where you do need a distributed engine, but it's very rare." }, { "start": 2444.14, "end": 2448.42, "text": "And I think that's also like, it's more the exception than the rule, even in organizations that do use this." }, { "start": 2448.88, "end": 2450.94, "text": "So there's still a lot of value to using DuckDB." }, { "start": 2451.66, "end": 2453.78, "text": "for like the ninety nine percent of use case that don't." }, { "start": 2455.08, "end": 2472.04, "text": "And is there one crazy use case like one crazy war story about a time when one laptop crushed a fifteen note cluster or something like where you were like super surprised that or a client was super surprised that it ultimately worked out." }, { "start": 2472.66, "end": 2472.66, "text": "I" }, { "start": 2473.98, "end": 2480.18, "text": "think I don't have the exact quiz, but there are some workloads where we beat BigQuery by quite some amount." }, { "start": 2481.54, "end": 2500.72, "text": "As for the laptop, we recently did a, my co-founder, Hannes, he has an old laptop lying around from a year old MacBook, and he ran the TPCH dataset benchmark on it with one terabyte of data on a ten-year-old machine." }, { "start": 2502.36, "end": 2503.0, "text": "And that worked." }, { "start": 2503.08, "end": 2504.22, "text": "That was quite funny to see." }, { "start": 2504.82, "end": 2510.48, "text": "And that was something that people in the past have used, like easily have used distributed computing for." }, { "start": 2511.14, "end": 2511.36, "text": "Okay." }, { "start": 2511.84, "end": 2515.1, "text": "And under the hood, I mean, you basically developed the system from scratch, right?" }, { "start": 2515.18, "end": 2520.36, "text": "So it uses, or you use C plus plus as far as I know, right?" }, { "start": 2520.8, "end": 2522.02, "text": "Like it's all written C plus plus." }, { "start": 2523.26, "end": 2524.68, "text": "What what led to that decision?" }, { "start": 2524.76, "end": 2524.96, "text": "like?" }, { "start": 2525.66, "end": 2530.26, "text": "just because you were used to it or because like the performance is still." }, { "start": 2531.58, "end": 2536.28, "text": "Crazy much better than if you would use I don't know go or something trendy." }, { "start": 2536.94, "end": 2540.24, "text": "Yeah, I mean the reason so for us as people suppose was actually." }, { "start": 2540.82, "end": 2546.1, "text": "modernizing because it's funny in the database world, everything was written in C for a long time." }, { "start": 2546.66, "end": 2558.9, "text": "So the database system we were working on was written in C and like all the other big database systems in the open source like SQLite, MySQL, Postgres, they're all written in C as well." }, { "start": 2559.36, "end": 2561.22, "text": "So for us, we were like, We're modern." }, { "start": 2561.3, "end": 2561.78, "text": "We're using C++." }, { "start": 2562.48, "end": 2563.74, "text": "Obviously, that's no longer true." }, { "start": 2563.8, "end": 2565.64, "text": "We can't claim that it's modern anymore." }, { "start": 2566.04, "end": 2567.56, "text": "But at the time, that was kind of the idea." }, { "start": 2568.78, "end": 2569.32, "text": "Why did we use C++?" }, { "start": 2570.04, "end": 2575.06, "text": "I think performance is the biggest factor by far, together with our own expertise." }, { "start": 2575.12, "end": 2585.84, "text": "Like, I mean, we were more C programmers than C++, but there's a lot of sort of skills that sort of transfer because C++ is, I mean, it's kind of in the name." }, { "start": 2585.9, "end": 2591.54, "text": "It's like C and so like... a lot of bells and whistles, but in its core it does like." }, { "start": 2591.6, "end": 2594.78, "text": "if you understand C, you understand C++ very fast as well." }, { "start": 2595.86, "end": 2597.02, "text": "Okay, okay, understood." }, { "start": 2597.36, "end": 2598.28, "text": "What else is under the hood?" }, { "start": 2598.48, "end": 2604.32, "text": "I mean, like, yeah, it seems to be like super small and super efficient." }, { "start": 2604.82, "end": 2607.02, "text": "Like, how did you get there?" }, { "start": 2607.28, "end": 2607.96, "text": "What else is there?" }, { "start": 2608.1, "end": 2610.38, "text": "Like what you really urgently depend on?" }, { "start": 2611.86, "end": 2613.98, "text": "So you mean in terms of like... Yeah," }, { "start": 2614.58, "end": 2618.82, "text": "like libraries, like technologies, like what's under the hood?" }, { "start": 2619.6, "end": 2622.72, "text": "We actually have a very strict no external dependencies policy." }, { "start": 2623.32, "end": 2627.18, "text": "So we don't really rely on many libraries." }, { "start": 2627.76, "end": 2634.22, "text": "And that's one of the reasons we have this extension ecosystem, because the core of DuckDB is supposed to be completely standalone." }, { "start": 2634.74, "end": 2636.96, "text": "That doesn't mean we don't use anybody's... code." }, { "start": 2637.16, "end": 2638.6, "text": "There are some dependencies there." }, { "start": 2638.88, "end": 2642.74, "text": "For example, one of the things we do is we have used the Postgres parser." }, { "start": 2643.88, "end": 2648.5, "text": "We are actually in the process of replacing it with our own, but that's still a working process." }, { "start": 2648.56, "end": 2651.62, "text": "We're still using the Postgres parser and have for many years." }, { "start": 2652.06, "end": 2652.48, "text": "It's great." }, { "start": 2652.72, "end": 2653.14, "text": "It works." }, { "start": 2654.96, "end": 2655.58, "text": "It does the trick." }, { "start": 2658.36, "end": 2661.2, "text": "The actual core number crunching, that's all." }, { "start": 2662.28, "end": 2664.5, "text": "we made it, we programmed it." }, { "start": 2665.14, "end": 2682.54, "text": "Obviously it's building on top of a lot of research in the database world and sort of like the fundamentals of the system are very much based on the state of the art in sort of how do you build a fast analytical database system, which is research that has been also partially carried out at the research institute we came from." }, { "start": 2682.64, "end": 2688.18, "text": "So we were very acquainted with the people that actually did that research and like obviously the research field." }, { "start": 2688.26, "end": 2702.1, "text": "Like there's a lot of amazing ideas in the database research field that for a long time had only been implemented in sort of the closed source, like high performance engines like Snowflake and BigQuery and not in the open." }, { "start": 2702.18, "end": 2705.46, "text": "And we have taken those ideas and pushed them all into the open source." }, { "start": 2706.48, "end": 2707.12, "text": "Yeah, super cool." }, { "start": 2707.82, "end": 2708.72, "text": "Really exciting story." }, { "start": 2710.64, "end": 2719.3, "text": "I think many many cto listeners will will ask themselves like hey how can I can I can I use it really for my my production workloads?" }, { "start": 2719.36, "end": 2723.82, "text": "and then like quickly people shed a light on resilience." }, { "start": 2724.7, "end": 2725.36, "text": "what like?" }, { "start": 2726.06, "end": 2729.86, "text": "I guess that's like just a mother duck use case that like if one." }, { "start": 2730.64, "end": 2731.92, "text": "duck box dies, then" }, { "start": 2732.3, "end": 2732.48, "text": "yeah," }, { "start": 2733.02, "end": 2737.74, "text": "like there's replicas and there's like automatic restarting and auto scaling, etc." }, { "start": 2738.6, "end": 2739.82, "text": "And that's just not what you." }, { "start": 2740.22, "end": 2743.6, "text": "what you what you what you worry about, I guess," }, { "start": 2744.16, "end": 2747.28, "text": "we worry about resilience in a different way, not in terms of like." }, { "start": 2747.34, "end": 2752.54, "text": "if one dies, we restart it because we don't have a server right like." }, { "start": 2753.08, "end": 2754.68, "text": "we don't take care of that for you." }, { "start": 2755.26, "end": 2757.6, "text": "So you have to kind of figure that out yourself." }, { "start": 2758.34, "end": 2765.26, "text": "Obviously, there are solutions that you may use elsewhere in your tech stack that work for this as well, like Kubernetes and things like that." }, { "start": 2766.56, "end": 2769.62, "text": "But we do care a lot about stability of the system." }, { "start": 2771.62, "end": 2771.84, "text": "Okay." }, { "start": 2771.96, "end": 2780.16, "text": "So your overall one box resilience is super important and you don't want your process to die and no core dumps, etc." }, { "start": 2781.22, "end": 2783.36, "text": "Yeah, it has to keep keep running." }, { "start": 2783.94, "end": 2790.5, "text": "That's a. we have a an internal model which is a never give up never surrender from a galaxy quest if you've seen the movie." }, { "start": 2792.52, "end": 2798.28, "text": "And the idea is that we should always keep running like we should never crash or running to out of memory exceptions things like that." }, { "start": 2799.56, "end": 2803.88, "text": "And then like as long as your machine still has resources available, that'll keep crunching." }, { "start": 2804.4, "end": 2808.72, "text": "So That's also one of the things that we try is that your RAM shouldn't be your bottleneck." }, { "start": 2808.76, "end": 2826.84, "text": "It should be your hard disk and your disk space so that if you run to a scenario where your query doesn't fit into RAM, the system will like, like spill out intermediates to disk and like gracefully degrade performance while still finishing instead of like just popping up and saying like out of memory." }, { "start": 2828.84, "end": 2833.54, "text": "And that that's something that In a lot of cases, we actually do a lot better on than Spark." }, { "start": 2833.82, "end": 2834.6, "text": "There's queries that Dr." }, { "start": 2834.78, "end": 2846.8, "text": "B can run that Spark can't run, because if Spark has to gather too much data onto a single machine, it will just die, whereas we are much more resilient in that regard." }, { "start": 2848.04, "end": 2848.52, "text": "Okay, cool." }, { "start": 2849.66, "end": 2850.52, "text": "Super exciting story." }, { "start": 2851.06, "end": 2855.64, "text": "I still have a little exciting surprise for you while slowly coming to the end." }, { "start": 2857.18, "end": 2865.84, "text": "While like recently poking around the duck to be CLI accidentally typed install quark load quark." }, { "start": 2868.2, "end": 2876.4, "text": "And then I don't know what led me to that, but I figured out that there's a dot time machine feature in the quark library." }, { "start": 2877.44, "end": 2880.96, "text": "Like you can just use it and travel back in time physically." }, { "start": 2881.04, "end": 2883.98, "text": "So imagine we now use that." }, { "start": 2884.6, "end": 2887.98, "text": "Little feature to type time dot time machine." }, { "start": 2888.04, "end": 2911.64, "text": "twenty twelve and you know like can watch yourself back in the days nerding around twenty twelve on the CWI lab with the very first embedded analytics prototype and you then you have ten seconds to whisper something into young marks marks years what what what sentence would it be." }, { "start": 2913.74, "end": 2918.84, "text": "I honestly, I think just keep doing what you're doing because it's working out." }, { "start": 2919.56, "end": 2920.82, "text": "I think so." }, { "start": 2920.96, "end": 2944.38, "text": "There's many things that I could tell myself at that point, like many lessons learned, but I feel that in some sense it's crucial that you Learn them yourself by like hitting your like a head against the wall like many of the things that we did right inductee be I think we did rights because we did them wrong first in our like initial prototype." }, { "start": 2944.44, "end": 2955.3, "text": "like there was many sort of like things that we learned because we tried them with like Menetebe and it didn't work and like People were complaining about it and that then more like work." }, { "start": 2955.68, "end": 2969.78, "text": "and I think If you just tell yourself that and you don't have the same sort of the lessons, the pain that comes along with that, then maybe it doesn't register in the same way, I feel." }, { "start": 2969.92, "end": 2970.0, "text": "So" }, { "start": 2970.16, "end": 2973.34, "text": "you actually heated the pain in your forehead because you smashed it against the wall." }, { "start": 2974.98, "end": 2976.44, "text": "Yeah, we call that the head bashing." }, { "start": 2977.24, "end": 2978.48, "text": "You need some head bashing." }, { "start": 2978.62, "end": 2980.8, "text": "Otherwise, it doesn't quite stick around." }, { "start": 2981.98, "end": 2983.78, "text": "What is your last head bashing moment?" }, { "start": 2988.44, "end": 2994.08, "text": "It's usually CI related, like anything sort of build system, continuous integration, things like that." }, { "start": 2994.14, "end": 2995.5, "text": "Those are the head bashing moments." }, { "start": 2996.22, "end": 2998.46, "text": "So we call a head bashing moment." }, { "start": 2999.54, "end": 3009.06, "text": "Whenever there's something that you know you're going to solve, but you need to just keep on going through it, just do it, do it, do it." }, { "start": 3009.9, "end": 3013.96, "text": "We call it the head bashing moments, and I think it's kind of critical to have those." }, { "start": 3014.7, "end": 3020.42, "text": "So, um, it's a little, little detour question, but, um, I mean, you're, you're quite low level, right?" }, { "start": 3020.52, "end": 3023.54, "text": "You, you, you develop C plus plus, I guess day to day." }, { "start": 3024.14, "end": 3024.24, "text": "Yeah." }, { "start": 3024.28, "end": 3029.08, "text": "Um, and I would say you're like, ninety-nine percent geek, um, or nerd." }, { "start": 3029.12, "end": 3030.32, "text": "I don't know how you differentiate." }, { "start": 3030.36, "end": 3030.72, "text": "That's very" }, { "start": 3030.84, "end": 3031.34, "text": "fair to say." }, { "start": 3031.86, "end": 3035.34, "text": "I'll say a hundred percent, but I don't know where you got this one percent from." }, { "start": 3037.12, "end": 3037.6, "text": "How do you," }, { "start": 3037.86, "end": 3040.96, "text": "how do you see a development for your personal use cases?" }, { "start": 3041.24, "end": 3045.88, "text": "Um, like, do you use, I don't know, anything like code code or stuff like that?" }, { "start": 3045.94, "end": 3047.44, "text": "And how excited are you about it?" }, { "start": 3047.94, "end": 3051.36, "text": "Um, and, and yeah, how many problems does it solve for you?" }, { "start": 3052.12, "end": 3052.74, "text": "Honest answer." }, { "start": 3053.52, "end": 3055.76, "text": "so personally I don't use this this at all." }, { "start": 3055.94, "end": 3060.54, "text": "But I think that's also because of the nature of like where I'm working." }, { "start": 3060.64, "end": 3069.82, "text": "like a lot of the stuff that I'm doing it's very Very core very low level and the AI just it can't really help you there." }, { "start": 3070.28, "end": 3074.74, "text": "like there's very specific things I need to build and program that just haven't been done before." }, { "start": 3075.3, "end": 3077.62, "text": "that I feel doesn't really help you with." }, { "start": 3078.16, "end": 3082.44, "text": "I think it does very much help if you're doing like higher level stuff." }, { "start": 3082.54, "end": 3093.04, "text": "Like I think some use case where I've seen where you're like basically composing other libraries, things like that, like writing Python code that composed different libraries, I think then it can be very helpful." }, { "start": 3093.48, "end": 3098.3, "text": "But for the sort of like low level C++ programming, I haven't really found it that useful in general." }, { "start": 3099.06, "end": 3100.52, "text": "Okay, yeah, that's just a survey." }, { "start": 3102.24, "end": 3105.5, "text": "I tend to query again and again." }, { "start": 3106.84, "end": 3107.28, "text": "Thanks a lot." }, { "start": 3108.48, "end": 3119.04, "text": "Very nice nice talk and really looking forward to the moment when I can send my congrats whenever Google acquires you for fifty billion or something." }, { "start": 3120.54, "end": 3127.28, "text": "Let's wait for that and really crazy what you've built there and really, really fun to observe that." }, { "start": 3128.4, "end": 3130.12, "text": "Let's see where the time takes you." }, { "start": 3131.18, "end": 3131.8, "text": "Thank you very much." }, { "start": 3131.92, "end": 3132.58, "text": "Thank you for having me." }, { "start": 3133.36, "end": 3133.8, "text": "Thanks a lot." }, { "start": 3133.9, "end": 3134.38, "text": "Have a great day." }, { "start": 3134.74, "end": 3134.94, "text": "Bye." }, { "start": 3135.42, "end": 3135.72, "text": "Thank you." }, { "start": 3137.7, "end": 3139.9, "text": "Thank you for listening to the AlphaList podcast." }, { "start": 3140.14, "end": 3142.58, "text": "If you liked this episode, share it with friends." }, { "start": 3142.82, "end": 3143.7, "text": "I'm sure they love it too." }, { "start": 3144.02, "end": 3150.76, "text": "Make sure to subscribe so you can hear deep insights into technical leadership and technology trends as they become available." }, { "start": 3151.14, "end": 3158.32, "text": "Also, please tell us if there is a topic you would like to hear more about or a technical leader whose brain you would like us to pick." }, { "start": 3158.74, "end": 3165.62, "text": "AlphaList is all about helping CTOs getting access to the insights they need to make the best decisions for their company." }, { "start": 3165.98, "end": 3172.1, "text": "Please send us suggestions to cto at alphalist.com, send me a message on LinkedIn or Twitter." }, { "start": 3172.52, "end": 3176.88, "text": "After all, the more knowledge we bring to CTOs, the more growth we see in tech." }, { "start": 3177.18, "end": 3181.2, "text": "Or, as we say on Alphalist, accumulated knowledge to accelerate growth." }, { "start": 3181.76, "end": 3192.2, "text": "See you in the next episode." } ]