[
  {
    "start": 0.14,
    "end": 1.52,
    "text": "We're ready for this."
  },
  {
    "start": 1.8,
    "end": 2.82,
    "text": "This is our moment!"
  },
  {
    "start": 3.0,
    "end": 4.86,
    "text": "All right, this is our time in the sunshine."
  },
  {
    "start": 5.94,
    "end": 23.68,
    "text": "and all that old silly stuff about reducing testing to checks of output from functions... That stuff doesn't have a chance against AI."
  },
  {
    "start": 25.06,
    "end": 30.9,
    "text": "Machine learning models are not developed by a conscious, intentional human being."
  },
  {
    "start": 31.12,
    "end": 38.1,
    "text": "They're more like selected from dishes of bacteria, Petri dishes."
  },
  {
    "start": 39.4,
    "end": 44.32,
    "text": "We can't write a chapter on AI because we know that it's going to be at a date before the ink is dry right?"
  },
  {
    "start": 44.6,
    "end": 48.56,
    "text": "That...that's just the way things go in this silly business."
  },
  {
    "start": 49.82,
    "end": 52.8,
    "text": "We are being paid to learn stuff on behalf other people!"
  },
  {
    "start": 52.96,
    "end": 54.04,
    "text": "What better job than"
  },
  {
    "start": 54.12,
    "end": 54.46,
    "text": "THAT?!"
  },
  {
    "start": 55.16,
    "end": 61.04,
    "text": "Welcome to Software Testing Unleash, the podcast for people who want to build great software."
  },
  {
    "start": 61.44,
    "end": 68.04,
    "text": "And also a warm welcome to the Eurostar community because this episode was recorded at The Eurostat."
  },
  {
    "start": 68.14,
    "end": 69.92,
    "text": "twenty-twenty six in Oslo."
  },
  {
    "start": 70.32,
    "end": 76.58,
    "text": "I hope you enjoyed the conference as much as i did and maybe we will see each other at Eurastar."
  },
  {
    "start": 76.7,
    "end": 78.74,
    "text": "Twenty-Twenty seven in Copenhagen."
  },
  {
    "start": 79.28,
    "end": 82.14,
    "text": "My guest was the one and only Michael Bowden."
  },
  {
    "start": 82.5,
    "end": 85.46,
    "text": "We talked about testing AI as a magic box."
  },
  {
    "start": 85.96,
    "end": 96.2,
    "text": "Why you should not depend on Magic when something really matters, why simple checks are hopeless with language models and how these models fool us like a mentalist on stage?"
  },
  {
    "start": 96.4,
    "end": 98.74,
    "text": "This and more your here in this episode!"
  },
  {
    "start": 98.92,
    "end": 99.96,
    "text": "And now enjoy!"
  },
  {
    "start": 113.34,
    "end": 120.28,
    "text": "The final day on Eurostar, nearly the end."
  },
  {
    "start": 120.9,
    "end": 121.6,
    "text": "It's fading out"
  },
  {
    "start": 122.7,
    "end": 123.9,
    "text": "and"
  },
  {
    "start": 123.94,
    "end": 126.04,
    "text": "it was just a reveal next year in Copenhagen."
  },
  {
    "start": 127.2,
    "end": 127.96,
    "text": "Yeah I hope to be there too!"
  },
  {
    "start": 130.0,
    "end": 130.06,
    "text": "You"
  },
  {
    "start": 131.96,
    "end": 132.34,
    "text": "never know!"
  },
  {
    "start": 132.84,
    "end": 145.36,
    "text": "It is great that you are here at this show now And i saw your tutorial not in person but in the abstract because of me being in the provical committee Testing Magic"
  },
  {
    "start": 147.32,
    "end": 147.48,
    "text": "Box."
  },
  {
    "start": 147.86,
    "end": 149.0,
    "text": "Yeah,"
  },
  {
    "start": 149.04,
    "end": 151.2,
    "text": "yeah and I want to elaborate it a little bit."
  },
  {
    "start": 153.02,
    "end": 154.26,
    "text": "what's behind that for you?"
  },
  {
    "start": 155.74,
    "end": 167.38,
    "text": "It started when we were writing our book Taking testing seriously the rapid software testing approach which We've been running For The last thirty years."
  },
  {
    "start": 167.62,
    "end": 172.18,
    "text": "yes how long it takes but also really Seriously for the last five years."
  },
  {
    "start": 172.26,
    "end": 187.9,
    "text": "we get right close to the end the process of writing the book and We've got to write a chapter on AI Except we can't read it chapter on a I because we know that It's gonna be out-of-date before the ink is dry."
  },
  {
    "start": 188.02,
    "end": 188.98,
    "text": "Yeah, right?"
  },
  {
    "start": 189.06,
    "end": 192.8,
    "text": "That's just The way things go in this silly business."
  },
  {
    "start": 192.84,
    "end": 198.52,
    "text": "yeah So we wrote a chapter On Magic cuz."
  },
  {
    "start": 198.58,
    "end": 214.5,
    "text": "we wanted they did the Chapter To Be Future proof to some degree So that prompted us to think about how we think about magic AI being a Magic Box."
  },
  {
    "start": 215.34,
    "end": 218.42,
    "text": "so first The zeroth principle here."
  },
  {
    "start": 218.66,
    "end": 219.46,
    "text": "Yeah, is it a box?"
  },
  {
    "start": 219.6,
    "end": 226.38,
    "text": "Is magical To the degree some but some boxes are strongly magical Some boxes or weakly Magical."
  },
  {
    "start": 226.56,
    "end": 235.98,
    "text": "yeah A box is magical to the degree That its behavior has neither controlled nor understood Nor otherwise known-to be safe."
  },
  {
    "start": 236.26,
    "end": 236.4,
    "text": "yet."
  },
  {
    "start": 237.04,
    "end": 238.6,
    "text": "Okay, now what does that mean?"
  },
  {
    "start": 238.76,
    "end": 252.02,
    "text": "That means for example the Python interpreter is a magic box to me because I don't know how it works and they...I mean philosophically conceptually."
  },
  {
    "start": 252.16,
    "end": 254.9,
    "text": "I know how work but i don't have the details of How Works."
  },
  {
    "start": 256.12,
    "end": 262.019,
    "text": "So in that sense It's fairly strongly magical To Me Controlled."
  },
  {
    "start": 262.5,
    "end": 266.02,
    "text": "Well, I guess i can control its behavior to some degree."
  },
  {
    "start": 266.76,
    "end": 271.44,
    "text": "So that makes it a little less strongly magical otherwise known to be safe."
  },
  {
    "start": 271.56,
    "end": 292.72,
    "text": "well in that sense although...I don't control or understand completely that thoroughly there's worldwide community of python users out here and they're also people for whom the internals They do understand it and they do control."
  },
  {
    "start": 292.82,
    "end": 292.88,
    "text": "It."
  },
  {
    "start": 293.26,
    "end": 305.5,
    "text": "so we have social proof that its safe And we have some kind of belief that if it were Fundamentally radically unsafe then the people responsible for curating in maintaining, it Would fix it."
  },
  {
    "start": 305.7,
    "end": 306.64,
    "text": "yeah?"
  },
  {
    "start": 307.2,
    "end": 311.7,
    "text": "I mean from time to time That's what happens We find out ways in which the magic isn't working or it."
  },
  {
    "start": 312.76,
    "end": 333.88,
    "text": "you know What we're dealing with at the moment is that AI in the broadest sense Mostly based on things like machine learning models."
  },
  {
    "start": 334.62,
    "end": 353.62,
    "text": "These things are constructed without a human Specifying, so it's unlike software regular software and that way Machine-learning models or not developed by a conscious, intentional human being."
  },
  {
    "start": 353.84,
    "end": 365.68,
    "text": "They're more like selected from dishes of bacteria, Petri dishes and we see one that kills off certain kind of nasty bacterium."
  },
  {
    "start": 365.72,
    "end": 368.52,
    "text": "when you say That one looks the best so pull it out."
  },
  {
    "start": 368.6,
    "end": 370.32,
    "text": "Yeah This is what I want."
  },
  {
    "start": 370.6,
    "end": 378.76,
    "text": "Okay poses special kinds of problems and risks because we don't know exactly how it works."
  },
  {
    "start": 378.82,
    "end": 383.22,
    "text": "It's like trying to figure out what good antibiotics were."
  },
  {
    "start": 383.28,
    "end": 391.44,
    "text": "before we had the capacity to do genetic analysis, DNA analysis in all these things."
  },
  {
    "start": 391.5,
    "end": 399.38,
    "text": "so now with proteomics and genomics We can sort of get a sense of The mechanisms that are driving a particular antibiotic."
  },
  {
    "start": 399.46,
    "end": 400.22,
    "text": "And now with CRISPR?"
  },
  {
    "start": 408.42,
    "end": 412.4,
    "text": "All that since these things are strongly magical boxes."
  },
  {
    "start": 413.26,
    "end": 414.54,
    "text": "Nobody knows how they work."
  },
  {
    "start": 414.96,
    "end": 416.64,
    "text": "Nobody has a source code for them."
  },
  {
    "start": 417.36,
    "end": 422.48,
    "text": "Even if we did have the source code, there's a certain amount of A stochasticness."
  },
  {
    "start": 426.56,
    "end": 428.9,
    "text": "Stochacity There you go."
  },
  {
    "start": 430.36,
    "end": 435.44,
    "text": "When your out having fun with Finnish people The night before Certain things get hard to say."
  },
  {
    "start": 439.26,
    "end": 444.78,
    "text": "These are strongly magical waxes which leads to a set of principles."
  },
  {
    "start": 444.92,
    "end": 456.88,
    "text": "Number one, if something matters very much you probably shouldn't depend on magic to get it."
  },
  {
    "start": 457.82,
    "end": 462.86,
    "text": "All right don't rely on Magic to obtain that."
  },
  {
    "start": 463.76,
    "end": 475.24,
    "text": "If Something Matters Very Much and You Actually Have No Other Way Of Obtaining It Well You Might As Well Try Magic But it behooves you to minimize your reliance on magic."
  },
  {
    "start": 475.6,
    "end": 477.82,
    "text": "You don't want to depend on it."
  },
  {
    "start": 479.6,
    "end": 489.64,
    "text": "All right, if something matters very much this is number three If something matters Very Much and you are using Magic to get it Responsibility demands that you test It"
  },
  {
    "start": 489.92,
    "end": 490.06,
    "text": "yeah?"
  },
  {
    "start": 491.14,
    "end": 504.06,
    "text": "If it matters too yourself or two other people especially if you have a duty of care To those People as we do people in the software business We have a duty of care to our clients and their clients, their customers."
  },
  {
    "start": 505.82,
    "end": 506.72,
    "text": "So we must test it."
  },
  {
    "start": 509.32,
    "end": 515.299,
    "text": "if we are going to test it... ...we want to maximize testability."
  },
  {
    "start": 516.4,
    "end": 527.18,
    "text": "now there's a lot of test ability in the traditional sense that is unavailable to us because once again.. ..we don't have the source code can instrument the sourcecode itself To make it more testable."
  },
  {
    "start": 527.26,
    "end": 540.72,
    "text": "but they're some things we could do to monitor inputs and outputs closely To understand the training data that went into these things, to understand the guardrails around them on both sides."
  },
  {
    "start": 542.14,
    "end": 547.46,
    "text": "To understand factors it might make a difference like the context window and its size."
  },
  {
    "start": 547.62,
    "end": 548.12,
    "text": "what's in it?"
  },
  {
    "start": 548.32,
    "end": 549.32,
    "text": "What got retrieved?"
  },
  {
    "start": 549.5,
    "end": 550.82,
    "text": "how we're accessing databases?"
  },
  {
    "start": 551.42,
    "end": 553.86,
    "text": "How are making calls for services outside world?"
  },
  {
    "start": 554.0,
    "end": 554.68,
    "text": "all those thing."
  },
  {
    "start": 556.14,
    "end": 560.68,
    "text": "If we're aware of those and we can control, those and observe Those and understand the behavior."
  },
  {
    "start": 560.74,
    "end": 565.2,
    "text": "Of those things then We're reducing the magic NASA magic box to some degree."
  },
  {
    "start": 565.84,
    "end": 572.0,
    "text": "so part of it is There's a GPT in the middle of a Magic Box."
  },
  {
    "start": 572.82,
    "end": 582.7,
    "text": "maybe we Can Put little windows In The bigger box Or Maybe you can put probes in It or a little camera inside."
  },
  {
    "start": 583.0,
    "end": 583.56,
    "text": "So all that stuff."
  },
  {
    "start": 586.2,
    "end": 588.62,
    "text": "So that's the fourth principle maximize testability."
  },
  {
    "start": 589.9,
    "end": 598.14,
    "text": "and Finally if you can't test it or you can test it thoroughly enough Then there are other things."
  },
  {
    "start": 598.2,
    "end": 600.5,
    "text": "You've got to do, you've gotta disclaim It."
  },
  {
    "start": 602.92,
    "end": 606.96,
    "text": "But even if you disclaim it And If you were foisting at on another people?"
  },
  {
    "start": 607.06,
    "end": 609.2,
    "text": "You have to accept responsibility for that."
  },
  {
    "start": 609.34,
    "end": 621.88,
    "text": "Yeah so we're into some pretty interesting areas for testing and for responsibility in ethics and for thinking about how we go about the business of understanding complicated stuff."
  },
  {
    "start": 624.34,
    "end": 631.52,
    "text": "Fortunately, uh... The kind of testing that we've always talked about is ready-for this"
  },
  {
    "start": 631.68,
    "end": 632.52,
    "text": "because"
  },
  {
    "start": 632.86,
    "end": 634.74,
    "text": "it's a kind of test thing We've always talked about."
  },
  {
    "start": 635.28,
    "end": 649.08,
    "text": "so testing isn't investigation with special focus on looking for problems testing for us Evaluating a product by learning about it through experiencing and exploring and experimenting."
  },
  {
    "start": 649.2,
    "end": 649.94,
    "text": "Yeah, yeah"
  },
  {
    "start": 650.78,
    "end": 652.02,
    "text": "We're ready for this!"
  },
  {
    "start": 652.38,
    "end": 653.4,
    "text": "This is our moment!"
  },
  {
    "start": 653.72,
    "end": 653.9,
    "text": "Alright?"
  },
  {
    "start": 653.94,
    "end": 672.92,
    "text": "This was our time in the sunshine And all that old silly stuff About reducing testing to checks of output from functions That's tough Doesn't have a chance?"
  },
  {
    "start": 673.02,
    "end": 674.24,
    "text": "hmm against AI."
  },
  {
    "start": 674.5,
    "end": 682.06,
    "text": "Yeah It's useful thing mostly for developers to do with deterministic software that they understand They can instrument."
  },
  {
    "start": 683.24,
    "end": 698.34,
    "text": "when it comes down to LLMs Its hopeless, yeah Now we can apply back kind of work to some degree usefully too all the bits of an AI based system That are good old-fashioned software."
  },
  {
    "start": 698.38,
    "end": 705.56,
    "text": "as usual We can't Apply to the LLM itself."
  },
  {
    "start": 705.76,
    "end": 710.64,
    "text": "Yeah, there are certain things that are checkable within a big LL M system."
  },
  {
    "start": 712.18,
    "end": 712.82,
    "text": "It's interesting."
  },
  {
    "start": 713.42,
    "end": 715.98,
    "text": "what comes to my mind is some clients of me."
  },
  {
    "start": 716.48,
    "end": 727.14,
    "text": "they have also this way going away from checking stuff with with this test cases, and now when the AI is coming they rethink what are my risks?"
  },
  {
    "start": 727.36,
    "end": 728.74,
    "text": "What quality do I want?"
  },
  {
    "start": 729.04,
    "end": 731.78,
    "text": "these old questions we should have asked a long time."
  },
  {
    "start": 732.14,
    "end": 733.92,
    "text": "They're not getting aware of that."
  },
  {
    "start": 733.96,
    "end": 734.04,
    "text": "We've"
  },
  {
    "start": 734.08,
    "end": 736.36,
    "text": "been asking those question forever right!"
  },
  {
    "start": 736.44,
    "end": 739.62,
    "text": "Is it...what's important?"
  },
  {
    "start": 739.74,
    "end": 747.22,
    "text": "Yeah help you understand how to run your business, and now... How can I help with that?"
  },
  {
    "start": 747.42,
    "end": 748.7,
    "text": "Yeah."
  },
  {
    "start": 748.96,
    "end": 751.94,
    "text": "What are the things they're gonna threaten value of product for people who matter?"
  },
  {
    "start": 752.4,
    "end": 752.64,
    "text": "To YOU!"
  },
  {
    "start": 752.88,
    "end": 754.14,
    "text": "And do the PEOPLE WHO MATTER TO"
  },
  {
    "start": 754.24,
    "end": 754.32,
    "text": "YOU?!"
  },
  {
    "start": 756.16,
    "end": 756.24,
    "text": "Yeah."
  },
  {
    "start": 756.46,
    "end": 758.0,
    "text": "That's it."
  },
  {
    "start": 758.36,
    "end": 760.74,
    "text": "It is much more about this conversation…."
  },
  {
    "start": 760.82,
    "end": 761.8,
    "text": "…about this stuff…."
  },
  {
    "start": 762.02,
    "end": 763.9,
    "text": "...and get this information out."
  },
  {
    "start": 764.18,
    "end": 767.26,
    "text": "after then we can deal with in any way."
  },
  {
    "start": 768.16,
    "end": 780.72,
    "text": "Plus.. I gotta add.... testing of language models recently and two things make it fun."
  },
  {
    "start": 782.18,
    "end": 801.8,
    "text": "It's iterative, spiraling in or another way to put it is like peeling an onion The more you peel the more there seems to be a layer down here Opening up the onion."
  },
  {
    "start": 801.94,
    "end": 804.98,
    "text": "Oh, there's another layer here and then he says nature is witty."
  },
  {
    "start": 806.36,
    "end": 807.4,
    "text": "Yeah Which is really cool?"
  },
  {
    "start": 808.68,
    "end": 816.5,
    "text": "So what I'm seeing Is patterns of inconsistency an unreliability in these things."
  },
  {
    "start": 817.16,
    "end": 819.26,
    "text": "that among other thing it's been occurring to me."
  },
  {
    "start": 820.32,
    "end": 829.32,
    "text": "Maybe this is at the bottom of what's going on when people say Claude code used to be good and lately it looks like."
  },
  {
    "start": 829.36,
    "end": 830.36,
    "text": "It's kind of sucking."
  },
  {
    "start": 830.5,
    "end": 834.56,
    "text": "well I'm seeing behavior at the very lowest levels in my querying the model directly."
  },
  {
    "start": 834.6,
    "end": 841.56,
    "text": "yeah, i've seen behaviors that Don't exactly Explain why?"
  },
  {
    "start": 841.66,
    "end": 843.06,
    "text": "That's happening or how its happening."
  },
  {
    "start": 843.84,
    "end": 845.02,
    "text": "but boy can I ever see that."
  },
  {
    "start": 845.1,
    "end": 851.62,
    "text": "play out the differences between three five clod three hi-ku Yeah, three five an hike."
  },
  {
    "start": 851.68,
    "end": 854.66,
    "text": "who four five are really, really interesting."
  },
  {
    "start": 854.74,
    "end": 865.24,
    "text": "Four or five tends to go off the rails in the experience that we've been doing a lot more quickly and a lot bizarrely than three-five did."
  },
  {
    "start": 865.8,
    "end": 867.94,
    "text": "but they all do it!"
  },
  {
    "start": 873.18,
    "end": 878.26,
    "text": "Some people saw this... I'm on The Verge of publishing some observations."
  },
  {
    "start": 880.6,
    "end": 882.4,
    "text": "There's so many and they're so weird."
  },
  {
    "start": 882.66,
    "end": 885.76,
    "text": "Yeah, yeah And I want to do a really good job of this."
  },
  {
    "start": 887.04,
    "end": 910.82,
    "text": "But the other the nice thing about it is we've got an experimental framework now that allows us To run these kinds of experiments over and over again and see patterns and see weird outliers in The refinement of the experiment allows us to see things in different ways and to look for different things, have software help us pick those thing up."
  },
  {
    "start": 911.02,
    "end": 911.94,
    "text": "That's lots of fun too!"
  },
  {
    "start": 913.2,
    "end": 916.48,
    "text": "So I'm having a good time with that kind stuff... And it is fun!"
  },
  {
    "start": 917.1,
    "end": 919.6,
    "text": "Testing has always been fun if you do right?"
  },
  {
    "start": 920.96,
    "end": 924.16,
    "text": "If not doing it right then its not fun You"
  },
  {
    "start": 924.22,
    "end": 924.34,
    "text": "know?"
  },
  {
    "start": 925.44,
    "end": 926.96,
    "text": "Thats what i love about my job."
  },
  {
    "start": 927.1,
    "end": 928.38,
    "text": "so since We're"
  },
  {
    "start": 929.22,
    "end": 931.96,
    "text": "being paid to learn stuff on behalf of other people."
  },
  {
    "start": 932.12,
    "end": 933.48,
    "text": "What's a better job than that?"
  },
  {
    "start": 935.94,
    "end": 951.28,
    "text": "So you are dealing with a lot AI stuff now, testing this and exploring it... And what do think is the problem for most testers?"
  },
  {
    "start": 952.28,
    "end": 958.22,
    "text": "they are complaining at Talks here or at Conversations that it's not testable."
  },
  {
    "start": 958.34,
    "end": 959.26,
    "text": "So what this?"
  },
  {
    "start": 960.16,
    "end": 961.88,
    "text": "What shift is needed for them"
  },
  {
    "start": 962.44,
    "end": 962.52,
    "text": "to?"
  },
  {
    "start": 963.04,
    "end": 963.94,
    "text": "oh man a few things."
  },
  {
    "start": 964.86,
    "end": 973.46,
    "text": "It keeps going back To This Thing That we have been refining For many years and now We're the point where we think we got it."
  },
  {
    "start": 973.56,
    "end": 980.8,
    "text": "testing is As I said evaluating your product by learning about it through experiencing exploring an experimenting."
  },
  {
    "start": 981.66,
    "end": 985.48,
    "text": "Now, that makes Anything testable."
  },
  {
    "start": 985.74,
    "end": 985.86,
    "text": "Yeah,"
  },
  {
    "start": 986.54,
    "end": 989.3,
    "text": "we can always experience something because get experienced with it."
  },
  {
    "start": 990.2,
    "end": 994.32,
    "text": "We can always Evaluate it by learning about it mm-hmm?"
  },
  {
    "start": 995.04,
    "end": 1006.62,
    "text": "We can also perform experiments within and that process is fundamentally an exploratory process cuz We need to be in control of it right."
  },
  {
    "start": 1007.58,
    "end": 1018.52,
    "text": "there is no preset a predetermined way Learning about something that you don't know about yet and the to agree nobody knows about right."
  },
  {
    "start": 1019.0,
    "end": 1022.02,
    "text": "so That's fun."
  },
  {
    "start": 1022.06,
    "end": 1023.6,
    "text": "Yeah, uh-uh."
  },
  {
    "start": 1024.14,
    "end": 1036.859,
    "text": "That is see at the excitement of our idea an observation And a procedure that allows you to enact it Prompting some new discovery."
  },
  {
    "start": 1037.14,
    "end": 1049.96,
    "text": "yeah when we see new discovery that prompts a new idea a new hypothesis A new procedure for experimenting with something, or a way of going deeper on the experimental process you've got."
  },
  {
    "start": 1050.04,
    "end": 1052.76,
    "text": "Yeah For what?"
  },
  {
    "start": 1052.8,
    "end": 1074.86,
    "text": "You have new patterns knew New observations of things like bias and behavior And also recognition that The least little thing Can cause no change whatsoever or a completely bizarre new thing."
  },
  {
    "start": 1075.52,
    "end": 1077.84,
    "text": "And it's exciting, and fun to do that!"
  },
  {
    "start": 1078.72,
    "end": 1104.6,
    "text": "I think this is great perspective because when we see there are these quotes like magic box technology as long as the magic disappears Decades in the past and I think we as a testers when we are going exploring this stuff And then the magic maybe little bit disappears, but we have fun on this process."
  },
  {
    "start": 1105.24,
    "end": 1108.88,
    "text": "Well You know i'm a big fan of Magic."
  },
  {
    "start": 1109.44,
    "end": 1112.26,
    "text": "whenever there's an opportunity to see him a magic show?"
  },
  {
    "start": 1112.72,
    "end": 1113.06,
    "text": "Take it."
  },
  {
    "start": 1113.48,
    "end": 1116.58,
    "text": "and one year There was a testing conference in Las Vegas."
  },
  {
    "start": 1118.04,
    "end": 1129.08,
    "text": "So I went to see pen and teller now Penn and tell her or my favorite magician because So often they will tell you exactly how a trick is done."
  },
  {
    "start": 1129.24,
    "end": 1139.1,
    "text": "Yeah, okay They do a wonderful routine with the cups and balls where they show You?"
  },
  {
    "start": 1139.76,
    "end": 1140.48,
    "text": "Exactly How."
  },
  {
    "start": 1140.98,
    "end": 1141.62,
    "text": "that's The one With."
  },
  {
    "start": 1141.76,
    "end": 1150.28,
    "text": "you know There's a couple of three Cups And then Up to Three Balls I think and they all move around and they All."
  },
  {
    "start": 1150.92,
    "end": 1152.52,
    "text": "they all seem To appear and disappear in so on."
  },
  {
    "start": 1153.26,
    "end": 1160.1,
    "text": "well the coolest thing about Their delivery of that trick is they do it with clear plastic cups."
  },
  {
    "start": 1160.28,
    "end": 1160.42,
    "text": "Yeah,"
  },
  {
    "start": 1162.04,
    "end": 1162.3,
    "text": "okay"
  },
  {
    "start": 1163.26,
    "end": 1165.82,
    "text": "So you get to see exactly how its being done?"
  },
  {
    "start": 1166.44,
    "end": 1175.38,
    "text": "But as a consequence again You get to See How Clever Magicians Are And You Get To See How Easy It Is For Us To Be Fooled."
  },
  {
    "start": 1176.02,
    "end": 1188.18,
    "text": "You Get The See the Specific Mechanisms Of How We're Fooled and That Defends You Against Being Fooled from being fooled in circumstances where there's another trick that they're doing."
  },
  {
    "start": 1190.16,
    "end": 1202.22,
    "text": "What it means is you can start thinking, in terms of human ingenuity... In terms what appears to be magical and that is a deeply humanizing kind thing right?"
  },
  {
    "start": 1204.14,
    "end": 1209.68,
    "text": "So the same with particular piece technology."
  },
  {
    "start": 1209.94,
    "end": 1218.88,
    "text": "if I understand how technologies like at work Then I can understand for instance with a large language model."
  },
  {
    "start": 1219.06,
    "end": 1228.84,
    "text": "I Can understand the basic principles behind its operation, which is basically to string one Dang token after another."
  },
  {
    "start": 1230.1,
    "end": 1235.02,
    "text": "But I can also from a psychological perspective because you look at that too."
  },
  {
    "start": 1235.84,
    "end": 1240.84,
    "text": "i can Look at how patterns of text fool us into believing it."
  },
  {
    "start": 1240.88,
    "end": 1247.48,
    "text": "there's insight here when they're isn't such insight that is there comes from us."
  },
  {
    "start": 1248.96,
    "end": 1264.28,
    "text": "So it's not the LLM saying anything interesting or profound, It is our effort to interpret and make sense of it causing neurons in our heads helping think about things we had never thought before."
  },
  {
    "start": 1265.02,
    "end": 1274.68,
    "text": "But what this allows for credit to ourselves, to give a credit to humans instead of giving credit the machine."
  },
  {
    "start": 1278.36,
    "end": 1281.8,
    "text": "There's another wonderful essay which I try and recommend for people."
  },
  {
    "start": 1282.02,
    "end": 1283.94,
    "text": "that is three or four them."
  },
  {
    "start": 1284.78,
    "end": 1290.98,
    "text": "one is Stephen Wolframs description what his chat GPT doing in."
  },
  {
    "start": 1291.08,
    "end": 1291.78,
    "text": "Why Does It Work?"
  },
  {
    "start": 1293.56,
    "end": 1302.36,
    "text": "That explains the mechanistic aspects the GPT's process."
  },
  {
    "start": 1303.22,
    "end": 1307.6,
    "text": "Very cool, but why does it work and how is that?"
  },
  {
    "start": 1307.68,
    "end": 1314.92,
    "text": "Work has another dimension That's a psychological dimension And that's really beautifully covered in an essay by guy named Balder Bjarneson."
  },
  {
    "start": 1317.34,
    "end": 1330.04,
    "text": "The title of that I say Is the large language mentalist effect How something like how Large-language models replicate the mechanisms Of the psychics con."
  },
  {
    "start": 1332.24,
    "end": 1333.08,
    "text": "And how does that work?"
  },
  {
    "start": 1334.92,
    "end": 1339.74,
    "text": "When people go see a cold reader, card reader, palm reader tarot card reader."
  },
  {
    "start": 1342.0,
    "end": 1344.12,
    "text": "Mentalist by the way."
  },
  {
    "start": 1344.78,
    "end": 1349.6,
    "text": "I just have to say thanks Alan Richardson who is on The Program Committee."
  },
  {
    "start": 1349.64,
    "end": 1357.68,
    "text": "That Year in twenty thirteen one of the People On the podium when i was the program chair Was A mentalist."
  },
  {
    "start": 1358.7,
    "end": 1363.94,
    "text": "So In Twenty Thirteen We were Ready for you know Preparing to deal with this moment."
  },
  {
    "start": 1364.08,
    "end": 1367.36,
    "text": "Yeah, but anyway how they work is?"
  },
  {
    "start": 1367.62,
    "end": 1377.98,
    "text": "They say something Quickly looking for a hit And I give this example in classes sometimes."
  },
  {
    "start": 1379.28,
    "end": 1382.1,
    "text": "now i'd like you to think back till last week."
  },
  {
    "start": 1384.88,
    "end": 1386.56,
    "text": "Last week was it fairly well."
  },
  {
    "start": 1386.68,
    "end": 1388.02,
    "text": "It's just little busy for you."
  },
  {
    "start": 1388.1,
    "end": 1396.34,
    "text": "of course You're preparing for Eurostar and there was that technological complication at some point."
  },
  {
    "start": 1396.4,
    "end": 1409.34,
    "text": "I'm thinking Wednesday or Thursday, there was some piece of gear and software that you had that frustrated for about half an hour to go back fix this thing?"
  },
  {
    "start": 1409.86,
    "end": 1411.0,
    "text": "You're remembering it right?"
  },
  {
    "start": 1415.16,
    "end": 1418.04,
    "text": "It's a productivity software."
  },
  {
    "start": 1419.12,
    "end": 1420.44,
    "text": "Now it doesn't matter whether I'm right or not."
  },
  {
    "start": 1420.5,
    "end": 1421.68,
    "text": "what's going on in your head, right?"
  },
  {
    "start": 1421.76,
    "end": 1429.92,
    "text": "Yeah is you're trying to stick that onto an event and That is the magic of of GPT says Balder points out in this."
  },
  {
    "start": 1430.2,
    "end": 1434.66,
    "text": "Yes really I was annoyed by his yeah essay here."
  },
  {
    "start": 1434.7,
    "end": 1438.24,
    "text": "Here's why I Wanted to do that essay."
  },
  {
    "start": 1438.7,
    "end": 1439.84,
    "text": "as one reason I was right."
  },
  {
    "start": 1439.96,
    "end": 1443.8,
    "text": "The other reason I thought if he did such a good job there i looked at him."
  },
  {
    "start": 1443.98,
    "end": 1445.6,
    "text": "I could never do something that good."
  },
  {
    "start": 1446.08,
    "end": 1455.72,
    "text": "so I recommend to testers who are working with these things understand how they work mechanistically Understand the effect."
  },
  {
    "start": 1455.76,
    "end": 1460.2,
    "text": "They have on people's minds and that defends us against being fooled by them."
  },
  {
    "start": 1460.9,
    "end": 1462.52,
    "text": "Bring that to our clients"
  },
  {
    "start": 1463.06,
    "end": 1463.12,
    "text": "Hmm,"
  },
  {
    "start": 1464.32,
    "end": 1480.9,
    "text": "and then say all right We've got a figure out whether this thing is doing a good job Whether it is satisfactorily going to provide us with consistent or reliable results, sometimes reliability doesn't matter."
  },
  {
    "start": 1481.54,
    "end": 1490.84,
    "text": "If the job for a GPT is to write a funny poem about a co-worker for The Christmas Party no problem!"
  },
  {
    "start": 1490.92,
    "end": 1494.9,
    "text": "No risk because if it produces one that you don't like he can throw out."
  },
  {
    "start": 1495.02,
    "end": 1498.88,
    "text": "yeah...if your body's giving medical advice big risks."
  },
  {
    "start": 1499.88,
    "end": 1505.86,
    "text": "so maybe we want to be cautious say and not say."
  },
  {
    "start": 1506.66,
    "end": 1519.6,
    "text": "And maybe it might be a good idea just to avoid using... Magical technology at all in places where the risk is elevated."
  },
  {
    "start": 1519.88,
    "end": 1523.16,
    "text": "Yeah, we're things are on the line when something matters very much yeah"
  },
  {
    "start": 1524.12,
    "end": 1527.9,
    "text": "And that's a principles that I saw true for years."
  },
  {
    "start": 1528.16,
    "end": 1530.24,
    "text": "but now much more important absolutely"
  },
  {
    "start": 1530.62,
    "end": 1531.6,
    "text": "Absolutely so."
  },
  {
    "start": 1531.76,
    "end": 1537.96,
    "text": "understand the technology engage with it encounter it Use tools."
  },
  {
    "start": 1539.34,
    "end": 1540.32,
    "text": "We have to do so much."
  },
  {
    "start": 1540.38,
    "end": 1547.3,
    "text": "statistical analysis of these things said anything other than a total first hurdle test."
  },
  {
    "start": 1547.98,
    "end": 1559.9,
    "text": "We're gonna have to use tools to generate large amounts of requests and responses, but we can use tools that help us induce variation into the request."
  },
  {
    "start": 1559.98,
    "end": 1581.36,
    "text": "A great application for an LLM ironically is to have it generate zillions of variations on the same question as in LLMs Because the introduction of a single word, or the change in a single words can dramatically affect output."
  },
  {
    "start": 1581.76,
    "end": 1592.5,
    "text": "What we want to see is how dramatically and often... And what degree might this completely invert meaning that people may be asking about?"
  },
  {
    "start": 1594.16,
    "end": 1595.04,
    "text": "So great insights."
  },
  {
    "start": 1595.28,
    "end": 1596.54,
    "text": "Thank you very much, Michael."
  },
  {
    "start": 1597.6,
    "end": 1604.7,
    "text": "I could talk with you for hours now But we have to continue this on another episode too."
  },
  {
    "start": 1605.42,
    "end": 1608.64,
    "text": "I would absolutely love that and anytime"
  },
  {
    "start": 1609.08,
    "end": 1609.32,
    "text": "Great!"
  },
  {
    "start": 1609.42,
    "end": 1611.8,
    "text": "And i think you gave so many tips here for testers."
  },
  {
    "start": 1613.0,
    "end": 1614.34,
    "text": "We have to reflect about that."
  },
  {
    "start": 1614.52,
    "end": 1616.52,
    "text": "Read the book, maybe get into paper."
  },
  {
    "start": 1616.9,
    "end": 1620.98,
    "text": "rethink what is magic and how can we deal with this problem?"
  },
  {
    "start": 1621.16,
    "end": 1622.04,
    "text": "The risks of it..."
  },
  {
    "start": 1623.6,
    "end": 1624.32,
    "text": "Magic is fun!"
  },
  {
    "start": 1627.4,
    "end": 1630.06,
    "text": "This one topic for today's episode."
  },
  {
    "start": 1630.4,
    "end": 1632.16,
    "text": "Thank you so much for being here on our show"
  },
  {
    "start": 1632.48,
    "end": 1634.34,
    "text": "My pleasure."
  }
]