[
  {
    "start": 0.22,
    "end": 7.88,
    "text": "Welcome to Software Testing Unleashed, the podcast for testers developers and all software people who want to create best quality"
  },
  {
    "start": 8.0,
    "end": 8.42,
    "text": "software."
  },
  {
    "start": 8.78,
    "end": 10.18,
    "text": "My name is Richie."
  },
  {
    "start": 10.24,
    "end": 18.74,
    "text": "I'm a software-quality coach author & keynote speaker And happy to bring you an episode live from The Hustaf Conference in Budapest."
  },
  {
    "start": 19.24,
    "end": 22.4,
    "text": "Hustef is one of the greatest conferences I've ever been to."
  },
  {
    "start": 22.6,
    "end": 24.62,
    "text": "Top quality talks, amazing people"
  },
  {
    "start": 25.02,
    "end": 25.68,
    "text": "and that unique"
  },
  {
    "start": 25.76,
    "end": 30.48,
    "text": "mix of friendly and intense atmosphere you only find at truly great events."
  },
  {
    "start": 31.22,
    "end": 32.619,
    "text": "If you have every get a chance."
  },
  {
    "start": 32.78,
    "end": 35.08,
    "text": "go there seriously You will love it."
  },
  {
    "start": 35.8,
    "end": 38.06,
    "text": "My guest today is Simon Stewart."
  },
  {
    "start": 38.66,
    "end": 41.12,
    "text": "Simon is professional software developer"
  },
  {
    "start": 41.7,
    "end": 42.56,
    "text": "since decades"
  },
  {
    "start": 43.0,
    "end": 46.58,
    "text": "And for more about ten years he was lead in Selenium project."
  },
  {
    "start": 47.04,
    "end": 47.98,
    "text": "We talked about"
  },
  {
    "start": 48.22,
    "end": 48.6,
    "text": "flaky"
  },
  {
    "start": 48.7,
    "end": 55.14,
    "text": "tests, we clarified what flaky test are and how to deal with them in our pipelines &"
  },
  {
    "start": 55.32,
    "end": 55.9,
    "text": "projects."
  },
  {
    "start": 56.24,
    "end": 58.24,
    "text": "And now enjoy the episode!"
  },
  {
    "start": 69.86,
    "end": 87.86,
    "text": "you brought such a great content."
  },
  {
    "start": 88.56,
    "end": 91.52,
    "text": "They said, okay that must be in the podcast here for the community."
  },
  {
    "start": 92.06,
    "end": 95.34,
    "text": "and we are here at Hustev, our fifteenth anniversary."
  },
  {
    "start": 95.7,
    "end": 95.8,
    "text": "What"
  },
  {
    "start": 95.84,
    "end": 97.0,
    "text": "a pleasure it is to be here."
  },
  {
    "start": 97.04,
    "end": 99.1,
    "text": "Yeah It's like a great event I liked so much."
  },
  {
    "start": 99.18,
    "end": 99.42,
    "text": "Yes"
  },
  {
    "start": 100.08,
    "end": 101.06,
    "text": "Fantastic event."
  },
  {
    "start": 101.98,
    "end": 106.6,
    "text": "And your topic yesterday was flaky tests"
  },
  {
    "start": 107.24,
    "end": 109.26,
    "text": "CI and The Great Flaky Test Adventure."
  },
  {
    "start": 111.74,
    "end": 119.12,
    "text": "We often don't care about flaky test because to get rid of and we have anger about it,"
  },
  {
    "start": 119.24,
    "end": 119.34,
    "text": "but"
  },
  {
    "start": 120.14,
    "end": 120.52,
    "text": "that's all."
  },
  {
    "start": 120.78,
    "end": 125.62,
    "text": "So what is a flaky test for you?"
  },
  {
    "start": 127.78,
    "end": 128.539,
    "text": "That's a good question!"
  },
  {
    "start": 129.82,
    "end": 132.98,
    "text": "There are some qualities that we expect from the tests that we write."
  },
  {
    "start": 134.58,
    "end": 136.46,
    "text": "when they have these quality we call them A Good Test."
  },
  {
    "start": 137.22,
    "end": 139.56,
    "text": "One of those is self-checking."
  },
  {
    "start": 140.14,
    "end": 143.12,
    "text": "You shouldn't need human beings come in verify this thing working."
  },
  {
    "start": 144.2,
    "end": 145.4,
    "text": "One of them is it's fast."
  },
  {
    "start": 145.8,
    "end": 151.36,
    "text": "We want our feedback to come back to us as quickly as possible, one of them if its isolated."
  },
  {
    "start": 151.9,
    "end": 154.62,
    "text": "in order to be fast we won't able run a test concurrently."
  },
  {
    "start": 155.18,
    "end": 161.88,
    "text": "so you do things like changing database states and ensure that another tests running at the same time your still okay?"
  },
  {
    "start": 162.8,
    "end": 166.82,
    "text": "And then final quality we wanted was a test needs to repeatable."
  },
  {
    "start": 167.42,
    "end": 175.1,
    "text": "That means again and again It might be a pass, it may fail but its always the same."
  },
  {
    "start": 175.9,
    "end": 179.96,
    "text": "A flaky test is one where that final property of repeatability isn't true!"
  },
  {
    "start": 180.58,
    "end": 184.46,
    "text": "It'll pass and will fail... And thats'a problem!"
  },
  {
    "start": 185.42,
    "end": 190.38,
    "text": "The reason why this is a problem Is that RCI systems set them up for a reason."
  },
  {
    "start": 191.68,
    "end": 194.24,
    "text": "That's to verify whether changes are safe."
  },
  {
    "start": 195.98,
    "end": 200.24,
    "text": "We don't know if they're perfectly safe But we do know If the build goes green."
  },
  {
    "start": 200.64,
    "end": 205.52,
    "text": "All the automated checks and tests that we have in place say, That changes is as safe it can be."
  },
  {
    "start": 207.18,
    "end": 213.44,
    "text": "When when you have flaky tests We no longer trust that piece of signal And that means that we waste time."
  },
  {
    "start": 214.5,
    "end": 220.42,
    "text": "I'm sure like most of your listeners Have been on a project where the testers see i run this flakey?"
  },
  {
    "start": 220.46,
    "end": 223.62,
    "text": "They're like oh yeah It just does that."
  },
  {
    "start": 224.0,
    "end": 224.5,
    "text": "Yeah!"
  },
  {
    "start": 224.56,
    "end": 226.96,
    "text": "Then you rerun that CI build."
  },
  {
    "start": 227.5,
    "end": 234.72,
    "text": "Sometimes it passes sometimes it fails And then you rerun it again and go, oh actually this might be a proper failure."
  },
  {
    "start": 235.36,
    "end": 239.02,
    "text": "It's only two or three runs in that you start investigating what happened?"
  },
  {
    "start": 240.04,
    "end": 245.6,
    "text": "You've destroyed the value that your CI brings and also lengthened to feedback loops on."
  },
  {
    "start": 245.7,
    "end": 250.82,
    "text": "one of things we want when doing software engineering is being able get feedback from changes quickly as possible."
  },
  {
    "start": 252.44,
    "end": 253.98,
    "text": "No, that's why a flaky test is"
  },
  {
    "start": 254.82,
    "end": 254.94,
    "text": "bad."
  },
  {
    "start": 255.02,
    "end": 261.72,
    "text": "And you mentioned also yesterday from the CI not just the value but also trust in the CI pipeline?"
  },
  {
    "start": 261.8,
    "end": 261.86,
    "text": "Yes!"
  },
  {
    "start": 261.959,
    "end": 265.42,
    "text": "I like this idea because we want to trust the CI and then have two."
  },
  {
    "start": 265.46,
    "end": 266.96,
    "text": "so it works fine."
  },
  {
    "start": 267.06,
    "end": 268.82,
    "text": "Yeah absolutely Like."
  },
  {
    "start": 270.22,
    "end": 277.24,
    "text": "anything which doesn't give consistent signal be that or pass-or-fail destroys trusts."
  },
  {
    "start": 277.74,
    "end": 281.58,
    "text": "If we don't have trust on the CI system we might as well not be running it."
  },
  {
    "start": 282.02,
    "end": 285.74,
    "text": "Yeah, that's true!"
  },
  {
    "start": 285.86,
    "end": 290.06,
    "text": "And when you go one step back what is the cause of a flaky test?"
  },
  {
    "start": 290.12,
    "end": 291.3,
    "text": "What can be there..."
  },
  {
    "start": 292.4,
    "end": 305.94,
    "text": "It's really interesting because I covered in the talk how a flakey test sometimes can actually be good tests just because its flaky doesn't necessarily mean it's bad and because it's highlighting things."
  },
  {
    "start": 307.98,
    "end": 312.88,
    "text": "It means that when the stars align and everything is functioning, The code is working."
  },
  {
    "start": 313.32,
    "end": 317.72,
    "text": "And as a great thing to know When it fails there's any number of reasons."
  },
  {
    "start": 318.68,
    "end": 321.0,
    "text": "One of them might be transient network errors."
  },
  {
    "start": 321.48,
    "end": 331.4,
    "text": "There may be Dependencies you have down like databases or message queues around available Could be race conditions could be Any number of things?"
  },
  {
    "start": 331.88,
    "end": 337.38,
    "text": "They can sometimes be flakiness in their in the underlying test framework, but it's rarely that."
  },
  {
    "start": 338.14,
    "end": 344.82,
    "text": "For those other reasons you know transient errors network problems dependencies being down Those are problem."
  },
  {
    "start": 344.88,
    "end": 351.28,
    "text": "so your production code will have to deal with and So finding flaky tests sometimes a bit of a blessing."
  },
  {
    "start": 351.66,
    "end": 355.04,
    "text": "Yeah because it encourages you to go and find out You know why?"
  },
  {
    "start": 355.2,
    "end": 357.48,
    "text": "That test is failing and try and fix the underlying cause."
  },
  {
    "start": 358.28,
    "end": 367.7,
    "text": "now The problem is when our developers find tests at a flakey They tend to go, the test is bad."
  },
  {
    "start": 368.72,
    "end": 375.04,
    "text": "And they blame the tests or they blame infrastructure and like oh well it should have just worked."
  },
  {
    "start": 375.38,
    "end": 381.16,
    "text": "so no you can't expect a hundred percent uptime of your dependencies if there's a transient failure."
  },
  {
    "start": 381.66,
    "end": 388.7,
    "text": "You really should have robust code that could deal with that because when app in hands of users will encounter those problems."
  },
  {
    "start": 393.24,
    "end": 397.74,
    "text": "So it's not just about to getting rid of the flaky test, but they use their information."
  },
  {
    "start": 398.12,
    "end": 401.92,
    "text": "if there is a test Flakey look behind that yes and To look what?"
  },
  {
    "start": 402.04,
    "end": 405.7,
    "text": "What's in it for me or for the robustness of my system precisely."
  },
  {
    "start": 406.02,
    "end": 415.04,
    "text": "Yeah one other problems with that Is it takes a lot of time yeah to track down A course of flakiness particularly If there are multiple causes of flaking us."
  },
  {
    "start": 415.48,
    "end": 421.24,
    "text": "well The flakeness doesn't happen very often like as a software engineer when you have a repeatable failure."
  },
  {
    "start": 421.8,
    "end": 426.24,
    "text": "That's far easier to debug than what happens every twenty test runs, for example."
  },
  {
    "start": 427.64,
    "end": 434.9,
    "text": "And so that is another reason why engineers are never particularly keen on working into sources of flakiness because it's a really difficult problem"
  },
  {
    "start": 435.98,
    "end": 438.44,
    "text": "Yeah and nobody wants to take care of it."
  },
  {
    "start": 438.56,
    "end": 443.34,
    "text": "Because yeah It's flaky not often repeatable when its flaky."
  },
  {
    "start": 445.4,
    "end": 452.88,
    "text": "But what strategies do get this information Relief from the rest."
  },
  {
    "start": 452.96,
    "end": 453.46,
    "text": "and to say"
  },
  {
    "start": 455.12,
    "end": 456.02,
    "text": "this was covered in a talk."
  },
  {
    "start": 456.64,
    "end": 457.54,
    "text": "Thank you for mentioning it."
  },
  {
    "start": 457.58,
    "end": 457.6,
    "text": "So,"
  },
  {
    "start": 459.02,
    "end": 464.82,
    "text": "so very first thing that you need to do is You need to take that test out of CI your see."
  },
  {
    "start": 464.94,
    "end": 468.3,
    "text": "I must give you strong signal And that means the tests must be removed."
  },
  {
    "start": 468.82,
    "end": 474.36,
    "text": "and there are couple of strategies He could use to do them as simplest might be To add a test annotation."
  },
  {
    "start": 474.54,
    "end": 477.44,
    "text": "most test frameworks have our adding more annotation you can add."
  },
  {
    "start": 478.74,
    "end": 485.62,
    "text": "if he'd do that linking to a bug in whatever bug tracking system you use is an excellent idea because it allows traceability."
  },
  {
    "start": 486.48,
    "end": 499.68,
    "text": "The other approach that can be used, if we could have a block list that maintains somewhere either in the repo or elsewhere and that information allows us sort of identify tests should be skipped for any particular run."
  },
  {
    "start": 500.28,
    "end": 501.78,
    "text": "That's what will do with Selenium projects."
  },
  {
    "start": 501.84,
    "end": 507.4,
    "text": "by way We have a skipped test file And feed into RCI System."
  },
  {
    "start": 507.94,
    "end": 512.12,
    "text": "Now, the advantage with having a skip test file is that we haven't got an annotation."
  },
  {
    "start": 512.179,
    "end": 514.44,
    "text": "And annotations are hard to find and they're hard to remember."
  },
  {
    "start": 515.059,
    "end": 522.179,
    "text": "A single text file with all of the crimes you have committed against stability in one place means that you can go back and find them."
  },
  {
    "start": 523.24,
    "end": 527.28,
    "text": "So once you've taken... The tests out of rotation?"
  },
  {
    "start": 527.68,
    "end": 528.92,
    "text": "A lot of companies stopped there!"
  },
  {
    "start": 529.42,
    "end": 529.56,
    "text": "Yes!"
  },
  {
    "start": 530.0,
    "end": 531.2,
    "text": "They say okay now.. We're"
  },
  {
    "start": 531.26,
    "end": 534.7,
    "text": "done!"
  },
  {
    "start": 536.02,
    "end": 537.7,
    "text": "And in a way that's fine, right?"
  },
  {
    "start": 537.76,
    "end": 539.68,
    "text": "Like at least their build is now stable."
  },
  {
    "start": 541.72,
    "end": 546.66,
    "text": "But here's the thing unless someone owns something it tends not to happen."
  },
  {
    "start": 547.92,
    "end": 552.84,
    "text": "so what we want to be able to do Is we want To assign The test to somebody."
  },
  {
    "start": 553.92,
    "end": 564.46,
    "text": "I really strongly believe That somebody should Be A person and Not a team Because otherwise you have like the tragedy of the commons Where everyone goes oh Somebody Else will fix It."
  },
  {
    "start": 565.26,
    "end": 572.88,
    "text": "Now assigning it to someone doesn't necessarily mean they do all the work, but it means that you are responsible for figuring out how this test should be fixed."
  },
  {
    "start": 573.46,
    "end": 591.26,
    "text": "And maybe what they need is a sign-in or whatever you can do with a bug right when it comes into your system... ...but once you have identified an owner then the next thing you'll need to do is resolve those tests and there's many strategies in order of doing so!"
  },
  {
    "start": 591.36,
    "end": 598.24,
    "text": "The first one is just run it repeatedly see whether you can identify the cause and put in a little fix."
  },
  {
    "start": 599.38,
    "end": 605.12,
    "text": "Another thing he could do is, that process of figuring out why this test is flaky."
  },
  {
    "start": 605.8,
    "end": 606.26,
    "text": "And then right?"
  },
  {
    "start": 606.38,
    "end": 608.34,
    "text": "The smallest possible tests You Can."
  },
  {
    "start": 608.94,
    "end": 627.84,
    "text": "That validates the fix your about to do fixes that if you Do that consistently you end up with it the classic test pyramid sort of wide on the bottom Yeah With the small tests and narrow one atop for the large tests Which was fantastic Another way of dealing with a problem is delete the test."
  },
  {
    "start": 629.22,
    "end": 631.4,
    "text": "Like, it's not stable... We cannot delete"
  },
  {
    "start": 631.46,
    "end": 631.88,
    "text": "tests!"
  },
  {
    "start": 632.94,
    "end": 633.44,
    "text": "You're right."
  },
  {
    "start": 633.8,
    "end": 642.12,
    "text": "but we can because hopefully were all using source control like if the test was really valuable and we could reviver fire then bring that back."
  },
  {
    "start": 643.04,
    "end": 645.74,
    "text": "One other reason why people don't want to delete tests."
  },
  {
    "start": 646.62,
    "end": 648.24,
    "text": "It covers an important workflow."
  },
  {
    "start": 648.68,
    "end": 652.28,
    "text": "I'm doing air quotes you cant see on the podcast."
  },
  {
    "start": 652.64,
    "end": 653.3,
    "text": "I'm doing quite well."
  },
  {
    "start": 653.34,
    "end": 654.34,
    "text": "Yeah, we have video too!"
  },
  {
    "start": 654.5,
    "end": 654.58,
    "text": "Oh"
  },
  {
    "start": 657.52,
    "end": 659.24,
    "text": "good so i've been talking a lot with my hands aswell."
  },
  {
    "start": 659.46,
    "end": 662.84,
    "text": "that's splendid yeah."
  },
  {
    "start": 662.94,
    "end": 665.54,
    "text": "So people go this test is really valuable for us."
  },
  {
    "start": 665.6,
    "end": 669.38,
    "text": "We need it because its our crucial workflow As like."
  },
  {
    "start": 669.42,
    "end": 677.8,
    "text": "Well then It's worth spending The engineering time to get This right and if Its not worth Spending The Engineering Time To Get This Right Is probably Not Worth Having The Test."
  },
  {
    "start": 678.74,
    "end": 682.7,
    "text": "The more important you say it is, the higher priority to get that thing rock solid."
  },
  {
    "start": 684.4,
    "end": 687.3,
    "text": "There are strategies for fixing tests."
  },
  {
    "start": 688.34,
    "end": 694.12,
    "text": "Flakiness in and of itself can be inescapable like there's always going to something unforeseen."
  },
  {
    "start": 695.8,
    "end": 697.28,
    "text": "And one of those strategies we can do?"
  },
  {
    "start": 698.52,
    "end": 700.52,
    "text": "You could rerun tests on same builds."
  },
  {
    "start": 701.24,
    "end": 701.92,
    "text": "That's okay."
  },
  {
    "start": 703.14,
    "end": 705.24,
    "text": "It not really ok but its an acceptable thing."
  },
  {
    "start": 706.2,
    "end": 712.16,
    "text": "And the reason why that works is if a test has a five percent chance of being flaky, first time you run it... Ah!"
  },
  {
    "start": 712.6,
    "end": 713.46,
    "text": "You hit that five per cent."
  },
  {
    "start": 713.78,
    "end": 723.48,
    "text": "The second time we run it ...you've got five percent to five percent chances at doing that and so you have this sort of exponentially decreasing probability of it failing because of flakiness."
  },
  {
    "start": 724.44,
    "end": 730.16,
    "text": ".The downside is by running that test again making them be making your build longer."
  },
  {
    "start": 731.98,
    "end": 736.94,
    "text": "maybe If there's unit tests like small tests took milliseconds That would be fine."
  },
  {
    "start": 738.24,
    "end": 743.18,
    "text": "But by nature, the tests that tend to be most flaky are the largest ones because they have the most moving pieces."
  },
  {
    "start": 743.26,
    "end": 749.56,
    "text": "and that's problematic Because if you take a test it takes four minutes for run And he'll run at five times."
  },
  {
    "start": 751.1,
    "end": 754.3,
    "text": "You've blown out the budget that you have forgetting results back."
  },
  {
    "start": 755.2,
    "end": 762.96,
    "text": "In my experience On average Roughly three times is good number To do test retries."
  },
  {
    "start": 763.04,
    "end": 768.44,
    "text": "but Take a look what you're doing, take a look at your time budget for your CI builds as well."
  },
  {
    "start": 769.22,
    "end": 779.06,
    "text": "You might have an SLO where we should have CI results within ten minutes and if you have an average test that takes three minutes then you want to rerun it?"
  },
  {
    "start": 779.48,
    "end": 784.14,
    "text": "You could theoretically run in three times but not four without blowing out the SLO."
  },
  {
    "start": 784.34,
    "end": 786.2,
    "text": "so there's a balancing act to be done now."
  },
  {
    "start": 787.0,
    "end": 787.64,
    "text": "Does that make sense?"
  },
  {
    "start": 787.84,
    "end": 788.54,
    "text": "Yeah, it makes sense."
  },
  {
    "start": 790.08,
    "end": 794.58,
    "text": "In your experience I assume you have often refactored such flaky tests."
  },
  {
    "start": 796.44,
    "end": 802.64,
    "text": "What was in your opinion or the most common cause of a flaky test?"
  },
  {
    "start": 803.12,
    "end": 811.78,
    "text": "Do you have any statistics from your experiences?"
  },
  {
    "start": 811.84,
    "end": 820.04,
    "text": "So big two are Concurrent access to shared data and mutable state."
  },
  {
    "start": 820.44,
    "end": 821.1,
    "text": "Yeah,"
  },
  {
    "start": 821.14,
    "end": 822.48,
    "text": "those are the big two okay."
  },
  {
    "start": 823.18,
    "end": 825.04,
    "text": "So I do a lot of work."
  },
  {
    "start": 825.1,
    "end": 829.04,
    "text": "in Java you can have static values that are shared across entire JVN."
  },
  {
    "start": 829.96,
    "end": 831.24,
    "text": "You can have tests at run."
  },
  {
    "start": 831.94,
    "end": 835.06,
    "text": "they modify some value stored in a static."
  },
  {
    "start": 835.96,
    "end": 847.96,
    "text": "A later test modifies our value or reads it And that's fine because when your on the test in that order It's ok but when you're running out-of-order or it's perturbed in some other way, suddenly the test starts failing."
  },
  {
    "start": 848.12,
    "end": 848.62,
    "text": "You're like what?"
  },
  {
    "start": 849.36,
    "end": 852.58,
    "text": "Like that's weird and you take forever to track down that stuff."
  },
  {
    "start": 853.46,
    "end": 857.34,
    "text": "The other thing is changing shared states."
  },
  {
    "start": 857.84,
    "end": 860.32,
    "text": "So have a database for example."
  },
  {
    "start": 860.72,
    "end": 869.82,
    "text": "And if your first line of tests drops everything on this table Something else midway through a test where they are modifying something in that table Disaster happens!"
  },
  {
    "start": 874.76,
    "end": 877.84,
    "text": "Works in isolation, but fails when run with other things."
  },
  {
    "start": 877.9,
    "end": 880.98,
    "text": "It's like ah I think i found something."
  },
  {
    "start": 882.44,
    "end": 885.48,
    "text": "The hardest ones to track down are race conditions."
  },
  {
    "start": 885.56,
    "end": 890.42,
    "text": "yeah Like timing things where everything needs line up just right?"
  },
  {
    "start": 890.78,
    "end": 891.36,
    "text": "Yeah anybody?"
  },
  {
    "start": 891.42,
    "end": 900.68,
    "text": "okay an example of that is I was once writing a custom zip Compression implementation in Java and it was fine."
  },
  {
    "start": 901.1,
    "end": 902.84,
    "text": "And I'd run my test into go green."
  },
  {
    "start": 903.62,
    "end": 903.78,
    "text": "No, no."
  },
  {
    "start": 903.9,
    "end": 904.82,
    "text": "Run it and it would be red."
  },
  {
    "start": 905.64,
    "end": 906.14,
    "text": "I'd run again."
  },
  {
    "start": 906.18,
    "end": 906.64,
    "text": "It'll Be Red."
  },
  {
    "start": 907.26,
    "end": 908.18,
    "text": "Run Again It Would Be Green."
  },
  {
    "start": 908.64,
    "end": 910.3,
    "text": "It's like what on earth is going On?"
  },
  {
    "start": 910.82,
    "end": 912.04,
    "text": "Like the code was rock solid."
  },
  {
    "start": 912.36,
    "end": 913.42,
    "text": "i read it And it's like."
  },
  {
    "start": 913.5,
    "end": 915.08,
    "text": "clearly there's nothing wrong here."
  },
  {
    "start": 916.6,
    "end": 924.2,
    "text": "So The Problem Is That The Zip Format The Time Stamps Are Accurate To Within Two Seconds Not One Second."
  },
  {
    "start": 924.54,
    "end": 924.96,
    "text": "Okay."
  },
  {
    "start": 925.04,
    "end": 931.04,
    "text": "so If I Started The Test I Think Was An Even Second, the test would pass because of granularity was right."
  },
  {
    "start": 931.44,
    "end": 940.0,
    "text": "But if I did it on an odd second he would fail Because we're round to either up or down and can't remember what It is like when you realize that."
  },
  {
    "start": 940.28,
    "end": 946.22,
    "text": "okay now i'm going to normalize my time stamps And suddenly rock solid text But it was a devil to try and down."
  },
  {
    "start": 947.38,
    "end": 948.98,
    "text": "Yeah, yeah great thought!"
  },
  {
    "start": 949.36,
    "end": 954.04,
    "text": "So I think you had a lot of tears in blood sweating when you were here at the factory."
  },
  {
    "start": 955.58,
    "end": 961.32,
    "text": "When we are all dealing with AI stuff what is your experience?"
  },
  {
    "start": 962.96,
    "end": 969.9,
    "text": "Did our tests get more flaky or did they make them more robust?"
  },
  {
    "start": 970.56,
    "end": 972.06,
    "text": "What's your direction?"
  },
  {
    "start": 974.94,
    "end": 976.0,
    "text": "That's a really good question."
  },
  {
    "start": 977.48,
    "end": 979.68,
    "text": "So I'm going to give the worst answer in the world."
  },
  {
    "start": 979.72,
    "end": 980.18,
    "text": "It depends?"
  },
  {
    "start": 981.1,
    "end": 982.72,
    "text": "It depends, I apologize."
  },
  {
    "start": 985.64,
    "end": 987.28,
    "text": "AI is a useful tool."
  },
  {
    "start": 988.02,
    "end": 992.7,
    "text": "right everything that an AI does should be reviewed by a human being."
  },
  {
    "start": 994.28,
    "end": 998.38,
    "text": "AI as very good at generating large number of test cases it seemed plausible."
  },
  {
    "start": 1000.02,
    "end": 1009.02,
    "text": "one other things we learned when doing extreme programming back then is like when after you've written some code, do some refactoring and remove the commonality."
  },
  {
    "start": 1009.82,
    "end": 1013.4,
    "text": "So quite often we end up with these quite long verbose tests... Yeah!"
  },
  {
    "start": 1013.68,
    "end": 1020.8,
    "text": "...and you can replace that with a parameterized test or other tests where you just sort of put things in loop And it becomes a lot clearer."
  },
  {
    "start": 1021.44,
    "end": 1022.9,
    "text": "That helps identify problems."
  },
  {
    "start": 1025.2,
    "end": 1031.22,
    "text": "I tend not to use AI to write my tests Like here are conditions for the thing."
  },
  {
    "start": 1032.42,
    "end": 1036.339,
    "text": "But where i do like using AI is where I'm debugging tests."
  },
  {
    "start": 1036.98,
    "end": 1039.16,
    "text": "It's like, why is this code failing?"
  },
  {
    "start": 1039.26,
    "end": 1040.04,
    "text": "What is going wrong?"
  },
  {
    "start": 1040.119,
    "end": 1040.92,
    "text": "Can you have a look at"
  },
  {
    "start": 1040.98,
    "end": 1041.06,
    "text": "it?\"."
  },
  {
    "start": 1042.099,
    "end": 1062.46,
    "text": "And AI are really good at analyzing quite intricate code and figuring out... This value here isn't ever set or there's a problem here or he might want to take a look that one of the problems is so like a drunk man in a bar with an opinion And they will say it very emphatically, and they might well be wrong."
  },
  {
    "start": 1063.5,
    "end": 1069.94,
    "text": "So the problem is on this line... ...and you can waste a certain amount of time tracking that down?"
  },
  {
    "start": 1071.32,
    "end": 1075.7,
    "text": "There's another mechanism which is to get different LLNs to go."
  },
  {
    "start": 1076.06,
    "end": 1077.66,
    "text": "I think there are problems with these lines."
  },
  {
    "start": 1078.0,
    "end": 1080.56,
    "text": "Can we verify if their test is failing because of those reasons?"
  },
  {
    "start": 1081.16,
    "end": 1082.38,
    "text": "If both agree then its like."
  },
  {
    "start": 1082.46,
    "end": 1086.98,
    "text": "OK That seems like a relatively same place to start."
  },
  {
    "start": 1087.18,
    "end": 1087.36,
    "text": "Yeah,"
  },
  {
    "start": 1087.5,
    "end": 1089.5,
    "text": "really use it as a tool for your debugging..."
  },
  {
    "start": 1091.14,
    "end": 1093.38,
    "text": "AI is not a replacement or something?"
  },
  {
    "start": 1093.7,
    "end": 1096.24,
    "text": "It's a power tool that we can use to help us do our work."
  },
  {
    "start": 1097.64,
    "end": 1105.82,
    "text": "So when fixing FlakyTest is one side but would be great if you could prevent it up front."
  },
  {
    "start": 1106.86,
    "end": 1107.34,
    "text": "What do you have?"
  },
  {
    "start": 1107.54,
    "end": 1108.5,
    "text": "One two three tips."
  },
  {
    "start": 1111.82,
    "end": 1117.12,
    "text": "One thing to do is gatekeep, like don't allow a flaky test into your build."
  },
  {
    "start": 1118.54,
    "end": 1120.88,
    "text": "A company that does this really well is actually Meta."
  },
  {
    "start": 1121.72,
    "end": 1124.86,
    "text": "so before any testers allowed go and affect the CI runs."
  },
  {
    "start": 1124.92,
    "end": 1131.16,
    "text": "at Meta they run one hundred times concurrently overnight with every other test And only if it's completely stable."
  },
  {
    "start": 1132.16,
    "end": 1135.0,
    "text": "So now we're going include this in the CI Runs."
  },
  {
    "start": 1135.4,
    "end": 1136.22,
    "text": "Is this automated?"
  },
  {
    "start": 1136.86,
    "end": 1138.98,
    "text": "Yes It's all completely automated."
  },
  {
    "start": 1140.28,
    "end": 1141.88,
    "text": "a human being would drown under the weight."
  },
  {
    "start": 1141.94,
    "end": 1150.42,
    "text": "They have so many engineers, I can only imagine some poor soul going like that one... it will be sold astraying anything!"
  },
  {
    "start": 1151.84,
    "end": 1154.58,
    "text": "You don't need to be meta but you can go okay."
  },
  {
    "start": 1154.64,
    "end": 1157.74,
    "text": "before i put this test into build..i'll run at ten times."
  },
  {
    "start": 1158.24,
    "end": 1164.02,
    "text": "That does require that you keep database of what tests do and it means that sort of trivial things are like."
  },
  {
    "start": 1164.18,
    "end": 1167.64,
    "text": "i need to rename a test or fix a typo because ive got better insight."
  },
  {
    "start": 1169.36,
    "end": 1174.34,
    "text": "That will cause your test to rerun, but that's okay because it should be stable if It's already in the build and if it's not."
  },
  {
    "start": 1175.1,
    "end": 1175.86,
    "text": "That's fine as well."
  },
  {
    "start": 1177.8,
    "end": 1180.5,
    "text": "So yeah gatekeeping is a sort of fantastic strategy."
  },
  {
    "start": 1181.2,
    "end": 1183.16,
    "text": "The other thing you can do Is?"
  },
  {
    "start": 1183.24,
    "end": 1191.7,
    "text": "You can take that the tests are known to be flaky And you can rerun them Because sometimes there's this sort of underlying systemic reason for failure."
  },
  {
    "start": 1192.3,
    "end": 1197.22,
    "text": "Somebody might have fixed that underlying course and suddenly a swathe of tests suddenly start passing."
  },
  {
    "start": 1198.56,
    "end": 1199.62,
    "text": "That's a really interesting thing."
  },
  {
    "start": 1200.08,
    "end": 1211.32,
    "text": "And that is another argument for having a separate file containing the list of tests that have failed rather than using an annotation, because it's quite cumbersome to modify code and go like, do I need to remove that?"
  },
  {
    "start": 1211.4,
    "end": 1212.82,
    "text": "Ignore annotation or do we keep it?"
  },
  {
    "start": 1213.12,
    "end": 1216.28,
    "text": "It was easy just by those."
  },
  {
    "start": 1217.74,
    "end": 1225.1,
    "text": "Then once happens you can put this test back into...I called in general population but your CI runs where you go."
  },
  {
    "start": 1225.18,
    "end": 1227.04,
    "text": "okay fine these are now providing value again."
  },
  {
    "start": 1227.5,
    "end": 1228.82,
    "text": "we can trust them and they give us signal."
  },
  {
    "start": 1231.0,
    "end": 1233.58,
    "text": "So valuable information about flaky tests."
  },
  {
    "start": 1233.66,
    "end": 1242.52,
    "text": "I like it so much because a lot of clients are dealing with flaky test and yeah, they often stop when removing them from the test suite And then do nothing with that."
  },
  {
    "start": 1242.78,
    "end": 1246.1,
    "text": "yes But there is their solutions and thoughts."
  },
  {
    "start": 1246.16,
    "end": 1248.26,
    "text": "you can invest into debugging It."
  },
  {
    "start": 1248.64,
    "end": 1252.0,
    "text": "one other things by the way That i've seen Is people put up a TTL?"
  },
  {
    "start": 1252.1,
    "end": 1254.1,
    "text": "A time to live on a flaky Test?"
  },
  {
    "start": 1254.18,
    "end": 1260.12,
    "text": "Yeah, and if its not being fixed before that time-to-live They delete it Because it's dead code."
  },
  {
    "start": 1260.76,
    "end": 1263.5,
    "text": "It isn't being used, and providing value is just dead weight."
  },
  {
    "start": 1264.24,
    "end": 1266.18,
    "text": "so deleting it a fine thing to do."
  },
  {
    "start": 1267.56,
    "end": 1272.48,
    "text": "The other thing as well is Linda at the same meetup gave an interesting talk."
  },
  {
    "start": 1273.52,
    "end": 1284.6,
    "text": "She was talking about having layered strategy for your testing And because earlier layers were covering much of their risk they didn't need certain tests."
  },
  {
    "start": 1285.04,
    "end": 1285.74,
    "text": "They could remove them."
  },
  {
    "start": 1287.0,
    "end": 1289.24,
    "text": "You quite often find flakiness in your large tests."
  },
  {
    "start": 1289.52,
    "end": 1305.76,
    "text": "Yeah, if you have covered all the things that a large test would've done with a variety of other tests It's okay to delete that test as well like We become very attached to our code because we spend a lot of time writing it and a lot Of time debugging it on A lot of Time making it pristine And lovely."
  },
  {
    "start": 1306.52,
    "end": 1309.34,
    "text": "unless regenerated it With AI which case will come attach to it?"
  },
  {
    "start": 1309.38,
    "end": 1310.62,
    "text": "Because look how much code we've written."
  },
  {
    "start": 1312.64,
    "end": 1314.58,
    "text": "But it's okay to delete things like."
  },
  {
    "start": 1314.62,
    "end": 1322.78,
    "text": "the aim of the game with software engineering is To get a change out in the hands of our users as quickly and safely as possible And that means the less code we do."
  },
  {
    "start": 1325.14,
    "end": 1327.4,
    "text": "Thank you very much Simon for this insights."
  },
  {
    "start": 1328.56,
    "end": 1334.66,
    "text": "It was really pleasure for me too, so yeah Thank you very much, enjoy the rest of the conference."
  },
  {
    "start": 1334.86,
    "end": 1335.82,
    "text": "It's going to be so much fun!"
  },
  {
    "start": 1336.06,
    "end": 1338.34,
    "text": "Yeah we have today a birthday party in evening."
  },
  {
    "start": 1338.38,
    "end": 1339.02,
    "text": "Fantastic"
  },
  {
    "start": 1339.3,
    "end": 1340.2,
    "text": "I am looking forward to that."
  },
  {
    "start": 1340.3,
    "end": 1340.72,
    "text": "yeah"
  },
  {
    "start": 1340.76,
    "end": 1342.82,
    "text": "i will maybe drink one beer there too."
  },
  {
    "start": 1343.92,
    "end": 1344.08,
    "text": "I'll"
  },
  {
    "start": 1344.7,
    "end": 1345.52,
    "text": "have a glass of water for us."
  },
  {
    "start": 1345.68,
    "end": 1345.74,
    "text": "Yes"
  },
  {
    "start": 1346.54,
    "end": 1348.74,
    "text": "okay thank you that were here and enjoy"
  },
  {
    "start": 1349.88,
    "end": 1349.92,
    "text": "it."
  },
  {
    "start": 1350.02,
    "end": 1351.02,
    "text": "all right take care."
  }
]