[
  {
    "startMs": 160,
    "text": "Hi everyone, my name is Rajat Kotial."
  },
  {
    "startMs": 2710,
    "text": "Hi everyone, my name is Rajat Kotial."
  },
  {
    "startMs": 2720,
    "text": "Hi everyone, my name is Rajat Kotial. I'm part of the automated reasoning"
  },
  {
    "startMs": 4390,
    "text": "I'm part of the automated reasoning"
  },
  {
    "startMs": 4400,
    "text": "I'm part of the automated reasoning group at AWS. And today I'm going to"
  },
  {
    "startMs": 7190,
    "text": "group at AWS. And today I'm going to"
  },
  {
    "startMs": 7200,
    "text": "group at AWS. And today I'm going to talk about a project I worked on"
  },
  {
    "startMs": 9509,
    "text": "talk about a project I worked on"
  },
  {
    "startMs": 9519,
    "text": "talk about a project I worked on recently called KO specification"
  },
  {
    "startMs": 11910,
    "text": "recently called KO specification"
  },
  {
    "startMs": 11920,
    "text": "recently called KO specification analyzer. And um we use certain"
  },
  {
    "startMs": 14950,
    "text": "analyzer. And um we use certain"
  },
  {
    "startMs": 14960,
    "text": "analyzer. And um we use certain techniques which involves formal methods"
  },
  {
    "startMs": 16870,
    "text": "techniques which involves formal methods"
  },
  {
    "startMs": 16880,
    "text": "techniques which involves formal methods and automated reasoning to prove"
  },
  {
    "startMs": 19750,
    "text": "and automated reasoning to prove"
  },
  {
    "startMs": 19760,
    "text": "and automated reasoning to prove specification that goes to your coding"
  },
  {
    "startMs": 21670,
    "text": "specification that goes to your coding"
  },
  {
    "startMs": 21680,
    "text": "specification that goes to your coding agent before it writes code."
  },
  {
    "startMs": 30720,
    "text": "So let's take a certain bug right"
  },
  {
    "startMs": 34709,
    "text": "So let's take a certain bug right"
  },
  {
    "startMs": 34719,
    "text": "So let's take a certain bug right this is a specification"
  },
  {
    "startMs": 37750,
    "text": "this is a specification"
  },
  {
    "startMs": 37760,
    "text": "this is a specification uh snippet and there are two rules for"
  },
  {
    "startMs": 40470,
    "text": "uh snippet and there are two rules for"
  },
  {
    "startMs": 40480,
    "text": "uh snippet and there are two rules for example and uh the rule two says when an"
  },
  {
    "startMs": 43830,
    "text": "example and uh the rule two says when an"
  },
  {
    "startMs": 43840,
    "text": "example and uh the rule two says when an order is submitted and this is for a a"
  },
  {
    "startMs": 46630,
    "text": "order is submitted and this is for a a"
  },
  {
    "startMs": 46640,
    "text": "order is submitted and this is for a a general e-commerce website let's say"
  },
  {
    "startMs": 49190,
    "text": "general e-commerce website let's say"
  },
  {
    "startMs": 49200,
    "text": "general e-commerce website let's say when an order is submitted and the"
  },
  {
    "startMs": 50950,
    "text": "when an order is submitted and the"
  },
  {
    "startMs": 50960,
    "text": "when an order is submitted and the inventory is unavailable we place the"
  },
  {
    "startMs": 53590,
    "text": "inventory is unavailable we place the"
  },
  {
    "startMs": 53600,
    "text": "inventory is unavailable we place the order as a back order which makes sense"
  },
  {
    "startMs": 55750,
    "text": "order as a back order which makes sense"
  },
  {
    "startMs": 55760,
    "text": "order as a back order which makes sense when you read it individually and"
  },
  {
    "startMs": 57670,
    "text": "when you read it individually and"
  },
  {
    "startMs": 57680,
    "text": "when you read it individually and there's also another rule in the same"
  },
  {
    "startMs": 59110,
    "text": "there's also another rule in the same"
  },
  {
    "startMs": 59120,
    "text": "there's also another rule in the same specification file uh which you hand to"
  },
  {
    "startMs": 61510,
    "text": "specification file uh which you hand to"
  },
  {
    "startMs": 61520,
    "text": "specification file uh which you hand to the coding agent which says u no order"
  },
  {
    "startMs": 64070,
    "text": "the coding agent which says u no order"
  },
  {
    "startMs": 64080,
    "text": "the coding agent which says u no order shall be placed as a back order and this"
  },
  {
    "startMs": 66789,
    "text": "shall be placed as a back order and this"
  },
  {
    "startMs": 66799,
    "text": "shall be placed as a back order and this is a rule we've set so individually read"
  },
  {
    "startMs": 70310,
    "text": "is a rule we've set so individually read"
  },
  {
    "startMs": 70320,
    "text": "is a rule we've set so individually read these two lines make sense right but in"
  },
  {
    "startMs": 73990,
    "text": "these two lines make sense right but in"
  },
  {
    "startMs": 74000,
    "text": "these two lines make sense right but in when an LLM reads this reads both the"
  },
  {
    "startMs": 76789,
    "text": "when an LLM reads this reads both the"
  },
  {
    "startMs": 76799,
    "text": "when an LLM reads this reads both the rules together both cannot hold true at"
  },
  {
    "startMs": 79109,
    "text": "rules together both cannot hold true at"
  },
  {
    "startMs": 79119,
    "text": "rules together both cannot hold true at the same time so we need to make sure"
  },
  {
    "startMs": 82630,
    "text": "the same time so we need to make sure"
  },
  {
    "startMs": 82640,
    "text": "the same time so we need to make sure the we resolve this ambiguity that the"
  },
  {
    "startMs": 84870,
    "text": "the we resolve this ambiguity that the"
  },
  {
    "startMs": 84880,
    "text": "the we resolve this ambiguity that the LLM is observing."
  },
  {
    "startMs": 88469,
    "text": "LLM is observing."
  },
  {
    "startMs": 88479,
    "text": "LLM is observing. And uh"
  },
  {
    "startMs": 91270,
    "text": "And uh"
  },
  {
    "startMs": 91280,
    "text": "And uh the scope and the reason we"
  },
  {
    "startMs": 96310,
    "text": "the scope and the reason we"
  },
  {
    "startMs": 96320,
    "text": "the scope and the reason we uh put an effort into this project is"
  },
  {
    "startMs": 98870,
    "text": "uh put an effort into this project is"
  },
  {
    "startMs": 98880,
    "text": "uh put an effort into this project is based on public research and recent"
  },
  {
    "startMs": 101109,
    "text": "based on public research and recent"
  },
  {
    "startMs": 101119,
    "text": "based on public research and recent research, we've observed there's a 20 to"
  },
  {
    "startMs": 102870,
    "text": "research, we've observed there's a 20 to"
  },
  {
    "startMs": 102880,
    "text": "research, we've observed there's a 20 to 40% drop with unclear prompts on the"
  },
  {
    "startMs": 106710,
    "text": "40% drop with unclear prompts on the"
  },
  {
    "startMs": 106720,
    "text": "40% drop with unclear prompts on the accuracy the LLM uh generates on the"
  },
  {
    "startMs": 109590,
    "text": "accuracy the LLM uh generates on the"
  },
  {
    "startMs": 109600,
    "text": "accuracy the LLM uh generates on the code accuracy, right? and 60 to 90% of"
  },
  {
    "startMs": 112710,
    "text": "code accuracy, right? and 60 to 90% of"
  },
  {
    "startMs": 112720,
    "text": "code accuracy, right? and 60 to 90% of valid code is still wrong after it"
  },
  {
    "startMs": 114469,
    "text": "valid code is still wrong after it"
  },
  {
    "startMs": 114479,
    "text": "valid code is still wrong after it writes it. And also in that case we"
  },
  {
    "startMs": 117749,
    "text": "writes it. And also in that case we"
  },
  {
    "startMs": 117759,
    "text": "writes it. And also in that case we observe two times the more more"
  },
  {
    "startMs": 119749,
    "text": "observe two times the more more"
  },
  {
    "startMs": 119759,
    "text": "observe two times the more more regressions when we uh deploy that code."
  },
  {
    "startMs": 125759,
    "text": "So where do bugs like these hide? Right?"
  },
  {
    "startMs": 128710,
    "text": "So where do bugs like these hide? Right?"
  },
  {
    "startMs": 128720,
    "text": "So where do bugs like these hide? Right? We want to detect these bugs before they"
  },
  {
    "startMs": 130949,
    "text": "We want to detect these bugs before they"
  },
  {
    "startMs": 130959,
    "text": "We want to detect these bugs before they reach the codebase. And the way only way"
  },
  {
    "startMs": 133750,
    "text": "reach the codebase. And the way only way"
  },
  {
    "startMs": 133760,
    "text": "reach the codebase. And the way only way to do that it's similar to how we write"
  },
  {
    "startMs": 135430,
    "text": "to do that it's similar to how we write"
  },
  {
    "startMs": 135440,
    "text": "to do that it's similar to how we write design docs. Similar to how we draw"
  },
  {
    "startMs": 138150,
    "text": "design docs. Similar to how we draw"
  },
  {
    "startMs": 138160,
    "text": "design docs. Similar to how we draw diagrams and flowcharts to make sure the"
  },
  {
    "startMs": 140710,
    "text": "diagrams and flowcharts to make sure the"
  },
  {
    "startMs": 140720,
    "text": "diagrams and flowcharts to make sure the flow is set before we write any code. We"
  },
  {
    "startMs": 142949,
    "text": "flow is set before we write any code. We"
  },
  {
    "startMs": 142959,
    "text": "flow is set before we write any code. We want to verify that without a human or"
  },
  {
    "startMs": 146470,
    "text": "want to verify that without a human or"
  },
  {
    "startMs": 146480,
    "text": "want to verify that without a human or anyone reading these files as the first"
  },
  {
    "startMs": 149270,
    "text": "anyone reading these files as the first"
  },
  {
    "startMs": 149280,
    "text": "anyone reading these files as the first reviewer"
  },
  {
    "startMs": 151190,
    "text": "reviewer"
  },
  {
    "startMs": 151200,
    "text": "reviewer in one sentence. Let's say the ambiguity"
  },
  {
    "startMs": 154790,
    "text": "in one sentence. Let's say the ambiguity"
  },
  {
    "startMs": 154800,
    "text": "in one sentence. Let's say the ambiguity and wording wrong level of detail but"
  },
  {
    "startMs": 157750,
    "text": "and wording wrong level of detail but"
  },
  {
    "startMs": 157760,
    "text": "and wording wrong level of detail but across sentences we can observe"
  },
  {
    "startMs": 159589,
    "text": "across sentences we can observe"
  },
  {
    "startMs": 159599,
    "text": "across sentences we can observe conflicts and gaps in the specification"
  },
  {
    "startMs": 162150,
    "text": "conflicts and gaps in the specification"
  },
  {
    "startMs": 162160,
    "text": "conflicts and gaps in the specification or rules. Right? sentences need reviews"
  },
  {
    "startMs": 165509,
    "text": "or rules. Right? sentences need reviews"
  },
  {
    "startMs": 165519,
    "text": "or rules. Right? sentences need reviews that need search in between them and"
  },
  {
    "startMs": 167990,
    "text": "that need search in between them and"
  },
  {
    "startMs": 168000,
    "text": "that need search in between them and ambiguity searches."
  },
  {
    "startMs": 170550,
    "text": "ambiguity searches."
  },
  {
    "startMs": 170560,
    "text": "ambiguity searches. So we have a three-step process to do"
  },
  {
    "startMs": 173350,
    "text": "So we have a three-step process to do"
  },
  {
    "startMs": 173360,
    "text": "So we have a three-step process to do this where the LLM will first refine it."
  },
  {
    "startMs": 176390,
    "text": "this where the LLM will first refine it."
  },
  {
    "startMs": 176400,
    "text": "this where the LLM will first refine it. It'll try to clean up the wording on its"
  },
  {
    "startMs": 177990,
    "text": "It'll try to clean up the wording on its"
  },
  {
    "startMs": 178000,
    "text": "It'll try to clean up the wording on its own and u if it succeeds then well and"
  },
  {
    "startMs": 181670,
    "text": "own and u if it succeeds then well and"
  },
  {
    "startMs": 181680,
    "text": "own and u if it succeeds then well and good but if it cannot we then formalize"
  },
  {
    "startMs": 184869,
    "text": "good but if it cannot we then formalize"
  },
  {
    "startMs": 184879,
    "text": "good but if it cannot we then formalize the um English language into SMT"
  },
  {
    "startMs": 188630,
    "text": "the um English language into SMT"
  },
  {
    "startMs": 188640,
    "text": "the um English language into SMT formulas. So, SMT lib is a certain set"
  },
  {
    "startMs": 192710,
    "text": "formulas. So, SMT lib is a certain set"
  },
  {
    "startMs": 192720,
    "text": "formulas. So, SMT lib is a certain set of type of function which we pass to a"
  },
  {
    "startMs": 195270,
    "text": "of type of function which we pass to a"
  },
  {
    "startMs": 195280,
    "text": "of type of function which we pass to a SAT solver a satisfiability checker"
  },
  {
    "startMs": 197830,
    "text": "SAT solver a satisfiability checker"
  },
  {
    "startMs": 197840,
    "text": "SAT solver a satisfiability checker where it's going to give out a boolean"
  },
  {
    "startMs": 199509,
    "text": "where it's going to give out a boolean"
  },
  {
    "startMs": 199519,
    "text": "where it's going to give out a boolean saying this expression that you've given"
  },
  {
    "startMs": 202229,
    "text": "saying this expression that you've given"
  },
  {
    "startMs": 202239,
    "text": "saying this expression that you've given me is valid true or false invalid"
  },
  {
    "startMs": 207110,
    "text": "me is valid true or false invalid"
  },
  {
    "startMs": 207120,
    "text": "me is valid true or false invalid and then once the solver gets back to us"
  },
  {
    "startMs": 209589,
    "text": "and then once the solver gets back to us"
  },
  {
    "startMs": 209599,
    "text": "and then once the solver gets back to us if that expression is valid then we"
  },
  {
    "startMs": 211750,
    "text": "if that expression is valid then we"
  },
  {
    "startMs": 211760,
    "text": "if that expression is valid then we don't do anything. But if it detects any"
  },
  {
    "startMs": 213430,
    "text": "don't do anything. But if it detects any"
  },
  {
    "startMs": 213440,
    "text": "don't do anything. But if it detects any ambiguity between two sentences or"
  },
  {
    "startMs": 215509,
    "text": "ambiguity between two sentences or"
  },
  {
    "startMs": 215519,
    "text": "ambiguity between two sentences or within a sentence where there's two"
  },
  {
    "startMs": 217830,
    "text": "within a sentence where there's two"
  },
  {
    "startMs": 217840,
    "text": "within a sentence where there's two meanings to a sentence, then we detect"
  },
  {
    "startMs": 219750,
    "text": "meanings to a sentence, then we detect"
  },
  {
    "startMs": 219760,
    "text": "meanings to a sentence, then we detect that and tell the LLM to recheck that"
  },
  {
    "startMs": 223670,
    "text": "that and tell the LLM to recheck that"
  },
  {
    "startMs": 223680,
    "text": "that and tell the LLM to recheck that sentence and it's going to reiterate on"
  },
  {
    "startMs": 225830,
    "text": "sentence and it's going to reiterate on"
  },
  {
    "startMs": 225840,
    "text": "sentence and it's going to reiterate on that."
  },
  {
    "startMs": 227750,
    "text": "that."
  },
  {
    "startMs": 227760,
    "text": "that. For example, the rules we saw"
  },
  {
    "startMs": 231350,
    "text": "For example, the rules we saw"
  },
  {
    "startMs": 231360,
    "text": "For example, the rules we saw we've seen when order is submitted and"
  },
  {
    "startMs": 233430,
    "text": "we've seen when order is submitted and"
  },
  {
    "startMs": 233440,
    "text": "we've seen when order is submitted and inventory is unavailable, we shall place"
  },
  {
    "startMs": 235910,
    "text": "inventory is unavailable, we shall place"
  },
  {
    "startMs": 235920,
    "text": "inventory is unavailable, we shall place the order as a back order. And in SMT"
  },
  {
    "startMs": 239670,
    "text": "the order as a back order. And in SMT"
  },
  {
    "startMs": 239680,
    "text": "the order as a back order. And in SMT format, it kind of looks like this. This"
  },
  {
    "startMs": 241350,
    "text": "format, it kind of looks like this. This"
  },
  {
    "startMs": 241360,
    "text": "format, it kind of looks like this. This is like a math formula, right?"
  },
  {
    "startMs": 248560,
    "text": "Once we go deeper, we"
  },
  {
    "startMs": 251270,
    "text": "Once we go deeper, we"
  },
  {
    "startMs": 251280,
    "text": "Once we go deeper, we once we detect this, we also do a"
  },
  {
    "startMs": 254470,
    "text": "once we detect this, we also do a"
  },
  {
    "startMs": 254480,
    "text": "once we detect this, we also do a three-step process again where we sample"
  },
  {
    "startMs": 257189,
    "text": "three-step process again where we sample"
  },
  {
    "startMs": 257199,
    "text": "three-step process again where we sample the translations again and again. We ask"
  },
  {
    "startMs": 259749,
    "text": "the translations again and again. We ask"
  },
  {
    "startMs": 259759,
    "text": "the translations again and again. We ask the LLM to translate it seven times or"
  },
  {
    "startMs": 261749,
    "text": "the LLM to translate it seven times or"
  },
  {
    "startMs": 261759,
    "text": "the LLM to translate it seven times or 10 times and to understand whether this"
  },
  {
    "startMs": 264870,
    "text": "10 times and to understand whether this"
  },
  {
    "startMs": 264880,
    "text": "10 times and to understand whether this LLM is behaving this way uh based on its"
  },
  {
    "startMs": 269830,
    "text": "LLM is behaving this way uh based on its"
  },
  {
    "startMs": 269840,
    "text": "LLM is behaving this way uh based on its inherent capabilities or is something"
  },
  {
    "startMs": 271670,
    "text": "inherent capabilities or is something"
  },
  {
    "startMs": 271680,
    "text": "inherent capabilities or is something affecting it to behave this way and then"
  },
  {
    "startMs": 274070,
    "text": "affecting it to behave this way and then"
  },
  {
    "startMs": 274080,
    "text": "affecting it to behave this way and then based on logical equivalence of these"
  },
  {
    "startMs": 276710,
    "text": "based on logical equivalence of these"
  },
  {
    "startMs": 276720,
    "text": "based on logical equivalence of these seven or 10 um different set of"
  },
  {
    "startMs": 279510,
    "text": "seven or 10 um different set of"
  },
  {
    "startMs": 279520,
    "text": "seven or 10 um different set of interpretations by the LLM we cluster"
  },
  {
    "startMs": 281749,
    "text": "interpretations by the LLM we cluster"
  },
  {
    "startMs": 281759,
    "text": "interpretations by the LLM we cluster them on logical equivalence which sets"
  },
  {
    "startMs": 284629,
    "text": "them on logical equivalence which sets"
  },
  {
    "startMs": 284639,
    "text": "them on logical equivalence which sets which uh um entries came under which uh"
  },
  {
    "startMs": 289670,
    "text": "which uh um entries came under which uh"
  },
  {
    "startMs": 289680,
    "text": "which uh um entries came under which uh assumptions that the LLM made for itself"
  },
  {
    "startMs": 292950,
    "text": "assumptions that the LLM made for itself"
  },
  {
    "startMs": 292960,
    "text": "assumptions that the LLM made for itself and then we accept or ask the user to"
  },
  {
    "startMs": 295909,
    "text": "and then we accept or ask the user to"
  },
  {
    "startMs": 295919,
    "text": "and then we accept or ask the user to clear out the ambiguity or abstain from"
  },
  {
    "startMs": 298469,
    "text": "clear out the ambiguity or abstain from"
  },
  {
    "startMs": 298479,
    "text": "clear out the ambiguity or abstain from implementing that in the codebase."
  },
  {
    "startMs": 305759,
    "text": "This is an example of asking the human"
  },
  {
    "startMs": 309189,
    "text": "This is an example of asking the human"
  },
  {
    "startMs": 309199,
    "text": "This is an example of asking the human um what you want to do with that spec"
  },
  {
    "startMs": 313110,
    "text": "um what you want to do with that spec"
  },
  {
    "startMs": 313120,
    "text": "um what you want to do with that spec rule that was um perhaps contradicting."
  },
  {
    "startMs": 316950,
    "text": "rule that was um perhaps contradicting."
  },
  {
    "startMs": 316960,
    "text": "rule that was um perhaps contradicting. remove the record is very a very vague"
  },
  {
    "startMs": 318870,
    "text": "remove the record is very a very vague"
  },
  {
    "startMs": 318880,
    "text": "remove the record is very a very vague statement. The LLM cannot can interpret"
  },
  {
    "startMs": 321189,
    "text": "statement. The LLM cannot can interpret"
  },
  {
    "startMs": 321199,
    "text": "statement. The LLM cannot can interpret anything it wants but we shouldn't allow"
  },
  {
    "startMs": 323909,
    "text": "anything it wants but we shouldn't allow"
  },
  {
    "startMs": 323919,
    "text": "anything it wants but we shouldn't allow that to happen and we either give two"
  },
  {
    "startMs": 326710,
    "text": "that to happen and we either give two"
  },
  {
    "startMs": 326720,
    "text": "that to happen and we either give two options we keep it as is the record is"
  },
  {
    "startMs": 328629,
    "text": "options we keep it as is the record is"
  },
  {
    "startMs": 328639,
    "text": "options we keep it as is the record is deleted or we retain it and but hide it"
  },
  {
    "startMs": 331430,
    "text": "deleted or we retain it and but hide it"
  },
  {
    "startMs": 331440,
    "text": "deleted or we retain it and but hide it from the users which one does the user"
  },
  {
    "startMs": 333510,
    "text": "from the users which one does the user"
  },
  {
    "startMs": 333520,
    "text": "from the users which one does the user prefer."
  },
  {
    "startMs": 337600,
    "text": "Um"
  },
  {
    "startMs": 339430,
    "text": "Um"
  },
  {
    "startMs": 339440,
    "text": "Um a conflict is when the examples we"
  },
  {
    "startMs": 341909,
    "text": "a conflict is when the examples we"
  },
  {
    "startMs": 341919,
    "text": "a conflict is when the examples we example we saw rule one is conflicting"
  },
  {
    "startMs": 343830,
    "text": "example we saw rule one is conflicting"
  },
  {
    "startMs": 343840,
    "text": "example we saw rule one is conflicting with rule three and a gap is where no"
  },
  {
    "startMs": 346950,
    "text": "with rule three and a gap is where no"
  },
  {
    "startMs": 346960,
    "text": "with rule three and a gap is where no rule covers an idle order in this"
  },
  {
    "startMs": 349590,
    "text": "rule covers an idle order in this"
  },
  {
    "startMs": 349600,
    "text": "rule covers an idle order in this example where the LLM detected an actual"
  },
  {
    "startMs": 353510,
    "text": "example where the LLM detected an actual"
  },
  {
    "startMs": 353520,
    "text": "example where the LLM detected an actual sorry"
  },
  {
    "startMs": 355189,
    "text": "sorry"
  },
  {
    "startMs": 355199,
    "text": "sorry process or pipeline detected there's no"
  },
  {
    "startMs": 358710,
    "text": "process or pipeline detected there's no"
  },
  {
    "startMs": 358720,
    "text": "process or pipeline detected there's no um entry for this rule where you are"
  },
  {
    "startMs": 361189,
    "text": "um entry for this rule where you are"
  },
  {
    "startMs": 361199,
    "text": "um entry for this rule where you are missing something"
  },
  {
    "startMs": 366160,
    "text": "all in Well, it cannot do a lot of"
  },
  {
    "startMs": 368550,
    "text": "all in Well, it cannot do a lot of"
  },
  {
    "startMs": 368560,
    "text": "all in Well, it cannot do a lot of things. End of the day, we want to make"
  },
  {
    "startMs": 372070,
    "text": "things. End of the day, we want to make"
  },
  {
    "startMs": 372080,
    "text": "things. End of the day, we want to make sure there is no oracle uh for for"
  },
  {
    "startMs": 375350,
    "text": "sure there is no oracle uh for for"
  },
  {
    "startMs": 375360,
    "text": "sure there is no oracle uh for for specification correctness other than the"
  },
  {
    "startMs": 377029,
    "text": "specification correctness other than the"
  },
  {
    "startMs": 377039,
    "text": "specification correctness other than the user. User needs to make sure to check"
  },
  {
    "startMs": 380309,
    "text": "user. User needs to make sure to check"
  },
  {
    "startMs": 380319,
    "text": "user. User needs to make sure to check every part of the snippet part of the"
  },
  {
    "startMs": 383270,
    "text": "every part of the snippet part of the"
  },
  {
    "startMs": 383280,
    "text": "every part of the snippet part of the rule book before sending it to the LLM."
  },
  {
    "startMs": 385909,
    "text": "rule book before sending it to the LLM."
  },
  {
    "startMs": 385919,
    "text": "rule book before sending it to the LLM. And um we as a tool only can help them"
  },
  {
    "startMs": 391430,
    "text": "And um we as a tool only can help them"
  },
  {
    "startMs": 391440,
    "text": "And um we as a tool only can help them and the user is more important here"
  },
  {
    "startMs": 393990,
    "text": "and the user is more important here"
  },
  {
    "startMs": 394000,
    "text": "and the user is more important here because of implicit domain knowledge."
  },
  {
    "startMs": 396390,
    "text": "because of implicit domain knowledge."
  },
  {
    "startMs": 396400,
    "text": "because of implicit domain knowledge. The LLM doesn't know a lot of domain"
  },
  {
    "startMs": 397990,
    "text": "The LLM doesn't know a lot of domain"
  },
  {
    "startMs": 398000,
    "text": "The LLM doesn't know a lot of domain knowledge and there's no guarantee I"
  },
  {
    "startMs": 400150,
    "text": "knowledge and there's no guarantee I"
  },
  {
    "startMs": 400160,
    "text": "knowledge and there's no guarantee I mean LLM only uh user can give that"
  },
  {
    "startMs": 404150,
    "text": "mean LLM only uh user can give that"
  },
  {
    "startMs": 404160,
    "text": "mean LLM only uh user can give that guarantee right and there's thin"
  },
  {
    "startMs": 406309,
    "text": "guarantee right and there's thin"
  },
  {
    "startMs": 406319,
    "text": "guarantee right and there's thin evidence right now for formalization and"
  },
  {
    "startMs": 408309,
    "text": "evidence right now for formalization and"
  },
  {
    "startMs": 408319,
    "text": "evidence right now for formalization and how it can be used in agents we are"
  },
  {
    "startMs": 410710,
    "text": "how it can be used in agents we are"
  },
  {
    "startMs": 410720,
    "text": "how it can be used in agents we are emerging in this technology"
  },
  {
    "startMs": 416960,
    "text": "what do we take back from this um talk"
  },
  {
    "startMs": 420309,
    "text": "what do we take back from this um talk"
  },
  {
    "startMs": 420319,
    "text": "what do we take back from this um talk check the spec always before you pass it"
  },
  {
    "startMs": 422710,
    "text": "check the spec always before you pass it"
  },
  {
    "startMs": 422720,
    "text": "check the spec always before you pass it down to the agent uh cluster the"
  },
  {
    "startMs": 425589,
    "text": "down to the agent uh cluster the"
  },
  {
    "startMs": 425599,
    "text": "down to the agent uh cluster the translation We what we do is we cluster"
  },
  {
    "startMs": 427909,
    "text": "translation We what we do is we cluster"
  },
  {
    "startMs": 427919,
    "text": "translation We what we do is we cluster the translation by logical equivalent to"
  },
  {
    "startMs": 431029,
    "text": "the translation by logical equivalent to"
  },
  {
    "startMs": 431039,
    "text": "the translation by logical equivalent to determine the different LLM"
  },
  {
    "startMs": 433350,
    "text": "determine the different LLM"
  },
  {
    "startMs": 433360,
    "text": "determine the different LLM interpretations and where it's genuinely"
  },
  {
    "startMs": 435589,
    "text": "interpretations and where it's genuinely"
  },
  {
    "startMs": 435599,
    "text": "interpretations and where it's genuinely going wrong. The LLM translates, the"
  },
  {
    "startMs": 438469,
    "text": "going wrong. The LLM translates, the"
  },
  {
    "startMs": 438479,
    "text": "going wrong. The LLM translates, the solver checks and the human decides. We"
  },
  {
    "startMs": 441749,
    "text": "solver checks and the human decides. We"
  },
  {
    "startMs": 441759,
    "text": "solver checks and the human decides. We give the human uh options and if there's"
  },
  {
    "startMs": 444950,
    "text": "give the human uh options and if there's"
  },
  {
    "startMs": 444960,
    "text": "give the human uh options and if there's no answer to that, we use LLM judges or"
  },
  {
    "startMs": 447749,
    "text": "no answer to that, we use LLM judges or"
  },
  {
    "startMs": 447759,
    "text": "no answer to that, we use LLM judges or abstain from implementing that piece."
  },
  {
    "startMs": 452070,
    "text": "abstain from implementing that piece."
  },
  {
    "startMs": 452080,
    "text": "abstain from implementing that piece. Here are the references and u this work"
  },
  {
    "startMs": 455430,
    "text": "Here are the references and u this work"
  },
  {
    "startMs": 455440,
    "text": "Here are the references and u this work has been majorly contributed by my team"
  },
  {
    "startMs": 458629,
    "text": "has been majorly contributed by my team"
  },
  {
    "startMs": 458639,
    "text": "has been majorly contributed by my team um Andrea Jin Stefan and Delmos. You can"
  },
  {
    "startMs": 462309,
    "text": "um Andrea Jin Stefan and Delmos. You can"
  },
  {
    "startMs": 462319,
    "text": "um Andrea Jin Stefan and Delmos. You can check out their blog in the website link"
  },
  {
    "startMs": 465110,
    "text": "check out their blog in the website link"
  },
  {
    "startMs": 465120,
    "text": "check out their blog in the website link provided and um del deeper into this."
  },
  {
    "startMs": 469029,
    "text": "provided and um del deeper into this."
  },
  {
    "startMs": 469039,
    "text": "provided and um del deeper into this. Thank you."
  }
]