[
  {
    "startMs": 1199,
    "text": "Uh hello everyone. Uh so my name is uh"
  },
  {
    "startMs": 4150,
    "text": "Uh hello everyone. Uh so my name is uh"
  },
  {
    "startMs": 4160,
    "text": "Uh hello everyone. Uh so my name is uh Zui and today uh together with uh Saloni"
  },
  {
    "startMs": 7430,
    "text": "Zui and today uh together with uh Saloni"
  },
  {
    "startMs": 7440,
    "text": "Zui and today uh together with uh Saloni so we are going to present the recursive"
  },
  {
    "startMs": 9750,
    "text": "so we are going to present the recursive"
  },
  {
    "startMs": 9760,
    "text": "so we are going to present the recursive self-improvement uh project that we are"
  },
  {
    "startMs": 12150,
    "text": "self-improvement uh project that we are"
  },
  {
    "startMs": 12160,
    "text": "self-improvement uh project that we are doing at linking and for this RSI work"
  },
  {
    "startMs": 15509,
    "text": "doing at linking and for this RSI work"
  },
  {
    "startMs": 15519,
    "text": "doing at linking and for this RSI work we focus on the user understanding and"
  },
  {
    "startMs": 19029,
    "text": "we focus on the user understanding and"
  },
  {
    "startMs": 19039,
    "text": "we focus on the user understanding and yeah super excited to share the uh RSI"
  },
  {
    "startMs": 22230,
    "text": "yeah super excited to share the uh RSI"
  },
  {
    "startMs": 22240,
    "text": "yeah super excited to share the uh RSI work we did at linking. So first uh just"
  },
  {
    "startMs": 25429,
    "text": "work we did at linking. So first uh just"
  },
  {
    "startMs": 25439,
    "text": "work we did at linking. So first uh just a high level or overview on what is a"
  },
  {
    "startMs": 27750,
    "text": "a high level or overview on what is a"
  },
  {
    "startMs": 27760,
    "text": "a high level or overview on what is a recursive self-improvement."
  },
  {
    "startMs": 30070,
    "text": "recursive self-improvement."
  },
  {
    "startMs": 30080,
    "text": "recursive self-improvement. So the recursive self-improvement is a"
  },
  {
    "startMs": 31990,
    "text": "So the recursive self-improvement is a"
  },
  {
    "startMs": 32000,
    "text": "So the recursive self-improvement is a build around some uh AI system and uh"
  },
  {
    "startMs": 34870,
    "text": "build around some uh AI system and uh"
  },
  {
    "startMs": 34880,
    "text": "build around some uh AI system and uh this is the purpose of doing this is we"
  },
  {
    "startMs": 37430,
    "text": "this is the purpose of doing this is we"
  },
  {
    "startMs": 37440,
    "text": "this is the purpose of doing this is we want to get a system that"
  },
  {
    "startMs": 39590,
    "text": "want to get a system that"
  },
  {
    "startMs": 39600,
    "text": "want to get a system that self-improvements uh self-improves. So"
  },
  {
    "startMs": 42150,
    "text": "self-improvements uh self-improves. So"
  },
  {
    "startMs": 42160,
    "text": "self-improvements uh self-improves. So it can learns from issues in production"
  },
  {
    "startMs": 44389,
    "text": "it can learns from issues in production"
  },
  {
    "startMs": 44399,
    "text": "it can learns from issues in production trace and that work. So the AI system"
  },
  {
    "startMs": 47750,
    "text": "trace and that work. So the AI system"
  },
  {
    "startMs": 47760,
    "text": "trace and that work. So the AI system can be some uh agent and I linking. So"
  },
  {
    "startMs": 50869,
    "text": "can be some uh agent and I linking. So"
  },
  {
    "startMs": 50879,
    "text": "can be some uh agent and I linking. So one example is uh our career uh agent to"
  },
  {
    "startMs": 54310,
    "text": "one example is uh our career uh agent to"
  },
  {
    "startMs": 54320,
    "text": "one example is uh our career uh agent to help uh our uh linking users members to"
  },
  {
    "startMs": 57670,
    "text": "help uh our uh linking users members to"
  },
  {
    "startMs": 57680,
    "text": "help uh our uh linking users members to achieve their career goals and from the"
  },
  {
    "startMs": 60470,
    "text": "achieve their career goals and from the"
  },
  {
    "startMs": 60480,
    "text": "achieve their career goals and from the agent so uh and there's uh many daily"
  },
  {
    "startMs": 63910,
    "text": "agent so uh and there's uh many daily"
  },
  {
    "startMs": 63920,
    "text": "agent so uh and there's uh many daily development work that we discover the"
  },
  {
    "startMs": 66310,
    "text": "development work that we discover the"
  },
  {
    "startMs": 66320,
    "text": "development work that we discover the issues we want to address and also we"
  },
  {
    "startMs": 69350,
    "text": "issues we want to address and also we"
  },
  {
    "startMs": 69360,
    "text": "issues we want to address and also we will get the production traces and that"
  },
  {
    "startMs": 72390,
    "text": "will get the production traces and that"
  },
  {
    "startMs": 72400,
    "text": "will get the production traces and that we can uh so they like we can log the"
  },
  {
    "startMs": 74710,
    "text": "we can uh so they like we can log the"
  },
  {
    "startMs": 74720,
    "text": "we can uh so they like we can log the issues we found during the production"
  },
  {
    "startMs": 76710,
    "text": "issues we found during the production"
  },
  {
    "startMs": 76720,
    "text": "issues we found during the production traffic and once we have this"
  },
  {
    "startMs": 79350,
    "text": "traffic and once we have this"
  },
  {
    "startMs": 79360,
    "text": "traffic and once we have this informations So we will have some"
  },
  {
    "startMs": 81109,
    "text": "informations So we will have some"
  },
  {
    "startMs": 81119,
    "text": "informations So we will have some evaluation engine for example the ar"
  },
  {
    "startMs": 83270,
    "text": "evaluation engine for example the ar"
  },
  {
    "startMs": 83280,
    "text": "evaluation engine for example the ar judge and to uh check our uh traces and"
  },
  {
    "startMs": 87270,
    "text": "judge and to uh check our uh traces and"
  },
  {
    "startMs": 87280,
    "text": "judge and to uh check our uh traces and to identify the issues and gives"
  },
  {
    "startMs": 89350,
    "text": "to identify the issues and gives"
  },
  {
    "startMs": 89360,
    "text": "to identify the issues and gives feedbacks and based on the feedbacks and"
  },
  {
    "startMs": 92310,
    "text": "feedbacks and based on the feedbacks and"
  },
  {
    "startMs": 92320,
    "text": "feedbacks and based on the feedbacks and we will uh use those feedback to keep uh"
  },
  {
    "startMs": 95590,
    "text": "we will uh use those feedback to keep uh"
  },
  {
    "startMs": 95600,
    "text": "we will uh use those feedback to keep uh continuously uh iterate our system. So"
  },
  {
    "startMs": 98950,
    "text": "continuously uh iterate our system. So"
  },
  {
    "startMs": 98960,
    "text": "continuously uh iterate our system. So we build this kind of like a loop to"
  },
  {
    "startMs": 101030,
    "text": "we build this kind of like a loop to"
  },
  {
    "startMs": 101040,
    "text": "we build this kind of like a loop to keep uh improve our um AI system. And uh"
  },
  {
    "startMs": 105270,
    "text": "keep uh improve our um AI system. And uh"
  },
  {
    "startMs": 105280,
    "text": "keep uh improve our um AI system. And uh one key thing we want to uh call out is"
  },
  {
    "startMs": 107910,
    "text": "one key thing we want to uh call out is"
  },
  {
    "startMs": 107920,
    "text": "one key thing we want to uh call out is uh during this uh recursive self"
  },
  {
    "startMs": 109749,
    "text": "uh during this uh recursive self"
  },
  {
    "startMs": 109759,
    "text": "uh during this uh recursive self improvement process the evaluation is"
  },
  {
    "startMs": 112710,
    "text": "improvement process the evaluation is"
  },
  {
    "startMs": 112720,
    "text": "improvement process the evaluation is very critical and it must be objective"
  },
  {
    "startMs": 115190,
    "text": "very critical and it must be objective"
  },
  {
    "startMs": 115200,
    "text": "very critical and it must be objective and well aligned with the user feedback"
  },
  {
    "startMs": 117510,
    "text": "and well aligned with the user feedback"
  },
  {
    "startMs": 117520,
    "text": "and well aligned with the user feedback because this is a key to uh like"
  },
  {
    "startMs": 119749,
    "text": "because this is a key to uh like"
  },
  {
    "startMs": 119759,
    "text": "because this is a key to uh like identify where the bottleneck is and"
  },
  {
    "startMs": 121830,
    "text": "identify where the bottleneck is and"
  },
  {
    "startMs": 121840,
    "text": "identify where the bottleneck is and guide the next step for the uh system AI"
  },
  {
    "startMs": 124709,
    "text": "guide the next step for the uh system AI"
  },
  {
    "startMs": 124719,
    "text": "guide the next step for the uh system AI system improvement."
  },
  {
    "startMs": 126950,
    "text": "system improvement."
  },
  {
    "startMs": 126960,
    "text": "system improvement. And uh next we want to uh uh like"
  },
  {
    "startMs": 129749,
    "text": "And uh next we want to uh uh like"
  },
  {
    "startMs": 129759,
    "text": "And uh next we want to uh uh like describe like show uh what kind of like"
  },
  {
    "startMs": 132309,
    "text": "describe like show uh what kind of like"
  },
  {
    "startMs": 132319,
    "text": "describe like show uh what kind of like uh which component of the AI system we"
  },
  {
    "startMs": 134949,
    "text": "uh which component of the AI system we"
  },
  {
    "startMs": 134959,
    "text": "uh which component of the AI system we can uh optimize. Uh so here take the"
  },
  {
    "startMs": 138229,
    "text": "can uh optimize. Uh so here take the"
  },
  {
    "startMs": 138239,
    "text": "can uh optimize. Uh so here take the harness uh as as an example or like as"
  },
  {
    "startMs": 141750,
    "text": "harness uh as as an example or like as"
  },
  {
    "startMs": 141760,
    "text": "harness uh as as an example or like as orchestrator. So there are different"
  },
  {
    "startMs": 143670,
    "text": "orchestrator. So there are different"
  },
  {
    "startMs": 143680,
    "text": "orchestrator. So there are different components we can uh use we can uh"
  },
  {
    "startMs": 146949,
    "text": "components we can uh use we can uh"
  },
  {
    "startMs": 146959,
    "text": "components we can uh use we can uh optimize by using the RSI. So for"
  },
  {
    "startMs": 149430,
    "text": "optimize by using the RSI. So for"
  },
  {
    "startMs": 149440,
    "text": "optimize by using the RSI. So for example the first is a planning and"
  },
  {
    "startMs": 151430,
    "text": "example the first is a planning and"
  },
  {
    "startMs": 151440,
    "text": "example the first is a planning and logics which is used to orchestrator"
  },
  {
    "startMs": 154229,
    "text": "logics which is used to orchestrator"
  },
  {
    "startMs": 154239,
    "text": "logics which is used to orchestrator orchestrates execution and make a"
  },
  {
    "startMs": 156390,
    "text": "orchestrates execution and make a"
  },
  {
    "startMs": 156400,
    "text": "orchestrates execution and make a decision uh past and the second is a"
  },
  {
    "startMs": 158949,
    "text": "decision uh past and the second is a"
  },
  {
    "startMs": 158959,
    "text": "decision uh past and the second is a context and the memory where we preserve"
  },
  {
    "startMs": 161750,
    "text": "context and the memory where we preserve"
  },
  {
    "startMs": 161760,
    "text": "context and the memory where we preserve and manage the session state and the"
  },
  {
    "startMs": 164309,
    "text": "and manage the session state and the"
  },
  {
    "startMs": 164319,
    "text": "and manage the session state and the third one is a two interface that"
  },
  {
    "startMs": 166790,
    "text": "third one is a two interface that"
  },
  {
    "startMs": 166800,
    "text": "third one is a two interface that interacts with external and internal"
  },
  {
    "startMs": 168550,
    "text": "interacts with external and internal"
  },
  {
    "startMs": 168560,
    "text": "interacts with external and internal APIs and data sets. The last uh last but"
  },
  {
    "startMs": 171750,
    "text": "APIs and data sets. The last uh last but"
  },
  {
    "startMs": 171760,
    "text": "APIs and data sets. The last uh last but not least is artifact storage uh that"
  },
  {
    "startMs": 174790,
    "text": "not least is artifact storage uh that"
  },
  {
    "startMs": 174800,
    "text": "not least is artifact storage uh that where we stores outputs and the"
  },
  {
    "startMs": 176390,
    "text": "where we stores outputs and the"
  },
  {
    "startMs": 176400,
    "text": "where we stores outputs and the generation logs and so they like uh and"
  },
  {
    "startMs": 179670,
    "text": "generation logs and so they like uh and"
  },
  {
    "startMs": 179680,
    "text": "generation logs and so they like uh and we are on top of those we definitely"
  },
  {
    "startMs": 181509,
    "text": "we are on top of those we definitely"
  },
  {
    "startMs": 181519,
    "text": "we are on top of those we definitely have some like evaluation uh logic and"
  },
  {
    "startMs": 184149,
    "text": "have some like evaluation uh logic and"
  },
  {
    "startMs": 184159,
    "text": "have some like evaluation uh logic and uh so that to help us to evaluate uh"
  },
  {
    "startMs": 186470,
    "text": "uh so that to help us to evaluate uh"
  },
  {
    "startMs": 186480,
    "text": "uh so that to help us to evaluate uh each um component and help us to keep uh"
  },
  {
    "startMs": 189509,
    "text": "each um component and help us to keep uh"
  },
  {
    "startMs": 189519,
    "text": "each um component and help us to keep uh improve the uh AI system and we want to"
  },
  {
    "startMs": 193430,
    "text": "improve the uh AI system and we want to"
  },
  {
    "startMs": 193440,
    "text": "improve the uh AI system and we want to uh like uh uh and another thing to uh"
  },
  {
    "startMs": 196070,
    "text": "uh like uh uh and another thing to uh"
  },
  {
    "startMs": 196080,
    "text": "uh like uh uh and another thing to uh keep in mind that all those uh like"
  },
  {
    "startMs": 198869,
    "text": "keep in mind that all those uh like"
  },
  {
    "startMs": 198879,
    "text": "keep in mind that all those uh like harness is a powered by the base uh iron"
  },
  {
    "startMs": 202070,
    "text": "harness is a powered by the base uh iron"
  },
  {
    "startMs": 202080,
    "text": "harness is a powered by the base uh iron model but for the RSI uh so it's"
  },
  {
    "startMs": 206149,
    "text": "model but for the RSI uh so it's"
  },
  {
    "startMs": 206159,
    "text": "model but for the RSI uh so it's currently the optimize of this base"
  },
  {
    "startMs": 209270,
    "text": "currently the optimize of this base"
  },
  {
    "startMs": 209280,
    "text": "currently the optimize of this base model uh itself is not in uh scope but"
  },
  {
    "startMs": 212710,
    "text": "model uh itself is not in uh scope but"
  },
  {
    "startMs": 212720,
    "text": "model uh itself is not in uh scope but we focus on the uh harness optimization"
  },
  {
    "startMs": 216470,
    "text": "we focus on the uh harness optimization"
  },
  {
    "startMs": 216480,
    "text": "we focus on the uh harness optimization uh first"
  },
  {
    "startMs": 218309,
    "text": "uh first"
  },
  {
    "startMs": 218319,
    "text": "uh first and uh here we want to uh like show uh"
  },
  {
    "startMs": 221430,
    "text": "and uh here we want to uh like show uh"
  },
  {
    "startMs": 221440,
    "text": "and uh here we want to uh like show uh like this is the ladders that we can"
  },
  {
    "startMs": 223750,
    "text": "like this is the ladders that we can"
  },
  {
    "startMs": 223760,
    "text": "like this is the ladders that we can climb to optimize the uh AI system by"
  },
  {
    "startMs": 226550,
    "text": "climb to optimize the uh AI system by"
  },
  {
    "startMs": 226560,
    "text": "climb to optimize the uh AI system by using the uh RSI. So uh at the beginning"
  },
  {
    "startMs": 230149,
    "text": "using the uh RSI. So uh at the beginning"
  },
  {
    "startMs": 230159,
    "text": "using the uh RSI. So uh at the beginning so typically so this is a start from"
  },
  {
    "startMs": 232390,
    "text": "so typically so this is a start from"
  },
  {
    "startMs": 232400,
    "text": "so typically so this is a start from simple uh process at the beginning we"
  },
  {
    "startMs": 234550,
    "text": "simple uh process at the beginning we"
  },
  {
    "startMs": 234560,
    "text": "simple uh process at the beginning we can focus on the prompts uh updates"
  },
  {
    "startMs": 237110,
    "text": "can focus on the prompts uh updates"
  },
  {
    "startMs": 237120,
    "text": "can focus on the prompts uh updates because this is a most uh this is the"
  },
  {
    "startMs": 239589,
    "text": "because this is a most uh this is the"
  },
  {
    "startMs": 239599,
    "text": "because this is a most uh this is the easiest and the most straightforward way"
  },
  {
    "startMs": 242229,
    "text": "easiest and the most straightforward way"
  },
  {
    "startMs": 242239,
    "text": "easiest and the most straightforward way to uh keep iterate our AI system and"
  },
  {
    "startMs": 246070,
    "text": "to uh keep iterate our AI system and"
  },
  {
    "startMs": 246080,
    "text": "to uh keep iterate our AI system and after that uh we can uh continue to"
  },
  {
    "startMs": 248869,
    "text": "after that uh we can uh continue to"
  },
  {
    "startMs": 248879,
    "text": "after that uh we can uh continue to optimize the structured context of the"
  },
  {
    "startMs": 251589,
    "text": "optimize the structured context of the"
  },
  {
    "startMs": 251599,
    "text": "optimize the structured context of the AI system and uh uh the and the for the"
  },
  {
    "startMs": 255589,
    "text": "AI system and uh uh the and the for the"
  },
  {
    "startMs": 255599,
    "text": "AI system and uh uh the and the for the this is the L1 uh ladder and for the L2"
  },
  {
    "startMs": 258229,
    "text": "this is the L1 uh ladder and for the L2"
  },
  {
    "startMs": 258239,
    "text": "this is the L1 uh ladder and for the L2 tool ladder and which is after the"
  },
  {
    "startMs": 260390,
    "text": "tool ladder and which is after the"
  },
  {
    "startMs": 260400,
    "text": "tool ladder and which is after the structure context optimization. We can"
  },
  {
    "startMs": 263110,
    "text": "structure context optimization. We can"
  },
  {
    "startMs": 263120,
    "text": "structure context optimization. We can uh expand our scope to design the auto"
  },
  {
    "startMs": 266550,
    "text": "uh expand our scope to design the auto"
  },
  {
    "startMs": 266560,
    "text": "uh expand our scope to design the auto automatically design the entire uh"
  },
  {
    "startMs": 268710,
    "text": "automatically design the entire uh"
  },
  {
    "startMs": 268720,
    "text": "automatically design the entire uh workflow and uh uh so this is kind of"
  },
  {
    "startMs": 271909,
    "text": "workflow and uh uh so this is kind of"
  },
  {
    "startMs": 271919,
    "text": "workflow and uh uh so this is kind of like uh like a meta information uh"
  },
  {
    "startMs": 274870,
    "text": "like uh like a meta information uh"
  },
  {
    "startMs": 274880,
    "text": "like uh like a meta information uh design and uh after that and we can"
  },
  {
    "startMs": 277830,
    "text": "design and uh after that and we can"
  },
  {
    "startMs": 277840,
    "text": "design and uh after that and we can automize the harness code uh itself and"
  },
  {
    "startMs": 280870,
    "text": "automize the harness code uh itself and"
  },
  {
    "startMs": 280880,
    "text": "automize the harness code uh itself and as at the end we can uh go very"
  },
  {
    "startMs": 283510,
    "text": "as at the end we can uh go very"
  },
  {
    "startMs": 283520,
    "text": "as at the end we can uh go very compactly to optimize the optimizer code"
  },
  {
    "startMs": 286150,
    "text": "compactly to optimize the optimizer code"
  },
  {
    "startMs": 286160,
    "text": "compactly to optimize the optimizer code uh itself. So this is kind of the"
  },
  {
    "startMs": 288469,
    "text": "uh itself. So this is kind of the"
  },
  {
    "startMs": 288479,
    "text": "uh itself. So this is kind of the different optimization ladders uh with"
  },
  {
    "startMs": 291590,
    "text": "different optimization ladders uh with"
  },
  {
    "startMs": 291600,
    "text": "different optimization ladders uh with increasing complexity and uh uh for our"
  },
  {
    "startMs": 295110,
    "text": "increasing complexity and uh uh for our"
  },
  {
    "startMs": 295120,
    "text": "increasing complexity and uh uh for our project. So currently uh our active"
  },
  {
    "startMs": 297510,
    "text": "project. So currently uh our active"
  },
  {
    "startMs": 297520,
    "text": "project. So currently uh our active focus is a prompt uh optimization but uh"
  },
  {
    "startMs": 301270,
    "text": "focus is a prompt uh optimization but uh"
  },
  {
    "startMs": 301280,
    "text": "focus is a prompt uh optimization but uh uh definitely like moving forward we are"
  },
  {
    "startMs": 303110,
    "text": "uh definitely like moving forward we are"
  },
  {
    "startMs": 303120,
    "text": "uh definitely like moving forward we are extend our scope to uh more and more"
  },
  {
    "startMs": 306070,
    "text": "extend our scope to uh more and more"
  },
  {
    "startMs": 306080,
    "text": "extend our scope to uh more and more complicated uh optimization uh in future"
  },
  {
    "startMs": 309830,
    "text": "complicated uh optimization uh in future"
  },
  {
    "startMs": 309840,
    "text": "complicated uh optimization uh in future and uh here we want to show our uh uh"
  },
  {
    "startMs": 313189,
    "text": "and uh here we want to show our uh uh"
  },
  {
    "startMs": 313199,
    "text": "and uh here we want to show our uh uh RSI uh strategy. So we break down our uh"
  },
  {
    "startMs": 317029,
    "text": "RSI uh strategy. So we break down our uh"
  },
  {
    "startMs": 317039,
    "text": "RSI uh strategy. So we break down our uh RSI uh uh process uh into seven uh"
  },
  {
    "startMs": 320790,
    "text": "RSI uh uh process uh into seven uh"
  },
  {
    "startMs": 320800,
    "text": "RSI uh uh process uh into seven uh steps. So the first step is to uh run"
  },
  {
    "startMs": 323909,
    "text": "steps. So the first step is to uh run"
  },
  {
    "startMs": 323919,
    "text": "steps. So the first step is to uh run the agent and uh generate some uh"
  },
  {
    "startMs": 326469,
    "text": "the agent and uh generate some uh"
  },
  {
    "startMs": 326479,
    "text": "the agent and uh generate some uh samples. So basically like for our"
  },
  {
    "startMs": 329350,
    "text": "samples. So basically like for our"
  },
  {
    "startMs": 329360,
    "text": "samples. So basically like for our career agent uh that we are optimizing"
  },
  {
    "startMs": 331830,
    "text": "career agent uh that we are optimizing"
  },
  {
    "startMs": 331840,
    "text": "career agent uh that we are optimizing for. So given a query we'll generate"
  },
  {
    "startMs": 334390,
    "text": "for. So given a query we'll generate"
  },
  {
    "startMs": 334400,
    "text": "for. So given a query we'll generate some uh output and we will lock those uh"
  },
  {
    "startMs": 337430,
    "text": "some uh output and we will lock those uh"
  },
  {
    "startMs": 337440,
    "text": "some uh output and we will lock those uh process in the uh traces for example the"
  },
  {
    "startMs": 339990,
    "text": "process in the uh traces for example the"
  },
  {
    "startMs": 340000,
    "text": "process in the uh traces for example the last miss traces and uh once so this is"
  },
  {
    "startMs": 343510,
    "text": "last miss traces and uh once so this is"
  },
  {
    "startMs": 343520,
    "text": "last miss traces and uh once so this is basically just the uh like the uh kind"
  },
  {
    "startMs": 345909,
    "text": "basically just the uh like the uh kind"
  },
  {
    "startMs": 345919,
    "text": "basically just the uh like the uh kind of the agent uh trajectories and the"
  },
  {
    "startMs": 348230,
    "text": "of the agent uh trajectories and the"
  },
  {
    "startMs": 348240,
    "text": "of the agent uh trajectories and the traces that we can uh uh use to uh"
  },
  {
    "startMs": 351110,
    "text": "traces that we can uh uh use to uh"
  },
  {
    "startMs": 351120,
    "text": "traces that we can uh uh use to uh optimize and once we got those"
  },
  {
    "startMs": 353830,
    "text": "optimize and once we got those"
  },
  {
    "startMs": 353840,
    "text": "optimize and once we got those information the next step step two is we"
  },
  {
    "startMs": 356870,
    "text": "information the next step step two is we"
  },
  {
    "startMs": 356880,
    "text": "information the next step step two is we uh runs through the evation uh engine"
  },
  {
    "startMs": 359590,
    "text": "uh runs through the evation uh engine"
  },
  {
    "startMs": 359600,
    "text": "uh runs through the evation uh engine for example the judge and to uh get some"
  },
  {
    "startMs": 362790,
    "text": "for example the judge and to uh get some"
  },
  {
    "startMs": 362800,
    "text": "for example the judge and to uh get some judge uh output to tell us uh where we"
  },
  {
    "startMs": 366070,
    "text": "judge uh output to tell us uh where we"
  },
  {
    "startMs": 366080,
    "text": "judge uh output to tell us uh where we are uh good at and we are where we are"
  },
  {
    "startMs": 368469,
    "text": "are uh good at and we are where we are"
  },
  {
    "startMs": 368479,
    "text": "are uh good at and we are where we are not good at. So to get the uh uh future"
  },
  {
    "startMs": 371830,
    "text": "not good at. So to get the uh uh future"
  },
  {
    "startMs": 371840,
    "text": "not good at. So to get the uh uh future iteration and then uh once we get the uh"
  },
  {
    "startMs": 375749,
    "text": "iteration and then uh once we get the uh"
  },
  {
    "startMs": 375759,
    "text": "iteration and then uh once we get the uh um judge output then we go uh we we are"
  },
  {
    "startMs": 379270,
    "text": "um judge output then we go uh we we are"
  },
  {
    "startMs": 379280,
    "text": "um judge output then we go uh we we are going to the RSI components and uh"
  },
  {
    "startMs": 382230,
    "text": "going to the RSI components and uh"
  },
  {
    "startMs": 382240,
    "text": "going to the RSI components and uh starting from the uh step three. So this"
  },
  {
    "startMs": 384469,
    "text": "starting from the uh step three. So this"
  },
  {
    "startMs": 384479,
    "text": "starting from the uh step three. So this is a kind of the RSI in the inside the"
  },
  {
    "startMs": 386629,
    "text": "is a kind of the RSI in the inside the"
  },
  {
    "startMs": 386639,
    "text": "is a kind of the RSI in the inside the RSI loop and the first step is we will"
  },
  {
    "startMs": 390070,
    "text": "RSI loop and the first step is we will"
  },
  {
    "startMs": 390080,
    "text": "RSI loop and the first step is we will uh we will do the agent reflection uh"
  },
  {
    "startMs": 393110,
    "text": "uh we will do the agent reflection uh"
  },
  {
    "startMs": 393120,
    "text": "uh we will do the agent reflection uh sometime we also call it a root causing"
  },
  {
    "startMs": 395110,
    "text": "sometime we also call it a root causing"
  },
  {
    "startMs": 395120,
    "text": "sometime we also call it a root causing so they like based on the judge uh"
  },
  {
    "startMs": 397189,
    "text": "so they like based on the judge uh"
  },
  {
    "startMs": 397199,
    "text": "so they like based on the judge uh output we want to understand what is a"
  },
  {
    "startMs": 399909,
    "text": "output we want to understand what is a"
  },
  {
    "startMs": 399919,
    "text": "output we want to understand what is a root cause uh that we don't have a good"
  },
  {
    "startMs": 403350,
    "text": "root cause uh that we don't have a good"
  },
  {
    "startMs": 403360,
    "text": "root cause uh that we don't have a good uh output for example is this is a agent"
  },
  {
    "startMs": 405830,
    "text": "uh output for example is this is a agent"
  },
  {
    "startMs": 405840,
    "text": "uh output for example is this is a agent issue a model issue u prompt issue or"
  },
  {
    "startMs": 409110,
    "text": "issue a model issue u prompt issue or"
  },
  {
    "startMs": 409120,
    "text": "issue a model issue u prompt issue or like a two uh two two issue or some"
  },
  {
    "startMs": 411749,
    "text": "like a two uh two two issue or some"
  },
  {
    "startMs": 411759,
    "text": "like a two uh two two issue or some other issues or sometimes it's Even it"
  },
  {
    "startMs": 414309,
    "text": "other issues or sometimes it's Even it"
  },
  {
    "startMs": 414319,
    "text": "other issues or sometimes it's Even it can even be a judge issue or rubric"
  },
  {
    "startMs": 416550,
    "text": "can even be a judge issue or rubric"
  },
  {
    "startMs": 416560,
    "text": "can even be a judge issue or rubric issue. And uh once we got the root cause"
  },
  {
    "startMs": 419430,
    "text": "issue. And uh once we got the root cause"
  },
  {
    "startMs": 419440,
    "text": "issue. And uh once we got the root cause and uh uh we will uh the next step is to"
  },
  {
    "startMs": 422710,
    "text": "and uh uh we will uh the next step is to"
  },
  {
    "startMs": 422720,
    "text": "and uh uh we will uh the next step is to uh based on the root cause uh we"
  },
  {
    "startMs": 425350,
    "text": "uh based on the root cause uh we"
  },
  {
    "startMs": 425360,
    "text": "uh based on the root cause uh we identified we want uh we can generate"
  },
  {
    "startMs": 427830,
    "text": "identified we want uh we can generate"
  },
  {
    "startMs": 427840,
    "text": "identified we want uh we can generate some uh addits proposal. So uh currently"
  },
  {
    "startMs": 431270,
    "text": "some uh addits proposal. So uh currently"
  },
  {
    "startMs": 431280,
    "text": "some uh addits proposal. So uh currently the addits proposal are just uh prompt"
  },
  {
    "startMs": 434150,
    "text": "the addits proposal are just uh prompt"
  },
  {
    "startMs": 434160,
    "text": "the addits proposal are just uh prompt but as we are climbing the climbing the"
  },
  {
    "startMs": 436469,
    "text": "but as we are climbing the climbing the"
  },
  {
    "startMs": 436479,
    "text": "but as we are climbing the climbing the ladder uh for the SI loop. So in future"
  },
  {
    "startMs": 439510,
    "text": "ladder uh for the SI loop. So in future"
  },
  {
    "startMs": 439520,
    "text": "ladder uh for the SI loop. So in future this uh additis can also be some like a"
  },
  {
    "startMs": 442390,
    "text": "this uh additis can also be some like a"
  },
  {
    "startMs": 442400,
    "text": "this uh additis can also be some like a new design of the workflow the harness"
  },
  {
    "startMs": 444469,
    "text": "new design of the workflow the harness"
  },
  {
    "startMs": 444479,
    "text": "new design of the workflow the harness code or even the optimizer optimizer"
  },
  {
    "startMs": 447510,
    "text": "code or even the optimizer optimizer"
  },
  {
    "startMs": 447520,
    "text": "code or even the optimizer optimizer code and uh uh so yeah that's uh step"
  },
  {
    "startMs": 451270,
    "text": "code and uh uh so yeah that's uh step"
  },
  {
    "startMs": 451280,
    "text": "code and uh uh so yeah that's uh step four to generate the edits proposal and"
  },
  {
    "startMs": 453749,
    "text": "four to generate the edits proposal and"
  },
  {
    "startMs": 453759,
    "text": "four to generate the edits proposal and the next step is to uh validate our"
  },
  {
    "startMs": 457189,
    "text": "the next step is to uh validate our"
  },
  {
    "startMs": 457199,
    "text": "the next step is to uh validate our edits because uh some edits can be uh"
  },
  {
    "startMs": 460230,
    "text": "edits because uh some edits can be uh"
  },
  {
    "startMs": 460240,
    "text": "edits because uh some edits can be uh good some can be bad uh so we will use"
  },
  {
    "startMs": 462870,
    "text": "good some can be bad uh so we will use"
  },
  {
    "startMs": 462880,
    "text": "good some can be bad uh so we will use the same uh evaluation engine for"
  },
  {
    "startMs": 464790,
    "text": "the same uh evaluation engine for"
  },
  {
    "startMs": 464800,
    "text": "the same uh evaluation engine for example the same judge with the same"
  },
  {
    "startMs": 467110,
    "text": "example the same judge with the same"
  },
  {
    "startMs": 467120,
    "text": "example the same judge with the same rubrics to uh to uh evaluate uh our"
  },
  {
    "startMs": 470469,
    "text": "rubrics to uh to uh evaluate uh our"
  },
  {
    "startMs": 470479,
    "text": "rubrics to uh to uh evaluate uh our addit validation and uh uh once we found"
  },
  {
    "startMs": 474150,
    "text": "addit validation and uh uh once we found"
  },
  {
    "startMs": 474160,
    "text": "addit validation and uh uh once we found some like a good candidates and we will"
  },
  {
    "startMs": 476469,
    "text": "some like a good candidates and we will"
  },
  {
    "startMs": 476479,
    "text": "some like a good candidates and we will uh naturally the next step is to uh"
  },
  {
    "startMs": 479029,
    "text": "uh naturally the next step is to uh"
  },
  {
    "startMs": 479039,
    "text": "uh naturally the next step is to uh evolve our uh prompt to adapt the uh the"
  },
  {
    "startMs": 483350,
    "text": "evolve our uh prompt to adapt the uh the"
  },
  {
    "startMs": 483360,
    "text": "evolve our uh prompt to adapt the uh the uh the proposed edits uh we have and uh"
  },
  {
    "startMs": 487270,
    "text": "uh the proposed edits uh we have and uh"
  },
  {
    "startMs": 487280,
    "text": "uh the proposed edits uh we have and uh uh once we uh once the prompt is uh"
  },
  {
    "startMs": 489589,
    "text": "uh once we uh once the prompt is uh"
  },
  {
    "startMs": 489599,
    "text": "uh once we uh once the prompt is uh updated. So the very last step which is"
  },
  {
    "startMs": 492150,
    "text": "updated. So the very last step which is"
  },
  {
    "startMs": 492160,
    "text": "updated. So the very last step which is a step seven here is we run the"
  },
  {
    "startMs": 494469,
    "text": "a step seven here is we run the"
  },
  {
    "startMs": 494479,
    "text": "a step seven here is we run the evaluation uh again on a hold up side"
  },
  {
    "startMs": 497749,
    "text": "evaluation uh again on a hold up side"
  },
  {
    "startMs": 497759,
    "text": "evaluation uh again on a hold up side and to confirm uh our uh this kind of"
  },
  {
    "startMs": 500550,
    "text": "and to confirm uh our uh this kind of"
  },
  {
    "startMs": 500560,
    "text": "and to confirm uh our uh this kind of the uh new system is indeed performs"
  },
  {
    "startMs": 503110,
    "text": "the uh new system is indeed performs"
  },
  {
    "startMs": 503120,
    "text": "the uh new system is indeed performs better uh than the old system after the"
  },
  {
    "startMs": 506070,
    "text": "better uh than the old system after the"
  },
  {
    "startMs": 506080,
    "text": "better uh than the old system after the SI optimization and uh uh this will and"
  },
  {
    "startMs": 510550,
    "text": "SI optimization and uh uh this will and"
  },
  {
    "startMs": 510560,
    "text": "SI optimization and uh uh this will and also like uh after this hold out"
  },
  {
    "startMs": 513190,
    "text": "also like uh after this hold out"
  },
  {
    "startMs": 513200,
    "text": "also like uh after this hold out validation uh like we will make the uh"
  },
  {
    "startMs": 516550,
    "text": "validation uh like we will make the uh"
  },
  {
    "startMs": 516560,
    "text": "validation uh like we will make the uh prompt updates and this will become our"
  },
  {
    "startMs": 519670,
    "text": "prompt updates and this will become our"
  },
  {
    "startMs": 519680,
    "text": "prompt updates and this will become our uh new baseline and uh and we are uh in"
  },
  {
    "startMs": 523190,
    "text": "uh new baseline and uh and we are uh in"
  },
  {
    "startMs": 523200,
    "text": "uh new baseline and uh and we are uh in future we will like continue to do this"
  },
  {
    "startMs": 525990,
    "text": "future we will like continue to do this"
  },
  {
    "startMs": 526000,
    "text": "future we will like continue to do this uh RSI uh iteration to keep improving uh"
  },
  {
    "startMs": 529350,
    "text": "uh RSI uh iteration to keep improving uh"
  },
  {
    "startMs": 529360,
    "text": "uh RSI uh iteration to keep improving uh our system uh with some uh cadence."
  },
  {
    "startMs": 533350,
    "text": "our system uh with some uh cadence."
  },
  {
    "startMs": 533360,
    "text": "our system uh with some uh cadence. Uh okay. Yeah. Next uh like uh we are we"
  },
  {
    "startMs": 535829,
    "text": "Uh okay. Yeah. Next uh like uh we are we"
  },
  {
    "startMs": 535839,
    "text": "Uh okay. Yeah. Next uh like uh we are we can uh like dig deeper into uh each step"
  },
  {
    "startMs": 539590,
    "text": "can uh like dig deeper into uh each step"
  },
  {
    "startMs": 539600,
    "text": "can uh like dig deeper into uh each step so that we can get uh so so to help you"
  },
  {
    "startMs": 541990,
    "text": "so that we can get uh so so to help you"
  },
  {
    "startMs": 542000,
    "text": "so that we can get uh so so to help you to understand more details on what we"
  },
  {
    "startMs": 543990,
    "text": "to understand more details on what we"
  },
  {
    "startMs": 544000,
    "text": "to understand more details on what we are uh doing here. for example, what is"
  },
  {
    "startMs": 545829,
    "text": "are uh doing here. for example, what is"
  },
  {
    "startMs": 545839,
    "text": "are uh doing here. for example, what is the agent we are optimizing and what is"
  },
  {
    "startMs": 548790,
    "text": "the agent we are optimizing and what is"
  },
  {
    "startMs": 548800,
    "text": "the agent we are optimizing and what is like a steps uh like a detail steps uh"
  },
  {
    "startMs": 551350,
    "text": "like a steps uh like a detail steps uh"
  },
  {
    "startMs": 551360,
    "text": "like a steps uh like a detail steps uh in each uh this high level steps to uh"
  },
  {
    "startMs": 554790,
    "text": "in each uh this high level steps to uh"
  },
  {
    "startMs": 554800,
    "text": "in each uh this high level steps to uh help us uh to help you to understand"
  },
  {
    "startMs": 556550,
    "text": "help us uh to help you to understand"
  },
  {
    "startMs": 556560,
    "text": "help us uh to help you to understand more of of our uh SI work here. So the"
  },
  {
    "startMs": 560150,
    "text": "more of of our uh SI work here. So the"
  },
  {
    "startMs": 560160,
    "text": "more of of our uh SI work here. So the first step I I want to show uh so this"
  },
  {
    "startMs": 563030,
    "text": "first step I I want to show uh so this"
  },
  {
    "startMs": 563040,
    "text": "first step I I want to show uh so this is the first uh the step we generated"
  },
  {
    "startMs": 565190,
    "text": "is the first uh the step we generated"
  },
  {
    "startMs": 565200,
    "text": "is the first uh the step we generated the output and the generated trace and"
  },
  {
    "startMs": 567670,
    "text": "the output and the generated trace and"
  },
  {
    "startMs": 567680,
    "text": "the output and the generated trace and uh so first we want to I want to uh help"
  },
  {
    "startMs": 570150,
    "text": "uh so first we want to I want to uh help"
  },
  {
    "startMs": 570160,
    "text": "uh so first we want to I want to uh help you to understand the agent uh we are"
  },
  {
    "startMs": 572310,
    "text": "you to understand the agent uh we are"
  },
  {
    "startMs": 572320,
    "text": "you to understand the agent uh we are optimizing here. So this is a uh user uh"
  },
  {
    "startMs": 575590,
    "text": "optimizing here. So this is a uh user uh"
  },
  {
    "startMs": 575600,
    "text": "optimizing here. So this is a uh user uh understanding uh agent uh at linking. So"
  },
  {
    "startMs": 579269,
    "text": "understanding uh agent uh at linking. So"
  },
  {
    "startMs": 579279,
    "text": "understanding uh agent uh at linking. So uh so like you can see we uh in this"
  },
  {
    "startMs": 581750,
    "text": "uh so like you can see we uh in this"
  },
  {
    "startMs": 581760,
    "text": "uh so like you can see we uh in this slide we are showing the input and the"
  },
  {
    "startMs": 583670,
    "text": "slide we are showing the input and the"
  },
  {
    "startMs": 583680,
    "text": "slide we are showing the input and the output of this uh agent. So the input is"
  },
  {
    "startMs": 586870,
    "text": "output of this uh agent. So the input is"
  },
  {
    "startMs": 586880,
    "text": "output of this uh agent. So the input is just a simply member ID also it's just a"
  },
  {
    "startMs": 590230,
    "text": "just a simply member ID also it's just a"
  },
  {
    "startMs": 590240,
    "text": "just a simply member ID also it's just a user ID and uh uh by uh given user ID we"
  },
  {
    "startMs": 594550,
    "text": "user ID and uh uh by uh given user ID we"
  },
  {
    "startMs": 594560,
    "text": "user ID and uh uh by uh given user ID we can fetch uh some information uh of this"
  },
  {
    "startMs": 597269,
    "text": "can fetch uh some information uh of this"
  },
  {
    "startMs": 597279,
    "text": "can fetch uh some information uh of this user for example this user search"
  },
  {
    "startMs": 599110,
    "text": "user for example this user search"
  },
  {
    "startMs": 599120,
    "text": "user for example this user search history uh profile data uh and also some"
  },
  {
    "startMs": 602790,
    "text": "history uh profile data uh and also some"
  },
  {
    "startMs": 602800,
    "text": "history uh profile data uh and also some like a location data and the preference"
  },
  {
    "startMs": 604630,
    "text": "like a location data and the preference"
  },
  {
    "startMs": 604640,
    "text": "like a location data and the preference data those kind of things and we have"
  },
  {
    "startMs": 607110,
    "text": "data those kind of things and we have"
  },
  {
    "startMs": 607120,
    "text": "data those kind of things and we have this user understanding agent to uh like"
  },
  {
    "startMs": 609829,
    "text": "this user understanding agent to uh like"
  },
  {
    "startMs": 609839,
    "text": "this user understanding agent to uh like take those users uh history or behavior"
  },
  {
    "startMs": 612630,
    "text": "take those users uh history or behavior"
  },
  {
    "startMs": 612640,
    "text": "take those users uh history or behavior data profile data as a input and uh do"
  },
  {
    "startMs": 615590,
    "text": "data profile data as a input and uh do"
  },
  {
    "startMs": 615600,
    "text": "data profile data as a input and uh do the uh do a lot like this kind of um uh"
  },
  {
    "startMs": 618470,
    "text": "the uh do a lot like this kind of um uh"
  },
  {
    "startMs": 618480,
    "text": "the uh do a lot like this kind of um uh agentic uh synthesize steps and we"
  },
  {
    "startMs": 621509,
    "text": "agentic uh synthesize steps and we"
  },
  {
    "startMs": 621519,
    "text": "agentic uh synthesize steps and we generate the uh output. So for this"
  },
  {
    "startMs": 624150,
    "text": "generate the uh output. So for this"
  },
  {
    "startMs": 624160,
    "text": "generate the uh output. So for this output we call it is a uh memor sending"
  },
  {
    "startMs": 626870,
    "text": "output we call it is a uh memor sending"
  },
  {
    "startMs": 626880,
    "text": "output we call it is a uh memor sending or the user understanding uh output. So"
  },
  {
    "startMs": 629590,
    "text": "or the user understanding uh output. So"
  },
  {
    "startMs": 629600,
    "text": "or the user understanding uh output. So here is a high level uh structure of the"
  },
  {
    "startMs": 632389,
    "text": "here is a high level uh structure of the"
  },
  {
    "startMs": 632399,
    "text": "here is a high level uh structure of the uh this output is so given a member. So"
  },
  {
    "startMs": 635750,
    "text": "uh this output is so given a member. So"
  },
  {
    "startMs": 635760,
    "text": "uh this output is so given a member. So we will generate some uh fields to"
  },
  {
    "startMs": 638710,
    "text": "we will generate some uh fields to"
  },
  {
    "startMs": 638720,
    "text": "we will generate some uh fields to describe uh this uh member based on our"
  },
  {
    "startMs": 641430,
    "text": "describe uh this uh member based on our"
  },
  {
    "startMs": 641440,
    "text": "describe uh this uh member based on our agent understanding. For example, what"
  },
  {
    "startMs": 643430,
    "text": "agent understanding. For example, what"
  },
  {
    "startMs": 643440,
    "text": "agent understanding. For example, what is the preferred uh job title uh of this"
  },
  {
    "startMs": 645990,
    "text": "is the preferred uh job title uh of this"
  },
  {
    "startMs": 646000,
    "text": "is the preferred uh job title uh of this member? This is just one uh field of"
  },
  {
    "startMs": 648710,
    "text": "member? This is just one uh field of"
  },
  {
    "startMs": 648720,
    "text": "member? This is just one uh field of this uh this user understanding uh"
  },
  {
    "startMs": 650949,
    "text": "this uh this user understanding uh"
  },
  {
    "startMs": 650959,
    "text": "this uh this user understanding uh output. So for this one, this is just a"
  },
  {
    "startMs": 653990,
    "text": "output. So for this one, this is just a"
  },
  {
    "startMs": 654000,
    "text": "output. So for this one, this is just a example of the preferred job titles. So"
  },
  {
    "startMs": 657190,
    "text": "example of the preferred job titles. So"
  },
  {
    "startMs": 657200,
    "text": "example of the preferred job titles. So it can be some like a network manager,"
  },
  {
    "startMs": 659269,
    "text": "it can be some like a network manager,"
  },
  {
    "startMs": 659279,
    "text": "it can be some like a network manager, account manager those kind of things and"
  },
  {
    "startMs": 661430,
    "text": "account manager those kind of things and"
  },
  {
    "startMs": 661440,
    "text": "account manager those kind of things and we'll give a confidence on like how"
  },
  {
    "startMs": 663829,
    "text": "we'll give a confidence on like how"
  },
  {
    "startMs": 663839,
    "text": "we'll give a confidence on like how confidence this uh agent think uh like"
  },
  {
    "startMs": 667350,
    "text": "confidence this uh agent think uh like"
  },
  {
    "startMs": 667360,
    "text": "confidence this uh agent think uh like this is indeed the preferred job titles"
  },
  {
    "startMs": 670150,
    "text": "this is indeed the preferred job titles"
  },
  {
    "startMs": 670160,
    "text": "this is indeed the preferred job titles for this user and uh also uh we'll get"
  },
  {
    "startMs": 673910,
    "text": "for this user and uh also uh we'll get"
  },
  {
    "startMs": 673920,
    "text": "for this user and uh also uh we'll get uh generate some uh citation data. So"
  },
  {
    "startMs": 676069,
    "text": "uh generate some uh citation data. So"
  },
  {
    "startMs": 676079,
    "text": "uh generate some uh citation data. So they like this is the uh like why the"
  },
  {
    "startMs": 678310,
    "text": "they like this is the uh like why the"
  },
  {
    "startMs": 678320,
    "text": "they like this is the uh like why the this is where we can tell why the agent"
  },
  {
    "startMs": 681030,
    "text": "this is where we can tell why the agent"
  },
  {
    "startMs": 681040,
    "text": "this is where we can tell why the agent think those uh titles is a preferred job"
  },
  {
    "startMs": 684230,
    "text": "think those uh titles is a preferred job"
  },
  {
    "startMs": 684240,
    "text": "think those uh titles is a preferred job title for this particular uh member. Uh"
  },
  {
    "startMs": 687030,
    "text": "title for this particular uh member. Uh"
  },
  {
    "startMs": 687040,
    "text": "title for this particular uh member. Uh for example it could be uh the reason"
  },
  {
    "startMs": 689269,
    "text": "for example it could be uh the reason"
  },
  {
    "startMs": 689279,
    "text": "for example it could be uh the reason could be like uh in the users uh save"
  },
  {
    "startMs": 692389,
    "text": "could be like uh in the users uh save"
  },
  {
    "startMs": 692399,
    "text": "could be like uh in the users uh save the search history like those job titles"
  },
  {
    "startMs": 695670,
    "text": "the search history like those job titles"
  },
  {
    "startMs": 695680,
    "text": "the search history like those job titles uh appears very frequently. So the agent"
  },
  {
    "startMs": 698150,
    "text": "uh appears very frequently. So the agent"
  },
  {
    "startMs": 698160,
    "text": "uh appears very frequently. So the agent think oh this is a strong evidence uh"
  },
  {
    "startMs": 700470,
    "text": "think oh this is a strong evidence uh"
  },
  {
    "startMs": 700480,
    "text": "think oh this is a strong evidence uh that to support that the preferred job"
  },
  {
    "startMs": 702470,
    "text": "that to support that the preferred job"
  },
  {
    "startMs": 702480,
    "text": "that to support that the preferred job title is a network uh manager uh for"
  },
  {
    "startMs": 704870,
    "text": "title is a network uh manager uh for"
  },
  {
    "startMs": 704880,
    "text": "title is a network uh manager uh for this uh for this uh user and uh uh as a"
  },
  {
    "startMs": 709350,
    "text": "this uh for this uh user and uh uh as a"
  },
  {
    "startMs": 709360,
    "text": "this uh for this uh user and uh uh as a uh and uh finally this agent can also"
  },
  {
    "startMs": 712150,
    "text": "uh and uh finally this agent can also"
  },
  {
    "startMs": 712160,
    "text": "uh and uh finally this agent can also output the reasoning on like why like"
  },
  {
    "startMs": 715829,
    "text": "output the reasoning on like why like"
  },
  {
    "startMs": 715839,
    "text": "output the reasoning on like why like just a reasoning like why the agent"
  },
  {
    "startMs": 717750,
    "text": "just a reasoning like why the agent"
  },
  {
    "startMs": 717760,
    "text": "just a reasoning like why the agent think those are preferred job titles"
  },
  {
    "startMs": 719670,
    "text": "think those are preferred job titles"
  },
  {
    "startMs": 719680,
    "text": "think those are preferred job titles like for example the state preference"
  },
  {
    "startMs": 721430,
    "text": "like for example the state preference"
  },
  {
    "startMs": 721440,
    "text": "like for example the state preference anchors the primary title save search"
  },
  {
    "startMs": 723590,
    "text": "anchors the primary title save search"
  },
  {
    "startMs": 723600,
    "text": "anchors the primary title save search agent uh adjacent one those kind of"
  },
  {
    "startMs": 727030,
    "text": "agent uh adjacent one those kind of"
  },
  {
    "startMs": 727040,
    "text": "agent uh adjacent one those kind of So this is a kind of uh uh one example"
  },
  {
    "startMs": 729990,
    "text": "So this is a kind of uh uh one example"
  },
  {
    "startMs": 730000,
    "text": "So this is a kind of uh uh one example of the user understanding agent that we"
  },
  {
    "startMs": 733110,
    "text": "of the user understanding agent that we"
  },
  {
    "startMs": 733120,
    "text": "of the user understanding agent that we are optimizing."
  },
  {
    "startMs": 735350,
    "text": "are optimizing."
  },
  {
    "startMs": 735360,
    "text": "are optimizing. So the step two is once we uh got this"
  },
  {
    "startMs": 738470,
    "text": "So the step two is once we uh got this"
  },
  {
    "startMs": 738480,
    "text": "So the step two is once we uh got this uh agents trace and we'll run the judge"
  },
  {
    "startMs": 742470,
    "text": "uh agents trace and we'll run the judge"
  },
  {
    "startMs": 742480,
    "text": "uh agents trace and we'll run the judge and to uh get a quality readings of this"
  },
  {
    "startMs": 746230,
    "text": "and to uh get a quality readings of this"
  },
  {
    "startMs": 746240,
    "text": "and to uh get a quality readings of this uh of the agent uh output and uh for"
  },
  {
    "startMs": 749430,
    "text": "uh of the agent uh output and uh for"
  },
  {
    "startMs": 749440,
    "text": "uh of the agent uh output and uh for this particular user agent uh the score"
  },
  {
    "startMs": 751990,
    "text": "this particular user agent uh the score"
  },
  {
    "startMs": 752000,
    "text": "this particular user agent uh the score is over uh 15 uh task of the dimensions"
  },
  {
    "startMs": 755350,
    "text": "is over uh 15 uh task of the dimensions"
  },
  {
    "startMs": 755360,
    "text": "is over uh 15 uh task of the dimensions by judge where are some um product uh"
  },
  {
    "startMs": 758310,
    "text": "by judge where are some um product uh"
  },
  {
    "startMs": 758320,
    "text": "by judge where are some um product uh defender uh rubrics. So here is one"
  },
  {
    "startMs": 761269,
    "text": "defender uh rubrics. So here is one"
  },
  {
    "startMs": 761279,
    "text": "defender uh rubrics. So here is one example of the uh judge uh output. So"
  },
  {
    "startMs": 764710,
    "text": "example of the uh judge uh output. So"
  },
  {
    "startMs": 764720,
    "text": "example of the uh judge uh output. So you can say for particular um member the"
  },
  {
    "startMs": 767910,
    "text": "you can say for particular um member the"
  },
  {
    "startMs": 767920,
    "text": "you can say for particular um member the user and we will generate a judge uh"
  },
  {
    "startMs": 770629,
    "text": "user and we will generate a judge uh"
  },
  {
    "startMs": 770639,
    "text": "user and we will generate a judge uh score which is 2.31 and uh we will show"
  },
  {
    "startMs": 773910,
    "text": "score which is 2.31 and uh we will show"
  },
  {
    "startMs": 773920,
    "text": "score which is 2.31 and uh we will show and it will also tell like this score is"
  },
  {
    "startMs": 775990,
    "text": "and it will also tell like this score is"
  },
  {
    "startMs": 776000,
    "text": "and it will also tell like this score is for uh which um and [clears throat]"
  },
  {
    "startMs": 778710,
    "text": "for uh which um and [clears throat]"
  },
  {
    "startMs": 778720,
    "text": "for uh which um and [clears throat] it'll show the breakdowns uh of the you"
  },
  {
    "startMs": 781750,
    "text": "it'll show the breakdowns uh of the you"
  },
  {
    "startMs": 781760,
    "text": "it'll show the breakdowns uh of the you know the each dimensions score for"
  },
  {
    "startMs": 783590,
    "text": "know the each dimensions score for"
  },
  {
    "startMs": 783600,
    "text": "know the each dimensions score for example the go summary dimensions and we"
  },
  {
    "startMs": 786389,
    "text": "example the go summary dimensions and we"
  },
  {
    "startMs": 786399,
    "text": "example the go summary dimensions and we also have some other dimension for the"
  },
  {
    "startMs": 788150,
    "text": "also have some other dimension for the"
  },
  {
    "startMs": 788160,
    "text": "also have some other dimension for the the second one is a prefer job title"
  },
  {
    "startMs": 790389,
    "text": "the second one is a prefer job title"
  },
  {
    "startMs": 790399,
    "text": "the second one is a prefer job title dimensions and for each this kind of"
  },
  {
    "startMs": 792949,
    "text": "dimensions and for each this kind of"
  },
  {
    "startMs": 792959,
    "text": "dimensions and for each this kind of dimensions we give a detailed uh score"
  },
  {
    "startMs": 795829,
    "text": "dimensions we give a detailed uh score"
  },
  {
    "startMs": 795839,
    "text": "dimensions we give a detailed uh score uh for each dimension and also the"
  },
  {
    "startMs": 797750,
    "text": "uh for each dimension and also the"
  },
  {
    "startMs": 797760,
    "text": "uh for each dimension and also the notes. So the notes is uh just the"
  },
  {
    "startMs": 800150,
    "text": "notes. So the notes is uh just the"
  },
  {
    "startMs": 800160,
    "text": "notes. So the notes is uh just the rationale also the uh vertx like why uh"
  },
  {
    "startMs": 804389,
    "text": "rationale also the uh vertx like why uh"
  },
  {
    "startMs": 804399,
    "text": "rationale also the uh vertx like why uh we give this score uh to this uh to this"
  },
  {
    "startMs": 808230,
    "text": "we give this score uh to this uh to this"
  },
  {
    "startMs": 808240,
    "text": "we give this score uh to this uh to this particular uh dimension and for this"
  },
  {
    "startMs": 810949,
    "text": "particular uh dimension and for this"
  },
  {
    "startMs": 810959,
    "text": "particular uh dimension and for this particular uh judge is the agent. So the"
  },
  {
    "startMs": 814389,
    "text": "particular uh judge is the agent. So the"
  },
  {
    "startMs": 814399,
    "text": "particular uh judge is the agent. So the uh judge score is one to three uh scale."
  },
  {
    "startMs": 817509,
    "text": "uh judge score is one to three uh scale."
  },
  {
    "startMs": 817519,
    "text": "uh judge score is one to three uh scale. So the score one means uh this is not a"
  },
  {
    "startMs": 820790,
    "text": "So the score one means uh this is not a"
  },
  {
    "startMs": 820800,
    "text": "So the score one means uh this is not a high uh good is not a good yeah the"
  },
  {
    "startMs": 825509,
    "text": "high uh good is not a good yeah the"
  },
  {
    "startMs": 825519,
    "text": "high uh good is not a good yeah the quality is not good and the two is a"
  },
  {
    "startMs": 827670,
    "text": "quality is not good and the two is a"
  },
  {
    "startMs": 827680,
    "text": "quality is not good and the two is a fair uh quality and the three means this"
  },
  {
    "startMs": 830230,
    "text": "fair uh quality and the three means this"
  },
  {
    "startMs": 830240,
    "text": "fair uh quality and the three means this is a very good quality. Yeah. And uh uh"
  },
  {
    "startMs": 833990,
    "text": "is a very good quality. Yeah. And uh uh"
  },
  {
    "startMs": 834000,
    "text": "is a very good quality. Yeah. And uh uh so yeah that's our uh um judge and uh"
  },
  {
    "startMs": 837350,
    "text": "so yeah that's our uh um judge and uh"
  },
  {
    "startMs": 837360,
    "text": "so yeah that's our uh um judge and uh after we got the judge score uh for each"
  },
  {
    "startMs": 841189,
    "text": "after we got the judge score uh for each"
  },
  {
    "startMs": 841199,
    "text": "after we got the judge score uh for each dimension of our user understanding"
  },
  {
    "startMs": 843350,
    "text": "dimension of our user understanding"
  },
  {
    "startMs": 843360,
    "text": "dimension of our user understanding agent and the next step is we will run"
  },
  {
    "startMs": 846310,
    "text": "agent and the next step is we will run"
  },
  {
    "startMs": 846320,
    "text": "agent and the next step is we will run this uh agent reflection process. So as"
  },
  {
    "startMs": 849829,
    "text": "this uh agent reflection process. So as"
  },
  {
    "startMs": 849839,
    "text": "this uh agent reflection process. So as we mentioned earlier so this is a kind"
  },
  {
    "startMs": 851829,
    "text": "we mentioned earlier so this is a kind"
  },
  {
    "startMs": 851839,
    "text": "we mentioned earlier so this is a kind of uh uh process for us to understand"
  },
  {
    "startMs": 855590,
    "text": "of uh uh process for us to understand"
  },
  {
    "startMs": 855600,
    "text": "of uh uh process for us to understand okay if this is a low score then what is"
  },
  {
    "startMs": 857910,
    "text": "okay if this is a low score then what is"
  },
  {
    "startMs": 857920,
    "text": "okay if this is a low score then what is the root cause uh lead to the uh this"
  },
  {
    "startMs": 861189,
    "text": "the root cause uh lead to the uh this"
  },
  {
    "startMs": 861199,
    "text": "the root cause uh lead to the uh this low score. So uh this is a so for this"
  },
  {
    "startMs": 864629,
    "text": "low score. So uh this is a so for this"
  },
  {
    "startMs": 864639,
    "text": "low score. So uh this is a so for this uh step three so actually uh internally"
  },
  {
    "startMs": 867110,
    "text": "uh step three so actually uh internally"
  },
  {
    "startMs": 867120,
    "text": "uh step three so actually uh internally there are four uh like we can call it a"
  },
  {
    "startMs": 870230,
    "text": "there are four uh like we can call it a"
  },
  {
    "startMs": 870240,
    "text": "there are four uh like we can call it a mini steps for the agent reflection. So"
  },
  {
    "startMs": 873670,
    "text": "mini steps for the agent reflection. So"
  },
  {
    "startMs": 873680,
    "text": "mini steps for the agent reflection. So the first step is a clustering. So we"
  },
  {
    "startMs": 876550,
    "text": "the first step is a clustering. So we"
  },
  {
    "startMs": 876560,
    "text": "the first step is a clustering. So we are group the failing uh rows because"
  },
  {
    "startMs": 879189,
    "text": "are group the failing uh rows because"
  },
  {
    "startMs": 879199,
    "text": "are group the failing uh rows because like in a production trace so we are we"
  },
  {
    "startMs": 881590,
    "text": "like in a production trace so we are we"
  },
  {
    "startMs": 881600,
    "text": "like in a production trace so we are we will have a lot of like uh examples a"
  },
  {
    "startMs": 884150,
    "text": "will have a lot of like uh examples a"
  },
  {
    "startMs": 884160,
    "text": "will have a lot of like uh examples a lot of users and we will group uh the"
  },
  {
    "startMs": 887990,
    "text": "lot of users and we will group uh the"
  },
  {
    "startMs": 888000,
    "text": "lot of users and we will group uh the those users also the traces by the"
  },
  {
    "startMs": 890629,
    "text": "those users also the traces by the"
  },
  {
    "startMs": 890639,
    "text": "those users also the traces by the failing uh dimension. So for this uh"
  },
  {
    "startMs": 894230,
    "text": "failing uh dimension. So for this uh"
  },
  {
    "startMs": 894240,
    "text": "failing uh dimension. So for this uh step so this is just a pure clustering"
  },
  {
    "startMs": 896629,
    "text": "step so this is just a pure clustering"
  },
  {
    "startMs": 896639,
    "text": "step so this is just a pure clustering uh without uh any um cost and uh once we"
  },
  {
    "startMs": 900470,
    "text": "uh without uh any um cost and uh once we"
  },
  {
    "startMs": 900480,
    "text": "uh without uh any um cost and uh once we get the clustering and uh uh so the next"
  },
  {
    "startMs": 903509,
    "text": "get the clustering and uh uh so the next"
  },
  {
    "startMs": 903519,
    "text": "get the clustering and uh uh so the next step is we call it L2 uh judge so b"
  },
  {
    "startMs": 907509,
    "text": "step is we call it L2 uh judge so b"
  },
  {
    "startMs": 907519,
    "text": "step is we call it L2 uh judge so b within each uh class. So we are run the"
  },
  {
    "startMs": 911110,
    "text": "within each uh class. So we are run the"
  },
  {
    "startMs": 911120,
    "text": "within each uh class. So we are run the um uh cause to diagnose uh each row and"
  },
  {
    "startMs": 916389,
    "text": "um uh cause to diagnose uh each row and"
  },
  {
    "startMs": 916399,
    "text": "um uh cause to diagnose uh each row and uh and then for each row we are assign a"
  },
  {
    "startMs": 919030,
    "text": "uh and then for each row we are assign a"
  },
  {
    "startMs": 919040,
    "text": "uh and then for each row we are assign a root cause and plus tag. So b like this"
  },
  {
    "startMs": 922710,
    "text": "root cause and plus tag. So b like this"
  },
  {
    "startMs": 922720,
    "text": "root cause and plus tag. So b like this is a u uh place that we uh can like rely"
  },
  {
    "startMs": 927110,
    "text": "is a u uh place that we uh can like rely"
  },
  {
    "startMs": 927120,
    "text": "is a u uh place that we uh can like rely on the to uh let us know uh uh if we"
  },
  {
    "startMs": 931829,
    "text": "on the to uh let us know uh uh if we"
  },
  {
    "startMs": 931839,
    "text": "on the to uh let us know uh uh if we think uh this is a for example this"
  },
  {
    "startMs": 934389,
    "text": "think uh this is a for example this"
  },
  {
    "startMs": 934399,
    "text": "think uh this is a for example this dimension has a low score is that due to"
  },
  {
    "startMs": 936150,
    "text": "dimension has a low score is that due to"
  },
  {
    "startMs": 936160,
    "text": "dimension has a low score is that due to the prompt issue uh agent issue harness"
  },
  {
    "startMs": 938790,
    "text": "the prompt issue uh agent issue harness"
  },
  {
    "startMs": 938800,
    "text": "the prompt issue uh agent issue harness issue code issue or like even the judge"
  },
  {
    "startMs": 941269,
    "text": "issue code issue or like even the judge"
  },
  {
    "startMs": 941279,
    "text": "issue code issue or like even the judge issue or rubric issue. So they will be"
  },
  {
    "startMs": 944069,
    "text": "issue or rubric issue. So they will be"
  },
  {
    "startMs": 944079,
    "text": "issue or rubric issue. So they will be covered by the root cause and tag uh"
  },
  {
    "startMs": 946949,
    "text": "covered by the root cause and tag uh"
  },
  {
    "startMs": 946959,
    "text": "covered by the root cause and tag uh here. So uh the then once we get the"
  },
  {
    "startMs": 951030,
    "text": "here. So uh the then once we get the"
  },
  {
    "startMs": 951040,
    "text": "here. So uh the then once we get the perole level root cause and the tag so"
  },
  {
    "startMs": 953509,
    "text": "perole level root cause and the tag so"
  },
  {
    "startMs": 953519,
    "text": "perole level root cause and the tag so the next step is to merge the uh issues"
  },
  {
    "startMs": 956949,
    "text": "the next step is to merge the uh issues"
  },
  {
    "startMs": 956959,
    "text": "the next step is to merge the uh issues because uh in the second step uh so we"
  },
  {
    "startMs": 960230,
    "text": "because uh in the second step uh so we"
  },
  {
    "startMs": 960240,
    "text": "because uh in the second step uh so we it's just a we is based right so so the"
  },
  {
    "startMs": 965509,
    "text": "it's just a we is based right so so the"
  },
  {
    "startMs": 965519,
    "text": "it's just a we is based right so so the root cost is more like the free tax but"
  },
  {
    "startMs": 968870,
    "text": "root cost is more like the free tax but"
  },
  {
    "startMs": 968880,
    "text": "root cost is more like the free tax but uh uh some root cause uh even the tax is"
  },
  {
    "startMs": 972470,
    "text": "uh uh some root cause uh even the tax is"
  },
  {
    "startMs": 972480,
    "text": "uh uh some root cause uh even the tax is not um uh exactly the same but they may"
  },
  {
    "startMs": 976470,
    "text": "not um uh exactly the same but they may"
  },
  {
    "startMs": 976480,
    "text": "not um uh exactly the same but they may just describe very similar thing or uh"
  },
  {
    "startMs": 979509,
    "text": "just describe very similar thing or uh"
  },
  {
    "startMs": 979519,
    "text": "just describe very similar thing or uh so that's why like we want to do this"
  },
  {
    "startMs": 981350,
    "text": "so that's why like we want to do this"
  },
  {
    "startMs": 981360,
    "text": "so that's why like we want to do this kind of uh dduping so we merge the"
  },
  {
    "startMs": 984870,
    "text": "kind of uh dduping so we merge the"
  },
  {
    "startMs": 984880,
    "text": "kind of uh dduping so we merge the issues if we think uh the root cause is"
  },
  {
    "startMs": 988230,
    "text": "issues if we think uh the root cause is"
  },
  {
    "startMs": 988240,
    "text": "issues if we think uh the root cause is uh near uh duplicates so uh I can give"
  },
  {
    "startMs": 991590,
    "text": "uh near uh duplicates so uh I can give"
  },
  {
    "startMs": 991600,
    "text": "uh near uh duplicates so uh I can give one example here so for example uh for"
  },
  {
    "startMs": 994790,
    "text": "one example here so for example uh for"
  },
  {
    "startMs": 994800,
    "text": "one example here so for example uh for one row for one member uh the root cause"
  },
  {
    "startMs": 997670,
    "text": "one row for one member uh the root cause"
  },
  {
    "startMs": 997680,
    "text": "one row for one member uh the root cause can be uh the members uh profile data is"
  },
  {
    "startMs": 1001590,
    "text": "can be uh the members uh profile data is"
  },
  {
    "startMs": 1001600,
    "text": "can be uh the members uh profile data is uh missing uh and another row uh uh so"
  },
  {
    "startMs": 1005829,
    "text": "uh missing uh and another row uh uh so"
  },
  {
    "startMs": 1005839,
    "text": "uh missing uh and another row uh uh so the arm can"
  },
  {
    "startMs": 1007110,
    "text": "the arm can"
  },
  {
    "startMs": 1007120,
    "text": "the arm can the profile data is uh not showing the"
  },
  {
    "startMs": 1010710,
    "text": "the profile data is uh not showing the"
  },
  {
    "startMs": 1010720,
    "text": "the profile data is uh not showing the full picture of this users uh like a"
  },
  {
    "startMs": 1013189,
    "text": "full picture of this users uh like a"
  },
  {
    "startMs": 1013199,
    "text": "full picture of this users uh like a browsing history or whatever. So b like"
  },
  {
    "startMs": 1015350,
    "text": "browsing history or whatever. So b like"
  },
  {
    "startMs": 1015360,
    "text": "browsing history or whatever. So b like they are uh both of them are describing"
  },
  {
    "startMs": 1018069,
    "text": "they are uh both of them are describing"
  },
  {
    "startMs": 1018079,
    "text": "they are uh both of them are describing the same thing. So the profile data is"
  },
  {
    "startMs": 1020310,
    "text": "the same thing. So the profile data is"
  },
  {
    "startMs": 1020320,
    "text": "the same thing. So the profile data is not supporting the uh this agent output."
  },
  {
    "startMs": 1024390,
    "text": "not supporting the uh this agent output."
  },
  {
    "startMs": 1024400,
    "text": "not supporting the uh this agent output. So uh but they just use a different"
  },
  {
    "startMs": 1026710,
    "text": "So uh but they just use a different"
  },
  {
    "startMs": 1026720,
    "text": "So uh but they just use a different text. So that's we are like that's why"
  },
  {
    "startMs": 1029189,
    "text": "text. So that's we are like that's why"
  },
  {
    "startMs": 1029199,
    "text": "text. So that's we are like that's why we need this um kind like a dduplication"
  },
  {
    "startMs": 1032150,
    "text": "we need this um kind like a dduplication"
  },
  {
    "startMs": 1032160,
    "text": "we need this um kind like a dduplication uh step to merge the similar issues so"
  },
  {
    "startMs": 1035189,
    "text": "uh step to merge the similar issues so"
  },
  {
    "startMs": 1035199,
    "text": "uh step to merge the similar issues so we can get some uh canonical uh issue I"
  },
  {
    "startMs": 1037909,
    "text": "we can get some uh canonical uh issue I"
  },
  {
    "startMs": 1037919,
    "text": "we can get some uh canonical uh issue I say the roo uh labels"
  },
  {
    "startMs": 1041029,
    "text": "say the roo uh labels"
  },
  {
    "startMs": 1041039,
    "text": "say the roo uh labels and uh once we finish the step three so"
  },
  {
    "startMs": 1043669,
    "text": "and uh once we finish the step three so"
  },
  {
    "startMs": 1043679,
    "text": "and uh once we finish the step three so the next step is to based on the uh"
  },
  {
    "startMs": 1045990,
    "text": "the next step is to based on the uh"
  },
  {
    "startMs": 1046000,
    "text": "the next step is to based on the uh canonical issues we found we are group"
  },
  {
    "startMs": 1048230,
    "text": "canonical issues we found we are group"
  },
  {
    "startMs": 1048240,
    "text": "canonical issues we found we are group and rank them by the uh frequency and uh"
  },
  {
    "startMs": 1051750,
    "text": "and rank them by the uh frequency and uh"
  },
  {
    "startMs": 1051760,
    "text": "and rank them by the uh frequency and uh this step uh there's no"
  },
  {
    "startMs": 1054630,
    "text": "this step uh there's no"
  },
  {
    "startMs": 1054640,
    "text": "this step uh there's no am required so this is just to help us"
  },
  {
    "startMs": 1057830,
    "text": "am required so this is just to help us"
  },
  {
    "startMs": 1057840,
    "text": "am required so this is just to help us to understand uh what is the major"
  },
  {
    "startMs": 1060230,
    "text": "to understand uh what is the major"
  },
  {
    "startMs": 1060240,
    "text": "to understand uh what is the major issues bottlenecks uh in our uh AI"
  },
  {
    "startMs": 1063029,
    "text": "issues bottlenecks uh in our uh AI"
  },
  {
    "startMs": 1063039,
    "text": "issues bottlenecks uh in our uh AI system and uh once we uh do the group"
  },
  {
    "startMs": 1066310,
    "text": "system and uh once we uh do the group"
  },
  {
    "startMs": 1066320,
    "text": "system and uh once we uh do the group and ranking and we are generated the uh"
  },
  {
    "startMs": 1068390,
    "text": "and ranking and we are generated the uh"
  },
  {
    "startMs": 1068400,
    "text": "and ranking and we are generated the uh scorec card for the human to uh get uh"
  },
  {
    "startMs": 1071830,
    "text": "scorec card for the human to uh get uh"
  },
  {
    "startMs": 1071840,
    "text": "scorec card for the human to uh get uh to just uh get a human to act on to uh"
  },
  {
    "startMs": 1075990,
    "text": "to just uh get a human to act on to uh"
  },
  {
    "startMs": 1076000,
    "text": "to just uh get a human to act on to uh continue uh iterate our uh AI system and"
  },
  {
    "startMs": 1079750,
    "text": "continue uh iterate our uh AI system and"
  },
  {
    "startMs": 1079760,
    "text": "continue uh iterate our uh AI system and all uh and also we are uh uh like"
  },
  {
    "startMs": 1082789,
    "text": "all uh and also we are uh uh like"
  },
  {
    "startMs": 1082799,
    "text": "all uh and also we are uh uh like basically just a cache I say save those"
  },
  {
    "startMs": 1085590,
    "text": "basically just a cache I say save those"
  },
  {
    "startMs": 1085600,
    "text": "basically just a cache I say save those root cause uh somewhere and it will"
  },
  {
    "startMs": 1088470,
    "text": "root cause uh somewhere and it will"
  },
  {
    "startMs": 1088480,
    "text": "root cause uh somewhere and it will continue use this for the self question"
  },
  {
    "startMs": 1090549,
    "text": "continue use this for the self question"
  },
  {
    "startMs": 1090559,
    "text": "continue use this for the self question loop. So that's why so you can see uh"
  },
  {
    "startMs": 1093190,
    "text": "loop. So that's why so you can see uh"
  },
  {
    "startMs": 1093200,
    "text": "loop. So that's why so you can see uh like there's a arrow here like a go back"
  },
  {
    "startMs": 1096070,
    "text": "like there's a arrow here like a go back"
  },
  {
    "startMs": 1096080,
    "text": "like there's a arrow here like a go back to the uh L2 judge. So be like it will"
  },
  {
    "startMs": 1099430,
    "text": "to the uh L2 judge. So be like it will"
  },
  {
    "startMs": 1099440,
    "text": "to the uh L2 judge. So be like it will guide us on like what is basically the"
  },
  {
    "startMs": 1102070,
    "text": "guide us on like what is basically the"
  },
  {
    "startMs": 1102080,
    "text": "guide us on like what is basically the pri prior diagnose prime on the next"
  },
  {
    "startMs": 1104630,
    "text": "pri prior diagnose prime on the next"
  },
  {
    "startMs": 1104640,
    "text": "pri prior diagnose prime on the next turn. Uh so that is easier for the uh"
  },
  {
    "startMs": 1107669,
    "text": "turn. Uh so that is easier for the uh"
  },
  {
    "startMs": 1107679,
    "text": "turn. Uh so that is easier for the uh like to judge to identify the issues uh"
  },
  {
    "startMs": 1110630,
    "text": "like to judge to identify the issues uh"
  },
  {
    "startMs": 1110640,
    "text": "like to judge to identify the issues uh in future runs. So that's uh uh like the"
  },
  {
    "startMs": 1114230,
    "text": "in future runs. So that's uh uh like the"
  },
  {
    "startMs": 1114240,
    "text": "in future runs. So that's uh uh like the what we do for the agent uh reflection."
  },
  {
    "startMs": 1117590,
    "text": "what we do for the agent uh reflection."
  },
  {
    "startMs": 1117600,
    "text": "what we do for the agent uh reflection. Uh so the next step is a generator edits"
  },
  {
    "startMs": 1120870,
    "text": "Uh so the next step is a generator edits"
  },
  {
    "startMs": 1120880,
    "text": "Uh so the next step is a generator edits proposal because from the step three uh"
  },
  {
    "startMs": 1123190,
    "text": "proposal because from the step three uh"
  },
  {
    "startMs": 1123200,
    "text": "proposal because from the step three uh the previous slide we know uh we already"
  },
  {
    "startMs": 1125590,
    "text": "the previous slide we know uh we already"
  },
  {
    "startMs": 1125600,
    "text": "the previous slide we know uh we already know okay so what is the root cause"
  },
  {
    "startMs": 1127510,
    "text": "know okay so what is the root cause"
  },
  {
    "startMs": 1127520,
    "text": "know okay so what is the root cause right why like the agent fails so the"
  },
  {
    "startMs": 1130630,
    "text": "right why like the agent fails so the"
  },
  {
    "startMs": 1130640,
    "text": "right why like the agent fails so the next step is naturally like given uh"
  },
  {
    "startMs": 1133510,
    "text": "next step is naturally like given uh"
  },
  {
    "startMs": 1133520,
    "text": "next step is naturally like given uh those root cause those issues we want to"
  },
  {
    "startMs": 1135909,
    "text": "those root cause those issues we want to"
  },
  {
    "startMs": 1135919,
    "text": "those root cause those issues we want to fix so what what can we do right what is"
  },
  {
    "startMs": 1139190,
    "text": "fix so what what can we do right what is"
  },
  {
    "startMs": 1139200,
    "text": "fix so what what can we do right what is u new edits we want to propose and uh"
  },
  {
    "startMs": 1142310,
    "text": "u new edits we want to propose and uh"
  },
  {
    "startMs": 1142320,
    "text": "u new edits we want to propose and uh here like"
  },
  {
    "startMs": 1144150,
    "text": "here like"
  },
  {
    "startMs": 1144160,
    "text": "here like we have uh again like this kind of we"
  },
  {
    "startMs": 1147270,
    "text": "we have uh again like this kind of we"
  },
  {
    "startMs": 1147280,
    "text": "we have uh again like this kind of we call it the proposal uh life cycle. So"
  },
  {
    "startMs": 1149830,
    "text": "call it the proposal uh life cycle. So"
  },
  {
    "startMs": 1149840,
    "text": "call it the proposal uh life cycle. So based on the judge uh evidence and the"
  },
  {
    "startMs": 1152470,
    "text": "based on the judge uh evidence and the"
  },
  {
    "startMs": 1152480,
    "text": "based on the judge uh evidence and the reflection artifact we will propose the"
  },
  {
    "startMs": 1155270,
    "text": "reflection artifact we will propose the"
  },
  {
    "startMs": 1155280,
    "text": "reflection artifact we will propose the input and uh we will run uh some"
  },
  {
    "startMs": 1158230,
    "text": "input and uh we will run uh some"
  },
  {
    "startMs": 1158240,
    "text": "input and uh we will run uh some methodology. So uh for this project uh"
  },
  {
    "startMs": 1161590,
    "text": "methodology. So uh for this project uh"
  },
  {
    "startMs": 1161600,
    "text": "methodology. So uh for this project uh we are using the uh JPA to uh given the"
  },
  {
    "startMs": 1165350,
    "text": "we are using the uh JPA to uh given the"
  },
  {
    "startMs": 1165360,
    "text": "we are using the uh JPA to uh given the uh you know the uh lookout and to uh"
  },
  {
    "startMs": 1168549,
    "text": "uh you know the uh lookout and to uh"
  },
  {
    "startMs": 1168559,
    "text": "uh you know the uh lookout and to uh propose the new prompt change and uh and"
  },
  {
    "startMs": 1171990,
    "text": "propose the new prompt change and uh and"
  },
  {
    "startMs": 1172000,
    "text": "propose the new prompt change and uh and like given a trace sample we do it in a"
  },
  {
    "startMs": 1175430,
    "text": "like given a trace sample we do it in a"
  },
  {
    "startMs": 1175440,
    "text": "like given a trace sample we do it in a mini batch uh fashion and uh and"
  },
  {
    "startMs": 1178310,
    "text": "mini batch uh fashion and uh and"
  },
  {
    "startMs": 1178320,
    "text": "mini batch uh fashion and uh and generated the new uh prompt updates"
  },
  {
    "startMs": 1180710,
    "text": "generated the new uh prompt updates"
  },
  {
    "startMs": 1180720,
    "text": "generated the new uh prompt updates proposal and once we generate the promp"
  },
  {
    "startMs": 1183350,
    "text": "proposal and once we generate the promp"
  },
  {
    "startMs": 1183360,
    "text": "proposal and once we generate the promp uh updates proposal we'll uh select the"
  },
  {
    "startMs": 1186070,
    "text": "uh updates proposal we'll uh select the"
  },
  {
    "startMs": 1186080,
    "text": "uh updates proposal we'll uh select the window prompts and do the uh validation"
  },
  {
    "startMs": 1189110,
    "text": "window prompts and do the uh validation"
  },
  {
    "startMs": 1189120,
    "text": "window prompts and do the uh validation validation uh set"
  },
  {
    "startMs": 1192470,
    "text": "validation uh set"
  },
  {
    "startMs": 1192480,
    "text": "validation uh set and uh once we uh found a good prompt"
  },
  {
    "startMs": 1195909,
    "text": "and uh once we uh found a good prompt"
  },
  {
    "startMs": 1195919,
    "text": "and uh once we uh found a good prompt candidates and we run the uh you know"
  },
  {
    "startMs": 1198310,
    "text": "candidates and we run the uh you know"
  },
  {
    "startMs": 1198320,
    "text": "candidates and we run the uh you know the validation set and we think uh it is"
  },
  {
    "startMs": 1201270,
    "text": "the validation set and we think uh it is"
  },
  {
    "startMs": 1201280,
    "text": "the validation set and we think uh it is uh become uh it is a good uh prompt"
  },
  {
    "startMs": 1204390,
    "text": "uh become uh it is a good uh prompt"
  },
  {
    "startMs": 1204400,
    "text": "uh become uh it is a good uh prompt updates and uh we are just um updates"
  },
  {
    "startMs": 1208230,
    "text": "updates and uh we are just um updates"
  },
  {
    "startMs": 1208240,
    "text": "updates and uh we are just um updates our prompt in our system. So this will"
  },
  {
    "startMs": 1210549,
    "text": "our prompt in our system. So this will"
  },
  {
    "startMs": 1210559,
    "text": "our prompt in our system. So this will be and into the production and this will"
  },
  {
    "startMs": 1213029,
    "text": "be and into the production and this will"
  },
  {
    "startMs": 1213039,
    "text": "be and into the production and this will become our new uh kind of like a"
  },
  {
    "startMs": 1215190,
    "text": "become our new uh kind of like a"
  },
  {
    "startMs": 1215200,
    "text": "become our new uh kind of like a baseline and we will continue to uh do"
  },
  {
    "startMs": 1217909,
    "text": "baseline and we will continue to uh do"
  },
  {
    "startMs": 1217919,
    "text": "baseline and we will continue to uh do the uh like the future uh RSI uh"
  },
  {
    "startMs": 1221270,
    "text": "the uh like the future uh RSI uh"
  },
  {
    "startMs": 1221280,
    "text": "the uh like the future uh RSI uh iterations. So yeah that's uh kind of"
  },
  {
    "startMs": 1224230,
    "text": "iterations. So yeah that's uh kind of"
  },
  {
    "startMs": 1224240,
    "text": "iterations. So yeah that's uh kind of like the and uh yeah that's a kind of"
  },
  {
    "startMs": 1226230,
    "text": "like the and uh yeah that's a kind of"
  },
  {
    "startMs": 1226240,
    "text": "like the and uh yeah that's a kind of like the entire life cycle of this RSI"
  },
  {
    "startMs": 1229830,
    "text": "like the entire life cycle of this RSI"
  },
  {
    "startMs": 1229840,
    "text": "like the entire life cycle of this RSI uh iteration. And uh next this slide uh"
  },
  {
    "startMs": 1233029,
    "text": "uh iteration. And uh next this slide uh"
  },
  {
    "startMs": 1233039,
    "text": "uh iteration. And uh next this slide uh we want to show our primary focus for"
  },
  {
    "startMs": 1235750,
    "text": "we want to show our primary focus for"
  },
  {
    "startMs": 1235760,
    "text": "we want to show our primary focus for this RSI work. So we so currently we"
  },
  {
    "startMs": 1239190,
    "text": "this RSI work. So we so currently we"
  },
  {
    "startMs": 1239200,
    "text": "this RSI work. So we so currently we have three major focuses. So the first"
  },
  {
    "startMs": 1241590,
    "text": "have three major focuses. So the first"
  },
  {
    "startMs": 1241600,
    "text": "have three major focuses. So the first is the quality, second is automation and"
  },
  {
    "startMs": 1244149,
    "text": "is the quality, second is automation and"
  },
  {
    "startMs": 1244159,
    "text": "is the quality, second is automation and the third is a production uh updates. So"
  },
  {
    "startMs": 1247029,
    "text": "the third is a production uh updates. So"
  },
  {
    "startMs": 1247039,
    "text": "the third is a production uh updates. So quality uh like we are focused on the"
  },
  {
    "startMs": 1250230,
    "text": "quality uh like we are focused on the"
  },
  {
    "startMs": 1250240,
    "text": "quality uh like we are focused on the different uh uh things uh on the"
  },
  {
    "startMs": 1253350,
    "text": "different uh uh things uh on the"
  },
  {
    "startMs": 1253360,
    "text": "different uh uh things uh on the quality. So um and we uh did many uh"
  },
  {
    "startMs": 1256710,
    "text": "quality. So um and we uh did many uh"
  },
  {
    "startMs": 1256720,
    "text": "quality. So um and we uh did many uh iterations to trying to improve the uh"
  },
  {
    "startMs": 1259270,
    "text": "iterations to trying to improve the uh"
  },
  {
    "startMs": 1259280,
    "text": "iterations to trying to improve the uh quality of our agent uh judge and this"
  },
  {
    "startMs": 1263350,
    "text": "quality of our agent uh judge and this"
  },
  {
    "startMs": 1263360,
    "text": "quality of our agent uh judge and this RSI loop. For example, we did extensive"
  },
  {
    "startMs": 1265990,
    "text": "RSI loop. For example, we did extensive"
  },
  {
    "startMs": 1266000,
    "text": "RSI loop. For example, we did extensive human annotations so that we can improve"
  },
  {
    "startMs": 1269270,
    "text": "human annotations so that we can improve"
  },
  {
    "startMs": 1269280,
    "text": "human annotations so that we can improve the judge both the L1 judge and the L2"
  },
  {
    "startMs": 1272070,
    "text": "the judge both the L1 judge and the L2"
  },
  {
    "startMs": 1272080,
    "text": "the judge both the L1 judge and the L2 judge uh quality and we did uh many"
  },
  {
    "startMs": 1275750,
    "text": "judge uh quality and we did uh many"
  },
  {
    "startMs": 1275760,
    "text": "judge uh quality and we did uh many study and analysis on assess agent judge"
  },
  {
    "startMs": 1278710,
    "text": "study and analysis on assess agent judge"
  },
  {
    "startMs": 1278720,
    "text": "study and analysis on assess agent judge uh uh variance so that we can uh make"
  },
  {
    "startMs": 1281430,
    "text": "uh uh variance so that we can uh make"
  },
  {
    "startMs": 1281440,
    "text": "uh uh variance so that we can uh make sure we can kind of mitigate the impact"
  },
  {
    "startMs": 1284470,
    "text": "sure we can kind of mitigate the impact"
  },
  {
    "startMs": 1284480,
    "text": "sure we can kind of mitigate the impact from the non nondeterministic nature of"
  },
  {
    "startMs": 1287350,
    "text": "from the non nondeterministic nature of"
  },
  {
    "startMs": 1287360,
    "text": "from the non nondeterministic nature of the LM and uh uh the third one is we"
  },
  {
    "startMs": 1290710,
    "text": "the LM and uh uh the third one is we"
  },
  {
    "startMs": 1290720,
    "text": "the LM and uh uh the third one is we push quality updates to both Asian judge"
  },
  {
    "startMs": 1293430,
    "text": "push quality updates to both Asian judge"
  },
  {
    "startMs": 1293440,
    "text": "push quality updates to both Asian judge and RSI reflection uh based on our"
  },
  {
    "startMs": 1296870,
    "text": "and RSI reflection uh based on our"
  },
  {
    "startMs": 1296880,
    "text": "and RSI reflection uh based on our annotation and the variance analysis uh"
  },
  {
    "startMs": 1299990,
    "text": "annotation and the variance analysis uh"
  },
  {
    "startMs": 1300000,
    "text": "annotation and the variance analysis uh mentioned uh above. So that's on the CI"
  },
  {
    "startMs": 1303590,
    "text": "mentioned uh above. So that's on the CI"
  },
  {
    "startMs": 1303600,
    "text": "mentioned uh above. So that's on the CI side and the second focus is uh"
  },
  {
    "startMs": 1305909,
    "text": "side and the second focus is uh"
  },
  {
    "startMs": 1305919,
    "text": "side and the second focus is uh automation uh because as we mentioned so"
  },
  {
    "startMs": 1308870,
    "text": "automation uh because as we mentioned so"
  },
  {
    "startMs": 1308880,
    "text": "automation uh because as we mentioned so there are seven steps in the RSI"
  },
  {
    "startMs": 1311510,
    "text": "there are seven steps in the RSI"
  },
  {
    "startMs": 1311520,
    "text": "there are seven steps in the RSI iteration and uh so that's why like here"
  },
  {
    "startMs": 1314549,
    "text": "iteration and uh so that's why like here"
  },
  {
    "startMs": 1314559,
    "text": "iteration and uh so that's why like here like I we build a uh this kind like"
  },
  {
    "startMs": 1317510,
    "text": "like I we build a uh this kind like"
  },
  {
    "startMs": 1317520,
    "text": "like I we build a uh this kind like workflow so that we can kick off this"
  },
  {
    "startMs": 1320230,
    "text": "workflow so that we can kick off this"
  },
  {
    "startMs": 1320240,
    "text": "workflow so that we can kick off this workflow with some uh cadence for weekly"
  },
  {
    "startMs": 1323110,
    "text": "workflow with some uh cadence for weekly"
  },
  {
    "startMs": 1323120,
    "text": "workflow with some uh cadence for weekly or bi-weekly or monthly based on the"
  },
  {
    "startMs": 1325669,
    "text": "or bi-weekly or monthly based on the"
  },
  {
    "startMs": 1325679,
    "text": "or bi-weekly or monthly based on the needs uh so that it can u automatically"
  },
  {
    "startMs": 1328549,
    "text": "needs uh so that it can u automatically"
  },
  {
    "startMs": 1328559,
    "text": "needs uh so that it can u automatically running and keep suggesting the uh like"
  },
  {
    "startMs": 1331270,
    "text": "running and keep suggesting the uh like"
  },
  {
    "startMs": 1331280,
    "text": "running and keep suggesting the uh like the prompt updates uh with some uh"
  },
  {
    "startMs": 1333590,
    "text": "the prompt updates uh with some uh"
  },
  {
    "startMs": 1333600,
    "text": "the prompt updates uh with some uh cadence and the third focus is"
  },
  {
    "startMs": 1336630,
    "text": "cadence and the third focus is"
  },
  {
    "startMs": 1336640,
    "text": "cadence and the third focus is production uh updates. So definitely"
  },
  {
    "startMs": 1338789,
    "text": "production uh updates. So definitely"
  },
  {
    "startMs": 1338799,
    "text": "production uh updates. So definitely like we want to uh build a uh useful uh"
  },
  {
    "startMs": 1342230,
    "text": "like we want to uh build a uh useful uh"
  },
  {
    "startMs": 1342240,
    "text": "like we want to uh build a uh useful uh RSI uh system that can help uh keep"
  },
  {
    "startMs": 1345990,
    "text": "RSI uh system that can help uh keep"
  },
  {
    "startMs": 1346000,
    "text": "RSI uh system that can help uh keep continuously uh improve our uh you know"
  },
  {
    "startMs": 1348710,
    "text": "continuously uh improve our uh you know"
  },
  {
    "startMs": 1348720,
    "text": "continuously uh improve our uh you know product. So that's why like we the third"
  },
  {
    "startMs": 1351590,
    "text": "product. So that's why like we the third"
  },
  {
    "startMs": 1351600,
    "text": "product. So that's why like we the third focus uh for the our our uh work is to"
  },
  {
    "startMs": 1355510,
    "text": "focus uh for the our our uh work is to"
  },
  {
    "startMs": 1355520,
    "text": "focus uh for the our our uh work is to make updates to the current agent in"
  },
  {
    "startMs": 1357669,
    "text": "make updates to the current agent in"
  },
  {
    "startMs": 1357679,
    "text": "make updates to the current agent in production with human in the loop uh"
  },
  {
    "startMs": 1360070,
    "text": "production with human in the loop uh"
  },
  {
    "startMs": 1360080,
    "text": "production with human in the loop uh like for now to uh ensure like the"
  },
  {
    "startMs": 1363110,
    "text": "like for now to uh ensure like the"
  },
  {
    "startMs": 1363120,
    "text": "like for now to uh ensure like the production uh uh quality uh is there but"
  },
  {
    "startMs": 1366630,
    "text": "production uh uh quality uh is there but"
  },
  {
    "startMs": 1366640,
    "text": "production uh uh quality uh is there but later like once we think uh the quality"
  },
  {
    "startMs": 1369430,
    "text": "later like once we think uh the quality"
  },
  {
    "startMs": 1369440,
    "text": "later like once we think uh the quality is good enough like we gradually uh uh"
  },
  {
    "startMs": 1373909,
    "text": "is good enough like we gradually uh uh"
  },
  {
    "startMs": 1373919,
    "text": "is good enough like we gradually uh uh like reduce the human effort and uh and"
  },
  {
    "startMs": 1377110,
    "text": "like reduce the human effort and uh and"
  },
  {
    "startMs": 1377120,
    "text": "like reduce the human effort and uh and hopefully we can achieve a higher"
  },
  {
    "startMs": 1379669,
    "text": "hopefully we can achieve a higher"
  },
  {
    "startMs": 1379679,
    "text": "hopefully we can achieve a higher automation uh kind of uh state."
  },
  {
    "startMs": 1383830,
    "text": "automation uh kind of uh state."
  },
  {
    "startMs": 1383840,
    "text": "automation uh kind of uh state. Uh cool. Yeah. Next, I will hand over uh"
  },
  {
    "startMs": 1386470,
    "text": "Uh cool. Yeah. Next, I will hand over uh"
  },
  {
    "startMs": 1386480,
    "text": "Uh cool. Yeah. Next, I will hand over uh to Saloni to uh share more details about"
  },
  {
    "startMs": 1389350,
    "text": "to Saloni to uh share more details about"
  },
  {
    "startMs": 1389360,
    "text": "to Saloni to uh share more details about the outcomes and uh learnings of our our"
  },
  {
    "startMs": 1393029,
    "text": "the outcomes and uh learnings of our our"
  },
  {
    "startMs": 1393039,
    "text": "the outcomes and uh learnings of our our work."
  },
  {
    "startMs": 1396880,
    "text": "I will share my screen."
  },
  {
    "startMs": 1398390,
    "text": "I will share my screen."
  },
  {
    "startMs": 1398400,
    "text": "I will share my screen. Sure. Yeah."
  },
  {
    "startMs": 1409039,
    "text": "Um, can you see my screen?"
  },
  {
    "startMs": 1411830,
    "text": "Um, can you see my screen?"
  },
  {
    "startMs": 1411840,
    "text": "Um, can you see my screen? Yeah. Yeah."
  },
  {
    "startMs": 1413430,
    "text": "Yeah. Yeah."
  },
  {
    "startMs": 1413440,
    "text": "Yeah. Yeah. Yeah. Um, so yeah. Hi, I'm Salon. And so"
  },
  {
    "startMs": 1415990,
    "text": "Yeah. Um, so yeah. Hi, I'm Salon. And so"
  },
  {
    "startMs": 1416000,
    "text": "Yeah. Um, so yeah. Hi, I'm Salon. And so far we described how the RSI system is"
  },
  {
    "startMs": 1419029,
    "text": "far we described how the RSI system is"
  },
  {
    "startMs": 1419039,
    "text": "far we described how the RSI system is designed. Uh, but one of the most"
  },
  {
    "startMs": 1420710,
    "text": "designed. Uh, but one of the most"
  },
  {
    "startMs": 1420720,
    "text": "designed. Uh, but one of the most interesting aspects of this work has"
  },
  {
    "startMs": 1422789,
    "text": "interesting aspects of this work has"
  },
  {
    "startMs": 1422799,
    "text": "interesting aspects of this work has been understanding what happens when we"
  },
  {
    "startMs": 1425669,
    "text": "been understanding what happens when we"
  },
  {
    "startMs": 1425679,
    "text": "been understanding what happens when we actually try to make this process"
  },
  {
    "startMs": 1427270,
    "text": "actually try to make this process"
  },
  {
    "startMs": 1427280,
    "text": "actually try to make this process reliable. And in particular, we learned"
  },
  {
    "startMs": 1429750,
    "text": "reliable. And in particular, we learned"
  },
  {
    "startMs": 1429760,
    "text": "reliable. And in particular, we learned that the quality of the evaluator is"
  },
  {
    "startMs": 1432070,
    "text": "that the quality of the evaluator is"
  },
  {
    "startMs": 1432080,
    "text": "that the quality of the evaluator is just as important as the quality of the"
  },
  {
    "startMs": 1434230,
    "text": "just as important as the quality of the"
  },
  {
    "startMs": 1434240,
    "text": "just as important as the quality of the optimization system. So let's walk"
  },
  {
    "startMs": 1436789,
    "text": "optimization system. So let's walk"
  },
  {
    "startMs": 1436799,
    "text": "optimization system. So let's walk through some of those um findings. Um so"
  },
  {
    "startMs": 1440230,
    "text": "through some of those um findings. Um so"
  },
  {
    "startMs": 1440240,
    "text": "through some of those um findings. Um so there are two major outcomes we want to"
  },
  {
    "startMs": 1441750,
    "text": "there are two major outcomes we want to"
  },
  {
    "startMs": 1441760,
    "text": "there are two major outcomes we want to highlight. So the first is that we"
  },
  {
    "startMs": 1443510,
    "text": "highlight. So the first is that we"
  },
  {
    "startMs": 1443520,
    "text": "highlight. So the first is that we successfully like deployed our initial"
  },
  {
    "startMs": 1445909,
    "text": "successfully like deployed our initial"
  },
  {
    "startMs": 1445919,
    "text": "successfully like deployed our initial RSI improvements into uh production"
  },
  {
    "startMs": 1448390,
    "text": "RSI improvements into uh production"
  },
  {
    "startMs": 1448400,
    "text": "RSI improvements into uh production which was like um landing a first PR"
  },
  {
    "startMs": 1451350,
    "text": "which was like um landing a first PR"
  },
  {
    "startMs": 1451360,
    "text": "which was like um landing a first PR which was optimized by Japa and through"
  },
  {
    "startMs": 1453029,
    "text": "which was optimized by Japa and through"
  },
  {
    "startMs": 1453039,
    "text": "which was optimized by Japa and through the loop which we just described. Um so"
  },
  {
    "startMs": 1455350,
    "text": "the loop which we just described. Um so"
  },
  {
    "startMs": 1455360,
    "text": "the loop which we just described. Um so these changes went through human review"
  },
  {
    "startMs": 1457430,
    "text": "these changes went through human review"
  },
  {
    "startMs": 1457440,
    "text": "these changes went through human review and quantitative measurement before"
  },
  {
    "startMs": 1459269,
    "text": "and quantitative measurement before"
  },
  {
    "startMs": 1459279,
    "text": "and quantitative measurement before deployment and it was an important"
  },
  {
    "startMs": 1461029,
    "text": "deployment and it was an important"
  },
  {
    "startMs": 1461039,
    "text": "deployment and it was an important milestone in moving RSI like beyond just"
  },
  {
    "startMs": 1463750,
    "text": "milestone in moving RSI like beyond just"
  },
  {
    "startMs": 1463760,
    "text": "milestone in moving RSI like beyond just an experimental workflow. The second"
  },
  {
    "startMs": 1466149,
    "text": "an experimental workflow. The second"
  },
  {
    "startMs": 1466159,
    "text": "an experimental workflow. The second major improvement was around like um"
  },
  {
    "startMs": 1468470,
    "text": "major improvement was around like um"
  },
  {
    "startMs": 1468480,
    "text": "major improvement was around like um evaluation quality where we invested"
  },
  {
    "startMs": 1470549,
    "text": "evaluation quality where we invested"
  },
  {
    "startMs": 1470559,
    "text": "evaluation quality where we invested substantial effort and like human"
  },
  {
    "startMs": 1472070,
    "text": "substantial effort and like human"
  },
  {
    "startMs": 1472080,
    "text": "substantial effort and like human annotations for the user understanding"
  },
  {
    "startMs": 1475269,
    "text": "annotations for the user understanding"
  },
  {
    "startMs": 1475279,
    "text": "annotations for the user understanding judge and as well as like analyzing"
  },
  {
    "startMs": 1476950,
    "text": "judge and as well as like analyzing"
  },
  {
    "startMs": 1476960,
    "text": "judge and as well as like analyzing where its judgment sort of like"
  },
  {
    "startMs": 1478470,
    "text": "where its judgment sort of like"
  },
  {
    "startMs": 1478480,
    "text": "where its judgment sort of like disagreed with like human reviewers. And"
  },
  {
    "startMs": 1481190,
    "text": "disagreed with like human reviewers. And"
  },
  {
    "startMs": 1481200,
    "text": "disagreed with like human reviewers. And this process revealed several recurring"
  },
  {
    "startMs": 1483190,
    "text": "this process revealed several recurring"
  },
  {
    "startMs": 1483200,
    "text": "this process revealed several recurring weaknesses in both our evaluation"
  },
  {
    "startMs": 1485510,
    "text": "weaknesses in both our evaluation"
  },
  {
    "startMs": 1485520,
    "text": "weaknesses in both our evaluation rubrics as well as our like our"
  },
  {
    "startMs": 1486950,
    "text": "rubrics as well as our like our"
  },
  {
    "startMs": 1486960,
    "text": "rubrics as well as our like our reflection system. And by investigating"
  },
  {
    "startMs": 1489350,
    "text": "reflection system. And by investigating"
  },
  {
    "startMs": 1489360,
    "text": "reflection system. And by investigating those disagreements, we were able to"
  },
  {
    "startMs": 1490870,
    "text": "those disagreements, we were able to"
  },
  {
    "startMs": 1490880,
    "text": "those disagreements, we were able to like make targeted improvements which"
  },
  {
    "startMs": 1493110,
    "text": "like make targeted improvements which"
  },
  {
    "startMs": 1493120,
    "text": "like make targeted improvements which further strengthen the reliability as"
  },
  {
    "startMs": 1495110,
    "text": "further strengthen the reliability as"
  },
  {
    "startMs": 1495120,
    "text": "further strengthen the reliability as well as like reduce the evaluation"
  },
  {
    "startMs": 1496789,
    "text": "well as like reduce the evaluation"
  },
  {
    "startMs": 1496799,
    "text": "well as like reduce the evaluation variance which we have. And uh before we"
  },
  {
    "startMs": 1499590,
    "text": "variance which we have. And uh before we"
  },
  {
    "startMs": 1499600,
    "text": "variance which we have. And uh before we could trust a system to like improve an"
  },
  {
    "startMs": 1501269,
    "text": "could trust a system to like improve an"
  },
  {
    "startMs": 1501279,
    "text": "could trust a system to like improve an agent automatically, we had to"
  },
  {
    "startMs": 1502950,
    "text": "agent automatically, we had to"
  },
  {
    "startMs": 1502960,
    "text": "agent automatically, we had to understand whether the feedback driving"
  },
  {
    "startMs": 1504390,
    "text": "understand whether the feedback driving"
  },
  {
    "startMs": 1504400,
    "text": "understand whether the feedback driving those improvements was also itself"
  },
  {
    "startMs": 1506310,
    "text": "those improvements was also itself"
  },
  {
    "startMs": 1506320,
    "text": "those improvements was also itself trustworthy. And this was like a major"
  },
  {
    "startMs": 1509350,
    "text": "trustworthy. And this was like a major"
  },
  {
    "startMs": 1509360,
    "text": "trustworthy. And this was like a major area of our engineering work. Um to so"
  },
  {
    "startMs": 1512549,
    "text": "area of our engineering work. Um to so"
  },
  {
    "startMs": 1512559,
    "text": "area of our engineering work. Um to so to understand the quality of our like um"
  },
  {
    "startMs": 1514870,
    "text": "to understand the quality of our like um"
  },
  {
    "startMs": 1514880,
    "text": "to understand the quality of our like um L1 which was our evaluation judge. We"
  },
  {
    "startMs": 1516789,
    "text": "L1 which was our evaluation judge. We"
  },
  {
    "startMs": 1516799,
    "text": "L1 which was our evaluation judge. We analyzed about 180 to 200 human reviews."
  },
  {
    "startMs": 1520710,
    "text": "analyzed about 180 to 200 human reviews."
  },
  {
    "startMs": 1520720,
    "text": "analyzed about 180 to 200 human reviews. We were particularly interested in how"
  },
  {
    "startMs": 1522230,
    "text": "We were particularly interested in how"
  },
  {
    "startMs": 1522240,
    "text": "We were particularly interested in how the judge agreed with human reviewers"
  },
  {
    "startMs": 1524230,
    "text": "the judge agreed with human reviewers"
  },
  {
    "startMs": 1524240,
    "text": "the judge agreed with human reviewers both across different score categories"
  },
  {
    "startMs": 1526950,
    "text": "both across different score categories"
  },
  {
    "startMs": 1526960,
    "text": "both across different score categories and across the individual evaluation"
  },
  {
    "startMs": 1529430,
    "text": "and across the individual evaluation"
  },
  {
    "startMs": 1529440,
    "text": "and across the individual evaluation dimensions. Starting with the chart on"
  },
  {
    "startMs": 1531110,
    "text": "dimensions. Starting with the chart on"
  },
  {
    "startMs": 1531120,
    "text": "dimensions. Starting with the chart on the left, we see basically that um for"
  },
  {
    "startMs": 1533750,
    "text": "the left, we see basically that um for"
  },
  {
    "startMs": 1533760,
    "text": "the left, we see basically that um for outputs we assigned a score of like one"
  },
  {
    "startMs": 1536950,
    "text": "outputs we assigned a score of like one"
  },
  {
    "startMs": 1536960,
    "text": "outputs we assigned a score of like one um a full judgment alignment was"
  },
  {
    "startMs": 1538710,
    "text": "um a full judgment alignment was"
  },
  {
    "startMs": 1538720,
    "text": "um a full judgment alignment was approximately 55% and for score two it"
  },
  {
    "startMs": 1541590,
    "text": "approximately 55% and for score two it"
  },
  {
    "startMs": 1541600,
    "text": "approximately 55% and for score two it was around 57%. But for score three the"
  },
  {
    "startMs": 1544470,
    "text": "was around 57%. But for score three the"
  },
  {
    "startMs": 1544480,
    "text": "was around 57%. But for score three the alignment increased to approximately"
  },
  {
    "startMs": 1546470,
    "text": "alignment increased to approximately"
  },
  {
    "startMs": 1546480,
    "text": "alignment increased to approximately like 80%. Which indicates an interesting"
  },
  {
    "startMs": 1549350,
    "text": "like 80%. Which indicates an interesting"
  },
  {
    "startMs": 1549360,
    "text": "like 80%. Which indicates an interesting pattern which was that the judge was"
  },
  {
    "startMs": 1551029,
    "text": "pattern which was that the judge was"
  },
  {
    "startMs": 1551039,
    "text": "pattern which was that the judge was considerably like better at recognizing"
  },
  {
    "startMs": 1552710,
    "text": "considerably like better at recognizing"
  },
  {
    "startMs": 1552720,
    "text": "considerably like better at recognizing stronger outputs than it was at like"
  },
  {
    "startMs": 1555350,
    "text": "stronger outputs than it was at like"
  },
  {
    "startMs": 1555360,
    "text": "stronger outputs than it was at like distinguishing between different sort of"
  },
  {
    "startMs": 1557029,
    "text": "distinguishing between different sort of"
  },
  {
    "startMs": 1557039,
    "text": "distinguishing between different sort of like weaker outputs. And we found that"
  },
  {
    "startMs": 1559830,
    "text": "like weaker outputs. And we found that"
  },
  {
    "startMs": 1559840,
    "text": "like weaker outputs. And we found that many of its uh lowest scores were like"
  },
  {
    "startMs": 1562230,
    "text": "many of its uh lowest scores were like"
  },
  {
    "startMs": 1562240,
    "text": "many of its uh lowest scores were like also overly harsh. And um now looking at"
  },
  {
    "startMs": 1565830,
    "text": "also overly harsh. And um now looking at"
  },
  {
    "startMs": 1565840,
    "text": "also overly harsh. And um now looking at the chart on right agreement also varied"
  },
  {
    "startMs": 1568149,
    "text": "the chart on right agreement also varied"
  },
  {
    "startMs": 1568159,
    "text": "the chart on right agreement also varied significantly across like different"
  },
  {
    "startMs": 1569590,
    "text": "significantly across like different"
  },
  {
    "startMs": 1569600,
    "text": "significantly across like different dimensions where higher preferred"
  },
  {
    "startMs": 1572310,
    "text": "dimensions where higher preferred"
  },
  {
    "startMs": 1572320,
    "text": "dimensions where higher preferred dimensions like titles had like the"
  },
  {
    "startMs": 1574070,
    "text": "dimensions like titles had like the"
  },
  {
    "startMs": 1574080,
    "text": "dimensions like titles had like the weakest observed alignment as well as"
  },
  {
    "startMs": 1575830,
    "text": "weakest observed alignment as well as"
  },
  {
    "startMs": 1575840,
    "text": "weakest observed alignment as well as like uh domains or like the expertise or"
  },
  {
    "startMs": 1578630,
    "text": "like uh domains or like the expertise or"
  },
  {
    "startMs": 1578640,
    "text": "like uh domains or like the expertise or area of work was around like 47% and the"
  },
  {
    "startMs": 1582310,
    "text": "area of work was around like 47% and the"
  },
  {
    "startMs": 1582320,
    "text": "area of work was around like 47% and the sort of like work type was around 50%."
  },
  {
    "startMs": 1585430,
    "text": "sort of like work type was around 50%."
  },
  {
    "startMs": 1585440,
    "text": "sort of like work type was around 50%. which told us like the evaluation"
  },
  {
    "startMs": 1587029,
    "text": "which told us like the evaluation"
  },
  {
    "startMs": 1587039,
    "text": "which told us like the evaluation quality was uniform across all the tasks"
  },
  {
    "startMs": 1589430,
    "text": "quality was uniform across all the tasks"
  },
  {
    "startMs": 1589440,
    "text": "quality was uniform across all the tasks which we cared about and certain"
  },
  {
    "startMs": 1591750,
    "text": "which we cared about and certain"
  },
  {
    "startMs": 1591760,
    "text": "which we cared about and certain dimensions required more clearer"
  },
  {
    "startMs": 1593669,
    "text": "dimensions required more clearer"
  },
  {
    "startMs": 1593679,
    "text": "dimensions required more clearer definitions better evidence handling"
  },
  {
    "startMs": 1595830,
    "text": "definitions better evidence handling"
  },
  {
    "startMs": 1595840,
    "text": "definitions better evidence handling rules as well as like more explicit"
  },
  {
    "startMs": 1597590,
    "text": "rules as well as like more explicit"
  },
  {
    "startMs": 1597600,
    "text": "rules as well as like more explicit scoring guidance."
  },
  {
    "startMs": 1600149,
    "text": "scoring guidance."
  },
  {
    "startMs": 1600159,
    "text": "scoring guidance. Um so when we look more closely at the"
  },
  {
    "startMs": 1601990,
    "text": "Um so when we look more closely at the"
  },
  {
    "startMs": 1602000,
    "text": "Um so when we look more closely at the human annotations several recurring"
  },
  {
    "startMs": 1603830,
    "text": "human annotations several recurring"
  },
  {
    "startMs": 1603840,
    "text": "human annotations several recurring patterns emerged. the first involved"
  },
  {
    "startMs": 1606149,
    "text": "patterns emerged. the first involved"
  },
  {
    "startMs": 1606159,
    "text": "patterns emerged. the first involved like using job titles. We found cases"
  },
  {
    "startMs": 1608950,
    "text": "like using job titles. We found cases"
  },
  {
    "startMs": 1608960,
    "text": "like using job titles. We found cases where the judge would accept like a"
  },
  {
    "startMs": 1610310,
    "text": "where the judge would accept like a"
  },
  {
    "startMs": 1610320,
    "text": "where the judge would accept like a return title because it was plausible"
  },
  {
    "startMs": 1612070,
    "text": "return title because it was plausible"
  },
  {
    "startMs": 1612080,
    "text": "return title because it was plausible would but would miss the fact that the"
  },
  {
    "startMs": 1613830,
    "text": "would but would miss the fact that the"
  },
  {
    "startMs": 1613840,
    "text": "would but would miss the fact that the agent had omitted an expected um signal"
  },
  {
    "startMs": 1617110,
    "text": "agent had omitted an expected um signal"
  },
  {
    "startMs": 1617120,
    "text": "agent had omitted an expected um signal or like an ignored and basically an"
  },
  {
    "startMs": 1619110,
    "text": "or like an ignored and basically an"
  },
  {
    "startMs": 1619120,
    "text": "or like an ignored and basically an ignored and important intent signal. Um"
  },
  {
    "startMs": 1621990,
    "text": "ignored and important intent signal. Um"
  },
  {
    "startMs": 1622000,
    "text": "ignored and important intent signal. Um in other words, sometimes the judge was"
  },
  {
    "startMs": 1623590,
    "text": "in other words, sometimes the judge was"
  },
  {
    "startMs": 1623600,
    "text": "in other words, sometimes the judge was sometimes evaluating the plausibility"
  },
  {
    "startMs": 1625430,
    "text": "sometimes evaluating the plausibility"
  },
  {
    "startMs": 1625440,
    "text": "sometimes evaluating the plausibility rather than the completeness of the"
  },
  {
    "startMs": 1627190,
    "text": "rather than the completeness of the"
  },
  {
    "startMs": 1627200,
    "text": "rather than the completeness of the facts which were stated. Uh the second"
  },
  {
    "startMs": 1629750,
    "text": "facts which were stated. Uh the second"
  },
  {
    "startMs": 1629760,
    "text": "facts which were stated. Uh the second issue was primarily around primarily"
  },
  {
    "startMs": 1631830,
    "text": "issue was primarily around primarily"
  },
  {
    "startMs": 1631840,
    "text": "issue was primarily around primarily around the fields of expertise. Here the"
  },
  {
    "startMs": 1633990,
    "text": "around the fields of expertise. Here the"
  },
  {
    "startMs": 1634000,
    "text": "around the fields of expertise. Here the distinction between like a domain,"
  },
  {
    "startMs": 1635430,
    "text": "distinction between like a domain,"
  },
  {
    "startMs": 1635440,
    "text": "distinction between like a domain, function and a skill wasn't like always"
  },
  {
    "startMs": 1637269,
    "text": "function and a skill wasn't like always"
  },
  {
    "startMs": 1637279,
    "text": "function and a skill wasn't like always clear where um a domain could be like um"
  },
  {
    "startMs": 1640710,
    "text": "clear where um a domain could be like um"
  },
  {
    "startMs": 1640720,
    "text": "clear where um a domain could be like um say software engineering and a function"
  },
  {
    "startMs": 1643110,
    "text": "say software engineering and a function"
  },
  {
    "startMs": 1643120,
    "text": "say software engineering and a function or a role could be like for example a"
  },
  {
    "startMs": 1645029,
    "text": "or a role could be like for example a"
  },
  {
    "startMs": 1645039,
    "text": "or a role could be like for example a machine learning engineer while skill"
  },
  {
    "startMs": 1647190,
    "text": "machine learning engineer while skill"
  },
  {
    "startMs": 1647200,
    "text": "machine learning engineer while skill being uh machine learning. Um the rubric"
  },
  {
    "startMs": 1649750,
    "text": "being uh machine learning. Um the rubric"
  },
  {
    "startMs": 1649760,
    "text": "being uh machine learning. Um the rubric also like lack consistent partial credit"
  },
  {
    "startMs": 1652149,
    "text": "also like lack consistent partial credit"
  },
  {
    "startMs": 1652159,
    "text": "also like lack consistent partial credit rules for output containing like a"
  },
  {
    "startMs": 1654070,
    "text": "rules for output containing like a"
  },
  {
    "startMs": 1654080,
    "text": "rules for output containing like a mixture of like correct and incorrect"
  },
  {
    "startMs": 1655669,
    "text": "mixture of like correct and incorrect"
  },
  {
    "startMs": 1655679,
    "text": "mixture of like correct and incorrect information. And another issue was"
  },
  {
    "startMs": 1658390,
    "text": "information. And another issue was"
  },
  {
    "startMs": 1658400,
    "text": "information. And another issue was involving around formatting and"
  },
  {
    "startMs": 1660310,
    "text": "involving around formatting and"
  },
  {
    "startMs": 1660320,
    "text": "involving around formatting and inference. So in some cases the judge"
  },
  {
    "startMs": 1662149,
    "text": "inference. So in some cases the judge"
  },
  {
    "startMs": 1662159,
    "text": "inference. So in some cases the judge penalized harmless formatting"
  },
  {
    "startMs": 1663830,
    "text": "penalized harmless formatting"
  },
  {
    "startMs": 1663840,
    "text": "penalized harmless formatting differences. In other words, uh it made"
  },
  {
    "startMs": 1666630,
    "text": "differences. In other words, uh it made"
  },
  {
    "startMs": 1666640,
    "text": "differences. In other words, uh it made it inconsistent assumptions about what"
  },
  {
    "startMs": 1668789,
    "text": "it inconsistent assumptions about what"
  },
  {
    "startMs": 1668799,
    "text": "it inconsistent assumptions about what could be inferred from like a member's"
  },
  {
    "startMs": 1670549,
    "text": "could be inferred from like a member's"
  },
  {
    "startMs": 1670559,
    "text": "could be inferred from like a member's existing employment information too. We"
  },
  {
    "startMs": 1673350,
    "text": "existing employment information too. We"
  },
  {
    "startMs": 1673360,
    "text": "existing employment information too. We also examined like the distribution of"
  },
  {
    "startMs": 1675350,
    "text": "also examined like the distribution of"
  },
  {
    "startMs": 1675360,
    "text": "also examined like the distribution of overly harsh judgments. Out of like the"
  },
  {
    "startMs": 1677750,
    "text": "overly harsh judgments. Out of like the"
  },
  {
    "startMs": 1677760,
    "text": "overly harsh judgments. Out of like the 37 um judgments identified as too harsh,"
  },
  {
    "startMs": 1680870,
    "text": "37 um judgments identified as too harsh,"
  },
  {
    "startMs": 1680880,
    "text": "37 um judgments identified as too harsh, 31 of them involved outputs that had"
  },
  {
    "startMs": 1683510,
    "text": "31 of them involved outputs that had"
  },
  {
    "startMs": 1683520,
    "text": "31 of them involved outputs that had been assigned a score of one, which"
  },
  {
    "startMs": 1685269,
    "text": "been assigned a score of one, which"
  },
  {
    "startMs": 1685279,
    "text": "been assigned a score of one, which suggested that the boundaries between"
  },
  {
    "startMs": 1687029,
    "text": "suggested that the boundaries between"
  },
  {
    "startMs": 1687039,
    "text": "suggested that the boundaries between our scores needed to be like"
  },
  {
    "startMs": 1688549,
    "text": "our scores needed to be like"
  },
  {
    "startMs": 1688559,
    "text": "our scores needed to be like recalibrated."
  },
  {
    "startMs": 1690710,
    "text": "recalibrated."
  },
  {
    "startMs": 1690720,
    "text": "recalibrated. Um, another recurring source of"
  },
  {
    "startMs": 1693269,
    "text": "Um, another recurring source of"
  },
  {
    "startMs": 1693279,
    "text": "Um, another recurring source of disagreement was whether an inferred"
  },
  {
    "startMs": 1694950,
    "text": "disagreement was whether an inferred"
  },
  {
    "startMs": 1694960,
    "text": "disagreement was whether an inferred member's preference should be considered"
  },
  {
    "startMs": 1696789,
    "text": "member's preference should be considered"
  },
  {
    "startMs": 1696799,
    "text": "member's preference should be considered valid when the evidence wasn't like"
  },
  {
    "startMs": 1698710,
    "text": "valid when the evidence wasn't like"
  },
  {
    "startMs": 1698720,
    "text": "valid when the evidence wasn't like explicitly stated. And the broader"
  },
  {
    "startMs": 1701029,
    "text": "explicitly stated. And the broader"
  },
  {
    "startMs": 1701039,
    "text": "explicitly stated. And the broader lesson here was that many evaluation"
  },
  {
    "startMs": 1702549,
    "text": "lesson here was that many evaluation"
  },
  {
    "startMs": 1702559,
    "text": "lesson here was that many evaluation disagreements were not simply like model"
  },
  {
    "startMs": 1704389,
    "text": "disagreements were not simply like model"
  },
  {
    "startMs": 1704399,
    "text": "disagreements were not simply like model periods. They were reflecting"
  },
  {
    "startMs": 1706070,
    "text": "periods. They were reflecting"
  },
  {
    "startMs": 1706080,
    "text": "periods. They were reflecting ambiguities in the rubric as well as"
  },
  {
    "startMs": 1707750,
    "text": "ambiguities in the rubric as well as"
  },
  {
    "startMs": 1707760,
    "text": "ambiguities in the rubric as well as like the underlying evaluation policy."
  },
  {
    "startMs": 1711350,
    "text": "like the underlying evaluation policy."
  },
  {
    "startMs": 1711360,
    "text": "like the underlying evaluation policy. Um, based on these findings, we"
  },
  {
    "startMs": 1712789,
    "text": "Um, based on these findings, we"
  },
  {
    "startMs": 1712799,
    "text": "Um, based on these findings, we introduced several targeted improvements"
  },
  {
    "startMs": 1714470,
    "text": "introduced several targeted improvements"
  },
  {
    "startMs": 1714480,
    "text": "introduced several targeted improvements to our evaluation rubrics. One was like"
  },
  {
    "startMs": 1717909,
    "text": "to our evaluation rubrics. One was like"
  },
  {
    "startMs": 1717919,
    "text": "to our evaluation rubrics. One was like basically fixing harmless formatting"
  },
  {
    "startMs": 1719669,
    "text": "basically fixing harmless formatting"
  },
  {
    "startMs": 1719679,
    "text": "basically fixing harmless formatting differences. Um, previously like a value"
  },
  {
    "startMs": 1722710,
    "text": "differences. Um, previously like a value"
  },
  {
    "startMs": 1722720,
    "text": "differences. Um, previously like a value such as like you know full-time could"
  },
  {
    "startMs": 1724389,
    "text": "such as like you know full-time could"
  },
  {
    "startMs": 1724399,
    "text": "such as like you know full-time could receive an incorrect score because of"
  },
  {
    "startMs": 1726070,
    "text": "receive an incorrect score because of"
  },
  {
    "startMs": 1726080,
    "text": "receive an incorrect score because of like capitalization or incorrect"
  },
  {
    "startMs": 1727750,
    "text": "like capitalization or incorrect"
  },
  {
    "startMs": 1727760,
    "text": "like capitalization or incorrect formatting. We updated the rubrics so"
  },
  {
    "startMs": 1730230,
    "text": "formatting. We updated the rubrics so"
  },
  {
    "startMs": 1730240,
    "text": "formatting. We updated the rubrics so that these differences would not be"
  },
  {
    "startMs": 1731590,
    "text": "that these differences would not be"
  },
  {
    "startMs": 1731600,
    "text": "that these differences would not be treated as like semantic failures."
  },
  {
    "startMs": 1733510,
    "text": "treated as like semantic failures."
  },
  {
    "startMs": 1733520,
    "text": "treated as like semantic failures. Similarly, we introduced normalization"
  },
  {
    "startMs": 1735350,
    "text": "Similarly, we introduced normalization"
  },
  {
    "startMs": 1735360,
    "text": "Similarly, we introduced normalization guidance for like titles. Um we also"
  },
  {
    "startMs": 1738470,
    "text": "guidance for like titles. Um we also"
  },
  {
    "startMs": 1738480,
    "text": "guidance for like titles. Um we also clarified how the judge should handle"
  },
  {
    "startMs": 1740149,
    "text": "clarified how the judge should handle"
  },
  {
    "startMs": 1740159,
    "text": "clarified how the judge should handle concurrent employment like you know sort"
  },
  {
    "startMs": 1742789,
    "text": "concurrent employment like you know sort"
  },
  {
    "startMs": 1742799,
    "text": "concurrent employment like you know sort of impairing seniority as well as like"
  },
  {
    "startMs": 1745190,
    "text": "of impairing seniority as well as like"
  },
  {
    "startMs": 1745200,
    "text": "of impairing seniority as well as like impairing geographic references of the"
  },
  {
    "startMs": 1747190,
    "text": "impairing geographic references of the"
  },
  {
    "startMs": 1747200,
    "text": "impairing geographic references of the user and more broadly we recalibrated"
  },
  {
    "startMs": 1749669,
    "text": "user and more broadly we recalibrated"
  },
  {
    "startMs": 1749679,
    "text": "user and more broadly we recalibrated the rules uh separating low scores from"
  },
  {
    "startMs": 1751830,
    "text": "the rules uh separating low scores from"
  },
  {
    "startMs": 1751840,
    "text": "the rules uh separating low scores from like partially correct outputs and these"
  },
  {
    "startMs": 1754549,
    "text": "like partially correct outputs and these"
  },
  {
    "startMs": 1754559,
    "text": "like partially correct outputs and these changes resulted like resulted in like"
  },
  {
    "startMs": 1756230,
    "text": "changes resulted like resulted in like"
  },
  {
    "startMs": 1756240,
    "text": "changes resulted like resulted in like measurable improvements uh where human"
  },
  {
    "startMs": 1758230,
    "text": "measurable improvements uh where human"
  },
  {
    "startMs": 1758240,
    "text": "measurable improvements uh where human alignment increased from like 57% to"
  },
  {
    "startMs": 1760470,
    "text": "alignment increased from like 57% to"
  },
  {
    "startMs": 1760480,
    "text": "alignment increased from like 57% to like 64%"
  },
  {
    "startMs": 1762230,
    "text": "like 64%"
  },
  {
    "startMs": 1762240,
    "text": "like 64% and um at the same time the mean error"
  },
  {
    "startMs": 1764310,
    "text": "and um at the same time the mean error"
  },
  {
    "startMs": 1764320,
    "text": "and um at the same time the mean error decreased from like 96 to 74 What we're"
  },
  {
    "startMs": 1767350,
    "text": "decreased from like 96 to 74 What we're"
  },
  {
    "startMs": 1767360,
    "text": "decreased from like 96 to 74 What we're trying to achieve here is to get our"
  },
  {
    "startMs": 1768950,
    "text": "trying to achieve here is to get our"
  },
  {
    "startMs": 1768960,
    "text": "trying to achieve here is to get our system as high or like as highly"
  },
  {
    "startMs": 1771510,
    "text": "system as high or like as highly"
  },
  {
    "startMs": 1771520,
    "text": "system as high or like as highly aligning to humans as much as possible."
  },
  {
    "startMs": 1774149,
    "text": "aligning to humans as much as possible."
  },
  {
    "startMs": 1774159,
    "text": "aligning to humans as much as possible. And what we find particularly"
  },
  {
    "startMs": 1775430,
    "text": "And what we find particularly"
  },
  {
    "startMs": 1775440,
    "text": "And what we find particularly interesting here is that we didn't"
  },
  {
    "startMs": 1776870,
    "text": "interesting here is that we didn't"
  },
  {
    "startMs": 1776880,
    "text": "interesting here is that we didn't achieve these improvements by only like"
  },
  {
    "startMs": 1778470,
    "text": "achieve these improvements by only like"
  },
  {
    "startMs": 1778480,
    "text": "achieve these improvements by only like handling the model, but we also improved"
  },
  {
    "startMs": 1780549,
    "text": "handling the model, but we also improved"
  },
  {
    "startMs": 1780559,
    "text": "handling the model, but we also improved specifications of what like the correct"
  },
  {
    "startMs": 1782070,
    "text": "specifications of what like the correct"
  },
  {
    "startMs": 1782080,
    "text": "specifications of what like the correct evaluation should look like. That is"
  },
  {
    "startMs": 1783750,
    "text": "evaluation should look like. That is"
  },
  {
    "startMs": 1783760,
    "text": "evaluation should look like. That is like rubric design as well as like um"
  },
  {
    "startMs": 1787269,
    "text": "like rubric design as well as like um"
  },
  {
    "startMs": 1787279,
    "text": "like rubric design as well as like um evidences which our evaluator is like"
  },
  {
    "startMs": 1789350,
    "text": "evidences which our evaluator is like"
  },
  {
    "startMs": 1789360,
    "text": "evidences which our evaluator is like spitting out. Um"
  },
  {
    "startMs": 1792870,
    "text": "spitting out. Um"
  },
  {
    "startMs": 1792880,
    "text": "spitting out. Um moving forward uh we perform like a"
  },
  {
    "startMs": 1794789,
    "text": "moving forward uh we perform like a"
  },
  {
    "startMs": 1794799,
    "text": "moving forward uh we perform like a similar human review analysis for our"
  },
  {
    "startMs": 1796789,
    "text": "similar human review analysis for our"
  },
  {
    "startMs": 1796799,
    "text": "similar human review analysis for our reflection agent which was described"
  },
  {
    "startMs": 1798389,
    "text": "reflection agent which was described"
  },
  {
    "startMs": 1798399,
    "text": "reflection agent which was described earlier in our pipeline. We should"
  },
  {
    "startMs": 1800549,
    "text": "earlier in our pipeline. We should"
  },
  {
    "startMs": 1800559,
    "text": "earlier in our pipeline. We should recall that our L2 takes an evaluation"
  },
  {
    "startMs": 1802950,
    "text": "recall that our L2 takes an evaluation"
  },
  {
    "startMs": 1802960,
    "text": "recall that our L2 takes an evaluation failure and then it tries to identify"
  },
  {
    "startMs": 1804789,
    "text": "failure and then it tries to identify"
  },
  {
    "startMs": 1804799,
    "text": "failure and then it tries to identify the root cause. For example, it"
  },
  {
    "startMs": 1807190,
    "text": "the root cause. For example, it"
  },
  {
    "startMs": 1807200,
    "text": "the root cause. For example, it basically identifies was the problem"
  },
  {
    "startMs": 1808710,
    "text": "basically identifies was the problem"
  },
  {
    "startMs": 1808720,
    "text": "basically identifies was the problem caused by the agent itself by the"
  },
  {
    "startMs": 1810470,
    "text": "caused by the agent itself by the"
  },
  {
    "startMs": 1810480,
    "text": "caused by the agent itself by the evaluation rubric or by some other issue"
  },
  {
    "startMs": 1812870,
    "text": "evaluation rubric or by some other issue"
  },
  {
    "startMs": 1812880,
    "text": "evaluation rubric or by some other issue with the available context. What we"
  },
  {
    "startMs": 1815350,
    "text": "with the available context. What we"
  },
  {
    "startMs": 1815360,
    "text": "with the available context. What we discovered was that root cause"
  },
  {
    "startMs": 1816950,
    "text": "discovered was that root cause"
  },
  {
    "startMs": 1816960,
    "text": "discovered was that root cause attribution was substantially harder"
  },
  {
    "startMs": 1818789,
    "text": "attribution was substantially harder"
  },
  {
    "startMs": 1818799,
    "text": "attribution was substantially harder than like simply identifying a failure."
  },
  {
    "startMs": 1821350,
    "text": "than like simply identifying a failure."
  },
  {
    "startMs": 1821360,
    "text": "than like simply identifying a failure. Human reviewers agreed more often when"
  },
  {
    "startMs": 1823990,
    "text": "Human reviewers agreed more often when"
  },
  {
    "startMs": 1824000,
    "text": "Human reviewers agreed more often when L2 attributed a problem to like the"
  },
  {
    "startMs": 1827110,
    "text": "L2 attributed a problem to like the"
  },
  {
    "startMs": 1827120,
    "text": "L2 attributed a problem to like the underlying model, but agreement was much"
  },
  {
    "startMs": 1829990,
    "text": "underlying model, but agreement was much"
  },
  {
    "startMs": 1830000,
    "text": "underlying model, but agreement was much lower for other causes, particularly"
  },
  {
    "startMs": 1831830,
    "text": "lower for other causes, particularly"
  },
  {
    "startMs": 1831840,
    "text": "lower for other causes, particularly like rubric related issues. Um, one"
  },
  {
    "startMs": 1834950,
    "text": "like rubric related issues. Um, one"
  },
  {
    "startMs": 1834960,
    "text": "like rubric related issues. Um, one important issue was that our original"
  },
  {
    "startMs": 1837029,
    "text": "important issue was that our original"
  },
  {
    "startMs": 1837039,
    "text": "important issue was that our original reflection taxonomy didn't include a way"
  },
  {
    "startMs": 1839029,
    "text": "reflection taxonomy didn't include a way"
  },
  {
    "startMs": 1839039,
    "text": "reflection taxonomy didn't include a way to explicitly say that or Elvin judge"
  },
  {
    "startMs": 1841350,
    "text": "to explicitly say that or Elvin judge"
  },
  {
    "startMs": 1841360,
    "text": "to explicitly say that or Elvin judge could have been wrong. Um, as a result,"
  },
  {
    "startMs": 1843750,
    "text": "could have been wrong. Um, as a result,"
  },
  {
    "startMs": 1843760,
    "text": "could have been wrong. Um, as a result, sometimes the reflection agent fell back"
  },
  {
    "startMs": 1846149,
    "text": "sometimes the reflection agent fell back"
  },
  {
    "startMs": 1846159,
    "text": "sometimes the reflection agent fell back to blaming the rubric even when the real"
  },
  {
    "startMs": 1848070,
    "text": "to blaming the rubric even when the real"
  },
  {
    "startMs": 1848080,
    "text": "to blaming the rubric even when the real problem was like an incorrect um, L1"
  },
  {
    "startMs": 1850230,
    "text": "problem was like an incorrect um, L1"
  },
  {
    "startMs": 1850240,
    "text": "problem was like an incorrect um, L1 judgment. Um, and like another issue was"
  },
  {
    "startMs": 1853909,
    "text": "judgment. Um, and like another issue was"
  },
  {
    "startMs": 1853919,
    "text": "judgment. Um, and like another issue was like the uh, reflection agent sometimes"
  },
  {
    "startMs": 1855990,
    "text": "like the uh, reflection agent sometimes"
  },
  {
    "startMs": 1856000,
    "text": "like the uh, reflection agent sometimes used the L1 judge's rationale without"
  },
  {
    "startMs": 1858149,
    "text": "used the L1 judge's rationale without"
  },
  {
    "startMs": 1858159,
    "text": "used the L1 judge's rationale without like independently validating or"
  },
  {
    "startMs": 1859830,
    "text": "like independently validating or"
  },
  {
    "startMs": 1859840,
    "text": "like independently validating or independently thinking over it against"
  },
  {
    "startMs": 1861909,
    "text": "independently thinking over it against"
  },
  {
    "startMs": 1861919,
    "text": "independently thinking over it against the agents output. And um we exposed an"
  },
  {
    "startMs": 1866070,
    "text": "the agents output. And um we exposed an"
  },
  {
    "startMs": 1866080,
    "text": "the agents output. And um we exposed an important risk here which was like a"
  },
  {
    "startMs": 1867430,
    "text": "important risk here which was like a"
  },
  {
    "startMs": 1867440,
    "text": "important risk here which was like a reflection system can produce convincing"
  },
  {
    "startMs": 1869590,
    "text": "reflection system can produce convincing"
  },
  {
    "startMs": 1869600,
    "text": "reflection system can produce convincing explanations while still like"
  },
  {
    "startMs": 1871350,
    "text": "explanations while still like"
  },
  {
    "startMs": 1871360,
    "text": "explanations while still like identifying misidentifying like the"
  },
  {
    "startMs": 1873669,
    "text": "identifying misidentifying like the"
  },
  {
    "startMs": 1873679,
    "text": "identifying misidentifying like the underlying problem and if we use those"
  },
  {
    "startMs": 1876230,
    "text": "underlying problem and if we use those"
  },
  {
    "startMs": 1876240,
    "text": "underlying problem and if we use those explanations to generate prompt changes"
  },
  {
    "startMs": 1877990,
    "text": "explanations to generate prompt changes"
  },
  {
    "startMs": 1878000,
    "text": "explanations to generate prompt changes we could like end up modifying like the"
  },
  {
    "startMs": 1879909,
    "text": "we could like end up modifying like the"
  },
  {
    "startMs": 1879919,
    "text": "we could like end up modifying like the wrong component of the system. So"
  },
  {
    "startMs": 1881669,
    "text": "wrong component of the system. So"
  },
  {
    "startMs": 1881679,
    "text": "wrong component of the system. So improving the L2 uh like reflection"
  },
  {
    "startMs": 1884549,
    "text": "improving the L2 uh like reflection"
  },
  {
    "startMs": 1884559,
    "text": "improving the L2 uh like reflection chart like became our next priority."
  },
  {
    "startMs": 1888470,
    "text": "chart like became our next priority."
  },
  {
    "startMs": 1888480,
    "text": "chart like became our next priority. Um so to talk about some of the quality"
  },
  {
    "startMs": 1890870,
    "text": "Um so to talk about some of the quality"
  },
  {
    "startMs": 1890880,
    "text": "Um so to talk about some of the quality improvements um these findings led us to"
  },
  {
    "startMs": 1893750,
    "text": "improvements um these findings led us to"
  },
  {
    "startMs": 1893760,
    "text": "improvements um these findings led us to like focus on improving both layers of"
  },
  {
    "startMs": 1895909,
    "text": "like focus on improving both layers of"
  },
  {
    "startMs": 1895919,
    "text": "like focus on improving both layers of evaluation. At L1 the goal was to make"
  },
  {
    "startMs": 1898389,
    "text": "evaluation. At L1 the goal was to make"
  },
  {
    "startMs": 1898399,
    "text": "evaluation. At L1 the goal was to make sure that the scoring was more accurate"
  },
  {
    "startMs": 1900870,
    "text": "sure that the scoring was more accurate"
  },
  {
    "startMs": 1900880,
    "text": "sure that the scoring was more accurate and more better aligned with human"
  },
  {
    "startMs": 1902470,
    "text": "and more better aligned with human"
  },
  {
    "startMs": 1902480,
    "text": "and more better aligned with human judgment. And at L2 the goal was to"
  },
  {
    "startMs": 1904630,
    "text": "judgment. And at L2 the goal was to"
  },
  {
    "startMs": 1904640,
    "text": "judgment. And at L2 the goal was to improve the reliability of the root"
  },
  {
    "startMs": 1906389,
    "text": "improve the reliability of the root"
  },
  {
    "startMs": 1906399,
    "text": "improve the reliability of the root cause diagnosis."
  },
  {
    "startMs": 1910480,
    "text": "And um the next this slide basically"
  },
  {
    "startMs": 1913190,
    "text": "And um the next this slide basically"
  },
  {
    "startMs": 1913200,
    "text": "And um the next this slide basically summarizes some of like the changes we"
  },
  {
    "startMs": 1914870,
    "text": "summarizes some of like the changes we"
  },
  {
    "startMs": 1914880,
    "text": "summarizes some of like the changes we made to the system and the impact that"
  },
  {
    "startMs": 1916310,
    "text": "made to the system and the impact that"
  },
  {
    "startMs": 1916320,
    "text": "made to the system and the impact that we observed. We introduced like several"
  },
  {
    "startMs": 1918789,
    "text": "we observed. We introduced like several"
  },
  {
    "startMs": 1918799,
    "text": "we observed. We introduced like several targeted changes to address the um the"
  },
  {
    "startMs": 1921669,
    "text": "targeted changes to address the um the"
  },
  {
    "startMs": 1921679,
    "text": "targeted changes to address the um the reflection failures"
  },
  {
    "startMs": 1923669,
    "text": "reflection failures"
  },
  {
    "startMs": 1923679,
    "text": "reflection failures um identified through like human review"
  },
  {
    "startMs": 1926789,
    "text": "um identified through like human review"
  },
  {
    "startMs": 1926799,
    "text": "um identified through like human review and um"
  },
  {
    "startMs": 1929029,
    "text": "and um"
  },
  {
    "startMs": 1929039,
    "text": "and um we we were basically like trying to"
  },
  {
    "startMs": 1931110,
    "text": "we we were basically like trying to"
  },
  {
    "startMs": 1931120,
    "text": "we we were basically like trying to measure the impact here in terms of like"
  },
  {
    "startMs": 1932789,
    "text": "measure the impact here in terms of like"
  },
  {
    "startMs": 1932799,
    "text": "measure the impact here in terms of like how do we translate all this evaluation"
  },
  {
    "startMs": 1935350,
    "text": "how do we translate all this evaluation"
  },
  {
    "startMs": 1935360,
    "text": "how do we translate all this evaluation and reflection work into like measurable"
  },
  {
    "startMs": 1937750,
    "text": "and reflection work into like measurable"
  },
  {
    "startMs": 1937760,
    "text": "and reflection work into like measurable impact. So there are two levels to"
  },
  {
    "startMs": 1939750,
    "text": "impact. So there are two levels to"
  },
  {
    "startMs": 1939760,
    "text": "impact. So there are two levels to consider here. The first is like"
  },
  {
    "startMs": 1941350,
    "text": "consider here. The first is like"
  },
  {
    "startMs": 1941360,
    "text": "consider here. The first is like improving the reliability of the RSI"
  },
  {
    "startMs": 1943350,
    "text": "improving the reliability of the RSI"
  },
  {
    "startMs": 1943360,
    "text": "improving the reliability of the RSI process itself. So by improving L1 we"
  },
  {
    "startMs": 1945669,
    "text": "process itself. So by improving L1 we"
  },
  {
    "startMs": 1945679,
    "text": "process itself. So by improving L1 we make the um evaluation scores more"
  },
  {
    "startMs": 1948470,
    "text": "make the um evaluation scores more"
  },
  {
    "startMs": 1948480,
    "text": "make the um evaluation scores more stable and like more better aligned with"
  },
  {
    "startMs": 1950389,
    "text": "stable and like more better aligned with"
  },
  {
    "startMs": 1950399,
    "text": "stable and like more better aligned with human expectations and by improving L2"
  },
  {
    "startMs": 1953029,
    "text": "human expectations and by improving L2"
  },
  {
    "startMs": 1953039,
    "text": "human expectations and by improving L2 we increase like the uh like the root"
  },
  {
    "startMs": 1955509,
    "text": "we increase like the uh like the root"
  },
  {
    "startMs": 1955519,
    "text": "we increase like the uh like the root cause diagnosis as we mentioned earlier."
  },
  {
    "startMs": 1958310,
    "text": "cause diagnosis as we mentioned earlier."
  },
  {
    "startMs": 1958320,
    "text": "cause diagnosis as we mentioned earlier. So we also use like majority voting sort"
  },
  {
    "startMs": 1960630,
    "text": "So we also use like majority voting sort"
  },
  {
    "startMs": 1960640,
    "text": "So we also use like majority voting sort of and like held out evaluations to like"
  },
  {
    "startMs": 1962389,
    "text": "of and like held out evaluations to like"
  },
  {
    "startMs": 1962399,
    "text": "of and like held out evaluations to like help mitigate um variance from like"
  },
  {
    "startMs": 1966070,
    "text": "help mitigate um variance from like"
  },
  {
    "startMs": 1966080,
    "text": "help mitigate um variance from like individual uh LLM judgments and together"
  },
  {
    "startMs": 1969669,
    "text": "individual uh LLM judgments and together"
  },
  {
    "startMs": 1969679,
    "text": "individual uh LLM judgments and together these improvements support like a more"
  },
  {
    "startMs": 1971350,
    "text": "these improvements support like a more"
  },
  {
    "startMs": 1971360,
    "text": "these improvements support like a more reliable optimization process."
  },
  {
    "startMs": 1974470,
    "text": "reliable optimization process."
  },
  {
    "startMs": 1974480,
    "text": "reliable optimization process. Um importantly like one of the observed"
  },
  {
    "startMs": 1977029,
    "text": "Um importantly like one of the observed"
  },
  {
    "startMs": 1977039,
    "text": "Um importantly like one of the observed improvement here was about the required"
  },
  {
    "startMs": 1978950,
    "text": "improvement here was about the required"
  },
  {
    "startMs": 1978960,
    "text": "improvement here was about the required adoption threshold and the reported"
  },
  {
    "startMs": 1980549,
    "text": "adoption threshold and the reported"
  },
  {
    "startMs": 1980559,
    "text": "adoption threshold and the reported confidence interval as well. Uh the"
  },
  {
    "startMs": 1983029,
    "text": "confidence interval as well. Uh the"
  },
  {
    "startMs": 1983039,
    "text": "confidence interval as well. Uh the second level of impact is where we want"
  },
  {
    "startMs": 1985029,
    "text": "second level of impact is where we want"
  },
  {
    "startMs": 1985039,
    "text": "second level of impact is where we want to measure the RSI where the RSI"
  },
  {
    "startMs": 1987430,
    "text": "to measure the RSI where the RSI"
  },
  {
    "startMs": 1987440,
    "text": "to measure the RSI where the RSI adoption grows. This includes the"
  },
  {
    "startMs": 1989509,
    "text": "adoption grows. This includes the"
  },
  {
    "startMs": 1989519,
    "text": "adoption grows. This includes the percentage of like agent prompt changes"
  },
  {
    "startMs": 1991269,
    "text": "percentage of like agent prompt changes"
  },
  {
    "startMs": 1991279,
    "text": "percentage of like agent prompt changes that can reliably be generated and"
  },
  {
    "startMs": 1992950,
    "text": "that can reliably be generated and"
  },
  {
    "startMs": 1992960,
    "text": "that can reliably be generated and validated through recursive"
  },
  {
    "startMs": 1994149,
    "text": "validated through recursive"
  },
  {
    "startMs": 1994159,
    "text": "validated through recursive self-improvements in like downstream"
  },
  {
    "startMs": 1996549,
    "text": "self-improvements in like downstream"
  },
  {
    "startMs": 1996559,
    "text": "self-improvements in like downstream response quality and also productivity"
  },
  {
    "startMs": 1998389,
    "text": "response quality and also productivity"
  },
  {
    "startMs": 1998399,
    "text": "response quality and also productivity gains for engineering for engineers like"
  },
  {
    "startMs": 2000549,
    "text": "gains for engineering for engineers like"
  },
  {
    "startMs": 2000559,
    "text": "gains for engineering for engineers like maintaining these systems. Um those"
  },
  {
    "startMs": 2002950,
    "text": "maintaining these systems. Um those"
  },
  {
    "startMs": 2002960,
    "text": "maintaining these systems. Um those broader metrics are a part of how we"
  },
  {
    "startMs": 2004470,
    "text": "broader metrics are a part of how we"
  },
  {
    "startMs": 2004480,
    "text": "broader metrics are a part of how we want to like basically think about"
  },
  {
    "startMs": 2006389,
    "text": "want to like basically think about"
  },
  {
    "startMs": 2006399,
    "text": "want to like basically think about evaluation evaluating these systems at"
  },
  {
    "startMs": 2008549,
    "text": "evaluation evaluating these systems at"
  },
  {
    "startMs": 2008559,
    "text": "evaluation evaluating these systems at scale because the ultimate goal isn't to"
  },
  {
    "startMs": 2010630,
    "text": "scale because the ultimate goal isn't to"
  },
  {
    "startMs": 2010640,
    "text": "scale because the ultimate goal isn't to like maximize the number of like"
  },
  {
    "startMs": 2012630,
    "text": "like maximize the number of like"
  },
  {
    "startMs": 2012640,
    "text": "like maximize the number of like automated prompt edits. It's to like"
  },
  {
    "startMs": 2014470,
    "text": "automated prompt edits. It's to like"
  },
  {
    "startMs": 2014480,
    "text": "automated prompt edits. It's to like produce measurable improvements in like"
  },
  {
    "startMs": 2016310,
    "text": "produce measurable improvements in like"
  },
  {
    "startMs": 2016320,
    "text": "produce measurable improvements in like agent quality while maintaining some"
  },
  {
    "startMs": 2018630,
    "text": "agent quality while maintaining some"
  },
  {
    "startMs": 2018640,
    "text": "agent quality while maintaining some sort of a confidence that those changes"
  },
  {
    "startMs": 2020310,
    "text": "sort of a confidence that those changes"
  },
  {
    "startMs": 2020320,
    "text": "sort of a confidence that those changes are like genuinely beneficial. And this"
  },
  {
    "startMs": 2023029,
    "text": "are like genuinely beneficial. And this"
  },
  {
    "startMs": 2023039,
    "text": "are like genuinely beneficial. And this brings up brings us to like the final"
  },
  {
    "startMs": 2025029,
    "text": "brings up brings us to like the final"
  },
  {
    "startMs": 2025039,
    "text": "brings up brings us to like the final part of our talk which is how do we"
  },
  {
    "startMs": 2027110,
    "text": "part of our talk which is how do we"
  },
  {
    "startMs": 2027120,
    "text": "part of our talk which is how do we scale these um capabilities"
  },
  {
    "startMs": 2030710,
    "text": "scale these um capabilities"
  },
  {
    "startMs": 2030720,
    "text": "scale these um capabilities um which is like through an acceleration"
  },
  {
    "startMs": 2032630,
    "text": "um which is like through an acceleration"
  },
  {
    "startMs": 2032640,
    "text": "um which is like through an acceleration plan. So so far we focused on like one"
  },
  {
    "startMs": 2035430,
    "text": "plan. So so far we focused on like one"
  },
  {
    "startMs": 2035440,
    "text": "plan. So so far we focused on like one implementation of RSI for user"
  },
  {
    "startMs": 2037110,
    "text": "implementation of RSI for user"
  },
  {
    "startMs": 2037120,
    "text": "implementation of RSI for user understanding but like the broader"
  },
  {
    "startMs": 2038710,
    "text": "understanding but like the broader"
  },
  {
    "startMs": 2038720,
    "text": "understanding but like the broader opportunity is like much larger. If we"
  },
  {
    "startMs": 2041269,
    "text": "opportunity is like much larger. If we"
  },
  {
    "startMs": 2041279,
    "text": "opportunity is like much larger. If we are able to like um kind of like"
  },
  {
    "startMs": 2043269,
    "text": "are able to like um kind of like"
  },
  {
    "startMs": 2043279,
    "text": "are able to like um kind of like standardize evaluation, reflection and"
  },
  {
    "startMs": 2045909,
    "text": "standardize evaluation, reflection and"
  },
  {
    "startMs": 2045919,
    "text": "standardize evaluation, reflection and optimization, we could potentially uh"
  },
  {
    "startMs": 2048470,
    "text": "optimization, we could potentially uh"
  },
  {
    "startMs": 2048480,
    "text": "optimization, we could potentially uh reuse this loop for like or the cycle"
  },
  {
    "startMs": 2050389,
    "text": "reuse this loop for like or the cycle"
  },
  {
    "startMs": 2050399,
    "text": "reuse this loop for like or the cycle for like many other agent teams. And um"
  },
  {
    "startMs": 2054710,
    "text": "for like many other agent teams. And um"
  },
  {
    "startMs": 2054720,
    "text": "for like many other agent teams. And um moving forward like um to understand the"
  },
  {
    "startMs": 2057510,
    "text": "moving forward like um to understand the"
  },
  {
    "startMs": 2057520,
    "text": "moving forward like um to understand the end toend evals the current state is"
  },
  {
    "startMs": 2059270,
    "text": "end toend evals the current state is"
  },
  {
    "startMs": 2059280,
    "text": "end toend evals the current state is like this diagram basically shows"
  },
  {
    "startMs": 2060550,
    "text": "like this diagram basically shows"
  },
  {
    "startMs": 2060560,
    "text": "like this diagram basically shows progression of capabilities that we"
  },
  {
    "startMs": 2062069,
    "text": "progression of capabilities that we"
  },
  {
    "startMs": 2062079,
    "text": "progression of capabilities that we believe are like necessary to support"
  },
  {
    "startMs": 2063750,
    "text": "believe are like necessary to support"
  },
  {
    "startMs": 2063760,
    "text": "believe are like necessary to support RSI at scale. At the foundation, we"
  },
  {
    "startMs": 2066230,
    "text": "RSI at scale. At the foundation, we"
  },
  {
    "startMs": 2066240,
    "text": "RSI at scale. At the foundation, we obviously have like the base agent where"
  },
  {
    "startMs": 2068069,
    "text": "obviously have like the base agent where"
  },
  {
    "startMs": 2068079,
    "text": "obviously have like the base agent where most teams already have agents that"
  },
  {
    "startMs": 2069510,
    "text": "most teams already have agents that"
  },
  {
    "startMs": 2069520,
    "text": "most teams already have agents that perform like useful tasks whether that's"
  },
  {
    "startMs": 2071270,
    "text": "perform like useful tasks whether that's"
  },
  {
    "startMs": 2071280,
    "text": "perform like useful tasks whether that's like question answering, summarizing,"
  },
  {
    "startMs": 2073030,
    "text": "like question answering, summarizing,"
  },
  {
    "startMs": 2073040,
    "text": "like question answering, summarizing, chat agents um or like just completing"
  },
  {
    "startMs": 2075990,
    "text": "chat agents um or like just completing"
  },
  {
    "startMs": 2076000,
    "text": "chat agents um or like just completing workflows as such. The next layer is"
  },
  {
    "startMs": 2077750,
    "text": "workflows as such. The next layer is"
  },
  {
    "startMs": 2077760,
    "text": "workflows as such. The next layer is agent evaluation and this is where we"
  },
  {
    "startMs": 2079669,
    "text": "agent evaluation and this is where we"
  },
  {
    "startMs": 2079679,
    "text": "agent evaluation and this is where we begin to like see gaps because some"
  },
  {
    "startMs": 2081750,
    "text": "begin to like see gaps because some"
  },
  {
    "startMs": 2081760,
    "text": "begin to like see gaps because some teams have built their own bespoke"
  },
  {
    "startMs": 2083750,
    "text": "teams have built their own bespoke"
  },
  {
    "startMs": 2083760,
    "text": "teams have built their own bespoke evaluation workflows. Other rely on like"
  },
  {
    "startMs": 2085909,
    "text": "evaluation workflows. Other rely on like"
  },
  {
    "startMs": 2085919,
    "text": "evaluation workflows. Other rely on like one-off analysis or like manually"
  },
  {
    "startMs": 2087510,
    "text": "one-off analysis or like manually"
  },
  {
    "startMs": 2087520,
    "text": "one-off analysis or like manually inspecting outputs. But only a few of"
  },
  {
    "startMs": 2090149,
    "text": "inspecting outputs. But only a few of"
  },
  {
    "startMs": 2090159,
    "text": "inspecting outputs. But only a few of them have like relatively standardized"
  },
  {
    "startMs": 2092149,
    "text": "them have like relatively standardized"
  },
  {
    "startMs": 2092159,
    "text": "them have like relatively standardized reliable evaluation processes that can"
  },
  {
    "startMs": 2093990,
    "text": "reliable evaluation processes that can"
  },
  {
    "startMs": 2094000,
    "text": "reliable evaluation processes that can be continuously reused. And above um"
  },
  {
    "startMs": 2098150,
    "text": "be continuously reused. And above um"
  },
  {
    "startMs": 2098160,
    "text": "be continuously reused. And above um evaluation we have agentic reflection"
  },
  {
    "startMs": 2100069,
    "text": "evaluation we have agentic reflection"
  },
  {
    "startMs": 2100079,
    "text": "evaluation we have agentic reflection which means being able to systematically"
  },
  {
    "startMs": 2101829,
    "text": "which means being able to systematically"
  },
  {
    "startMs": 2101839,
    "text": "which means being able to systematically investigate or do some sort of like root"
  },
  {
    "startMs": 2104150,
    "text": "investigate or do some sort of like root"
  },
  {
    "startMs": 2104160,
    "text": "investigate or do some sort of like root cause diagnosis. Um and finally at the"
  },
  {
    "startMs": 2106950,
    "text": "cause diagnosis. Um and finally at the"
  },
  {
    "startMs": 2106960,
    "text": "cause diagnosis. Um and finally at the top we have the RSI loop itself which"
  },
  {
    "startMs": 2108950,
    "text": "top we have the RSI loop itself which"
  },
  {
    "startMs": 2108960,
    "text": "top we have the RSI loop itself which connects evaluation and reflection to"
  },
  {
    "startMs": 2110710,
    "text": "connects evaluation and reflection to"
  },
  {
    "startMs": 2110720,
    "text": "connects evaluation and reflection to like optimization and validation. The"
  },
  {
    "startMs": 2113829,
    "text": "like optimization and validation. The"
  },
  {
    "startMs": 2113839,
    "text": "like optimization and validation. The the important observation here is that"
  },
  {
    "startMs": 2115349,
    "text": "the important observation here is that"
  },
  {
    "startMs": 2115359,
    "text": "the important observation here is that these layers like they depend on one"
  },
  {
    "startMs": 2117349,
    "text": "these layers like they depend on one"
  },
  {
    "startMs": 2117359,
    "text": "these layers like they depend on one another and you can't build like a"
  },
  {
    "startMs": 2119270,
    "text": "another and you can't build like a"
  },
  {
    "startMs": 2119280,
    "text": "another and you can't build like a trustworthy loop without like reliable"
  },
  {
    "startMs": 2121829,
    "text": "trustworthy loop without like reliable"
  },
  {
    "startMs": 2121839,
    "text": "trustworthy loop without like reliable diagnosis. That's why we see like"
  },
  {
    "startMs": 2124150,
    "text": "diagnosis. That's why we see like"
  },
  {
    "startMs": 2124160,
    "text": "diagnosis. That's why we see like leveling up evaluation infrastructure as"
  },
  {
    "startMs": 2126310,
    "text": "leveling up evaluation infrastructure as"
  },
  {
    "startMs": 2126320,
    "text": "leveling up evaluation infrastructure as a prerequisite to like scaling RSI and"
  },
  {
    "startMs": 2129430,
    "text": "a prerequisite to like scaling RSI and"
  },
  {
    "startMs": 2129440,
    "text": "a prerequisite to like scaling RSI and rather than asking every team to like"
  },
  {
    "startMs": 2131829,
    "text": "rather than asking every team to like"
  },
  {
    "startMs": 2131839,
    "text": "rather than asking every team to like independently build all these"
  },
  {
    "startMs": 2133030,
    "text": "independently build all these"
  },
  {
    "startMs": 2133040,
    "text": "independently build all these components, there's an opportunity over"
  },
  {
    "startMs": 2134470,
    "text": "components, there's an opportunity over"
  },
  {
    "startMs": 2134480,
    "text": "components, there's an opportunity over here to like provide some sort of like a"
  },
  {
    "startMs": 2136069,
    "text": "here to like provide some sort of like a"
  },
  {
    "startMs": 2136079,
    "text": "here to like provide some sort of like a shared um infrastructure."
  },
  {
    "startMs": 2139270,
    "text": "shared um infrastructure."
  },
  {
    "startMs": 2139280,
    "text": "shared um infrastructure. So uh what would it like basically take"
  },
  {
    "startMs": 2141589,
    "text": "So uh what would it like basically take"
  },
  {
    "startMs": 2141599,
    "text": "So uh what would it like basically take to make RSI like a reusable capability"
  },
  {
    "startMs": 2143670,
    "text": "to make RSI like a reusable capability"
  },
  {
    "startMs": 2143680,
    "text": "to make RSI like a reusable capability across an enterprise is like uh we've"
  },
  {
    "startMs": 2146390,
    "text": "across an enterprise is like uh we've"
  },
  {
    "startMs": 2146400,
    "text": "across an enterprise is like uh we've identified like five strong areas. First"
  },
  {
    "startMs": 2148790,
    "text": "identified like five strong areas. First"
  },
  {
    "startMs": 2148800,
    "text": "identified like five strong areas. First is like standardizing evaluation"
  },
  {
    "startMs": 2150310,
    "text": "is like standardizing evaluation"
  },
  {
    "startMs": 2150320,
    "text": "is like standardizing evaluation infrastructure. So we need reusable"
  },
  {
    "startMs": 2152150,
    "text": "infrastructure. So we need reusable"
  },
  {
    "startMs": 2152160,
    "text": "infrastructure. So we need reusable evaluation frameworks, metrics, judges"
  },
  {
    "startMs": 2154550,
    "text": "evaluation frameworks, metrics, judges"
  },
  {
    "startMs": 2154560,
    "text": "evaluation frameworks, metrics, judges which can be calibrated with human"
  },
  {
    "startMs": 2156069,
    "text": "which can be calibrated with human"
  },
  {
    "startMs": 2156079,
    "text": "which can be calibrated with human feedback and um the second is like"
  },
  {
    "startMs": 2158790,
    "text": "feedback and um the second is like"
  },
  {
    "startMs": 2158800,
    "text": "feedback and um the second is like automating feedback loops. So we want"
  },
  {
    "startMs": 2160550,
    "text": "automating feedback loops. So we want"
  },
  {
    "startMs": 2160560,
    "text": "automating feedback loops. So we want systems that can continuously collect"
  },
  {
    "startMs": 2162310,
    "text": "systems that can continuously collect"
  },
  {
    "startMs": 2162320,
    "text": "systems that can continuously collect execution traces, detect recurring"
  },
  {
    "startMs": 2164790,
    "text": "execution traces, detect recurring"
  },
  {
    "startMs": 2164800,
    "text": "execution traces, detect recurring failures and perform some sort of loop"
  },
  {
    "startMs": 2166470,
    "text": "failures and perform some sort of loop"
  },
  {
    "startMs": 2166480,
    "text": "failures and perform some sort of loop case analysis."
  },
  {
    "startMs": 2168150,
    "text": "case analysis."
  },
  {
    "startMs": 2168160,
    "text": "case analysis. Um the third one is we need to scale um"
  },
  {
    "startMs": 2170790,
    "text": "Um the third one is we need to scale um"
  },
  {
    "startMs": 2170800,
    "text": "Um the third one is we need to scale um automated optimization that includes"
  },
  {
    "startMs": 2172870,
    "text": "automated optimization that includes"
  },
  {
    "startMs": 2172880,
    "text": "automated optimization that includes prompt optimization, agent reflection."
  },
  {
    "startMs": 2175589,
    "text": "prompt optimization, agent reflection."
  },
  {
    "startMs": 2175599,
    "text": "prompt optimization, agent reflection. So instead of making like ad hoc"
  },
  {
    "startMs": 2177109,
    "text": "So instead of making like ad hoc"
  },
  {
    "startMs": 2177119,
    "text": "So instead of making like ad hoc changes, we want improvement processes"
  },
  {
    "startMs": 2179430,
    "text": "changes, we want improvement processes"
  },
  {
    "startMs": 2179440,
    "text": "changes, we want improvement processes where candidates where candidate updates"
  },
  {
    "startMs": 2181589,
    "text": "where candidates where candidate updates"
  },
  {
    "startMs": 2181599,
    "text": "where candidates where candidate updates can be like generated and tested in more"
  },
  {
    "startMs": 2183190,
    "text": "can be like generated and tested in more"
  },
  {
    "startMs": 2183200,
    "text": "can be like generated and tested in more like a repeatable way. Um the fourth"
  },
  {
    "startMs": 2185750,
    "text": "like a repeatable way. Um the fourth"
  },
  {
    "startMs": 2185760,
    "text": "like a repeatable way. Um the fourth area is like validation and trust. So"
  },
  {
    "startMs": 2187750,
    "text": "area is like validation and trust. So"
  },
  {
    "startMs": 2187760,
    "text": "area is like validation and trust. So any automated improvement needs evidence"
  },
  {
    "startMs": 2189750,
    "text": "any automated improvement needs evidence"
  },
  {
    "startMs": 2189760,
    "text": "any automated improvement needs evidence that it actually you know works which"
  },
  {
    "startMs": 2191990,
    "text": "that it actually you know works which"
  },
  {
    "startMs": 2192000,
    "text": "that it actually you know works which means like having robust hold out"
  },
  {
    "startMs": 2194069,
    "text": "means like having robust hold out"
  },
  {
    "startMs": 2194079,
    "text": "means like having robust hold out evaluations and some sort of regression"
  },
  {
    "startMs": 2196069,
    "text": "evaluations and some sort of regression"
  },
  {
    "startMs": 2196079,
    "text": "evaluations and some sort of regression testing. And finally, we want to like"
  },
  {
    "startMs": 2198390,
    "text": "testing. And finally, we want to like"
  },
  {
    "startMs": 2198400,
    "text": "testing. And finally, we want to like obviously democratize RSI across teams"
  },
  {
    "startMs": 2200550,
    "text": "obviously democratize RSI across teams"
  },
  {
    "startMs": 2200560,
    "text": "obviously democratize RSI across teams where the goal is to develop reusable"
  },
  {
    "startMs": 2202310,
    "text": "where the goal is to develop reusable"
  },
  {
    "startMs": 2202320,
    "text": "where the goal is to develop reusable components and self-service infra so"
  },
  {
    "startMs": 2204950,
    "text": "components and self-service infra so"
  },
  {
    "startMs": 2204960,
    "text": "components and self-service infra so that teams can adopt these capabilities"
  },
  {
    "startMs": 2206870,
    "text": "that teams can adopt these capabilities"
  },
  {
    "startMs": 2206880,
    "text": "that teams can adopt these capabilities without having to build an entire RSI"
  },
  {
    "startMs": 2209510,
    "text": "without having to build an entire RSI"
  },
  {
    "startMs": 2209520,
    "text": "without having to build an entire RSI pipeline themselves which is what our"
  },
  {
    "startMs": 2211270,
    "text": "pipeline themselves which is what our"
  },
  {
    "startMs": 2211280,
    "text": "pipeline themselves which is what our broader vision is. And I think that this"
  },
  {
    "startMs": 2213990,
    "text": "broader vision is. And I think that this"
  },
  {
    "startMs": 2214000,
    "text": "broader vision is. And I think that this is an important shift in how we approach"
  },
  {
    "startMs": 2215750,
    "text": "is an important shift in how we approach"
  },
  {
    "startMs": 2215760,
    "text": "is an important shift in how we approach the engineering of like AI systems."
  },
  {
    "startMs": 2219270,
    "text": "the engineering of like AI systems."
  },
  {
    "startMs": 2219280,
    "text": "the engineering of like AI systems. Um so in recursive self-improvement we"
  },
  {
    "startMs": 2222150,
    "text": "Um so in recursive self-improvement we"
  },
  {
    "startMs": 2222160,
    "text": "Um so in recursive self-improvement we basically just to summarize we have like"
  },
  {
    "startMs": 2224230,
    "text": "basically just to summarize we have like"
  },
  {
    "startMs": 2224240,
    "text": "basically just to summarize we have like two key components which is the"
  },
  {
    "startMs": 2225589,
    "text": "two key components which is the"
  },
  {
    "startMs": 2225599,
    "text": "two key components which is the improvement engine and the target agent."
  },
  {
    "startMs": 2227750,
    "text": "improvement engine and the target agent."
  },
  {
    "startMs": 2227760,
    "text": "improvement engine and the target agent. The target agent in the AI system is"
  },
  {
    "startMs": 2229670,
    "text": "The target agent in the AI system is"
  },
  {
    "startMs": 2229680,
    "text": "The target agent in the AI system is something we're trying to make better"
  },
  {
    "startMs": 2230950,
    "text": "something we're trying to make better"
  },
  {
    "startMs": 2230960,
    "text": "something we're trying to make better while the improvement engine is you know"
  },
  {
    "startMs": 2232710,
    "text": "while the improvement engine is you know"
  },
  {
    "startMs": 2232720,
    "text": "while the improvement engine is you know responsible for driving that change by"
  },
  {
    "startMs": 2235109,
    "text": "responsible for driving that change by"
  },
  {
    "startMs": 2235119,
    "text": "responsible for driving that change by evaluating performance identifying"
  },
  {
    "startMs": 2237030,
    "text": "evaluating performance identifying"
  },
  {
    "startMs": 2237040,
    "text": "evaluating performance identifying failures root causes and proposing"
  },
  {
    "startMs": 2239109,
    "text": "failures root causes and proposing"
  },
  {
    "startMs": 2239119,
    "text": "failures root causes and proposing improvements. But the objective of the"
  },
  {
    "startMs": 2241270,
    "text": "improvements. But the objective of the"
  },
  {
    "startMs": 2241280,
    "text": "improvements. But the objective of the RSI system isn't simply to improve the"
  },
  {
    "startMs": 2243829,
    "text": "RSI system isn't simply to improve the"
  },
  {
    "startMs": 2243839,
    "text": "RSI system isn't simply to improve the agents performance at like a given point"
  },
  {
    "startMs": 2245829,
    "text": "agents performance at like a given point"
  },
  {
    "startMs": 2245839,
    "text": "agents performance at like a given point in time. It's to like accelerate how"
  },
  {
    "startMs": 2247750,
    "text": "in time. It's to like accelerate how"
  },
  {
    "startMs": 2247760,
    "text": "in time. It's to like accelerate how quickly the agent gets better. And if we"
  },
  {
    "startMs": 2250310,
    "text": "quickly the agent gets better. And if we"
  },
  {
    "startMs": 2250320,
    "text": "quickly the agent gets better. And if we think of like the agent performance as a"
  },
  {
    "startMs": 2252630,
    "text": "think of like the agent performance as a"
  },
  {
    "startMs": 2252640,
    "text": "think of like the agent performance as a function of time, we are interested in"
  },
  {
    "startMs": 2254630,
    "text": "function of time, we are interested in"
  },
  {
    "startMs": 2254640,
    "text": "function of time, we are interested in increasing its first like its first"
  },
  {
    "startMs": 2256710,
    "text": "increasing its first like its first"
  },
  {
    "startMs": 2256720,
    "text": "increasing its first like its first derivative. If we think about it"
  },
  {
    "startMs": 2258150,
    "text": "derivative. If we think about it"
  },
  {
    "startMs": 2258160,
    "text": "derivative. If we think about it essentially like making the performance"
  },
  {
    "startMs": 2260790,
    "text": "essentially like making the performance"
  },
  {
    "startMs": 2260800,
    "text": "essentially like making the performance um improvement curve steeper and in our"
  },
  {
    "startMs": 2263750,
    "text": "um improvement curve steeper and in our"
  },
  {
    "startMs": 2263760,
    "text": "um improvement curve steeper and in our case like the target agent is our user"
  },
  {
    "startMs": 2265510,
    "text": "case like the target agent is our user"
  },
  {
    "startMs": 2265520,
    "text": "case like the target agent is our user understanding agent while the"
  },
  {
    "startMs": 2266710,
    "text": "understanding agent while the"
  },
  {
    "startMs": 2266720,
    "text": "understanding agent while the improvement engine is like the RSI"
  },
  {
    "startMs": 2268470,
    "text": "improvement engine is like the RSI"
  },
  {
    "startMs": 2268480,
    "text": "improvement engine is like the RSI pipeline which brings together"
  },
  {
    "startMs": 2270550,
    "text": "pipeline which brings together"
  },
  {
    "startMs": 2270560,
    "text": "pipeline which brings together evaluation, reflection and firmed"
  },
  {
    "startMs": 2272310,
    "text": "evaluation, reflection and firmed"
  },
  {
    "startMs": 2272320,
    "text": "evaluation, reflection and firmed optimization."
  },
  {
    "startMs": 2274630,
    "text": "optimization."
  },
  {
    "startMs": 2274640,
    "text": "optimization. Um before I close, we want to like"
  },
  {
    "startMs": 2276790,
    "text": "Um before I close, we want to like"
  },
  {
    "startMs": 2276800,
    "text": "Um before I close, we want to like acknowledge the people who helped make"
  },
  {
    "startMs": 2278390,
    "text": "acknowledge the people who helped make"
  },
  {
    "startMs": 2278400,
    "text": "acknowledge the people who helped make this work possible. This has been like"
  },
  {
    "startMs": 2280230,
    "text": "this work possible. This has been like"
  },
  {
    "startMs": 2280240,
    "text": "this work possible. This has been like an collaborative effort across the core"
  },
  {
    "startMs": 2282069,
    "text": "an collaborative effort across the core"
  },
  {
    "startMs": 2282079,
    "text": "an collaborative effort across the core team, our evaluation team and our"
  },
  {
    "startMs": 2284150,
    "text": "team, our evaluation team and our"
  },
  {
    "startMs": 2284160,
    "text": "team, our evaluation team and our engineering partners and leadership. Uh"
  },
  {
    "startMs": 2286470,
    "text": "engineering partners and leadership. Uh"
  },
  {
    "startMs": 2286480,
    "text": "engineering partners and leadership. Uh we we particularly like to thank all the"
  },
  {
    "startMs": 2288630,
    "text": "we we particularly like to thank all the"
  },
  {
    "startMs": 2288640,
    "text": "we we particularly like to thank all the team members and collaborators listed"
  },
  {
    "startMs": 2290470,
    "text": "team members and collaborators listed"
  },
  {
    "startMs": 2290480,
    "text": "team members and collaborators listed here for their contributions."
  },
  {
    "startMs": 2292710,
    "text": "here for their contributions."
  },
  {
    "startMs": 2292720,
    "text": "here for their contributions. Um a lot of the work we have described"
  },
  {
    "startMs": 2294550,
    "text": "Um a lot of the work we have described"
  },
  {
    "startMs": 2294560,
    "text": "Um a lot of the work we have described for evaluation quality to reflection and"
  },
  {
    "startMs": 2296870,
    "text": "for evaluation quality to reflection and"
  },
  {
    "startMs": 2296880,
    "text": "for evaluation quality to reflection and production required collaborations"
  },
  {
    "startMs": 2298390,
    "text": "production required collaborations"
  },
  {
    "startMs": 2298400,
    "text": "production required collaborations across these different areas of"
  },
  {
    "startMs": 2300630,
    "text": "across these different areas of"
  },
  {
    "startMs": 2300640,
    "text": "across these different areas of expertise and we're grateful for the"
  },
  {
    "startMs": 2302550,
    "text": "expertise and we're grateful for the"
  },
  {
    "startMs": 2302560,
    "text": "expertise and we're grateful for the support and um feedback we have"
  },
  {
    "startMs": 2304230,
    "text": "support and um feedback we have"
  },
  {
    "startMs": 2304240,
    "text": "support and um feedback we have received. Um"
  },
  {
    "startMs": 2307109,
    "text": "received. Um"
  },
  {
    "startMs": 2307119,
    "text": "received. Um yep so thank you and uh I guess we just"
  },
  {
    "startMs": 2310790,
    "text": "yep so thank you and uh I guess we just"
  },
  {
    "startMs": 2310800,
    "text": "yep so thank you and uh I guess we just want to conclude by saying that um most"
  },
  {
    "startMs": 2313910,
    "text": "want to conclude by saying that um most"
  },
  {
    "startMs": 2313920,
    "text": "want to conclude by saying that um most important part of like improvement"
  },
  {
    "startMs": 2315430,
    "text": "important part of like improvement"
  },
  {
    "startMs": 2315440,
    "text": "important part of like improvement processes is like being able to you know"
  },
  {
    "startMs": 2317349,
    "text": "processes is like being able to you know"
  },
  {
    "startMs": 2317359,
    "text": "processes is like being able to you know trust the changes we are making to make"
  },
  {
    "startMs": 2319349,
    "text": "trust the changes we are making to make"
  },
  {
    "startMs": 2319359,
    "text": "trust the changes we are making to make the system better. So that's some of the"
  },
  {
    "startMs": 2321589,
    "text": "the system better. So that's some of the"
  },
  {
    "startMs": 2321599,
    "text": "the system better. So that's some of the directions we're exploring with um"
  },
  {
    "startMs": 2323510,
    "text": "directions we're exploring with um"
  },
  {
    "startMs": 2323520,
    "text": "directions we're exploring with um recursive self-improvement here. So"
  },
  {
    "startMs": 2325190,
    "text": "recursive self-improvement here. So"
  },
  {
    "startMs": 2325200,
    "text": "recursive self-improvement here. So yeah, thank you so much for listening."
  }
]