# Self-Evolving Code with AI: Enhancing Quality and Security in CI

Gunjan Patel, Palo Alto Networks

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/prttz9_1XDc-self-evolving-code-ai-enhancing-quality-security) · [Video](https://www.youtube.com/watch?v=prttz9_1XDc)

## Self-Evolving Code with AI: Enhancing Quality and Security in CI

A CI-based assistant can clarify code, build a behavioral test baseline, and debate security fixes before handing a reasoned proposal to a human reviewer.

Familiarity with pull requests, CI, and unit tests will help you follow the workflow; no prior experience building AI agents is required.

### Keeping routine work out of the coding flow

[0:15](https://www.youtube.com/watch?v=prttz9_1XDc&t=15s)

How much of developing software is actually writing application code, and what happens to your concentration when the work shifts to tests, documentation, and deployment? Gunjan Patel explores that problem through an independent side project, not a Palo Alto Networks product. At the time of the talk, he was a director of engineering there and had spent the preceding year and a half to two years moving into generative AI. He dedicates the talk to his close friend Nikhil, who had been enthusiastic about the conference and died unexpectedly two weeks earlier.

Patel starts with the relationship between skill and challenge described in the book he calls *The Flow*. A green flow zone sits between anxiety and boredom. A developer begins with an idea and writes code; a difficult problem raises the challenge and can push the developer toward anxiety. Then unit tests and documentation arrive. In Patel’s framing, those tasks can pull the same developer toward boredom because the work no longer matches their skill level.

The surrounding work is substantial: Kubernetes manifests, Dockerfiles, deployment, and tests all compete with application development. A copilot helps overcome immediate technical hurdles, but the developer still has to steer each interaction. Patel’s proposed change is to **move routine work into CI**, where it can proceed without continuously interrupting the person writing the code.

![Slide titled “What’s the problem?” compares three emoji-based workflows: Reality of the SDLC, The Copilot Duct tape, and Copilot + Boring Tasks in CI.](https://ai.engineer/api/talk-essay-media/b4e283971ad6a04c226acd0bf1f8f5dabc13bb3e9d26a351093c987d575044e0)

The development workflow, with Copilot and with boring tasks moved into CI.

A ChatGPT meme supplies the joke behind that ambition: people at both ends of the spectrum simply use ChatGPT, while the person in the middle—Patel includes himself—juggles tools to get the work done. Moving tasks offline is also an attempt to reduce that juggling.

### A ghostpilot has time to reconsider

[3:29](https://www.youtube.com/watch?v=prttz9_1XDc&t=209s)

Patel calls the offline assistant a **ghostpilot**. Sourcegraph Cody and GitHub Copilot illustrate the live alternative: they work beside a developer who expects a response in seconds. Code review has a different interaction pattern. A reviewer needs context, time to reflect, and opportunities to revise an answer.

| Dimension | Copilot | Ghostpilot |
| --- | --- | --- |
| Interaction | Live assistance | Offline review |
| Immediate goal | Help with the next decision | Assess a change in context |
| Reasoning pattern | Quick response | Reflection and iteration |

Patel connects this distinction to Daniel Kahneman’s [Thinking, Fast and Slow](https://us.macmillan.com/books/9780374275631/thinkingfastandslow/): the copilot fills the fast role, while the ghostpilot provides a slower, more deliberate process.

![Slide contrasts quick just-in-time Copilot decisions with deliberate code review, reflection, and full-context answers, alongside the cover of Thinking, Fast and Slow.](https://ai.engineer/api/talk-essay-media/acfae68265b2c0cacaa6bbe6204730ae6e718a193376afef6f9eadc55aa540ad)

Co-pilot versus Ghost-pilot: quick decisions versus deliberate review.

### Establish intent and behavior before proposing fixes

[4:57](https://www.youtube.com/watch?v=prttz9_1XDc&t=297s)

The workflow begins with a developer writing code using an IDE assistant, then submitting it through the pull-request process. CI performs an ordered sequence:

1. **Clarify intent.** Improve variable names and comments so subsequent model calls have a better account of what the code is supposed to do.
2. **Establish behavior.** Generate and run unit tests in a loop, creating a baseline of expected outputs for given inputs.
3. **Identify security issues.** Use environmental context to look for missing security practices and relevant vulnerabilities.
4. **Propose fixes and rerun tests.** Prepare changes for human review, then check them against the earlier baseline.

The order matters: clearer intent informs the tests, and the tests provide a behavioral check on later changes. Patel explicitly corrects himself when describing the last stage: the system proposes fixes; a human reviews them.

### Names and comments supply domain context

[6:29](https://www.youtube.com/watch?v=prttz9_1XDc&t=389s)

Why spend a CI stage on names and comments? Patel recalls a line from a conference session by Manuel: “LLMs are cultural technology.” The point is that a model’s knowledge extends beyond programming into domains such as finance, psychology, and art. Meaningful identifiers can connect the code to that broader knowledge.

In a finance application, `rev` may mean revenue to its author. Naming it `annual_revenue` makes both the domain and the time period explicit. Patel suggests that subsequent model calls—even calls to different providers—can then draw on financial knowledge and consider corner cases that an abbreviated name might not evoke. The name becomes context passed between stages.

Comments serve a related purpose, but they can become stale when someone changes code in a hurry. The useful information is the intended behavior, which may differ from what buggy code currently does. Improving comments should make that intention legible to the next model.

In Patel’s CI example, the assistant adds a high-level description of a code block’s expected flow. It also makes a server’s port identifier explicitly refer to a port number, distinguishing it from a port name. That small clarification becomes useful when the next stage decides which inputs and behaviors to test.

### List test cases before writing test code

[8:59](https://www.youtube.com/watch?v=prttz9_1XDc&t=539s)

The test stage builds coverage in layers: ordinary behavior first, corner cases second, and regression precautions drawn from historical bugs third. A project’s Jira history can reveal failures that deserve explicit coverage even when they are not obvious from the current code.

Patel reports that a single request to generate unit tests does a poor job in his workflow. CI offers room to separate the reasoning into multiple steps. People still care about CI completion time, but they are not waiting for each response in the same way they wait for an IDE suggestion.

The sequence is deliberately staged:

1. **Describe expected behavior.** Have the model read the code, comments, README, and possibly a product requirements document, or PRD. It first states what the code should do. Requirements may describe overall intent without reaching function-level detail.
2. **Enumerate adversarial cases.** Give the same model or another model an adversarial mindset. Ask for unusual characters, minimum and maximum values, and boundary conditions. List cases without writing tests yet.
3. **Personalize the list.** Add cases based on team priorities in `context.yaml` and the project’s historical bugs.
4. **Generate and run tests.** Turn the assembled case list into test code, then iterate until the tests pass.

The intermediate case list makes the coverage decision explicit before the model commits to test implementation. Patel describes repeated model work as cheap or free labor, but supplies no cost measurements for this loop.

### Combine declared context with inferred priorities

[11:36](https://www.youtube.com/watch?v=prttz9_1XDc&t=696s)

The context file describes the application’s operating environment: cloud provider, service, deployment location, whether the tool is internal or customer-facing, whether it handles personally identifiable information, and whether it is a frontend or backend application. Networking details, the PRD, and the Jira project connect that environment to requirements and bug history.

A Slack channel has a more ambitious proposed role: the assistant could ask a person for missing context. Patel explicitly says this interaction is not implemented. He compares the intended experience to Perplexity Pro asking a clarifying question when a request is unclear. Company security policies and InfoSec guidelines complete the declared context used for review.

A second portion of the context is inferred by AI. The model identifies the language and kind of code—such as infrastructure as code, backend code, or shell code—because each implies different review priorities. Historical bugs and escalations identify focus areas such as SQL injection, timeouts, and concurrent users. Patel also points to Quinn’s [OpenCtx](https://github.com/sourcegraph/openctx) as a potential way to bring in additional context, rather than an integration already present in the project.

### Debate which security findings warrant a fix

[13:20](https://www.youtube.com/watch?v=prttz9_1XDc&t=800s)

Security review uses that context to look for company security practices that are not being followed and for problems in logical flow. Patel distinguishes this goal from running static application security testing, or SAST. The task includes **prioritizing findings**, because asking a model to find problems tends to produce a list even when some issues are minor. A useful review must decide which ones warrant action.

![Finding security issues slide lists security best practices not followed, potential logical or flow-based issues, prioritization, and a distinction from SAST or static code analysis.](https://ai.engineer/api/talk-essay-media/6943d40a005b115845c5cf22b371751109feb2383db3054e0dec18b5d413dc5a)

Security review checks best practices, logical issues, and finding priorities.

To make that decision, Patel simulates three employees. For Python code, the roles receive different context:

| Role | Context supplied |
| --- | --- |
| Red-team engineer | Security policies and reviewed code |
| Python developer | Code author’s perspective and product requirements |
| Engineering manager | Business context, Zoom transcripts, Slack conversations, and PRD |

The developer represents the intent behind the implementation, while the manager brings in business constraints that a security-only reading could miss.

A prompt has the roles take turns debating each issue: what risk it creates, whether it needs fixing, and how much effort a fix requires. After the debate, a reflection step checks whether the answer still matches the original intent. That check can use a separate model. The output is therefore a considered priority decision, rather than merely the first reviewer’s list of concerns.

Patel then suggests an optional experiment he recalls from a podcast as an “Odyssey Journey”: imagine three possible future paths, then return to advise your past self. This is a proposed prompting exercise, not another established pipeline stage. He mentions research suggesting such approaches can outperform standard LLM use, but does not identify the study or give comparison conditions.

### A vulnerability in the placement of a condition

[16:07](https://www.youtube.com/watch?v=prttz9_1XDc&t=967s)

The concrete security example is a vulnerability in Kubernetes Go code that Patel describes as critical. He says he selected an issue from after **GPT-3.5**’s training cutoff. The relevant repair moves an `if` statement from one place in the control flow to another: the problem lies in when a condition is checked.

Patel reports that his workflow finds the issue and argues that SAST tools would miss it because it is a logical-flow problem. The talk does not identify the vulnerability, exact patch, model snapshot, or detection output, so this remains one reported example rather than a measured detection rate or a verified comparison with SAST.

### Hand the reviewer a proposal, rationale, and test result

[16:43](https://www.youtube.com/watch?v=prttz9_1XDc&t=1003s)

The final stage produces outcome-oriented suggestions, with the human reviewer deciding which fixes to accept. For each proposal, three bullet points summarize the simulated employees’ discussion and explain the priority. Security-policy citations give the reviewer a concrete basis for the recommendation. Before that handoff, CI reruns the unit tests established earlier to check the proposed change against the behavioral baseline.

Patel closes the technical demonstration with a deliberately simple example: a hard-coded API key. The review comment includes a risk score, an effort score, a recommendation, and a proposed code snippet. A Python recommendation for removing an embedded key can take this form:

```python
import os


def load_api_key() -> str:
    api_key = os.environ["SERVICE_API_KEY"]
    if not api_key.strip():
        raise ValueError("SERVICE_API_KEY must not be empty")
    return api_key
```

This proposal moves the credential into runtime configuration and fails when it is missing or empty. The review still needs to explain the relevant policy, the risk being addressed, and the effort of supplying that configuration. The snippet is the proposed change; acceptance remains the human reviewer’s decision.

The closing slide points to Patel’s contact information and a GitHub repository. At the time of the recording, he says the code, GitLab CI files, and GitHub Actions files will be uploaded in the next few days. The implementation handoff is therefore a promised release, following a workflow whose final artifact is a tested, explained proposal for human review.

![Thanks slide showing Gunjan Patel, Palo Alto Networks, LinkedIn, Twitter and GitHub contact links, and an underlined repository URL.](https://ai.engineer/api/talk-essay-media/ee00e39739ec11aebbcb0525959d4a5fd69d52adf99a10409ac31a348f25bad4)

Gunjan Patel’s closing contact and GitHub repository slide.

## Resources

- [OpenCtx](https://github.com/sourcegraph/openctx) — Experimental standard and source code for bringing external developer-tool context into code views. The repository was archived in May 2026.
- [Thinking, Fast and Slow](https://us.macmillan.com/books/9780374275631/thinkingfastandslow/) — Daniel Kahneman's account of intuitive and deliberative thinking supplies Patel's copilot-versus-ghostpilot analogy.
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://arxiv.org/abs/2305.14325) — Research on improving reasoning and factual answers through repeated debate among language-model instances.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=prttz9_1XDc&t=5s) [Microsoft presenting sponsor slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/7d8470549143b666e7cf2df33780479bb20cd81328a14b580fc13aaccb7137ae.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=prttz9_1XDc&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/846dc2dd9aef9a53afbdddfdf83ea7260dbe15448544d647afac5905e4d78006.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Self-Evolving Code with AI: Enhancing Quality and Security in CI

Gunjan Patel explores self-evolving code as a side project, not a Palo Alto Networks product.

[0:15](https://www.youtube.com/watch?v=prttz9_1XDc&t=15s) [Title slide naming Gunjan Patel, Director of Engineering at Palo Alto Networks.](https://assets.ai.engineer/sites/aiecode2025/sha256/b931a06d764c9fbfe6e8846c7e90c377f9998dd0f7712aba0886b193269421e0.webp)

🚀 Self-Evolving Code with AI:
Enhancing Quality and Security in CI 💻

Gunjan Patel
Director of Engineering @ paloalto NETWORKS

### The coding flow

The chart places flow between anxiety and boredom as abilities and challenges increase. A dashed path rises through the green flow zone.

[1:08](https://www.youtube.com/watch?v=prttz9_1XDc&t=68s) [A chart of challenges against abilities shows a green flow zone between anxiety and boredom, with a dashed path rising through the flow zone.](https://assets.ai.engineer/sites/aiecode2025/sha256/a804c0cbb1df3be7db6481ff94bb623bca46beebc966416e66466cedf427ac8c.webp)

The coding flow

ANXIETY
FLOW
BOREDOM

Y-axis: CHALLENGES — Low, High
X-axis: ABILITIES — Low, High

[1:30](https://www.youtube.com/watch?v=prttz9_1XDc&t=90s) [A chart places flow between anxiety and boredom as challenges and abilities increase.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a64b58c275653edb5cfd1bd21a884fa86d9007085ed4e0849056f785c02d386.webp)

The coding flow

ANXIETY
FLOW
BOREDOM

Y-axis: CHALLENGES — High
X-axis: ABILITIES — High

Interface text:
AIE
Microsoft
smol ai
World’s Fair
neo4j
Galileo
neptune.ai
groq

### The coding flow: the beginning

A project begins with an idea and initial code. The arrow moves upward through the flow zone as abilities and challenges increase, between anxiety and boredom.

[1:33](https://www.youtube.com/watch?v=prttz9_1XDc&t=93s) [The flow chart shows an initial arrow alongside the steps idea and initial code.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e6256501ae815307aceb1cd05a32cf2393d1521d15b8a8cef3d2f3991a38796.webp)

The coding flow: the beginning

0. Idea
1. Initial code

ANXIETY
FLOW
BOREDOM

Vertical axis: CHALLENGES
Vertical axis labels: Low, High
Horizontal axis: ABILITIES
Horizontal axis label: High

Event branding: AIE; Microsoft; smol ai; Google Cloud; AI Engineer World’s Fair; neo4j; Lambda; Covalent

### The coding flow: the challenge

A hard problem steepens the path through the flow zone toward anxiety as challenge rises faster than ability.

[1:41](https://www.youtube.com/watch?v=prttz9_1XDc&t=101s) [A purple arrow rises toward anxiety as a hard problem is added to the coding steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b4502ce65499157476e58cd2f5654b09b741282c564aa40e3db5c40031c9d69.webp)

The coding flow: the challenge

0. Idea
1. Initial code
2. Hard problem

CHALLENGES: Low, High
ABILITIES: High
ANXIETY
FLOW
BOREDOM

Interface text: AIE, Google Cloud, World's Fair, neo4j, Lambda, Covalent, aws

### What's the problem?

Coding is only part of the development lifecycle. The slide contrasts Copilot helping with coding hurdles with moving boring tasks into CI to make better use of developer time.

[2:13](https://www.youtube.com/watch?v=prttz9_1XDc&t=133s) [Three illustrated workflows compare the software development lifecycle, Copilot assistance, and moving boring tasks to CI to make better use of developer time.](https://assets.ai.engineer/sites/aiecode2025/sha256/40a69118f1c3c71f3a89a9da4c369420183b10c68557a291cadf3f53a6dc6618.webp)

What’s the problem?

Reality of the SDLC
CODE

The Copilot Duct tape
CODE

Copilot + Boring Tasks in CI
CI  CD
CODE

Make Better Use of Human [struck through] Developer Time

### How do you code?

A bell-curve meme puts ChatGPT at both extremes and a jumble of coding tools in the middle. The speaker places himself among those juggling tools.

[3:12](https://www.youtube.com/watch?v=prttz9_1XDc&t=192s) [A bell-curve meme places ChatGPT at both ends and a collection of coding tools in the middle.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0b2de3c16c93629a689206bee5e57f9b9648a39be4999c32a3d28184bf15506.webp)

How do you code?

ChatGPT
ChatGPT
CI  CD

Chart percentages: 0.1%, 2%, 14%, 34%, 34%, 14%, 2%, 0.1%
68%
95%
IQ score ticks: 55, 70, 85, 100, 115, 130, 145

### Co-pilot vs Ghost-pilot

The speaker compares a copilot’s quick decisions with a ghost-pilot’s slower code review, which reflects on answers and updates them using full context.

[4:18](https://www.youtube.com/watch?v=prttz9_1XDc&t=258s) [A slide contrasts quick copilot decisions with deliberate, context-aware code review, alongside the book Thinking, Fast and Slow.](https://assets.ai.engineer/sites/aiecode2025/sha256/40566ffda294f52db39c0a352cd8e3edce15837f8841db12ec193ba41aac36fc.webp)

Co-pilot vs Ghost-pilot

• Co-pilot is made for quick just-in-time decisions
• Code-reviewer is made for deliberate full context decisions
  a. Deliberate thinking vs “guess the next word”
  b. Reflecting on the answer
  c. Being aware of the full context to update the answer

Book cover:
THE NEW YORK TIMES BESTSELLER
THINKING, FAST AND SLOW
DANIEL KAHNEMAN
WINNER OF THE NOBEL PRIZE IN ECONOMICS

Event branding: AIE, AWS, Google Cloud, neo4j, AI Engineer World’s Fair, Lambda, Covalent.

### The Ghost-pilot

The proposed CI workflow improves names and comments, establishes a unit-test baseline, then uses environmental context to identify vulnerabilities and propose fixes for human review.

[4:58](https://www.youtube.com/watch?v=prttz9_1XDc&t=298s) [A CI workflow diagram moves from code comments and tests through context-aware vulnerability identification and proposed fixes with human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/66cf91a71e5da2d636b4b9aeda301c64d34fc829bc59636eaca2a309c69e01f8.webp)

The Ghost-pilot

Code Files
[Code snippet: illegible]
Code change
Pull Request
Dependencies

1. code comments, variable names
Human
Update Changelog
Improved readability and docs

2. Add & Run Unit Tests
Pass
Fail
Fix Code
UTs to set a baseline for code integrity before making changes
Update Pull Request

3a. Environment Context
3b. Identify Vulnerabilities
Reflect on the answer & update
Identify code/flow based vulns with context for ranking
Comment on Pull Request

4. Propose a fix & run Unit Tests
UT Pass
UT Fail
Update Code
Human
Fix the vulns and validate code integrityis maintained using UTs
Update Pull Request

CI/CD

### Improve the variable names and code comments first

Replacing rev with annual_revenue gives an LLM more context for later analysis. Comments explain developer intent, though they can become outdated or diverge from the code.

[6:25](https://www.youtube.com/watch?v=prttz9_1XDc&t=385s) [Slide explaining how descriptive variable names and code comments provide context for language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a06c7f0dc9598e327334534d81e1aa685c01bf2808ee50921c9238515be1599.webp)

1 Improve the variable names and code comments first

• LLMs = Cultural Technology[1]
• var rev VS var annual_revenue makes a difference for LLMs to fetch the relevant training knowledge
• Good code comments give additional context about the developer’s intent (...which may or maynot match the code reality)
• Doing this first helps with the later steps

[1] Manuel Odendahl - AI Engineer World’s Fair

### Code comments and variable names example

The CI example adds a high-level flow and renames port to port_number, making the code’s intent clearer before AI-generated unit tests.

[8:28](https://www.youtube.com/watch?v=prttz9_1XDc&t=508s) [Side-by-side code diff highlighting added explanatory comments and a clearer port variable name.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d1ffde9a9044a70844194419e453f026313bf4fe2fa02e564b652b50f63f0fa.webp)

1

Before:
import sys
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

HandlerClass = SimpleHTTPRequestHandler
ServerClass = BaseHTTPServer.HTTPServer
Protocol = "HTTP/1.0"

if sys.argv[1:]:
    port = int(sys.argv[1])
else:
    port = 8000
server_address = ('127.0.0.1', port)

After:
import sys
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

# This code sets up a simple HTTP server to serve files from the current directory
# using the specified port number, or 8000 by default.
#
# High-level flow:
# 1. Import necessary libraries
# 2. Set up the handler, server, and protocol
# 3. Parse the command line arguments for the port number
# 4. Set up the server address and protocol version
# 5. Start the server and serve files forever
#
# Dependencies: Python 2.x, BaseHTTPServer, and SimpleHTTPServer

# Set the handler, server, and protocol
Handler = SimpleHTTPRequestHandler
HTTPServer = BaseHTTPServer.HTTPServer
ProtocolVersion = "HTTP/1.0"

# Parse the command line arguments for the port number
if sys.argv[1:]:
    port_number = int(sys.argv[1])
else:
    port_number = 8000

# Set up the server address and protocol version
server_address = ('127.0.0.1', port_number)
Handler.protocol_version = ProtocolVersion

Interface text:
server.py
+39 -11
Viewed

### Adding Unit Tests

Start with tests for expected behavior, then cover corner cases and use past bugs to identify cases that need additional protection.

[8:57](https://www.youtube.com/watch?v=prttz9_1XDc&t=537s) [Slide describing unit tests for baseline behavior, corner cases and historical bugs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a88637c62c1873fdf4bb0eca87fa082a186c9de34ad48f0009946a9229816b9.webp)

2 Adding Unit Tests

• Adding UTs to verify the “expected” behavior and set the baseline
• UTs for checking the corner cases
• UTs based on historic bugs and context

### Adding Unit Tests: Steps

The LLM describes expected behavior, lists edge cases, and adds project context before generating and running unit tests in a loop until they pass.

[9:31](https://www.youtube.com/watch?v=prttz9_1XDc&t=571s) [Four steps for generating unit tests: establish a baseline, identify edge cases, add project context, then generate and run tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/89b03840665916c3a1044b4ab4b1eb231a887b43ab532735f126b79dd1bf29c4.webp)

2 Adding Unit Tests: Steps

Set the baseline: LLM to output the expected behavior of this code with context {code comments for the function, README for the project, PRD}

Adversarial mindset: LLM to assume adversarial mindset and list out tests for edge cases like {min/max, weird characters...}

Personalize to the code: Add context from the past bugs, team priorities and deployment context to list UTs that affect this code

Generate UTs: based on the input from the previous steps, LLM can finally generate and run UTs until they pass in a loop

### Adding Unit Tests: Steps

The proposed workflow defines expected behavior and lists edge cases, then uses past bugs and team priorities to tailor test cases before generating and running tests in a loop until they pass.

[11:05](https://www.youtube.com/watch?v=prttz9_1XDc&t=665s) [Four steps for adding unit tests: establish expected behavior, identify edge cases, personalize tests to the code, and generate and run tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c46121018f77490e603d5845ec34c20ced624bc61bf0cf0a99beb6af4777437.webp)

2 Adding Unit Tests: Steps

Set the baseline: LLM to output the expected behavior of this code with context {code comments for the function, README for the project, PRD}

Adversarial mindset: LLM to assume adversarial mindset and list out tests for edge cases like {min/max, weird characters...}

Personalize to the code: Add context from the past bugs, team priorities and deployment context to list UTs that affect this code

Generate UTs: based on the input from the previous steps, LLM can finally generate and run UTs until they pass in a loop

Interface text: AIE; Google Cloud; neo4j; AI Engineer World's Fair; Lambda; Covalent; Microsoft; smol ai

[11:25](https://www.youtube.com/watch?v=prttz9_1XDc&t=685s) [Four steps for generating unit tests: establish expected behavior, identify edge cases, add project context, and generate and run tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/d8e7fe1ed893e4a443e88337a2779bed78c249dc2bf9568941e3ad1a4b28caa6.webp)

2 Adding Unit Tests: Steps

Set the baseline: LLM to output the expected behavior of this code with context {code comments for the function, README for the project, PRD}

Adversarial mindset: LLM to assume adversarial mindset and list out tests for edge cases like {min/max, weird characters...}

Personalize to the code: Add context from the past bugs, team priorities and deployment context to list UTs that affect this code

Generate UTs: based on the input from the previous steps, LLM can finally generate and run UTs until they pass in a loop

Interface text: AIE; Google Cloud; neo4j; AI Engineer World's Fair; Lambda; Covalent

### Context: User provided

The context file helps tailor unit tests to the application and its deployment, with Jira for bug reports and security policies for guidance. The speaker also proposes Slack questions to fill context gaps; this is not implemented yet.

[11:37](https://www.youtube.com/watch?v=prttz9_1XDc&t=697s) [Example context file listing deployment, application, network, documentation, project, communication, and security-policy fields.](https://assets.ai.engineer/sites/aiecode2025/sha256/97978bd08e253132f94beddc5ca06959277564f6f86b3c31df8d76daf088ffb1.webp)

3a Context: User provided

Context.yaml

cloud: "GCP | AWS | Azure"
service: "Kubernetes | VM | lambda"
environment: "company-tool | customer-facing"
pii_data: "True | False"
type: "backend | frontend"
modality: "internal | external"
network: "private-cloud | internal-vpn | public"
prd_doc: “https://link-to-the-prd-doc”
jira_project_id: “AIEWF”
slack_channel_id: “aiewf-devs”
security_policies: “https://link-to-infosec-guidelines”

### Context: AI inferred

AI infers the language and code type to guide priorities, then draws focus areas from historical bugs. OpenCtx.org is a potential integration point for additional context.

[12:40](https://www.youtube.com/watch?v=prttz9_1XDc&t=760s) [AI-inferred context includes programming language, code type, and focus areas informed by historical bugs.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c490b908a70850bb70905c80a53e9f29e93bac2635c9f70defc226adbb5e468.webp)

3a Context: AI inferred

language: “Go | Python | Java | Rust”

code_type: “kubernetes | dockerfile | helm | server | shell”

focus_areas: “sql_injection | timeout_errors |
concurrent_users | timezones” //based on historic bugs

PS: OpenCtx.org

### Finding security issues

The review looks for missed security practices and logical flow issues, then prioritizes what needs fixing. The speaker distinguishes this from static analysis and SAST tools.

[13:19](https://www.youtube.com/watch?v=prttz9_1XDc&t=799s) [Slide lists security practices, logical and flow issues, prioritization, and the distinction from static analysis.](https://assets.ai.engineer/sites/aiecode2025/sha256/f5ffe63fb1ae3ff3a67f1ebcb0fb14c81024b31a4a0c781a1d27e645756a51ee.webp)

3b Finding security issues

• Look for security best practices not followed
• Identify potential logical/flow based issues
• Prioritize the relevant ones
• Different from SAST/ static code analysis tools

### Finding security issues: prompting

Three simulated roles receive different context and debate which security issues to prioritize. A different model reflects on the outcome before recommendations are posted.

[13:58](https://www.youtube.com/watch?v=prttz9_1XDc&t=838s) [Slide depicts three simulated roles with different context debating and prioritizing security issues, followed by reflection.](https://assets.ai.engineer/sites/aiecode2025/sha256/64f5f5e79abc811112dcfe53147254803d7750503a66fd2cbb1ccd229db2c2ec.webp)

3b Finding security issues: prompting

Red Team Engineer
Security policies + Code
[Code screenshot: illegible]

Python Developer
Code + PRD
[Code screenshot: illegible]

Engineering Manager
PRD + Business context

• Simulate AI employees and have them debate among themselves to identify and prioritize issues
• Multi-turn discussions with different context and priority for each
• Self-reflect on the outcome before posting the recommendations (different model)
• Fun: The odyssey journey, time travel, code survivor

### Fixing security issues

The workflow suggests security fixes and explains their priority with policy citations. Baseline unit tests run before a human reviewer decides which fixes to make.

[16:41](https://www.youtube.com/watch?v=prttz9_1XDc&t=1001s) [Slide describes outcome-based fix suggestions, reasons and policy citations, and unit tests for baseline behavior.](https://assets.ai.engineer/sites/aiecode2025/sha256/b12799f577ff642872711301e4bfeb0ee060bf8eaea7a44a0d26060dc5d6d1ee.webp)

4 Fixing security issues

• Outcome based fix suggestions from the previous stage
• Explain reasoning behind prioritization and cite policies / best practices
• Run Unit Tests written in step 2 to make sure the fix doesn’t change the code baseline behavior

[17:02](https://www.youtube.com/watch?v=prttz9_1XDc&t=1022s) [Slide describing outcome-based security fixes, prioritization and policy citations, and baseline unit tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe784fbf42df78fb7ed92ffac961d296aa56f49f3b26a202acc281fc391ee4e9.webp)

4 Fixing security issues

• Outcome based fix suggestions from the previous stage
• Explain reasoning behind prioritization and cite policies / best practices
• Run Unit Tests written in step 2 to make sure the fix doesn’t change the code baseline behavior

Interface text: AIE; AI Engineer World’s Fair; Google Cloud; Neo4j; Lambda; Covalent; Microsoft; smol ai

### Thanks!

At the talk, Patel said he would upload the code and GitLab CI and GitHub Actions files to the project repository in the next few days.

[17:50](https://www.youtube.com/watch?v=prttz9_1XDc&t=1070s) [Closing slide with Gunjan Patel’s role, social profiles, and project repository.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e79b8f6a5b6759768840b0f19664cb28879987b0df0d948b9b34677fc9652d4.webp)

Thanks!

Gunjan Patel
Director of Engineering @ paloalto NETWORKS

linkedin.com/in/gunjanpatel1
twitter.com/@Gunjan_Patel1
github.com/gunjan5

https://github.com/gunjan5/self-improving-code

## Full transcript

[0:00](https://www.youtube.com/watch?v=prttz9_1XDc&t=0s) [on-hold music] My name is Gunjan Patel.

[0:15](https://www.youtube.com/watch?v=prttz9_1XDc&t=15s) Um, I'm a director of engineering at Palo Alto Networks. Uh, like many of you in the last year and a half or two years, I've pivoted to working on generative AI.

[0:26](https://www.youtube.com/watch?v=prttz9_1XDc&t=26s) Um, the talk today is about self-evolving code. I'm not here to sell you anything. None of this is a product from our company. Uh, this is, um, a side project that I've been exploring.

[0:40](https://www.youtube.com/watch?v=prttz9_1XDc&t=40s) Before we get started with this talk, um, I would like to dedicate this talk to my close friend, Nikhil, who passed away two weeks ago. Um, it was unexpected, uh, but, um, he was very excited for me to come and do this talk.

[0:57](https://www.youtube.com/watch?v=prttz9_1XDc&t=57s) Uh, and he was very enthusiastic about this conference itself. Um, so I wanna dedicate this talk to him.

[1:06](https://www.youtube.com/watch?v=prttz9_1XDc&t=66s) Okay. Let's start with the coding flow. Uh, if you read the book called The Flow, um, it talks about this zone where, uh, your skills on one axis, your, your skills and how challenging the problem is on the next, on x-axis.

[1:25](https://www.youtube.com/watch?v=prttz9_1XDc&t=85s) As you're starting to work on a project, you should be somewhere within that green zone to stay in the flow state.

[1:34](https://www.youtube.com/watch?v=prttz9_1XDc&t=94s) In reality, what happens is, in the beginning, you have an idea, you start writing some code, initially crank out some code.

[1:42](https://www.youtube.com/watch?v=prttz9_1XDc&t=102s) Then you have a little bit of a difficult problem, so it cranks up the, uh, direction a little bit to the anxiety, may even go a little bit into anxiety.

[1:53](https://www.youtube.com/watch?v=prttz9_1XDc&t=113s) And then you have to write unit tests, and then it goes straight to the boredom part, uh, because it's, uh, your skill levels are quite higher, but nobody likes writing unit tests, uh, or code documentation.

[2:10](https://www.youtube.com/watch?v=prttz9_1XDc&t=130s) So what's the problem there? As a developer, you're... When you're writing code, uh, when you're developing software, writing code is a small portion of that entire flow, right? Uh, you have to work with a lot of stuff like, you know, uh, building Kubernetes, uh, like manifest, building Dockerfile, et cetera, deploying stuff, writing unit

[2:35](https://www.youtube.com/watch?v=prttz9_1XDc&t=155s) tests. So then there is this copilot duct tape. Yes, uh, having something like a copilot that would help you get through a lot of technical hurdles on, uh, coding side.

[2:48](https://www.youtube.com/watch?v=prttz9_1XDc&t=168s) Whenever you see something challenging, you can overcome that.

[2:52](https://www.youtube.com/watch?v=prttz9_1XDc&t=172s) Um, but we are trying to outsource more and more of that to do offline, right? So you have, you as a developer, have more time to write code. So the key part in this talk is about outsourcing a boring task, uh, to CI.

[3:11](https://www.youtube.com/watch?v=prttz9_1XDc&t=191s) Uh, [chuckles] uh, had to insert this meme here, um, that like on both ends of the spectrum, people use ChatGPT. In the middle, the mid, uh, such as myself, is like juggling all these different tools, trying to figure out how to write the code.

[3:29](https://www.youtube.com/watch?v=prttz9_1XDc&t=209s) Okay, let's get into this. Uh, so what's the difference between copilot and, uh, I just coined this term ghostpilot. Uh, copilots are made for quick just-in-time dec-- uh, decisions, right?

[3:41](https://www.youtube.com/watch?v=prttz9_1XDc&t=221s) Uh, copilot examples are, uh, Sourcegraph Cody, like Quinn just talked about earlier, uh, and GitHub Copilot.

[3:50](https://www.youtube.com/watch?v=prttz9_1XDc&t=230s) They work with developers live. Developers are impatient. You need response in seconds. Uh, uh, how good of a tool it is, partly measured by how quick the responses are, right?

[4:04](https://www.youtube.com/watch?v=prttz9_1XDc&t=244s) Um, code review-- A code reviewer is made to have a deliberate thinking, right? When you write code and then a code reviewer is reviewing your code, it's deliberate thinking.

[4:17](https://www.youtube.com/watch?v=prttz9_1XDc&t=257s) It's not just guess the next word, uh, at this point. Uh, you need to ref-reflect on the answers that, uh,

[4:25](https://www.youtube.com/watch?v=prttz9_1XDc&t=265s) that you're providing or, uh, like human code reviewer, uh, being aware of the full context and, uh, to update the answer, right? So some of those things require... They need time, they require iteration.

[4:41](https://www.youtube.com/watch?v=prttz9_1XDc&t=281s) Uh, so one, one example is, uh, this book called Thinking, Fast and Slow. Uh, so if you think about copilots as thinking fast, uh, the slow system is this, uh, ghostpilot system.

[4:57](https://www.youtube.com/watch?v=prttz9_1XDc&t=297s) So this is the high-level architecture, uh, I'm talking about here. Developer writes, um, code using, uh, a copilot or IDE assistant, uh, tool.

[5:11](https://www.youtube.com/watch?v=prttz9_1XDc&t=311s) They check in the code in, in CI, in the-- during the pull request process. Uh, the first step is, uh, improving code vari-- uh, code comments and variable names.

[5:24](https://www.youtube.com/watch?v=prttz9_1XDc&t=324s) Second step is adding and running, running unit tests in a loop. Uh, and I'll go into each of these steps in more detail. Uh, adding unit tests here early on...

[5:35](https://www.youtube.com/watch?v=prttz9_1XDc&t=335s) Uh, sorry. Let me talk about the first step. Doing the first step properly means adding unit tests makes more sense, right? Like you need AI to understand what is the intent of this piece of code.

[5:49](https://www.youtube.com/watch?v=prttz9_1XDc&t=349s) Uh, adding unit tests early on sets a baseline behavior for the code that this is-- if this is the input, this is the expected output. So it's set- it's setting the baseline.

[5:59](https://www.youtube.com/watch?v=prttz9_1XDc&t=359s) Then third step is based on the environmental context, identify security issues, security best practices that are not followed. Um-

[6:08](https://www.youtube.com/watch?v=prttz9_1XDc&t=368s) And fourth step is fixing them. And after fix-- proposing a fix, not fixing them, proposing a fix for a human to review, uh,

[6:18](https://www.youtube.com/watch?v=prttz9_1XDc&t=378s) run the unit test again. So let's go into each of these steps. So first one, why improve variable names, uh, and code comments?

[6:29](https://www.youtube.com/watch?v=prttz9_1XDc&t=389s) Uh, yesterday there was a session here from Manuel, uh, that he, he wrote this line: "LLMs are cultural technology," right? LLMs are built based on all the knowledge in the world, not just coding knowledge.

[6:43](https://www.youtube.com/watch?v=prttz9_1XDc&t=403s) Like most of us here, we know a lot more about coding than art or how humans behave, uh, psychology, et cetera. But LLMs have all that knowledge stuffed into them.

[6:56](https://www.youtube.com/watch?v=prttz9_1XDc&t=416s) We should figure out how to utilize that completely, right? So an example is if you're working at a finance company, for example, and use a variable name rev, it makes sense to you, like revenue, in that con- small context of your code.

[7:15](https://www.youtube.com/watch?v=prttz9_1XDc&t=435s) Uh, but improving it to annual revenue, and then in the subsequent steps, if, uh, LLM is run, uh, using different, um, different LLM providers,

[7:27](https://www.youtube.com/watch?v=prttz9_1XDc&t=447s) it will pull in its existing knowledge from all the finance world.

[7:32](https://www.youtube.com/watch?v=prttz9_1XDc&t=452s) So there may be some corner cases that may not have been discovered if you just use the variable name rev versus annual revenue.

[7:41](https://www.youtube.com/watch?v=prttz9_1XDc&t=461s) Um, the second part is, uh, improving good, uh, improving code comments, right? Uh, code com-comments can get out of date quickly. If someone is in a hurry, makes change to a piece of code without improving the code comment, uh, then it's out of date, right?

[7:58](https://www.youtube.com/watch?v=prttz9_1XDc&t=478s) Like, it doesn't state the developer's intent. So improving that, um, to make sure that LLMs understand what is intended behavior of this code, even though-- even if the code may not reflect it, and that's why we have bugs, right?

[8:14](https://www.youtube.com/watch?v=prttz9_1XDc&t=494s) There's the intent. Nobody intends to have a bug. Uh, so making that intention of the code block clear, that's why we improve the code comments.

[8:25](https://www.youtube.com/watch?v=prttz9_1XDc&t=505s) So this is a small example of that. Um, you can see, uh, in CI, it's adding high-level flow of what is expected from this code, uh, code block. Um, here, like, it's a server, so I just put port, and it improved the port number because it could be port name or port number.

[8:48](https://www.youtube.com/watch?v=prttz9_1XDc&t=528s) Uh, I don't think about those things, but, uh, when it comes to adding unit tests, AI-generated unit tests, it ma- it becomes important.

[8:59](https://www.youtube.com/watch?v=prttz9_1XDc&t=539s) Second one is adding unit tests. So adding unit tests, as I said earlier, it's to set the baseline behavior. Um, uh, then

[9:09](https://www.youtube.com/watch?v=prttz9_1XDc&t=549s) o-once we have the baseline behavior, we can add second layer of unit tests co- uh, for covering corner cases. Third one is to get context from historical bugs. Uh, what, uh, looking at Jira fetching for this project, we have seen these kind of bugs in the, in the past.

[9:25](https://www.youtube.com/watch?v=prttz9_1XDc&t=565s) So adding additional precautions to make sure those cases are covered.

[9:31](https://www.youtube.com/watch?v=prttz9_1XDc&t=571s) So what are the steps like? It's not asking an LLM to say, "Generate unit test for this code." It doesn't do a very good job. Uh, and this is where having iterations comes into play, right?

[9:45](https://www.youtube.com/watch?v=prttz9_1XDc&t=585s) So first one we do is, um, because we have all the time in the world, people are not rushing for CI jobs to complete. I mean, yeah, a lot of people are impatient, uh, but it's not like when you're writing code, uh, with an IDE-based assistant, you need response quickly.

[10:02](https://www.youtube.com/watch?v=prttz9_1XDc&t=602s) So what we do in, in this one is set the baseline. So, uh, LLM outputs the expected behavior of this code. N-not the person who wrote it. LLM will read it and say, "Okay, this is the expected behavior."

[10:15](https://www.youtube.com/watch?v=prttz9_1XDc&t=615s) Uh, it will take things into consideration like, uh, code comments, README file, and maybe a PRD document. It may not go into function-level details, but overall intent is there.

[10:29](https://www.youtube.com/watch?v=prttz9_1XDc&t=629s) Uh, then in the next step, uh, we added adversarial mindset. Uh, you can use the same model or a different model and say, uh, system prompt is, uh, you assume adversarial mindset and what are the edge cases.

[10:44](https://www.youtube.com/watch?v=prttz9_1XDc&t=644s) A- think about weird characters, min, max, all the boundary conditions, and add, list out the unit test cases. Don't write the unit test yet, just list it out. Then, uh, personalize the code.

[10:56](https://www.youtube.com/watch?v=prttz9_1XDc&t=656s) Personalizing happens from team priorities that, uh, are written in context.yaml file, which I'll cover in a second. Um, and historic bugs again. So personalizing which unit tests to add based on that.

[11:11](https://www.youtube.com/watch?v=prttz9_1XDc&t=671s) So we'll add few more unit tests at the end that are specifically applicable to this code and this application.

[11:18](https://www.youtube.com/watch?v=prttz9_1XDc&t=678s) And then the last step is, here's the unit test cases we need to cover, generate unit tests. And then it will do a much better job, and then it can go in a loop until they pass, right?

[11:31](https://www.youtube.com/watch?v=prttz9_1XDc&t=691s) Uh, because again, it's cheap labor, free labor.

[11:36](https://www.youtube.com/watch?v=prttz9_1XDc&t=696s) Uh, this is what the context looks like. Uh, context is used in the n- uh, step three, but things like what is the cloud provider? What is the service?

[11:45](https://www.youtube.com/watch?v=prttz9_1XDc&t=705s) Where is it deployed? Is it a company internal tool or a customer-facing tool? Is there PII data in it? Um, is it a front-end or back-end, uh, application?

[11:56](https://www.youtube.com/watch?v=prttz9_1XDc&t=716s) Networking, PRD. Where, where is the Jira project to pull all the bug reports and, uh, more information on this project? Slack channel. Now, this is important. I haven't implemented this part yet, but where it could go and ask follow-up questions to the user saying, "Hey, I'm not sure about this context.

[12:15](https://www.youtube.com/watch?v=prttz9_1XDc&t=735s) It's missing this context." Similar to, uh, if you use Perplexity, uh, and use Perplexity Pro, you ask a question, it's not clear, it'll ask you a question back. Uh, so that's what this is designed to do.

[12:28](https://www.youtube.com/watch?v=prttz9_1XDc&t=748s) Uh, and your company security policies, um, guidelines from your InfoSec team, if you have any.

[12:36](https://www.youtube.com/watch?v=prttz9_1XDc&t=756s) Then second part of the context is built, uh, using AI. So AI infers this context. Uh, what language is this? Is this IaC code or is this back-end code, uh, shell code, et cetera?

[12:49](https://www.youtube.com/watch?v=prttz9_1XDc&t=769s) Uh, because you will have different priorities based on that. Uh, what are the focus areas? That comes from historical bugs, uh, SQL injection, timeout errors, concurrent users, uh, whatever issues as, and escalations you've had in the past.

[13:05](https://www.youtube.com/watch?v=prttz9_1XDc&t=785s) Um, here s- uh, Quinn has done a great job on openctx.org. Uh, that's a potential integration point here, uh, as well for bringing in additional context.

[13:20](https://www.youtube.com/watch?v=prttz9_1XDc&t=800s) Finding security bugs. Now, uh, security issues. So security best pract-- This is not just looking... It's different from running static code analysis and, uh, diff- running SAST-based tools. It's finding sec- which security best practices are not followed that your company may have policies around.

[13:38](https://www.youtube.com/watch?v=prttz9_1XDc&t=818s) Uh, logical flow based issues. Uh, I'll show you an example in a bit. Uh, and prioritize which ones need fixing, right? Not just identify, because when you ask AI to find issues, it's going to find issues.

[13:52](https://www.youtube.com/watch?v=prttz9_1XDc&t=832s) Uh, it's never gonna say, "No, it's fine. I mean, it's a small issue," but it's gonna say something is wrong.

[14:00](https://www.youtube.com/watch?v=prttz9_1XDc&t=840s) So how do we do that? This is, this is a little bit of a weird slide now, right? Uh, what I'm doing here is simulating three AI employees, right?

[14:13](https://www.youtube.com/watch?v=prttz9_1XDc&t=853s) Um, one is a red team engineer, one is a Python developer, assuming this is Python code, and an engineering manager. Uh, engineering managers are useful for something, uh, other than, uh, talking in meetings.

[14:28](https://www.youtube.com/watch?v=prttz9_1XDc&t=868s) Uh. [laughing] So what context am I providing the red team engineer? Security policies and the code that's written, uh, or the code that's being reviewed. Python developer will assume the identity of the person who wrote this code, uh, and the PRD, uh, product requirements.

[14:46](https://www.youtube.com/watch?v=prttz9_1XDc&t=886s) Engineering manager will have business side of things, uh, business context, so, uh, Zoom meeting transcripts, uh, Slack channel conversations and PRD. Now,

[15:00](https://www.youtube.com/watch?v=prttz9_1XDc&t=900s) there is a prompt to have these three people take turn, multi-turn, and then debate among each other on each of the security issues on identifying what is the risks associated with it and which ones need to be fixed, and what is the effort for it.

[15:16](https://www.youtube.com/watch?v=prttz9_1XDc&t=916s) And at the end, after they're done with the debate, uh, self-reflect on the answer and say, "Okay, this is," you know. And that could be a s-separate model. It could look at the answer and say, "Okay, this is matching the original intent or not."

[15:32](https://www.youtube.com/watch?v=prttz9_1XDc&t=932s) Uh, some fun things you could try. Um, I was listening to a podcast recently, and they talked about Odyssey Journey, which is like you think about like three potential paths you could take, uh, and then time travel back and then talk to your past self.

[15:47](https://www.youtube.com/watch?v=prttz9_1XDc&t=947s) Now, that's not really possible in real life, but in here we can do that. Um, with AI, it's, it's all fun, um, but it, uh, there is some research paper that show that this is actually more effective than just using LLMs, uh, as a standard tool.

[16:07](https://www.youtube.com/watch?v=prttz9_1XDc&t=967s) Uh, this is an example. Uh, this was a Kubernetes bug, uh, in the Go code. It was a critical severity bug. Uh, and I made sure to find this issue from after the training end, so GPT-3.5, uh, training date end.

[16:22](https://www.youtube.com/watch?v=prttz9_1XDc&t=982s) Uh, this was-- This wouldn't have been picked up in any of the SAST tools because it's a logical thing. This if sta-statement needs to go from here to here, and that caused the, uh, security vulnerability.

[16:35](https://www.youtube.com/watch?v=prttz9_1XDc&t=995s) Now, that's something that's when I run it through this, it's, it finds that issue.

[16:43](https://www.youtube.com/watch?v=prttz9_1XDc&t=1003s) Last one is fixing the security issues. So this is outcome based, uh, and it's again, suggestion, uh, of what to fix, and then human reviewer comes in at the end and, uh, decides which ones to actually fix.

[16:57](https://www.youtube.com/watch?v=prttz9_1XDc&t=1017s) For each of the fix, explain the reasoning why it's prioritized. Uh, that all the conversation that the, the three virtual employees had ha- have had, uh, it's summarized in three bullet points.

[17:10](https://www.youtube.com/watch?v=prttz9_1XDc&t=1030s) Uh, so human reviewer has the context and security, uh, policy, uh, citation.

[17:18](https://www.youtube.com/watch?v=prttz9_1XDc&t=1038s) And at the end, make sure before a human comes in, run the unit tests that were establishing the baseline early on, so a human doesn't have to waste their time, uh, with verifying the fix.

[17:32](https://www.youtube.com/watch?v=prttz9_1XDc&t=1052s) Um, this is not a great example, but a simple example of what it would look like in a, a code comment, uh, hard-coded API key. I mean, most people wouldn't have that.

[17:43](https://www.youtube.com/watch?v=prttz9_1XDc&t=1063s) Uh, what is the risk score? What is the effort score? Uh, what is the recommendation? Which is the code snippet? And then fix it afterwards.

[17:52](https://www.youtube.com/watch?v=prttz9_1XDc&t=1072s) Uh, that's it. This, these are, this is my information. This is the GitHub repo. Uh, I will upload the code, uh, the GitLab CI and GitHub Actions, uh, files there in the next few days, uh, so bear with me.

[18:06](https://www.youtube.com/watch?v=prttz9_1XDc&t=1086s) Uh, but thank you for coming. [upbeat music]
