AI Engineer World's Fair 2025
How to Hire AI Engineers When Everyone Is Cheating With AI
Read the talk
Hiring AI Engineers When Coding Puzzles Measure the Assistant
AI assistance changes what a coding interview reveals. Beth Glenfield proposes workplace simulations that expose collaboration, technical judgment, and the ability to ship useful AI products.
From a talk by Beth Glenfield
What does a successful coding interview reveal?
Are you using AI in recruitment—as an interviewer or a candidate? Are you competing with Google and Meta for talent? And does solving a LeetCode puzzle still tell you whether someone can build a great AI product? These are Beth Glenfield’s opening questions about technical hiring in 2025. For smaller companies, the problem combines an increasingly questionable assessment signal with competition for candidates they cannot easily outspend.
Looking back over the preceding eighteen months, Glenfield points to Cluely, which she describes as an AI cheating service from a former Columbia student. Glenfield reports a $5.3 million funding round and says Cluely was approaching $1 million in annual recurring revenue. The funding and revenue figures have different evidentiary weight: the revenue trajectory was a founder claim, not an audited result.
Glenfield also cites a 93% success rate for LeetCode Wizard in Google and Meta interviews, and says one in three interviews involves an AI assistant. The Leetcode Wizard figure should be read as a vendor-reported interview-test pass rate, not an independently established hiring or offer rate. Its current page dates its testing claim to August 2026 and cannot establish the historical 2025 testing conditions; the one-in-three figure remains an attributed claim.
A completed puzzle can reveal the strength of the candidate’s assistant more than the candidate’s engineering judgment. That is the assessment failure Glenfield identifies: the output remains visible, while the contribution being evaluated becomes harder to distinguish.
Meanwhile, employers increasingly expect engineers to use those tools. Glenfield quotes Sam Altman: “The tactical thing to do is to learn the best AI tools.” She then attributes to Marc Benioff a halt to Salesforce software-engineer hiring for 2025 and a claimed 30% productivity boost from replacing people with AI. That attribution does not establish replacement as the cause of the gain. Glenfield immediately questions the evidence: “I would like to see the data on that.” Her broader point is that expectations of engineering work have changed, even while interviews continue to test older proxies.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The capabilities puzzles overlook
Compensation is only part of the competition for an engineer. Brand recognition, career prestige, and perceived stability also matter. After layoffs, candidates may place particular weight on security. A Series A startup asking someone to choose it over Google’s AI division therefore has to offer more than a competitive interview process.
At the same time, the people a company needs for AI development may spend little effort optimizing for algorithm puzzles. Glenfield describes creative problem solvers and collaborative leaders who can work effectively with AI. Their relevant activity includes building AI tools, using AI libraries, contributing to open source, and understanding the business impact of their work. A puzzle score does not directly expose those capabilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the interview resemble the work
The alternative begins with a different observation: watch a candidate collaborate with AI teammates on an actual business scenario. Instead of using algorithm recall as the main signal, observe how the candidate delegates tasks, handles ambiguity, and responds when requirements change mid-sprint. AI becomes part of the assessment environment, so the interviewer can examine how the candidate works with it.
That environment also gives the candidate information. Showing how a team works early in the process can make its engineering culture tangible, giving a smaller company something specific to demonstrate beyond brand recognition. Glenfield introduces DevDay as her proposed way to build this kind of technical hiring process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
AI teammates make trade-offs observable
In Glenfield’s description of DevDay, candidates enter workplace simulations with AI agents that have different personalities:
- The perfectionist brings an emphasis on getting the work right.
- The pragmatist supplies a contrasting practical perspective.
- The security expert brings security concerns into the discussion.
- The junior developer needs extensive mentoring.
The candidate has to navigate these colleagues while making everyday engineering trade-offs. The work involves building features for the hiring company’s business domain, giving those decisions a concrete purpose.
The assessment unit becomes behavior during the work, not just the finished code. Glenfield identifies several signals to observe:
| Capability | Evidence in the simulation |
|---|---|
| AI collaboration | How the candidate works with AI teammates |
| Handling ambiguity | How the candidate proceeds with unclear requirements |
| Technical communication | Decisions explained in pull requests and ticket comments |
| Mentorship | How the candidate guides others |
| Adaptability | How the candidate responds when circumstances change |
Pull requests and ticket comments matter because they make technical reasoning visible to teammates. Mentoring and adaptation extend the assessment beyond producing an individual solution: the candidate must help the surrounding team make progress as the situation changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small companies have less room for a hiring mistake
Glenfield illustrates the resources of a large employer with a hypothetical funnel: interview 100 candidates, hire five, and offer large paychecks. This is an illustration of hiring capacity, not reported Google or Meta recruiting data. Most companies cannot absorb hiring mistakes with the same ease.
Glenfield puts the potential loss from a bad hiring process at 20–60K, without specifying currency or cost components. The practical requirement behind that estimate is more direct: a company hiring for AI development needs someone who can ship AI products. An assessment that rewards interview performance without exposing that ability leaves the central hiring risk unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hire for the responsibilities that are changing
The closing argument returns to changing engineering roles. Glenfield cites Mark Zuckerberg’s prediction that AI would handle mid-level engineering work later in 2025, then invokes TechCrunch reporting about disappearing entry-level engineering jobs. The first is a forecast; reduced entry-level hiring does not by itself demonstrate that AI has eliminated an occupation.
Glenfield’s expectation is that engineering jobs will become different rather than necessarily fewer. The responsibilities she emphasizes are creativity, collaboration, and business judgment alongside coding: deciding what to build and working effectively with AI to deliver it. That is the connection back to the interview design. A workplace simulation aims to expose those responsibilities while the candidate is exercising them.
At the time of the talk, Glenfield says DevDay is working with select design partners. She closes by inviting people to learn about the process or discuss how they approach hiring, pointing to her on-screen contact details and offering to talk at the event.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The interview-assistance product's feature descriptions and self-reported testing claims.
Further reading
An analysis of technology hiring trends, including entry-level opportunities and changing demand for skills.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi everyone.
- 0:16
So I'm Beth Glenfield. I actually flew in from Ireland, so, uh, I will try and slow down my accent for everybody. I have heard that I do get a bit American sometimes, so, uh, yeah, it may not be the [REDACTED:origin] accent you're expecting.
- 0:30
So I'm gonna talk to you today about how I believe AI is breaking how we hire technically.
- 0:38
So I know everyone's very busy, but just a few questions to think about. Is that okay? Perfect. Is, are you using AI today in your recruitment process, both as an interviewee and an interviewer?
- 0:54
Are you competing against Google or Meta employees, both for positions but also for candidates? And are you recognizing that LeetCode puzzles just aren't really up to scratch like they used to be for knowing if people will build great AI products?
- 1:13
So welcome to twenty twenty-five, where the technical recruitment process is very broken and small companies are getting crushed in the talent war.
- 1:23
So just to refresh our memories, let's look back at what happened over the last eighteen months. So we had Clue come to the table. Uh, they have raised a five point three million round, and it is a AI cheating service from a ex-Columbia student.
- 1:37
Um, they're currently heading towards one million in ARR. And, um, we're also seeing ninety-three percent LeetCode wizard success rates for Goog- Google and Meta interviews, and we're seeing, like, one in three interviews now are having AI assistants.
- 1:54
So really whenever you're interviewing candidates, you're just interviewing for who has the best AI coding assistant.
- 2:03
So Sam Altman said this very plainly, "The tactical thing to do is to learn the best AI tools." We also saw that Marc Benioff announced that Salesforce will no longer be hiring software engineers this year because according to them, they have seen a thirty percent boost in their productivity since they've replaced people with AI.
- 2:24
I would like to see the data on that. But really, jobs have fundamentally changed.
- 2:31
And you're not just competing on compensation anymore. You're competing on brand recognition, career prestige, and perceived stability. There's a lot of people being laid off, and people are looking for security.
- 2:43
So when a candidate has to choose between your Series A startup and Google's AI division, we kinda know what really happens there.
- 2:53
So the candidates you actually want for AI development, the creative problem solvers, the collaborative leaders, and the ones who can work with AI instead of re- being replaced by AI, they are not being optimized for LeetCode performance.
- 3:09
Instead, they're building AI tools and using AI libraries. They're contributing to open source. They are understanding how the business impact will be impacted, and that is just not what LeetCode is indexing for.
- 3:24
So what if instead of asking candidates to solve puzzles that they will never actually use in a, a job, you could observe how they collaborate with AI teammates on actual business scenarios?
- 3:41
What if instead of measuring their ability to memorize algorithms, you can see how they delegate tasks, how they handle ambiguity, and how they deal whenever requirements change mid-sprint? So you're not competing on brand recognition.
- 3:55
You can demonstrate to a candidate at the beginning of a process what your engineering culture is and see how they would perform in your environment.
- 4:06
So at DevDay, we are completely reimagining the technical hiring process in the AI era. Here are some examples of what we're seeing today, like I mentioned earlier, and then we'll get into what we're considering is the new kind of process you should be looking for in a hiring process.
- 4:26
What we, what we do instead of getting candidates to code interviews is we create real-life workplace simulations. Candidates work alongside AI agents with very different personalities. You can have the perfectionist, the pragmatist, the security expert, and even the junior developer that needs extensive mentoring, so that now they have to make trade-offs on day in, day out situations.
- 4:50
They're not solving contrived problems. They are building features specifically for your business domain.
- 4:57
So instead of measuring how c- they can code, we're actually, actually measuring them on skills that matter. How do they collaborate with AI? How do they handle ambiguity? How do they communicate these technical decisions in pull requests, in comments on tickets?
- 5:14
And how do they mentor others and adapt when everything changes on a daily basis?
- 5:20
Now, if you do work at Google or Meta, you can just brute force your hiring process. You interview a hundred candidates, you hire five, and you give them great paychecks.
- 5:32
But for the majority of companies, we don't have the luxury. You cannot afford to make one bad hire, and you can lose up to twenty to sixty K in that bad hiring process.
- 5:43
And you definitely can't afford to hire someone who doesn't know how to ship AI products.
- 5:49
So look, the future is coming for us whether we like it or not. Uh, Mark Zuckerberg says that AI will handle mid-level engineering work by later this year. TechCrunch is also reporting that we're wiping out engi- entry-level engineering jobs.
- 6:05
But that doesn't mean fewer engineering jobs. They're just very different engineering jobs, jobs that require creati- creativity, collaboration, and the ability to work according to business judgment and not just code, jobs that require working with AI instead of being replaced by it.
- 6:26
So we're currently working with select design partners. If you're interested in thinking about this process or learning more or just want to chat about how you're thinking about hiring, let me know.
- 6:36
My details are on the screen, or just come find me at the event. Thank you. [upbeat music]