AI Engineer World's Fair 2025

How to Hire AI Engineers When Everyone Is Cheating With AI

Read the talk

How to Hire AI Engineers When Everyone Is Cheating With AI

AI assistance changes what a coding interview reveals. Beth Glenfield proposes workplace simulations that expose collaboration, technical judgment, and the ability to ship useful AI products.

From a talk by Beth Glenfield

What does a successful coding interview reveal?

Are you using AI in recruitment—as an interviewer or a candidate? Are you competing with Google and Meta for talent? And does solving a LeetCode puzzle still tell you whether someone can build a great AI product? These are Beth Glenfield’s opening questions about technical hiring in 2025. For smaller companies, the problem combines an increasingly questionable assessment signal with competition for candidates they cannot easily outspend.

Looking back over the preceding eighteen months, Glenfield points to Cluely, which she describes as an AI cheating service from a former Columbia student. Glenfield reports a $5.3 million funding round and says Cluely was approaching $1 million in annual recurring revenue. The funding and revenue figures have different evidentiary weight: the revenue trajectory was a founder claim, not an audited result.

Glenfield also cites a 93% success rate for LeetCode Wizard in Google and Meta interviews, and says one in three interviews involves an AI assistant. The Leetcode Wizard figure should be read as a vendor-reported interview-test pass rate, not an independently established hiring or offer rate. Its current page dates its testing claim to August 2026 and cannot establish the historical 2025 testing conditions; the one-in-three figure remains an attributed claim.

Slide titled “Over the last 18 months...” lists $5.3M in Cluely funding, $1M ARR from AI interview assistance, a 93% LeetCode Wizard pass rate, and one in three technical interviews involving AI.Open full source frame
The slide lists four indicators of AI use in technical interviews.

A completed puzzle can reveal the strength of the candidate’s assistant more than the candidate’s engineering judgment. That is the assessment failure Glenfield identifies: the output remains visible, while the contribution being evaluated becomes harder to distinguish.

Meanwhile, employers increasingly expect engineers to use those tools. Glenfield quotes Sam Altman: “The tactical thing to do is to learn the best AI tools.” She then attributes to Marc Benioff a halt to Salesforce software-engineer hiring for 2025 and a claimed 30% productivity boost from replacing people with AI. That attribution does not establish replacement as the cause of the gain. Glenfield immediately questions the evidence: “I would like to see the data on that.” Her broader point is that expectations of engineering work have changed, even while interviews continue to test older proxies.

0:300:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:30 · section reference included

The capabilities puzzles overlook

Compensation is only part of the competition for an engineer. Brand recognition, career prestige, and perceived stability also matter. After layoffs, candidates may place particular weight on security. A Series A startup asking someone to choose it over Google’s AI division therefore has to offer more than a competitive interview process.

At the same time, the people a company needs for AI development may spend little effort optimizing for algorithm puzzles. Glenfield describes creative problem solvers and collaborative leaders who can work effectively with AI. Their relevant activity includes building AI tools, using AI libraries, contributing to open source, and understanding the business impact of their work. A puzzle score does not directly expose those capabilities.

2:312:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:31 · section reference included

Make the interview resemble the work

The alternative begins with a different observation: watch a candidate collaborate with AI teammates on an actual business scenario. Instead of using algorithm recall as the main signal, observe how the candidate delegates tasks, handles ambiguity, and responds when requirements change mid-sprint. AI becomes part of the assessment environment, so the interviewer can examine how the candidate works with it.

That environment also gives the candidate information. Showing how a team works early in the process can make its engineering culture tangible, giving a smaller company something specific to demonstrate beyond brand recognition. Glenfield introduces DevDay as her proposed way to build this kind of technical hiring process.

3:243:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:24 · section reference included

AI teammates make trade-offs observable

In Glenfield’s description of DevDay, candidates enter workplace simulations with AI agents that have different personalities:

  • The perfectionist brings an emphasis on getting the work right.
  • The pragmatist supplies a contrasting practical perspective.
  • The security expert brings security concerns into the discussion.
  • The junior developer needs extensive mentoring.

The candidate has to navigate these colleagues while making everyday engineering trade-offs. The work involves building features for the hiring company’s business domain, giving those decisions a concrete purpose.

The assessment unit becomes behavior during the work, not just the finished code. Glenfield identifies several signals to observe:

CapabilityEvidence in the simulation
AI collaborationHow the candidate works with AI teammates
Handling ambiguityHow the candidate proceeds with unclear requirements
Technical communicationDecisions explained in pull requests and ticket comments
MentorshipHow the candidate guides others
AdaptabilityHow the candidate responds when circumstances change

Pull requests and ticket comments matter because they make technical reasoning visible to teammates. Mentoring and adaptation extend the assessment beyond producing an individual solution: the candidate must help the surrounding team make progress as the situation changes.

Slide titled “What we believe you should measure” describes delegating, prompting and evaluating AI output; explaining technical decisions; handling changing requirements; and guiding junior team members. The speaker inset partly obscures the final judgment item.Open full source frame
Assessment criteria include AI collaboration, communication, adaptability, and mentorship.
4:264:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

Small companies have less room for a hiring mistake

Glenfield illustrates the resources of a large employer with a hypothetical funnel: interview 100 candidates, hire five, and offer large paychecks. This is an illustration of hiring capacity, not reported Google or Meta recruiting data. Most companies cannot absorb hiring mistakes with the same ease.

Glenfield puts the potential loss from a bad hiring process at 20–60K, without specifying currency or cost components. The practical requirement behind that estimate is more direct: a company hiring for AI development needs someone who can ship AI products. An assessment that rewards interview performance without exposing that ability leaves the central hiring risk unresolved.

5:205:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:20 · section reference included

Hire for the responsibilities that are changing

The closing argument returns to changing engineering roles. Glenfield cites Mark Zuckerberg’s prediction that AI would handle mid-level engineering work later in 2025, then invokes TechCrunch reporting about disappearing entry-level engineering jobs. The first is a forecast; reduced entry-level hiring does not by itself demonstrate that AI has eliminated an occupation.

White slide quoting “AI will handle mid-level engineering work this year,” attributed to Mark Zuckerberg, with the speaker inset at bottom left.Open full source frame
A slide attributes a prediction about AI handling mid-level engineering work to Mark Zuckerberg.

Glenfield’s expectation is that engineering jobs will become different rather than necessarily fewer. The responsibilities she emphasizes are creativity, collaboration, and business judgment alongside coding: deciding what to build and working effectively with AI to deliver it. That is the connection back to the interview design. A workplace simulation aims to expose those responsibilities while the candidate is exercising them.

At the time of the talk, Glenfield says DevDay is working with select design partners. She closes by inviting people to learn about the process or discuss how they approach hiring, pointing to her on-screen contact details and offering to talk at the event.

5:496:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:49 · section reference included

Resources

From the talk

Transcript download
Read the transcript and explore slides

Loading transcript and slides…