# Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

Diego Rodriguez, Krea

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/h5ItAJuB3Fc-perceptual-evaluations-evals-aesthetics) · [Video](https://www.youtube.com/watch?v=h5ItAJuB3Fc)

## Perceptual Evaluations: Evals for Aesthetics

A malformed hand, perceptual compression and a fragile image metric expose the gap between measuring visual properties and judging whether an image works for a person.

### The hand a model almost accepts

[0:16](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=16s)

An AI-generated hand has distorted fingers and joints. To a person, the anatomy looks immediately wrong. Diego Rodriguez, Krea’s co-founder, opens with a simple test: ask **o3** what it thinks of the image. In Rodriguez’s account, o3 spends 17 seconds on a tool-assisted assessment involving Python and OpenCV. It notices melting joints, yet largely accepts the hand as natural. The elaborate inspection misses the force of the immediate human reaction.

![Chat screenshot showing hands with distorted fingers, the question “What do you think of this image?”, and “Thought for 17 seconds.”](https://ai.engineer/api/talk-essay-media/9bde80ebf7b15140e325bee7cb5eab17f60c7c20573731cc92066804b346112f)

An unnatural-looking hand image accompanies the prompt “What do you think of this image?”

This is one anecdote, not a benchmark of the model’s visual abilities. But it poses a useful evaluation problem: **recognizing visual features is not the same as judging how an image appears to a person**. Models learn from human-produced data and human preferences, while the data itself reflects what people can perceive and choose to record. Why does that training still leave such a gap in judgment?

### Which information reaches the recipient?

[2:14](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=134s)

Rodriguez approaches the problem through Claude Shannon. Shannon’s early work on switching circuits and his later communication theory are distinct contributions; the relevant paper here is [A Mathematical Theory of Communication](https://www.princeton.edu/~wbialek/rome/refs/shannon_48.pdf). Its framing matters: information travels from a source through a channel to a destination, with noise potentially entering along the way. The system exists to communicate something to a recipient.

For generative media, that framing connects representation to perception. Rodriguez draws a conceptual resemblance between communication diagrams and neural networks or variational autoencoders, rather than claiming they are equivalent systems. Compression supplies the more concrete connection: a representation can lose information in transit while preserving what matters to the recipient. That makes the recipient’s sensitivity part of the engineering problem—and, eventually, part of evaluation.

### Deleting information without changing what people see

[4:09](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=249s)

JPEG makes this relationship tangible. Human vision is more sensitive to fine brightness detail than to fine color detail, so an image representation need not preserve both at the same resolution. Alongside this explanation, Rodriguez shows an illusion in which regions A and B have the same color but appear different until the reveal. The illusion illustrates how context affects appearance; the compression example exploits a particular difference in visual sensitivity.

The demonstrated sequence separates the image into components before reducing information:

1. **Separate brightness and color.** Convert the RGB representation into components that distinguish brightness from color differences. The slide labels these `Y`, `Cb` and `Cr`.
2. **Downsample the color components.** Reduce their spatial resolution while retaining the brightness detail to which vision is more sensitive.
3. **Reconstruct the image.** Combine the components again and compare the result with the original.

The pixelated letter R makes the separation visible: the brightness component and the two color components carry different parts of the same image.

![A yellow and purple pixelated R above three component grids labeled Y, Cb, and Cr.](https://ai.engineer/api/talk-essay-media/96601a928f5047554f95271cbbc8076f3feab4b58380a09d5add5d3b397956bd)

A pixelated letter R separated into Y, Cb, and Cr components.

Rodriguez describes the displayed reconstruction as looking the same with roughly 50 percent less information. That figure belongs to this example, not to a universal JPEG file-size guarantee. The essential result is that information can be removed without producing a difference the viewer readily notices.

Other compression steps do different work. Huffman coding encodes symbols efficiently; it is distinct from deliberately discarding less noticeable visual information. Rodriguez then extends the perceptual principle to MP3: what can listeners hear, and what can be removed without a noticeable loss? Video adds opportunities to reduce redundancy across time. His MP4 analogy is shorthand: [MP4 is a container format](https://www.loc.gov/preservation/digital/formats/fdd/fdd000155.shtml), while the audiovisual codecs inside it perform the compression. Across these examples, human perception helps determine which losses are acceptable.

### Compression enters both the dataset and the metric

[5:58](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=358s)

Studying audiovisual systems engineering made this limitation concrete for Rodriguez. He could delete information in code, render the image again and still see the same thing. Knowing that data had disappeared did not make the difference perceptible. The exercise turned an abstract claim about the limits of the senses into something he could directly experience.

Now move that observation into a training pipeline. Internet images have often already been compressed. Before a model encounters them, their representations contain decisions about what human viewers are unlikely to miss. Rodriguez asks whether training on that material passes some of our perceptual limitations into AI. It is a question about the consequences of data selection and representation, not a demonstrated causal result.

Evaluation introduces another mismatch. Rodriguez points to a JPEG example associated with [clean-fid](https://github.com/GaParmar/clean-fid): four displayed images appear essentially alike to him, while the FID results react strongly to compression artifacts. **Fréchet Inception Distance (FID)** compares image distributions through image features; it is not an aesthetic rating of an individual picture. The distinction matters because a distributional difference can be real without being the difference a human evaluator cares about.

The lesson is not that JPEG must always worsen FID. Its effect depends on the reference distribution and the evaluation pipeline. Rather, a metric can be sensitive to changes that barely affect perceived quality. Before using such a score to select a generative model, the practical question is whether its sensitivity matches the judgment the selection is supposed to serve.

### An incorrect clock can be successful art

[7:42](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=462s)

It is tempting to evaluate what is easy to specify: prompt adherence using **[CLIP](https://github.com/openai/CLIP)**, object counts, or whether an object is blue or red. These are useful but limited targets. Image–text matching addresses correspondence between an image and language; counting and color checks address particular visible properties. None alone establishes whether an image succeeds aesthetically.

Rodriguez then turns to art with an unusual clock and sky. A literal realism check could treat both as generation failures. In an artwork, however, distortion may be intentional and central to the meaning. The viewer’s interpretation depends on what the author is communicating, not just whether the depicted objects conform to ordinary physical appearances. **A departure from realism can be the point of the image.**

That changes the objective for a creative tool. Krea’s purpose, as Rodriguez describes it, is to help artists and other creatives express themselves through imagery and video. An evaluation system that penalizes intentional departures from realism can work against that purpose. Expressive usefulness belongs in the definition of success, even when it is harder to specify than an object count.

### From predicting cars to anticipating traffic

[9:05](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=545s)

The discussion then widens from evaluating a capability to understanding its consequences. Rodriguez recalls an analogy from his friend Cheng Lu, whom he identifies as working at Midjourney: predicting the car in a world of horses is comparatively straightforward. Keep the vehicle, replace the horse’s energy with an engine, and the basic idea is there. Predicting traffic is harder. Traffic emerges when many people adopt and use the invention together.

What is the equivalent of traffic for AI? Rodriguez asks engineers, researchers and founders to look beyond isolated capabilities toward the changes those capabilities enable at scale. Converting `JSON` to `YAML` is his example of a narrow task that can absorb attention. He concedes that such tasks matter; the concern is that measuring them can leave the larger consequences unexplored.

### Translation changes who can work together

[10:19](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=619s)

The Tower of Babel supplies the next analogy. In Rodriguez’s retelling, incompatible languages prevent people from coordinating and completing their shared project. He compares that breakdown to infrastructure meetings where arguments over Kubernetes and other choices prevent anything from getting built. Communication failures can block collective action even when the participants have the necessary technical abilities.

Increasingly capable translation changes that constraint. Rodriguez describes translation as solved to a high degree, rather than perfectly solved, and asks what follows when people can speak their own languages while still communicating with one another. His example is already practical: he uses translation to provide customer support to Japanese Krea users despite not speaking Japanese fluently. It extends direct support from a founder to people he otherwise could not serve in that way.

The consequence reaches beyond exchanging factual instructions. Better communication also helps people convey opinions and understand one another’s preferences. That brings the discussion back to aesthetics: if a tool is meant to support expression, it needs to understand more of what a person means by a successful result.

### Evaluate what the evaluation captures

[12:26](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=746s)

A cat image exposes several different questions that an evaluation might collapse into one:

| Criterion | Question |
|---|---|
| Count | Are there four cats? |
| Color description | Is this cat blue or teal? |
| Personal preference | Do I like this particular blue? |

A correct count does not settle the hue, and an accurate hue description does not establish whether the viewer likes it. Preference adds a target beyond descriptive accuracy.

**Evaluating an evaluation means asking whether it captures the intended person’s judgment.** Rodriguez wants metrics that account for his opinion, rather than assuming one score exhausts what makes an image good. His reference to being a visual learner expresses that personal perspective. The broader design requirement is to account for how people perceive images and for the nature of the data used to train the system—including JPEG artifacts already present in internet imagery.

### The research priorities behind the product

[13:30](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=810s)

Rodriguez closes the prepared talk by connecting these questions to Krea’s research agenda. He says the company achieved its presented progress with eight people and had grown to twelve at the time of the talk.

The work he invites people to pursue includes aesthetics research, hyper-personalization and scaling real-time generative models across image, video, audio and 3D for users around the world. These are research and hiring priorities, not a deployment architecture demonstrated in the talk. They make the evaluation problem commercially concrete: a creative system must support many kinds of media while responding to individual tastes.

### Learning acceptable variation from examples

[15:10](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=910s)

In the Q&A, Rodriguez restates the audience’s question as a search for perceptually aware metrics. Can a useful metric remain nearly unchanged when JPEG introduces artifacts that barely affect the viewer’s experience? He says such metrics have precedents in traditional encoding techniques, then turns from existing measures to something that could be learned.

The proposed approach is to train a classifier or continuous scoring model using examples of the intended judgment. Show it five images and label all five as good, even though they contain different artifacts. The acceptable variation need not be limited to JPEG compression. Those examples would give the evaluator evidence about which differences should not disqualify an image. This is a proposal for learning the criterion, not a reported result from a validated evaluator.

The ending returns to the opening hand from another direction. People can often recognize whether an image works before they can articulate a complete rule for it. Rodriguez’s wager is that this difficulty of specification makes the problem suitable for machine learning: examples can communicate a judgment that a checklist struggles to express. The task is to learn what people mean when they recognize a good image on sight.

## Resources

- [clean-fid](https://github.com/GaParmar/clean-fid) — Reproducible FID computation and examples showing how resizing and JPEG compression affect evaluation.
- [A Mathematical Theory of Communication](https://www.princeton.edu/~wbialek/rome/refs/shannon_48.pdf) — Shannon's foundational paper on communication, information, channels and noise.
- [A New Krea](https://www.krea.ai/blog/new-krea) — Rodriguez and Perez describe Krea's creative-tool philosophy, company growth and research priorities.
- [OpenAI CLIP](https://github.com/openai/CLIP) — Official code and examples for matching images with text and performing zero-shot image classification.
- [Learned Perceptual Image Patch Similarity](https://richzhang.github.io/PerceptualSimilarity/) — Research on learned image similarity, with human judgments, LPIPS implementation links and a dataset.

## Slides

### Event sponsors

[0:10](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=10s) [AWS innovation partner and platinum sponsor logos on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/754e34bbd60ed061c5e53c1696244fd25e8161a4e85ca7f82b115ba5883b88ac.webp)

INNOVATION PARTNER

aws

PLATINUM SPONSORS

Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Perceptual Evaluations

Krea co-founder Diego Rodriguez introduces how his team thinks about evaluations that account for human perception, opinion, and aesthetics.

[0:30](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=30s) [Title slide introducing Diego Rodriguez from Krea.](https://assets.ai.engineer/sites/aiecode2025/sha256/c3e221408795bb0e03ff11def5b4c70b210520625fc3338fef402367ac411f1a.webp)

Perceptual Evaluations

Diego Rodriguez from Krea
(@asciidiego - @krea_ai)

### Evaluating an image of hands

Rodriguez asks o3 to assess an AI-generated image with distorted hands. He says the model called it mostly natural, contrasting with people’s immediate reaction.

[0:49](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=49s) [A chat prompt asks what the model thinks of an image showing distorted hands.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d4fb0f4eedc9ead3a6f1b8a715497d0f586ce448a4fd42f2800a456eb1dcc7a.webp)

What do you think of this image?

Interface text:
Thought for 17 seconds

### A bit of history

Rodriguez begins a brief history with Claude Shannon and information theory.

[2:15](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=135s) [An orange slide reads “A bit of history.”](https://assets.ai.engineer/sites/aiecode2025/sha256/d6cc2918147d48bdfba37ec1583297c951584b0e789e08551c0588887eca4ee4.webp)

A bit of history

### Claude Shannon

Rodriguez introduces Shannon’s foundational work in information theory and emphasizes its focus on communication.

[2:22](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=142s) [An orange slide reads “Claude Shannon.”](https://assets.ai.engineer/sites/aiecode2025/sha256/7b7c348c79d7924bf3945ea7736a60f15df77a548cd0a08d293e602f41c79bbf.webp)

Claude Shannon

### General communication system

A message passes from source through transmitter to receiver and destination, with noise entering the signal. Rodriguez sees a parallel to neural networks and connects compression to evaluation.

[3:09](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=189s) [Diagram showing an information source, transmitter, noisy channel, receiver, and destination.](https://assets.ai.engineer/sites/aiecode2025/sha256/e666964a45cb559678a0db200a0e2d842b9338fc0755d1e01bbc76da8a7ceae6.webp)

INFORMATION SOURCE
TRANSMITTER
RECEIVER
DESTINATION

MESSAGE
SIGNAL
RECEIVED SIGNAL
MESSAGE

NOISE SOURCE

Fig. 1—Schematic diagram of a general communication system.

### Compression

Rodriguez introduces compression as a way to think about evaluation, with JPEG as an example.

[3:59](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=239s) [Title slide reading Compression.](https://assets.ai.engineer/sites/aiecode2025/sha256/713b39177d6f3ae2a4310f436932efef9492cb28c7de6d35c10bf0724c23eb31.webp)

Compression

### Checker-shadow illusion

The cylinder’s shadow makes squares A and B appear different, although they are the same color. Rodriguez uses the illusion to introduce how perception relates to image compression.

[4:12](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=252s) [A cylinder casts a shadow across a checkerboard with squares A and B marked.](https://assets.ai.engineer/sites/aiecode2025/sha256/7465f4673cd1b2eba1e2a9dd8ca473c4cd62b5a3f23484c4cb622dbffb196b83.webp)

A
B

### Checker-shadow comparison

A connecting strip reveals that squares A and B are the same color, though the shadow makes them appear different.

[4:29](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=269s) [The checker-shadow image with a connecting strip between squares A and B.](https://assets.ai.engineer/sites/aiecode2025/sha256/6571ed1287c4a865ae46ff1b98742071f5d321e4e32387da367bd8591b49b037.webp)

A
B

### RGB color space

The RGB cube illustrates a color space used to represent images on computers. JPEG can convert it to a space that separates brightness from color, then downsample the color channels.

[4:36](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=276s) [A three-dimensional cube illustrating colors in RGB space.](https://assets.ai.engineer/sites/aiecode2025/sha256/484d08b23725724bc1d293db126c571d086428329374d3cd297f32fd0d196223.webp)

RGB color space

### RGB color space

The diagonal through RGB space tracks brightness, from dark to light. JPEG can separate brightness from color and downsample color channels, which people are less sensitive to.

[4:42](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=282s) [A diagonal sequence of cubes progresses from dark to light under the heading RGB color space.](https://assets.ai.engineer/sites/aiecode2025/sha256/b6f1b1238f2e4c878331b6f40ad0352a1a8ffcee04929eb287d8f8a5ace22a1f.webp)

RGB color space

Interface text:
AIE
World's Fair
Microsoft
smol ai

[4:47](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=287s) [A diagonal sequence of cubes increases in brightness.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab3974cb39bcfbf2c34ca0e87dedfecbc452345d3f32ab33a362c9afff3793db.webp)

RGB color space

Interface text:
AIE
World's Fair
Microsoft
smol ai

### Brightness and chroma

JPEG separates brightness from color, then downsamples the color channels because we are less sensitive to color detail.

[4:50](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=290s) [The labels Y, Chroma blue, and Chroma red appear on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/f41c235db9717bc5272d0520b48c1e0ef9901081f0c82477282f41aa3117ec45.webp)

Y
Chroma blue
Chroma red

Interface text:
AIE
Microsoft
smol ai
World’s Fair

### Color and brightness visualization

A column of brightness levels rises through a plane of colors. Rodriguez explains that JPEG can separate brightness from color and downsample color channels, which people are less sensitive to.

[4:54](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=294s) [Colored cubes form a horizontal plane, intersected by a vertical column of cubes at different brightness levels.](https://assets.ai.engineer/sites/aiecode2025/sha256/e558da87fc864c9ad7a677544daf7c1ad1b4cfc79bfe8f2061d1796efacf9d23.webp)

1.00

Interface text:
i
AIE
World’s Fair
Microsoft
smol ai

### Image components

The pixelated R is separated into brightness and color components. Its shape remains visible in Y and Cb, while Cr appears uniform.

[5:03](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=303s) [A pixelated letter R appears above three component grids labeled Y, Cb, and Cr.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7103f20eed513180cde6a72398b5bd7c515a46d7e42b75f0d71f8a5c43272f9.webp)

R

Y   Cb   Cr

### Original image and 4:2:0 subsampling

The rose with downsampled color looks much like the original. The speaker says the image uses 50% less information.

[5:13](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=313s) [Two images of a red rose are labeled Original image and Subsampling 4:2:0.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3ed8915ff91af41a7a2970c4d8788a2a51641959793757b654dba150bacd3f9.webp)

Original image
Subsampling 4:2:0

### Plus other stuff

Rodriguez mentions Huffman coding, then extends the perceptual idea to what people can hear in MP3 and across time in MP4.

[5:26](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=326s) [The words Plus other stuff appear centered on an orange background.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e581fda8764d2df997e6c754b72eb8bdf22221e41ed57ee77abc56f6b739788.webp)

Plus other stuff

### But this made me think about myself

Rodriguez deleted image information and re-rendered it, yet saw no difference. The experience made the limits of his senses tangible.

[5:57](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=357s) [Orange slide reading “But this made me think about myself.”](https://assets.ai.engineer/sites/aiecode2025/sha256/668ffd32156088d00dd3c39c1f8a2b2447524b6fbff3b615e0e547b475413916.webp)

But this made me think about myself

### Human (AI too?) Perception

Rodriguez asks whether training on compressed internet images could pass human perceptual limitations on to AI.

[6:31](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=391s) [Orange slide reading “Human (AI too?) Perception.”](https://assets.ai.engineer/sites/aiecode2025/sha256/eb53943a452d38b841e78097591c506b5cbc1b40e83a711ace6c36f85c55ce92.webp)

Human (AI too?) Perception

### Things that are easy to measure

Rodriguez questions the focus on easy-to-measure diffusion model metrics, from CLIP prompt adherence to counting, color, and spatial relationships.

[7:42](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=462s) [Slide lists prompt adherence with CLIP, object counting, color attribution, and spatial reasoning as measures used for diffusion models.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6199b425e9cf46d4fddba180120a1bb6fc7280921f38a80afebbe4c6111aa46.webp)

With diffusion models and in academia we are sometimes focused on measuring "things that are easy to measure" such as

1. General prompt adherence with CLIP
2. Object counting (are there 4 cats?)
3. Color attribution (are the cats blue?)
4. Spatial reasoning (is the cat to the right of the desk?)

Interface text:
AIE
Microsoft
smol ai
World's Fair

### Melting clocks

Rodriguez uses melting clocks to show how image metrics can mistake artistic choices for errors and miss meaning a human viewer can grasp.

[7:59](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=479s) [A surreal landscape painting with soft, melting clocks draped over a branch and other forms.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae70d36d2924f28e9e711ceb04f04e1b3216289fd56dd856f8a575f10f5581e8.webp)

AIE
aws
World’s Fair

### Tower of Babel

Rodriguez uses the Tower of Babel to illustrate how language can obstruct coordination, then describes supporting Japanese-speaking Krea users despite not speaking Japanese himself.

[9:07](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=547s) [A painting of a large tiered tower rising over a landscape.](https://assets.ai.engineer/sites/aiecode2025/sha256/b219f57a77de5fe1acc629544e44bdbc1463c2a076a907accd0cd37fff429842.webp)

Interface text:
AIE
World’s Fair
Microsoft

### Let's eval our evals

Rodriguez argues that aesthetic evaluations should account for personal opinions and human visual perception, as well as artifacts in training data such as internet JPEGs.

[12:44](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=764s) [The slide calls for evaluations that account for personal opinions and visual learning.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8056da2888fc3b42f45a92c3fd4e28d0bfc7dde50b6c5c6bf2083cf72205a7d.webp)

Let’s eval our evals;
take into account my opinions;
And consider I may be a visual learner

AIE
World’s Fair

### Company metrics and applications

Krea reports more than 25m users and over $83M, achieved with about eight people. Rodriguez says the team has grown to 12 and invites applications for generative media research.

[13:34](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=814s) [Company metrics and customer logos appear beside an invitation for priority applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a597f16d060ced443a66ac64b414b571204f13ef6f3b94dd9d2cd47a5fe9af6.webp)

>25m
>83$M
~8 ppl

High-priority applications
d+wf@krea.ai

PIXAR
Microsoft
SAMSUNG
perplexity
COLDPLAY
LEGO
Foster + Partners
BOSS
HUGO BOSS
lululemon

Interface text:
AIE
Microsoft
smol ai

[13:47](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=827s) [Closing slide showing three company metrics, customer logos, and a contact for high-priority applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7e31cfd12b85ae91f07d35fbcfd50e757fe5b0f3a6e0eceed0bedd9622d9e6a.webp)

>25m
>83$M
~8 ppl

High-priority applications
d+wf@krea.ai

PIXAR
Microsoft
SAMSUNG
perplexity
COLDPLAY
LEGO
Foster + Partners
BOSS
HUGO BOSS
lululemon

Interface text:
AIE

## Full transcript

[0:00](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=0s) [on hold music] Okay, so hello everyone.

[0:16](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=16s) Uh, my name is Diego Rodriguez. Uh, I'm the co-founder of Krea, a startup in the AI space like many others, uh, in particular generative media, multimedia, multimodal, and all the, uh, buzzwords.

[0:30](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=30s) But, uh, I, I come here mainly to tell a story about,

[0:35](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=35s) hmm, how we think about evaluations when we have to take into account human perception and human opinion and aesthetics, uh, into the mix, right? So I'm gonna start with a very simple story, is like I put an AI-generated image of a hand, obviously it looks horrible.

[0:56](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=56s) Uh, and then I ask o3, "What do you think of this image?" Then he thought for seventeen seconds, obviously tool calling, does Python analysis, OpenCV goes crazy. And then after he charges me a few cents, it's like, "Oh, just a couple of melting joints there."

[1:11](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=71s) It's like, it's mostly natural, but like... [chuckles] And it's like, okay, we have like what many people claim is basically AGI, and it is completely unable of answering a very simple question.

[1:23](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=83s) And, and like th-does it... That's a surprising thing if you think about it because we as humans, uh, when people see that image, it's like we just react so naturally, right?

[1:36](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=96s) Against that it's like, "Ugh, what is that?" Like, that, that, that's not natural. And, and I feel like that's precisely what AI models are being trained on. A, on human data, right?

[1:45](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=105s) Uh, uh, second on human preference data. Uh, and, and third, like in a way limited by the data that we humans ba-ba-based on our preconceived notions and perception and all of that.

[2:01](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=121s) So that's what this talk is about, about why, what, um, what can we do better, and honestly to ask ourselves some questions that I think are not being asked enough in the field.

[2:14](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=134s) Um, cool. So a tiny, tiny bit of history. Uh, there's, uh... We all know about Claude, uh, Claude Shannon that is, the [chuckles] the father of information theory. Uh, and according to many, his master thesis is one of the most important master thesis in the world, where he laid foundations for digital circuits and then eventually communication and all...

[2:39](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=159s) And, uh, to a degree, we can say that even LLMs, uh, nowadays, right? If we fast-forward. Um, and I wanna,

[2:49](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=169s) I wanna focus on the infor-- a-all the, like the fact that we

[2:56](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=176s) call his work foundational in information theory, while when he published it, it was actually called ma- like Mathematical Foundation for Communication Theory, and he was always focused on communication.

[3:09](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=189s) Um, there's this image, uh, appears on that work, uh, and it's all about, okay, this is the source, this is the channel, this is destination, there can be some noise there.

[3:21](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=201s) Um, and as a, well, as a founder of a company that is focusing on, on media,

[3:29](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=209s) to me it's interesting to realize like these parallels between classic information theory and communication. Uh, let me see. Did I put the image? Let's see. Well, if you... I didn't put the image, but if you have any context around variational autoencoders or neural networks and whatever, you, you can squint and be like, "Oh, is that a neural

[3:51](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=231s) network?" Right? Um, and in the context of information and communication, I wanna talk about how compression is going to be, uh, related to how we think about evaluation, right?

[4:09](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=249s) And I'm gonna talk in, for an example, for, for example, on JPEG. Um, JPEG exploits

[4:18](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=258s) like human nature in the sense that we are very sensitive to brightness, but not so much to color, and this is a i-illusion that also talks about that, where A and B is actually the same color, but we are basically unable to perceive it until we do this, and then suddenly it's like, "Oh, really?"

[4:33](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=273s) Um, and it's kinda like, what's going on there, right? Um, and so JPEG just does the same thing where, okay, we have RGB color space to represent images with computers.

[4:44](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=284s) Uh, we notice that there's a diagonal that represents the brightness of the images. We can change into a different color space, uh, that separates color versus brightness, and then we can downsample the channels around color because we are actually not even that sensitive to it, so we can remove that or parts of it.

[5:04](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=304s) And then, uh, once we do that, this is, uh, an image where we can see the, uh, brightness and color components separated. Once we downsample, we can try to recreate the image, and this is an example of like basically original image and then the image with the downsampled color looks the same to us, uh, and the image

[5:23](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=323s) is like fifty percent less information, right? Mm.

[5:28](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=328s) And other stuff. There's Hussman-- uh, Huffman coding and more stuff, but like the point is the same, right? And then the thing is if you exploit the same for audio, like what can we hear?

[5:42](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=342s) What can we not hear? Well, you, you do the same and we have MP3. And then if you do the exact same thing across time, well, congrats, now you have MP4.

[5:50](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=350s) It's like it's all this principle of like let's exploit how we humans perceive the world, right? Um,

[5:58](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=358s) but this made me think about myself because I studied audiovisual systems engineering, which is engineering around all of these microph- how microphones work, how speakers work. And it was just interesting to me that I was coding.

[6:13](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=373s) I start deleting information. I know for a fact that I'm deleting information, yet-- And then I re-render the image, and I see the same. It's like, like philosophers always tell you about like, "Oh, we are limited by our senses," but like this is the first time that I was like, I'm seeing it, right?

[6:28](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=388s) Like, I am not seeing the difference. Um,

[6:33](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=393s) but then if all-- i- if, if a lot of our data is the Internet, right? Like we're scraping data from the Internet, bunch of those images are also compressed, right?

[6:44](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=404s) Like, are we taking into account that perhaps our AIs are limited too because we're kinda like cont-- like

[6:53](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=413s) we have some sort of contagion going on of our flaws into the AI? Um,

[7:00](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=420s) and then it gets more tricky because, for instance, uh, this is a, just a screenshot I took from a paper. I think it's called clean-FID. And FID scores, for all of you who don't have context, is one of the standard metrics used for, uh, how well, for instance, diffusion models are, are reproducing an image.

[7:20](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=440s) But then you start adding JPEG artifacts, and the score is like, "Oh, no, no, no. This is horrible. Horrible image." And it's like perceptually, the four images are basically the same, yet the FID score is like, "No, no, no.

[7:31](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=451s) This is really bad." So then it's like, why are we using FID scores or metrics along those lines to deci- to decide how this generative AI model is good or bad, right?

[7:42](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=462s) Um, so the thing is sometimes I feel like we are focused on measuring just things that are easy to measure, right? Like prompt adheren- adherence with CLIP. Hmm, how many objects are there?

[7:55](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=475s) Uh, is this blue? Is this red? Et cetera.

[7:59](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=479s) But what about here? Oh, it's like, oh, no. Really bad, really bad generator because that's not how clock looks and the sky, that makes no sense. And it's like, okay, how, h-

[8:13](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=493s) not only are we limiting our AIs by our human perceptions, uh, on top of that, we forget about r- the relativity of metrics, right? Like, uh, no, actually this is art, and this is great.

[8:29](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=509s) And, and, and, and there's sometimes meaning behind the work that is not-- Like, it is conveyed in the image, but only if you're human. You, you get it, right?

[8:39](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=519s) Like, oh, this is what the author is trying to tell me. But I feel like the metrics don't show that. And kinda like

[8:46](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=526s) commercially and professionally, my job is kinda like, okay, how can we make a company that

[8:53](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=533s) allows creatives, artists of all sorts, uh, we can start with imagery, we can start with, with video, but to better express themselves. But how are we supposed to do that if this is kinda like the state of the art, right?

[9:05](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=545s) Um, then a friend of mine, uh, Cheng Lu, actually he works at, at Midjourney. He was, uh-- He has, um,

[9:18](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=558s) like he has great talks that you should all check, but he told me once, a little bit over a year ago, a quote that I just can't stop thinking about, which goes something like,

[9:30](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=570s) "Hey, man, if you think about it, like predicting the car back when everything was horses, it's not that hard." I was like, "What?" Like, "Yeah, it's not that hard to, to like...

[9:41](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=581s) Oh, cars are the future," and whatever. It's like we have s- we have a thing that goes like this. We have horses that make energy, so you swap the thing for the engine.

[9:49](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=589s) That's essentially a car, right? It's like, come on, how hard is that? He's like, "You know what's hard to predict? Traffic." [laughs]

[9:56](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=596s) Right? And, and then I just kept thinking about it. I was like, oh, man, like as engineers, as researchers, as founders, what are the traffics that we're missing now?

[10:06](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=606s) 'Cause I feel like everyone's focused on like, yeah, but you can, you can, I don't know, transform from JSON to YAML. I'm like, who cares? Like, dude, who cares? [laughs]

[10:13](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=613s) Like, or, or yes, it's important, right? Like, but

[10:19](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=619s) what kind of big picture are we all missing, right? Um, then he talks about, well, you know, the myth of the Tower of Babel, where i- i- i- in a nutshell, it's like

[10:36](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=636s) God-- Like we, we wanna go and meet God, and then he's like, "No, I don't want that. So instead, I'm just gonna confuse all of you, and then you're not gonna be able to, uh, coordinate.

[10:50](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=650s) Uh, and then you're all-- each one is gonna speak a different language, and then it's just g- basically gonna be impossible to keep the thing going." Which like reminds me of like standard infrastructure meetings w- with backend engineers.

[11:03](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=663s) It's like, "No, we should use Kubernetes." "No, we should use..." And it's like it's just all fighting and whatever, and nothing get to be-- nothing gets built. And I'm like, "Dude, God is winning.

[11:10](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=670s) Goddammit." Um, but then this makes me think about like

[11:17](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=677s) we are now in-- We just entered the age where you can have

[11:23](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=683s) models... Essentially, they solve translation, right? Or they solve it to a very high degree. So, so what happens now that we, that we can all speak our own languages, yet at the same time communicate with each other?

[11:39](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=699s) I'm already doing it. For instance, I do sometimes customer support manually for Krea, and I literally speak Japanese with some of my users, and I don't speak Japanese. I learn a little bit, but I don't speak it.

[11:51](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=711s) And, and, and like I'm now able to provide an excellent founder-led, whatever that means, uh, customer support level to a country that otherwise I would be unable to do, right?

[12:04](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=724s) And- And so I invite us all to think about what that really means, um,

[12:15](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=735s) because this for, for instance, means that we can now understand better or transmit our own opinion better to others. And

[12:26](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=746s) on the previous point that I was talking about with the art,

[12:30](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=750s) that's kinda like an opinion, right? Like, evals are not just about, are there four cats here? It's about this cat is blue, and it's like, yeah, but is it blue or is it teal?

[12:41](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=761s) What kind of blue? And I don't like this blue, and all of that. Um, so

[12:48](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=768s) like, in a nutshell, it's like how, how do we eval our evals, right? Like, from my opinion, like from my opinion, this is bad. Um, then I want metrics that take into account my op- my opinion too, and then it's like, okay, consider myself I may be a visual learner.

[13:05](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=785s) What that means is, like, maybe your eval should take into account how we humans perceive images, right? So, and, and, and also the, the nature of the data, such as, oh, it's all trained on JPEG on the internet, so take into account the artifacts, take into account, uh, like all of these while,

[13:30](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=810s) while training your data. Um, okay, I guess mandatory slide before the thank you. Uh, bunch of users, bunch of money. We did all of that with eight people, now we're 12.

[13:41](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=821s) And this is an email that I set up today for high priority applications, uh, for every- anyone who wants to work on research around, uh, aesthetics research, uh, hyper-personalization, scaling generative AI models in real time for multimedia, image, video, audio, 3D, uh, across the globe.

[13:59](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=839s) Uh, we have customers like those, and that's it. Thank you. Oh, here. [audience applauding]

[14:09](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=849s) Q&A? Okay, perfect. Any questions? Uh... [speaking]

[14:33](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=873s) Yeah, okay. There's many points there. Can you, like, reframe the question? Like...

[14:40](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=880s) So we were working on a proof of concept. So by showing them, uh, uh, if it's, uh, if the bias is, like, less visual, what would it do to the

[14:49](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=889s) Yeah. [speaking] Yeah. [speaking]

[15:10](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=910s) Yeah. So, so the question, like, in a nutshell is, like, are there, uh, perceptually aware metrics, right? Like-

[15:19](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=919s) Yeah.

[15:19](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=919s) Like, okay, you... I showed an example of FID score. It changes a lot with JPEG artifacts. Are those where it's almost like the opposite, bare- barely changes, uh, and the metric is still good?

[15:29](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=929s) Like, there are some, and many of these are used also in traditional, uh, encoding, uh, techniques. Um, but in a way, I'm here to invite us all to start thinking about those.

[15:43](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=943s) Like, like to-- We can actually train.

[15:52](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=952s) Right.

[15:52](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=952s) Like, we can train, uh... I mean, it's, it's ca- it's called a classifier, right? Like, or, or, or, or a continuous classifier. We can train so that it understands what we mean, and it's like, "Hey, I show you these five images.

[16:03](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=963s) These five images are actually all good." And then they can have all sorts of artifact, not just JPEG artifact. And this is exactly where machine learning excels, right? When it's all about opinions, and it's like, "Let me just show, and you will know," you know?

[16:16](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=976s) You know what? You, you will know when you see it. That's precisely the type of question that AI is amazing at. [outro music]
