AI Engineer World's Fair 2025
ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski
Read the talk
Inside ComfyUI: From a Diffusion Graph to Reusable Creative Workflows
Follow ComfyUI’s workshop from local image generation through guidance, latent compression, regional control and the engineering needed to turn workflows into applications.
From a talk by Jedrick Kosinski and ComfyAnonymous
Before you start: No prior ComfyUI experience is required; familiarity with text-to-image generation will help when following the workflow examples.
What should a generative interface let you control?
How do you move beyond typing a prompt when you need control over an image’s structure, a model’s behavior or an entire generation pipeline? ComfyUI exposes that pipeline as a graph of connected nodes. Jedrick Kosinski and its original creator, ComfyAnonymous, introduce an open-source canvas for image, video, audio, 3D and text workflows. The graph provides a common working surface for local models and models reached through remote APIs.
Kosinski describes rapid support for new models through implementations redesigned for ComfyUI, including a goal of supporting releases on day one. Local execution spans NVIDIA, AMD and Intel hardware; closed-source models can be accessed through APIs. Community-maintained custom node packs extend the system beyond what the core team can implement.
A generated artifact can also carry its recipe. When an image or video contains intact workflow metadata, dragging it onto the canvas can restore the graph and generation parameters. That makes sharing an output a way to share an editable experiment. Kosinski says most workflows circulating from the preceding year and a half remained compatible with the version shown in the workshop.
Kosinski reports that, after two years, ComfyUI had 78,000 GitHub stars and ranked among the top 150 repositories. He also reports 3–4 million active users, 20,000 daily downloads, and 22,000 custom nodes from 3,000 public developers. These are workshop-time figures; activity windows and counting methods are not specified. He names Amazon, Apple, Tencent and Netflix as adopters and describes widespread use in visual-AI startup backends, without quantifying that broader claim.
The practical attraction is access to controls that a prompt box cannot expose by itself: depth maps, line art, masks and model-specific operations. A creative user can explore combinations interactively; a developer can automate a graph or extend it with custom nodes. Those roles overlap, and the open-source ecosystem lets users contribute a missing capability instead of waiting for the core team.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a personal experiment to Comfy Org
ComfyAnonymous dates the personal project to January 2023. Six months later, he joined Stability AI, where ComfyUI served as a tool for internal model experimentation. After a year there, he left in June 2024 and joined Jolin and Robin to form the company behind Comfy. The sequence helps explain the software’s emphasis on exposing model internals: it developed alongside hands-on experimentation with those models.
The team photograph comes from a climb up Mount Fuji, which ComfyAnonymous remembers as difficult enough not to recommend. Kosinski missed it after his flight to Japan was rerouted to Alaska for a day; he reached Tokyo only five hours before the others were due to wake up for the climb. The team closes this introduction with an invitation to people interested in open-source generative AI to visit the Comfy careers page.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the galaxy-bottle graph
The first live example is ComfyUI’s familiar galaxy-bottle workflow. Kosinski changes the seed to produce a fresh result, with generation running locally. The initial model is Stable Diffusion 1.5, or SD 1.5. ComfyAnonymous distinguishes its relatively weak base model from the fine-tunes that originally inspired his work: the base model makes a convenient fast demonstration, but it does not represent the image quality of newer models.
At the workshop, the presenters describe SD 1.5 as nearly three years old. Its age does not obscure the graph’s useful structure. ComfyUI splits a Stable Diffusion pipeline into a diffusion model, a text encoder and a variational autoencoder, or VAE, then connects those components to sampling and output operations.
| Graph component | Responsibility |
|---|---|
MODEL | Supplies the diffusion model |
CLIP | Supplies the text encoder |
VAE | Converts between image and latent representations |
| Sampler | Uses the model and conditioning to produce latents |
VAE Decode | Converts sampled latents into an image |
Save Image | Writes the image output |
Following these connections is more useful than treating the canvas as a collection of unrelated boxes. The sampler’s output is not yet the final visible image: it passes through VAE decoding before saving.
The presenters then replace the checkpoint with SDXL. Loading it and running the graph takes time on the laptop; disk access and local execution are visible constraints on the demonstration. The resulting bottle looks better to them, although they describe SDXL itself as about two years old. The useful comparison is structural: the model changes while the same basic pipeline remains recognizable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse the pipeline for video and sampling patches
Opening a video template makes that continuity more obvious. Kosinski says the selected template cannot run because roughly 60 GB of model files are absent from the laptop. That is a description of the missing files, not a VRAM requirement. The template uses Wan2.1, which ComfyAnonymous calls probably the best open video model available at the time. Its graph still uses a sampling node and VAE decoding, with settings appropriate to video.
ComfyAnonymous next adds a node that he describes as a CFG trick. It patches the sampling code’s guidance calculations, illustrating a second kind of reuse: a graph can change how sampling behaves without replacing the whole pipeline. A user can insert an existing node or write a new one to perform that modification.
Kosinski introduces classifier-free guidance, or CFG, through positive and negative conditioning; the negative prompt in this example targets text and watermarks. His subtraction explanation is an intuition for combining the model’s predictions under those conditions. The operation is on predictions during sampling, rather than on finished images or literal prompt strings. The visible graph shows the connected patch above the prompt and sampler nodes, with a purple bottle image on the right.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the two text-encoding branches actually do
An audience question about CLIP Text Encode leads into the division of labor inside the model. A diffusion model does not receive the prompt as raw language. A text encoder first turns it into embeddings, and the diffusion model learns to use those representations. In ComfyUI, CLIP is a historical name for the text-encoder interface: early Stable Diffusion models used CLIP, while later models can use other encoders under the same label. Depending on the model, conditioning commonly uses the final or penultimate hidden state.
Why are there two text-encoding nodes? They encode the positive and negative prompts used for CFG. In the setup being discussed, setting the guidance scale to 1 leaves positive conditioning alone. ComfyAnonymous tries that setting, but the first unexpectedly poor result is confounded by the patch node he had added earlier. He identifies that interference before continuing; the subsequent positive-only image is still described as chaotic and poorly defined.
The guidance calculation can be written in terms of a negative-conditioned prediction ε₋, a positive-conditioned prediction ε₊ and scale s:
For arrays or tensors of predictions, the same operation is compact in Python:
python
def cfg_prediction(positive_prediction, negative_prediction, scale):
return negative_prediction + scale * (
positive_prediction - negative_prediction
)
Increasing the scale extends the prediction farther in the direction from negative to positive conditioning. That explains the presenters’ description of pushing toward what the prompt asks for and away from what the negative prompt specifies. It gives a more defined result in their example; increasing guidance indefinitely is not a general quality rule, because guidance also changes the quality–diversity tradeoff.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why the sampler works in latent space
The next question concerns the VAE. ComfyAnonymous explains that an important source of Stable Diffusion’s efficiency is generating in a compressed latent representation. Earlier pixel-space diffusion models worked directly over the image’s pixels and were comparatively slow. With latent diffusion, the repeated sampling work happens on a smaller spatial grid.
For the Stable Diffusion architecture discussed here, the VAE downsamples each spatial axis by 8: a 512 × 512 × 3 image becomes a 64 × 64 × 4 latent representation. The fourth latent channel is a reminder that this is a learned representation, not simply a small RGB picture. During text-to-image generation, the graph samples in that latent space and then decodes to pixels. The dimensions explain the architectural reduction; they do not specify a measured runtime speedup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automating a workflow does not solve evaluation
Once generation is a workflow, it is natural to ask whether evaluation can become another step: can a node decide whether an output is a good image? ComfyAnonymous starts with the difficulty of defining good. A preference learned from one group of users can favor a style that another group does not want. In his account, adding user-preference data during training can make results worse for those other tastes.
The questioner suggests critique models or multimodal prompts that inspect an image. ComfyAnonymous recalls experiments at Stability AI that generated an image from a workflow and then asked another model to rate it. He says those experiments did not work particularly well. The exchange supplies a caution about the evaluation target, rather than a working automated judging recipe.
A separate question asks how to take a graph built in the UI, run it headlessly and scale it as an application. The canvas is an editor; the backend executes the workflow. ComfyAnonymous points to existing third-party inference and workflow-to-app services, without naming a particular provider. At the time of the workshop, an official hosted application-serving solution remained future work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remote models, 3D and the boundary of core support
Kosinski returns to the canvas to show API nodes, introduced in the month preceding the workshop. These spend paid credits to run generation remotely. He opens a template for Black Forest Labs’ FLUX.1 Kontext image-editing model. In the recording, Kontext is available through APIs, and the presenters say support is ready for an anticipated local release. The laptop has no internet connection, so this demonstration cannot run. Remote execution avoids local model execution but introduces a network dependency.
The template browser then moves to Hunyuan 3D. ComfyAnonymous describes a model that produces a voxel-like representation, followed by an algorithm that converts it to a mesh. He finds the generation impressive but the resulting mesh quality limited. The gallery shows example scenes and figures; it is not a live mesh-quality comparison. The required models are absent from the laptop, so this workflow is also inspected rather than executed.
Support for a modality can exist at different layers of the ecosystem:
- Local LLMs: available through community custom nodes, rather than core ComfyUI in the workshop’s version.
- Audio generation: supported, though ComfyAnonymous describes the available model as weaker than some proprietary alternatives.
- Text-to-speech: recently released models already have custom-node support; inclusion in core remains a separate decision.
For ComfyAnonymous, core integration should expose useful additional control—extra parameters or ways to manipulate the model. If it merely wraps an operation already handled well by a custom node, the benefit is less clear.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Virtual try-on and character consistency
An audience member asks about virtual try-on: provide a person or mannequin image and an image of a garment, then generate the person wearing it. ComfyAnonymous points to both local and API-based workflows and suggests trying Kontext first. His recommendation is tentative, and he warns that searchable workflows can age quickly as models improve. During the exchange, he clarifies the name as Kontext, with a K.
At this point in the recording, local Kontext execution is still contingent on the expected release. That historical distinction matters when following the workshop today: BFL subsequently released Kontext dev weights on June 26, 2025 under a non-commercial open-weight license, rather than an unrestricted open-source license. The offline template shown here remains an API-era demonstration.
Kosinski contrasts this with workflows he encountered at a Japan meetup. Older try-on systems could require a few dozen nodes: one model found the region to change, and another inpainted the garment into that mask. Newer edit models accept an image and an instruction directly. Masks still have a role, because they can constrain the operation when the model changes parts of the image that should remain untouched.
The next question asks how to preserve a character across a sequence of story images. ComfyAnonymous offers two paths: train a character LoRA, or use an edit model such as Kontext. He places the recent edit-model trend in the wake of GPT-4o image editing and recommends Kontext for this use at the time. He describes its character preservation as very strong, but the workshop does not run a consistency evaluation.
The proposed reference-image procedure is straightforward:
- Generate a character image that you are happy with.
- Supply that image as the reference for an edit model.
- Request the character in a particular scene or activity.
- Reuse the approved reference for the other scenes in the story.
The reference supplies identity information that would otherwise need to be reconstructed from text each time. This is the workflow ComfyAnonymous describes for the audience, rather than a sequence generated live on the disconnected laptop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a reference image is not enough
Kosinski expands the character example into a more controlled graph. A LoRA, or low-rank adaptation, modifies a model’s behavior. He describes assigning a character A adaptation to one image region and a character B adaptation to another, with separate strengths and potentially a schedule. A pair of prompt branches can thus grow into a larger graph that specifies where particular concepts should act. The workshop describes this regional-LoRA capability without identifying the node implementation; it should not be confused with simply loading two LoRAs globally.
The two approaches trade preparation for control:
| Approach | Preparation | Main attraction |
|---|---|---|
| Edit model | Reference image and instruction | Simple setup for an unfamiliar user |
| Subject LoRA | Training images and adaptation training | Can teach a subject the base model does not know |
Kosinski recommends Kontext to beginners because the instruction does much of the work. For subject LoRA training, Kosinski suggests roughly 10–30 images, compared with one reference image for the newer edit-model approach. That is his practical guidance for this discussion, not a fixed requirement for every LoRA training task.
ComfyAnonymous then explains the weight-level mechanism: instead of training the full model, train a small patch to its weights. That patch can represent a style, a character or multiple concepts. In the graph, the LoRA is chained into the model path so that downstream sampling uses the adapted model.
The available example is an anime-character LoRA prepared for a Japan presentation. It belongs to the SD 1.5 model family, and the laptop has no corresponding SDXL LoRA. The presenters consider using an anime checkpoint, while warning that the other available models could give poor results. They ultimately decide not to press Run. The canvas shows how to connect the adaptation, but no generated LoRA result is demonstrated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Night at the top, daytime at the bottom
The presenters browse ControlNet examples next. The template system provides example inputs and generated images in place of a fresh input demonstration. ControlNet is another way to condition generation more explicitly, but ComfyAnonymous speculates that improving edit models may reduce the need for some of these specialized control workflows. His point is that the usefulness of a technique can change even when the capability remains available.
They then open a regional-prompting workflow that applies different prompts to different parts of an image. Kosinski selects the default SD 1.5 model and fixes the seed. They expect it to work if it does not depend on missing input images, though ComfyAnonymous initially qualifies that at least half the workflow should work. The resulting discussion identifies the intended spatial division: night at the top and daytime at the bottom.
Regional conditioning turns location into part of the instruction. Instead of asking one global prompt to express both times of day, the graph associates each condition with an area. ComfyAnonymous believes this old workflow can still work with recent models because the underlying technique belongs to the diffusion pipeline. Once a new diffusion model is implemented, many older sampling and conditioning techniques remain usable. That compatibility is not demonstrated across newer models here, and some techniques become unnecessary or make larger models slower on constrained local hardware.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Local control includes responsibility for the output
Kosinski returns to the earlier hesitation about running the anime example. In the early image-generation ecosystem, community fine-tunes could substantially improve a base model, but their datasets could also produce content unsuitable for a conference screen. ComfyAnonymous offers a possible failure path: a slightly broken model might begin with a skin-colored blob and converge toward nudity. That is his explanation of the risk, not a diagnosed trace of the unexecuted example.
The practical rule is to review generated outputs before presenting them, especially when using unfamiliar community checkpoints. Local execution gives users freedom over the models and outputs they run, but it also leaves that review with them. Kosinski attributes some improvement in newer models to more constrained training sets; ComfyAnonymous emphasizes that better models make fewer random mistakes. Both connect that progress to more dependable prompt adherence.
The team reiterates hiring needs across backend, frontend, core model inference and cloud deployment. For users getting started, the two practical entry paths are a local GPU or API nodes. Kosinski reports that some early models have run on eight-year-old GPUs. ComfyAnonymous says models will usually run on a broad range of NVIDIA hardware, while warning that execution can be extremely slow. Being able to load and run a model is a different threshold from making it comfortable to use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control, extensibility and the cost of learning
Asked why someone should choose ComfyUI over alternatives, ComfyAnonymous argues for its depth of control, community support and extensions. He also identifies the main drawback: it can be difficult to get into. Kosinski agrees that an unfamiliar node canvas can be intimidating. He describes roughly 3,000 node packs in the ecosystem, alongside the previously cited custom-node and developer counts. The breadth that makes the system powerful also increases what a new user has to understand.
For an enterprise, Kosinski presents local operation and inspectable source code as data-control advantages. A team can examine the implementation and contribute fixes rather than depending entirely on an opaque remote service. He also claims leading inference performance and rapid adoption of new optimizations, though the workshop supplies no comparative performance measurements. These are the presenters’ reasons for choosing the platform, rather than a tested ranking of competing products.
For support, Kosinski points to Comfy Org’s Discord and the large body of YouTube tutorials. He also notes the emergence of paid tutoring. He treats that market as feedback for the product team: if users need to pay someone to make the software approachable, onboarding needs improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make local workflows reliable before building the cloud
The final questions turn from individual graphs to product architecture. Asked whether there is a published roadmap, ComfyAnonymous does not identify one. He describes official cloud workflow execution as work that has not really started yet. Before it can succeed, custom-node installation and management need to become smoother. The node interface will remain, but he expects an additional layer that lets users build a more traditional interface from a workflow graph. New model releases can change those priorities, so he offers a direction without delivery promises.
An audience member sketches the team product that could sit on top: one power user builds a template that preserves style, brand, character or design rules, while other users access it through an API or a restricted interface. Role-based permissions would expose approved parameters without allowing arbitrary graph edits. ComfyAnonymous endorses this direction, but places reliable cloud inference—with custom nodes working correctly—before those higher-level services. The proposed permissions and template controls are not presented as existing features.
Kosinski says a blog post about upcoming work is planned for the end of the event week. One proposed feature is subgraphs: combine several nodes into a single node, then double-click to enter the nested workflow. That would let a graph retain its internal detail while presenting a simpler outer surface.
The other foundation is dependency management. Custom node packs can request different Python packages, creating conflicts in the environment that executes the graph. Kosinski describes isolation or better compatibility reporting as ways to make those dependencies manageable. The workshop’s closing engineering priority is therefore local reliability: once workflows and their custom nodes become easier to run consistently on one machine, building a dependable cloud product becomes easier too.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and installation instructions for the node-based generative workflow system.
Model weights and documentation covering latent compression, text conditioning, training and limitations.
Official implementation and instructions for the Wan2.1 video-generation models.
Black Forest Labs introduces image-and-text editing and character-preserving generation.
Further reading
Example workflows that can be restored from images containing embedded workflow metadata.
Examples of inserting and chaining LoRAs in a ComfyUI workflow.
Workflows for applying different conditioning to different image regions.
The original paper explaining guidance through conditional and unconditional score estimates.
Tencent's public implementation for generating textured 3D assets.
A June 2025 proposal for stable custom-node APIs, dependency isolation and future distributed execution.
The June 2025 announcement explains nested workflows and the planned public beta.
Updates since the talk
The June 26, 2025 release announcement includes local inference resources and non-commercial licensing terms.
Read the complete timestamped transcript
- 0:14
All right. Uh, good morning, everyone. I am Jedrick Kosinski, and this is...
- 0:20
Yeah. Hello, I am, uh, known online as, uh, [REDACTED:username], the original creator of ComfyUI. [audience applauding]
- 0:30
And we are part of the Comfy Org, the organization that, uh, is in charge of ComfyUI.
- 0:38
So, uh, I guess now that I have a mic, I'll ask again. Who here has heard of ComfyUI?
- 0:44
All right. All right. This half of the room, very knowledgeable. Very nice. Very nice. Um, for those unaware, we are an open source note-based design canvas intended for, uh, generative AI purposes for, uh, multimodal, um, creative applications.
- 1:01
We support image, video, audio, 3D, text, and more, uh, generative AI models.
- 1:09
Um, ComfyUI supports the absolute bleeding edge of generative AI tech on day one. We have [REDACTED:username] here implementing it from not quite scratch, but it is redesigned from the original implementations.
- 1:24
Uh, we offer open source locally hosted models that support NVIDIA, AMD, and Intel hardware, and we also support closed source API accessible models that we, as the name suggests, just use an API to deliver to the user. [lip smack]
- 1:38
All of this functionality is also extendable with community-supported custom node packs, so anything we do not have the time to get to ourselves, the community does for us.
- 1:49
A big part of what makes ComfyUI special is the shareability of the workflows. Any image or video that was generated by ComfyUI has embedded metadata that lets you drag it back into the canvas and brings you the original workflow with all of the parameters that was used to generate it.
- 2:08
Uh, this sort of shareability and virality has really helped ComfyUI's traction. If you do a simple Google search on, uh, ComfyUI workflows, you will find pages and pages and pages of results from the past year and a half, most of which are still compatible with modern ComfyUI versions.
- 2:29
Um, in terms of pure numbers, uh, this sort of shareability and virality has, over the past two years, taken us to the position of top 150 most popular GitHub repos of all time with 78,000 stars.
- 2:49
Any comments, Comfy?
- 2:50
Uh, no.
- 2:52
All right. All right. [chuckles] Uh, with more of the traction numbers, we have three to four million active users. We have 20K daily downloads. We have 22,000 custom nodes made by 3,000 public developers that we enable in our ecosystem, and we've been adopted by Amazon, Apple, Tencent, Netflix, and more.
- 3:13
And pretty much any startup these days built around visual generative AI probably has ComfyUI working somewhere on their backend.
- 3:23
Um, why is ComfyUI popular? Um, it gives maximal control. You can go beyond prompts and interact with models that give you access to depth maps, line art, uh, masks, anything like that that is out in the space.
- 3:40
If it's out-- If it's open source, we probably either support it directly or the community has, uh, made it possible. Uh, we are an all-in-one platform for both exploration for creatives and automation for developers.
- 3:53
Sometimes those roles can also be switched, where developers want to explore tweaking models and seeing how things can be extended, and so we offer that as well through our custom node, uh, feature.
- 4:05
And because we are open source, we do not only depend on the output of the core team. We can trust the community to let us know anything they'd want us to work on and also make anything we do not have the time to work on on our team.
- 4:23
Any comments, Comfy?
- 4:25
Uh, well, I think, uh, yeah, I think we haven't shown the, the interface yet, so...
- 4:31
Yes. We have only shown one screenshot of the interface at the very start. We will, uh-
- 4:36
Yeah
- 4:36
... show that off [chuckles] as well. You wanna do it... Just do it later. [chuckles]
- 4:42
Yeah.
- 4:43
All right. Story behind ComfyUI.
- 4:44
Yes.
- 4:44
You wanna get into this, Comfy?
- 4:45
Yeah. The quick story is that, uh, basically, ComfyUI started as my own personal project, and then I... And then, uh, [lip smack] yeah. And then, which... Yeah. I started it in, uh, January, January 2023, and then
- 5:02
six months later, I was hired at Stability AI, so I spent one year at Stability AI. They were using Comfy for, uh, for more, uh, like experimentation with, uh, internal experimentation with the models.
- 5:18
And then I left Stability AI in, uh, June 2024, and then I joined up with, uh, Jolin and Robin, and, uh, we, uh, [lip smack] we made, uh, like the Com- Comfy, uh, company.
- 5:31
And, uh, yeah, that's, uh... Things have been, uh, going pretty well since then, so...
- 5:38
Yeah. Yep. And this picture was taken on the-
- 5:42
Yeah. We, we went, uh... That, that picture, we went on top of, uh, Mount Fuji, uh, which, uh, I don't recommend. It's, uh, ver- very, very difficult. But, uh, yeah.
- 5:55
But we did it, so yeah.
- 5:57
I, I lucked out, and my flight to Japan, uh, got rerouted to Alaska for 24 hours. So I landed in Tokyo Five hours before they were going to be waking up to go to Mount Fuji.
- 6:11
So I got to-
- 6:12
Yeah, so-
- 6:12
Yeah
- 6:12
... yeah, so you, you missed the, the fun.
- 6:15
I missed the fun, and then still got sick for a week right afterwards. [chuckles]
- 6:19
Yeah. Oh, yeah. Yeah, so.
- 6:23
All right. And I'd like to announce that Comfy Org is indeed hiring. Uh, you can s- look at any opportunities on comfyorg/careers.
- 6:32
Yeah. Yeah, we-- yeah, we're hiring for a bunch of stuff, so if you're interested in, uh, joining us, if you're interested in open source, uh, generative AI, well, that's, uh...
- 6:43
Maybe, uh, yeah, maybe we have a, a spot for you on our team, so.
- 6:50
Definitely check out the website. Uh, that is all for the official slides, but now is the fun part of showing-
- 6:58
Yeah
- 6:58
... the UI and taking any questions you may have.
- 7:08
Nice. I'm sure many on this side of the room who are familiar with ComfyUI know this standard, uh, galaxy bottle workflow.
- 7:17
Unfortunately, this spoils the results. I'll just shake up the seed.
- 7:22
Anyone not familiar with ComfyUI, this is all being locally rendered.
- 7:26
Yeah. But this is a very old model. This is SD 1.5.
- 7:30
Yes, this is-
- 7:31
So th- that's why the results are not, not very good.
- 7:34
Yes. This model was, I think, the one that inspired your initial work on ComfyUI at the time.
- 7:41
Yeah, well, fine tunes of this model.
- 7:43
Mm-hmm.
- 7:43
This is the base model, which isn't very good, but-
- 7:47
Yeah, this is-
- 7:48
But it's very fast, so.
- 7:50
It's very fast, but it is ancient tech at this point.
- 7:55
Yeah, it's almost three years old at this point.
- 7:58
Yeah. [chuckles] It is-
- 8:00
Yeah
- 8:00
... ancient.
- 8:02
Yeah.
- 8:06
All right. And there's the UI. Like, you're, you're, you're, you're looking at it. [chuckles]
- 8:11
Yeah. So basically, what, uh, Comfy does, for those who are not familiar, it kind of splits the whole diffusion pipeline into these different components. Like a Stable Diffusion model is a diffusion model, a text encoder, and a VAE, which is why you have those three things
- 8:34
right here. So yeah, model, diffusion model. Clip is the text encoder. VAE, the VAE. And then, so have the sampler,
- 8:47
VAE decode, and save image, and that's, uh, that's basically a basic, uh, diffusion model pipeline. And what that lets you do is you can... Let me check which models you have on here.
- 9:04
Okay.
- 9:04
Not many. [clears throat]
- 9:05
Well, maybe SDXL is, uh-
- 9:08
Yeah, that one should work
- 9:09
... a bit better for, for the-
- 9:11
I can, uh, type those numbers in for you.
- 9:16
Uh, need-
- 9:18
I, in my, uh, rookie mistake of turning off my Num Lock.
- 9:31
Let's see how quick my disk drive is.
- 9:36
Yeah, this is all running on the laptop. That's why it's, uh,
- 9:40
a bit slow. But, uh...
- 9:43
Once it samples, it'll get there.
- 9:58
Yeah, so, yeah, so this does look a lot better than, uh-
- 10:02
Yeah, this model is still also ancient.
- 10:04
Yeah.
- 10:04
I think this one's two years old, uh-
- 10:06
Yeah
- 10:06
... at this point.
- 10:06
This one's, uh, two years old.
- 10:10
Yeah. But we have more exciting workflows though. If we browse the templates, if we wanna go a little advanced, we've got... There we go. This will not run 'cause I do not have, like, 60 gigabytes worth of models.
- 10:25
But here's what that [chuckles] workflow looks like.
- 10:28
Yeah. This is, uh, what a video workflow looks like, which you
- 10:32
can see it's very similar from, uh, from one of the image workflows. It's just you still have the,
- 10:40
the sam- the same sampling node with different settings, and same VAE decode node, which is kind of hidden here. And yeah. So,
- 10:57
so this is, this is the, the WAN 2.1 model. That's probably the best open video model at the moment. And you can see the, the pipeline is still very similar to even the first, uh, Stable Diffusion 1.5 model that was, that we were presenting earlier.
- 11:16
So yeah. But what that lets you do, the fact that you can, uh, you can go and change things. Like, say, if I want to, uh...
- 11:34
Like... Like, this is, uh, one of, uh, a technique that, uh... This is basically, uh, this node where, that I just added, what it does is it, um, it's a, what I call a CFG trick.
- 11:46
So it, it will add something to the, uh, to the sampling, t-to the CFG calculations of the sampling code. So basically, it's, you can easily write these nodes which, uh,
- 12:04
which will change. So you can go and just patch the pipeline this way just by add- just by either writing your own nodes or using ... nodes that already exist
- 12:18
And for anyone unfamiliar with CFG, it is a AI trick where you take the positive prompt, you sample on that. You take the negative prompt, in this case text and watermark, you sample on that, and with the magic of AI, you literally subtract the results from each other, and that in some way improves the image result.
- 12:40
Yeah. I think we can... Yeah. We-- I think we can take, uh... Does anyone have any questions about anything?
- 12:50
Like, uh, or anything in general related to ComfyUI?
- 12:54
We have a question right here.
- 12:56
Yeah. So I see, I can see on the screen some- something about clip passing code.
- 13:01
Yeah.
- 13:02
What is that relationship with the model that we-
- 13:05
Uh, yeah. Those are basically th- those CLIP, uh... You know what CLIP is?
- 13:11
It's a model.
- 13:11
Yeah. It's, uh, basically the diffusion models, they use the text encoder part of the CLIP model to... It's, uh, instead of-- So instead of passing the text directly to the model, they use this, uh, a text encoder because that way the model doesn't have to...
- 13:30
The diffusion model doesn't have to learn, like, all the-- to understand human language. It can just learn the output embeddings of whatever text encoder you use. So yeah. So the, the, this is, uh...
- 13:45
Basically, the CLIP in ComfyUI represents the text encoder. The reason it's named CLIP is because before, like on the Stable Diffusion models, they were only using CLIP as the text encoder.
- 14:00
But in later models, it's more they started... Later models started using different text encoders that were not CLIP. So the name, I should ... Yeah, the name should be changed.
- 14:13
But, uh, yeah. So what this does is it ... Essentially, what this node does is it passes the text through the text encoder, and then the output would essentially be the output embeddings of, or the last hidden, the last hidden state essentially of the text encoder.
- 14:32
And that's usually ... Well, it depends. Uh, it's slightly different for every, for every model, but essentially it's the ... Most of them, it's the last hidden state or the penultimate hidden state that is passed to the diffusion model.
- 14:48
So why there are two CLIP encoders?
- 14:51
Yeah, because this is a positive and negative prompt. Uh, this is how, uh, the CF... Like, 'cause the models, how you sample most of these diffusion models is with, uh, a positive and a negative prompt, and that's s- using CFG, something called classifier-free guidance, CFG.
- 15:16
And what it ... Basically, the, the idea is that if you only sample with a positive prompt. So yeah, if I put CFG to one, that's essentially just sampling with a positive prompt.
- 15:32
And you, you can see what happens when you, you only sample with a positive prompt. It's, uh
- 15:40
... You can see that the image is ... Wait. This is worse than... Well, no, okay, it's 'cause I have this node. Well, yeah, this is worse than it should be.
- 15:51
But, uh, okay. Yeah. If I, yeah, if I sample with just, uh,
- 16:00
just a posit- you see that it's, uh, the image is not very well defined. It's very chaotic if you only... So what CFG does, it's a trick 'cause if you think, uh, of all the possibilities of what the model can generate, it's a kind ...
- 16:19
If you ... It's kind of a way to push for, like ... The CFG scale does, when sampling, it does positive minus negative prompt, and it's a way to push the sampling ver- more towards your positive and away from your negative.
- 16:39
So the higher the scale, the more it will do that, which means you get a more defined image. I don't know if, uh, my explanation makes sense, but, uh-
- 16:53
Yep.
- 16:54
Yeah.
- 16:54
Yep.
- 16:56
Uh, and then the VAE, V-A-E encoder is part of the model or the diffusion state-
- 17:01
Yeah. The, the VAE is 'cause the... What made Stable Diffusion be ex- work extremely well-
- 17:09
For any phone
- 17:09
... and, uh, yeah. What, what made Stable Diffusion be extremely popular is the fact that the, the image generation happens in compressed latent space. So instead of doing it in pixel space on a, like, let's say a 5,000, a 512 times 512 image in pixel space, that's, uh, that's a lot of pixels.
- 17:33
Uh, some earlier diffusion models did that, but they were pretty slow. Stable Diffusion, it did this in a latent space, which, uh, for a Stable Diffusion, the VAE is 8X compressed on every, uh, on every, on the two s- two dimensions.
- 17:51
So yeah. So instead of, uh, sampling a, uh,
- 17:57
yeah, a 512 times 512, you would be sampling a 64 times 64 image, which is, which is why these models are, got so popular because they were a lot more efficient than, uh, what came before.
- 18:14
So yeah. So that's what the VAE... The VAE is just a, yeah, it's a VAE. It in- input is, uh, like 512 time 512 time three channel, and output would be, uh, would be, yeah, 64 times 64 times four channel in the case of, uh, of this model.
- 18:34
Awesome. Thank you so much.
- 18:35
Thank you.
- 18:35
No problem.
- 18:39
So, all right, we have a question right here, and I'll, I'll give you the mic.
- 18:53
Thank you. So, um, ComfyUI is really in a lot of the examples is focused on the image generation as such, you know, kind of all kind of, uh, cool plugins.
- 19:03
Um, I wonder if you have, uh, good suggestions or ideas about evaluating the results, kinda like verifying or kinda like saying this is good image or not a good image, uh, to, to kind of automate that workflow as well.
- 19:17
Uh, that's, uh, that's a difficult thing to do usually 'cause if, uh,
- 19:24
it's the problem where, like, how do you define a good image? 'Cause, uh, yeah. The, yeah, that there's some problems with, uh, 'cause, uh, people's taste is very subjective, so what is a good image for one person might not be good image for another person.
- 19:47
So yeah, it's a, it's a problem they have. It's actually a big problem with, uh, like user people who do, who train these diffusion models, like user preference. Uh, they, uh, uh, when they, when they actually add the user preference data, their results get a bit worse because users like, uh, s- like the average user likes a
- 20:11
certain type of image, which is not maybe, might not be what, uh, what most, what most people want. So it's, uh, yeah.
- 20:25
But, uh ...
- 20:28
Any follow-ups?
- 20:29
No, it's more like I've seen kinda like critique models that you bring in or-
- 20:35
Yeah
- 20:35
... you kind of have a prompt that looks at the image m- uh, like a multimodal. But anyway, if there's nothing there.
- 20:40
Yeah, there's a, yeah, the I've, yeah. We've had like, at least back when I was at Stability, we did have some, uh, we did experiment with some models that tried to just see, oh, like get the output im- output the image from the workflow, get some kind of rating from a model.
- 21:01
But, uh, it didn't work that well. So it's, uh-
- 21:06
Okay. Fair enough. Good question.
- 21:12
All right. Do we have any other questions right now from anyone?
- 21:17
Raise your hand so I can see. [chuckles] Gotcha. Do you have another one? Awesome.
- 21:29
So this is predominantly a workflow, and once you kinda like, uh, develop it, you do it in the UI. Um, any good tools around then, uh, running this more headless and kind of scaling this out and maybe building this into an app for kind of people using it?
- 21:48
Yeah. This is, uh, just, uh, yeah, this is one thing that, uh, 'cause well,
- 21:56
is this com- what ComfyUI is, it's actually you have this interface, but you also have a powerful backend behind it, which, uh, executes the workflows. And right now there's, there's actually, uh, a lot of, uh, [tsking] a lot of different inference service for these workflows, and eventually we'll be building our own.
- 22:19
So, and yeah, and there's, there's already some, uh, [tsking] a lot of, uh, third-party services that I saw that, uh, you can take your workflow, make an app out of it, and, uh, yeah.
- 22:32
So y-you can already, you can already do that, uh, but, uh, just there's no, just no official way of doing it. But there might, uh, there might be one in, in the future, so.
- 22:46
Okay. Thanks for clarifying.
- 22:49
Thank you for the question. All right. Any questions? 'Cause we'll keep on talking about other stuff if there are no more questions, so be prepared. [chuckles]
- 23:07
All righty. Uh, one of the more recent additions to ComfyUI, for a long time, we only supported open source local models. In the past month, we've introduced API nodes, which for paid credits allow you to generate remotely.
- 23:24
Um, let me just open up a template.
- 23:29
We can do... There we go. One of the models that recently came out was a, uh, Black Forest Labs, uh, context model.
- 23:39
Uh, currently not out for open source usage in terms of being able to run locally, but they have made the APIs available.
- 23:47
Yeah. Eventually they're supposed to release an open source version, which, uh, well, we, we already support. They just haven't, haven't released it yet.
- 23:57
Yes. We are waiting for the green light.
- 23:59
Yeah.
- 24:02
And I would run this, but I have no internet connection, and [chuckles] that's one of the limitations of API nodes. You need to, you know, they're not r-ran locally.
- 24:11
Yeah. So yeah, I think there's some interesting
- 24:18
flows. Yeah. So we have, uh, yeah, yeah, we have a lot of different, uh... So the models that, so that we support image, video, yeah, is 3D.
- 24:33
So we have a basic support for like a- Hunyuan 3D model, which is, uh, basically it's an interesting model. It basically outputs a voxel type, uh...
- 24:50
Yeah, like the, the 3D mo-model all these output is a kind of a voxel format, and then you, and then so that's why in the workflow there's, uh,
- 25:02
yeah, there's some, uh, code too, so. But the, the problem with these models, since it's kind of it generates some voxel format, and then you, you need to use an algorithm to convert it to mesh, is that the mesh isn't very high quality, but it's still, uh,
- 25:21
pretty impressive.
- 25:22
I do not have any of these models, Comfy.
- 25:25
Yeah. Uh, so.
- 25:32
And we're currently, I guess, not l- we only have local support for LLMs yet?
- 25:37
Well, we, well, there, there's a bunch of custom nodes with, uh, local LLM support. It's just not a core Comfy thing yet. It's just, uh, we're, we're more focused on the, on like image and video and all these, uh, more visual...
- 25:55
Oh, we also support audio and audio model now, so.
- 25:59
Mm-hmm.
- 26:00
So yeah, it's not as good as some of the, uh, proprietary models out there, but it's, uh, yeah, it's pretty fun to play with.
- 26:11
And there were some more, I think, uh, audio models that came out this week.
- 26:15
Yeah, but those are, those are text-to-speech models.
- 26:19
Gotcha.
- 26:19
Yeah, those which we, we may support. We'll, we'll have to see if, uh, 'cause they're al- they're already supported as custom nodes. But, uh, yeah, before to... Yeah, it's just to integrate them in core Comfy, there needs to be like a reason to, like, uh, if, uh, give them some extra control or some extra,
- 26:44
uh, like extra knobs to turn, or else there's not much point.
- 26:53
Yeah.
- 26:55
I'm interested in any questions from this side of the room that maybe wasn't too familiar with ComfyUI at the start. Uh, do you have any questions, comments, inquiries?
- 27:07
All right.
- 27:08
Yeah.
- 27:08
I will hand you the mic.
- 27:10
Yeah.
- 27:13
Um, sorry, it's me again. So does ComfyUI have a use case for the virtual try-on where, you know, we upload the image of the model, uh, uh, the mannequin and the garment, the clothes, so that it generate the virtual try-on images?
- 27:29
Yeah, like for example, the New Flux context model can, can do that, uh, I think. So there, yeah, there's a few different, there's some open source ways and there's some, uh, some ways using, uh, the API nodes.
- 27:46
But, uh, yeah, virtual try-ons, it's something that seems, uh, very popular, so there are, uh, there are a bunch of workflows for it.
- 27:54
Okay. So we can find it on the ComfyUI and try it out?
- 27:57
Uh, yeah. Yeah, if you, if you search, you can find, uh, you can probably easily find a workflow for it. The only thing you might, uh, it's just some of the...
- 28:09
It's just that the field, uh, evolves so fast that, uh, sometimes, uh, m- workflows you find might be slightly outdated, so. But if I was doing that, I would first try the New Flux context model since that seems to be, uh,
- 28:29
the best one for that. But, uh, yeah.
- 28:33
Uh, the name is New Conf... Uh, what's the model name? New?
- 28:38
Uh, Flux Concept.
- 28:40
Flux Concept.
- 28:41
Context.
- 28:42
Context.
- 28:42
Yeah, I keep... Okay, yeah. Flux, sorry, Flux Context.
- 28:45
Context with a K.
- 28:46
Yeah, context with a K.
- 28:48
Context.
- 28:49
So.
- 28:49
Okay. Thank you. Thank you so much.
- 28:51
And to also follow up on that-
- 28:53
Yeah, and-
- 28:53
Um, the r-
- 28:54
... right now, yeah, right now it's an API node only, but they're g- they should release the, uh, the open source version soon. So yeah. So once that's, once that's released, you'll be able to run it on your, on your, on your machine with the ComfyUI.
- 29:11
Yeah. To follow up on virtual try-on, this is actually something that people have made workflows in the past year. When we were in Japan, when we had a meet and greet there, there were some people who actually made workflows specifically for that.
- 29:22
Back then, there weren't, uh, so-some of the models like context now are very good at a, "Hey, change this one thing." At the time, there weren't, so the workflows you'd find probably have a few dozen nodes basically finding, using, uh, one model to find the masks of like what to change, then another model to in-paint those masks
- 29:44
of the actual thing you wanna change. Now, the models are a bit more, uh, advanced, where you can just say, "Hey, I wanna edit this," and it does it.
- 29:53
And you, of course, combine up the masks as well in case the model gets a little,
- 29:58
uh, a little rowdy and tries to change things you don't want. You can always add masks to keep it contained.
- 30:04
Mm-hmm.
- 30:06
Right. Any more questions on this side of the room?
- 30:10
Uh, thanks for this. A quick question. So sorry, I joined the session very la-late, but, um, if we want to generate any kind of image, I think this allows us to write the prompts, and then it allows us to generate image.
- 30:30
Is that correct?
- 30:32
Yes.
- 30:33
Okay. So for example, if you wanna have a tool that automate Building multiple images based on, let's say, character. Like if, if I want to have defined a character and if, if I want to generate the stories based on the characters, does this allow it?
- 30:52
Yes. Well, yeah. What you need is, uh, there, there's a few different ways to... 'Cause I assume, yeah, you want to generate a consistent-
- 31:00
Yes
- 31:01
... character. 'Cause depending on what you want, you can either, uh, train a LoRA for your character or use one of the newer model like, uh, like the Flux Context model.
- 31:14
Like these, uh, like very recently, there's all these, uh, edit models that have what I call edit models, which are basically, uh, they, they got very inspired what, what the 40 was doing.
- 31:30
So-
- 31:31
Which one do you suggest?
- 31:32
What?
- 31:33
Which one do you suggest?
- 31:35
Uh, right now the, the best one is, uh, the Flux, uh, the, yeah, the Flux, uh, Context model.
- 31:43
But, uh, like I said, it's only, right now it's only available through an API and, but, uh, should be open source, uh, soon. And then there's some other ones too, but, uh, at when, yeah.
- 31:59
Well, what you can do with, uh, with the, the context is just, uh, like some-- You give it a reference image of a character and you say, "Oh, make that character do this," and it actually keeps the character consistency extremely well.
- 32:19
Just, uh-
- 32:20
So there is, there is a way to, uh, maintain character throughout the story generation, right?
- 32:27
Yes. Yeah. Well, what you would do is you would have, uh, yeah, first you generate a, your character of, an image of your character that you're happy with, and then you would, uh, you would pass it to this model and say, "Oh, put this character in this scene, put this character in that scene," and then you generate
- 32:50
your image b- is based on this reference image of the character. [lips smack]
- 32:56
Okay. Thank you.
- 32:59
And to follow up on that, one of the advantages of a node-based system is with the way that is set up, all you can currently edit in it are some of the parameters and the text prompts, but you could also apply the LoRAs.
- 33:12
Uh, LoRAs are, uh, l- low rank adaptations to the model. Um, and b- 'cause it's node-based, you can also mask the specific area each of those low-rank adaptations would apply to.
- 33:23
So let's say you have two LoRAs trained, one for character A, one for character B. Uh, what our node-based system allows is to say, "Hey, in this area of the image, I'd like this LoRA to be active, maybe at this strength."
- 33:36
You could even schedule it in terms of that. And on the other area of an image, you can have, "Oh, I want this other character LoRA to be active."
- 33:44
So if y- even if you, uh, if the, like an all-in-one model like Context doesn't quite do what you want, there are multiple ways you can sort of coerce these models to kind of do it.
- 33:55
With a basic, uh, prompt-based system, there are of course limitations, but because we are node-based, you can do, you know... There's two things for the prompts there. You could set that up to be 10 nodes, and some of those nodes apply a specific LoRA to a particular image.
- 34:11
Sorry, to a particular area of an image.
- 34:15
Do you also, uh, do you, do you also recommend LoRA or, uh, the other one?
- 34:22
Um, if you don't have, uh, like much-
- 34:26
Yeah
- 34:26
... experience in the space, I'd recommend the Context model, mainly because you just, you just have to type in the prompt and it does the work for you. The other one, especially back before these sort of, you know, edit via text models existed, was sort of the brute force way of getting what you want.
- 34:43
But you could really get what you want 'cause you could train it on anything you want. The models don't have to be aware of what it is, and the, the only disadvantage is you need to have enough training images, so like between 10 to 30, to actually get your subject to appear the way you want them to.
- 35:00
With these newer edit models, you only need to give the one image.
- 35:05
No problem.
- 35:07
Yeah.
- 35:09
All right. Any questions here? Or back on that area of the room? I can walk.
- 35:20
All righty, Comfy, what do you wanna talk about next?
- 35:22
Oh, well, yeah. Yeah. Well, we, since we mentioned LoRAs, like LoRAs are one of the... Basically what they, what they are is a, a patch on, I call them a, yeah, they're basically a patch on the model weights, which is, uh, or a more efficient way to train a concept or multiple concept in, in a, in a
- 35:44
model. And yeah, right now we don't, it's basically just if you want to train a model, instead of training the full model, you would train this small patch on the model.
- 35:59
And this allows you to, well, you can train styles, specific characters, anything. So yeah.
- 36:11
Yeah, we can, we can skip showing it off. Uh, this was for the, uh, Japan presentation-
- 36:16
Oh, okay
- 36:16
... where this, this LoRA is for, for a anime character. That goes hard in Japan. It probably doesn't go very hard at a AI conference.
- 36:24
Oh. [chuckles] So.
- 36:27
But this is how you would do it. You would just chain the model there.
- 36:32
And th- these are-
- 36:33
Examples running in ComfyTrade possible?
- 36:36
Uh, wh- which model?
- 36:37
Uh, do I have any SD XL ones?
- 36:39
Which model is this?
- 36:39
I do not.
- 36:39
Okay. This is for 1.5?
- 36:42
Yes.
- 36:42
Okay.
- 36:43
And, uh, s- They're for Japan. [chuckles]
- 36:46
All right. Well, we can still show them off, but, uh-
- 36:49
All right. We, we can try. [chuckles]
- 36:51
Yeah.
- 36:51
Oh, and these would probably look very poorly on these models, but we can give it a shot.
- 36:55
Well, use the anime one.
- 37:00
Okay. We can use an anime one. That works.
- 37:01
Okay. That's the anime one.
- 37:03
Yeah, it works.
- 37:04
Good. So yeah, just... I mean, if you tried that prompt, it's, well, probably not gonna...
- 37:15
Yeah. We can, uh, we c- we, we can do that in a bit. All right.
- 37:20
Okay.
- 37:21
What else would you like to talk about?
- 37:25
Uh, yeah. Well, we can just try, see. Yeah. Well, we can press Run and-
- 37:32
I, I don't know if we should. [both laugh] I don't know if we should press Run.
- 37:36
Okay. Yeah, you... Well, okay. [laughs]
- 37:39
Yeah, we can skew it. Yeah, this is, uh, assignment to do at home, I suppose.
- 37:43
Oh, well. Yeah.
- 37:47
Uh, but we have other models that we support. Let's see. Yeah, apologies that we do not have much live demos. Uh,
- 37:56
uh, th- there were some setup last minute in terms of us attending the conference, so...
- 38:03
Um, but we are here. [chuckles] Sorry about that.
- 38:12
Uh, here are some ControlNet examples where... can't show the inputs, but we can actually-
- 38:19
Yeah
- 38:19
... I, I guess we can sh- we can trust the template system to kinda show what that's about.
- 38:24
Yeah. ControlNets are just one of the many ways to have more control of these, uh, of the, the models.
- 38:32
Yeah. So the examples here would be the inputs that were used to actually generate these images.
- 38:36
Yeah. But those might be... Like, ControlNets might no longer be very useful 'cause now there's all these edit models that are coming out. So yeah. It just means that the space is, uh, is evolving.
- 38:52
But, uh...
- 38:54
Uh, so here's a more advanced workflow where it applies, I believe, different prompts, different areas of the image.
- 39:02
Yeah. This is, uh, different prompts to different areas.
- 39:06
Yeah. We can actually make this one go on the default SD 1.5 model. That one will j- will work.
- 39:12
Mm.
- 39:14
Okay. Ah, yes. This is the old way of prompting things when the models kind of, you had to really coerce them. [chuckles]
- 39:24
We will fix the seed. Okay. And let's see how the laptop handles this.
- 39:32
Yes.
- 39:32
See, assuming there's no loaded images, this should just work.
- 39:36
Yeah. At least half the workflow should work. [chuckles]
- 39:47
Yeah. So... Yeah, this is a very old workflow, but, uh,
- 40:00
I think it still works on even the most recent models.
- 40:07
Uh... Yeah. [chuckles]
- 40:14
And we can, uh, we can change the prompts, maybe it's more obvious. But I believe the prompts are basically doing a different time of day on some of these.
- 40:20
Yeah. Yeah. It's basically different time of day on, like, if you go...
- 40:29
Uh, yeah. Top is, like, night and bottom is daytime.
- 40:36
Yeah. Just, uh... Yeah. So this is just one of many ways you can get, like, more control. This is just a way of applying different prompts in different areas of the image.
- 40:49
And like I said, it's, I think it, it still works even on the most recent models.
- 40:55
Yep. Yep. Everything that basically started fr- from the foundation, uh, Comfy set up two years ago-
- 41:00
Yeah
- 41:00
... most of those, any of those tricks or applications still apply to newer models.
- 41:05
Yeah. Yeah, 'cause they're general, like, diffusion model tricks, and we're still using diffusion. So, you know, yeah. So that's what makes Comfy nice, is that if once a, if a new diffusion model is implemented, usually you can use all the old tricks if you want.
- 41:22
Some of them might not be useful anymore, but you can still use them.
- 41:27
Yeah. The models have also gotten bigger and harder to run locally in some cases on some hardware. So, uh, some of these tricks would, you know, make things run quite a bit slower.
- 41:39
Um, in the early days of image generation, a lot of the improvements were with community fine-tunes who would take, you know, vast datasets and improve the base model. You may have noticed I was a little nervous running a model, uh, a few minutes ago.
- 41:56
The reason for that, that was one of those fine... Th- that was one of the sort of days of back of community fine-tunes. Uh, the datasets they used may not always produce, uh-
- 42:07
Yeah
- 42:07
... the, the, the most, uh-
- 42:09
Yeah
- 42:09
... conference-friendly content.
- 42:10
Yeah. Yeah. There's, there's some interesting things that happen when a model is slightly broken because since it's a diffusion model, uh, if it's slightly broken and you're generating a, like a h- a character, the first, like, the first step might produce a, like a skin color blob, which means it might converge to s- to a naked
- 42:35
person, basically. So yeah.
- 42:39
Yeah. And given there are community fine-tunes that basically everyone trusted to produce better quality images Those are usually generations that you first review and then show rather than press Q and then, uh-
- 42:52
Yeah, but that's the power-
- 42:53
... trust things
- 42:53
... of, uh, running things locally. You don't have, uh, any problem. Uh, you can do whatever you want. So,
- 43:01
yeah.
- 43:01
Yeah. With, with newer models and bigger ones, the training sets are a bit more constrained. So you, you have-
- 43:07
Ah
- 43:08
... the pro- the pros and cons of that.
- 43:09
Well, it's just they're, they're better. They make less, uh, random mistakes. But, uh-
- 43:13
Yeah, you can be more... You can trust more that when you put in a specific prompt, it will not hallucinate as much.
- 43:19
Mm. Yeah. Uh.
- 43:28
All right. So in terms of... We mentioned that we are hiring. I believe we're looking for positions on-
- 43:34
Well, everything pretty much, I think.
- 43:36
Yeah. Everything. Back end, front end-
- 43:38
Yeah, core
- 43:39
... uh, cloud deployment
- 43:40
... core model inference, uh, yeah, cloud. Yeah, just, yeah, go look at our careers page and, uh-
- 43:48
Yeah. It's c- it's comfy.org/careers.
- 43:50
Yeah, and if you haven't tried the software, go try it. You can just... If y- you j- all you need is a g- a g- decent GPU, and you can run it locally, or you can use the API nodes, and yeah.
- 44:04
Yeah. People have gotten some of the early models to work on extremely old GPUs, like-
- 44:11
Yeah
- 44:12
... s- 80-year-old GPUs.
- 44:12
Yeah, yeah. One of the strengths of Comfy is that pretty much any hardware, well, any NVIDIA hardware, the model will r- usually run. It might be extremely slow, but it will usually run.
- 44:28
So, yeah. So are there any final questions?
- 44:34
Oh, right there.
- 44:36
I can just scream it from here.
- 44:38
Uh, hold up. I'll give you the mic. [chuckles] There's a process to this thing.
- 44:42
Thank you.
- 44:43
Yeah.
- 44:43
Um, I've tried using Comfy, and I was just wondering, like, if you could give us, like, a quick synopsis of what do you think about Comfy versus the alternatives that exist.
- 44:51
Like, why would you sort of say Comfy is the one that people should start with or stick to? I have no idea, like, of the depth of it, so just give me, like, a seminar of that, please.
- 44:59
Uh, Comfy is, uh... You should use it because it's the, it's the mo- basically, it's the most powerful one, so if you, uh... Like, everyone who y- like, it's basically the, the end game for, for these, these types of interfaces.
- 45:16
So th- there's nothing that gives you more control, that has more community support, that has more extensions. So the only downside it has right now is it's, uh, it's a bit, uh, difficult to get into, but, uh, we are working on that.
- 45:34
So-
- 45:35
Thank you
- 45:35
... yeah. Yeah, node-based systems, especially if you're not used to them at first, can be quite intimidating. And as, uh, Comfy mentioned, one of the greatest assets of ComfyUI is that it is community extendable, and it is open source in that anything that the core team may not be able to get to, there probably exists a community
- 45:57
solution for that or to do something. Like, like we mentioned in the slide, there are, I believe, 22,000 custom nodes within, like, 3,000 node packs made by, you know, 3,000 separate developers who are all passionate.
- 46:11
Uh, if you go to other places, you will not always have, you know, the certainty as, like, "Oh, can I run this locally? Do I know all my data's safe?"
- 46:21
Inter-- If you are in a, for example, an enterprise setting, data security might be a big thing to avoid becoming the next headline in terms of a data leak or a ransomware attack.
- 46:30
So being, being able to actually look at the source code, if that's your thing, or having your team be able to look at the source code. You can contribute any fixes.
- 46:38
Uh, in terms of optimization and performance, we are pretty much state-of-the-art. Uh, Comfy over there, when the new model comes out, and he hears that there is a way to run it faster, he implements it, or one of us on the team implements it.
- 46:53
So.
- 46:54
Is there any recommendation you'd have for getting support? Like, where would you, like... Is there a Discord channel-
- 46:58
Mm-hmm
- 46:58
... or-
- 46:58
Um, there's a Discord channel that we have for Comfy Org. We also, as we s- s- close the slides, if you just Google ComfyUI, there will be most likely thousands of YouTube videos.
- 47:08
Um, there's even some people who have taken, uh... They've seen the opportunity of the difficulty of ComfyUI, um, and they are, for example, having paid, uh, tutoring classes for it, which is a bit of a eye-opener for us 'cause that says we should probably do a better job onboarding [chuckles] users if, uh, people are, you know, making money
- 47:27
that way. But there should be a lot of resources out there for you.
- 47:34
All right. Any other questions? All right. Over there.
- 47:45
Yeah. Thank you. Is there currently a published product roadmap?
- 47:51
Uh, if you mean, uh, uh, what we are currently, um... Well, we haven't started, really started, actually started yet, but eventually we'll have a solution to run these workflows in the cloud.
- 48:06
And, uh, yeah, how exactly and it's gonna work 'cause we... The, the thing is w- before doing that, we want to fix, there's a, a few issues we have to fix, like the...
- 48:18
For example, we want to make, uh, installing and dealing with the custom node that you install, we want to make that a lot smoother, make, uh, the interface better, add a...
- 48:31
Yeah, we're, what we're gonna do is, uh, improve the interface. Okay, well, the, th- there's always gonna be the node interface, but, uh, we are most likely going to add an other layer on top of it where you can have a more, build a more traditional interface out of your workflow graph And that will fit in
- 48:56
with the, well, with the cloud stuff that we're gonna be doing eventually. So yeah, so that's, that's the direction where we're going in. But, uh, the thing is in this space is that things change a lot.
- 49:11
So a new model that comes out tomorrow might, uh, might mean we need to, uh, pivot a bit. So, uh, that's why I'm not, uh, I'm not giving any promises.
- 49:22
So [chuckles] yeah.
- 49:23
Yeah, 'cause like the, the first thing that went to my mind is we had the gentleman ask a question about, can we serve these workflows up? So it's like, if you can access a workflow through an API, you can have like a single power user building out massive templates-
- 49:34
Yeah
- 49:34
... that maintain like style and brand guidelines or, or story or character or design. And they'd be like role-based access control. You could have like just a general user in there saying, "Hey, I need to generate this workflow based on these parameters.
- 49:46
I can't touch anything else in there." It's like, is that, is like being more enterprise or team-
- 49:53
Yeah
- 49:53
... team-ready com-
- 49:54
Like this is one of... This is a direction we are going into. So like having, uh, just the, the basics for that would be first a good cloud inference service where you can run workflows very well and have all the custom nodes work.
- 50:10
And once we solve that, then all that other, all that other service becomes a lot easier. So yeah.
- 50:17
Awesome. Thank you.
- 50:18
Yep. And to follow up on that, at the end of the week, we will have a blog post about some of the things we are working on, um, for the...
- 50:24
We are planning to allow cloud services, but first as these, as, uh, Comfy said, we need to work out dependency issues, so we'll have a bunch of features being announced there.
- 50:33
For example, we'll have a subgraph option where you can combine a bunch of nodes, put it into one node, and you can double-click into it as like a separate workflow.
- 50:42
Uh, solving dependency issues where custom nodes right now can request different Python packages, making sure all of those could get either properly isolated or have more ways for them to report their compatibility.
- 50:54
'Cause once local becomes much better to run, that means our life trying to get this as a cloud product will also become smoother.
- 51:01
Yeah. I think, yeah, we are out of time now, so-
- 51:04
Yep
- 51:05
... I would, uh, would like to thank everyone for coming. We, uh, yeah. And I hope you, uh, you learned something. [chuckles]
- 51:14
Yeah. Thank you for all the questions. Greatly appreciated.
- 51:17
Yeah. [upbeat music]