ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

Talk transcript

1. 0:14

   All right. Uh, good morning, everyone. I am Jedrick Kosinski, and this is...

2. 0:20

   Yeah. Hello, I am, uh, known online as, uh, [REDACTED:username], the original creator of ComfyUI. [audience applauding]

3. 0:30

   And we are part of the Comfy Org, the organization that, uh, is in charge of ComfyUI.

4. 0:38

   So, uh, I guess now that I have a mic, I'll ask again. Who here has heard of ComfyUI?

5. 0:44

   All right. All right. This half of the room, very knowledgeable. Very nice. Very nice. Um, for those unaware, we are an open source note-based design canvas intended for, uh, generative AI purposes for, uh, multimodal, um, creative applications.

6. 1:01

   We support image, video, audio, 3D, text, and more, uh, generative AI models.

7. 1:09

   Um, ComfyUI supports the absolute bleeding edge of generative AI tech on day one. We have [REDACTED:username] here implementing it from not quite scratch, but it is redesigned from the original implementations.

8. 1:24

   Uh, we offer open source locally hosted models that support NVIDIA, AMD, and Intel hardware, and we also support closed source API accessible models that we, as the name suggests, just use an API to deliver to the user. [lip smack]

9. 1:38

   All of this functionality is also extendable with community-supported custom node packs, so anything we do not have the time to get to ourselves, the community does for us.

10. 1:49

    A big part of what makes ComfyUI special is the shareability of the workflows. Any image or video that was generated by ComfyUI has embedded metadata that lets you drag it back into the canvas and brings you the original workflow with all of the parameters that was used to generate it.

11. 2:08

    Uh, this sort of shareability and virality has really helped ComfyUI's traction. If you do a simple Google search on, uh, ComfyUI workflows, you will find pages and pages and pages of results from the past year and a half, most of which are still compatible with modern ComfyUI versions.

12. 2:29

    Um, in terms of pure numbers, uh, this sort of shareability and virality has, over the past two years, taken us to the position of top 150 most popular GitHub repos of all time with 78,000 stars.

13. 2:49

    Any comments, Comfy?

14. 2:50

    Uh, no.

15. 2:52

    All right. All right. [chuckles] Uh, with more of the traction numbers, we have three to four million active users. We have 20K daily downloads. We have 22,000 custom nodes made by 3,000 public developers that we enable in our ecosystem, and we've been adopted by Amazon, Apple, Tencent, Netflix, and more.

16. 3:13

    And pretty much any startup these days built around visual generative AI probably has ComfyUI working somewhere on their backend.

17. 3:23

    Um, why is ComfyUI popular? Um, it gives maximal control. You can go beyond prompts and interact with models that give you access to depth maps, line art, uh, masks, anything like that that is out in the space.

18. 3:40

    If it's out-- If it's open source, we probably either support it directly or the community has, uh, made it possible. Uh, we are an all-in-one platform for both exploration for creatives and automation for developers.

19. 3:53

    Sometimes those roles can also be switched, where developers want to explore tweaking models and seeing how things can be extended, and so we offer that as well through our custom node, uh, feature.

20. 4:05

    And because we are open source, we do not only depend on the output of the core team. We can trust the community to let us know anything they'd want us to work on and also make anything we do not have the time to work on on our team.

21. 4:23

    Any comments, Comfy?

22. 4:25

    Uh, well, I think, uh, yeah, I think we haven't shown the, the interface yet, so...

23. 4:31

    Yes. We have only shown one screenshot of the interface at the very start. We will, uh-

24. 4:36

    Yeah

25. 4:36

    ... show that off [chuckles] as well. You wanna do it... Just do it later. [chuckles]

26. 4:42

    Yeah.

27. 4:43

    All right. Story behind ComfyUI.

28. 4:44

    Yes.

29. 4:44

    You wanna get into this, Comfy?

30. 4:45

    Yeah. The quick story is that, uh, basically, ComfyUI started as my own personal project, and then I... And then, uh, [lip smack] yeah. And then, which... Yeah. I started it in, uh, January, January 2023, and then

31. 5:02

    six months later, I was hired at Stability AI, so I spent one year at Stability AI. They were using Comfy for, uh, for more, uh, like experimentation with, uh, internal experimentation with the models.

32. 5:18

    And then I left Stability AI in, uh, June 2024, and then I joined up with, uh, Jolin and Robin, and, uh, we, uh, [lip smack] we made, uh, like the Com- Comfy, uh, company.

33. 5:31

    And, uh, yeah, that's, uh... Things have been, uh, going pretty well since then, so...

34. 5:38

    Yeah. Yep. And this picture was taken on the-

35. 5:42

    Yeah. We, we went, uh... That, that picture, we went on top of, uh, Mount Fuji, uh, which, uh, I don't recommend. It's, uh, ver- very, very difficult. But, uh, yeah.

36. 5:55

    But we did it, so yeah.

37. 5:57

    I, I lucked out, and my flight to Japan, uh, got rerouted to Alaska for 24 hours. So I landed in Tokyo Five hours before they were going to be waking up to go to Mount Fuji.

38. 6:11

    So I got to-

39. 6:12

    Yeah, so-

40. 6:12

    Yeah

41. 6:12

    ... yeah, so you, you missed the, the fun.

42. 6:15

    I missed the fun, and then still got sick for a week right afterwards. [chuckles]

43. 6:19

    Yeah. Oh, yeah. Yeah, so.

44. 6:23

    All right. And I'd like to announce that Comfy Org is indeed hiring. Uh, you can s- look at any opportunities on comfyorg/careers.

45. 6:32

    Yeah. Yeah, we-- yeah, we're hiring for a bunch of stuff, so if you're interested in, uh, joining us, if you're interested in open source, uh, generative AI, well, that's, uh...

46. 6:43

    Maybe, uh, yeah, maybe we have a, a spot for you on our team, so.

47. 6:50

    Definitely check out the website. Uh, that is all for the official slides, but now is the fun part of showing-

48. 6:58

    Yeah

49. 6:58

    ... the UI and taking any questions you may have.

50. 7:08

    Nice. I'm sure many on this side of the room who are familiar with ComfyUI know this standard, uh, galaxy bottle workflow.

51. 7:17

    Unfortunately, this spoils the results. I'll just shake up the seed.

52. 7:22

    Anyone not familiar with ComfyUI, this is all being locally rendered.

53. 7:26

    Yeah. But this is a very old model. This is SD 1.5.

54. 7:30

    Yes, this is-

55. 7:31

    So th- that's why the results are not, not very good.

56. 7:34

    Yes. This model was, I think, the one that inspired your initial work on ComfyUI at the time.

57. 7:41

    Yeah, well, fine tunes of this model.

58. 7:43

    Mm-hmm.

59. 7:43

    This is the base model, which isn't very good, but-

60. 7:47

    Yeah, this is-

61. 7:48

    But it's very fast, so.

62. 7:50

    It's very fast, but it is ancient tech at this point.

63. 7:55

    Yeah, it's almost three years old at this point.

64. 7:58

    Yeah. [chuckles] It is-

65. 8:00

    Yeah

66. 8:00

    ... ancient.

67. 8:02

    Yeah.

68. 8:06

    All right. And there's the UI. Like, you're, you're, you're, you're looking at it. [chuckles]

69. 8:11

    Yeah. So basically, what, uh, Comfy does, for those who are not familiar, it kind of splits the whole diffusion pipeline into these different components. Like a Stable Diffusion model is a diffusion model, a text encoder, and a VAE, which is why you have those three things

70. 8:34

    right here. So yeah, model, diffusion model. Clip is the text encoder. VAE, the VAE. And then, so have the sampler,

71. 8:47

    VAE decode, and save image, and that's, uh, that's basically a basic, uh, diffusion model pipeline. And what that lets you do is you can... Let me check which models you have on here.

72. 9:04

    Okay.

73. 9:04

    Not many. [clears throat]

74. 9:05

    Well, maybe SDXL is, uh-

75. 9:08

    Yeah, that one should work

76. 9:09

    ... a bit better for, for the-

77. 9:11

    I can, uh, type those numbers in for you.

78. 9:16

    Uh, need-

79. 9:18

    I, in my, uh, rookie mistake of turning off my Num Lock.

80. 9:31

    Let's see how quick my disk drive is.

81. 9:36

    Yeah, this is all running on the laptop. That's why it's, uh,

82. 9:40

    a bit slow. But, uh...

83. 9:43

    Once it samples, it'll get there.

84. 9:58

    Yeah, so, yeah, so this does look a lot better than, uh-

85. 10:02

    Yeah, this model is still also ancient.

86. 10:04

    Yeah.

87. 10:04

    I think this one's two years old, uh-

88. 10:06

    Yeah

89. 10:06

    ... at this point.

90. 10:06

    This one's, uh, two years old.

91. 10:10

    Yeah. But we have more exciting workflows though. If we browse the templates, if we wanna go a little advanced, we've got... There we go. This will not run 'cause I do not have, like, 60 gigabytes worth of models.

92. 10:25

    But here's what that [chuckles] workflow looks like.

93. 10:28

    Yeah. This is, uh, what a video workflow looks like, which you

94. 10:32

    can see it's very similar from, uh, from one of the image workflows. It's just you still have the,

95. 10:40

    the sam- the same sampling node with different settings, and same VAE decode node, which is kind of hidden here. And yeah. So,

96. 10:57

    so this is, this is the, the WAN 2.1 model. That's probably the best open video model at the moment. And you can see the, the pipeline is still very similar to even the first, uh, Stable Diffusion 1.5 model that was, that we were presenting earlier.

97. 11:16

    So yeah. But what that lets you do, the fact that you can, uh, you can go and change things. Like, say, if I want to, uh...

98. 11:34

    Like... Like, this is, uh, one of, uh, a technique that, uh... This is basically, uh, this node where, that I just added, what it does is it, um, it's a, what I call a CFG trick.

99. 11:46

    So it, it will add something to the, uh, to the sampling, t-to the CFG calculations of the sampling code. So basically, it's, you can easily write these nodes which, uh,

100. 12:04

     which will change. So you can go and just patch the pipeline this way just by add- just by either writing your own nodes or using ... nodes that already exist

101. 12:18

     And for anyone unfamiliar with CFG, it is a AI trick where you take the positive prompt, you sample on that. You take the negative prompt, in this case text and watermark, you sample on that, and with the magic of AI, you literally subtract the results from each other, and that in some way improves the image result.

102. 12:40

     Yeah. I think we can... Yeah. We-- I think we can take, uh... Does anyone have any questions about anything?

103. 12:50

     Like, uh, or anything in general related to ComfyUI?

104. 12:54

     We have a question right here.

105. 12:56

     Yeah. So I see, I can see on the screen some- something about clip passing code.

106. 13:01

     Yeah.

107. 13:02

     What is that relationship with the model that we-

108. 13:05

     Uh, yeah. Those are basically th- those CLIP, uh... You know what CLIP is?

109. 13:11

     It's a model.

110. 13:11

     Yeah. It's, uh, basically the diffusion models, they use the text encoder part of the CLIP model to... It's, uh, instead of-- So instead of passing the text directly to the model, they use this, uh, a text encoder because that way the model doesn't have to...

111. 13:30

     The diffusion model doesn't have to learn, like, all the-- to understand human language. It can just learn the output embeddings of whatever text encoder you use. So yeah. So the, the, this is, uh...

112. 13:45

     Basically, the CLIP in ComfyUI represents the text encoder. The reason it's named CLIP is because before, like on the Stable Diffusion models, they were only using CLIP as the text encoder.

113. 14:00

     But in later models, it's more they started... Later models started using different text encoders that were not CLIP. So the name, I should ... Yeah, the name should be changed.

114. 14:13

     But, uh, yeah. So what this does is it ... Essentially, what this node does is it passes the text through the text encoder, and then the output would essentially be the output embeddings of, or the last hidden, the last hidden state essentially of the text encoder.

115. 14:32

     And that's usually ... Well, it depends. Uh, it's slightly different for every, for every model, but essentially it's the ... Most of them, it's the last hidden state or the penultimate hidden state that is passed to the diffusion model.

116. 14:48

     So why there are two CLIP encoders?

117. 14:51

     Yeah, because this is a positive and negative prompt. Uh, this is how, uh, the CF... Like, 'cause the models, how you sample most of these diffusion models is with, uh, a positive and a negative prompt, and that's s- using CFG, something called classifier-free guidance, CFG.

118. 15:16

     And what it ... Basically, the, the idea is that if you only sample with a positive prompt. So yeah, if I put CFG to one, that's essentially just sampling with a positive prompt.

119. 15:32

     And you, you can see what happens when you, you only sample with a positive prompt. It's, uh

120. 15:40

     ... You can see that the image is ... Wait. This is worse than... Well, no, okay, it's 'cause I have this node. Well, yeah, this is worse than it should be.

121. 15:51

     But, uh, okay. Yeah. If I, yeah, if I sample with just, uh,

122. 16:00

     just a posit- you see that it's, uh, the image is not very well defined. It's very chaotic if you only... So what CFG does, it's a trick 'cause if you think, uh, of all the possibilities of what the model can generate, it's a kind ...

123. 16:19

     If you ... It's kind of a way to push for, like ... The CFG scale does, when sampling, it does positive minus negative prompt, and it's a way to push the sampling ver- more towards your positive and away from your negative.

124. 16:39

     So the higher the scale, the more it will do that, which means you get a more defined image. I don't know if, uh, my explanation makes sense, but, uh-

125. 16:53

     Yep.

126. 16:54

     Yeah.

127. 16:54

     Yep.

128. 16:56

     Uh, and then the VAE, V-A-E encoder is part of the model or the diffusion state-

129. 17:01

     Yeah. The, the VAE is 'cause the... What made Stable Diffusion be ex- work extremely well-

130. 17:09

     For any phone

131. 17:09

     ... and, uh, yeah. What, what made Stable Diffusion be extremely popular is the fact that the, the image generation happens in compressed latent space. So instead of doing it in pixel space on a, like, let's say a 5,000, a 512 times 512 image in pixel space, that's, uh, that's a lot of pixels.

132. 17:33

     Uh, some earlier diffusion models did that, but they were pretty slow. Stable Diffusion, it did this in a latent space, which, uh, for a Stable Diffusion, the VAE is 8X compressed on every, uh, on every, on the two s- two dimensions.

133. 17:51

     So yeah. So instead of, uh, sampling a, uh,

134. 17:57

     yeah, a 512 times 512, you would be sampling a 64 times 64 image, which is, which is why these models are, got so popular because they were a lot more efficient than, uh, what came before.

135. 18:14

     So yeah. So that's what the VAE... The VAE is just a, yeah, it's a VAE. It in- input is, uh, like 512 time 512 time three channel, and output would be, uh, would be, yeah, 64 times 64 times four channel in the case of, uh, of this model.

136. 18:34

     Awesome. Thank you so much.

137. 18:35

     Thank you.

138. 18:35

     No problem.

139. 18:39

     So, all right, we have a question right here, and I'll, I'll give you the mic.

140. 18:53

     Thank you. So, um, ComfyUI is really in a lot of the examples is focused on the image generation as such, you know, kind of all kind of, uh, cool plugins.

141. 19:03

     Um, I wonder if you have, uh, good suggestions or ideas about evaluating the results, kinda like verifying or kinda like saying this is good image or not a good image, uh, to, to kind of automate that workflow as well.

142. 19:17

     Uh, that's, uh, that's a difficult thing to do usually 'cause if, uh,

143. 19:24

     it's the problem where, like, how do you define a good image? 'Cause, uh, yeah. The, yeah, that there's some problems with, uh, 'cause, uh, people's taste is very subjective, so what is a good image for one person might not be good image for another person.

144. 19:47

     So yeah, it's a, it's a problem they have. It's actually a big problem with, uh, like user people who do, who train these diffusion models, like user preference. Uh, they, uh, uh, when they, when they actually add the user preference data, their results get a bit worse because users like, uh, s- like the average user likes a

145. 20:11

     certain type of image, which is not maybe, might not be what, uh, what most, what most people want. So it's, uh, yeah.

146. 20:25

     But, uh ...

147. 20:28

     Any follow-ups?

148. 20:29

     No, it's more like I've seen kinda like critique models that you bring in or-

149. 20:35

     Yeah

150. 20:35

     ... you kind of have a prompt that looks at the image m- uh, like a multimodal. But anyway, if there's nothing there.

151. 20:40

     Yeah, there's a, yeah, the I've, yeah. We've had like, at least back when I was at Stability, we did have some, uh, we did experiment with some models that tried to just see, oh, like get the output im- output the image from the workflow, get some kind of rating from a model.

152. 21:01

     But, uh, it didn't work that well. So it's, uh-

153. 21:06

     Okay. Fair enough. Good question.

154. 21:12

     All right. Do we have any other questions right now from anyone?

155. 21:17

     Raise your hand so I can see. [chuckles] Gotcha. Do you have another one? Awesome.

156. 21:29

     So this is predominantly a workflow, and once you kinda like, uh, develop it, you do it in the UI. Um, any good tools around then, uh, running this more headless and kind of scaling this out and maybe building this into an app for kind of people using it?

157. 21:48

     Yeah. This is, uh, just, uh, yeah, this is one thing that, uh, 'cause well,

158. 21:56

     is this com- what ComfyUI is, it's actually you have this interface, but you also have a powerful backend behind it, which, uh, executes the workflows. And right now there's, there's actually, uh, a lot of, uh, [tsking] a lot of different inference service for these workflows, and eventually we'll be building our own.

159. 22:19

     So, and yeah, and there's, there's already some, uh, [tsking] a lot of, uh, third-party services that I saw that, uh, you can take your workflow, make an app out of it, and, uh, yeah.

160. 22:32

     So y-you can already, you can already do that, uh, but, uh, just there's no, just no official way of doing it. But there might, uh, there might be one in, in the future, so.

161. 22:46

     Okay. Thanks for clarifying.

162. 22:49

     Thank you for the question. All right. Any questions? 'Cause we'll keep on talking about other stuff if there are no more questions, so be prepared. [chuckles]

163. 23:07

     All righty. Uh, one of the more recent additions to ComfyUI, for a long time, we only supported open source local models. In the past month, we've introduced API nodes, which for paid credits allow you to generate remotely.

164. 23:24

     Um, let me just open up a template.

165. 23:29

     We can do... There we go. One of the models that recently came out was a, uh, Black Forest Labs, uh, context model.

166. 23:39

     Uh, currently not out for open source usage in terms of being able to run locally, but they have made the APIs available.

167. 23:47

     Yeah. Eventually they're supposed to release an open source version, which, uh, well, we, we already support. They just haven't, haven't released it yet.

168. 23:57

     Yes. We are waiting for the green light.

169. 23:59

     Yeah.

170. 24:02

     And I would run this, but I have no internet connection, and [chuckles] that's one of the limitations of API nodes. You need to, you know, they're not r-ran locally.

171. 24:11

     Yeah. So yeah, I think there's some interesting

172. 24:18

     flows. Yeah. So we have, uh, yeah, yeah, we have a lot of different, uh... So the models that, so that we support image, video, yeah, is 3D.

173. 24:33

     So we have a basic support for like a- Hunyuan 3D model, which is, uh, basically it's an interesting model. It basically outputs a voxel type, uh...

174. 24:50

     Yeah, like the, the 3D mo-model all these output is a kind of a voxel format, and then you, and then so that's why in the workflow there's, uh,

175. 25:02

     yeah, there's some, uh, code too, so. But the, the problem with these models, since it's kind of it generates some voxel format, and then you, you need to use an algorithm to convert it to mesh, is that the mesh isn't very high quality, but it's still, uh,

176. 25:21

     pretty impressive.

177. 25:22

     I do not have any of these models, Comfy.

178. 25:25

     Yeah. Uh, so.

179. 25:32

     And we're currently, I guess, not l- we only have local support for LLMs yet?

180. 25:37

     Well, we, well, there, there's a bunch of custom nodes with, uh, local LLM support. It's just not a core Comfy thing yet. It's just, uh, we're, we're more focused on the, on like image and video and all these, uh, more visual...

181. 25:55

     Oh, we also support audio and audio model now, so.

182. 25:59

     Mm-hmm.

183. 26:00

     So yeah, it's not as good as some of the, uh, proprietary models out there, but it's, uh, yeah, it's pretty fun to play with.

184. 26:11

     And there were some more, I think, uh, audio models that came out this week.

185. 26:15

     Yeah, but those are, those are text-to-speech models.

186. 26:19

     Gotcha.

187. 26:19

     Yeah, those which we, we may support. We'll, we'll have to see if, uh, 'cause they're al- they're already supported as custom nodes. But, uh, yeah, before to... Yeah, it's just to integrate them in core Comfy, there needs to be like a reason to, like, uh, if, uh, give them some extra control or some extra,

188. 26:44

     uh, like extra knobs to turn, or else there's not much point.

189. 26:53

     Yeah.

190. 26:55

     I'm interested in any questions from this side of the room that maybe wasn't too familiar with ComfyUI at the start. Uh, do you have any questions, comments, inquiries?

191. 27:07

     All right.

192. 27:08

     Yeah.

193. 27:08

     I will hand you the mic.

194. 27:10

     Yeah.

195. 27:13

     Um, sorry, it's me again. So does ComfyUI have a use case for the virtual try-on where, you know, we upload the image of the model, uh, uh, the mannequin and the garment, the clothes, so that it generate the virtual try-on images?

196. 27:29

     Yeah, like for example, the New Flux context model can, can do that, uh, I think. So there, yeah, there's a few different, there's some open source ways and there's some, uh, some ways using, uh, the API nodes.

197. 27:46

     But, uh, yeah, virtual try-ons, it's something that seems, uh, very popular, so there are, uh, there are a bunch of workflows for it.

198. 27:54

     Okay. So we can find it on the ComfyUI and try it out?

199. 27:57

     Uh, yeah. Yeah, if you, if you search, you can find, uh, you can probably easily find a workflow for it. The only thing you might, uh, it's just some of the...

200. 28:09

     It's just that the field, uh, evolves so fast that, uh, sometimes, uh, m- workflows you find might be slightly outdated, so. But if I was doing that, I would first try the New Flux context model since that seems to be, uh,

201. 28:29

     the best one for that. But, uh, yeah.

202. 28:33

     Uh, the name is New Conf... Uh, what's the model name? New?

203. 28:38

     Uh, Flux Concept.

204. 28:40

     Flux Concept.

205. 28:41

     Context.

206. 28:42

     Context.

207. 28:42

     Yeah, I keep... Okay, yeah. Flux, sorry, Flux Context.

208. 28:45

     Context with a K.

209. 28:46

     Yeah, context with a K.

210. 28:48

     Context.

211. 28:49

     So.

212. 28:49

     Okay. Thank you. Thank you so much.

213. 28:51

     And to also follow up on that-

214. 28:53

     Yeah, and-

215. 28:53

     Um, the r-

216. 28:54

     ... right now, yeah, right now it's an API node only, but they're g- they should release the, uh, the open source version soon. So yeah. So once that's, once that's released, you'll be able to run it on your, on your, on your machine with the ComfyUI.

217. 29:11

     Yeah. To follow up on virtual try-on, this is actually something that people have made workflows in the past year. When we were in Japan, when we had a meet and greet there, there were some people who actually made workflows specifically for that.

218. 29:22

     Back then, there weren't, uh, so-some of the models like context now are very good at a, "Hey, change this one thing." At the time, there weren't, so the workflows you'd find probably have a few dozen nodes basically finding, using, uh, one model to find the masks of like what to change, then another model to in-paint those masks

219. 29:44

     of the actual thing you wanna change. Now, the models are a bit more, uh, advanced, where you can just say, "Hey, I wanna edit this," and it does it.

220. 29:53

     And you, of course, combine up the masks as well in case the model gets a little,

221. 29:58

     uh, a little rowdy and tries to change things you don't want. You can always add masks to keep it contained.

222. 30:04

     Mm-hmm.

223. 30:06

     Right. Any more questions on this side of the room?

224. 30:10

     Uh, thanks for this. A quick question. So sorry, I joined the session very la-late, but, um, if we want to generate any kind of image, I think this allows us to write the prompts, and then it allows us to generate image.

225. 30:30

     Is that correct?

226. 30:32

     Yes.

227. 30:33

     Okay. So for example, if you wanna have a tool that automate Building multiple images based on, let's say, character. Like if, if I want to have defined a character and if, if I want to generate the stories based on the characters, does this allow it?

228. 30:52

     Yes. Well, yeah. What you need is, uh, there, there's a few different ways to... 'Cause I assume, yeah, you want to generate a consistent-

229. 31:00

     Yes

230. 31:01

     ... character. 'Cause depending on what you want, you can either, uh, train a LoRA for your character or use one of the newer model like, uh, like the Flux Context model.

231. 31:14

     Like these, uh, like very recently, there's all these, uh, edit models that have what I call edit models, which are basically, uh, they, they got very inspired what, what the 40 was doing.

232. 31:30

     So-

233. 31:31

     Which one do you suggest?

234. 31:32

     What?

235. 31:33

     Which one do you suggest?

236. 31:35

     Uh, right now the, the best one is, uh, the Flux, uh, the, yeah, the Flux, uh, Context model.

237. 31:43

     But, uh, like I said, it's only, right now it's only available through an API and, but, uh, should be open source, uh, soon. And then there's some other ones too, but, uh, at when, yeah.

238. 31:59

     Well, what you can do with, uh, with the, the context is just, uh, like some-- You give it a reference image of a character and you say, "Oh, make that character do this," and it actually keeps the character consistency extremely well.

239. 32:19

     Just, uh-

240. 32:20

     So there is, there is a way to, uh, maintain character throughout the story generation, right?

241. 32:27

     Yes. Yeah. Well, what you would do is you would have, uh, yeah, first you generate a, your character of, an image of your character that you're happy with, and then you would, uh, you would pass it to this model and say, "Oh, put this character in this scene, put this character in that scene," and then you generate

242. 32:50

     your image b- is based on this reference image of the character. [lips smack]

243. 32:56

     Okay. Thank you.

244. 32:59

     And to follow up on that, one of the advantages of a node-based system is with the way that is set up, all you can currently edit in it are some of the parameters and the text prompts, but you could also apply the LoRAs.

245. 33:12

     Uh, LoRAs are, uh, l- low rank adaptations to the model. Um, and b- 'cause it's node-based, you can also mask the specific area each of those low-rank adaptations would apply to.

246. 33:23

     So let's say you have two LoRAs trained, one for character A, one for character B. Uh, what our node-based system allows is to say, "Hey, in this area of the image, I'd like this LoRA to be active, maybe at this strength."

247. 33:36

     You could even schedule it in terms of that. And on the other area of an image, you can have, "Oh, I want this other character LoRA to be active."

248. 33:44

     So if y- even if you, uh, if the, like an all-in-one model like Context doesn't quite do what you want, there are multiple ways you can sort of coerce these models to kind of do it.

249. 33:55

     With a basic, uh, prompt-based system, there are of course limitations, but because we are node-based, you can do, you know... There's two things for the prompts there. You could set that up to be 10 nodes, and some of those nodes apply a specific LoRA to a particular image.

250. 34:11

     Sorry, to a particular area of an image.

251. 34:15

     Do you also, uh, do you, do you also recommend LoRA or, uh, the other one?

252. 34:22

     Um, if you don't have, uh, like much-

253. 34:26

     Yeah

254. 34:26

     ... experience in the space, I'd recommend the Context model, mainly because you just, you just have to type in the prompt and it does the work for you. The other one, especially back before these sort of, you know, edit via text models existed, was sort of the brute force way of getting what you want.

255. 34:43

     But you could really get what you want 'cause you could train it on anything you want. The models don't have to be aware of what it is, and the, the only disadvantage is you need to have enough training images, so like between 10 to 30, to actually get your subject to appear the way you want them to.

256. 35:00

     With these newer edit models, you only need to give the one image.

257. 35:05

     No problem.

258. 35:07

     Yeah.

259. 35:09

     All right. Any questions here? Or back on that area of the room? I can walk.

260. 35:20

     All righty, Comfy, what do you wanna talk about next?

261. 35:22

     Oh, well, yeah. Yeah. Well, we, since we mentioned LoRAs, like LoRAs are one of the... Basically what they, what they are is a, a patch on, I call them a, yeah, they're basically a patch on the model weights, which is, uh, or a more efficient way to train a concept or multiple concept in, in a, in a

262. 35:44

     model. And yeah, right now we don't, it's basically just if you want to train a model, instead of training the full model, you would train this small patch on the model.

263. 35:59

     And this allows you to, well, you can train styles, specific characters, anything. So yeah.

264. 36:11

     Yeah, we can, we can skip showing it off. Uh, this was for the, uh, Japan presentation-

265. 36:16

     Oh, okay

266. 36:16

     ... where this, this LoRA is for, for a anime character. That goes hard in Japan. It probably doesn't go very hard at a AI conference.

267. 36:24

     Oh. [chuckles] So.

268. 36:27

     But this is how you would do it. You would just chain the model there.

269. 36:32

     And th- these are-

270. 36:33

     Examples running in ComfyTrade possible?

271. 36:36

     Uh, wh- which model?

272. 36:37

     Uh, do I have any SD XL ones?

273. 36:39

     Which model is this?

274. 36:39

     I do not.

275. 36:39

     Okay. This is for 1.5?

276. 36:42

     Yes.

277. 36:42

     Okay.

278. 36:43

     And, uh, s- They're for Japan. [chuckles]

279. 36:46

     All right. Well, we can still show them off, but, uh-

280. 36:49

     All right. We, we can try. [chuckles]

281. 36:51

     Yeah.

282. 36:51

     Oh, and these would probably look very poorly on these models, but we can give it a shot.

283. 36:55

     Well, use the anime one.

284. 37:00

     Okay. We can use an anime one. That works.

285. 37:01

     Okay. That's the anime one.

286. 37:03

     Yeah, it works.

287. 37:04

     Good. So yeah, just... I mean, if you tried that prompt, it's, well, probably not gonna...

288. 37:15

     Yeah. We can, uh, we c- we, we can do that in a bit. All right.

289. 37:20

     Okay.

290. 37:21

     What else would you like to talk about?

291. 37:25

     Uh, yeah. Well, we can just try, see. Yeah. Well, we can press Run and-

292. 37:32

     I, I don't know if we should. [both laugh] I don't know if we should press Run.

293. 37:36

     Okay. Yeah, you... Well, okay. [laughs]

294. 37:39

     Yeah, we can skew it. Yeah, this is, uh, assignment to do at home, I suppose.

295. 37:43

     Oh, well. Yeah.

296. 37:47

     Uh, but we have other models that we support. Let's see. Yeah, apologies that we do not have much live demos. Uh,

297. 37:56

     uh, th- there were some setup last minute in terms of us attending the conference, so...

298. 38:03

     Um, but we are here. [chuckles] Sorry about that.

299. 38:12

     Uh, here are some ControlNet examples where... can't show the inputs, but we can actually-

300. 38:19

     Yeah

301. 38:19

     ... I, I guess we can sh- we can trust the template system to kinda show what that's about.

302. 38:24

     Yeah. ControlNets are just one of the many ways to have more control of these, uh, of the, the models.

303. 38:32

     Yeah. So the examples here would be the inputs that were used to actually generate these images.

304. 38:36

     Yeah. But those might be... Like, ControlNets might no longer be very useful 'cause now there's all these edit models that are coming out. So yeah. It just means that the space is, uh, is evolving.

305. 38:52

     But, uh...

306. 38:54

     Uh, so here's a more advanced workflow where it applies, I believe, different prompts, different areas of the image.

307. 39:02

     Yeah. This is, uh, different prompts to different areas.

308. 39:06

     Yeah. We can actually make this one go on the default SD 1.5 model. That one will j- will work.

309. 39:12

     Mm.

310. 39:14

     Okay. Ah, yes. This is the old way of prompting things when the models kind of, you had to really coerce them. [chuckles]

311. 39:24

     We will fix the seed. Okay. And let's see how the laptop handles this.

312. 39:32

     Yes.

313. 39:32

     See, assuming there's no loaded images, this should just work.

314. 39:36

     Yeah. At least half the workflow should work. [chuckles]

315. 39:47

     Yeah. So... Yeah, this is a very old workflow, but, uh,

316. 40:00

     I think it still works on even the most recent models.

317. 40:07

     Uh... Yeah. [chuckles]

318. 40:14

     And we can, uh, we can change the prompts, maybe it's more obvious. But I believe the prompts are basically doing a different time of day on some of these.

319. 40:20

     Yeah. Yeah. It's basically different time of day on, like, if you go...

320. 40:29

     Uh, yeah. Top is, like, night and bottom is daytime.

321. 40:36

     Yeah. Just, uh... Yeah. So this is just one of many ways you can get, like, more control. This is just a way of applying different prompts in different areas of the image.

322. 40:49

     And like I said, it's, I think it, it still works even on the most recent models.

323. 40:55

     Yep. Yep. Everything that basically started fr- from the foundation, uh, Comfy set up two years ago-

324. 41:00

     Yeah

325. 41:00

     ... most of those, any of those tricks or applications still apply to newer models.

326. 41:05

     Yeah. Yeah, 'cause they're general, like, diffusion model tricks, and we're still using diffusion. So, you know, yeah. So that's what makes Comfy nice, is that if once a, if a new diffusion model is implemented, usually you can use all the old tricks if you want.

327. 41:22

     Some of them might not be useful anymore, but you can still use them.

328. 41:27

     Yeah. The models have also gotten bigger and harder to run locally in some cases on some hardware. So, uh, some of these tricks would, you know, make things run quite a bit slower.

329. 41:39

     Um, in the early days of image generation, a lot of the improvements were with community fine-tunes who would take, you know, vast datasets and improve the base model. You may have noticed I was a little nervous running a model, uh, a few minutes ago.

330. 41:56

     The reason for that, that was one of those fine... Th- that was one of the sort of days of back of community fine-tunes. Uh, the datasets they used may not always produce, uh-

331. 42:07

     Yeah

332. 42:07

     ... the, the, the most, uh-

333. 42:09

     Yeah

334. 42:09

     ... conference-friendly content.

335. 42:10

     Yeah. Yeah. There's, there's some interesting things that happen when a model is slightly broken because since it's a diffusion model, uh, if it's slightly broken and you're generating a, like a h- a character, the first, like, the first step might produce a, like a skin color blob, which means it might converge to s- to a naked

336. 42:35

     person, basically. So yeah.

337. 42:39

     Yeah. And given there are community fine-tunes that basically everyone trusted to produce better quality images Those are usually generations that you first review and then show rather than press Q and then, uh-

338. 42:52

     Yeah, but that's the power-

339. 42:53

     ... trust things

340. 42:53

     ... of, uh, running things locally. You don't have, uh, any problem. Uh, you can do whatever you want. So,

341. 43:01

     yeah.

342. 43:01

     Yeah. With, with newer models and bigger ones, the training sets are a bit more constrained. So you, you have-

343. 43:07

     Ah

344. 43:08

     ... the pro- the pros and cons of that.

345. 43:09

     Well, it's just they're, they're better. They make less, uh, random mistakes. But, uh-

346. 43:13

     Yeah, you can be more... You can trust more that when you put in a specific prompt, it will not hallucinate as much.

347. 43:19

     Mm. Yeah. Uh.

348. 43:28

     All right. So in terms of... We mentioned that we are hiring. I believe we're looking for positions on-

349. 43:34

     Well, everything pretty much, I think.

350. 43:36

     Yeah. Everything. Back end, front end-

351. 43:38

     Yeah, core

352. 43:39

     ... uh, cloud deployment

353. 43:40

     ... core model inference, uh, yeah, cloud. Yeah, just, yeah, go look at our careers page and, uh-

354. 43:48

     Yeah. It's c- it's comfy.org/careers.

355. 43:50

     Yeah, and if you haven't tried the software, go try it. You can just... If y- you j- all you need is a g- a g- decent GPU, and you can run it locally, or you can use the API nodes, and yeah.

356. 44:04

     Yeah. People have gotten some of the early models to work on extremely old GPUs, like-

357. 44:11

     Yeah

358. 44:12

     ... s- 80-year-old GPUs.

359. 44:12

     Yeah, yeah. One of the strengths of Comfy is that pretty much any hardware, well, any NVIDIA hardware, the model will r- usually run. It might be extremely slow, but it will usually run.

360. 44:28

     So, yeah. So are there any final questions?

361. 44:34

     Oh, right there.

362. 44:36

     I can just scream it from here.

363. 44:38

     Uh, hold up. I'll give you the mic. [chuckles] There's a process to this thing.

364. 44:42

     Thank you.

365. 44:43

     Yeah.

366. 44:43

     Um, I've tried using Comfy, and I was just wondering, like, if you could give us, like, a quick synopsis of what do you think about Comfy versus the alternatives that exist.

367. 44:51

     Like, why would you sort of say Comfy is the one that people should start with or stick to? I have no idea, like, of the depth of it, so just give me, like, a seminar of that, please.

368. 44:59

     Uh, Comfy is, uh... You should use it because it's the, it's the mo- basically, it's the most powerful one, so if you, uh... Like, everyone who y- like, it's basically the, the end game for, for these, these types of interfaces.

369. 45:16

     So th- there's nothing that gives you more control, that has more community support, that has more extensions. So the only downside it has right now is it's, uh, it's a bit, uh, difficult to get into, but, uh, we are working on that.

370. 45:34

     So-

371. 45:35

     Thank you

372. 45:35

     ... yeah. Yeah, node-based systems, especially if you're not used to them at first, can be quite intimidating. And as, uh, Comfy mentioned, one of the greatest assets of ComfyUI is that it is community extendable, and it is open source in that anything that the core team may not be able to get to, there probably exists a community

373. 45:57

     solution for that or to do something. Like, like we mentioned in the slide, there are, I believe, 22,000 custom nodes within, like, 3,000 node packs made by, you know, 3,000 separate developers who are all passionate.

374. 46:11

     Uh, if you go to other places, you will not always have, you know, the certainty as, like, "Oh, can I run this locally? Do I know all my data's safe?"

375. 46:21

     Inter-- If you are in a, for example, an enterprise setting, data security might be a big thing to avoid becoming the next headline in terms of a data leak or a ransomware attack.

376. 46:30

     So being, being able to actually look at the source code, if that's your thing, or having your team be able to look at the source code. You can contribute any fixes.

377. 46:38

     Uh, in terms of optimization and performance, we are pretty much state-of-the-art. Uh, Comfy over there, when the new model comes out, and he hears that there is a way to run it faster, he implements it, or one of us on the team implements it.

378. 46:53

     So.

379. 46:54

     Is there any recommendation you'd have for getting support? Like, where would you, like... Is there a Discord channel-

380. 46:58

     Mm-hmm

381. 46:58

     ... or-

382. 46:58

     Um, there's a Discord channel that we have for Comfy Org. We also, as we s- s- close the slides, if you just Google ComfyUI, there will be most likely thousands of YouTube videos.

383. 47:08

     Um, there's even some people who have taken, uh... They've seen the opportunity of the difficulty of ComfyUI, um, and they are, for example, having paid, uh, tutoring classes for it, which is a bit of a eye-opener for us 'cause that says we should probably do a better job onboarding [chuckles] users if, uh, people are, you know, making money

384. 47:27

     that way. But there should be a lot of resources out there for you.

385. 47:34

     All right. Any other questions? All right. Over there.

386. 47:45

     Yeah. Thank you. Is there currently a published product roadmap?

387. 47:51

     Uh, if you mean, uh, uh, what we are currently, um... Well, we haven't started, really started, actually started yet, but eventually we'll have a solution to run these workflows in the cloud.

388. 48:06

     And, uh, yeah, how exactly and it's gonna work 'cause we... The, the thing is w- before doing that, we want to fix, there's a, a few issues we have to fix, like the...

389. 48:18

     For example, we want to make, uh, installing and dealing with the custom node that you install, we want to make that a lot smoother, make, uh, the interface better, add a...

390. 48:31

     Yeah, we're, what we're gonna do is, uh, improve the interface. Okay, well, the, th- there's always gonna be the node interface, but, uh, we are most likely going to add an other layer on top of it where you can have a more, build a more traditional interface out of your workflow graph And that will fit in

391. 48:56

     with the, well, with the cloud stuff that we're gonna be doing eventually. So yeah, so that's, that's the direction where we're going in. But, uh, the thing is in this space is that things change a lot.

392. 49:11

     So a new model that comes out tomorrow might, uh, might mean we need to, uh, pivot a bit. So, uh, that's why I'm not, uh, I'm not giving any promises.

393. 49:22

     So [chuckles] yeah.

394. 49:23

     Yeah, 'cause like the, the first thing that went to my mind is we had the gentleman ask a question about, can we serve these workflows up? So it's like, if you can access a workflow through an API, you can have like a single power user building out massive templates-

395. 49:34

     Yeah

396. 49:34

     ... that maintain like style and brand guidelines or, or story or character or design. And they'd be like role-based access control. You could have like just a general user in there saying, "Hey, I need to generate this workflow based on these parameters.

397. 49:46

     I can't touch anything else in there." It's like, is that, is like being more enterprise or team-

398. 49:53

     Yeah

399. 49:53

     ... team-ready com-

400. 49:54

     Like this is one of... This is a direction we are going into. So like having, uh, just the, the basics for that would be first a good cloud inference service where you can run workflows very well and have all the custom nodes work.

401. 50:10

     And once we solve that, then all that other, all that other service becomes a lot easier. So yeah.

402. 50:17

     Awesome. Thank you.

403. 50:18

     Yep. And to follow up on that, at the end of the week, we will have a blog post about some of the things we are working on, um, for the...

404. 50:24

     We are planning to allow cloud services, but first as these, as, uh, Comfy said, we need to work out dependency issues, so we'll have a bunch of features being announced there.

405. 50:33

     For example, we'll have a subgraph option where you can combine a bunch of nodes, put it into one node, and you can double-click into it as like a separate workflow.

406. 50:42

     Uh, solving dependency issues where custom nodes right now can request different Python packages, making sure all of those could get either properly isolated or have more ways for them to report their compatibility.

407. 50:54

     'Cause once local becomes much better to run, that means our life trying to get this as a cloud product will also become smoother.

408. 51:01

     Yeah. I think, yeah, we are out of time now, so-

409. 51:04

     Yep

410. 51:05

     ... I would, uh, would like to thank everyone for coming. We, uh, yeah. And I hope you, uh, you learned something. [chuckles]

411. 51:14

     Yeah. Thank you for all the questions. Greatly appreciated.

412. 51:17

     Yeah. [upbeat music]