Transcript of SeC Next-Gen Video Auto-Masking | Even Works with Scene Cuts!

Hey everyone, the artificial trainer here. Welcome back for another video. I've been gone for a little while. I've been working on some other projects. I've got projects I'm working on with studios, some of my own technical projects that I'm working on that I'll be excited to share with all of you. But I've found some really cool projects in the meantime and I want to share them with all of you. Let's get them up and running in Comfy UI. This video is going to be about SEC automasking. So it's similar to SAM 2 segment and anything too, but it uses a VLM on top of it to give the model better context and ultimately mask things better and more consistently and even birectionally and if something goes in and out of frame, which is awesome because that was a big limitation of the SAM 2 nodes and Comfy. So let's hop into it and see how it works. Okay, so the first thing that you're going to need to do is get your Comfy UI up and running. If you don't have Comfy Comfy UI working yet, head to the link in the description. I have a tutorial on how to install it. It's a few months old. There may be a couple outdated things in it, but it's still, I think, one of the best tutorials out there on how to install Comfy UI. So once you have it up, go to the manager and go to custom nodes manager and look for sec nodes and you're going to install this one. Once you install it, a restart, it'll prompt you to restart Comfy UI. Restart Comi, refresh your page and then you'll have the nodes ready to download the workflow. So, head into my Patreon, grab the workflow, drag it into Comfy Live. All right. And then I also provided a few a couple videos down here. I'm going to use them in the examples here. You saw the examples at the beginning. They're the same ones. So, this one is going to be a good example of even when there's scene cuts, you're able to mask the scene cuts. And I will also show you. So, the output videos, we're basically drawing red over the mask. and drawing it onto the video itself. I'll also show you where the mask output is to be able to use this for something like vase or you know w fun or something like that because this is going to be super useful especially for something like vase. This will be super useful. Okay, so we load up our video in here. Just drag it in to the load video node. Then we're then we're looking at annotation frame. So annotation frame is the frame that we're going to select our points to mask. This does not have to be the start frame which is really cool. So you can if you have a you know a 347 frame video and you put the annotation frame as 300 then it'll birectionally mask. So it'll it'll start at that 300 frame and it'll go forward to the end of the video and it'll go backwards to the beginning of the video and you'll get all of the masks. Okay. And then let's I'll clear my points editor here and then just walk you quickly through how this works. So if you don't have the frame that you want to mask, just run the workflow. So you can either click the run button or controll enter and wait until this picture shows up here. So it'll be blank until it runs through this node. So it's connecting to background image here. And then once it once you see that image pop up, you can stop the workflow. So in order to mask this, we have green nodes are what we want to mask. Red nodes are what we don't want to mask. And then so I want to mask the guy in the blue shirt. And I don't want to mask the other guy. Let's just mask out the or make sure we're not masking the background. Okay. And then if we run this, we should get a pretty good result of just the blue guy masked throughout the whole video. All right. And there is our video. So really awesome. It only masks that one particular guy. See it doesn't mask this guy in the brown shirt, but it it always masks the guy in the blue shirt. Even that little piece of him there. It's a really really cool results. Awesome model. Super useful for video masking and for any, you know, video workflows that require masking. I'll be replacing SAM 2 nodes in all of my workflows with these nodes. All right, let's try another one. That's that would typically be pretty challenging for Sam, too. So, I'm going to mask one of these like a really small people down here. And you can even mask them like someone that's not in frame from the start, like one of these people down here that shows up at the end. You can mask them as well if you change the annotation frame. So, let's change the annotation frame to something like 150. It's a 260 frame video. So, let's do 150. And then we'll mask one of these people that's not in the frame at the beginning. All right. So, we want to change the points editor image. So, we'll run it and then we'll stop once the image changes. Okay. So, the image changed. And now I stopped it. All right. So, let's do a new canvas. And then I want to mask one of the people down here that wasn't in the frame at the beginning. So this guy here what shouldn't have been in the frame at the beginning. Just make sure you usually want to mask like their lower body, their upper body, and then their head at a minimum. And then just throw some other masks around here for things you don't want to mask. Okay, I think that should be pretty good. And now we can run it. And just again to note, you know, we're starting at or we're using frame 150. We're going forward and backwards from that point to do the masking. So, really cool. The subject doesn't have to start in frame. It can go in and out of frame, which is pretty awesome. All right, so here you go. So, when the subject first comes in frame, especially for smaller things like this, you can get a little bit of pixely frame. It's not 100% great, but if you use the grow mask and you um just grow the mask, it should help with that pixelation. There's also a fill holes in the KJ node um what is it? Grow mask with blur. He has a fill holes um option. So you can use that as well to help with that inconsistent mask right at the beginning. But you can see it does a really good job of keeping the mask on him throughout the whole video. And this is something that Sam 2 typically struggled with for me when it was something smaller like that. So really pretty impressive how much this is able to improve SAM 2. This is actually built on top of SAM 2. The segmentation is SAM 2, but it adds a VL VLM aspect on top of SAM 2 to help with consistency and improve the masking. All right, so that is it for this one. Really awesome new masking technique. Like I said, I'll probably be replacing all of my SAM 2 masking with this masking just because it has that birectional. It's more consistent and you can do scene cuts. So, if you use even something like the Phantom Vase merge, you could mask a whole scene out and then potentially use context windows and replace characters throughout an entire scene rather than needing to chunk out the scenes one by one. That's something I haven't tried yet, but it would be a really interesting experiment to try. So, that is it for this video. I hope it's useful to you. If you make anything cool with this, share it in the Discord. Always love to get more people in the Discord and see what they're working on. Helps me tailor my content better to my subscribers. Speaking of which, if you haven't subscribed yet, hit the like and subscribe button below. Click the bell for notifications to get all the latest AI news. Appreciate you watching this video and I'll talk to you again in the next one.