In this video, I want to show you how you can create on your local computer with comfy UI and LTX. Literally unlimited lip syncing from music video to just speech. Create your storyboard.
All this links to workflows will be down below as well. Still to the end and check out this video that was created with storyboard inside CompuI and LTX 2. 3.
For this video, I'm using Comfy UI0. 17. 2.
And inside this Confui, we have our templates. If we look on our video, you can see right here we have it LTX 2. 3 image audio to video.
This workflow was original inspiration. However, it was did little bit modifications add some custom nodes to make it work. So, let's go ahead and start from beginning.
We're going ahead and open this workflow. And inside this workflow, you'll notice we have it our load audio where we can put it. And as an example, they have a small audio about 9 seconds long.
Where is the fuzzy frog going around if we're going inside the our magic happen? It's a similar what we have before in LTX. And right here we have it our durations where we can control it.
So this is will be kind of inspiration and this is what workflow will be. So it will be unpacked already and ready to use. Right here you can see this is workflow that first I created.
This was a limitations on how much we can render because I can create I can create some animations but we constrain with some time limits based on our resources and as well on the performance and in some case I notice if I do even 30 seconds yes it can do up to this but degradations happen and as well we have it worse and worse but sometimes music a little bit longer or for example in my case I want to create a song. I want to create a long song and I want this create up to it's kind of looping. So what is happening here?
We have it our image that we're going to use over and over again as our reference and here we have it for example loading our music loader and in the end I changed little bit we have our audio segment looper. So actually what it does is it determine how long is our audio and base it on this and also based on what duration you want it it will regenerate stop on this time and continue generating next segment. So it will create segment over the period and stopped and from that stop create another and created with this um looking it will actually will create over and over from same point but using this techniques it will reset to original pause because it was starting.
So right here is example. You can see she animating moving and look right here. There you go.
That was reset to original pose and that maybe work okay with talking head. The reason is why you want to do this if you utilize like multi- camera render then you can create it segments for example you can create segments for about maybe 15 seconds and another ones about 20 seconds. Take different cameras.
It's what render in this case and based on this you can kind of render all song along and uh after in editing you can create multi frames and kind of those moments where is the jump overcome with a different type of the camera position. So this is one approach. No way is a loop happen in everything.
Our next logical steps. What if we can specify on different image at this time come to help our story board type of the video. So right here you can see it's a special segment dynamics where you can select how many images you want to connect.
Next we for example says okay let's connect to image two will be this camera. Next we have it this other and you connect all of them this way with our music. Next you can say what durations you want for each of the segments.
For example, first one maybe like 15 seconds. Next, maybe you want only 10 seconds and it does it actually create start with this and this kind of save that problem because usually when you do music video it's a bit boring to have a person stand on one place. Technically you can do this one but it is boring.
So in this case we'll just change our camera position. Tell you true. I did work a little bit on the first last frames and I'm getting some result to this.
So please subscribe, like and follow the channel to see what's happening when with LTX we can go work with last and first and last frame. But beside right here you can see we have a two uh whatever cameras we have it angles and we can just generate nice things about this down below right here we have it how long link and what we have it left like 30 seconds. So if you needed to add more you can add another image.
I'm again just showing example and for example right here we can say 30 seconds so we can add more and more different uh images and I think it is actually support about 50 or something see now we over by 30 seconds by the way if it's over the person still standing there and you can see in our video and just moving but not singing not with a lip sync going in this case and yes right here and also I have another versions where this is one generic prompt apply applied for everything. Another versions it's where individual prompts you can specify for each um frame from our storyboard. So for example you can tell she can just look around or something.
So you can mix around different prompts and frames. And most important of course this is when generated as a result you will have it generating as a single frames like right here and also in the end it will combine together. So here is example it's combine all of them and in some you can see they have it separate frames.
So this is actually it's what magic happened inside this our segment loop which in a end kind of combine them together and manage. So you still have it segments if you want to rerender separately or create and sometimes maybe you do maybe you want to create just specific segment on this case I have it another workflow or you can easy modify already existing because they do have it range it's like this range where you can say start ending time and if you want just under specific segment of your song you can do this way or speech for example you know sometimes glitch happen and I notice If I do very long the I do have it some very weird type of the visual glitch. I want overlap this.
So I can go ahead and create this. By the way that is also coming if you look on original um sometimes they have it also range selector but if you need it this workflow by the way all these workflows has a link down below and you can download all of them from my Patreon website. Um we also I did modify slightly uh overall um for example right here math calculations was little bit change so we can apply for not just cut straight so we can properly make all this link so this work this way so and right here for example different workflow that is specifically designed to work with the music and you can see I have it way from here it's downloaded on the song and as we're playing and we can add the same way we can add images only with little bit different.
So I'm going around and I say as well around this area with a 6 seconds I feel like by music it should change composition. So we go add key frame and it's create additional image and now I can take this image and connect to new. So I know on this time it will render this segment.
Next my camera jump to this segment and I can continue listening. Uh maybe around here like right there. And yeah, we can we can also just probably just expand if you need see bigger so we can make this wavelength little bit uh wave and let's add another ones right here.
We have it another ones and we'll just connect to the next and so on. You can also just create all of these multiple key frames uh during all of your segment and kind of reassign um specifically your storyboard only visible for music and I say I just like to create this music video and I think it's kind of fun was it does have it a little bit problem I'll let you know uh sometimes it's may not lips moving in some conditions actually mostly it's with music when it's algorithm maybe does not pick up it work very well with a just speech with talking head uh with music it depend uh sometimes uh singing person maybe going can speak with a back singers because it does not find this way so it uh it does not know who is who there another version with this where we adding like right here you can see this is one prompt so we have another ones where we also adding prompt to this as file. So depend how you want you want complex or simplicity.
And uh this is again this is my workflows based on original from LTX from this workflow. So I just a little bit modify them uh make sure they fit my interest what I'm doing with this and I hope you actually also guys will have it fun to create. Be sure you watch all video down below.
Let me know what you think and we'll see you next time. Bye. Who am I to disagree?
I travel the world of the seven. Everybody's looking for something. Some of them want to [music] use you.
Some of them want to get used by you. Some of them want to abuse you. Some of them want to be abused.
Sweet dreams are made to do. This is who am I to disagree? I travel [music] the world in seven.
Everybody's looking for something. All your head moving [music] on. Keep your head and moving on.
All you got to hit and moving on. Keep your head moving on your [music] head and moving on. Keep your head moving on.
All you do moving on. Keep your head moving on. Some more want to use you.
[music] Some more want to get just by you. Some more to abuse you. Some more than want [music] to be abused.
Sweet dreams are made of this. Who am I to disagree? I [music] travel the world in the seven seas.
Everybody's looking for something. Sweet dreams are made [music] of this. Who am I to disagree?
I travel the world in the seven seas. Everybody's looking for something. [music] [music] Drams [music] are made.
Who am I to disagree? I travel the world in the seven seas. Everybody's looking for something.
Sweet dreams are made [music] over this. Who am I to disagree? I travel the world in the seven seas.
Everybody's looking for something. Sweet dreams are made over this. [music] Who am I to disagree?
I travel the world and the simp. [music] Who am I to disagree? Who am I to disagree?
Everybody's looking for something. [music] Everybody's looking for something. [music] In this video, I want to show you how you can create on your local computer with comfy UI and LTX.
Literally unlimited lip syncing from music video to just speech. Create your storyboard. All this links to workflows will be down below as well.
Still to the end and check out this video that was created with storyboard inside CompuI and LTX 2. 3. For this video, I'm using Comfy UI0.
17. 2. And inside this Confui, we have our templates.
If we look on our video, you can see right here we have it LTX 2. 3 image audio to video. This workflow was original inspiration.
However, it was did little bit modifications add some custom nodes to make it work. So, let's go ahead and start from beginning. We're going ahead and open this workflow.
And inside this workflow, you'll notice we have it our load audio where we can put it. And as an example, they have a small audio about 9 seconds long. Where is the fuzzy frog going around if we're going inside the our magic happen?
It's a similar what we have before in LTX. And right here we have it our durations where we can control it. So this is will be kind of inspiration and this is what workflow will be.
So it will be unpacked already and ready to use. Right here you can see this is workflow that first I created. This was a limitations on how much we can render because I can create I can create some animations but we constrain with some time limits based on our resources and as well on the performance and in some case I notice if I do even 30 seconds yes it can do up to this but degradations happen and as well we have it worse and worse but sometimes music a little bit longer or for example in my case I want to create a song.
I want to create a long song and I want this create up to it's kind of looping. So what is happening here? We have it our image that we're going to use over and over again as our reference and here we have it for example loading our music loader and in the end I changed little bit we have our audio segment looper.
So actually what it does is it determine how long is our audio and base it on this and also based on what duration you want it it will regenerate stop on this time and continue generating next segment. So it will create segment over the period and stopped and from that stop create another and created with this um looking it will actually will create over and over from same point but using this techniques it will reset to original pause because it was starting. So right here is example.
You can see she animating moving and look right here. There you go. That was reset to original pose and that maybe work okay with talking head.
The reason is why you want to do this if you utilize like multi- camera render then you can create it segments for example you can create segments for about maybe 15 seconds and another ones about 20 seconds. Take different cameras. It's what render in this case and based on this you can kind of render all song along and uh after in editing you can create multi frames and kind of those moments where is the jump overcome with a different type of the camera position.
So this is one approach. No way is a loop happen in everything. Our next logical steps.
What if we can specify on different image at this time come to help our story board type of the video. So right here you can see it's a special segment dynamics where you can select how many images you want to connect. Next we for example says okay let's connect to image two will be this camera.
Next we have it this other and you connect all of them this way with our music. Next you can say what durations you want for each of the segments. For example, first one maybe like 15 seconds.
Next, maybe you want only 10 seconds and it does it actually create start with this and this kind of save that problem because usually when you do music video it's a bit boring to have a person stand on one place. Technically you can do this one but it is boring. So in this case we'll just change our camera position.
Tell you true. I did work a little bit on the first last frames and I'm getting some result to this. So please subscribe, like and follow the channel to see what's happening when with LTX we can go work with last and first and last frame.
But beside right here you can see we have a two uh whatever cameras we have it angles and we can just generate nice things about this down below right here we have it how long link and what we have it left like 30 seconds. So if you needed to add more you can add another image. I'm again just showing example and for example right here we can say 30 seconds so we can add more and more different uh images and I think it is actually support about 50 or something see now we over by 30 seconds by the way if it's over the person still standing there and you can see in our video and just moving but not singing not with a lip sync going in this case and yes right here and also I have another versions where this is one generic prompt apply applied for everything.
Another versions it's where individual prompts you can specify for each um frame from our storyboard. So for example you can tell she can just look around or something. So you can mix around different prompts and frames.
And most important of course this is when generated as a result you will have it generating as a single frames like right here and also in the end it will combine together. So here is example it's combine all of them and in some you can see they have it separate frames. So this is actually it's what magic happened inside this our segment loop which in a end kind of combine them together and manage.
So you still have it segments if you want to rerender separately or create and sometimes maybe you do maybe you want to create just specific segment on this case I have it another workflow or you can easy modify already existing because they do have it range it's like this range where you can say start ending time and if you want just under specific segment of your song you can do this way or speech for example you know sometimes glitch happen and I notice If I do very long the I do have it some very weird type of the visual glitch. I want overlap this. So I can go ahead and create this.
By the way that is also coming if you look on original um sometimes they have it also range selector but if you need it this workflow by the way all these workflows has a link down below and you can download all of them from my Patreon website. Um we also I did modify slightly uh overall um for example right here math calculations was little bit change so we can apply for not just cut straight so we can properly make all this link so this work this way so and right here for example different workflow that is specifically designed to work with the music and you can see I have it way from here it's downloaded on the song and as we're playing and we can add the same way we can add images only with little bit different. So I'm going around and I say as well around this area with a 6 seconds I feel like by music it should change composition.
So we go add key frame and it's create additional image and now I can take this image and connect to new. So I know on this time it will render this segment. Next my camera jump to this segment and I can continue listening.
Uh maybe around here like right there. And yeah, we can we can also just probably just expand if you need see bigger so we can make this wavelength little bit uh wave and let's add another ones right here. We have it another ones and we'll just connect to the next and so on.
You can also just create all of these multiple key frames uh during all of your segment and kind of reassign um specifically your storyboard only visible for music and I say I just like to create this music video and I think it's kind of fun was it does have it a little bit problem I'll let you know uh sometimes it's may not lips moving in some conditions actually mostly it's with music when it's algorithm maybe does not pick up it work very well with a just speech with talking head uh with music it depend uh sometimes uh singing person maybe going can speak with a back singers because it does not find this way so it uh it does not know who is who there another version with this where we adding like right here you can see this is one prompt so we have another ones where we also adding prompt to this as file. So depend how you want you want complex or simplicity. And uh this is again this is my workflows based on original from LTX from this workflow.
So I just a little bit modify them uh make sure they fit my interest what I'm doing with this and I hope you actually also guys will have it fun to create. Be sure you watch all video down below. Let me know what you think and we'll see you next time.
Bye. Who am I to disagree? I travel the world of the seven.
Everybody's looking for something. Some of them want to [music] use you. Some of them want to get used by you.
Some of them want to abuse you. Some of them want to be abused. Sweet dreams are made to do.
This is who am I to disagree? I travel [music] the world in seven. Everybody's looking for something.
All your head moving [music] on. Keep your head and moving on. All you got to hit and moving on.
Keep your head moving on your [music] head and moving on. Keep your head moving on. All you do moving on.
Keep your head moving on. Some more want to use you. [music] Some more want to get just by you.
Some more to abuse you. Some more than want [music] to be abused. Sweet dreams are made of this.
Who am I to disagree? I [music] travel the world in the seven seas. Everybody's looking for something.
Sweet dreams are made [music] of this. Who am I to disagree? I travel the world in the seven seas.
Everybody's looking for something. [music] [music] Drams [music] are made. Who am I to disagree?
I travel the world in the seven seas. Everybody's looking for something. Sweet dreams are made [music] over this.
Who am I to disagree? I travel the world in the seven seas. Everybody's looking for something.
Sweet dreams are made over this. [music] Who am I to disagree? I travel the world and the simp.
[music] Who am I to disagree? Who am I to disagree? Everybody's looking for something.
[music] Everybody's looking for something.
