is written in front of
@行人一kun is an active and popular creator in the aigc circle. He often works tirelessly to research new tools and share new works. Most importantly, he likes to share his experience and methods of using technology and products, and is always passionate about encouraging everyone's creation in the community.
When we saw the launch of the "first AIGC short drama" that Kun Dao and Ke Ling AI collaborated with, we were sincerely happy for the progress of his and AI's film and television creations. I reprint the creative notes he wrote, hoping to inspire friends who are curious about AI creation.
From January 4th to July 13th, in half a year, we went from "striving to present" to "already presented". The first part of the short play series "Mountains and Seas Strange Mirror" is finally released to everyone. We met, you can see it for free on Kuaishou Xingmang.
△ The first five episodes of "The Strange Mirror of Mountains and Seas: Cutting Through Waves" have been updated by @kuaishouxingmang.
This article, I want to talk to you about the behind-the-scenes of this project - maybe you want to know some of the process that I want to record. , just think of it as a summary of the past six months. The
article may be a bit long, and you don’t have to read it to the end, but I guarantee that every word is exchanged for liver cells.
Let me first answer the most frequently asked question:
In the past six months, I have changed from a traditional film and television person who has been in the industry for more than 20 years to an AI film and television worker. The company I founded, Xingxian Culture, has also completed its transition from traditional film and television. The transformation of the company into an AI-native film and television company. Why make this change?
I want to say, because I believe in . The emergence of
in December last year made me realize that AI film and television has become a possibility, and based on my intuition, the impact that AI film and television technology will bring to the film and television industry in the future will be disruptive. This intuition, Today, I am becoming more and more determined.
Many people see imperfection, but what I see is infinite possibilities.
Because these imperfections can be solved through technical means in the future, it is only a matter of time. And I know that if I switch from traditional film and television to AI film and television, it will be an increasingly easier process - every day will be the most difficult day, because tomorrow's technology will definitely be further than today's, and tomorrow's technology will be better than today's. The creative tools used will definitely be more accurate and advanced than those used today;
then the most difficult day is the day I turn around, and every day after that, I will move forward with hope. (The self-pua logic is perfectly established)
The second most asked question I am asked is: Why did choose to make the original IP "Mountains and Seas Strange Mirror"?
ai In the early stages of development of film and television, it is necessary to choose the pain points of traditional film and television to create .
Among traditional film and television, the two major tracks of fantasy and science fiction are subject to the high cost and long cycle of CG special effects. In fact, their development is limited, and it is precisely these two tracks that have relatively strong demand from the audience.
I remembered that when we were working on "The Snow King" many years ago, director Zhang Yimou was the chief director and I was the executive director. The movie "The Great Wall" was released during the recording of the program. We were lucky enough to watch the premiere with Director Zhang. On the way back to the base after watching the premiere, I asked Director Zhang: "Why not make Taotie more intelligent, similar to Alien?” I still remember the answer that Director Zhang gave me:
After we discussed it, we couldn’t support it under the current budget conditions. Every special effects shot is accurate. If Taotie was smarter, he would definitely need more Lots of special effects shots.
Even for a director like Director Zhang, special effects budget is an important factor to consider, not to mention other film and television workers.
△Director Zhang Yimou (left) and @行人一kun (right)
Among science fiction and fantasy, science fiction has a relatively high threshold; and fantasy has this gene in many traditional Chinese cultures, so there will be no threshold.Just like no one in China will ask: Why did Sun Wukong jump out of the stone?
In the field of fantasy, there are three traditional classics that have relatively large room for development - "Journey to the West", "Fengshen" and "Mountains and Seas" "The Classic of Mountains and Seas", the first two already have insurmountable mountains, and "The Classic of Mountains and Seas" is precisely more suitable for new IP development.
"The Classic of Mountains and Seas" itself does not have a big story logic line. It is more like an encyclopedia, recording many mythical beasts, which provides us with a lot of creative materials.
In addition, I borrowed the "Feng Yue Bao Jian" from "A Dream of Red Mansions": a strange mirror that reflects the mountains and seas in everyone's heart and the little monster in everyone's heart. Each of us needs to have a wonderful journey with this little monster, either reconciliation or devouring it. Everyone is in the mirror and comes out to see fate. The big logic of the ip
is built. Under this, there is a lot of development that can be done, and short plays are just the first step. The forward development of
ai is inevitable. Joining it is not a temporary enthusiasm or a desire to realize money quickly, but the hope that it can become the second growth curve of my life. I will be patient in this industry for at least ten years. Looking at this time cycle, this half year, or even the next two years, is a process of experimentation and exploration.
I am from Hunan, and I like Zeng Guofan’s sentence very much: A strong enemy will fight a war . The development of AI technology is changing with each passing day, and it is difficult to catch up with various waves, large and small, but when every big wave comes, at least we are the "conspicuous package" standing on the wave.
So for every work I launch, I will have some "motivations" of my own, either for new business attempts, or to seize the wave of technological innovation, and this time "Mountains and Seas Strange Mirror: Cutting the Waves" is exactly that Combine the two.
Next, as an AI film and television creator, I would like to report to you some of my creative experiences in the past six months. How to choose a theme in
? How to establish a creative framework?
After 2 months of polishing the script, a new team of screenwriters was even writing it, and finally the script "Cracking the Waves" was finalized - it's not because the previous team didn't write well. I also liked the stories that were not broadcast, but "Breaking Waves" is that is most suitable for the current AI technology capabilities.
At the beginning of writing the script, I set several principles:
. Familiar strangeness (in order to allow the audience to have a sense of involvement when watching)
. Simple plot but interesting points in each episode (in order to reduce the viewing frequency) Threshold)
. Short and concise, each episode is 2-3 minutes, 5 episodes in total, no more than 15 minutes per unit (for small steps to run quickly)
. Deliver positive energy (AI technology is inherently bad)
These principles are based on There are several major pain points in AI technology at present:
, character consistency
, scene consistency
, character performance
, action interaction
, range of motion
Half a year ago, this was an insurmountable bottleneck for almost all AI technologies. Under the constraints of these pain points, it is difficult for AI films to express complex narratives with vivid performances, and many action scenes also need to be realized through montage editing. During the production process of
, due to the 3-4 second limit on the length of a single shot (this was the length at the time), the rhythm of the entire film would be affected.
Based on this, our story chose the relationship between mother and child that everyone can relate to, and set "cutting the sea to save the mother" as the main line of the story.
The plot of the story requires some mythical beasts to "star". We found the mythical beasts related to water from "The Classic of Mountains and Seas". The protagonists of Jiuying, Yuyu, Kunpeng, and Water God Gonggong were successively determined.
△ Jiuying
△ Kunpeng
△ Water God Gonggong
The story of young Li Xingzhou, who braved dangers across mountains and seas to save his mother, took shape in this way.I would like to thank Kuaishou Xingmang and Ziranhuo for their tolerance. I sent the script and the reply came back like this:
Really, just three comments. From a creator's perspective, I am very grateful to the platform for such trust. The next step of
is to create pictures and videos. This basic working logic is certain. At this stage, the direct generation of video from text is basically just a toy, and there is no way to deliver it commercially. How does
overcome difficulties in the creative process?
In the Wensheng diagram stage, we clearly put forward a principle that we cannot use any artist's prompt words or any real person's photos as background pictures. This kind of infringement risk will endanger the security of the entire project .
After many rounds of "casting", we finally determined the "fixed makeup photo" of the male lead Li Xingzhou (actually it was just a big look, and the facial details were modified later).
△ The "hero" Li Xingzhou
including Xingzhou Niang, Shui Ye, and Zhong Ye were actually selected through this round of "casting".
△ Boating Girl
△ Shui Ye (the details of the pores on the face are amazing)
But even if we use lora and various auxiliary mat images (the mat images are all generated by ourselves), in today's technology, We still can't achieve 100% perfect character consistency.
△ The appearance of the male protagonist from different angles cannot yet achieve 100% consistency.
This technology may only be solved by digital people in the future, but digital people also involve the problem of integration with the background, and the consistency of the background When it comes to the issue of scene consistency, the solution to scene consistency depends on 3dai (but I learned in this paragraph that it may be solved through other ways), so this becomes a chain within a chain, and each link is difficult to solve. Nine rings of rings.
There is a lot of discussion about the settings of mythical beasts and mythical characters, because the actual descriptions in "The Classic of Mountains and Seas" are relatively limited, such as the turtle fish. The original text: "The body of a fish has the wings of a bird, and its sound is like a mandarin duck. See Then its city is flooded. "
only described the shape of the body, but did not describe the color or appearance. Based on the characteristics of its ferocious beast, including some color combinations combined with traditional Chinese aesthetics, we finally determined that it is a ferocious-looking creature with teeth. The image of a flying fish with a sharp beak, mainly red, and bright colors with a feathery texture.
At the same time, if the audience is careful, they will also find that the consistency of the monsters is not as good as the characters.
This is not because we don’t want to do better, but because the ai model still feeds less material to the monsters than humans, so no matter how we improve the accuracy of the prompt words, it is still difficult Ensure the consistency of monsters just like maintaining the consistency of human faces. The improvement of the consistency of
may depend on the improvement of the consistency function in graphics software like midjourney in the future, or the future implementation of 3dai.
During the Tusheng video stage, from April to May, we tried various technical paths, using various Tusheng video AI software and comfyui to build workflows, and even considered green screen motion capture transfer. , but no matter which one it is, I am not very satisfied.
is either a dynamic ppt, or it cannot be perfectly integrated with the background. No matter how you draw cards or edit, it can never achieve the desired effect. In fact, the first version of
was released as early as the beginning of May, but it has been jumping back and forth among various modifications - . Until June 7, Keling's Wensheng Video turned out to be , a new model under the dit framework. It was a huge shock to the AI film and television industry. The video I made in one hour actually received 100,000 views on social platforms.
Everyone on the Internet is shouting: China's Sora is coming! This made me realize: the next generation of AI film and television is here.
I quickly contacted Kuaishou’s technical department. After a whole afternoon of technical exchanges with Keling’s technical team, I decided: remake!
Even if it needs to be edited again after being remade, and the music and sound effects need to be adjusted, it is not a big deal, because the improvement in image quality that this new model of Keling Tusheng Video can bring is a generational difference. It is reflected in several aspects:
, native high-definition
, no longer sliced screen content driven
, positive response to prompt words
, the main body movement is reasonable, the range of movement is not only large but also smooth
What is even more impressive is that the Keling team not only provides We provide dedicated personnel to work with us and also automate the process. When the final film came out, we all felt that the remake was worth it.
Take this shot in the trailer as an example:
△ The dynamic effect of Water God Gonggong is amazing
Water God Gonggong stood up from the mountain. Not only did the action of standing up be completed, the entire character also realized a three-dimensional turning action. The texture of the muscles and the flowing hair are very suitable, and the effect of the surrounding rocks collapsing is also very shocking.
In the original traditional film and television industry, it would take a long time and a lot of capital to make such a special effects lens; now, it only takes a picture and can be realized through the driver of Keling AI.
After we replaced the new material and pressed the gourd to lift the ladle, there was another problem with the original mouth shape:
The lip shape problem seems to be a simple lip sync, but in fact it is related to character performance. Most lip-syncing technology drivers on the market now target the triangle area of the nose and mouth, but if the driver is only in this area, it will give people a strange feeling of "pictures talking".
Because when a real person speaks, in addition to the movement of the triangle area, the entire body will have a sense of breathing, including changes in micro-expressions on the face. There is no good solution for these in the mature technologies that are now open.
We have also thought of many ways, such as using a certain technology to drive the face first (only close-ups are possible) and then expand the image. Or use face-changing technology, etc., but none of them can achieve the desired effect.
had to turn to the Keling team for help. Fortunately, they have such technical reserves. After several rounds of communication and debugging, we finally got the best lip-sync drive effect at present, which also gave the characters a certain performance. Of course, it is still far from the performance of a truly good actor.
△ ai "lip-syncing"
On July 12, the day before it was released, we did a premiere for the media on the big screen in the theater. At first, I refused...
Although before When doing variety shows, we have also used the big screens in theaters for premieres, but after all, they are real-shot content;
the content produced by AI now can be viewed on mobile phones, and it can cover some weaknesses. Once it is put on the big screen It is like giving a magnifying glass to all viewers, all weaknesses will be exposed.
But the colleagues who promoted it felt that the effect on the big screen was better, and more media were invited. After discussion, we decided to give it a try, so I went to the film post-production company Manying to help me produce a DCP. Bag.
At around 10 o'clock in the evening, when we walked into the theater we selected, I felt for the first time how big the screen was...
After actually watching the entire movie on the big screen, I asked with trepidation Colleague who recommended: Is it okay? They all said: Great! According to my own assessment, 80% of the pictures are currently on the big screen. For the remaining 20%, I plan to kneel down and thank all the media teachers and relatives and friends for their tolerance... For a new species, I hope everyone will give us some time. I believe that if we have a chance to hold a premiere for the next film, I will strive to make all the pictures worthy of this big screen.
has talked so much. I want to conclude the efforts of our team in the past six months and also cut the ribbon for the birth of this new species of AI fantasy short drama. I hope you can connect three times with one click. Thank you for your attention and forwarding. We will next See you in one piece.
