"Love Never Ends", China Central Radio and Television's first drama-style mini-series that combines AI with real-person performances, was refreshing after its broadcast. The whole drama uses AI technology to generate intentional scenes. The pictures are beautiful and delicate, and the plot has ups and downs. It tells the story of a family facing sudden changes and using artificial intelligence to create digital human videos to soothe the soul. What is striking is that Liu Ze, Liu Yijia, Zhang Yiqin, and Zhang Lingze, a student team from the School of Experimental Art of Tianjin Academy of Fine Arts, under the leadership of Professor Liu Shuming, participated in the entire creative and production work of the play's AI generation. This art practice is an industry exploration of AI-generated art by the Science and Technology Art Major of Tianjin Academy of Fine Arts. What kind of exploration did this team of teachers and students make during this production process? What valuable practical experience has been accumulated in AI generative art? A reporter from Tianjin Daily conducted an exclusive interview with Professor Liu Shuming from the Department of Science and Technology, School of Experimental Art, Tianjin Academy of Fine Arts.
From space scenes to facial expression details, enrich the branches of AI-generated videos
In the mini-series "Love Never Ends", the male protagonist Baohua is middle-aged and an excellent corporate engineer. He is going to celebrate the diamond for his old mother. On the way to their wedding anniversary, Baohua's son Ajian and his wife Guiying died while trying to save others. They were worried that the elderly could not bear such a huge blow, so they used artificial intelligence to create digital human videos to soothe the hearts of their loved ones, showing the tenacity and love of three generations of a family. The story of the endless power of love. The play is joined by powerful actors Wei Qing, Tao Huimin, Cao Li, and Chen Guanning.
Professor Liu Shuming of Tianjin Academy of Fine Arts led a team of students to undertake part of the creative tasks in the play, including using aigc (artificial intelligence generated content) technology to generate storyboard scripts, space scenes, real-person photos, virtual props, subjective and objective aerial shots, and film. Name, logo, etc. The team used AIGC technology to generate an image of the father who died unexpectedly, and used technology to make up for the lost family ties and make love last in the virtual world, adding strong technological attributes and humanistic connotations to the play.
"At the beginning of creation, we first determined the visual tone of the film based on the storyline. The story takes place in early spring. Then, we considered that the overall color tone of this season in the north is cooler, while the environmental tone in the south at this time is warmer. We put the story The environment was set to a city in southern China, and then we began to collect visual reference pictures that matched it. Finally, we communicated with the director to determine the visual tone of the entire short film. At the same time, these pictures also provided the basic vision for subsequent AIGC generation. "Professor Liu Shuming introduced, "Using AIGC to generate spatial scenes is mainly to match the story line development of the characters in the plot, and generate corresponding indoor and outdoor scenes and room furniture objects based on the identity, age, occupation, lifestyle, etc. of the characters in the plot. Furnishings, including living rooms, kitchens, bedrooms, hospital corridors, wards, aerial day and night views of urban spaces, etc., play a role in promoting the development of the plot and increasing visual expression. For example, the scene at grandma’s house was determined through repeated generation and debugging. The old houses in the southern towns are simply decorated, and there are some old objects in the houses, which are suitable for the living scenes of the elderly. They provide lighting, composition, scenery, props, camera movements and other pre-performance support for the subsequent performances. "
" In order to meet the shooting requirements. Based on the front view of the selected scene, Professor Liu Shuming led the team to generate other storyboard scenes in different scenes, different angles and different times while maintaining the consistency of the scene. Realizes the generation of multi-scenery background sets for the same scene. At this stage, AIGC's detail processing capabilities are average. In order to pursue the consistency of details, students collaborated with multiple image processing software for deep processing to achieve refined detail effects.
During the creation process, it was necessary to generate photos of different age groups for the key role of grandma in the play, including black and white photos of her youth, and in order to celebrate grandma’s diamond wedding in the play, AI was used to restore grandma’s wedding dress when she was young. According to.The real problem is that the age of actors is irreversible. In traditional filming, actors of different ages who look similar will be invited to play the same character in different life stages. However, with the assistance of AIGC technology, the team finally made many attempts and adjustments. Using only one candid photo of the actor, photos of different eras and ages were generated. Including only using the portrait of the male protagonist Baohua in the play and the African work photos on the Internet, the work photos of Baohua and African workers in the same frame were generated.
ai assists creation in the entire film and television process, greatly improving the fault tolerance of shooting work
Professor Liu Shuming has participated in the creative work of virtual images in many CCTV documentaries for many years. However, this is the first time for students to participate in CCTV program production. In early April this year, when Director Hu Zhaohui was preparing for CCTV's first "AI + real person" short drama "Love Never Ends", he happened to learn that Liu Shuming was leading students to conduct teaching practices related to AIGC, so after in-depth communication , the two parties finally reached this cooperation. Professor Liu Shuming said: "The work process of this project is different from other crews. There are almost no precedents for reference. Its experimental and exploratory value is self-evident. Young students of Tianjin Academy of Fine Arts have high artistic accomplishment and innovative ability, especially I have strong comprehensive learning ability and the courage to try challenges, which is also what CCTV and the director team value. In addition, the professionalism and tacit understanding I have established with them over many years of cooperation have led to this cooperation through various opportunities. I Leading four students to participate in this very challenging CCTV project "
" In this production, Professor Liu Shuming and the student team faced four challenges.
The first point is how to apply the virtual images generated by aigc to real-life film and television dramas, and how to organically combine the two. For the team, and even for the entire film and television industry, the method of combining AIGC technology with real-person shooting is still blank in China, and CCTV is exploring it for the first time. Therefore, there is a relative lack of reference materials and practical experience for the team. How to make AIGC technology better intervene in the workflow of the film and television industry and integrate and innovate? The team is faced with the challenge of exploring, correcting, and even trial and error in the process.
The second challenge faced by the team is that, unlike the traditional shooting workflow, in the past, photography would have greater room for creativity. Now, photography must be highly coordinated with the angle, scene, and perspective relationship of the scene generated by AI to ensure that the later stage The smoothness of editing. Therefore, the team needs to gradually explore a working model that is highly nested and coordinated with various types of work on the crew.
The third point is how to seamlessly combine the virtual scenes generated by aigc with real-life shots, and the generated scenes must be natural enough and consistent with the role positioning of the characters themselves and the background setting of the entire script. This requires the team to summarize the characteristics of different characters before actually generating the scene, and design the unique elements and characteristics of the scenes where different characters appear. After having the design direction of the scene, the students continued to adjust the parameters generated by AIGC, and finally achieved a perfect presentation with a high degree of matching between the real shots of the characters and the AIGC generated shots.
The fourth challenge is that the creation time is very tight. From joining the team to prepare software and hardware, to cooperating with the whole process of photography, lighting, costumes, props, and makeup, to post-editing, there are only ten days.
Liu Shuming said: “Through the discussions and practices of the students day and night, we finally figured out a complete set of workflows, and also introduced a variety of software collaboration methods to solve the problems of single aigc technology in image details and screen consistency. Defects. This complete work process also greatly improved the production efficiency of the project through the students' reasonable division of labor and cooperation, and completed the production tasks on time."
" This artistic practice has accumulated valuable practical experience in AI-generated art. Professor Liu Shuming said, "Our work, in a nutshell, is to explore the full-process AI-assisted creation in the film and television shooting process; we have accumulated AIGC technology in art Practical experience in the industry beyond expression; explored the multiple possibilities and workflows of combining AIGC technology with real shooting; overturned the practical limitations of traditional setting requiring a large number of props and space. With the help of AIGC technology, any scene can be transformed They can be produced and replaced more efficiently, and the final composite effect can be seen in real time during the shooting process, which greatly improves the fault tolerance of the shooting work. "
" The team actively explored the artistry and emotional expression of aigc, followed the principle of echoing the protagonist's mood more delicately, gave full play to the unconstrained expressiveness and creativity of aigc technology, and created multiple subjective empty shots. The background or subjective empty lens of the character's emotions and mood strengthens the rendering of the protagonist's mood. "
" For example, when expressing the joy of grandma waiting for the arrival of her son, in order to further enhance the warm and tranquil atmosphere, through aigc technology, the production team generated a kitten on grandma's bedside and interacted with grandma, which is consistent with the character's environment. The setting also increases the flexibility of the lens. In order to highlight Ajian's sadness after losing his father, the final picture forms a rainy night scene. The shadows of the trees on the rainy night are reflected in Ajian's window, which not only increases the lonely mood of the window, but also highlights the loneliness of the window. The image of Ajian alone made the audience resonate with the character. After the grandmother gradually relieved the news of her son's death, she used the apricot blossoms blooming outside the window to symbolize hope.
"Through these explorations, we have accumulated many valuable practices. Experience, how to optimize AI generation parameters, how to deal with details in the generation results, how to integrate with real-shot materials, etc. This practical experience also laid a solid foundation for our future creation and research. "Professor Liu Shuming said.
Break the limitations of traditional creation and create new visual effects
The Science and Technology Art Major of the School of Experimental Art of Tianjin Academy of Fine Arts not only provides students with professional knowledge, but also encourages and practices the integration of the latest scientific and technological means with artistic creation concepts. Combining technology and art to explore more diverse application fields. In this production practice of CCTV, we have achieved many innovations and transcendences. "In this practical project, we combined aigc technology with traditional film and television. The combination of production and pre-production breaks the limitations of traditional creation and creates new visual effects. In the application of AIGC technology, we have carried out many optimizations and innovations, and tried to use different software technologies, such as Tusheng pictures, Tusheng videos, Vincent videos, targeted training of lora models, etc., to make the generated effects more realistic and vivid . At the same time, in artistic creation, we pay more attention to achieving unique artistic expression through AI technology, so that the works have both technical advantages and artistic beauty. "Professor Liu Shuming said.
Liu Ze, a 2021 graduate student at the School of Experimental Art, Tianjin Academy of Fine Arts, is one of the main creative members of this project. Liu Ze sighed with emotion: "As an art practitioner who continues to create in the field of digital media art, I always I am all experiencing the relationship between technology and art. In this practice of aigc technology, I realized this more deeply: the collision between aigc and film and television micro-short dramas has gone far beyond the effect of '1+1>2', and even It brings geometric efficiency improvements to the workflow of the traditional film and television industry. We only need to generate scenes based on the established script settings. In just one to two days, we can iterate dozens of plans, and the generated images only need to be fine-tuned with local details before they can be put into actual film and television productions. Synthesis work in progress. For situations where the financial budget for such micro-short dramas is insufficient, aicg technology will also have greater room for development, which will undoubtedly have a huge impact on our traditional workflow. "
" "In addition to optimizing scene costs, aigc even has great potential in improving shooting skills and actor performances.For example, in a car accident scene that was filmed at night, an elderly man who was temporarily in place as extras had to show the panic expression before being hit by a speeding car. This close-up shot needed to reflect the suddenness of the situation through the actor's facial expressions. Explosiveness and sense of crisis. However, after the filming, the expressions of the elderly actors did not achieve the expected performance feeling. Due to time and many other reasons, this footage could not be reshot. Therefore, team members proposed using aigc's graph-generating graph technology to optimize the image based on the original image. We extracted single frames of group actors' expressions, optimized the actors' facial expressions through prompt words, and used the character consistency function to preserve the actors' original facial features as much as possible. Finally, after a few iterations, a facial close-up with a very vivid expression of surprise was completed. Finally, we exported this frame as a close-up shot through Tusheng video, which perfectly solved the problem of the actor's expression not being in place. "Classmate Liu Ze still remembers this.
ai technology promotes the intelligent transformation of the film and television industry, and scientific and technological art works need to adhere to emotion and temperature
Nowadays, the combination of AI technology and content creation breaks the traditional boundaries of image creation, presenting A new visual style and narrative method. Has the use of AI technology in film and television become an unstoppable trend?
In this regard, Professor Liu Shuming believes that, “In this practice, Our team deeply feels the power of AIGC as a new technological force in empowering future film and television production. It has obvious advantages in special effects production, scene generation, real-time preview, etc. It also saves a lot of manpower, material resources and time costs, eliminating the fatigue of traveling to multiple places for shooting. At the same time, it also breaks the restrictions of shooting seasons, shooting locations and equipment, and can generate swallows on branches in spring even in winter. The whispering scene achieves maximum cost reduction and efficiency increase. Moreover, AI models have the ability to continuously learn from massive data, which continuously provides new inspiration and ideas for the film and television industry, helping to create more innovative and market-competitive works. The application of AI technology in the film and television industry is promoting the transformation of the entire industry into intelligence. "
" "Of course, the technical limitations of AIGC's empowerment of film and television are also obvious. For example, the widespread application of AIGC technology may cause employment pressure for some employees, leading to the replacement of related positions. In addition, copyright issues related to the content it generates also need to be resolved urgently. More importantly, the content generated by AI may have deficiencies in emotional expression and creative uniqueness. "
Professor Liu Shuming said that the charm of film and television creation has not diminished after more than a hundred years since its birth, because it carries human emotions and dreams. It is an art form full of humanistic care and aesthetic value. Although aigc technology can Improve production efficiency and effects. However, if in all future film and television creation processes, actors perform in a green screen environment without real emotions, and the audience watches the movie with subjective presets virtually generated by AIGC, then the image Is it still the indispensable spiritual link that carries countless people’s dreams and emotions? “Therefore, I think the more advanced technology is, the more we must hold on to the emotion and warmth of technological art works, think deeply about technological humanity, advocate technology for good, and be full of moderation. Make full use of AIGC technology and carefully protect the timeless morality and emotions of mankind under the cloak of technology, in order to achieve the original intention of scientific and technological progress to benefit mankind. "(Tianjin Daily reporter: Guo Xiaoying)
