On the first anniversary of the release of the "Tiangong" large model, Kunlun Worldwide announced that the "Tiangong 3.0" base model and the "Tiangong skymusic" music model have officially launched public beta.
Since AI allowed humans to achieve freedom in music creation, even quarrels have become interesting.
Some time ago, Aran Komatsuzaki, a well-known AI blogger on the You know, in the past, the war of words between these big guys was mainly to post a post, and then you and I would follow up. This time, aran komatsuzaki's approach can be said to have played a new trick. I don't know if it was inspired by "Thank you, thank you, I want to diss you".
Like aran komatsuzaki, many people at home and abroad who know music but do not understand music are trying out AI music creation tools such as suno, and have generated many very interesting music works.
However, many netizens have reported that the Chinese generated by suno is sometimes unstable, and there are problems such as Chinese songs having an English feel and uncommon words being sung incorrectly.
b website netizens commented on a Chinese song generated by suno. Video address: https://b23.tv/gvqtuou
So, is there an AI music generation model specifically optimized for Chinese?
Kunlun Worldwide’s “Tiangong skymusic”, which is open to the public for public beta testing today, is such a model. The Chinese vocals it generates are clear, authentic, and have no abnormal sounds, and there are no acclimatization problems such as "Broadway-style Chinese songs". Moreover, it is not only optimized for Mandarin, but also takes care of dialects such as Cantonese, Chengdu, and Beijing dialect.
video link: https://mp.weixin.qq.com/s?__biz=mza3mzi4mjgzmw==&mid=2650914847&idx=1&sn=c41a2beca4285222d1aa5b8764097c55&chksm=84e40461b3938d773 9f50e17b0313bfb461b268bccd3a5c1ef0f87f6dbf88c4c93b7e59dd081&token=1501617723&lang=zh_cn#rd
So, how does Tiangong skymusic perform compared with suno? ? Horizontal evaluation data shows that in terms of vocal and BGM sound quality, vocal naturalness, pronunciation intelligibility and other indicators, Tiangong SkyMusic is better, and its overall performance surpasses Suno V3, becoming China's first music AIGC sota The model also made China's self-developed large model technology lead the world in the AIGC field for the first time.
Such excellent performance is naturally inseparable from the powerful base model, that is, the large model "Tiangong 3.0" released and open sourced by Kunlun Wanwei at the same time. This model has 400 billion parameters, surpassing grok-1 with 314 billion parameters, and is the world's largest open source moe model.
In many authoritative multi-modal evaluation results such as mmbench, "Tiangong 3.0" surpassed gpt-4v and took the lead in the world.
With the support of this base model, skymusic has a deeper understanding of songs. It can control emotional changes through lyrics and implement various singing techniques such as vibrato, opera, chanting, etc., making the generated musical works more emotionally rich and suitable for the situation.
Audio link: https://mp.weixin.qq.com/s?__biz=mza3mzi4mjgzmw==&mid=2650914847&idx=1&sn=c41a2beca4285222d1aa5b8764097c55&chksm=84e40461b3938d7739f 50e17b0313bfb461b268bccd3a5c1ef0f87f6dbf88c4c93b7e59dd081&token=1501617723&lang=zh_cn#rd
So, how to use this model specifically? What is the technical route? What’s innovative about “Tiangong 3.0” behind it? Let’s look at them one by one.
The first domestic music sota model with unlimited ways to play
In fact, it is very simple to use skymusic to generate songs: you only need to enter the song title, lyrics, and select a reference track, and it will generate songs with similar styles and singing styles.
If you don’t want to write lyrics yourself, you can also try the “ai lyrics writing” function in the lower right corner of the input box. It can be written from the first sentence, and only one sentence is generated at a time. Unsatisfactory sentences can be deleted in time until the entire song is completed.
Of course, you can also try to use "Tiangong 3.0" to write songs. For example, this song "Heart of the Machine" was written by us using "Tiangong 3.0":
The next step is to choose a reference song, which is also Tiangong skymusic The unique feature is the ability to generate music based on sample sound sources.
In this step, Tiangong skymusic provides many reference tracks, you can choose from them, or you can choose to upload song files. Here, we have uploaded a song by Luo Tianyi to see how the generation effect is: This ability to generate music based on sample sound sources has greatly enriched the gameplay of SkyMusic. In the user works display area, we can see that "New Man" (the episode of the movie "Zhou Chu Eliminates Three Evils") has five versions including ancient rock and DJ. During the trial, we also discovered that, in fact, the music generated by Tiangong skymusic covers rap, folk, funk, ancient style, electronic and other genres. In the next step, the team also plans to allow users to generate songs based on hummed melodies, which will be of great help to professionals.
Video link: https://mp.weixin.qq.com/s?__biz=mza3mzi4mjgzmw==&mid=2650914847&idx=1&sn=c41a2beca4285222d1aa5b8764097c55&chksm=84e4 0461b3938d7739f50e17b0313bfb461b268bccd3a5c1ef0f87f6dbf88c4c93b7e59dd081&token=1501617723&lang=zh_cn#rd
Currently, skymusic is fully open, download "天公skymusic" You can experience it through the "Work" app. This is the only large-scale AI music generation model currently publicly available in China, and its emergence fills the gap in domestic AIGC tools in this field.
Although this model is still in its infancy, it has already made many people feel the joy of music creation. Everyone uses it to transform divine comedies, create second hits, rewrite ancient poems to support education... and develop various new directions for music creation.
Self-developed sora architecture, the technical roadmap has been made public
Skymusic is an end-to-end music generation model, so it feels very simple for us to use it. However, the development of the entire model is not that simple.
First of all, from a technical perspective, Skymusic chose the large-model music audio generation route, which means that it directly learns and generates audio waveforms instead of using symbolic music generation routes (such as midi) to generate musical scores. This approach allows for integrated end-to-end generation of elements such as instruments, vocals, melodies, volumes, notes, etc., providing a more direct, higher-quality music creation experience. However, this direction is also more difficult and requires high computing power and funds, so there are very few people doing it.
Moreover, in this direction, there are even fewer people who are determined to conquer the field of "vocal songs". Most research focuses on the field of BGM without voices, because there is almost no open information or open source models for reference in the former.
Faced with these pressures, Kunlun Wanwei conducted countless research and development experiments, invested a lot of computing power, built a data set containing 20 million songs (the largest music data set in human history), and finally explored a method with good results. , reproducible solutions. Moreover, they also made the technical schematic diagram of this solution public.
Tiangong skymusic technical schematic diagram: the large-scale transformer is responsible for composing music, learning the contextual dependencies of music patches, and at the same time achieving music controllability; the diffusion transformer is responsible for singing, allowing music patches to be restored to high-quality audio through ldm. This model architecture works extremely well when processing video, audio, and music.
As can be seen from the picture, the framework of Skymusic is a dit technology path similar to sora, but the development time was before sora came out, so it is inevitable to step on many pitfalls.
For the industry, this schematic diagram is very valuable, because there is no large-scale AI music model company on the market that has disclosed its own technical path, including Suno, and Kunlun Wanwei is the only one.
The powerful base behind it - Tiangong 3.0
The success of skymusic is inseparable from the base model behind it - Tiangong 3.0. Its core upgrade is reflected in "independent thinking".This is reflected in the model's new multi-round search and comprehensive tool calling, chart drawing, research mode, enhancement mode and other functions.
is given a data statistics task, it can not only help you collect all the data, but also write your own code and call various functions to draw charts. Each intermediate step has been dismantled in a clear manner, and subsequent execution requires little human intervention. Even details such as "avoiding label overlap" and "centering text display" have been taken into consideration. This is the embodiment of the ability of "independent thinking".
video link: https://mp.weixin.qq.com/s?__biz=mza3mzi4mjgzmw==&mid=2650914847&idx=1&sn=c41a2beca4285222d1aa5b8764097c55&chksm=84e40461b3938d7739f 50e17b0313bfb461b268bccd3a5c1ef0f87f6dbf88c4c93b7e59dd081&token=1501617723&lang=zh_cn#rd
This kind of "independent thinking" ability cannot be improved without "Tiangong 3.0" is optimized in terms of semantic understanding and logical reasoning. Compared with the previous generation "Tiangong 2.0" moe large model, "Tiangong 3.0" has amazing performance improvements in areas such as model semantic understanding, logical reasoning, versatility, generalization, uncertainty knowledge, and learning capabilities. The model technical knowledge ability has increased by more than 20%, and the mathematics/reasoning/coding/cultural and creative abilities have increased by more than 30%.
takes the search task as an example. In the "Search Enhancement" mode, given a simple search request, "Tiangong 3.0" will not only give a summary answer, but also refine some important information into charts.
video link: https://mp.weixin.qq.com/s?__biz=mza3mzi4mjgzmw==&mid=2650914847&idx=1&sn=c41a2beca4285222d1aa5b8764097c55&chksm=84e40461b3938d7739f 50e17b0313bfb461b268bccd3a5c1ef0f87f6dbf88c4c93b7e59dd081&token=1501617723&lang=zh_cn#rd
In "Research" mode, it also provides a The "In-depth Research" module allows you to discuss extended issues not mentioned in the search query, giving you the feeling of reading a paper. Finally, it also organizes this information into a mind map for quick reference.
video link: https://mp.weixin.qq.com/s?__biz=mza3mzi4mjgzmw==&mid=2650914847&idx=1&sn=c41a2beca4285222d1aa5b8764097c55&chksm=84e40461b3938d7739 f50e17b0313bfb461b268bccd3a5c1ef0f87f6dbf88c4c93b7e59dd081&token=1501617723&lang=zh_cn#rd
has greatly improved semantic understanding and logical reasoning capabilities. In addition, “Tiangong 3.0” also conducted special training on the model’s ability to independently plan, call, and combine external tools and information. By independently planning and calling and combining external tools and information, it can help you accurately and efficiently complete various complex needs such as industrial research, product reviews, and information analysis.
This ability to think independently is crucial for large artificial intelligence models. Firstly, this ability enables AI to conduct autonomous reasoning in the absence of direct instructions, improving its ability to handle complex problems; secondly, independently thinking AI models can design innovative solutions to meet personalized and scenario-based needs. demand; finally, this ability enables AI to continuously optimize its performance through self-learning and adaptation when encountering novel or changing environments. The cumulative effect of these three aspects has greatly promoted the breadth and depth of application of AI technology, making it more intelligent and efficient in a variety of practical applications.
"Tiangong 3.0" includes many large model capabilities such as ai music, ai search, ai writing, ai painting, etc. When talking about the relationship between the 400 billion parameter moe large model "Tiangong 3.0" and Tiangong skymusic, Kunlun Wanwan Fang Han, chairman and CEO of Weiwei, explained, "Everyone knows that the AI base large model is the solid foundation of AIGC, especially the text large model. The capability basis of Vincent Picture, Vincent Music and Vincent Video (these AIGC models) are all text large models . If the text model capability is not strong enough, the aigc capability will be greatly limited. "This effect of
is also reflected in the AI painting capabilities of "Tiangong 3.0". "Tiangong 3.0" has added new functions such as image size expansion, image orientation adjustment, mat image generation, mat image evolution, and mat image expansion, and the actual test results are excellent.
"Our 400 billion large model is the base model that provides support for all our C-end products. The better my base model is, the better my music, games, videos and animation products will be. . So we have a very strong motivation to make large base models." Fang Han said.
realizes general artificial intelligence,
allows everyone to better shape and express themselves
In the vision of agi, we often hear some ai business executives mention that they want to use ai tools to improve the productivity and efficiency of human society . Therefore, they mostly focus on the expansion and enhancement of model intelligence. But in Fang Han’s view, an important issue has been overlooked, that is, how to use AI to help people better understand and express emotions.
In the music works section of SkyMusic, we have seen many such examples: the sorrow of students about to graduate, the sadness of young people unable to love, and the fatigue of middle-aged people supporting their families are all expressed through singing. This is the real way of "clearing one's will through song".
The support for dialects is more like a kind of cultural equality, which is something Fang Han values very much. In the future, they hope to add more languages so that every person speaking a small language can easily create their own cultural content.
"In the aigc field, our ambitious goal is to hope that everyone in the world can create content equally. We want to lower the creative threshold for everyone so that everyone can better shape and express themselves." Fang Han express.
Recently, these contents were also written into Kunlun Worldwide’s latest mission.
In fact, this approach also makes sense commercially. "Once everyone can create music, I believe that any public place, such as every restaurant and bar, can create their own background music to meet their own business needs." Fang Han explained.
With continued optimization and improvement in the future, Skymusic will gradually evolve into a professional and easy-to-use music creation platform for everyone.
Of course, Kunlun Wanwei’s efforts are not limited to music. Based on "Tiangong 3.0", they have formed six major AI business matrices. In the future, these matrices will form an AI UGC platform.
This platform can not only help ordinary people express themselves, but also help creators who want to use AI to create content to complete the fully closed loop of IP creation. This closed loop is centered on "good stories (IP)" and spans various forms such as text, comics, music and videos. Consumers' content consumption is also completed on this platform. This is the business logic built by Kunlun Wanwei .
"Our essence is to allow more people to join the team of creators. The premise is that you must be able to tell a good story. If you can create a good IP, then you can create content. It is foreseeable that, All content industries will be redone. The number of creators will expand hundreds of times, and the content that can be consumed will also expand a hundred times. This is the logic of our "all in agi and aigc" strategy." Fang Han said.
What will this era be shaped like? Let’s wait and see
