The industrial collaborative paper entitled “Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training” is selected as a highlight at the IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR) 2023. With more than nine thousand of submissions, only the top 2.5% of them are selected as highlights. This paper was the outcome of the collaboration between , Mr LIN Junfan, COMP Research Assistant and the first author of this paper; Prof. CHEN Changwen, our Chair Professor; and other five collaborators from PolyU, Sun Yat-sen University and Huawei Cloud respectively. The project is supported by the funding sponsored from Huawei Technologies Co. Ltd to PolyU.
This paper addresses the emerging and challenging problem of text-to-motion generation, which aims to synthesize motion with the same semantics as the input text. However, due to the lack of diverse labeled training data, most approaches either limit to specific types of text annotations or require online optimizations to cater to the texts during inference at the cost of efficiency and stability.
In this paper, an offline open-vocabulary text-to-motion generation in a zero-shot learning manner that neither requires paired training data nor extra online optimization to adapt for unseen texts is investigated. Inspired by the prompt learning in NLP, the paper pretrains a motion generator that learns to reconstruct the full motion from the masked motion. During inference, instead of changing the motion generator, the method proposed in this paper reformulates the input text into a masked motion as the prompt for the motion generator to ``reconstruct'' the motion. Read the paper here.
CVPR is a premier annual computer vision event since 1985, which is regarded as one of the most important conferences in its field. The conference is highly selective in paper acceptance, and only the papers with the highest impact would be selected by the programme committee as the highlights or for oral presentations. This year, CVPR 2023 will be held from 18 to 22 June at the Vancouver Convention Center in Canada.
