PAIR Public Forum for Research and Innovation: Prof. YANG Hongxia of PolyU delivers "DeepSeek and Beyond"

Abstract

DeepSeek’s recently released model has demonstrated that very strong AI capabilities can be achieved without the need of particularly large models. Their AI models perform in a manner comparable to top AI models in the United States, but require much fewer computing resources. More importantly, they have made this technology sharable through open sourcing. This breakthrough has changed the way how people think about AI development, and has also triggered concerns over privacy security, technological competition and other issues. Our Co-Gen AI project aims to build on this idea to help enhance Hong Kong’s AI competitiveness. Our platform has three innovative features. The first one is Domain-Adaptive Continual Pretraining (DACP), which allows AI models to become more specialised through learning data from specific domains (e.g., industry, scientific research) and demonstrates better training outcomes that open-source models and ChatGPT. The second one is Advanced Model Fusion Infrastructure, which comprises a “Model Fusion” over existing specialised models, significantly reducing the use of computing resources—training a 7B model requires only 64–128 GPU cards, and a 100B large model requires just 512–1,024 cards, thus bringing over 90% resource saving. The third one is Resource-Efficient Architecture, which allows us to leverage ordinary computing resources in Cyberport, Science Park, Pengcheng Lab and other places to develop AI, thus enabling the efficient training of large models by integrating small models, and avoiding the problems in traditional approaches that require large clusters of identical high-end GPUs for training. Through this innovative approach, AI development can be made more accessible and less dependent on massive centralised computational resources. This project expects to help Hong Kong occupy an important position in the global generative AI development landscape.