Abstract: Large language models (LLMs) based on deep learning have achieved tremendous breakthroughs in many applications with increasing model sizes, e.g., BERT with 340 million parameters, GPT-3 with 175 billion parameters, PaLM with 540 billion parameters. However, scaling LLMs requires a linear increase of compute with the model size. Recently, sparsely activated Mixture-of-Experts (MoE) layers are widely exploited to scale the model size to trillions of parameters while improving its generalization ability with only a sub-linear increase of computations. For example, Switch Transformer scales the size of parameters from several billion to 1.5 trillion with 15 MoE layers, each of which has 2048 experts. However, training MoE models on a large-scale GPU/TPU cluster introduces critical performance issues that make the distributed training system extremely inefficient due to the iterative data communication in expert parallelism. This talk discusses sparsely activated Mixture-of-Experts (MoE) LLMs and our recent efforts in accelerating its distributed training through various optimization techniques.

Bio: Bo Li is a Chair Professor in the Department of Computer Science and Engineering, Hong Kong University of Science and Technology. He held the position of Cheung Kong Visiting Chair Professorship in Shanghai Jiao Tong University between 2010 and 2016, and was the Chief Technical Advisor for ChinaCache Corp. (NASDAQ:CCIH), a world leading CDN provider. He made pioneering contributions in multimedia communications and the Internet video broadcast.He and his two PhD students developed Coolstreaming system, which was credited as first large-scale Peer-to-Peer live video streaming system in the world. It attracted significant attention from both industries with over USD 25M investment (2006), and academia received the Test-of-Time Best Paper Award from IEEE INFOCOM (2015).

He is a Fellow of IEEE. He received the Young Investigator Award from the National Natural Science Foundation of China in 2004, and the State Natural Science Award (2nd Class) in 2011. He was a co-recipient of 8 Best Paper Awards from IEEE including IEEE INFOCOM (2021) Best Paper Award. He received his PhD in the Electrical and Computer Engineering from University of Massachusetts at Amherst, and his B. Eng. (summa cum laude) in the Computer Science and Technology from Tsinghua University, Beijing, China.

Abstract: Crowd sensing is a new sensing paradigm that uses individual sensing capability to accomplish complex social sensing tasks. In this speech, I will introduce our work in crowd sensing in various aspects, such as theory, methods, and platform. Furthermore, I will present the main idea of crowd sensing 2.0, including the featues and enabling technologies of heterogeneous crowd sensing.

Bio: Dr. Zhiwen Yu is currently a vice president of Harbin Engineering University, China and a professor of Northwestern Polytechnical University, China. He has worked as an Alexander Von Humboldt Fellow at Mannheim University, Germany from Nov. 2009 to Oct. 2010, a research fellow at Kyoto University, Japan from Feb. 2007 to Jan. 2009, and a post-doctoral researcher at Nagoya University, Japan in 2006-2007. His research interests cover Ubiquitous Computing, Internet of Things, and Crowd Sensing and Computing. He is the Editor-in-Chief of CCF Transactions on Pervasive Computing and Interaction. He has served as an associate/guest editor for a number of international journals, such as IEEE Transactions on Human-Machine Systems, IEEE Communications Magazine, and ACM Transactions on Intelligent Systems and Technology. He received the CCF Young Scientist Award in 2011, the Humboldt Fellowship in 2008, and the CCF Excellent Doctoral Dissertation Award in 2006. He got the National Science Fund for Distinguished Young Scholars in 2017.

Abstract: With the tremendous growth of Internet videos and their analytics techniques, various challenges emerge particularly those concerning mobility and analytics efficiency. How to mitigate these challenges and how can we embrace the opportunities in this era? In this talk, I will introduce some of our recent efforts and outline a vision towards integrating sensing, networking and data analytics in a supporting infrastructure in the digital world.

Bio: Xiaoming Fu received his Ph.D. degree in computer science from Tsinghua University, China, in 2000. In 2002 he joined the faculty of the University of Göttingen, Germany where he became a Professor and the Head of the Computer Networks Group in 2007. He directs the Sino-German Institute for Social Computing at the University of Göttingen and is a principal scientist at the Center for Internet & Society, Fudan University, China. He has also held visiting positions at ETSI, University of Cambridge, Columbia University, Tsinghua University, Sydney, Uppsala, UMPC and UCLA. Prof. Fu's research interests include network architectures, protocols, and applications.

He is an IEEE Fellow, an ACM Distinguished Member, and a member of Academia Europaea, and is currently an editorial board member of IEEE Network and IEEE Transactions on Network and Service Management, and co-editor-in-chief of the Journal of Social Computing. He has served on committee conferences of leading conferences such as SIGCOMM, CoNEXT, MOBICOM, MOBIHOC, INFOCOM, ICNP, ICDCS, IWQoS and COSN. He has served as secretary (2008-2010) and vice chair (2010-2012) of the IEEE Communications Society Technical Committee on Computer Communications (TCCC), and chair (2011-2013) of the Internet Technical Committee (ITC) of the IEEE Communications Society and the Internet Society. He has been a PI in the EU FP6 ENABLE, VIDIOS, Daidalos-II, MING-T, FP7 GreenICN, H2020 COSAFE, HE CODECO and COVER projects, as well as the coordinator of EU FP7 MobileCloud, GreenICN and CleanSky projects and H2020 ICN2020 project.