Longbo Huang (Tsinghua University)
Title: Double Quantization for Communication-Efficient Distributed Optimization
Abstract:
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose double quantization, a general scheme for quantizing both model parameters and gradients. Three communication-efficient algorithms are proposed under this general scheme. Specifically, (i) we propose a low-precision algorithm AsyLPG with asynchronous parallelism, (ii) we explore integrating gradient sparsification with double quantization and develop Sparse-AsyLPG, (iii) we show that double quantization can also be accelerated by momentum technique and design accelerated AsyLPG. We establish rigorous performance guarantees for the algorithms, and conduct experiments on a multi-server test-bed to demonstrate that our algorithms can effectively save transmitted bits without performance degradation.
Bio:
Dr. Longbo Huang is an associate professor at the Institute for Interdisciplinary Information Sciences (IIIS) at Tsinghua University, Beijing, China. He received his Ph.D. in EE from the University of Southern California, and worked as a postdoctoral researcher in the EECS dept. at University of California at Berkeley before joining IIIS. Dr. Huang currently serves as an editor for IEEE Transactions on Communications (TCOM), and an associate editor for ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) and IEEE/ACM Transactions on Networking (TON). Dr. Huang has held visiting positions at the LIDS lab at MIT, the Chinese University of Hong Kong, Bell-labs France, and Microsoft Research Asia (MSRA). He was a visiting scientist at the Simons Institute for the Theory of Computing at UC Berkeley in Fall 2016. Dr. Huang received the Outstanding Teaching Award from Tsinghua university, the Google Research Award and the Microsoft Research Asia Collaborative Research Award in 2014, and was selected into the MSRA StarTrack Program in 2015. Dr. Huang won the ACM SIGMETRICS Rising Star Research Award in 2018.
Dr. Huang’s current research interests are in the areas of stochastic modeling and analysis, reinforcement learning and control, optimization and machine learning, and big data analytics.
Javad Ghaderi (Columbia University)
Title: Bin Packing with Queue: Scheduling Resource-Constrained Jobs in the Cloud
Abstract:
Many problems of both theoretical and practical importance that arise in optimizing stochastic networks are hard combinatorial problems. This talk will focus on a natural combinatrial scheduling problem which arises in distributed computing frameworks. Jobs with diverse resource requirements (e.g. CPU, memory) arrive over time and must be served by a cluster of servers. To improve throughput and delay, the scheduler can pack as many jobs as possible in the servers, however the sum of the resource requirements of the jobs placed in a server cannot exceed its capacity. We are interested in scalable scheduling algorithms that can provide optimal or near-optimal throughput guarantees with low complexity. We first show that greedy bin packing heuristics are not throughput optimal, but Best-Fit can be modified to achieve at least 1/2 of the optimal throughput. Then we propose randomized scheduling algorithms that can provably achieve maximum throughput with low complexity. The algorithms are naturally distributed and each server needs to perform only a constant number of operations per time unit. Time permitting, I will overview another combinatorial scheduling problem related to scheduling deadline-constrained packets in wireless networks. The talk will demonstrate the power of randomization for solving combinatorial scheduling problems in various stochastic networks.