This book investigates compressive sensing techniques to provide a robust and general framework for network data analytics. The goal is to introduce a compressive sensing framework for missing data interpolation, anomaly detection, data segmentation and activity recognition, and to demonstrate its benefits. Chapter 1 introduces compressive sensing, including its definition, limitation, and how it supports different network analysis applications. Chapter 2 demonstrates the feasibility of compressive sensing in network analytics, the authors we apply it to detect anomalies in the customer care call dataset from a Tier 1 ISP in the United States. A regression-based model is applied to find the relationship between calls and events. The authors illustrate that compressive sensing is effective in identifying important factors and can leverage the low-rank structure and temporal stability to improve the detection accuracy. Chapter 3 discusses that there are several challenges in applying compressive sensing to real-world data. Understanding the reasons behind the challenges is important for designing methods and mitigating their impact. The authors analyze a wide range of real-world traces. The analysis demonstrates that there are different factors that contribute to the violation of the low-rank property in real data. In particular, the authors find that (1) noise, errors, and anomalies, and (2) asynchrony in the time and frequency domains lead to network-induced ambiguity and can easily cause low-rank matrices to become higher-ranked. To address the problem of noise, errors and anomalies in Chap. 4, the authors propose a robust compressive sensing technique. It explicitly accounts for anomalies by decomposing real-world data represented in matrix form into a low-rank matrix, a sparse anomaly matrix, an error term and a small noise matrix. Chapter 5 addresses the problem of lack of synchronization, and the authors propose a data-driven synchronization algorithm. It can eliminate misalignment while taking into account the heterogeneity of real-world data in both time and frequency domains. The data-driven synchronization can be applied to any compressive sensing technique and is general to any real-world data. The authors illustrates that the combination of the two techniques can reduce the ranks of real-world data, improve the effectiveness of compressive sensing and have a wide range of applications.
The networks are constantly generating a wealth of rich and diverse information. This information creates exciting opportunities for network analysis and provides insight into the complex interactions between network entities. However, network analysis often faces the problems of (1) under-constrained, where there is too little data due to feasibility and cost issues in collecting data, or (2) over-constrained, where there is too much data, so the analysis becomes unscalable. Compressive sensing is an effective technique to solve both problems. It utilizes the underlying data structure for analysis. Specifically, to solve the under-constrained problem, compressive sensing technologies can be applied to reconstruct the missing elements or predict the future data. Also, to solve the over-constraint problem, compressive sensing technologies can be applied to identify significant elements
To support compressive sensing in network data analysis, a robust and general framework is needed to support diverse applications. Yet this can be challenging for real-world data where noise, anomalies and lack of synchronization are common. First, the number of unknowns for network analysis can be much larger than the number of measurements. For example, traffic engineering requires knowing the complete traffic matrix between all source and destination pairs, in order to properly configure traffic and avoid congestion. However, measuring the flow between all source and destination pairs is very expensive or even infeasible. Reconstructing data from a small number of measurements is an underconstrained problem. In addition, real-world data is complex and heterogeneous, and often violate the low-level assumptions required by existing compressive sensing techniques. These violations significantly reduce the applicability and effectiveness of existing compressive sensing methods. Third, synchronization of network data reduces the data ranks and increases spatial locality. However, periodic time series exhibit not only misalignment but also different frequencies, which makes it difficult to synchronize data in the time and frequency domains.
The primary audience for this book is data engineers, analysts and researchers, who need to deal with big data with missing anomalous and synchronization problems. Advanced level students focused on compressive sensing techniques will also benefit from this book as a reference.
Guangtao Xue (IEEE M'11) received the Ph.D. degree from the Department of Computer Science and Engineering, Shanghai Jiao Tong University, China, in 2004. He is currently a Professor with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include vehicular ad hoc networks, wireless networks, mobile computing, and distributed computing. He is a member of the IEEE Computer Society and the IEEE Communication Society.
Yi-Chao Chen (IEEE M'16) received his Ph.D. in Computer Science at the University of Texas at Austin in 2015, and B.S. and M.S. in the Department of Computer Science and Information Engineering at National Taiwan University in 2004 and 2006, respectively. He is a tenure-track Associate Professor in the Department of Computer Science and Engineering at Shanghai Jiao Tong University. His research interests focus on networked systems and span the areas of wireless networking, network measurement and analytics, and mobile computing. He is a member of the IEEE Computer and IEEE Communication Society.
Feng Lyu (IEEE S'16-M'18) received the Ph.D. degree from the Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, in 2018 and the B.S. degree in software engineering from Central South University, Changsha, China, in 2013. From September 2018 to December 2019 and from October 2016 to October 2017, he was a Postdoctoral Fellow and a Visiting Ph.D. Student with BBCR Group, Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada. He is currently a Professor with the School of Computer Science and Engineering, Central South University, Changsha, China. His research interests include vehicular networks, beyond 5G networks, big data measurement and application design, and could or edge computing.
Minglu Li (IEEE M'08-SM'19) received the Ph.D. degree from Shanghai Jiao Tong University, China, in 1996. He is currently a Distinguished Professor with the College of Mathematics and Computer Science, Zhejiang Normal University, China, and a Professor with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His main research topics include cloud computing, image processing, and e-commerce.
