Anomaly detection in large-scale data stream networks
MetadataShow full item record
This paper addresses the anomaly detection problem in large-scale data mining applications using residual subspace analysis. We are specifically concerned with situations where the full data cannot be practically obtained due to physical limitations such as low bandwidth, limited memory, storage, or computing power. Motivated by the recent compressed sensing (CS) theory, we suggest a framework wherein random projection can be used to obtained compressed data, addressing the scalability challenge. Our theoretical contribution shows that the spectral property of the CS data is approximately preserved under a such a projection and thus the performance of spectral-based methods for anomaly detection is almost equivalent to the case in which the raw data is completely available. Our second contribution is the construction of the framework to use this result and detect anomalies in the compressed data directly, thus circumventing the problems of data acquisition in large sensor networks. We have conducted extensive experiments to detect anomalies in network and surveillance applications on large datasets, including the benchmark PETS 2007 and 83 GB of real footage from three public train stations. Our results show that our proposed method is scalable, and importantly, its performance is comparable to conventional methods for anomaly detection when the complete data is available.
The final publication is available at Springer via http://dx.doi.org/10.1007/s10618-012-0297-3
Showing items related by title, author, creator and subject.
Pham, DucSon; Saha , Budhaditya; Lazarescu, Mihai; Venkatesh, Svetha (2009)This paper addresses a major challenge in datamining applications where the full information about the underlying processes, such as sensor networks or large online database, cannot be practically obtained due to physical ...
Pham, DucSon; Saha, Budhaditya; Phung, Dinh; Venkatesh, Svetha (2012)The data deluge has created a great challenge for data mining applications wherein the rare topics of interest are often buried in the flood of major headlines. We identify and formulate a novel problem: cross-channel ...
Pham, DucSon; Saha, Budhaditya; Phung, Dinh; Venkatesh, Svetha (2011)We identify and formulate a novel problem: cross channel anomaly detection from multiple data channels. Cross channel anomalies are common amongst the individual channel anomalies, and are often portent of significant ...