ISTC-ARSA

MLSploit Extended Abstract to Appear in KDD 2019

2019-06-11T09:30:00-04:00

An extended abstract authored by ISTC-ARSA researchers has been accepted to the 25th Conference on Knowledge Discovery and Data Mining (KDD'19) in August.

Title: MLsploit: A Framework for Interactive Experimentation with Adversarial Machine Learning Research

Authors: Nilaksh Das, Siwei Li, Chanil Jeon, Jinho Jung, Shang-Tse Chen, Carter Yagemann, Evan Downing, Haekyu Park, Evan Yang, Li Chen, Michael Kounavis, Ravi Sahita, David Durham, Scott Buck, Polo Chau, Taesoo Kim, Wenke Lee

Abstract:

We present MLsploit, the first user-friendly, cloud-based system that enables researchers and practitioners to rapidly evaluate and compare state-of-the-art adversarial attacks and defenses for machine learning (ML) models. As recent advances in adversarial ML have revealed that many ML techniques are highly vulnerable to adversarial attacks, MLsploit meets the urgent need for practical tools that facilitate interactive security testing of ML models. MLsploit is jointly developed by researchers at Georgia Tech and Intel, and is open-source. Designed for extensibility, MLsploit accelerates the study and development of secure ML systems for safety-critical applications. In this showcase demonstration, we highlight the versatility of MLsploit in performing fast-paced experimentation with adversarial ML research that spans a diverse set of modalities, such as bypassing Android and Linux malware, or attacking and defending deep learning models for image classification. We invite the audience to perform experiments interactively in real time by varying different parameters of the experiments or using their own samples, and finally compare and evaluate the effects of such changes on the performance of the ML models through an intuitive user interface, all without writing any code.

Barnum to Appear in Information Security Conference 2019

2019-06-08T15:30:00-04:00

A paper authored by ISTC-ARSA researchers has been accepted to the 22nd Information Security Conference (ISC'19) in September.

Title: Barnum: Detecting Document Malware via Control Flow Anomalies in Hardware Traces.

Authors: Carter Yagemann (Georgia Tech), Salmin Sultana (Intel Labs), Li Chen (Intel Labs), Wenke Lee (Georgia Tech).

Abstract:

This paper proposes Barnum, an offline control flow attack detection system that applies deep learning on hardware execution traces to model a program's behavior and detect control flow anomalies. Our implementation analyzes document readers to detect exploits and ABI abuse. Recent work has proposed using deep learning based control flow classification to build more robust and scalable detection systems. These proposals, however, were not evaluated against different kinds of control flow attacks, programs, and adversarial perturbations.

We investigate anomaly detection approaches to improve the security coverage and scalability of control flow attack detection. Barnum is an end-to-end system consisting of three major components: 1) trace collection, 2) behavior modeling, and 3) anomaly detection via binary classification. It utilizes Intel^® Processor Trace for low overhead execution tracing and applies deep learning on the basic block sequences reconstructed from the trace to train a normal program behavior model. Based on the path prediction accuracy of the model, Barnum then determines a decision boundary to classify benign vs. malicious executions.

We evaluate against 8 families of attacks to Adobe Acrobat Reader and 9 to Microsoft Word on Windows 7. Both readers are complex programs with over 50 dynamically linked libraries, just-in-time compiled code and frequent network I/O. Barnum shows its effectiveness with 0% false positive and 2.4% false negative on a dataset of 1,250 benign and 1,639 malicious PDFs. Barnum is robust against evasion techniques as it successfully detects 500 adversarially perturbed PDFs.

uCFI Accepted to ACM CCS 2018

2018-07-23T22:00:00-04:00

A paper authored by ISTC-ARSA researchers has been accepted to the 25th ACM Conference on Computer and Communications Security (CCS'18) being held in Toronto, Canada from October 15, 2018 to October 19, 2018.

Title: Enforcing Unique Code Target Property for Control-Flow Integrity

Authors: Hong Hu, Chenxiong Qian, Carter Yagemann, Simon Pak Ho Chung, Bill Harris, Taesoo Kim, Wenke Lee

Abstract:

The goal of control-flow integrity (CFI) is to stop control-hijacking attacks by ensuring that each indirect control-flow transfer (ICT) jumps to its legitimate target. However, existing implementations of CFI have fallen short of this goal because their approaches are inaccurate and as a result, the set of allowable targets for an ICT instruction is too large, making illegal jumps possible.

In this paper, we propose the Unique Code Target (UCT) property for CFI. Namely, for each invocation of an ICT instruction, there should be one and only one valid target. We develop a prototype called uCFI to enforce this new property. During compilation, uCFI identifies the sensitive instructions that influence ICT and instruments the program to record necessary execution context. At runtime, uCFI monitors the program execution in a different process, and performs points-to analysis by interpreting sensitive instructions using the recorded execution context in a memory safe manner. It checks runtime ICT targets against the analysis results to detect CFI violations. We apply uCFI to SPEC benchmarks and 2 servers (nginx and vsftpd) to evaluate its efficacy of enforcing UCT and its overhead. We also test uCFI against control-hijacking attacks, including 5 real-world exploits, 1 proof of concept COOP attack, and 2 synthesized attacks that bypass existing defenses. The results show that uCFI strictly enforces the UCT property for protected programs, successfully detects all attacks, and introduces less than 10% performance overhead.

Researchers gather May 9-10 for second annual retreat

2018-05-15T10:15:00-04:00

Researchers from Intel Labs and Georgia Tech gathered at Intel's campus in Portland, Oregon for a two-day annual retreat dedicated to the advancement of machine learning (ML) cybersecurity.

Following a review of the multi-year project vision and goals for the Intel ISTC-ARSA, students gave a demo of the upcoming MLSploit framework, which combines all the research from ISTC-ARSA into an intuitive web interface.

Although the retreat is closed to the public, we are excited to announce that MLSploit is slated for public release in the coming year. By releasing MLSploit, we aim to create a standardized framework and dataset for researchers to evaluate and compare ideas, as well as offer students a way to learn about ML cybersecurity.

Robust Physical Adversarial Attack on Faster R-CNN Object Detector

2018-04-16T09:00:00-04:00

We have release a new code repository for physically attacking Faster R-CNN.

In this work, we tackle the more challenging problem of crafting physical adversarial perturbations to fool image-based object detectors like Faster R-CNN. Attacking an object detector is more difficult than attacking an image classifier, as it needs to mislead the classification results in multiple bounding boxes with different scales. Our approach can generate perturbed stop signs that are consistently mis-detected by Faster R-CNN as other objects, posing a potential threat to autonomous vehicles and other safety-critical computer vision systems.

The arXiv paper is available here.

Shield: Fast, Practical Defense and Vaccination for Deep Learning using JPEG Compression

2018-02-19T09:00:00-05:00

We published a new defense for deep learning that uses JPEG compression. The paper is available on arXiv and the code is on Github.

Defending AI with JPEG Compression

2017-12-12T17:30:00-05:00

The field of machine learning has witnessed tremendous success in the recent years across multiple domains. It is not uncommon to witness the state of the art being challenged nearly every month, more so in the domain of computer vision. Many deep neural networks have been proposed that can beat even humans at certain tasks like image recognition.

However, it has recently been shown that even though these mathematical models appear to be so adept at natural tasks, the way they make certain decisions are extremely uninterpretable and somewhat precarious. For instance, given a model that does very well at the task of traffic sign recognition (e.g., a self-driving car would use something like this to make its decisions on the fly), the pixels of the input image that is consumed by the model can be changed so slightly that the change is completely invisible to humans, but confuses the model with embarrassing accuracy. Precise methods of constructing such input perturbations have been proposed in the recent literature that can target a fully observable model, but it’s adversarial effect transfers well even to other models which are not observed or targeted by the attack. This is especially the case with deep neural networks (DNNs), which lends researchers the reason to introspect how fragile DNNs really are in representing the input internally and how these models can be made more robust.

At the Intel Science and Technology Center for Adversary-Resilient Security Analytics (ISTC-ARSA), one of our many endeavors is to identify ways in which we can protect DNNs from such attacks. Part of my research here focuses on experimenting with preprocessing techniques that don’t require explicitly modifying the DNN architectures that work well. Instead, we focus on transforming the input to the network so as to preserve the original semantics and destroy any adversarial perturbations that may have been added to confuse the system. In this post, I will present empirical evidence that showcases the power of one such image preprocessing technique - JPEG compression - as a model agnostic defense to adversarial attacks.

State-of-the-Art Adversarial Attacks

Before we dive deeper into how JPEG compression works as a defense, let us take a look at some state-of-the-art adversarial attacks.

Fast Gradient Sign Method (FGSM)

Proposed by Goodfellow et al., the FGSM attack is one of the fastest ways of computing adversarial perturbations. This attack simply computes the sign of the gradient of the loss with respect to each pixel and scales it by a constant factor in the opposite direction. Put simply, it adds a constant magnitude perturbation to each pixel in an image; the sign of the perturbation corresponds to the direction which increases the overall classification loss.

DeepFool

Moosavi-Dezfooli et al. presented an optimal attack that efficiently computes the minimal perturbation for a given image that is enough to fool a model. It does so by linearizing the decision boundary of the model and iteratively perturbing the image so that it moves closer to the boundary, until it just crosses over. Since this attack computes the minimal perturbation required, it results in a very low magnitude of perturbation and is virtually invisible to the naked eye.

JPEG Compression as a Defense Against Adversarial Attacks

JPEG is a standard and widely-used image encoding and compression technique which mainly consists of the following steps:

converting the given image from RGB to YCbCr (chrominance + luminance) color space: this is done because the human visual system relies more on spatial content and acuity than it does on color for interpretation. Converting the color space isolates these components which are of more importance.
performing spatial subsampling of the chrominance channels in the YCbCr space: the human eye is much more sensitive to changes in luminance, and downsampling the chrominance information does not affect the human perception of the image very much.
transforming a blocked representation of the YCbCr spatial image data to a frequency domain representation using Discrete Cosine Transform (DCT): this step allows the JPEG algorithm to further compress the image data as outlined in the next step by computing DCT coefficients.
performing quantization of the blocked frequency domain data according to a user-defined quality factor: this is where the JPEG algorithm achieves the majority of the compression, at the expense of image quality. This step suppresses higher frequencies more since these coefficients contribute less to the human perception of the image.

The core principle motivating JPEG compression is based on the human psychovisual system. It aims to suppress high frequency information like sharp transitions in intensity and color hue using Discrete Cosine Transform. As adversarial attacks often introduce perturbations that are not compatible with human psychovisual awareness (hence these attacks are mostly imperceptible to humans), we hypothesize that JPEG compression has the potential to remove such perturbations.

Thus, we propose to use JPEG compression as a preprocessing step in the classification pipeline and experiment with varying the compression quality to see its effect on the misclassification success of adversarial attacks. Moreover, since applying JPEG compression might introduce artifacts of its own during classification, we also propose to vaccinate a DNN by retraining it on JPEG compressed images of a particular quality. A model is retrained multiple times on multiple qualities, and we use an ensemble of these models to get the final classification label. We tested our ensemble on the CIFAR-10 and GTSRB (German Traffic Sign Recognition Benchmark) datasets and present the results in the next section.

Results

The misclassification success of an adversarial attack is defined as the proportion of instances whose labels were successfully flipped by the attack, amongst all the instances which were correctly classified by the model.

We see that varying the image quality of JPEG compression on the CIFAR-10 and GTSRB datasets reduces the misclassification success of FGSM and DeepFool attacks. The horizontal dashed lines show further drop in misclassification success using our ensemble of vaccinated DNNs.

Table 1: Classification accuracies with our approach on the respective test sets when original, non-vaccinated model is under attack.

To summarize, we see that we are able to recover the classification accuracy significantly using our ensemble of vaccinated models under attack, and that the accuracy actually increases with benign images!

Discussion

Benign, everyday images lie in a very constrained subspace. An image with completely random pixel colors is highly unlikely to be perceived as natural by human beings. However, the objective basis of classification models like DNNs often is not aligned with such considerations. DNNs may be viewed as constructing decision boundaries that linearly separate the data in high dimensional spaces. In doing so, these models assume that the subspaces of natural images exist beyond the actual subspace. Adversarial attacks take advantage of this by perturbing images just enough so that they cross over the decision boundary of the model. However, this crossing over does not guarantee that the perturbed images would lie in the original narrow subspace. Indeed, perturbed images could lie in artificially expanded subspaces where natural images would not be found.

Since JPEG compression takes the human psychovisual system into account, we believe that the subspace in which JPEG images occur would have some semblance with the subspace of naturally occurring images, and that using JPEG compression as a preprocessing step during classification would re-project any adversarially perturbed instances back onto this subspace.

To gain further insights on this approach and for more details on our experiments, you can refer to our full paper found here.

CCS 2017 Accepted Papers

2017-10-30T09:00:00-04:00

We have three papers appearing in CCS 2017:

Yang Ji, Sangho Lee, Evan Downing, Weiren Wang, Mattia Fazzini, Taesoo Kim, Alessandro Orso, Wenke Lee. RAIN: Refinable Attack Investigation with On-demand Inter-Process Information Flow Tracking. Appeared in ACM Conference on Computer and Communications Security (CCS 2017). Dallas, USA. October 2017. [Paper]
Ruian Duan, Ashish Bijlani, Meng Xu, Taesoo Kim, Wenke Lee. Checking Open-Source License Violation and 1-day Security Risk at Large Scale. Appeared in ACM Conference on Computer and Communications Security (CCS 2017). Dallas, USA. October 2017. [Paper]
Wen Xu, Sanidhya Kashyap, Changwoo Min, Taesoo Kim. Designing New Operating Primitives to Improve Fuzzing Performance. Appeared in ACM Conference on Computer and Communications Security (CCS 2017). Dallas, USA. October 2017. [Paper]

Congratulations to the above authors!

Intel PT Data at Rest: A Compression Experiment

2017-10-28T10:30:00-04:00

At the Intel Science and Technology Center for Adversarial Resilient Security Analytics (ISTC-ARSA), one of our ongoing goals is to identify and explore new data sources for more robust machine learning. One of the new sources we're interested in is Intel Processor Trace (PT), which is able to efficiently record instruction level traces. However, there are a number of technical challenges that need to be solved before PT can be used in the context of machine learning for security. The following is a small one-day experiment I conducted to explore the memory overhead of storing PT traces. Machine learning will require access to many traces, which makes the exploration of simple ways to conserve space a worthy endeavor.

Intel Processor Trace (PT) is a powerful hardware feature for recording the behavior of CPUs. With it, developers and researchers can monitor the control-flow path taken by threads, hardware interrupts, and more, all with cycle-accurate timing. However, this rich stream of data comes at the cost of size. Depending on what PT is configured to trace, it can output hundreds of megabytes of data per second per core. PT does take steps to save bandwidth by only recording changes in control-flow, excluding redundant high-order bits in target addresses, and compressing returns leading to predictable locations. However, despite this compression, the volume of data is still massive.

As a consequence, much of the work published so far handles tracing in one of two ways. One option is to consume the trace as it is generated. This works as long as the consumer can keep up with the producer, which is the case in the control-flow integrity (CFI) system Griffin. The other common approach is to configure PT to write in a circular buffer. This option is suitable for crash dump analysis systems like Snorlax, which only need a fixed size window into a thread's past.

However, while some applications are feasible using the two previous methods, there are still situations were it is desirable to store the entire trace for postmortem analysis. If nothing else, it is useful for repeatable experiments. With this in mind, I performed a naive experiment last night to explore if more can be done to compress PT traces when the data is at rest. Based on the observations that the compression PT applies is highly localized (i.e. a target address verses the previously recorded target address and a return verses the previously recorded call) and that programs often execute repetitive loops, I hypothesized that even a general purpose compression algorithm should be able to compress traces with a good ratio.

Procedure

The overall idea for the experiment is very simple: gather some PT traces, compress them with a commonly used algorithm, and compare the sizes. For a subject I used the simple HTTP server that comes with Python 2.7 to host a copy of this blog. For each trial I had a crawler request pages from the server for a set duration. Once the time expired, I terminated the server and crawler and stopped the tracing. I then compressed the trace using the GNU/Linux utility gzip, which uses Lempel-Ziv coding. I also fed it through a disassembler that matches the PT packets to the binary's static code to produce a linear sequence of instructions. From this I counted the number of unique basic blocks executed during the trace to serve as a rough proxy for code coverage. To summarize the procedure:

Configure and enable PT tracing.
Start the Python HTTP server.
Start the crawler.
Wait for a specified duration.
Terminate the crawler and server.
Stop PT tracing.
Compress the resulting trace and count the number of unique basic blocks executed.

Results

Comparing the original size of the PT trace to the size after compression produces the above graph. Both plots best match linear regressions and are increasing over time. However, the size of the compressed traces increases at a slower rate than the uncompressed traces, meaning these two plots are diverging as time increases.

Another observation to note is the large volume of trace data produced during the server's startup. This explains why even the shortest trial produced a 1GB trace. For the same reason, counting the number of unique basic blocks turned out to not be useful. The number of new basic blocks executed while serving requests was small.

The next graph shows the relationship between the compressed and uncompressed sizes as a space savings percentage. The plot best fits a linear regression and shows the savings decreasing over time. This is likely due to the design of the underlying compression algorithm, which is intended for general use and does not take into consideration the unique characteristics of PT traces.

To summarize, this experiment shows that more can be done to compress PT traces for storage at rest.

Discussion

It is understandable that the compression used by PT would produce small space savings compared to general compression algorithms given the limitations of hardware memory and Intel's very strict performance overhead requirements. In practice, PT produces an overhead of less than 4% in the worst case, and less than 2% on average. These numbers are based on my own observations and the results published by other researchers. In short, PT has very few clock cycles and very little space available for performing compression.

Another factor that deserves consideration is compression's impact on processing time. For systems that consume PT traces on the fly, the largest source of performance overhead is not PT tracing itself but rather the time spent buffering and consuming it. In CFI, for example, the PT trace has to be matched with the executed code in order to reconstruct control-flow. This is why the authors of Griffin report a 11.9% overhead on the SPECint benchmark despite the 4% overhead of PT itself. Adding better space saving compression could increase this overhead further.

That said, for storing PT traces at rest, more can be done to better conserve space.

AVPass Code Release

2017-09-15T09:00:00-04:00

The code for AVPass is available now on Github!

Researchers to gather June 7-8 for first annual retreat

2017-06-06T09:14:00-04:00

Researchers from Intel Labs and Georgia Tech will converge in Atlanta for a two-day annual retreat dedicated to the advancement of machine learning (ML) cybersecurity.

Following a review of the multi-year project vision and goals for the Intel ISTC-ARSA, recent learnings will be presented about each of the five research themes underway:

Attacks on ML: machine teaching and active learning (by Le Song, Polo Chau, Jinho Jung, Erkam Uzun, Simon Chung)
Improvements to ML (by Polo Chau, Le Song, Shang-Tse Chen)
Malware analysis (by Evan Downing, Yeongjin Jang, Kyuhong Park)
Intel PT (by Carter Yagemann, Chenxiong Qian, Bill Harris)
Intel SGX (by Taesoo Kim)

as well as additional research by Professor Irfan Essa of Georgia Tech’s interdisciplinary research center ML@GT and by Chief Research Scientist Michael Farrell, who co-directs Georgia Tech’s $17M cyber attribution study for the U.S. Department of Defense.

Although the retreat is closed to the public, outcomes including posters and presentations will be made available on this website and housed under the “Outcomes” section.

Site Update and New Publications

2017-06-03T08:23:00-04:00

We have pushed a lot of great new content to the ISTC-ARSA website:

Our About page has been updated with more specifics regarding our research activities.
Under the new Themes tab we have a listing of our current projects.
The Publications tab is now an Outcomes dropdown menu including Publications, Presentations, and Software.

We are also happy to report a handful of accepted and published works including:

Adversarial machine learning

Weiyang Liu, Bo Dai, James M. Rehg, and Le Song. Iterative Machine Teaching. To appear in International Conference on Machine Learning (ICML 2017). Sydney, Australia. August 2017. [Paper]
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. SphereFace: Deep Hypersphere Embedding for Face Recognition. To appear in CVPR 2017. Honolulu, Hawaii. July, 2017. [Paper] [Results]

Adversarial-resillience

Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Li Chen, Michael E. Kounavis, and Duen Horng Chau. Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compression. [Paper]

MLSPLOIT

Steffen Maass, Changwoo Min, Sanidhya Kashyap, Woonhak Kang, Mohan Kumar, and Taesoo Kim. Mosaic: Processing a Trillion-Edge Graph on a Single Machine. In Proceedings of the 12st ACM European Conference on Computer Systems (EuroSys 2017). Belgrade, Serbia. April, 2017. [Paper] [Slides]

Next-generation security analytics

Ren Ding, Chenxiong Qian, Chengyu Song, Bill Harris, Taesoo Kim, and Wenke Lee. Efficient Protection of Path-Sensitive Control Security. To appear in Proceedings of the 26th USENIX Security Symposium (Security 2017). Vancouver, Canada. August 2017.

Robust security analytics

Sangho Lee, Ming-Wei Shih, Prasun Gera, Taesoo Kim, Hyesoon Kim, and Marcus Peinado. Inferring Fine-grained Control Flow Inside SGX Enclaves with Branch Shadowing. To appear in Proceedings of the 26th USENIX Security Symposium (Security 2017). Vancouver, Canada. August 2017.
Jaehyuk Lee, Jinsoo Jang, Yeongjin Jang, Nohyun Kwak, Yeseul Choi, Changho Choi, Taesoo Kim, Marcus Peinado, and Brent B. Kang. Hacking in Darkness: Return-oriented Programming against Secure Enclaves. To appear in Proceedings of the 26th USENIX Security Symposium (Security 2017). Vancouver, Canada. August 2017.
Jinho Jung, Chanil Jeon, Max Wolotsky, Insu Yun, and Taesoo Kim. AVPASS: Leaking and Bypassing Antivirus Detection Model Automatically. To appear in BlackHat USA 2017. Las Vegas, NV. Auguest 2017.
Seongmin Kim, Juhyeng Han, Jaehyeong Ha, Taesoo Kim, and Dongsu Han. Enhancing Security and Privacy of Tor's Ecosystem by using Trusted Execution Environments. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2017). Boston, MA. March 2017. [Paper] [Slides] [Code]
Jaebaek Seo, Byoungyoung Lee, Sungmin Kim, Ming-Wei Shih, Insik Shin, Dongsu Han, and Taesoo Kim. SGX-Shield: Enabling Address Space Layout Randomization for SGX Programs. In Proceedings of the 2017 Network and Distributed System Security Symposium (NDSS 2017). San Diego, CA. February 2017. [Paper] [Slides]
Ming-Wei Shih, Sangho Lee, Taesoo Kim, and Marcus Peinado. T-SGX: Eradicating Controlled-Channel Attacks Against Enclave Programs. In Proceedings of the 2017 Network and Distributed System Security Symposium (NDSS 2017). San Diego, CA. February 2017. [Paper] [Slides] [Code]

Hello World!

2017-01-27T11:30:00-05:00

This marks the beginning of our site to track the progress of the Intel Science & Technology Center for Adversary-Resilient Security Analytics (ISTC-ARSA) housed at Georgia Tech’s Institute for Information Security & Privacy (IISP).