Get Start with STYX

1. Setup

Environment Requirement:

Python Package:

2. Example

Examples of using STYX can be found in Github Repo. It provides an example for training the MLP models w.r.t. MNIST benchmark under three different methods (traditional training, adversarial training, and STYX) and then making the robustness evaluation. The example can be run with the following command, which takes about 20 minutes:

cd Examples
bash run_example.sh

In more detail, we consider that in three steps:

(1) model’s description

cd Tool/traditional_training

First we need to provide the model’s structure like Train_mnist_MLP.py in the current folder. Currently, we supprot five model types: MLP for MNIST (mnist_MLP), MLP for Fashion-MNIST (fmnist_MLP), CNN for MNIST (mnist_CNN), CNN for Fashion-MNIST (fmnist_CNN), and CNN for CIFAR-10 (cifar10_CNN).

(2) model’s generation

Given the paramters like data_type=”mnist”, model_type=”MLP” and train_epoch=20. Next we use the following command to generate the models by different training method.

(2.1) traditional_training

cd Tool/traditional_training
python traditional_training.py <data_type> <model_name> <train_epoch>

(2.2) adversarial_training

cd Tool/adversarial_training
python adversarial_training.py <data_type> <model_name> <train_epoch>

(2.3) STYX

cd Tool/STYX
python styx.py <data_type> <model_name> <train_epoch>

(3) model’s evaluation

After that, we use the following command to evaluate the models by different attacking method. Take the attacking method “FGM” and the model trained by STYX as example (i.e. attacking_method=”FGM”, training_method=”STYX”):

cd Tool/evaluation  
python main_evaluation.py <data_type> <model_name> <training_method> <attacking_method>

And the evaluation result will be consist of three parts: Accuracy, Robustness, and Time-cost.