# Curiousily

## Credit Card Fraud Detection using Autoencoders in Keras | TensorFlow for Hackers (Part VII)

Deep Learning, Neural Networks, TensorFlow, Python6 min read

Share

It’s Sunday morning, it’s quiet and you wake up with a big smile on your face. Today is going to be a great day! Except, your phone rings, rather “internationally”. You pick it up slowly and hear something really bizarre - “Bonjour, je suis Michele. Oops, sorry. I am Michele, your personal bank agent.”.

What could possibly be so urgent for someone from Switzerland to call you at this hour? “Did you authorize a transaction for $3,358.65 for 100 copies of Diablo 3?” Immediately, you start thinking of ways to explain why you did that to your loved one. “No, I didn’t !?“. Michele’s answer is quick and to the point - “Thank you, we’re on it”. Whew, that was close! But how did Michele knew that this transaction was suspicious? After all, you did order 10 new smartphones from that same bank account, last week - Michele didn’t call then. source: Tutsplus Annual global fraud losses reached$21.8 billion in 2015, according to Nilson Report. Probably you feel very lucky if you are a fraud. About every 12 cents per $100 were stolen in the US during the same year. Our friend Michele might have a serious problem to solve here. In this part of the series, we will train an Autoencoder Neural Network (implemented in Keras) in unsupervised (or semi-supervised) fashion for Anomaly Detection in credit card transaction data. The trained model will be evaluated on pre-labeled and anonymized dataset. The source code and pre-trained model are available on GitHub here. # Setup We will be using TensorFlow 1.2 and Keras 2.0.4. Let’s begin: 1import pandas as pd2import numpy as np3import pickle4import matplotlib.pyplot as plt5from scipy import stats6import tensorflow as tf7import seaborn as sns8from pylab import rcParams9from sklearn.model_selection import train_test_split10from keras.models import Model, load_model11from keras.layers import Input, Dense12from keras.callbacks import ModelCheckpoint, TensorBoard13from keras import regularizers1415%matplotlib inline1617sns.set(style='whitegrid', palette='muted', font_scale=1.5)1819rcParams['figure.figsize'] = 14, 82021RANDOM_SEED = 4222LABELS = ["Normal", "Fraud"] # Loading the data The dataset we’re going to use can be downloaded from Kaggle. It contains data about credit card transactions that occurred during a period of two days, with 492 frauds out of 284,807 transactions. All variables in the dataset are numerical. The data has been transformed using PCA transformation(s) due to privacy reasons. The two features that haven’t been changed are Time and Amount. Time contains the seconds elapsed between each transaction and the first transaction in the dataset. 1df = pd.read_csv("data/creditcard.csv") # Exploration 1df.shape 1(284807, 31) 31 columns, 2 of which are Time and Amount. The rest are output from the PCA transformation. Let’s check for missing values: 1df.isnull().values.any() 1False 1count_classes = pd.value_counts(df['Class'], sort = True)2count_classes.plot(kind = 'bar', rot=0)3plt.title("Transaction class distribution")4plt.xticks(range(2), LABELS)5plt.xlabel("Class")6plt.ylabel("Frequency"); We have a highly imbalanced dataset on our hands. Normal transactions overwhelm the fraudulent ones by a large margin. Let’s look at the two types of transactions: 1frauds = df[df.Class == 1]2normal = df[df.Class == 0] 1frauds.shape 1(492, 31) 1normal.shape 1(284315, 31) How different are the amount of money used in different transaction classes? 1frauds.Amount.describe() 1count 492.0000002mean 122.2113213std 256.6832884min 0.000000525% 1.000000650% 9.250000775% 105.8900008max 2125.8700009Name: Amount, dtype: float64 1normal.Amount.describe() 1count 284315.0000002mean 88.2910223std 250.1050924min 0.000000525% 5.650000650% 22.000000775% 77.0500008max 25691.1600009Name: Amount, dtype: float64 Let’s have a more graphical representation: 1f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)2f.suptitle('Amount per transaction by class')34bins = 5056ax1.hist(frauds.Amount, bins = bins)7ax1.set_title('Fraud')89ax2.hist(normal.Amount, bins = bins)10ax2.set_title('Normal')1112plt.xlabel('Amount ($)')13plt.ylabel('Number of Transactions')14plt.xlim((0, 20000))15plt.yscale('log')16plt.show();

Do fraudulent transactions occur more often during certain time?

1f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)2f.suptitle('Time of transaction vs Amount by class')34ax1.scatter(frauds.Time, frauds.Amount)5ax1.set_title('Fraud')67ax2.scatter(normal.Time, normal.Amount)8ax2.set_title('Normal')910plt.xlabel('Time (in Seconds)')11plt.ylabel('Amount')12plt.show()

Doesn’t seem like the time of transaction really matters.

# Autoencoders

Autoencoders can seem quite bizarre at first. The job of those models is to predict the input, given that same input. Puzzling? Definitely was for me, the first time I heard it.

More specifically, let’s take a look at Autoencoder Neural Networks. This autoencoder tries to learn to approximate the following identity function:

$\textstyle f_{W,b}(x) \approx x$

While trying to do just that might sound trivial at first, it is important to note that we want to learn a compressed representation of the data, thus find structure. This can be done by limiting the number of hidden units in the model. Those kind of autoencoders are called undercomplete.

Here’s a visual representation of what an Autoencoder might learn:

### Reconstruction error

We optimize the parameters of our Autoencoder model in such way that a special kind of error - reconstruction error is minimized. In practice, the traditional squared error is often used:

$\textstyle L(x,x') = ||\, x - x'||^2$

If you want to learn more about Autoencoders I highly recommend the following videos by Hugo Larochelle:

# Preparing the data

First, let’s drop the Time column (not going to use it) and use the scikit’s StandardScaler on the Amount. The scaler removes the mean and scales the values to unit variance:

1from sklearn.preprocessing import StandardScaler23data = df.drop(['Time'], axis=1)45data['Amount'] = StandardScaler().fit_transform(data['Amount'].values.reshape(-1, 1))

Training our Autoencoder is gonna be a bit different from what we are used to. Let’s say you have a dataset containing a lot of non fraudulent transactions at hand. You want to detect any anomaly on new transactions. We will create this situation by training our model on the normal transactions, only. Reserving the correct class on the test set will give us a way to evaluate the performance of our model. We will reserve 20% of our data for testing:

1X_train, X_test = train_test_split(data, test_size=0.2, random_state=RANDOM_SEED)2X_train = X_train[X_train.Class == 0]3X_train = X_train.drop(['Class'], axis=1)45y_test = X_test['Class']6X_test = X_test.drop(['Class'], axis=1)78X_train = X_train.values9X_test = X_test.values
1X_train.shape
1(227451, 29)

# Building the model

Our Autoencoder uses 4 fully connected layers with 14, 7, 7 and 29 neurons respectively. The first two layers are used for our encoder, the last two go for the decoder. Additionally, L1 regularization will be used during training:

1input_dim = X_train.shape[1]2encoding_dim = 14
1input_layer = Input(shape=(input_dim, ))23encoder = Dense(encoding_dim, activation="tanh",4                activity_regularizer=regularizers.l1(10e-5))(input_layer)5encoder = Dense(int(encoding_dim / 2), activation="relu")(encoder)67decoder = Dense(int(encoding_dim / 2), activation='tanh')(encoder)8decoder = Dense(input_dim, activation='relu')(decoder)910autoencoder = Model(inputs=input_layer, outputs=decoder)

Let’s train our model for 100 epochs with a batch size of 32 samples and save the best performing model to a file. The ModelCheckpoint provided by Keras is really handy for such tasks. Additionally, the training progress will be exported in a format that TensorBoard understands.

1nb_epoch = 1002batch_size = 3234autoencoder.compile(optimizer='adam',5                    loss='mean_squared_error',6                    metrics=['accuracy'])78checkpointer = ModelCheckpoint(filepath="model.h5",9                               verbose=0,10                               save_best_only=True)11tensorboard = TensorBoard(log_dir='/media/old-tf-hackers-7/logs',12                          histogram_freq=0,13                          write_graph=True,14                          write_images=True)1516history = autoencoder.fit(X_train, X_train,17                    epochs=nb_epoch,18                    batch_size=batch_size,19                    shuffle=True,20                    validation_data=(X_test, X_test),21                    verbose=1,22                    callbacks=[checkpointer, tensorboard]).history
1autoencoder = load_model('model.h5')

# Evaluation

1plt.plot(history['loss'])2plt.plot(history['val_loss'])3plt.title('model loss')4plt.ylabel('loss')5plt.xlabel('epoch')6plt.legend(['train', 'test'], loc='upper right');

The reconstruction error on our training and test data seems to converge nicely. Is it low enough? Let’s have a closer look at the error distribution:

1predictions = autoencoder.predict(X_test)
1mse = np.mean(np.power(X_test - predictions, 2), axis=1)2error_df = pd.DataFrame({'reconstruction_error': mse,3                        'true_class': y_test})
1error_df.describe()
reconstruction_errortrue_class
count56962.00000056962.000000
mean0.7426130.001720
std3.3968380.041443
min0.0501390.000000
25%0.2370330.000000
50%0.3905030.000000
75%0.6262200.000000
max263.9099551.000000

### Reconstruction error without fraud

1fig = plt.figure()2ax = fig.add_subplot(111)3normal_error_df = error_df[(error_df['true_class']== 0) & (error_df['reconstruction_error'] < 10)]4_ = ax.hist(normal_error_df.reconstruction_error.values, bins=10)

### Reconstruction error with fraud

1fig = plt.figure()2ax = fig.add_subplot(111)3fraud_error_df = error_df[error_df['true_class'] == 1]4_ = ax.hist(fraud_error_df.reconstruction_error.values, bins=10)

1from sklearn.metrics import (confusion_matrix, precision_recall_curve, auc,2                             roc_curve, recall_score, classification_report, f1_score,3                             precision_recall_fscore_support)

ROC curves are very useful tool for understanding the performance of binary classifiers. However, our case is a bit out of the ordinary. We have a very imbalanced dataset. Nonetheless, let’s have a look at our ROC curve:

1fpr, tpr, thresholds = roc_curve(error_df.true_class, error_df.reconstruction_error)2roc_auc = auc(fpr, tpr)34plt.title('Receiver Operating Characteristic')5plt.plot(fpr, tpr, label='AUC = %0.4f'% roc_auc)6plt.legend(loc='lower right')7plt.plot([0,1],[0,1],'r--')8plt.xlim([-0.001, 1])9plt.ylim([0, 1.001])10plt.ylabel('True Positive Rate')11plt.xlabel('False Positive Rate')12plt.show();

The ROC curve plots the true positive rate versus the false positive rate, over different threshold values. Basically, we want the blue line to be as close as possible to the upper left corner. While our results look pretty good, we have to keep in mind of the nature of our dataset. ROC doesn’t look very useful for us. Onward…

### Precision vs Recall

source: Wikipedia

Precision and recall are defined as follows:

$\text{Precision} = \frac{\text{true positives}}{\text{true positives} + \text{false positives}}$

$\text{Recall} = \frac{\text{true positives}}{\text{true positives} + \text{false negatives}}$

Let’s take an example from Information Retrieval in order to better understand what precision and recall are. Precision measures the relevancy of obtained results. Recall, on the other hand, measures how many relevant results are returned. Both values can take values between 0 and 1. You would love to have a system with both values being equal to 1.

Let’s return to our example from Information Retrieval. High recall but low precision means many results, most of which has low or no relevancy. When precision is high but recall is low we have the opposite - few returned results with very high relevancy. Ideally, you would want high precision and high recall - many results with that are highly relevant.

1precision, recall, th = precision_recall_curve(error_df.true_class, error_df.reconstruction_error)2plt.plot(recall, precision, 'b', label='Precision-Recall curve')3plt.title('Recall vs Precision')4plt.xlabel('Recall')5plt.ylabel('Precision')6plt.show()

A high area under the curve represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate. High scores for both show that the classifier is returning accurate results (high precision), as well as returning a majority of all positive results (high recall).

1plt.plot(th, precision[1:], 'b', label='Threshold-Precision curve')2plt.title('Precision for different threshold values')3plt.xlabel('Threshold')4plt.ylabel('Precision')5plt.show()

You can see that as the reconstruction error increases our precision rises as well. Let’s have a look at the recall:

1plt.plot(th, recall[1:], 'b', label='Threshold-Recall curve')2plt.title('Recall for different threshold values')3plt.xlabel('Reconstruction error')4plt.ylabel('Recall')5plt.show()

Here, we have the exact opposite situation. As the reconstruction error increases the recall decreases.

# Prediction

Our model is a bit different this time. It doesn’t know how to predict new values. But we don’t need that. In order to predict whether or not a new/unseen transaction is normal or fraudulent, we’ll calculate the reconstruction error from the transaction data itself. If the error is larger than a predefined threshold, we’ll mark it as a fraud (since our model should have a low error on normal transactions). Let’s pick that value:

1threshold = 2.9

And see how well we’re dividing the two types of transactions:

1groups = error_df.groupby('true_class')2fig, ax = plt.subplots()34for name, group in groups:5    ax.plot(group.index, group.reconstruction_error, marker='o', ms=3.5, linestyle='',6            label= "Fraud" if name == 1 else "Normal")7ax.hlines(threshold, ax.get_xlim()[0], ax.get_xlim()[1], colors="r", zorder=100, label='Threshold')8ax.legend()9plt.title("Reconstruction error for different classes")10plt.ylabel("Reconstruction error")11plt.xlabel("Data point index")12plt.show();

I know, that chart might be a bit deceiving. Let’s have a look at the confusion matrix:

1y_pred = [1 if e > threshold else 0 for e in error_df.reconstruction_error.values]2conf_matrix = confusion_matrix(error_df.true_class, y_pred)34plt.figure(figsize=(12, 12))5sns.heatmap(conf_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, fmt="d");6plt.title("Confusion matrix")7plt.ylabel('True class')8plt.xlabel('Predicted class')9plt.show()

Our model seems to catch a lot of the fraudulent cases. Of course, there is a catch (see what I did there?). The number of normal transactions classified as frauds is really high. Is this really a problem? Probably it is. You might want to increase or decrease the value of the threshold, depending on the problem. That one is up to you.

# Conclusion

We’ve created a very simple Deep Autoencoder in Keras that can reconstruct what non fraudulent transactions looks like. Initially, I was a bit skeptical about whether or not this whole thing is gonna work out, bit it kinda did. Think about it, we gave a lot of one-class examples (normal transactions) to a model and it learned (somewhat) how to discriminate whether or not new examples belong to that same class. Isn’t that cool? Our dataset was kind of magical, though. We really don’t know what the original features look like.

Keras gave us very clean and easy to use API to build a non-trivial Deep Autoencoder. You can search for TensorFlow implementations and see for yourself how much boilerplate you need in order to train one. Can you apply a similar model to a different problem?

The source code and pre-trained model are available on GitHub here.

# References

Share

## Want to be a Machine Learning expert?

Join the weekly newsletter on Data Science, Deep Learning and Machine Learning in your inbox, curated by me! Chosen by 10,000+ Machine Learning practitioners. (There might be some exclusive content, too!)

You'll never get spam from me

© 2020 Curiousily by Venelin Valkov