<jupyter_start><jupyter_text>Deep Learning Malware Detector with stringsThis malware detector uses the strings-command and the Feature Hasher to create a dataset of malware and benignware. The dataset is split up into training and test and we use a (not so deep) deep neural network to classify. Thus, we solve the task from chapter 9 with deep learning now :)<jupyter_code>#  These are the imports we are going to need
import subprocess
import os
import numpy
from sklearn.ensemble import RandomForestClassifier
from sklearn import tree
from sklearn.feature_extraction import FeatureHasher
from sklearn.model_selection import train_test_split
# Create the dataset using the files in data

benignware = os.listdir('data/benignware') # <---------- CHANGE PATH HERE
malware = os.listdir('data/malware') # <---------------- CHANGE PATH HERE
hasher = FeatureHasher(1000) # We initialize the featurehasher using 1,000 features

def extract_strings(filepath):
    '''This methods extracts the strings from a file using the strings command in unix os'''
    strings = subprocess.Popen(['strings', filepath], stdout=subprocess.PIPE).communicate()[0].decode('utf-8').split('\n')
    return strings

benign_strings = [extract_strings('data/benignware/' + benignware[i]) for i in range(len(benignware))] # All strings from benignfiles
malware_strings = [extract_strings('data/malware/' + malware[i]) for i in range(len(malware))] # All strings from malwarefiles
# This creates the dataset by using the FeatureHasher to squash the strings of the file into a featuremap
# The code is mostly taken from the malware datascience book

benign_features = [] # This list later has all the benign_featuremaps
for bs in benign_strings:
    # store string features in dictionary form
    string_features = {}
    for string in bs:
        string_features[string] = 1

    # hash the features using the hashing trick
    hashed_features = hasher.transform([string_features])
    # do some data munging to get the feature array
    hashed_features = hashed_features.todense()
    hashed_features = numpy.asarray(hashed_features)
    hashed_features = hashed_features[0]
    benign_features.extend([hashed_features])
    
malware_features = []
for ms in malware_strings:
    # store string features in dictionary form
    string_features = {}
    for string in ms:
        string_features[string] = 1

    # hash the features using the hashing trick
    hashed_features = hasher.transform([string_features])
    # do some data munging to get the feature array
    hashed_features = hashed_features.todense()
    hashed_features = numpy.asarray(hashed_features)
    hashed_features = hashed_features[0]
    malware_features.extend([hashed_features])

# Let's create a trainingset
X = benign_features + malware_features
Y = [1 for _ in range(len(benignware))] + [0 for _ in range(len(malware))]

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)
# Let's print the statistics (showing that we have more benignware than malware)
# It already shows, that we should use some kind of weighting in our algorithms later, to deal with the skewed data

print('Statistics:')
print('Benign-Files:', len(benignware))
print('Malware-Files:', len(malware))<jupyter_output>Statistics:
Benign-Files: 991
Malware-Files: 428<jupyter_text>TrainingSo after we created our dataset, we finally can start training.Firstly, we create new numpy arrays to be used by keras. We then define our model<jupyter_code>X_trainN = numpy.array(X_train)
Y_trainN = numpy.array(Y_train)
X_testN  = numpy.array(X_test)
Y_testN  = numpy.array(Y_test)
import keras
from keras.layers import Dense, Activation, Dropout
from keras.models import Sequential

model = Sequential()
model.add(Dense(10, input_dim=1000))
model.add(Activation('relu'))
model.add(Dropout(3))
model.add(Dense(10, input_dim=1000))
model.add(Activation('relu'))
model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.fit(X_trainN, Y_trainN, epochs=10, batch_size=32)
print('Score on unseen data:', model.evaluate(X_testN, Y_testN, batch_size=32), '(corresponding to)', *model.metrics_names)<jupyter_output>Using TensorFlow backend.
StarcoderdataJupyterStructuredCleanDedup