<filename>8_conclusion.tex
\chapter{Conclusion and Future Work}\label{sec:summary} \label{sec:future_work}
%I first experimented with white-box model inference of MicroPilot's source code using existing state of the art tools. 
%However, the algorithms' implementations showed quite a number of scalability issues when tested with such a large scale software system. I was able to mitigate or even solve some of the issues through various methods and tricks, but still they were not usable as a robust model inference technique. These attempts were published in ICPC '19 (co-located with ICSE) in negative results track. \footnote{I referenced that paper \cite{mashhadi2019empirical} in this thesis but did not include it.}
%Thereafter, considering the partner company's needs, I took a higher level black-box approach in modeling the software. 

In this thesis, I developed a novel method for inferring black box models for autopilot software systems.
My method at its core is a deep neural network that combines convolutions and recurrent cells: hybrid CNN-RNN model. 
This design is inspired by deep neural network architectures that showed good performance in other fields (such as speech recognition, sleep phase detection, and human activity recognition). 
It can be used for both CPD and state classification problems in multivariate time series. 

This method be used as a black-box state model inference for variety of use cases such as testing, debugging, and anomaly detection in control software systems, where there are several input signals that control output states. 

I have trained and evaluated this neural network on two case studies of a UAV autopilot softwares, one from the FOSS\footnote{Free (as in freedom) and Open-Source software.} community and one from our industry partner.
It showed promising results in inferring a behavioural model from the autopilot execution data; showing significant improvement in both change point detection and state classification as compared with several baselines on 10 comparison metrics.

% I created a tool that automatically generated fuzz test scenarios for fixed wing aircraft that use Paparazzi. It could also execute the tests and collect telemetry data for training the model down the way. 
% I managed to replicate the study and get results as good as (in some cases even better) the first study on MicroPilot. In addition, I created a hyper-parameter tuning pipeline that significantly improved the results on Paparazzi's data; the default hyper-parameter values were [manually] tuned for MicroPilot.

Some potential extensions to this work include:
(a) Examining if and how transfer learning can be employed to make this method work with smaller datasets and also reduce the labeling overhead. 
(b) Adapting this method to work with other data types such as images in video surveillance systems, or sensor data + stream of images in self-driving cars, could be another interesting line of work for continuing this work
(c) Using the inferred model to perform a downstream task such as test generation or validation. 
(d) This approach at its current form is an offline method which is more useful for a postmortem analysis. To make it beneficial in more settings, it could be expanded into being an online (or even real-time) method.
