text:
Why is my neural network giving me wildly incorrect error and not changing accuracy?
My full code is as follows. I have tried to whittle it down to just the code that matters, but the problem I have is that i'm not sure what part of my network code is producing the problem. I've removed my code that loads and sifts through the CSV data because then my code would be too long.
#include <iostream>
#include <array>
#include <random>
#include <chrono>
#include <iomanip>
#include <fstream>
#include <algorithm>
#include <iomanip>
#include <variant>
#include <unordered_set>

typedef std::variant<std::string,std::uint_fast16_t,bool,float> CSVType;

/* ... functions to load CSV data ... */

typedef float DataType;
typedef DataType (*ActivationFuncPtr)(const DataType&);

DataType step(const DataType& x, const DataType& threshold)
{
    return x >= threshold ? 1 : 0;
}

DataType step0(const DataType& x)
{
    return step(x,0);
}

DataType step05(const DataType& x)
{
    return step(x,0.5);
}

DataType sigmoid(const DataType& x)
{
    return DataType(1) / (DataType(1) + std::exp(-x));
}

DataType sigmoid_derivative(const DataType& x)
{
    return x * (DataType(1) - x);
}

template<std::size_t NumInputs>
class Neuron
{
public:

    Neuron()
    {
        RandomiseWeights();
    }

    void RandomiseWeights()
    {
        std::generate(m_weights.begin(),m_weights.end(),[&]()
        {
            return m_xavierNormalDis(m_mt);
        });
        m_biasWeight = 0;

        for(std::size_t i = 0; i < NumInputs+1; ++i)
            m_previousWeightUpdates[i] = 0;
    }

    DataType FeedForward(const std::array<DataType,NumInputs> inputValues)
    {
        DataType output = m_biasWeight;
        for(std::size_t i = 0; i < inputValues.size(); ++i)
            output += inputValues[i] * m_weights[i];

        m_inputValues = inputValues;

        return output;
    }

    std::array<DataType,NumInputs> Backpropagate(const DataType& error)
    {
        std::array<DataType,NumInputs> netInputOverWeight;
        for(std::size_t i = 0; i < NumInputs; ++i)
        {
            netInputOverWeight[i] = m_inputValues[i];
        }

        DataType netInputOverBias = DataType(1);

        std::array<DataType,NumInputs> errorOverWeight;
        for(std::size_t i = 0; i < NumInputs; ++i)
        {
            errorOverWeight[i] = error * netInputOverWeight[i];
        }

        DataType errorOverBias = error * netInputOverBias;

        for(std::size_t i = 0; i < NumInputs; ++i)
        {
            m_outstandingWeightAdjustments[i] = errorOverWeight[i];
        }
        m_outstandingWeightAdjustments[NumInputs] = errorOverBias;

        DataType errorOverNetInput = error;

        std::array<DataType,NumInputs> errorWeights;
        for(std::size_t i = 0; i < NumInputs; ++i)
        {
            errorWeights[i] = errorOverNetInput * m_weights[i];
        }

        return errorWeights;
    }

    void AdjustWeights(const DataType& learningRate, const DataType& momentum)
    {
        for(std::size_t i = 0; i < NumInputs; ++i)
        {
            DataType adjustment = learningRate * m_outstandingWeightAdjustments[i] + momentum * m_previousWeightUpdates[i];
            m_weights[i] = m_weights[i] - adjustment;
            m_previousWeightUpdates[i] = adjustment;
        }
        DataType adjustment = learningRate * m_outstandingWeightAdjustments[NumInputs] + momentum * m_previousWeightUpdates[NumInputs];
        m_biasWeight = m_biasWeight - adjustment;
        m_previousWeightUpdates[NumInputs] = adjustment;
    }

    const std::array<DataType,NumInputs>& GetWeights() const
    {
        return m_weights;
    }

    const DataType& GetBiasWeight() const
    {
        return m_biasWeight;
    }

protected:

    static std::mt19937 m_mt;
    static std::uniform_real_distribution<DataType> m_uniformDisRandom;
    static std::uniform_real_distribution<DataType> m_xavierUniformDis;
    static std::normal_distribution<DataType> m_xavierNormalDis;

    std::array<DataType,NumInputs> m_weights;
    DataType m_biasWeight;

    std::array<DataType,NumInputs+1> m_previousWeightUpdates;
    std::array<DataType,NumInputs+1> m_outstandingWeightAdjustments;

    std::array<DataType,NumInputs> m_inputValues;
};

template<std::size_t NumInputs>
std::mt19937 Neuron<NumInputs>::m_mt(std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()).count());

template<std::size_t NumInputs>
std::uniform_real_distribution<DataType> Neuron<NumInputs>::m_uniformDisRandom(-1,1);

template<std::size_t NumInputs>
std::uniform_real_distribution<DataType> Neuron<NumInputs>::m_xavierUniformDis(-std::sqrt(6.f / NumInputs+1),std::sqrt(6.f / NumInputs+1));

template<std::size_t NumInputs>
std::normal_distribution<DataType> Neuron<NumInputs>::m_xavierNormalDis(0,std::sqrt(2.f / NumInputs+1));

template<std::size_t NumNeurons>
class ActivationLayer
{
public:

    ActivationLayer()
    :
        m_outputs({})
    {}

    virtual std::array<DataType,NumNeurons> GetOutputs() const final
    {
        return m_outputs;
    }

    virtual void CompleteBackprop(const DataType& learningRate, const DataType& momentum) final
    {
    }

protected:
    std::array<DataType,NumNeurons> m_outputs;
};

template<std::size_t NumNeurons>
class SigmoidActivation : public ActivationLayer<NumNeurons>
{
public:

    virtual std::array<DataType,NumNeurons> FeedForward(const std::array<DataType,NumNeurons>& inputValues)
    {
        for(std::size_t i = 0; i < NumNeurons; ++i)
            ActivationLayer<NumNeurons>::m_outputs[i] = sigmoid(inputValues[i]);
        return ActivationLayer<NumNeurons>::m_outputs;
    }

    virtual std::array<DataType,NumNeurons> Backpropagate(const std::array<DataType,NumNeurons> errors)
    {
        std::array<DataType,NumNeurons> backpropErrors;
        for(std::size_t i = 0; i < NumNeurons; ++i)
            backpropErrors[i] = errors[i] * sigmoid_derivative(ActivationLayer<NumNeurons>::m_outputs[i]);
        return backpropErrors;
    }
};

template<std::size_t NumInputs, std::size_t NumNeurons>
class FullyConnectedLayer
{
public:

    FullyConnectedLayer()
    :
        m_neurons([=]()
        {
            std::array<Neuron<NumInputs>,NumNeurons> neurons;
            for(Neuron<NumInputs>& n : neurons)
                n = Neuron<NumInputs>();
            return neurons;
        }())
    {
    }

    virtual std::array<DataType,NumNeurons> FeedForward(const std::array<DataType,NumInputs>& inputValues)
    {
        std::array<DataType,NumNeurons> outputValues;
        for(std::size_t i = 0; i < NumNeurons; ++i)
            outputValues[i] = m_neurons[i].FeedForward(inputValues);
        return outputValues;
    }

    /** \brief Take a sum of errors for each node and produce the errors for each input node in the previous layer.
     *
     */

    virtual std::array<DataType,NumInputs>
    Backpropagate(const std::array<DataType,NumNeurons> errors)
    {
        std::array<std::array<DataType,NumInputs>,NumNeurons> errorValues;
        for(std::size_t i = 0; i < NumNeurons; ++i)
        {
            errorValues[i] = m_neurons[i].Backpropagate(errors[i]);
        }
        std::array<DataType,NumInputs> returnErrors;
        std::fill(returnErrors.begin(),returnErrors.end(),0);
        for(std::size_t i = 0; i < NumNeurons; ++i)
        {
            for(std::size_t j = 0; j < NumInputs; ++j)
            {
                returnErrors[j] += errorValues[i][j];
            }
        }
        return returnErrors;
    }

    virtual void CompleteBackprop(const DataType& learningRate, const DataType& momentum)
    {
        for(Neuron<NumInputs>& n : m_neurons)
            n.AdjustWeights(learningRate, momentum);
    }

    const Neuron<NumInputs>& operator[](const std::size_t& index) const
    {
        return m_neurons[index];
    }

    std::array<std::array<DataType,NumInputs>,NumNeurons> GetWeights() const
    {
        std::array<std::array<DataType,NumInputs>,NumNeurons> weights;
        for(std::size_t i = 0; i < NumNeurons; ++i)
        {
            weights[i] = m_neurons[i].GetWeights();
        }
        return weights;
    }

protected:
    std::array<Neuron<NumInputs>,NumNeurons> m_neurons;
};

template<std::size_t I = 0, typename FuncT, typename... Tp>
inline typename std::enable_if<I == sizeof...(Tp)>::type for_each(std::tuple<Tp...> &, FuncT)
{
}

template<std::size_t I = 0, typename FuncT, typename... Tp>
inline typename std::enable_if<I < sizeof...(Tp)>::type for_each(std::tuple<Tp...>& t, FuncT f)
{
    f(std::get<I>(t)); // call f, passing the Ith element of the std::tuple t and the existing output O
    for_each<I + 1, FuncT, Tp...>(t, f); // process the next element of the tuple with the new output
}

template<std::size_t I = 0, typename FuncT, typename... Tp>
inline typename std::enable_if<I == sizeof...(Tp)>::type for_each(const std::tuple<Tp...> &, FuncT)
{
}

template<std::size_t I = 0, typename FuncT, typename... Tp>
inline typename std::enable_if<I < sizeof...(Tp)>::type for_each(const std::tuple<Tp...>& t, FuncT f)
{
    f(std::get<I>(t)); // call f, passing the Ith element of the std::tuple t and the existing output O
    for_each<I + 1, FuncT, Tp...>(t, f); // process the next element of the tuple with the new output
}

template<std::size_t I = 0, typename FuncT, typename O, typename FinalOutput, typename... Tp>
inline typename std::enable_if<I == sizeof...(Tp)>::type for_each_get_final_output(std::tuple<Tp...> &, FuncT, O o, FinalOutput& finalOutput)
{
    finalOutput = o;
}

template<std::size_t I = 0, typename FuncT, typename O, typename FinalOutput, typename... Tp>
inline typename std::enable_if<I < sizeof...(Tp)>::type for_each_get_final_output(std::tuple<Tp...>& t, FuncT f, O o, FinalOutput& finalOutput)
{
    auto newO = f(std::get<I>(t),o); // call f, passing the Ith element of the std::tuple t and the existing output O
    for_each_get_final_output<I + 1, FuncT, decltype(newO), FinalOutput, Tp...>(t, f, newO, finalOutput); // process the next element of the tuple with the new output
}

template<std::size_t I = 0, typename FuncT, typename O, typename... Tp>
inline typename std::enable_if<I == 0>::type for_each_reverse_impl(std::tuple<Tp...>& t, FuncT f, O o)
{
    f(std::get<0>(t),o);
}

template<std::size_t I = 0, typename FuncT, typename O, typename... Tp>
inline typename std::enable_if<(I > 0)>::type
for_each_reverse_impl(std::tuple<Tp...>& t, FuncT f, O o)
{
    auto newO = f(std::get<I>(t),o); // call f, passing the Ith element of the std::tuple t and the existing output O
    for_each_reverse_impl<I - 1, FuncT, decltype(newO), Tp...>(t, f, newO); // process the next element of the tuple with the new output
}

template<typename FuncT, typename O, typename... Tp>
inline void for_each_reverse(std::tuple<Tp...>& t, FuncT f, O o)
{
    for_each_reverse_impl<sizeof...(Tp)-1, FuncT, O, Tp...>(t, f, o);
}

enum class LOSS_FUNCTION : std::uint_fast8_t
{
    MEAN_SQUARE_ERROR,
    CROSS_ENTROPY
};

class ValidationOptions
{
public:
    enum class METRIC : std::uint_fast8_t { NONE, ACCURACY, LOSS };

    ValidationOptions()
    :
        m_validationSplit(.3f),
        m_enableLoss(false),
        m_lossFunction(LOSS_FUNCTION::MEAN_SQUARE_ERROR),
        m_enableAccuracy(false),
        m_outputFilter([](const DataType& x){ return x; }),
        m_earlyStoppingMetric(METRIC::NONE),
        m_earlyStoppingPatience(1.f),
        m_earlyStoppingDelta(1.f),
        m_earlyStoppingNumEpochsAverage(1)
    {}

    ValidationOptions& Loss(const bool enable = true, LOSS_FUNCTION lossFunction = LOSS_FUNCTION::MEAN_SQUARE_ERROR)
    {
        m_enableLoss = enable;
        m_lossFunction = lossFunction;
        return *this;
    }

    ValidationOptions& Split(const float dataSplitValidation)
    {
        m_validationSplit = dataSplitValidation;
        return *this;
    }

    ValidationOptions& Accuracy(const bool enable = true, ActivationFuncPtr outputFilter = [](const DataType& x){return x;})
    {
        m_enableAccuracy = enable;
        m_outputFilter = outputFilter;
        return *this;
    }

    ValidationOptions& EarlyStop(const bool enable = true,
                                 const METRIC metric = METRIC::ACCURACY,
                                 const float patience = .1f,
                                 const DataType delta = .01,
                                 const std::size_t epochNumToAverage = 10)
     {
         if(enable == false)
            m_earlyStoppingMetric = METRIC::NONE;
         else
            m_earlyStoppingMetric = metric;

         m_earlyStoppingPatience = patience;
         m_earlyStoppingDelta = delta;
         m_earlyStoppingNumEpochsAverage = epochNumToAverage;

         return *this;
     }

     float GetValidationSplit() const { return m_validationSplit; }
     bool Loss() const { return m_enableLoss; }
     LOSS_FUNCTION GetLossFunction() const { return m_lossFunction; }
     bool Accuracy() const { return m_enableAccuracy; }
     ActivationFuncPtr GetOutputFilter() const { return m_outputFilter; }
     METRIC GetEarlyStoppingMetric() const { return m_earlyStoppingMetric; }
     float GetEarlyStoppingPatience() const { return m_earlyStoppingPatience; }
     DataType GetEarlyStoppingDelta() const { return m_earlyStoppingDelta; }
     std::size_t GetEarlyStoppingNumEpochsAvg() const { return m_earlyStoppingNumEpochsAverage; }

protected:

    float m_validationSplit;        /**< Percentage of the data set aside for validation */

    bool m_enableLoss;
    LOSS_FUNCTION m_lossFunction;   /**< Loss function to use */

    bool m_enableAccuracy;
    ActivationFuncPtr m_outputFilter;   /**< When measuring accuracy data is passed through this */

    METRIC m_earlyStoppingMetric;                   /**< The metric used to stop early */
    float m_earlyStoppingPatience;                  /**< Percentage of total epochs to wait before stopping early */
    DataType m_earlyStoppingDelta;                  /**< The amount that the early stopping metric needs to change in a single step before stopping */
    std::size_t m_earlyStoppingNumEpochsAverage;    /**< The number of epochs averaged over to smooth out the stopping metric */
};

template<typename... Layers>
class NeuralNetwork
{
public:

    NeuralNetwork(Layers... layers)
    :
        m_layers(std::make_tuple(layers...))
    {

    }

    template<std::size_t NumFeatures, std::size_t NumOutputs, std::size_t NumTrainingRows>
    void Fit(const std::size_t& numberEpochs,
             const std::size_t& batchSize,
             DataType learningRate,
             const DataType& momentum,
             std::array<std::array<DataType,NumFeatures>,NumTrainingRows>& trainingData,
             std::array<std::array<DataType,NumOutputs>,NumTrainingRows>& trainingOutput,
             const ValidationOptions validationOptions,
             const bool linearDecayLearningRate = true,
             std::ostream& outputStream = std::cout)
    {
        std::size_t epochNumber = 0;

        // need to support more than just MSE to measure loss
        std::vector<DataType> lastEpochLoss(validationOptions.GetEarlyStoppingNumEpochsAvg(),0);
        DataType lastEpochLossAverage = std::numeric_limits<DataType>::max();

        std::vector<DataType> lastValidationAccuracys(validationOptions.GetEarlyStoppingNumEpochsAvg(),0);
        DataType lastValidationAccuraryAvg = 0;

        std::vector<std::size_t> randomIndices(NumTrainingRows,0);
        for(std::size_t i = 0; i < NumTrainingRows; ++i)
            randomIndices[i] = i;

        std::random_shuffle(randomIndices.begin(),randomIndices.end());
        // take some percentage as validation split
        // we do this by taking the first percentage of already shuffled indices and removing them
        // from what is available
        std::size_t numValidationRecords = NumTrainingRows*validationOptions.GetValidationSplit();
        std::size_t numTrainingRecords = NumTrainingRows - numValidationRecords;
        std::vector<std::size_t> validationRecords(numValidationRecords);
        for(std::size_t i = 0; i < numValidationRecords; ++i)
        {
            std::size_t index = randomIndices.back();
            randomIndices.pop_back();
            validationRecords[i] = index;
        }

        while(epochNumber < numberEpochs)
        {
            // shuffle the indices so that they are pulled into each batch randomly each time
            std::random_shuffle(randomIndices.begin(),randomIndices.end());

            DataType epochLoss = 0;

            std::tuple<Layers...> backupLayers = m_layers;

            for(std::size_t batchNumber = 0; batchNumber < std::ceil(numTrainingRecords / batchSize); ++batchNumber)
            {
                std::array<DataType,NumOutputs> propagateError = {0};

                std::size_t startIndex = batchNumber * batchSize;
                std::size_t endIndex = startIndex + batchSize;
                if(endIndex > numTrainingRecords)
                    endIndex = numTrainingRecords;

                DataType batchLoss = 0;

                for(std::size_t index = startIndex; index < endIndex; ++index)
                {
                    std::size_t row = randomIndices[index];
                    const std::array<DataType,NumFeatures>& dataRow = trainingData[row];
                    const std::array<DataType,NumOutputs>& desiredOutputRow = trainingOutput[row];

                    // Feed the values through to the output layer
                    // use of "auto" is so this lambda can be used for all layers without
                    // me needing to do any fucking around
                    std::array<DataType,NumOutputs> finalOutput;
                    for_each_get_final_output(m_layers, [](auto& layer, auto o)
                    {
                        return layer.FeedForward(o);
                    }, dataRow, finalOutput);

                    DataType totalError = 0;
                    for(std::size_t i = 0; i < NumOutputs; ++i)
                    {
                        if(validationOptions.GetLossFunction() == LOSS_FUNCTION::MEAN_SQUARE_ERROR)
                            totalError += std::pow(desiredOutputRow[i] - finalOutput[i],2.0);
                        else if(validationOptions.GetLossFunction() == LOSS_FUNCTION::CROSS_ENTROPY)
                        {
                            if(NumOutputs == 1)
                            {
                                // binary cross entropy
                                totalError += (desiredOutputRow[i] * std::log(1e-15 + finalOutput[i]));
                            }
                            else
                            {
                                // cross entropy
                            }
                        }
                    }

                    batchLoss += totalError;
                }

                batchLoss *= DataType(1) / (endIndex - startIndex);

                for(std::size_t i = 0; i < NumOutputs; ++i)
                    propagateError[i] = batchLoss;

                // update after every batch
                for_each_reverse(m_layers, [](auto& layer, auto o)
                {
                    auto errors = layer.Backpropagate(o);
                    return errors;
                }, propagateError);

                // once backprop is finished, we can adjust all the weights
                for_each(m_layers, [&](auto& layer)
                {
                    layer.CompleteBackprop(learningRate,momentum);
                });

                epochLoss += batchLoss;
            }

            epochLoss *= DataType(1) / numTrainingRecords;

            lastEpochLoss.erase(lastEpochLoss.begin());
            lastEpochLoss.push_back(epochLoss);
            DataType avgEpochLoss = 1.f * std::accumulate(lastEpochLoss.begin(),lastEpochLoss.end(),0.f) / (epochNumber < validationOptions.GetEarlyStoppingNumEpochsAvg() ? epochNumber+1 : lastEpochLoss.size());

            if(validationOptions.GetEarlyStoppingMetric() == ValidationOptions::METRIC::LOSS
               && epochNumber > numberEpochs * validationOptions.GetEarlyStoppingPatience()
               && avgEpochLoss > lastEpochLossAverage + validationOptions.GetEarlyStoppingDelta())
            {
                // the loss average has decreased, so we should go back to the previous run and exit
                std::cout   << "Early exit Loss Avg \n"
                            << "Last Epoch: " << lastEpochLossAverage << "\n"
                            << "This Epoch: " << avgEpochLoss << std::endl;
                m_layers = backupLayers;
                break;
            }

            lastEpochLossAverage = avgEpochLoss;

            // check for the error against the reserved validation set
            std::size_t numCorrect = 0;
            for(std::size_t row = 0; row < validationRecords.size(); ++row)
            {
                const std::array<DataType,NumFeatures>& dataRow = trainingData[row];
                const std::array<DataType,NumOutputs>& desiredOutputRow = trainingOutput[row];

                std::array<DataType,NumOutputs> finalOutput;
                for_each_get_final_output(m_layers, [](auto& layer, auto o)
                {
                    return layer.FeedForward(o);
                }, dataRow, finalOutput);

                bool correct = true;
                for(std::size_t i = 0; i < NumOutputs; ++i)
                {
                    if(validationOptions.GetOutputFilter()(finalOutput[i]) != desiredOutputRow[i])
                        correct = false;
                }
                if(correct)
                    ++numCorrect;
            }

            DataType validationAccuracy = DataType(numCorrect) / numValidationRecords;

            lastValidationAccuracys.erase(lastValidationAccuracys.begin());
            lastValidationAccuracys.push_back(validationAccuracy);
            DataType avgValidationAccuracy = std::accumulate(lastValidationAccuracys.begin(),lastValidationAccuracys.end(),0.f) / (epochNumber < validationOptions.GetEarlyStoppingNumEpochsAvg() ? epochNumber+1 : lastValidationAccuracys.size());

            if(validationOptions.GetEarlyStoppingMetric() == ValidationOptions::METRIC::ACCURACY
               && epochNumber > numberEpochs * validationOptions.GetEarlyStoppingPatience()
               && avgValidationAccuracy < lastValidationAccuraryAvg - validationOptions.GetEarlyStoppingDelta())
            {
                // the accuracy has decreased, so we should go back to the previous run and exit
                std::cout   << "Early exit validation accuracy \n"
                            << "Last Epoch: " << lastValidationAccuraryAvg << "\n"
                            << "This Epoch: " << avgValidationAccuracy << std::endl;
                m_layers = backupLayers;
                break;
            }

            lastValidationAccuraryAvg = avgValidationAccuracy;

            outputStream << epochNumber << "," << epochLoss << "," << avgEpochLoss << "," << validationAccuracy << "," << avgValidationAccuracy << std::endl;

            learningRate -= learningRate / (numberEpochs-epochNumber);

            ++epochNumber;
        }
    }

    template<std::size_t NumFeatures, std::size_t NumOutputs, std::size_t NumEvaluationRows>
    void Evaluate(std::array<std::array<DataType,NumFeatures>,NumEvaluationRows> inputData,
                  std::array<std::array<DataType,NumOutputs>,NumEvaluationRows> correctOutputs,
                  DataType& loss,
                  DataType& accuracy,
                  ActivationFuncPtr outputFilter = [](const DataType& x){return x;})
    {
        loss = 0;

        std::size_t numCorrect = 0;

        for(std::size_t row = 0; row < NumEvaluationRows; ++row)
        {
            const std::array<DataType,NumFeatures>& dataRow = inputData[row];
            const std::array<DataType,NumOutputs>& outputRow = correctOutputs[row];

            // Feed the values through to the output layer

            std::array<DataType,NumOutputs> finalOutput;
            for_each_get_final_output(m_layers, [](auto& layer, auto o)
            {
                layer.FeedForward(o);
                return layer.GetOutputs();
            }, dataRow, finalOutput);

            DataType thisLoss = 0;
            for(std::size_t i = 0; i < NumOutputs; ++i)
                thisLoss += outputRow[i] - finalOutput[i];
            loss += thisLoss * thisLoss;

            bool correct = true;
            for(std::size_t i = 0; i < NumOutputs; ++i)
            {
                if(outputFilter(finalOutput[i]) != outputRow[i])
                    correct = false;
            }
            if(correct)
                ++numCorrect;
        }

        loss *= DataType(1) / NumEvaluationRows;
        accuracy = DataType(numCorrect) / NumEvaluationRows;
    }

    template<std::size_t NumFeatures, std::size_t NumOutputs, std::size_t NumRecords>
    void Predict(std::array<std::array<DataType,NumFeatures>,NumRecords> inputData,
                 std::array<std::array<DataType,NumOutputs>,NumRecords>& predictions,
                 ActivationFuncPtr outputFilter = [](const DataType& x){return x;})
    {
        for(std::size_t row = 0; row < NumRecords; ++row)
        {
            const std::array<DataType,NumFeatures>& dataRow = inputData[row];

            // Feed the values through to the output layer

            std::array<DataType,NumOutputs> finalOutput;
            for_each_get_final_output(m_layers, [](auto& layer, auto o)
            {
                return layer.FeedForward(o);
            }, dataRow, finalOutput);

            for(std::size_t i = 0; i < NumOutputs; ++i)
                predictions[row][i] = outputFilter(finalOutput[i]);
        }
    }

protected:
    std::tuple<Layers...> m_layers;
};

main()
{
    std::vector<std::vector<CSVType>> trainingCSVData;
    /* load training CSV data */

    std::vector<std::vector<CSVType>> testCSVData;
    /* load test CSV data */

    std::cout << std::fixed << std::setprecision(80);

    std::ofstream file("error_out.csv", std::ios::out | std::ios::trunc);
    if(!file.is_open())
    {
        std::cout << "couldn't open file" << std::endl;
        return 0;
    }

    file << std::fixed << std::setprecision(80);

    /*
        Features
        1   pClass 1
        2   pClass 2
        3   pClass 3
        4   Sex female 1, male 0
        5   Age normalised between 0 and 1  age range 0 to 100
        6   Number siblings between 0 and 1   num range 0 to 8
        7   Number of parents / children        num range 0 to 9
        8   Ticket cost   between 0 and 1       num range 0 to 512.3292
        9   embarked S
        10  embarked Q
        11  embarked C
    */

    std::array<std::array<DataType,29>,891> inputData;
    std::array<std::array<DataType,1>,891> desiredOutputs;

    /* ... data that loads the titanic data into a series of features. Either class labels or normalised values (like age) */

    NeuralNetwork neuralNet{
        FullyConnectedLayer<29,256>(),
        SigmoidActivation<256>(),
        FullyConnectedLayer<256,1>(),
        SigmoidActivation<1>()
    };

    neuralNet.Fit(300,
                  1,
                  0.05,
                  0.25f,
                  inputData,
                  desiredOutputs,
                  ValidationOptions().Accuracy(true,step05).Loss(true,LOSS_FUNCTION::CROSS_ENTROPY).Split(0.3),
                  false,
                  file);

    file.close();

    return 0;
}

The data used is from the titanic problem that you can download from Kaggle here.
The typical output file that's being generated is like this:
0,-4.91843843460083007812500000000000000000000000000000000000000000000000000000000000,-4.91843843460083007812500000000000000000000000000000000000000000000000000000000000,0.65168541669845581054687500000000000000000000000000000000000000000000000000000000,0.65168541669845581054687500000000000000000000000000000000000000000000000000000000
1,-6.14257431030273437500000000000000000000000000000000000000000000000000000000000000,-6.14257431030273437500000000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
2,-6.43130302429199218750000000000000000000000000000000000000000000000000000000000000,-6.43130302429199218750000000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
3,-6.58864736557006835937500000000000000000000000000000000000000000000000000000000000,-6.58864736557006835937500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
4,-6.70884752273559570312500000000000000000000000000000000000000000000000000000000000,-6.70884752273559570312500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
5,-6.78206682205200195312500000000000000000000000000000000000000000000000000000000000,-6.78206682205200195312500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
6,-6.86832284927368164062500000000000000000000000000000000000000000000000000000000000,-6.86832284927368164062500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
7,-6.92110681533813476562500000000000000000000000000000000000000000000000000000000000,-6.92110681533813476562500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
8,-6.96584081649780273437500000000000000000000000000000000000000000000000000000000000,-6.96584081649780273437500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
9,-7.02414274215698242187500000000000000000000000000000000000000000000000000000000000,-7.02414274215698242187500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000
10,-7.06421041488647460937500000000000000000000000000000000000000000000000000000000000,-7.06421041488647460937500000000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000,0.65543073415756225585937500000000000000000000000000000000000000000000000000000000

(shortened for space)
This was previously working when the error I fed back through backprop was just the difference between the correct result and the prediction. But I've since been told that I should be propagating back the Loss Functions error, which I then implemented as Binary Cross Entropy.
So either:

I should not feed back the error from the loss function
I am calculating the loss incorrectly
My back propagation code is wrong
I've got something horrible happening in my validation loop

Do a forward and backrpop by hand and compare to the values you get in your code. Speaking of which, that is a huge slab of code. I would say you would be better posting this to data science exchange as ai is generally for philosophical and theory, but this is such a large amount of code I don't think anyone at data science exchange would want to go over it. I think your best bet is to debug your code in segments, and see how that goes.

Recessive seems correct to me. I don't usually mind reading a fair amount of code - I've answered similar implementation questions before, and have implemented neural networks in C (not C++ as here). But the size of the challenge to find your bug(s) in the attached code looks like too much effort for me. I will though have one guess.

@NeomerArcana No worries!  (The overwhelming majority of my questions across stack get close;)

This is a guess, as I am not reading all that code!

This was previously working when the error I fed back through backprop was just the difference between the correct result and the prediction. But I've since been told that I should be propagating back the Loss Functions error, which I then implemented as Binary Cross Entropy.

You may have been right before, and the advice you received wrong. Sort of.
Backpropagation does not feed back the error value directly. It works exclusively with gradients of the error. So you typically start with a gradient value based on the objective function.
If you backprop each function in the network a single step at a time, then you would start with $\nabla_{\hat{y}}\mathcal{L}$, the gradient of the loss function with respect to the estimated values (i.e. output of the NN). Then from that you could calculate $\nabla_{z}\mathcal{L}$, which is the gradient of the loss function with respect to the pre-activation values of the output layer.
However, it is possible to combine steps analytically, and really common to start with the first gradient being $\nabla_{z}\mathcal{L}$. That's because these pre-activation values are what you use to then calculate $\nabla_{W}\mathcal{L}$ and $\nabla_{b}\mathcal{L}$ - the gradients with respect to the layer's weights and biases, plus also to calculate $\nabla_{z'}\mathcal{L}$ - the gradients with respect to pre-activation values of the previous layer.
Either way, once you have $\nabla_{z}\mathcal{L}$ for the output layer you can just run back though each layer in turn, repeating the same calcuation steps again and again. It is this repetition over each layer that looks like classic backprop in code.
Assuming your neural network outputs a single value, the probablility of survival in the case of this dataset:

Your objective/loss function for binary classification should be Binary Cross Entropy.
Your output layer should have a sigmoid activation function.

If you take the gradient of the loss function and backpropagate it throught that output layer's activation function, you end up with a starting delta $\nabla_{z} \mathcal{L}$ i.e. the gradient of the loss function with respect to the pre-activation value of the first layer. This is a nice place to start the recursive backprop code.
The value of this gradient is mathematically, per item:
$$\nabla_{z} \mathcal{L} = \hat{y}_i - y_i$$
i.e. the difference the neural network estimate and the ground truth value. It takes this value because complex terms in the gradient of the loss function and the sigmoid activation function cancel out exactly. This is one reason why you often see sigmoid activation used with binary cross entropy - it is very convenient that the combination simplifies the gradient like this. You need to backpropagate each data point separately, and average the gradients for each batch/minibatch (actually you don't need to take this mean value, but doing so means that you don't need to adjust your learning rate as much to account for the number of items being processed per update step).
It appears to of worked for you.
Whether or not you were accidentally correct depends on whether you were getting an estimate for $\nabla_{z}\mathcal{L}$ to start your backprop routine, or $\nabla_{\hat{y}}\mathcal{L}$. I cannot find what that assumption is in your code, but the fact that it worked before and does not now suggests that you may of been correct, even though you may not of fully understood the maths.

Neil, I'm disinclined to close this question since time went into this answer.  Do you think would could reform the question to be more suitable.  (The OP themself voted to close and tried to delete, but was blocked once there was an answer, so I think they'd be ok with editing the question.)


text:
What is the underlying model of IBM Watson Assistant and Microsoft LUIS?
As I stated in my question, I would like to know the underlying pipeline and machine learning models that are used to classify intents and identify entities in IBM Watson Assistant and Microsoft LUIS services.
I searched on different websites and the documentations of those services, but I did not find anything. However, there are some blogs mentioned that IBM Watson is trained using one billion words from Wikipedia but there is no reference to support that claim.
I highly appreciate if anyone could refer me to a doc/blog that answers my question.
Thanks in advance :)

Hi and welcome to this community! Please, ask just about one tool (unless they are related), otherwise, this will be too broad.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Unable to achieve expected outputs using NEAT for the snake game
I am trying to implement NEAT for the snake game. My game logic is ready, which is working properly and NEAT configured. But even after 100 generations with 200 genomes per generation, the snakes perform very poorly. It barely ever eats more than 2 food. Below is the snip of the eval_genome function:
def eval_genome(genomes, config):
    clock = pygame.time.Clock()
    win = pygame.display.set_mode((WIN_WIDTH, WIN_HEIGHT))
    for genome_id, g in genomes:
        net = neat.nn.FeedForwardNetwork.create(g, config)
        g.fitness = 0
        snake = Snake()
        food = Food(snake.body)
        run = True
        UP = DOWN = RIGHT = LEFT = MOVE_SNAKE = False
        moveToFood = 0
        score = 0
        moveCount = 0
        while run:
            pygame.time.delay(50)
            clock.tick_busy_loop(10)
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    run = False
            snakeHeadX = snake.body[0]['x']
            snakeHeadY = snake.body[0]['y']
            snakeTailX = snake.body[len(snake.body)-1]['x']
            snakeTailY = snake.body[len(snake.body)-1]['y']
            snakeLength = len(snake.body)
            snakeHeadBottomDist = WIN_HEIGHT - snakeHeadY - STEP
            snakeHeadRightDist = WIN_WIDTH - snakeHeadX - STEP
            foodBottomDist = WIN_HEIGHT - food.y - STEP
            foodRightDist = WIN_WIDTH - food.x - STEP
            snakeFoodDistEuclidean = math.sqrt((snakeHeadX - food.x)**2 + (snakeHeadY - food.y)**2)
            snakeFoodDistManhattan = abs(snakeHeadX - food.x) + abs(snakeHeadY - food.y)
            viewDirections = snake.checkDirections(food, UP, DOWN, LEFT, RIGHT)
            deltaFoodDist = snakeFoodDistEuclidean

            outputs = net.activate((snakeHeadX, snakeHeadY, snakeHeadBottomDist, snakeHeadRightDist, snakeTailX, snakeTailY, snakeLength, moveCount, moveToFood, food.x, food.y, foodBottomDist, foodRightDist, snakeFoodDistEuclidean, snakeFoodDistManhattan, viewDirections[0], viewDirections[1], viewDirections[2], viewDirections[3], viewDirections[4], viewDirections[5], viewDirections[6], viewDirections[7], deltaFoodDist))

            if (outputs[0] == max(outputs) and not DOWN):
                snake.setDir(0,-1)
                UP = True
                LEFT = False
                RIGHT = False
                MOVE_SNAKE = True
            elif (outputs[1] == max(outputs) and not UP):
                snake.setDir(0,1)
                DOWN = True
                LEFT = False
                RIGHT = False
                MOVE_SNAKE = True
            elif (outputs[2] == max(outputs) and not RIGHT):
                snake.setDir(-1,0)
                LEFT = True
                UP = False
                DOWN = False
                MOVE_SNAKE = True
            elif (outputs[3] == max(outputs) and not LEFT):
                snake.setDir(1,0)
                RIGHT = True
                UP = False
                DOWN = False
                MOVE_SNAKE = True
            elif (not MOVE_SNAKE):
                if (outputs[0] == max(outputs)):
                    snake.setDir(0,-1)
                    UP = True
                    MOVE_SNAKE = True
                elif (outputs[1] == max(outputs)):
                    snake.setDir(0,1)
                    DOWN = True
                    MOVE_SNAKE = True
                elif (outputs[2] == max(outputs)):
                    snake.setDir(-1,0)
                    LEFT = True
                    MOVE_SNAKE = True
                elif (outputs[3] == max(outputs)):
                    snake.setDir(1,0)
                    RIGHT = True
                    MOVE_SNAKE = True

            win.fill((0, 0, 0))
            food.showFood(win)
            if(MOVE_SNAKE):
                snake.update()
                newSnakeHeadX = snake.body[0]['x']
                newSnakeHeadY = snake.body[0]['y']
                newFoodDist = math.sqrt((newSnakeHeadX - food.x)**2 + (newSnakeHeadY - food.y)**2)
                deltaFoodDist = newFoodDist - snakeFoodDistEuclidean
                moveCount += 1
                if (newFoodDist <= snakeFoodDistEuclidean):
                    g.fitness += 1
                else:
                    g.fitness -= 10
            snake.show(win)
            if(snake.collision()):
                if score != 0:
                    print('FINAL SCORE IS: '+ str(score))
                g.fitness -= 50
                break

            if(snake.eat(food,win)):
                g.fitness += 15
                score += 1
                if score == 1 :
                    moveToFood = moveCount
                    # foodEatenMove = pygame.time.get_ticks()/1000
                else:
                    moveToFood = moveCount - moveToFood
                food.foodLocation(snake.body)
                food.showFood(win)

Additionally, I am putting the definition of the checkDirections function. What it does is returns an array of size 8 corresponding to 8 directions where each value can be either 0 (not food or body), 1(food found but no body), 3(body found but no food), or 4(both body and food found).
def checkDirections(self, food, up, down, left, right):
        '''
        x+STEP, y-STEP
        x+STEP, y+STEP
        x-STEP, y-STEP
        x-STEP, y+STEP
        x+STEP, y
        x, y-STEP
        x, y+STEP
        x-STEP, y
        '''
        view = []
        x = self.xdir
        y = self.ydir

        view.append(self.check(x, y, STEP, -STEP, food.x, food.y))
        view.append(self.check(x, y, STEP, STEP, food.x, food.y))
        view.append(self.check(x, y, -STEP, -STEP, food.x, food.y))
        view.append(self.check(x, y, -STEP, STEP, food.x, food.y))
        view.append(self.check(x, y, STEP, 0, food.x, food.y))
        view.append(self.check(x, y, 0, -STEP, food.x, food.y))
        view.append(self.check(x, y, 0, STEP, food.x, food.y))
        view.append(self.check(x, y, -STEP, 0, food.x, food.y))

        if up == True:
            view[6] = -999
        elif down == True:
            view[5] = -999
        elif left == True:
            view[4] == -999
        elif right == True:
            view[7] == -999
        return view

    def check(self, x, y, xIncrement, yIncrement, foodX, foodY):
        value = 0
        foodFound = False
        bodyFound = False
        while (x >= 0 and x <= WIN_WIDTH and y >= 0 and y <= WIN_HEIGHT):
            x += xIncrement
            y += yIncrement
            if (not foodFound):
                if (foodX == x and foodY == y):
                    foodFound = True
            if (not bodyFound):
                for i in range(1, len(self.body)):
                    if ((x == self.body[i]['x']) and (y == self.body[i]['y'])):
                        bodyFound = True
            if (not bodyFound and not foodFound):
                value = 0
            elif (not bodyFound and foodFound):
                value = 1
            elif (bodyFound and not foodFound):
                value = 2
            else:
                value = 3
        return value

I am using sigmoid as the activation function. Although I have tried with tanh and relu as well with no luck. Below is the NEAT config file that I am using:
[NEAT]
fitness_criterion     = max
fitness_threshold     = 10000
pop_size              = 200
reset_on_extinction   = False

[DefaultGenome]
# node activation options
activation_default      = sigmoid
activation_mutate_rate  = 0.0
activation_options      = sigmoid

# node aggregation options
aggregation_default     = sum
aggregation_mutate_rate = 0.0
aggregation_options     = sum

# node bias options
bias_init_mean          = 0.0
bias_init_stdev         = 1.0
# was 30 max and -30 for min bias
bias_max_value          = 100.0
bias_min_value          = -100.0
bias_mutate_power       = 0.5
bias_mutate_rate        = 0.7
bias_replace_rate       = 0.3

# genome compatibility options
compatibility_disjoint_coefficient = 1.0
compatibility_weight_coefficient   = 0.5

# connection add/remove rates
conn_add_prob           = 0.8
conn_delete_prob        = 0.56

# connection enable options
enabled_default         = True
# below was 0.01
enabled_mutate_rate     = 0.3

feed_forward            = True
initial_connection      = full

# node add/remove rates
node_add_prob           = 0.7
node_delete_prob        = 0.4

# network parameters
num_hidden              = 0
num_inputs              = 24
num_outputs             = 4

# node response options
response_init_mean      = 1.0
response_init_stdev     = 0.0
response_max_value      = 30.0
response_min_value      = -30.0
response_mutate_power   = 0.0
response_mutate_rate    = 0.0
response_replace_rate   = 0.0

# connection weight options
weight_init_mean        = 0.0
weight_init_stdev       = 1.0
weight_max_value        = 30
weight_min_value        = -30
weight_mutate_power     = 0.5
weight_mutate_rate      = 0.8
weight_replace_rate     = 0.1

[DefaultSpeciesSet]
compatibility_threshold = 3.0

[DefaultStagnation]
species_fitness_func = max
max_stagnation       = 20
species_elitism      = 2

[DefaultReproduction]
elitism            = 2
survival_threshold = 0.2

If anyone has any insights or thoughts that could help improve the performance of the snake AI, please let me know.

As I can see, at each frame, you're giving 1 fitness if the snake gets closer to the food, and you remove 10 fitness if it gets away from the food. Am I right? Also, you're removing 50 fitness when a snake dies, and giving 15 fitness when it eats a food?
Try to remove the -10 fitness at each frame where the snake gets away from food, it's not necessary. Not getting fitness is also a penalty, you don't need to exaggerate the penalty as it's making your -50 fitness being negligible. Because of that, dying quickly is better than surviving for 6 frames without getting closer to the food. So the first strategy your AIs are going to find is spinning around to die quickly (because surviving is bad for them). This is not what you want.
But these are only assumptions, try it, and tell me if it works!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Which neural network algorithms can be used to map motion vectors in image processing?
I'm working on finding out the motion vectors of objects in images. The inputs are the images of objects in motion. The outputs of neural network are the object name, direction of object vector and prediction of next vector change.
There are different 3D ConvNets I'm considering as a baseline like ReMotENet. I would appreciate if you would recommend any interesting papers in MoCap domain and any existing neural networks performing similar task.


text:
Is there any time-varying directed graph dataset?
I am interested in the node classification task for graph data. So far,I've tried it with the Cora dataset, but it is an undirected graph and has word attributes as features. I want to extend this task to a time-varying directed graph. Does anybody know about this kind of dataset?


text:
Why they use KL divergence in Natural gradient?
Natural gradient aims to do a steepest descent on the "function" space, a manifold that is independent from how the function is parameterized. It argues that the steepest descent on this function space is not the same as steepest descent on the parameter space. We should favor the former.
Since, for example in a regression task, a neural net could be interpreted as a probability function (Gaussian with the output as mean and some constant variance), it is "natural" to form a distance on the manifold under the KL-divergence (and a Fisher information matrix as its metric).
Now, if I want to be creative, I could use the same argument to use "square distance" between the outputs of the neural nets (distance of the means) which I think is not the same as the KL.
Am I wrong, or it is just another legit way? Perhaps, not as good?

The KL divergence has slightly different interpretations depending on the context. The related Wikipedia article contains a section dedicated to these interpretations. Independently of the interpretation, the KL divergence is always defined as a specific function of the cross-entropy (which you should be familiar with before attempting to understand the KL divergence) between two distributions (in this case, probability mass functions)
\begin{align}
D_\text{KL}(P\parallel Q)
&= -\sum_{x\in\mathcal{X}} p(x) \log q(x) + \sum_{x\in\mathcal{X}} p(x) \log p(x) \\
 &= H(P, Q) - H(P)
\end{align}
where $H(P, Q)$ is the cross-entropy of the distribution $P$ and $Q$ and $H(P) = H(P, P)$.
The KL is not a metric, given that it does not obey the triangle inequality. In other words, in general, $D_\text{KL}(P\parallel Q) \neq D_\text{KL}(Q\parallel P) $.
Given that a neural network is trained to output the mean (which can be a scalar or a vector) and the variance (which can be a scalar, a vector or a matrix), why don't we use a metric like the MSE to compare means and variances? When you use the KL divergence, you don't want to compare just numbers (or matrices), but probability distributions (more precisely, probability densities or mass functions), so you will not compare just the mean and the variance of two different distributions, but you will actually compare the distributions. See the example of the application of the KL divergence in the related Wikipedia article.

MSE doesn't compare variance, I know, but I could argue that my neural net is not a probabilistic function (it is a deterministic one), hence there is no variance. A natural gradient is then should be formed in MSE distance of the function output?

Yes, Squared distances & KL Divergence are not the same. Squared distance between means is not a useful metric as it doesn't gauge the amount of similarity between 2 distributions.
When we compute \begin{align}
D_\text{KL}(P\parallel Q)
\end{align}
We are computing the amount of information that is lost when we approximate P as Q. Ideally, we would want the KL divergence to be as low as possible.
Here is an interesting article https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained
where the author has explained KL Divergence with a toy example.
I hope it helps :)

It is true if the assumption is that the neural network is a probability function. However, the assumption is arbitrary. I could do equally well if I assume that the neural net is a deterministic function. The square distance makes much more sense now.

I have been reading a lot about Natural Gradient and its use to find a descent direction. I found that this post was the most clear.
Consider a model $p$ parameterized by some parameters $\theta$ and we want to maximize the likelihood of observing our data $x$ under this model: $p(x|\theta)$. To optimise this likelihood we can take steps in the the distribution space. Updating the parameters $\theta$ we need to measure how our likelihood changes and this is measure using the KLK divergence.
Even though the KL divergence is not a "proper" distance metric as it is not symmetric, it is still quite informative about the similarity between distributions. It's practical because it can capture differences between distributions that the Euclidean metric (parameter-dependent) could not (see the same post for a simple example).
So answering your question is essentially answering which is the best between Natural Gradient Descent and "Normal" Gradient Descent in the Euclidean space where your loss is measured with a L2 norm.
You can train the same model using both methods and you will just find different descent directions.
Hopefully though, both will converge but in my opinion Natural Gradient descent should be superior in nature. It is just very expensive to actually compute because to find the direction in distribution space you need to compute the inverse Fisher matrix $F^{-1}$ or approximate it and that's quite costly as it is of size $n\times n$ where $n$ is the size of $\theta$ which is typically high in neural networks.


text:
Solving the dead time problem for control using reinforcement learning
There are several occasion that reinforcement learning can be used as a control mean.
The action is for example the set target temperature (which in many occasions change with time) and the state is for example the current temperature and other variables. The policy is then the control mean that is going to be learnt using the reinforcement learning.
As there is a dead time (input lag) and time delay in the real world, how can one propose to tackle this problem when using reinforcement learning as a control mean? Thank you.

Could you explain why you think a delay between action and observable effects of the action would be a problem? Generally in reinforcement learning it is not an issue, but maybe there is some special case for your example that I am missing.

Related (maybe a duplicate?) https://ai.stackexchange.com/questions/8267/dealing-with-lags-in-reinforcement-learning

I think you are right. This is the same problem that I ask here. I didn't find it before, but thank you.

Possible duplicate of Dealing with Lags in Reinforcement Learning

OK. If lag effects are extreme, such that observable state does not change at all, but "momentum" is building due to actions taken on each time step, and the target may overshoot, then you may need to make some changes to the state representation. If you think that applies in your case, then it may not be a duplicate. If so, add that detail to the question  . . . otherwise it is good to have duplicates because it is another way to word the same question, so more chance of someone else looking to find the same answer.

Yes, it is right. The lag effects are so huge that after a certain time steps, it will be noticable but it will not overshoot in my case.

You need lag+ "momentum" for this to be a worry. If the subsystem that the agent is controlling is a traditional control with damping already built in, the lag won't matter. If the agent is directly controlling a heating element a few rooms away though, it may matter more.

According to your terminology maybe you are right to add "momentum". I used the term dead-time  in control system because normally a system has this property and when the dead-time is big then it poses a problem while using reinforcement learning to control it. Time delay corresponds to the delay due to the dynamics of the system in my terminology. Thanks


text:
How to handle classification with label updates?
Suppose that my task is to label news articles; that is, to classify which category a news article belongs to. Using the labelled data (with old labels) that I have, I have trained a model for this.
For relevancy purposes, certain labels may be split into multiple new labels. For example, 'Sports' may split into 'Sports' and 'E-Sports'. Because of these new labels, I will need to retrain my model. However, my training data is labelled with the old labels. What can I do to address these 'label updates'?
My idea: Perhaps use some unsupervised clustering method (K-means?) to split the data with the old labels into the new labels. (But how can we be certain that which cluster has what new label?) Then use this 'updated' data to train a model. Is this correct?

so you want to train your new model without having the new data? My first guess would be to relabel the old data.

Yes, I want to train my new model without having the new data. Manually relabelling the old data seems impractical to me if my dataset is large, hence I thought of using K-means to do the splitting. I am unsure if this is correct.

Doing some unsupervised learning will give you some divisions within each label but you can't be sure it will be split on "E-sports" and "Sports". It might as wel split sports into "land sports" and "water sports" or "long sports" and "short sports".
The only way to reliably split "sports" in 2 subsections of your desire is to relabel this manualy I am afraid.


text:
Neural network for reinforcement learning
I’m using a simple neural network to solve a reinforcement learning problem.
The configuration is:
X-inputs: The current state
Y-outputs: The possible actions
Whenever the network yields a “good” solution, i “reward” the network by training it a number of times.
Whenever the network yields a “bad” or “neutral” solution, i ignore it.
This seems to be working somewhat, but from what i read, everyone else (in broad terms) seems to be using a 2 neural network configuration for similar tasks. (Policy network and value network)
Am i missing something? - and are there any obvious caveats of the “single network” method i am using?
Supplemental question: Are there other methods of “rewarding” a network, aside from simply training it?
Thanks,

Is the output in your case a softmax over action choices? If that's the case, you are using a well-known RL method, and an answer can give its strengths and weaknesses compared to other methods.

From your description, it seems that you are implementing a version of an algorithm called REINFORCE. This algorithm belongs to a family called Policy Gradient methods, which directly optimizes the policy network $\pi(a_t|s_t)$ from rewards without ever worrying about estimating a value function. This type of algorithm is usually pretty slow and presents high variance.
The methods that you recognize as the ones using two neural networks correspond to a family called Actor-Critic methods. This type of algorithm uses the trajectories of rewards to estimate a value network $q(s_t,a_t)$ (called the critic), and, contrary to the previous family of methods, it uses the value network to train the policy network $\pi(a_t|s_t)$ (called the actor), instead of directly using the trajectory of rewards. This indirect dependence usually makes variance smaller and also learning faster. I recommend you have a look at chapter 13 of the book An Introduction to Reinforcement Learning.
So, to answer your first question: it seems you are missing the family of Actor-Critic methods. I recommend you learn about them since they are very powerful (e.g., read about DDPG or SAC).
About your second question, the standard method to "reward" a policy network is not by training it. Usually, you have a reward function $r(s_t,a_t)$ that depends on your current state $s_t$ and action $a_t$ and you modify the parameters $\theta$ of your network in such a way that the probability of an action $\pi(a_t|s_t)$ increases if the reward is positive or decreases if it is negative. More specifically, you perform stochastic gradient ascend steps like this one:
$$\theta_t\leftarrow\theta_t+\alpha\mathbb E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]\nabla \log\pi(a_t|s_t,\theta_t)$$
What this formula says is that if in the time-step $t$ you take an action $a_t$ in the state $s_t$, wait $T$ steps and collect the rewards from $r(s_t,a_t)$ to $r(s_{t+T},a_{t+T})$, then, you should modify your parameters in the direction that the policy increases the most (i.e., $\nabla \log\pi(a_t|s_t,\theta_t)$) if the expected return $E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]$ is positive, or in the direction that the policy decreases the most if that expected return is negative.

So, @NeilSlater are you saying that there exists a target distribution that we know beforehand? I considered that, but I found it very weird that one could know that target distribution, which is precisely the solution of the RL problem. If you take your reward as 1 for "good" results and 0 for "bad" ones, you can see that the only difference is that in REINFORCE the loss function is the loglikelihood of a specific action, while in the Cross Entropy Method it is the dot product between the loglikelihood and that target distribution.

@NeilSlater, but according to you exactly what is the loss function being used to train the network? If it is the Cross Entropy with the action itself (a vector with 0s in all positions except for the position corresponding to the action taken, say the i-th position, and a 1 in this  i-th position), then as I already explained in the previous comment, it is exactly the same as REINFORCE using a reward function that has outcomes 0 and 1. This is the case since the dot product between the loglikelihood and a vector of 0s and a 1 in the i-th entry is precisely log pi(a_t|s_t).

I still do not understand what would be the loss function. As far as I understand, it is some kind of distance between your output distribution and the target distribution that you build with your actions. Well, a typical loss function when your output is a distribution and your target is also a distribution is the Cross Entropy. Whether you derive CEM or not from the policy gradient theorem is not relevant. I'm telling you (just look at the formula) that CEM turns out to be the same as REINFORCE in the especial case that you choose a specific binary reward function and the Cross Entropy loss.

I see your point in calling my characterization "innacurate" and I will edit those parts. I could argue that CEM is a funny version of REINFORCE in the context of RL, though XD. However, I think it is more appropriate to the principal question to introduce the Actor-Critic methods as opposed to the Policy Gradient methods, not the particular CEM algorithm that turns out to be equivalent to a specific version of REINFORCE

To make CEM weight updates look the same as basic REINFORCE, you need to filter down to episodes where the only and final reward is +1 (which is not a great use for CEM - it works much better when returns can be differentiated), and also set discount factor to 1. Thanks for the edit anyway - it doesn't go quite as far as I'd like by identifying the algorithm the OP appears to be using, but at least it doesn't tell the OP that they are doing something "funny" :-)


text:
CNN multi output scores and evaluation
I am building a CNN with two outputs. I still have to put time in the network itself, but I was trying to get a good evaluation/classification report of the results. My code is the following:
scores = model.evaluate(data_test, [Y1_test, Y2_test], verbose=0)

for i, j in zip(model.metrics_names, scores):
    print(i,'=', j)

Output:
loss = 5.124477842579717
Y1_output_loss = 1.3782909
Y2_output_loss = 4.10769
Y1_output_accuracy = 0.6304348
Y2_output_accuracy = 0.54347825

Not great, but that is not the point. My code for the classification repot is as follows:
Y1_pred, Y2_pred = model.predict(data_test)
Y1_true, Y2_true = Y1_test.argmax(axis=-1), Y2_test.argmax(axis=-1)
Y1_pred, Y2_pred = Y1_pred.argmax(axis=-1), Y2_pred.argmax(axis=-1)

print(classification_report(Y1_true, Y1_pred))
print(classification_report(Y2_true, Y2_pred))

Output:
Classification report Y1
              precision    recall  f1-score   support

           0       0.20      0.33      0.25         6
           3       0.00      0.00      0.00         3
           6       0.00      0.00      0.00         6
           8       0.00      0.00      0.00         2
           9       0.00      0.00      0.00         7
          10       0.03      0.50      0.06         2
          11       0.00      0.00      0.00         3
          12       0.00      0.00      0.00         7
          13       0.00      0.00      0.00         2
          14       0.00      0.00      0.00         7
          15       0.00      0.00      0.00         1

    accuracy                           0.07        46
   macro avg       0.02      0.08      0.03        46
weighted avg       0.03      0.07      0.04        46

Classification report Y2
              precision    recall  f1-score   support

           0       0.00      0.00      0.00         9
           2       0.00      0.00      0.00        10
           3       0.15      1.00      0.26         7
           4       0.00      0.00      0.00         9
           5       0.00      0.00      0.00         6
           6       0.00      0.00      0.00         2
           7       0.00      0.00      0.00         3

    accuracy                           0.15        46
   macro avg       0.02      0.14      0.04        46
weighted avg       0.02      0.15      0.04        46

Now the average accuracy is extremely low suddenly, so I have the feeling it isn't lining up correctly. But I don't see where?
Thank you all

It looks like your model was trained to reproduce output vector of input and we have accuracy only a bit higher than random guesses. The chance that an index of a maximum element of prediction and from a sample will be the same will depend mainly on a size of your vector in this case. Other than that, I think that it's worth trying MLP with SoftMax as an output on top of CNN you have.


text:
Why are researchers focused on deep learning based stereo depth/disparity methods instead of non deep learning ones?
In recent years if you are working on stereo depth/disparity algorithms, it seems like you will only ever get your paper accepted to CVPR/ICCV/ECCV if there's some deep learning involved in it. A lot of authors published their code on github and I've tried out multiple of them and here is what I observed. None of these deep learning based methods generalized well. Almost all methods trained on the KITTI dataset (street images) or the scene flow dataset (synthetic images). These methods perform well when the test data is similar to the training data, but fails miserably on other kinds of test data (e.g. close up human) whereas a classical traditional computer vision based method like PatchMatch would generate decent results. In my opinion, no matter how well these new deep learning methods perform on the KITTI benchmark, it's nearly useless in the real world.
I understand deep learning has the potential to approximate any non-linear function when there's enough quality training data and unlimited computation, but ground truth depth/disparity cannot be labeled by manual labor like a cat-dog classification problem. That means the ground truth training data has to come from traditional computer vision algorithms or hardware or be synthetic. Traditional computer vision algorithms are not even close to perfect yet but the research pretty much stifled because of deep learning. The ground truth of the KITTI dataset comes from a hardware LIDAR, but it's extremely sparse. If we align multiple scans from LIDAR in order to form a dense result, that's relying on some type of SLAM which again is relying on an imperfect traditional computer vision algorithm. There is no sign of hardware that can generate accurate dense depth that is coming out soon. As for synthetic data, it doesn't accurately represent real data. Since there isn't even a good way to obtain training data for stereo depth/disparity, why are the researchers so fixated on building complex deep neural nets to solve stereo depth/disparity nowadays?


text:
Analyzing vibration using machine learning
I would like a few suggestions on an idea that I have -
I am trying to make a musical instrument (percussion), whilst just having a PVC disc. I am hitting the disc in a variety of styles (in order to produce a variety of sounds correspondingly), just like the way the actual percussion instrument is hit. I am converting the mechanical vibrations on the PVC disc to an electrical signal using a transducer, performing an FFT analysis of the different strokes, and trying to identify the stroke which is hit. Using this technique, I could get an accuracy of only 80 percent. I would like it to be extremely accurate ( more than 95 percent recognition). I was using only frequency as the parameter used to distinguish the sounds.
Now, I am thinking that if I could use other parameters too in order to identify the stroke, I might be able to get the required accuracy. I am thinking of resorting to Machine Learning for this. I am kind of new to this and would like to know what I might need to know before I proceed with this idea.
Any help would be greatly appreciated.

If you want to use machine learning for such a project, you can use vibrations data directly, and treat the problem as a regular audio classification problem.
A simple approach would be to use a Neural Network with Convolutions. This would take care of features extraction for you. And maybe follow these by dense layers at the end.
Given that, it would be easier to make suggestions if you posted samples of your data.
Edit: Also, keep in mind that machine learning usually requires large datasets so if you are collecting data all by yourself in the way you describe, you might not have enough sample to run a good model. In such a case when number of samples is limited one can use transfer learning - using a pre-trained model - but I am not aware of any such pre-trained models for wave data.


text:
The best way of classifying a dataset including classes with high similarity?
I have a dataset which has two very similar classes (men wrestling, women wrestling). I've used InceptionV3 as a classifier to solve the problem of classifying this dataset. Unfortunately, the accuracy of this classifier doesn't hit more than 70%. Is there any suggestion about how I can overcome this problem or any other similar problems?

How much data do you have?

About ten thousand.
Regardless of amount of data, I would be grateful for any suggestion

If you want to classify data with similar characteristics, it would often helps if you hand craft features. For classifying women or men wrestling, you may want to try using cv2 to track human faces and feed that to your CNN as input. Example: https://realpython.com/face-recognition-with-python/
If the data is not images, you may want to do a analysis to see which feature have no clear relationship with whether it is men or women wrestling and remove them. Example: https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0
Hope that I can help you and have a nice day!


text:
TensorFlow 2.0 - Normalizing input to DNN (on structured data)
I have a structured dataset of around 100 gigs, and I am using DNN for classification in TF 2.0. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.
Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have been reading TensorFlow docs to get info on how to normalize features when reading data in batches. But, couldn't find one. though I found this article, it is only for the case where entire data can be loaded in memory.
So, If any of you have worked on creating such a TensorFlow data pipeline for normalizing input features while loading data in batches and training model, It would be helpful.

The central limit theorem tells us that the error in an estimate of the mean or standard deviation of a dataset will decline as $\frac{1}{\sqrt{n}}$, where $n$ is the number of samples taken at random from the set, and combined together to compute the mean and deviation.
If you select at random, for example, $10^6$ examples (probably a few megabytes), then the mean and standard deviation you compute from those examples will be within $\frac{1}{sqrt(10^6)} = \frac{1}{10^3}$ of the "true" answer. That's one part in 1,000, which is certainly accurate enough to use for re-scaling the dataset.

I was thinking along the same lines, To use certain number of observation for getting a ball park number, and now I could also quantity the error! But isn't there any more elegant solution to this?

You can also calculate mean and standard deviation using single-pass "streaming" approaches. Books like http://numerical.recipes/ cover these algorithms, and many more statistical algorithms for single-passes through batches of a dataset. However, the central limit theorem tells us these will probably not improve your accuracy enough to be worthwhile. Why spend many minutes (or perhaps hours) reading through all the data for a 0.1% improvement in a statistic you use only for normalization (which doesn't need to be that accurate)?

Indeed, its better to invest that time doing something  else that could actually help improve accuracy. Thank you

One way could be to first iterate over the dataset in batches, just to get the mean and sd. Then when running training, use the true population parameters obtained before.


text:
What is the maximum number of dichotomies in a square?
I am new to machine learning. I am reading this blog post on the VC dimension.
$\mathcal H$ consists of all hypotheses in two dimensions $h: R^2 → \{−1, +1 \}$, positive inside some square boxes and negative elsewhere.
An example.

My questions:

What is the maximum number of dichotomies for the 4 data points? i.e calculate mH(4)
It seems that the square can shatter 3 points but not 4 points. The $\mathcal V \mathcal C$ VC dimension of a square is 3. What is the proof behind this?

"What is the maximum number of dichotomies for the 4 data points?", but in the picture you're showing, there are 5 data points (or numbers).

The picture is just an example showing one of the possibilities of data points distributions. For point INSIDE the square, it is +1, For point OUTSIDE the square, it is -1.  My question is asking 4 points, it is correct and unrelated to 5 points in my drawing.


text:
Segmentation of a static object in a video
I've videos from a mounted camera on a helmet and the manually segmented labels (mask) of them.
The mask is valid through the entire video, only the scene vary.
In different videos the camera is mounted differently on top of the helmet.
Things I've tried:

Training semantic segmentation on frame-mask pairs
Training semantic segmentation of concatenated frames with the mask.
Averaging consecutive frames and calculating the time-wise std of the pixels, and feeding the NN with this as an input.
ensembling (averaging) segmentation results from N frames
Usage of classic background subtraction techniques such as MOG2 (worse)

Although DNN models achieve 99% accuracy during training, in some of my test videos the model is missing large part of the helmet..
I'm certain this task can achieve ~100% accuracy even for never-before-seen examples.
Do you have some ideas?


text:
Can a Video Game Characters Behavior be directed by a NN?
So, I’m looking into some dynamic ways in which one can drive the behavior of a video game character. Specifically an NPC (Non playable character) that will be observable from the players point of view. Something I’d like to clarify from the start is what I mean by behavior. Since video games are understood visually, then I would qualify behavior to be anything visual, such as gestures, mannerisms or actions in any local space.
Let’s take a common archetype as an example. We’ll say we want the behavior of a villain. My first thought, was to use videos as training data. Videos of specific subjects or actors in a villainous scene (think Frankenstein, Dracula, Emperor Palpatine in Star Wars etc...) in hopes that an understanding of their mannerisms, body language and gestures could be captured and later applied to 3D animation dynamically.
I do understand that anything 3D typically requires rigging and animation. I’m currently not exactly sure how to marshal the data from one format (video analyses) to 3D animation. I thought I’d start working on the concept from high level first.
Any thoughts?

Is it safe to assume that this is just about animation, and not action choices? So you might be interested in things like motion style transfer e.g. https://www.youtube.com/watch?v=AvVQZjygQBQ - it's not the only thing that might be of interest in this topic area.

Right, but for those of us who do not want to use a traditional motion capture method, couldn't one use semantic segmentation mapping for video footage to create the needed labels for input into an algorithm? A google search of "Video Semantic Segmentation" reveals how this can be done. Its just, I don't know which algorithm to use for capturing the expressiveness of the motion. Can we approach this in a non-traditional way?


text:
Deep audio fingerprinting for word search
Simply speaking, I'm trying to somehow search an audio clip for a list of words, and if found, I mark the time stamps. My use-case is profanity check with a list of pre-defined profane words.
Is there any successfull approaches, samples, tools or APIs, possibly through deep learning, to perform this? I'm new to audio processing.

You would need to perform some kind of speech-to-text to get the audio transcription with the corresponding synchronization wrt the audio. Then search in the transcription.
You could use DSAlign by mozilla


text:
How to create an AI to solve a word search?
This at first sounds ridiculous. Of course there is an easy way to write a program to solve a wordsearch.
But what I would like to do is write a program that solves a word-search like a human.
That is, use or invent different strategies. e.g. search randomly for the starting letter; go line-by-line;
Probably the AI will eventually find out that going line by line looking for a given starting letter of a word is a good strategy.
Any idea how you would write such a strategy-finding AI?
I think the main "moves" would be things like "move right one letter in the grid", "store this word in memory", "compare this letter with first letter in memory" and a few more.

This sounds like a problem that might be solvable with a LSTM-DQN approach, as described in Language Understanding for Text-based Games using Deep
Reinforcement Learning by Narasimhan et al., 2015, and then extended to a domain very similar to your problem in Deep Reinforcement Learning for Syntactic Error Repair in Student Programs by Gupta et al., 2019.
The basic idea is to treat the array of letters each of which can be 'circled' or not, plus the position of a cursor as a state. An action is to move the cursor or toggle the state of a letter. You then model the problem as a reinforcement learning problem, with rewards given every time an entire new word becomes 'circled', probably with a caveat about not circling any invalid letters (otherwise, it'll just learn to circle everything).
This is just a guess, but it seems like it's very closely related to your problem, so it's likely to be effective.

Thanks. They seem like very modern papers. Surprising! I'll take a look.

This kind of AI (reinforcement learning) has only really come into its own recently. The field has existed for a long time, but wasn't been feasible for real problems until 2015.

@zooby There are other (older) ways you could try to solve this problem, but this is the approach I think is most promising given the modern state-of-the-art. Hope you find it useful.

I think yes, it is theoretically a reinforcement leaning problem. I wonder, however, whether such methods would solve the problem in a realistically human timeframe given the amount of combinations involved.... Well, I may have to just try it!

An LSTM in theory should be able to solve the problem, but again I wonder about scaling... I feel like it needs some semantic knowledge somehow.

@zooby This approach will probably require many times more exposure to word-search problems than a human would (keep in mind: it's going to have to learn English from scratch, as well as to take the right actions). The beauty of the approach, to my mind, is that it doesn't need semantic knowledge. This is increasingly true of most of the best approaches to language-related problems, and usually these systems scale better than semantic ones, rather than worse.


text:
AI with conflicting objectives?
A recent question on AI and acting recalled me to the idea that in drama, there are not only conflicting motives between agents (characters), but a character may themselves have objectives that are in conflict.
The result of this in performance is typically nuance, but also carries the benefit of combinatorial expansion, which supports greater novelty, and it occurs to me that this would be a factor in affective computing.
(The actress Eva Green is a good example, where her performances typically involve indicating two or more conflicting emotions at once.)
It occurs to me that this can even arise in the context of a formal game where achieving the most optimal outcome requires managing competing concerns.

Is there literature or examples of AI with internal conflicting objectives?

MOEAs sounds very cool, but I feel that you can't really talk about conflict in AI without discussing generative adversarial networks (GANs), which have been shown to have amazing performance by training a model to say detect in-between pictures of cats and dogs and an adversarial network being trained to create pictures to attempt to trick the training network as much as possible. The completely conflicting objectives of the networks enable both to be trained very well so the models, in the end, are much more robust and able to handle sometimes bizarrely generated edge cases.
I also found this paper Evolutionary Multi-Objective Optimization Driven by Generative Adversarial Networks (GANs) (2019), which combines MOEAs and GANs, but there are potentially more related papers.

There are multi-objective optimization problems, where the objective functions may be in conflict with each other, which can potentially have multiple Pareto-optimal solutions. The paper Multi-objective optimization using genetic algorithms: A tutorial (2006) gives a good overview of the multi-objective optimization problem with genetic algorithms, which can be called evolutionary multi-objective optimization (EMO) or multi-objective optimization evolutionary algorithms (MOEAs).
A common multi-objective genetic algorithm is NSGA (or NSGA-2 and NSGA-3), which stands for Non-dominated Sorting Genetic Algorithm, which is based on the concepts of non-dominated sorting, Pareto front and optimality, niches (sub-populations), and elitism (the best individuals of the current population are carried over to the next generation).
If you want to play with MOEAs, you may wanna try the Python deap package, which supports, for example, the NSGA-2 algorithm.

I've only fully read the paper NSGA-2, which I advise you to read, if you want to get into MOEAs. I've skimmed through the Multi-objective optimization using genetic algorithms: A tutorial.


text:
AlphaGo Zero: Does the policy head give a probability for every possible move?
If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v.
My question is: in this vector that it is outputted, do we have a probability for every possible action in the game? If so: does it apply a probability of 0 to actions that are not possible in that particular state? If this is true, how does the network know which actions are valid?
If not: then the network will output vectors of different sizes according to each state. Is this even feasible? And again, how will the network know which actions are valid?
Related questions but none of them covers this question in specific: 1, 2 and 3.

Not a duplicate, but more directly answers your question: https://ai.stackexchange.com/questions/2980/how-to-handle-invalid-moves-in-reinforcement-learning I recall answering something that would be a duplicate, but cannot find it. Might be on Data Science instead, perhaps . . . anyway, short anser is yes that the NN has every move encoded, and no the NN it never learns "not to play" invalid moves, instead validity is something that the environment is expected to supply as a callable function that the agent can use to filter choices.

Thanks for the link @NeilSlater, it is very helpful! One last question, to see if I understood correctly: so the neural network will return probabilities for all possible actions. After I get the vector from the network, I will have to deal with impossible moves and distribute the probabilities assigned to them to all other valid moves (proportionally). Is that right?

That is one way to do it, yes. If you still need details for how to handle it in your case, then keep the question open because I could not find a full duplicate, so perhaps someone will answer here.


text:
Defect Detection System using Deep Learning
What is the general approach to defect detection in deep learning?
Would the approach be better if we try to learn the positive images (defects in images) as much as possible or we try to learn the negative images (images without blemishes) and try to single out the defects as some anomalies
Can someone point me to some architecture?
REgards


text:
Purpose of using actor-critic algorithms under deterministic MDP dynamics?
One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from episode to episode). Therefore it is perfectly reasonable to use a critic to reduce variance and this is what for example Deep Deterministic Policy Gradient (DDPG) does.
Now, let's assume that we're given an MDP with completely deterministic dynamics. In that case, if we start from a specific state and follow a certain deterministic policy we will always obtain the exact same return (therefore we have zero bias and zero variance). If we follow a certain stochastic policy the return will vary depending on how much we explore, but under an almost-deterministic policy our variance will be quite small. In any case, there's no contribution to the variance from the deterministic MDP dynamics.
In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely deterministic (if historical prices are used as states) and there's no real sequentiality involved. Consequently, all return variance stems from the stochasticity of the policy itself. However, most researchers still use DDPG as a variance reduction mechanism.
What's the point of using DDPG for variance reduction when the underlying MDP used for training has deterministic dynamics? Why not simply use the Reinforce algorithm?

In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely deterministic (if historical prices are used as states) and there's no real sequentiality involved.

Whilst I cannot comment on the specific financial model, I think it unlikely that these researchers would apply RL without there being a sequence.
More likely in my opinion, the historic data feed forms a major part of the environment, but that there are still time steps and a state which depends on an agent's actions. For instance, in a trading simulation, provided the values of trades are below levels that would significantly alter the market itself, it may be a reasonable approximation to use the history of prices and other factual information that progress like a recording, plus have state include the agent's current portfolio of investment and working funds.

What's the point of using DDPG for variance reduction when the underlying MDP used for training has deterministic dynamics? Why not simply use the Reinforce algorithm?

Variance in returns occurs due to stochastic dynamics (if they are present) and the behaviour policy. You cannot use any RL control algorithm with a deterministic behaviour policy*. It would never gain any data that allowed it to assess alternative behaviour.
So in REINFORCE, which is on-policy (the behaviour policy and target policy are the same), and typically starts with near equiprobable action choices, there is high variance. It could be very high when measured over a long episode with many action choices. In basic REINFORCE, the variance is not controlled for, and training uses individual Monte Carlo style returns.
In DDPG, which is off-policy (the target policy is deterministic**, the behaviour policy is stochastic), there is still variance, but it is much reduced with the actor-critic mechanism, plus can be constrained by choice of noise function that relates the behaviour policy to the target policy. In addition, updates to policy and value functions can be made independently of episode ends, which can significantly speed up learning.
To determine what difference this makes for any experiment, you would need to compare the two algorithms on the same task. In practice DDPG will significantly out-perform REINFORCE on many tasks, including those with deterministic environments. However, there might be specific combinations where the simplicity of REINFORCE wins out, if only because there are less hyperparameters to tune.
On one point:

under an almost-deterministic policy our variance will be quite small

That's true, but how do you get to that stage of training with REINFORCE? It is by testing and working through more stochastic policies, which is what will take time. Your statement only applies to REINFORCE when the control problem is nearly complete, or if you take a short-cut and force the policy function into what you hope is a near-optimal policy. In which case you are engaging in a form of variance reduction - it may even work for some scenarios, but is likely not as general as applying an Actor-Critic algoirthm.

* Actually technically you can if the environment is stochastic in the right way so that you effectively explore all state/action combinations. But we are talking about deterministic environments here, and obviously the stochastic environment would introduce variance in returns.
** The target policy changes over time, and this introduces non-stationarity and bias for the critic component to deal with, but not technically variance.

"plus have state include the agent's current portfolio of investment and working funds."
Yes, this is a common way of (in my opinion artificially) introducing sequentiality (in a sense that now an action at time t will affect the state at time t+1). However, if we ignore the trading costs (which is commonly done in many papers), the optimal action at time t+1 will clearly NOT depend on the part of the state at time t+1 that represents current portfolio weights (as one should invest in stocks that will grow in the future no matter what the current portfolio weights are).

@BGa: I cannot really comment further on the stock trading scenario, I don't know it well enough. I hope the rest of my answer explains why DDPG would be preferred over REINFORCE for deterministic environments.

"That's true, but how do you get to that stage of training with REINFORCE?"
Why can't I simply start out with a stochastic Gaussian policy with a very small variance, and then adjust its mean while learning but keep the variance fixed.

@BGa: You can, and this is also a form of variance reduction (I just added a sentence to that effect). However, you may limit exploration too much by doing so, and the more complex algorithms have more general mechanisms to cope.

OK, clear now, thanks. But another thing just crossed my mind. Let's forget about REINFORCE for a moment. Why is DDPG preferred over simply having a NN (which represents a policy) output a deterministic action, calculating the loss function for this policy  (say negative Sharpe ratio or whatever) and simply using the gradient descent to find optimal parameters of the NN? This way we don't have to deal with either stochastic policies or the intricacies of actor-critic approaches.

@BGa: In brief because you have no ground truth of the correctness of the action. Actually many RL algorithms can be viewed as "wrappers" around NNs that generate training data as you suggest - very easy to see in something like DQN. Alternatively, you can think of the NN as the simple part of this, and the RL design as more critical. A really simple RL-ish algorithm that does what you suggest and is quite good in deterministic environments is Cross Entropy Method (CEM) which collects its data from "elite" episodes for training


text:
Keyword spotting with custom keywords and why not use speech recognition instead
My question regards performing keyword spotting for custom keywords and justifying the use of keyword spotting models instead of speech recognition.
I have been doing some searching around Keyword Spotting and I realized there is not so much work out there. Probably the most common dataset I have found people using is the Speech Commands Dataset. However, this dataset has only 30 keywords.
If I want keyword spotting for my own custom application, then to the best of my knowledge I need either a pre-trained model or my own data to train a model on. However, to the best of my knowledge, there is no model pre-trained on a dataset with a large enough set of keywords that is likely to cover a lot of applications. Correct me if I am wrong in this.
I have come to the conclusion that I need to train my own model and the only two ways I could train models on custom keywords is to get that data myself, either by crowdsourcing or by performing speech recognition on large datasets, picking up segments which include the words of interest and then doing some manual work to check if these segments truly include the keywords I want. Does someone think that this would be a good or bad idea and why?
Lastly, why would I even bother going the keyword detection route and not just use a speech recognition model that will recognize the words a human speaks and see if any of them match my keyword? Is the performance that much better with keyword detection?


text:
Two Models vs One Model for Person Detection and Object Detection
Is it possible to do person detection and object detection within one model? The training data would be images annotated with bounding boxes for objects and people.  Because normally object detection and person detection are done separately? Is there any research about models that simultaneously detect both people and objects?

By person detection you mean outputting a bounding box for people or face recognition? Thanks @PrimeNumber

@ClementHui: bounding box for people.

object detection also can detect persons. People can be a class of object to classify.

Hi @PrimeNumber I have added an answer, hope that it can help you.

Yes you can classify people bounding box with object detection. State of the art object detection model have people as one of the class in the object detection, as shown here:

As you can see the image have both object bounding box and people bounding box.


text:
How to measure the size of an crack which is segmented from an image using Mask-RCNN?
I am a masters student going to work in a project to analyze the cracks in underwater concrete structures.
I need some suggestions for data acquisition and length measurement of the crack.
I have decided to do crack segmentation using Mask-RCNN. But I don't know which methodology is best to measure the length of the cracks. While searching about this, I found many ways to measure the crack size when there is another reference object of known size in the image. But in my case, there won't be any reference object and also it is not possible to know the distance between the camera and target since it is underwater.
If the images are of stereotype, Will that solve this issue?
Can anyone help?

With "stereotype" you probably mean stereoscopic, and yes, given a known and stable camera arrangement, it should be possible to calculate distance and therefore absolute size.


text:
What is the reason AMD Radeon is not widely used for machine learning and deep learning?
What is the reason AMD Radeon is not widely used for machine learning and deep learning? Is it mainly an issue of lack of software? Or is Radeon's GPU not as good as NVIDIA's?

The main reason that AMD Radeon graphics card is not used for deep learning is not the hardware and raw speed. Instead it is because the software and drivers for deep learning on Radeon GPU is not actively developed. NVIDIA have good drivers and software stack for deep learning such as CUDA, CUDNN and more. Many deep learning library also have CUDA support. However for AMD there is little support on software of GPU. There is ROCM but it is not well optimized and also a lot of deep learning libraries don't have ROCM support.
Also on the hardware side, AMD lacks deep learning specific features like tensor cores. AMD also don't have data centre specific cards like the Tesla lineup. The performing of AMD cards also is not very good until recently with the RDNA architecture.
Lastly, data centres don't change hardware very often. NVIDIA flourished in the deep learning field very early on so many companies bought a lot of Tesla GPU. Even if AMD caught up in the deep learning field it is very hard as many companies have used NVIDIA from a long time ago, and switching to a very different architecture of GPU is troublesome, especially for a data centre with couple hundred or more servers.
AMD has also "given up" on deep learning market. They don't actively tries to make effort in deep learning data centres using AMD hardware. Many data centre also use very old Tesla hardwares like Tesla K80 of M series.
Hope this can help you.