text:
I need help understanding general back propagation algorithm
In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as:

The back-propagation algorithm is very simple. To compute the gradient of some
scalar z with respect to one of its ancestors x in the graph, we begin by observing
that the gradient with respect to z is given by dz = 1. We can then compute dz
the gradient with respect to each parent of z in the graph by multiplying the current gradient by the Jacobian of the operation that produced z. We continue multiplying by Jacobians traveling backwards through the graph in this way until we reach x. For any node that may be reached by going backwards from z through two or more paths, we simply sum the gradients arriving from different paths at that node.

To be specific I don't get this part:

We can then compute dz the gradient with respect to each parent of z in the graph by multiplying the current gradient by the Jacobian of the operation that produced z.

Can anyone help me understand this with some illustration? Thank you.

Hey. 3 blue 1 brown made amazing series on neural networks, https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi, the 3rd and 4th part have good illustrations of back propagation algorithm

Please, ask a more specific question. The current question is too broad.


text:
Interpretability of feature weights from Gaussian process classifier
Suppose I trained a Gaussian process classifier with a linear kernel (using GPML toolbox) and got some feature weights for each input feature.
My question is then:
Does it/When does it make sense to interpret the weights to indicate the real-life importance of each feature or interpret at group level the average over the weights of a group of features?

Hi Alex and welcome to this community! I suggest you ask only one question. I already doubt that there's anyone here that has a better knowledge of Gaussian processes than me, so, if you want to attract the attention of people to answer your question (by facilitating the reader's life), then simplify your post and ask just one question. Ask the other questions in separate posts (each in its own post).

Thank you for the advice, @nbro ! :) I have edited the question accordingly. Also if you have any insights I would be very grateful to be enlightened! ;)

Cross-posted: https://ai.stackexchange.com/q/16865/1794, https://stats.stackexchange.com/q/438683/2921, https://cs.stackexchange.com/q/118114/755.  Please do not post the same question on multiple sites.

How can you get the weights? The weights are given as an input, aren't they? I assume by "weight" you mean this part of input which is used to calculate weight of each input in the likelihood? Or what exactly you mean by "weight"?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Should my agent be taking varying number of steps?
My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the $\epsilon$-greedy stage, it still kills/crashes itself in subsequent episodes.
I would like to know if this common in RL (D3QN) scenarios.
A graph showing episodes vs steps has been placed below.

As one can see, the agent reaches 400 steps in episode 1000. But, in the subsequent episode, it falls down below 50 steps.

Does your graph show steps during training?

Right, although the epsilon is gradually decreasing and therefore it is also exploiting instead of just exploring.

For the graph, can you say what epsilon starts at for epsilon starts at for episode 1, and ends at for episode ~2200?

What agent are you using? Which environment (and maybe library)? Which hyper-parameters? I think you should provide also these details, because they may be useful.

@NeilSlater At episode 1, the epsilon should be 1. I do not know when it would have reached 0, although it reached there before episode 2200. Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?

@nbro The agent is a self driving vehicle. Regarding hyperparameters, I am not sure how describing them would help.
Honestly, this is my first independent RL project, therefore my questions might be a little silly :)
Would adding the episode vs reward help?

@desert_ranger Which RL model/algorithm are you using to guide the self-driving car? Well, in these cases, it may be useful to provide as much info as possible about your configuration, although I don't really know if it will turn out to be useful.

@nbro It is a slightly modified D3QN

Thanks for accepting the answer. I have extended the answer a little with another key point that you may want to consider.

Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning.
Looking at it in detail I can say:

There is clearly some learning occurring.

There is a strong random element throughout. As you say the epsilon is reduced to near zero by episode 2000, and I would assume a driving simulation is mostly deterministic but with a lot of state hidden from the agent, this mainly implies that episodes are started in different states.

There may be an effect from continued learning that causes the agent performance to vary so much. However, it is more likely towards the end that the agent is encountering new never-seen-before states and making poor decisions in them.

The resulting graph matches to my experience of training DQN agents where some of the hyperparameters are off. Probably you need to do some kind of exploration or search through those parameters.

From your comments:

Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?

These are possibilities that I cannot rule out by looking at the graph.
However, I think it is more likely that you have some agent design or hyperparameters that are not a good fit to the problem. Those hyperparameters could be almost anything - not enough episodes to cover variation, neural network too simple, neural network too complicated, epsilon decay too fast, poor choice of regularisation, experience replay memory too small, poor choice of optimiser, etc etc.
It is also possible you have limited the state model to the point that learning is hard or even impossible. A common beginner's mistake here is to use static sensor information such as a single current screenshot on each timestep, so the agent has no way to assess how fast it is going, or which way it is already turning etc (if your input includes direct knowledge of current speed, turning etc then this is probably not a problem for you, I am just raising one of many possibilities).
One thing that may help you understand the agent's learning performance better is instead of plotting the learning episode graph, plot a test graph, perhaps once every ~100 episodes, where you run some number of episodes without learning and with $\epsilon = 0$, and take average of the metrics you are interested in. This will remove some of the randomness from the values you are plotting and give you a much better sense of training progress than your current plot.
With Q learning, you may also get significant benefit by setting a non-zero minimum $\epsilon$, e.g. $\epsilon = \text{max}(\text{eps_decay}(n), 0.01)$ where $\text{eps_decay}(n)$ is your current epsilon decay function for step $n$. That is because the off-policy updates with exploration are still helpful at all stages of learning.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is it always a good idea to use deterministic policies during testing?
I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this case, as per game theory, a stochastic (random) policy is required (as per my understanding)
Edit - Let me clarify my understanding of stochastic and deterministic policies. A deterministic policy will always select an action given some state. The stochastic policy will sample the action given a state. Therefore in the latter case, an optimal policy for Rock, Paper and Scissor will always choose an action with probability of 0.33.

It's worth noting deterministic policy doesn't necessarily mean it always select a unique action given some state, it's possible in finite MDP at some state several actions equally solves the Bellman optimality equation, thus the greedy deterministic optimal policy can randomly follow any such action.

Is it always a good idea to use deterministic policies during testing?

No, it's not, and you correctly point out that in certain scenarios the optimal policy is stochastic.
Specifically, the optimal policy in Rock-Paper-Scissors is a stochastic policy. However, this game can't really be represented as a stationary MDP, but as a Markov Game with 2 players. If you wanted to represent it as an MDP (Markov Game with 1 player), then it might not be stationary, i.e. the dynamics could change.
Having said that, there's a known result in the theory of MDPs that says that there's at least one optimal policy that is stationary and deterministic, but there are other environments and models other than finite MDPs. You also have POMDPs, Markov Games, etc., which are generalisations of finite MDPs to different scenarios (partial observability of the state and multiple players, respectively).
So, when should you use deterministic policies? Well, most likely, when you are sure that your environment is a finite MDP. It's possible that, when they use deterministic policies, people assume the environment can be represented as an MDP, while, in reality, maybe it's not. Then, of course, in practice, you lose all theoretical guarantees.


text:
How does the hidden activation differ from the output, at any time step for a SimpleRNN?
I am watching the Sequence models course taught by Andrew Ng. I am a bit lost on the SimpleRNN lecture. As per the lecture, at each time step, there's an output y from a hidden layer and an input activation from the preceding layer. However, I do not understand what's the difference between the activation and output $y$ at any time step (except the first).
As per my understanding -
$$y = activation(w*x+b)$$
Therefore $y_t$ and $activation_t$ should be the same. I have included a figure to describe my issue.

In the picture, what's the difference between $\hat{a}^{\langle t \rangle}$ and $\hat{y}^{\langle t \rangle}$?
Here's the link of the lesson - https://www.coursera.org/learn/nlp-sequence-models/lecture/ftkzt/recurrent-neural-network-model

They are not the same always, the activation (as you called it) here is the hidden state (I think it's a in your slide), now you can have the output is the same as the hidden state (for example that may represent a word embedding) but for most problems you need to do more operations on the hidden state to calculate the output.
We do that because what you need as output in the current step may not be the whole hidden state (which may contatin info that required in future steps), also you may need to project the hidden state, for example if you want to predict a word you need to project the hidden state into a vector in $\mathbb{R}^{|V|}$ where $V$ is your vocabulary, and then apply a softmax over it (like in the equations above).

So, if I understand correctly, the weight wya(in the image) or V(in eq 10.10) can be considered analogous to inserting a dense layer after the RNN layer.

@desert_ranger Yes.


text:
How does CURL extract labels from logits?
While going over the pseudocode of the CURL paper, the method to identify labels from the logits wasn't clear to me.  I believe this technique might be common in other PyTorch/Deep Learning tasks. I have attached the pseudocode below -

Given arange returns a 1-D tensor with values from 0 to logits.shape[0], then the labels is a vector of $0$ to $N$ where $N$ is the number of classes predicted by the output layer of f_q.
The CrossEntropyLoss then finds the difference between the predictions and the target labels, which the encoder weights f_q.params is updated according to. I haven't read the paper, but this particular part is a standard multi-class classification approach.


text:
How is GPT 4 able to solve math?
How can GPT 4 solve complex calculus and other math problems. I believe these problems require analytical reasoning and ability to compute numbers. Does it still use a LLM to complete this process or does it add on to this?
Here is the link to the official results published by OpenAI

But you cannot solve mathematics by predicting the next word technique. If that were the case, all English majors could be mathematicians.

@desert_ranger Your argument is basically "Reductio ad absurdum".

And more importantly, there are not many details of how GPT-4 works or was trained, so this is not objectively answerable.

@user253751 An example would be GPT-4's score in GRE Math and AP Math exams.

@user253751 My apologies. I have updated my question with the official link.

@Dr.Snoopy I am not familiar with these argument terms. Is there a flaw with this type of argument?

@Dr.Snoopy I wasn't aware that the training details weren't released. Despite that, generative models are usually built using some core algorithm - Autoregressive, Flows, VAE, etc. Therefore, I was hoping that maybe there's a explanation about GPT-4 capabilities and incapabilities.

This is basically the only technical information available: https://arxiv.org/abs/2303.08774

"But you cannot solve mathematics by predicting the next word technique." I don't think this is true. If mathematics can be laid out as text in a textbook, it can be predicted using next word prediction.

They can't do math. They answers to mathematical questions are based on what is most likely given the training data. Yes, there are plugins, and you can query Wolfram Alpha, but the models per se are not doing math.

Large Language Models actually can do math. It's an "emergent" property, i.e. it appears only at larger scales. Understanding complex English language does require some analytical ability, which can carry over to math tasks like calculus and even arithmetic. Numbers can be represented as words, so it's definitely not unthinkable that an LLM could learn to add and subtract if it were to see enough examples.
The graph below from the 2022 paper "Emergent Abilities of Large Language Models" shows that these properties "spontaneously" emerge as models get larger. We're interested in subgraph (A) here. Upto 10^22 FLOPS, the models studied (the largest models available at the time) have basically no arithmetic ability, but scaling the models further rapidly improves their capabilities. We don't know the internals of GPT-4, but it should be larger than these models, so it was expected that it would better at arithmetic.
It also goes the other way around, Numeracy enhances the Literacy of Language Models

Language models model the probability P(w_k | w_{k - i}, ..., w_{k - 1} for words w_k, that is the probability of a word w_k appearing in context w_{k - i}, ..., w_{k - 1}. Furthermore, models have been trained to do math computations directly. If the training dataset contains enough examples of math tasks, the model may be able to reproduce these examples, and with enough parameters and enough samples even find some rules. However, it's not very reliable (for now), it's always based on statistics in the end.

Do we have any reason to believe that they're still using a basic LLM?  A mixed architecture seems like such low-hanging fruit.

Saying that they can do math makes no sense when they can't even count. Doing math is not just choosing an answer based on what is most likely. You need to do it with probability 1 or 0. There's nothing in between. Everything in between is not math, and that's what all these models do. So, these models do everything, except math.

Damn . . . . . .

ChatGPT now uses Wolfram Alpha to deal with math as well as other factual information.
https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/

Wolfram is offered as a plugin and not used by default to solve mathematical equations with ChatGPT.

As far as we know, GPT-4's core capabilities are still based mainly on a Large Language Model (LLM).
If so, then the apparent capabilities to reason are a somewhat surprising emergent phenomenom from a well-trained sequence prediction engine that has been trained on large amounts of data, and has capacity to create highly complex rules that approximate a "next symbol oracle".
Again, assuming this assertion is correct, then maths and logic capabilities of ChatGPT divide into a few different possibilities (these are not formal classifications, just my amateur analysis):

Rote learning of symbol associations. This is likely to occur with commonly-occurring items in the training data. Special values of trigonometry functions for example.
Things that look like logical reasoning, but are simply well-formed sentences that are on-topic. This is something we can easily be fooled by. When ChatGPT gives an explanation for a thing, it may not have any representation of it beyond being in an "explainy" state, and generating text that fits.
Approximate rules and processes. The LLM is a complex neural network, and can in principle learn arbitrary internal functions in order to predict sequence continuation. It learns these statistically, and there will be limitations - for example it is unlikely that it could learn to produce crytpographic hashes given only examples. But it may really learn to add two numbers across a wide range of numerical values given thousands of examples.
Logic processes embedded in the form of output text. I have seen many examples where a LLM gets a correct answer when it is allowed to "think things through" by showing the working out, whilst forcing a direct answer will be wrong.
Accurate rules and processes. Some rules in maths are very language-like and could be learned very well by an LLM. That could include some mathematical symbol manipulations such as variable substitution.

I expect that all the above are occurring in some mix.
For example, you could conjecture that there is a reasonable chance that GPT can internally count accurately up to some moderate number, and re-use that ability in different contexts to predict numerical symbols and associated words (e.g. one has also representation 1) It may also contain more than one such semi-accurate counter, using them in different contexts.
The sheer quantity of training material - more than any single person could consume in their lifetime - plus learning capacity of the neural network means that there are probably a lot of simple rote rules that are subject-dependent. However, in some areas the model will have a "deeper understanding" in that it will have learned reasonably complex manipulations, and used them to predict a sequence of symbols as accurately as possible, using as little of its learning capacity as possible (because it is being asked to predict text in a huge range of contexts, so benefits when it compresses its rules)
GPT has not learned primarily by reasoning and from first principles though. Its inner representations and logical units of work are likely to be quite alien to humans, and may freely combine grammar, sentiment, mathematical building blocks and other symbollic context in ways that could seem very odd if they could even be explained. This heady mix of things that occurs in most neural networks during training, is one reason why it is unlikely that OpenAI have wired in separate logic modules for highly structured processing such as math symbols or calculations. Providing such modules is possible, but detecting when to use them, and how to wire them into the network are both hard problems.

OpenAI's CEO explicitly mentioned last month in the GPT-4 announcement video that GPT-4 isn't hooked up to a calculator.
One can however install plugins on top of ChatGPT, which may connect it to some other resources such as Wolfram as mentioned in Jaume Oliver Lafont's answer.

There is a folk story about J.W. Gibbs that goes something like:

Being a famous scientist, Gibbs was a member of a number of scientific bodies. He was bored by those and never took a podium. Except for one time. The discussion was about redirecting some effort from teaching mathematics towards more effort at teaching foreign languages. Gibbs decided to give a speech that one time. He said: "Mathematics is a language."

I don't know is this story is true or not, but I share the attitude.

Finally an answer which makes sense. At least GPT's understood it.. Don't know when humans will.


text:
Which AI conference presentation on predicting terrorist movement inside buildings caused protest in the audience and media?
I remember reading an online news article a while ago on AI ethics that described a conference presentation on the subject of a military AI system that predicted terrorist movement inside buildings and used drones to shoot them when they exited the building. The presentation caused protests and sharp criticism in the audience due to the dehumanizing nature of the AI usage presented.
The article was in mainstream media, but I am unable to find it with Google search.
Which AI conference presentation was it and are any articles on it available?


text:
Retraining ResNet-50 for iris flower classification
I am trying to retrain ResNet-50 for iris flower classification in tensorflow (TensorFlow version: 2.3.0) using the following code
import tensorflow as tf
import cv2, random
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from random import shuffle
from IPython.display import SVG
import numpy as np # linear algebra
import pandas as pd
import shutil
import matplotlib.pyplot as plt
%matplotlib inline
from IPython.display import Image, display
from sklearn.model_selection import train_test_split
import os
print(os.listdir("./iris recognition/flowers"))

labels = os.listdir("./iris recognition/flowers")
num_classes = len(set(labels))
IMAGE_SIZE= 224

# Create model
model = tf.keras.Sequential()
model.add(tf.keras.applications.ResNet50(include_top=False, weights='imagenet'))
model.add(tf.keras.layers.GlobalAveragePooling2D())
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))

# Do not train first layer (ResNet) as it is already pre-trained
model.layers[0].trainable = False

# Compile model
from tensorflow.python.keras import optimizers

sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()
train_folder = './iris recognition/flowers'

image_size = 224
data_generator =  tf.keras.preprocessing.image.ImageDataGenerator(preprocessing_function=tf.keras.applications.resnet50.preprocess_input,
                                    horizontal_flip=True,
                                    width_shift_range=0.2,
                                    height_shift_range=0.2,
                                    validation_split=0.2)# set validation split

train_generator = data_generator.flow_from_directory(
    train_folder,
    target_size=(image_size, image_size),
    batch_size=10,
    class_mode='categorical',
    subset='training'
    )
validation_generator = data_generator.flow_from_directory(
    train_folder,
    target_size=(image_size, image_size),
    batch_size=10,
    class_mode='categorical',
    subset='validation'
    )

NUM_EPOCHS = 70
EARLY_STOP_PATIENCE = 5
from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint

cb_early_stopper = EarlyStopping(monitor = 'val_loss', patience = EARLY_STOP_PATIENCE)
cb_checkpointer = ModelCheckpoint(filepath = './working/best.hdf5',
                                  monitor = 'val_loss',
                                  save_best_only = True,
                                  mode = 'auto')
import math

fit_history = model.fit(
    train_generator,
    steps_per_epoch=10,
    validation_data=validation_generator,
#     validation_steps=10,
    epochs=NUM_EPOCHS,
    callbacks=[cb_checkpointer, cb_early_stopper])
model.load_weights("./working/best.hdf5")

I tried to change the number of epochs , steps_per_epoch and validation_steps however the model accuracy does not improved as shown in
After trying several essais while changing batch size from 32, 64 and 128.For each essai I tried with and without using :

steps_per_epoch=10.

sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9,
nesterov = True)

The best accuracy I achieved is 72.29% in which the training was the most stable which was when using batch size=128 as shwon in
Any suggestions please?!

Have you tried decreasing the learning rate or increasing the batch size? Your model is fluctuating widely and the lr. or the batch size might be reasons for that.

I only tried to change the batch size form 24 to 10.

You could try batch sizes of 32 or 64 (or even 128), this might stabilize the training already.

Actually, I tried as you said changing with these parameters as follow:  I tried changing the batch size from 32 ,64 and 128 with sgd (with and without using steps per epoch) and without sgd(with and without using steps per epoch) and the best accuracy I achieved is 72.29%

Has training become more steady? Have you tried a smaller learning rate as well?

@Mariusmarten, I have edited the question with the last updates. Considering the learning rate which is in my code sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True) I just comment this line and run my code.

It looks like you're defining an optimizer in the line sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True) and then not using it, since you compile the model with optimizer='adam'. So changing any of the sgd parameters, or even commenting out the line, changes nothing for your model.
you'll need to change to model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy']) to use the sgd optimizer object in the model.

Oh,  I didn't pay attention to that. Ok I will try it.


text:
How to use PCA to reduce the dimension of output features of convolutional layers of CNN
I am working on a hybrid CNN-SVM classification using python. I tried to get the final output features of the CNN model(flatten layer) to be fed to the SVM classifier. The output shape obtained from the flatten layer was (600,32786).For computational reasons, I wanted to reduce the shape of the obtained features to (600,250). I was looking for a way to do that without losing relevant features, and I read about the PCA technique.
I saved the output features in a(.npy) file. I tried the following:
X= np.load('features.npy')
pca_250 = PCA(n_components=250)
pca_250.fit(test_images)
Y = pca_250.transform(test_images)

Is this correct?
Regards

You’re basically guaranteed to lose something when you drop into a lower dimension, whether you use PCA or just lower the number of neurons in the pre-SVM layer.

This code here, assuming X is your (600, 32786) matrix, reduces your feature space to (600, 250). However, you have to keep in mind that PCA does not remove redundant or less important features so that your are left with the 250 of your most important original features. This is merely a projection of your data onto 250 principal axes. If that's what you wanted to do then yes your code is correct save for the fact that I am not sure why you have performed PCA on test_images rather than X.

@Minuano, I was mistaken I should use X instead as you said. Actually I wanted to reduce  the dimension of the feature space while keeping the most relevant features. Is there any solution to do that?

There are several ways you can go about doing that. Most commonly used methods, as far as I know, involve calculating the correlation of your variables with each other ( a (32786, 32786) correlation matrix) and keeping only the top 250 most uncorrelated features. You can also calculate the correlation between your features and your target variable and try to remove uncorrelated features. For further insight I encourage you to read this article.


text:
How does Lateral Inhibition Provide Competition among Neurons?
I stumbled upon a paper from P.Diehl and M.Cook with the title "Unsupervised learning of digit recognition using spike-timing-dependent plasticity" and I'm trying to understand the logic behind the network connection they made.
The network is as follows. The inputs (size of an image 28x28) are connected to the usual all-to-all fashion with positive weights to an NxN layer of neurons. They are decoded using poisson random distribution in which the frequency of spikes of a pixel is set accordingly to the pixel value. The NxN layer is connected 1 on 1 to an NxN layer of inhibitory neurons. These neurons inhibit all other neurons except the one they are connected to. Thus they are connected all-to-all with an exception.
According to the paper this provides competition among neurons. I cannot understand how, in this particular connection, competition is provided. How can different neurons inherit different properties? To me it seems that all neurons will inherit the same properties, thus no differences in weights will be made in the training session among all neurons. For example, if the input 5 is passed to the network all weights of all neurons will try to adjust according to 5. Then if input 7 is passed next, all the weights will be updated according to the new number (7). It is expected, though, that some weights will keep the previous adjustment ie that some weights will have the properties of 5 and the others the properties of 7.

The architecture described in the paper is organized into three
layers. The first layer contains 784 input neurons, the second layer
$N$ excitatory neurons, and the third layer $N$ inhibitory neurons. In
their S-sized network $N=200$.
The weight all synapses from an input neuron to its excitatory
neurons decrease when it spikes. The weight of all synapses to an
excitatory neuron from its input neurons increase when it
spikes. Thus, spiking makes it easier for excitatory neurons to spike
again when receiving stimuli on the same synapses.
Meanwhile, the weight between excitatory and inhibitory neurons is so
strong so that when an excitatory neuron spikes its inhibitory
counterpart also spikes, depressing the other neurons and making it
harder for them to spike. The winner neuron(s) slaps the loser neurons
in the face.
What I think you are missing is that it is all dependent on
chance. Given some stimuli, one neuron will spike first and due to
literal inhibition its advantage for that particular stimuli will grow
larger and larger. That neuron will "win" that stimuli, while allowing
the remaining neurons to compete for the remaining stimuli. To prevent
one neuron from winning everything weights have to be normalized
between samples.


text:
structure of neural network for classification problems with large amounts of null classifications
I am building a Convolution neural network to predict certain categories based on images (the location of a pointer on a surface) . However in many cases there will be no pointer in the view or something that is not the pointer. Initially I was just going to train it with outputs of the different classifications including the null classification. However given that the null classification is far more common than the others (perhaps 1000 times more likely) would it be better to have a separate null classifier and then if this outputted non null then the second classifier would be used.
Any suggestions?

Hi, can you share some more details on the area of application? What is it that you are trying to locate in the images? And what exactly does it mean when "nothing" is in view?

I've updated the question to reflect the problem is about locating a pointer on a surface in an image

You'll be fine so long as you use softmax and cross entropy loss. Take Alexnet. It has 1000 classes to classify, and since it only ever outputs 1, 999 times out of 1000, the desired output is 0. Despite this, it still operates fantastically due to its final activation and loss function.

I think there is a significant difference between an example where there are many classes each reasonably unlikely to an example where one state has around a 99.9% chance of occurring. In my example my main worry is making sure there are very few errors in this class.

I see multiple reasons to take a different route:

While in "classical" pattern recognition you might have done things like feature engineering outside of your model, one idea of deep learning was to "insource" it into the model and let the model take care of "everything". Following that, I have seen a general tendency in deep net architectures to let your deep net handle all the work in one single model. So your idea kind of contradicts the architectural mainstream.
There are so many deep net architectures being published which are well engineered and tested for all kinds of tasks. Therefore, I would check if there is anything "ready off the shelf" for a task like yours in the very first place. If there is, then it will probably be better than your self-engineered model architecture. And if there is not, you might want to check related tasks and see how they solved this problem, e.g. medical applications where specific cells need to be detected with most examples being negative (and if there is a positive example the area in a picture might still be mostly negative as it is just a single positive cell among many others).
Your thought of feeding the "present/not present" binary classification to your task of locating an object is very closely related to what has been discussed as auxiliary tasks. I don't know if it has been applied to exactly your problem but there are similar applications. For example there is an application where "name recognition" is complemented by the auxiliary task of deciding whether any name is present in a given sentence or not. Which is somewhat similar to your case. The paper An Overview of Multi-Task Learning in Deep Neural Networks provides an overview. A famous example using auxiliary tasks would be GoogleNet. This also goes back to my first point of letting the deep net handle "everything" internally.

However, as a disclaimer, this is a theoretical perspective as I cannot speak from experience regarding your problem.


text:
Autoencoder make spectrogram important parts more pronounced with a "log loss"
Hi I want to create a neural network that essentially picks out the most pronounced parts of a spectrogram.
Assume this is the True spectrogram:
[0.0, 0.25, 0.95, 0.1, 0.0, ...]
I want the auto encoder output to tend towards outputs like this:
[0.0, 0.05, 1.0, 0.0, 0.0, ...]
I essentially want to punish the neural network more for getting larger values more incorrect.
True 0.2, Predicted 0.1
True 0.9, Predicted 0.8 <---- Far greater loss
Since I don't care about background sounds this will make the important parts of the spectrogram more pronounced.

Will this work? What should I use?

From what you have posted, this looks like a task more easily solved with a simple filter.  Is there a reason you are preferring a NN?

Because within the spectrogram there also random background noises that I add which will be removed @DavidHoelzer

Why not do a simple SVD to remove the noise?


text:
Can learned feature vectors be considered a good encryption?
Considering I have some neural network that, using supervised learning, transforms a string into a learned feature vector where "close" strings will result into more close vectors.
I know that since a NN is no one-way-function there is a way to retrieve the input data from my output if I have the entire network at hand (if I know biases, weights, etc.)
My question is if the network is not known, is there a way for me (using ie. some probabilistic distributions) to make assumptions or even reconstruct the input data?

even if you have weights and biases, unless the network falls into some reversable scheme (most used in practice do not) then you still wont be able to reconstruct the input

In principle, if an attacker can query the network as often as they want, they can simply build a dataset of encodings and sources and learn a decoder, that should get pretty good. I also remember that there was a paper a few years ago, that could even reconstruct a model by querying it enough. They noted, that this could for example be used to reconstruct recommendation engines from amazon.
However, if the network is trained in a specific way (like concatenating a random key to the input), it might be possible.

Beside that, there exists a wikipedia article concerned with this question: https://en.wikipedia.org/wiki/Neural_cryptography


text:
Finding the right model
Let us say that i have two ball throwing machines which has some algorithm running in the back-end for releasing the balls. One machine shows it throws 5 balls in 1 sec. Other shows the exact distribution of how many balls were thrown in each 0.2 secs (say the distribution is: 2,1,0,1,1) but the sum is 5 balls/sec for this machine too. Can i use this data and some other independent parameters like speed, direction etc as inputs and predict the similar distribution for lower accuracy machine.?
Re-framing my question:
I am searching for an apt supervised model for the following use case:
If I have a  sum (say 10) and it can be distributed in a predefined number of bins (say 5) in a number of ways for instance:
1. 1, 2, 5, 0, 2
2. 0, 0, 3, 7, 0
etc.
The distribution bins always have whole numbers and the sum is also a whole number. The distribution depends on a number of factors and patterns which can be learned in the volumetric data. Hence, if I am able to load more than one sum (say n sums) and output the corresponding n*5 distributions it will be better for precise prediction (as per my intuitions). I tried using some networks but they are not doing much good.


text:
DQN card game - how to represent the actions?
I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players (if they have this card in hand). The one who has the card with the highest number gets the played cards. Each collected red card gives you a -1 point.
With respect to this answer I setup the inputs / state of the NN as binary (meaning I have 180 bool values (60 card x is currently on the table, 60 card x is in the ai hand, 60 card x is already been played)
How to design the outputs / actions?

If the ai is the first player of a round it can play any card
If the ai is not first player it has to respect the first played card (or play a wizzard)

This means there is actually a list of available options. I then sort this list and have 60 Output bools which I set to 1 if this option is possible. Among these options the ai should then decide what the correct option is? Is this the correct procedure?
Inconsistent / Varying Action Space
This is what we have to deal here with. As explained in here I think a DQN as well as Policy Gradient Methods is not the correct architecture to choose for solving such multi-agent card games. What architecture would you choose?
General procedure?
Assume I have 4 players, so do I have to get the old state before the ai is the next player and the new state is directly after this round is finished?
my_game = game(["Tim", "Bob", "Lena", "Anja"])
while True:
    #1. Play unti AI has to move:
    my_game.play_round_until_ai()

    #2. Get old state:
    state_old = agent.get_state(my_game)

    #3. Get the action the AI should perform
    action_ = agent.get_action(state_old, my_game)

    #4. perform new Action and get new state
    #reward rates how good the current action was
    #score is the actual score of this game!
    reward, done, score = my_game.finishRound(action_)

    # 5: Calculate new state
    state_new = agent.get_state(my_game)

    #6. train short memory base on the new action and state
    agent.train_short_memory(state_old, action_, reward, state_new, done)

    #7. store the new data into a long term memory
    agent.remember(state_old, action_, reward, state_new, done)

    if done == True:
        # One game is over, train on the memory and plot the result.
        sc = my_game.reset()

My code so far is available here: https://github.com/CesMak/witches_ai

Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action.
Why is this better?
You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.

Could you point me to some examples?

sure, I found this answer on this website : https://ai.stackexchange.com/questions/9491/inconsistent-action-space-in-reinforcement-learning

I just read this article but I cannot find what you are suggesting. What I could directly re-implement is choice (1): Such that the network learns possible actions but this is as said not the best option. Do you know some code example / what would be the name of this algorithm you suggested  (I think it is not DQN anymore). Sry I am completely new to in this topic.

Try looking up on state value functions vs state action value functions. I remember learning this from the book 'Reinforcement learning : an introduction'. from A quick internet search maybe this will help you understand: https://www.cs.dartmouth.edu/~lorenzo/teaching/cs134/Archive/Spring2009/final/PengTao/final_report.pdf


text:
Would it be possible to implement the principals of the K means clustering algorithm in a Neural Network
During a Machine Learning course which I have done I have learnt about the K means algorithm. Is it possible to use the principals of K means within a neural network?

Do you know what a neural network is? Or are you asking if a neural network can be used to cluster?

I do know what neural networks are, what I am asking is if you are able to use the principals of K means (the choosing of the cluster centroids and the updating of the centroids position) and use those principals in order to find the optimal centroid position via a neural network.

Yes it is possible, here is a paper that describes the process: https://arxiv.org/pdf/1808.07292.pdf

@razvanc92 Feel free to provide a formal answer below. Maybe you could just summarize their abstract or so if you are not familiar with the paper.


text:
Can a deep neural network be trained to classify an integer N1 as being divisible by another integer N2?
So I’ve been working on my own little dynamic architecture for a deep neural network (any number of hidden layers with any number of nodes in every layer) and got it solving the XOR problem efficiently. I moved on to trying to see if I could train my network on how to classify a number as being divisible by another number or not while experimenting with different network structures and have noticed some odd things. I know this is a weird thing to try and train a neural network to do but I just thought it might be easy because I can simply generate the training data set and test data set programmatically.
From what I’ve tested, it seems that my network is only really good at identifying whether or not a number is divisible by a number who is a power of 2. If you test divisibility by a power of two, it converges on a very good solution very quickly. And it generalizes well on numbers outside of the training set - which I guess it kind of makes sense, as I’m inputting the numbers into the network in binary representation, so all the network has to learn is that a number n is only divisible by 2^m if the last m digits in the binary input vector are 0 (i.e. fire the output neuron if the last m neurons on the input layer don't fire, else don't). When checking divisibility by non-powers of two, however, there does not seem to be as much of a "positional" (maybe that's the word, maybe not) relationship between the input bits and whether or not the number is divisible.  I thought though, that if I threw more neurons and layers at the problem that it might be able to solve classifying divisibility by other numbers – but that is not the case. The network seems to converge on not-so-optimal local minima on the cost function (for which I am using mean-squared-error) when dividing by numbers that are not powers of 2. I’ve tried different learning rates as well to no avail.
Do you have any idea what would cause something like this or how to go about trying to fix it? Or are plain deep neural networks maybe just not good at solving these types of problems?
Note: I should also add that I've tried using different activation functions for different layers (like having leaky-relu activation for your first hidden layer, then sigmoid activation for your output layer, etc.) which has also not seem to have made a difference
Here is my code if you feel so inclined as to look at it: https://github.com/bigstronkcodeman/Deep-Neural-Network/blob/master/Neural.py
(beware: it was all written from scratch by me in the quest to learn so some parts (namely the back-propagation) are not very pretty - I am really new to this whole neural network thing)

When you represent a number in base 2 (binary), you have already divided the number by 2 many times.  If there is no remainder at the end, the number is obviously evenly divisible by 2.  This hints that your AI could test for divisibility by dividing.  Hmm-- nothing gained there!

My two cents on this problem: I believe a step that makes factorization difficult is the usual mod b representation. For instance, if we say that 9 is 121 (in "binary", 1x2^2+2x2^1+1x2^0) instead of the usual 1001 it may be easier to find that its factors are 11 and 11.

There is a recent development in research that was looking into effectiveness of neural networks on arithmetic. Interestingly, feed-forward neural networks (MLPs) with various activation functions as well as LSTMs (RNNs which are Turing-complete) are not able to model simple arithmetic operations (e.g. addition/multiplication), they designed a new logic unit which can solve all of the simple arithmetic problems.
See: Neural Arithmetic Logic Units
More recently, DL can solve symbolic maths: Deep Learning for Symbolic Mathematics

When you represent a number in base 2 (binary), you have already divided the number by 2 many times.  If there is no remainder at the end, the number is obviously evenly divisible by 2.  This hints that your AI could test for divisibility by dividing.  Hmm-- not much gained there!
Unfortunately the problem is not one suited to solving via AI.  That's why factorization of large numbers is a good basis for hard encryption schemes.  I'd suggest finding a different sort of problem on which to test your AI.


text:
What's the best architecture for time series prediction with a long dataset?
I have to build a neural network without any architecture limitations which have to predict the next value of a time series.
The dataset is composed of 400.000 values, which are given in hex format. For example
0xbfb22b14
0xbfb22b10
0xbfb22b0c
0xbfb22b18
0xbfb22b14

I think LSTM is suitable for this problem, but I am worried about the length of the input. Would it be a good idea to use CNN?
def structure(step,n_features):
    # define model
    model = Sequential()
    model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(step, n_features)))
    model.add(Dense(1))
    model.compile(optimizer='adam', loss='mse')
    return model

What about this one ?
"model": {
        "loss": "mse",
        "optimizer": "adam",
    "save_dir": "saved_models",
        "layers": [
            {
                "type": "lstm",
                "neurons": 999,
                "input_timesteps": 998,
                "input_dim": 1,
                "return_seq": true
            },
            {
                "type": "dropout",
                "rate": 0.05
            },
            {
                "type": "lstm",
                "neurons": 100,
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.05
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }

Yes, LSTM are ideal for this. For even stronger representational capacity, make your LSTM's multi-layered.
Using 1-dimensional convolutions in a CNN is a common way to exctract information from time series too, so there's no harm in trying. Typically, you'll test many models out and take the one that has best validation performance.

We have tried this models but the loss is extremely big (quintillion scale). What do you suggest to do further ?

I added a new one , what do you think about it ? Whatever architecture I choose the training time is extremely big . Note that sequence length is 999 and I want to predict the 1000th .


text:
Convert input dataset given in hex addresses to int
I have created an LSTM Neural Network which take as input the following format in an .csv file
sinewave
0.841470985
0.873736397
0.90255357
0.927808777
0.949402346
0.967249058
0.98127848
0.991435244

How can I write some code so it can take as input hex addresses and convert them to int ?
eg the following .xlsx file containing 400.000 samples
0xbfb22b18
0xbfb22b14
0xbfb22b10
0xbfb22b0c
0xbfb22b18
0xbfb22b14
0xbfb22b10
0xbfb22b0c
0xbfb22b18
0xbfb22b14
0xbfb22b10
0xbfb22b0c

Add train_data 1st
import pandas as pd

df = pd.read_excel('data.xlsx', index_col=None, names=['train_data'])
df['train_data'] = df['train_data'].apply(lambda x: int(x, 16))
df.to_csv('data.csv', index=False)


text:
How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?
How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.
I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.
This is a question from Neural Network Design (2nd Edition) book by
Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .
Resource : E7.5 p 224-225

I found this but I don't think it's what is needed for my question .

Some more theory about autoassociative nets can be found here p 48 .

In addition some relative paper p 16 - 20

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
\begin{equation}
    T = P
\end{equation}
\begin{equation}
    W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI
\end{equation}
Applying a prototype vector as input :
\begin{equation}
    \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k
\end{equation}
Because they are orthogonal we have :
\begin{equation}
    \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I})
\end{equation}
where $R-Q\cdot I$ is the length of vectors
So since ,
\begin{equation}
    W \cdot p_k= (R - Q \cdot I) \cdot p _k
\end{equation}
prototype vectors
continue to be eigenvectors of the new weight matrix .
It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
ability to generalize or may increase the biological
plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used


text:
Ghost camera or video overlays for example in sports
Secondary camera, ghost overlay, video merge... I do not know if what I mean has a more specific name.
I wonder if this is a thing. This could be insightful for example in racing sports where participants race one after another e.g. alpine skiing, downhill mountain bike, showjumping etc. E.g. comparing the current starter to the leader.
Given the camera position is fixed and only the camera angle and zoom is varying to focus on the current starter, the tasks to be able to overlay videos would be to:

match the timing, i.e. both videos start when the timer starts
align and overlay the videos according to specific marker points. Keypoint detection and tracking.
get the opacity right so that both videos are visible

My question is if there is any research on this. If so, what keywords do I need to search for?

Edit:
My seach led me to SIFT (Scale Invariant Feature Transform) and SURF (Speeded-Up Robust Features). Feature matching should be possible with kNN or brute force. A lot can be done with OpenCV.


text:
Can a neural network be used to predict a sequence of integers based on dataset of previously produced random numbers?
What i really want to do, is to predict an integer sequence of (5 numbers with values from 1 to 50) for example based on a big dataset of other 5 numbers sequences with same values range created by the same random number generator. I suppose there is a way to train based on the dataset and the program will find a pattern or based on the most common numbers predict the next number sequence. The more numbers will predict in the sequence correctly the better of course. Any help, directions and preferably python code would be greatly appreciated.
I recently read the following  can-a-neural-network-be-used-to-predict-the-next-pseudo-random-number and i am new to the AI field. The proposed code while it creates a sequence of 25 numbers it ends showing 20 numbers i do not understand why. It seems they try to do something similar if i understand correctly
I tried The code here can-a-neural-network-be-used-to-predict-the-next-pseudo-random-number
It shows always the same numbers no matter how many epochs and or iterations i do is that normal?
Is the last code close to what i want to accomplish?
Thanks in advance.

The post you linked to clearly states that pseudo random number cannot be predicted. Their randomness is made to be nearly perfect, and if you ever found a way to even predict a pseudo random number with 20% chance of correct, the security of the entire world would be vulnerable to attacks, as things ranges from cryptocurrency and secure data transfer is all protected by pseudo random number.

If is a truly a random number, and you could guess each of the next successive five in sequence, then you could win the lottery consistently.
This is one of the first tasks many people try to do when first learning machine learning.  If the lottery is truly a random physical process with fair, i.e., balanced ping pong balls, then you cannot predict which 5 or 6 numbers will come up next.  The Lottery Commissions around the world go through great pains to ensure that the lotteries are fair and not fraudulent.
It looks like you are using a random seed number, that is why you are getting the next number always the same.
Good luck learning your numbers !


text:
Good model and training algorithm to store texture data for fast gpu inference
Now, the following may sound silly, but I want to do it for my better understanding of performance and implementation of GPU inference for a set of deep learning problems.
What I want to do is to replace a surface texture for a 3d model by a NN that stores that texture data in some way and allows to infere the rgb color of an arbitrary texel from its UV coordinates. So basically it should offer the same functionality as the texture itself.
A regular texture lookup takes a UV coordinate and returns the (possibly filtered) RGB color at these texture coordinates.
So, I want to train a network that takes two floats in [0,1] range as input and outputs three floats of rgb color.
I further want to then train that network to store my 4096x4096 texture. So the training data I have available are 4096*4096=16777216 of <float2, float3> pairs
Finally I want to evaluate the trained network in my (OpenGL 4 or directX11) pixel shader, feeding it for every rendered pixel the interpolated UV coordinates at this pixel and retrieving the RGB value from it.
It's clear that this will

have lower fidelity than just using the texture directly
use likely more memory than just using the texture directly
be slower than using the texture directly

and as such may be silly to do, but I'd still like to try to do this somewhat optimally, especially in terms of inference performance (I'd like to be able to run it at interactive framerates at 1080p resolutions).
Can someone point me to a class of networks or articles or describe a model and training algorithm that would be well suited for this task (especially in terms of implementing inference for the pixel shader)?

authors: Роман Фролов


text:
What class of problem is this?
If I have a lot of input output pairs as training data
<float Xi, float Yi>
and I have a parametrized approximation function (I know the function algorithm, but not the values of the many many parameters it contains) which shall approximate the process by which the original data pairs were generated. The function takes two input values:
// c is a precomputed classifier for x and can have values from 0 to 255, so there can be up to 256 different classes
y = f(float x, int c)

the hidden parameters of the function are some big lookup tables (a lot of free parameters, but still much fewer than the amount of data points in the training data)
Now I want to fit all the hidden parameters that f contains AND compute for each Xi a ci, such that for the fitted function the error over all i of Yi - f(Xi, ci) is minimized
So, using some algorithm I want to fit the parameters of f and also classify the inputs Xi so that f(Xi, ci) aproximates Yi
How is this kind of problem called and what kind of algorithm is used to solve it?
I assume it's possible to initialize all hidden parameters as well as all ci with random values and then somehow use back propagation of the error to iteratively find  parameters and ci such that the function works well.
What I don't know is whether this is a well known class of problem and I just don't know the name of it, so I'm asking for pointers.
Or maybe in other words:
I have a function that has a certain layout (for performance reasons) which I want to use to approximate and interpolate my training data, I want to tune the parameters of this function such that it approximates the original data well. since the data points fall into some 'categories', I want to pre-classify the data-points to make it easier for the function to do its job. What kind of algorithm do I use to find the function's parameters and to pre-classify the input?

This problem is typically called parameter estimation or inverse modelling, and there are a variety of techniques to solve it.
If your free parameters are all continuous (i.e. none are discrete, such as integers), and your model function is differentiable, then you can turn the model into a computation graph in e.g. TensorFlow and use gradient descent methods with MSE loss, learning parameters on the data in much the same way as a neural networks. Most of the toolkits built for neural networks can do this, you just have to ignore the pre-packaged layer models and the utilities to manage them. You will need to take care with parameter initialisation, ideally you already have some rough starting values.
You can use other approaches, such as search methods too. Even when you have some discrete-valued parameters then you can use genetic algorithms for instance.
The best method to use will depend on details of your model, it is not possible to point out a generic "best". If you can use a gradient-based method directly, as you suggest in the question, then that might be the most efficient in terms of computation, provided you have some way to set reasonable initial parameters.


text:
How to cluster data points such that the number of clusters is kept minimal and each cluster projects well onto a lower-dimensional subspace?
If I want to find a (linear) subspace onto which a data-set projects well, I can simply use PCA. However, often the data can project with much smaller error if I first separate it into a couple of classes and then perform PCA for each class individually. But what If I don't know what kind of classes there might be in my data and into how many classes it would make sense to split the data? What kind of machine learning algorithm can do this well?
Example:

If I'd just cluster first based on distance in the high-dimensional space, I would arrive at the bad clustering. There are 5 clusters and the green and red clusters don't project very well onto a 2D subspace.
As a human looking at the data, I see however that if I separate the data as indicated, red and blue will project very well onto a plane each and green will project very well onto a line, so I can run PCA for each group individually.
How can I automate this clustering based on how well it will project onto as low-imensional subspaces as possible?
Something like minimize E = SumOverClusters(SumOverPoints(SquaredDist(projected_point, original_point)) * (number_dims_projected / number_dims_original)) + C * number_of_clusters
What technique is well suited to do that?
(edit: while the example shows a 3d space, I'm more interested in doing that in about 64dimensional spaces)

Searching for "PCA mixture model" gives quite a few hits, is that maybe what you are looking for? (Effectively just combining mixture of experts + PCA)


text:
A generalized quadratic loss and Newton iteration for Support Vector Regression, why doesn't it generalize well?
I'm comparing the results of an Newton optimizer for a modified version of SVM ( a generalized quadratic loss, similar to the one stated in:
A generalized quadratic loss for SVM
) with classic SVM^light for regression. The problem is that it's able to overfit the data (UCI Yacht data-set) but I can't reach the generalization results of SVM^light. I've tried several hyper-parameters grids. I'm solving the primal problem. I'll send you my code if you need it. Any suggestion?

I figured out I've not shuffled the whole data-set (training+validation+test).
Now it generalizes in a better way (at least on the UCI Yacht data-set)

I was wrong. I tried a different shuffle and the problem still persist.


text:
What's wrong with our loss and PyTorch?
Given the samples $\vec{x_i} \in \mathbb{R}^d, i \in [1,..,l]$ where $l$ is the number of training samples, $d$ is the number of input features, the related target values $y_i \in \mathbb{R}$, and the $l \times l$ matrix defined below:
$$ S_{i,j} =  e^{-\gamma_S ||\vec{x}_i - \vec{x}_j||^2} = e^{-\gamma_S (\vec{x}_i' \vec{x}_i -2 \vec{x}_i' \vec{x}_j + \vec{x}_j'\vec{x}_j)}
$$
where $i \in [1,..,l], j \in [1,..,l]$, and $\gamma_S$ is another hyper-parameter, we would like to use with PyTorch the following custom loss for a regression task:
$$\sum_{i=1}^l \sum_{j=1}^l \sqrt{|p_i-y_i|}\sqrt{|p_j-y_j|} S_{i,j}$$
where $p_i$ is the $i$-th estimation.
Our loss is implemented with this code:
def ourLoss(out, lab):
    global stra, sc
    abserr = torch.abs(out - lab).flatten().float()
    serr = torch.sqrt(abserr)
    bm = stra[sc : sc + out.shape[0], sc : sc + out.shape[0]].float()
    loss = torch.dot(serr, torch.matmul(bm, serr))
    return loss

where 'stra' is $S$, sc is a counter used for batch evaluations, and then the Adam optimizer returns a nan loss value...

Can you try to lower your learning rate (substantially). It might be that it just explodes becuase of exploding gradients.

We tried, other nan's compare.
We think the problem is related with the fact that as some serr goes to $0$, its derivative goes to infinity.
We tried to clip the gradient but it did not work.

We tried also to add a small value ( 1E-7 ) to serr, but we still have nan loss values.


text:
Glass Degradation Video Prediction
We are working on the subject above, where a sequence of $n$ glass frames forms an example with an associate target that is a video of the glass future state.
We would like to know if there exists an algorithm (ConvLSTM, Trasformer, etc…), that could be trained and that it is able to predict glasses breaking videos. In this moment, we are testing SimVPv2 (on MMNIST).

How do the glass images look like, are there like initial fractures somewhere, giving a hint that it may break in near future?

Thank you! Yes, the glasses have already fractures. One of the initial tasks is to classify them between "broken" and "intact". Then a regression algorithm could be used to estimate the degree of rupture. The final task is to have a predictor that will give the next frame of the glass given a degradation video (perhaps with a ConvLSTM or a Transformer algorithm).

At present we are trying OpenSTL-SimVPv2 on MMNIST. We are unable to run in parallel the VPTR learner, due to our 32 GB RAM limit.


text:
How to implement RL policies learned on a finite horizon?
I am modelling a ride-hailing system where passenger requests continuously arrive into the system. An RL model is developed to learn how to match those requests with drivers efficiently.
Basically, the system can run infinitely as long as there are requests arriving (infinite horizon reality). However, in order for the RL training to conduct, the episode length should be restricted to some finite duration, say $[0,T]$ (finite horizon training).

My question is how to implement the learned policy based on finite horizon $[0,T]$ to the real system with infinite horizon $[0,\infty]$?

I expect there would be a conflict of objectives. The value function near $T$ is partially cut off in a finite horizon and would become an underestimate and affect policy performance in an infinite horizon implementation. To this end, I doubt the applicability of the learned policy.

A normal way to deal with training on infinite horizon (aka "continuing" or "non-episodic") problems is to use TD learning or other bootstrapping methods (of which Q-learning in DQN is one example), and to treat the cutoff at $T$ for pseudo-episodes as a training artefact.
If the state at time $T$ was really a terminal state, the TD target would be just $r$ because $q(s^T,\cdot) = 0$ by definition, but that doesn't apply in your case.
So always use the bootstrapped TD target - e.g. $r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$ for single step TD target with a Q estimate having $\theta$ as learned params - and don't treat the horizon data any differently.
If you do this, then your concerns about under-estimates should not be an issue. Your pseudo-episodes do need to be long enough to observe the impact of multiple requests in the long term is the main issue (setting $T$ too low so that the system does not reach any kind of equilibrium would be a problem).
You could also use an average reward setting and differential value functions. It is slightly better from a theoretical standpoint, but Q-learning and DQN is fine if you don't want to be bothered with that. The same basic rule applies - ignore "episode end" for constructing TD targets because it is just a training artefact. Also ensure you set $T$ high enough that the long term impacts of a policy are observable.
If your starting state is special (e.g. cars all in fixed places, and no requests in progress) this could also be an issue, because the real world system will rarely be in that state but you will have many episode starts with it. If you cannot start the system in a reasonable random initial state for your problem, you may also want to discard data from episode starts and allow a run-in time before using data for training from the pseudo-episodes.

For infinite horizon, i.e. non-episodic tasks, can I use DQN with no episodes, γ=0.99, and only one terminal state that is the timestep limit ?

@ddaedalus: Yes, sort of, that is what this answer describes. Technically the termination condition is not real or part of the state, so you just use it to trigger a stop to collecting training data and re-set the state for convenience. You should use value function updates as if that last state is not terminal. Because it isn't.

Awesome point - ignore "episode end" for constructing TD targets because it is just a training artefact. Helping me understand instantly, thanks. Found your solution agrees with this paper I was reading. It is all about time limits that are not regarded terminal states from the perspective of the environment. However, it involves overwriting the agent training processes, kind of tricky since I am using agents from Stable Baselines. Do you have any experience for such a modification?

Your remark about the lengh of $T$ is also helpful. However, my system is time-dependent since it is a traffic system, supply and demand are constantly changing and stochastic. which means the system is non-stationary. For a non-stationary system, if I train the agent in $[0, T]$, does it mean the policy is only applicable for $[0, T]$ since the traffic conditions change after $T$?

@GuoyangQin It depends on what you mean by "non-stationary" and "time dependent". If rates of traffic depend reliably on time of day for instance, make that a feature in the state. If conditions are otherwise detectable or predictable through some feature, make that feature part of the state. If you want to deploy a trained agent that can cope with varying traffic conditions, you should train it to be optimal over those variations in conditions.

@GuoyangQin Finally, one alternative if you really do mean "non-stationary" as opposed to "depends on more factors that could become part of the state", then one solution is to deploy the system in a continuously learning setup. If this is important to you, then you could ask a new question about it


text:
How are batch statistics computed in Recurrent Batch Normalization?
I'm implementing recurrent BN per this paper in Keras, but looking at it and those citing it, a detail remains unclear to me: how are batch statistics computed? Authors omit explicit clarification, but state (pg. 3) (emphasis mine):

At training time, the statistics E[h] and Var[h] are estimated by the sample mean and sample variance of the current minibatch

Yet another paper (pg. 3) using and citing it describes:

We subscript BN by time (BN_t) to indicate that each time step tracks its own mean and variance. In practice, we track these statistics as they change over the course of training using an exponential moving average (EMA)

My question's thus two-fold:

Are minibatch statistics computed per immediate minibatch, or as an EMA?
How are the inference parameters, shared across all timesteps, gamma and beta computed? Is the computation in (1) simply averaged across all timesteps? (e.g. average EMA_t for all t)

Existing implementations: in Keras and TF below, but are all outdated, and am unsure regarding correctness

Keras, TF-A, and TF-B
All above agree that during training, immediate minibatch statistics are used, and that beta and gamma are updated as an EMA of these minibatches
Problem: the bn operation (in A, and presumably B & C) is applied on a single timestep slice, to be passed to the K.rnn control flow for re-iteration. Hence, EMA is computed w.r.t. minibatches and timesteps - which I find questionable:

EMA is used in place of a simple average when population statistics are dynamic (e.g. minibatch-to-minibatch), whereas we have access to all timesteps in a minibatch prior having to update gamma and beta
EMA is a worse but at times necessary alternative to a simple average, but per above, we can use latter - so why don't we? Timestep statistics can be cached, averaged at the end, then discarded - holds also for stateful=True


text:
How to handle extremely 'long' images?
After transforming timeseries into an image format, I get a width-height ratio of ~135. Typical image CNN applications involve either square or reasonably-rectangular proportions - whereas mine look nearly like lines:

Example dimensions: (16000, 120, 16) = (width, height, channels).
Are 2D CNNs expected to work well with such aspect ratios? What hyperparameters are appropriate - namely, in Keras/TF terms, strides, kernel_size (is 'unequal' preferred, e.g. strides=(16, 1))? Relevant publications would help.

Clarification: width == timesteps. The images are obtained via a transform of the timeseries, e.g. Short-time Fourier Transform. channels are the original channels. height is the result of the transform, e.g. frequency information. The task is binary classification of EEG data (w/ sigmoid output).
Relevant thread

Are the spatial correlations in the vertical (height) dimension relevant? I.e. does the vertical distance have physical meaning?

@MPA Question updated

What is the type of the desired output? A class label (e.g. for detection), a time series (e.g. for filtering), etc.?

@MPA Binary classification, updated again

Depending on the physical nature of the input data, you could first perform 1D convolutions with dimensionality reduction along the time axis to extract the most relevant features for each frequency band, and then combine the features of each frequency band into a class prediction. Or if the data are sufficiently smooth, you could try an auto-encoder to compress the time axis into a smaller latent space representation, and perform additional analysis on the latent space.

@MPA 16000 is precisely from the autoencoder approach; the full original timeseries spans 200,000+ timesteps. Though compressing it even further is unrealistic without loss of information - and the timeseries transform used is promising for discrimitative feature extraction. The linked thread has some sound suggestions I might summarize in an answer.

I suppose the EEG data is quasi-periodic. What about Dynamic Mode Decomposition (DMD) and performing the classification on the dominant modes?

I had recently used a slightly unorthodox method to process such images, which involved using RNNs.
Assume the image dimensions to be (16000, 120, 16) = (width, height, channels), as in the question.
Apply a 2D convolution (or multiple such convolutions) of shape(1, k, c),
such that the output of the convolutions becomes (16000, 1, c). So if you only use a single convolutional layer, k=120.
Then, squeeze the extra dimension, to get the shape (16000, c).
The problem has now been transformed back into a sequence problem! You can use RNN variants for further processing.

This defeats the purpose of the transform; see updated question.


text:
What's the intuition behind contrastive learning?
Recently, I have seen a surge of papers w.r.t contrastive learning (a subset of semi-supervised learning).
Can anyone give a detailed explanation of this approach with its advantages/disadvantages and what are the cases in which it gives better results?
Also, why it's gaining traction amongst the ML research community?

As I understand it contrastive learning is a type of data augmentation. For classification you have labeled dataset, two augmentation of the same sample should have the same label. Idea of contrastive learning: not only   two different augmentation of the same sample should have the same label, but the label should be different from the sample augmented from different label. Contrastive learning use loss which encourage that difference (therefore "contrastive") Advantages and disadvantages are the same as for common augmentation. Based on https://arxiv.org/abs/2002.05709

Contrastive learning is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a dictionary look-up problem.
If I conceptually compare the loss mechanisms for:

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations @ https://arxiv.org/abs/2002.05709
MoCo: Momentum Contrast for Unsupervised Visual Representation Learning @ https://arxiv.org/abs/1911.05722

Both MoCo and SimCLR use varients of a contrastive loss function, like InfoNCE from the paper Representation Learning with Contrastive Predictive Coding
\begin{eqnarray*}
\mathcal{L}_{q,k^+,\{k^-\}}=-log\frac{exp(q\cdot k^+/\tau)}{exp(q\cdot k^+/\tau)+\sum\limits_{k^-}exp(q\cdot k^-/\tau)}
\end{eqnarray*}
Here q is a query representation, $k^+$ is a representation of the positive (similar) key sample, and ${k^−}$ are representations of the negative (dissimilar) key samples. $\tau$ is a temperature hyper-parameter. In the instance discrimination pretext task (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.

The contrastive loss can be minimized by various mechanisms that differ in how the keys are maintained.
In an end-to-end mechanism (Fig. 1a), the negative keys are from the same batch and updated end-to-end by back-propagation. SimCLR, is based on this mechanism and requires a large batch to provide a large set of negatives.
In the MoCo mechanism i.e. Momentum Contrast (Fig. 1b), the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch.


text:
How to prove that a regularisation method simplified a neural network?
There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural network. The Dropout deactivates nodes and the L1 shrinks the weights of the model, and so on.
The main argument in favour of regularising a neural network is that by simplifying the model you are forcing it to learn more general functions and thus making the neural network more robust to overfitting or noisy input.
Once you have a model trained with, and without, regularisation, it is possible to compare their performance by calculating the error metrics on their outputs. This will prove whether the regularised model performs better than the standard model or not.
However, considering that the regularised model achieved better performance on its error metrics, how to prove that the weights of the regularised model have less variance (simpler) than the standard neural network?

This question doesn't have an answer unless you define what you mean by complexity. If by complexity you mean the number of parameters, then both regularization methods you mentioned do nothing for complexity. If by complexity you mean the variance in parameters you might have more luck, but I imagine that dropout wouldn't show any reduced complexity.

@Recessive Thank you I updated my question. But how does dropout not reduce complexity if it deactivates nodes during the training process? Having less nodes seems like a simpler model.

It inconsistently deactivates nodes during training which means you need just as many parameters, and then at testing time it doesn't deactivate them at all

Hello. I still think that this question is not clear enough, because of the part. "how to prove that the weights of the regularised model have less variance (simpler) than the standard neural network?". It seems that you're using "simpler" as a synonym for having less variance, but what do you mean by "less variance" here? You need to clarify this part, because, once we have 2 neural networks, we have 2 specific sets of weights, so the only thing that you can compute is really the variance of the values of these parameters.

So, are you looking for a theoretical result that shows that, if using e.g. L2 regularization, your parameters will be limited to a smaller set of values, so will have less variance generally?


text:
How to handle a high dimensional video (large number of frames per video) data for training a video classification network
I have a video dataset as follows.
Dataset size: 1k videos
Frames per video: 4k (average) and 8k (maximum)
Labels: Each video has one label.
So the size of my input will be (N, 8000, 64, 64, 3)
64 is height and width of video. I use keras. I am not really sure how to do an end-to-end training with this kind of dataset. I was thinking of dividing each input in blocks of frames (N, 80, 100, 64, 64, 3) for training. But still it wont work for an end-to-end network training.
I am not in favor of  dropping the frames. That might be my last choice.
Any help will be appreciated. Thanks in advance.


text:
Making generated texts from "data-to-text" more variable
I am diving in data-to-text generation for long articles (> 1000 words). After creating a template and fill it with data I am currently going down on paragraph level and adding different paragraphs, which are randomly selected and put together. I also added on a word level different outputs for date, time and number formats.
The challenge I see is, that when creating large amounts of such generated texts they become boring to read as the uniqueness for the reader goes down.
Furthermore, I also think it's easy to detect that such texts have been autogenerated. However, I still have to validate this hypotheses.
I was wondering if there is an even better method to bring in variability in such a text?
Can you suggest any methods, papers, resources or share your experience within this field.
I highly appreciate your replies!

You could handwrite different templates and choose probabilistically, according to writing style or pragmatic effects like irony and so on, but that very much depends on the domain.
If you have tabular data, from which you want to generate text, you should probably forget about GPT and so on. You only have few control (despite copy mechanism) over the generation process, you actually predict the next most probable word sequence for a given length - GPTs don't author coherent text across paragraphs, especially not when text length is more than a few hundred words.
Check the linguistic counterpart (https://arxiv.org/abs/1703.09902) to end2end generation systems. Breaking up the networks, pipelining again and using networks for controllable tasks.
e.g. build a module that selects which attributes to produce. Create RDF triples from the column head words and values in your data base. Take a Text to text Transfer model (Google's T5) and transform into surface text. You should also have a look into the webNLG challenge (https://webnlg-challenge.loria.fr/challenge_2020/). This might help. Much of this is still open research. I am quite busy in this topic, so feel free to ask.

Thx for your reply! I am also a researcher. Is there a way to get in touch with you?

VIa Mail? Don't know whether stackexchange offers private messages, I think these comments will soon get deleted:)

Btw if you are interested for an exchange message me on maggregor.samsa@gmail.com

The state of the art in text generation is the GPT model. GPT-3, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. This article and this one should give you an example of how powerful it is at text generation.
GPT is a transformer based architecture, somewhat like BERT. The main difference is it only takes into account left context, which is why it is so well suited for text generation.
GPT-3 is still very new and is not available for free. However, GPT-2, the previous release of the model, is available for free. While obviously not as advanced as GPT-3, it is still quite impressive in its own right, and for someone trying to generate text, it is the clear choice.
Here is a link to a tutorial explaining the basics to get you started with GPT-2.
If you are interested in diving into the relevant research papers:
Here are the open AI papers on GPT 1-3

Improving Language Understanding by Generative Pre-Training
Language Models are Unsupervised Multitask Learners
Language Models are Few-Shot Learners

Additionally, if you have never seen transformers before, take a look at:

Attention Is All You Need


text:
Sample size for the evaluation of Deep Learning Models
I'm evaluating the performance and accuracy in detecting objects for my data set using three deep learning algorithms. In total there are 24,085 images. I measure the performance in terms of time taken to detect the objects. To measure the accuracy, I manually count the number of objects in each image and then calculate recall and precision values for three algorithms.
However, since I'm manually counting to get actual object count, I selected only 30 images. Will that sample be enough to make a conclusion that algorithm 1 is better than others in terms performance and accuracy?


text:
Reasoning behind $Zero$ validation accuracy in the following ResNet50 model for classification
I have written this code to classify Cats and dogs using Resnet50. Actually while studying I came to the conclusion that Transfer learning gives very good accuracy for deep learning models, but I ended getting a far worse result and I didn't understand the cause for it. Any description with reasoning would be very helpful. The dataset contains 2000 images of cats and dogs as training and 1000 images as the validation set.
The following summarises my model
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, InputLayer, Flatten, GlobalAveragePooling2D
num_classes = 2
IMG_SIZE = 224
IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)
my_new_model=tf.keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=IMG_SHAPE, pooling='avg', classes=2)
my_new_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.resnet50 import preprocess_input
train_datagen = ImageDataGenerator(
 preprocessing_function=preprocess_input,
 rotation_range=40,
 width_shift_range=0.2,
 height_shift_range=0.2,
 shear_range=0.2,
 zoom_range=0.2,
 horizontal_flip=True,)

# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_generator = train_datagen.flow_from_directory(
     train_dir,  # This is the source directory for training images
     target_size=(224,224),  # All images will be resized to 224x224
     batch_size=20,
     class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
     validation_dir,
     target_size=(224, 224),
     class_mode='binary')

my_new_model.fit_generator(
     train_generator,
     epochs = 8,
     steps_per_epoch=100,
     validation_data=validation_generator)

For this I get the training logs as,
Train for 100 steps, validate for 32 steps
Epoch 1/8
100/100 - 49s - loss: 7889.4051 - accuracy: 0.0000e+00 - val_loss: 7834.5318 - val_accuracy: 0.0000e+00
Epoch 2/8
100/100 - 35s - loss: 7809.7583 - accuracy: 0.0000e+00 - val_loss: 7775.1556 - val_accuracy: 0.0000e+00
Epoch 3/8
100/100 - 35s - loss: 7808.4858 - accuracy: 0.0000e+00 - val_loss: 7765.3964 - val_accuracy: 0.0000e+00
Epoch 4/8
100/100 - 35s - loss: 7808.0520 - accuracy: 0.0000e+00 - val_loss: 7764.0735 - val_accuracy: 0.0000e+00
Epoch 5/8
100/100 - 35s - loss: 7807.7891 - accuracy: 0.0000e+00 - val_loss: 7762.4891 - val_accuracy: 0.0000e+00
Epoch 6/8
100/100 - 35s - loss: 7807.6872 - accuracy: 0.0000e+00 - val_loss: 7762.1766 - val_accuracy: 0.0000e+00
Epoch 7/8
100/100 - 35s - loss: 7807.6633 - accuracy: 0.0000e+00 - val_loss: 7761.9766 - val_accuracy: 0.0000e+00
Epoch 8/8
100/100 - 35s - loss: 7807.6514 - accuracy: 0.0000e+00 - val_loss: 7761.9346 - val_accuracy: 0.0000e+00
<tensorflow.python.keras.callbacks.History at 0x7f5adff722b0>

If I change the class_mode='categorical' it's giving error as
Incompatible shapes: [20,2] vs. [20,2048].


text:
Is reinforcement learning suited for real-time systems?
From what I have seen, any results involving RL almost always take a massive number of simulations to reach a remotely good policy.
Will any form of RL be viable for real-time systems?

Short answer: Yes, it is.
Explanation
Reinforcement learning can be considered as a online learning. That is, you can train your model with a single data/reward pairs.  As with any online learning algorithm, there are a few things to consider.
The model tends to forget the knowledge gained. To overcome this problem, one can save new data in a circular buffer called history and train the model with a portion of mix of new and old data. This is actually the common way to train an RL model and can be adopted to real-time systems. There are also others techniques to overcome it.
Another problem is that if only one data point is fed to the network, it will be impossible to apply some techniques, such as Batch normalization.

This doesn't address the massive, massive number of simulations required. IIRC the number of walking simulations for robots is way, way higher than the real time constraints imposed on animals learning to walk.

Where does this requirement come from? You can pre-train your model and then apply online learning so that it is always up to date.

The issue is IMO, real systems have built in priors I think(or models which better lead to correct parameter estimation) and it seems this is missing from the RL paradigm. Like the convergence rate for animal/human behavior is astronomically faster than any type of RL policy.

All this simulation are required to collect data. Theoretically, your model have to visit all states of the world and learn reward for each action in given state. If we train a model using Q-learning algorithm, it doesn't matter how you train your model. You have a bunch of (input, action, reward) tuples. You may feed this tuple as long as you get new one. In this case a network just an approximation of a Q-function

But do you dispute the fact that humans don't need nearly the same number of trials to reach a viable policy?

We don't know how our brain works. Q-learning has Markov assumption. It means that current state doesn't depend on previous states, which means that the model doesn't have memory.

Humans may not need as much online training as we are capable of transferring knowledge from different domains to help in achieving the task


text:
Accuracy dropped when I ran the program the second time
I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN :
class FirstFFNetwork:

  #intialize the parameters
  def __init__(self):
    self.w1 = np.random.randn()
    self.w2 = np.random.randn()
    self.w3 = np.random.randn()
    self.w4 = np.random.randn()
    self.w5 = np.random.randn()
    self.w6 = np.random.randn()
    self.b1 = 0
    self.b2 = 0
    self.b3 = 0

  def sigmoid(self, x):
    return 1.0/(1.0 + np.exp(-x))

  def forward_pass(self, x):
    #forward pass - preactivation and activation
    self.x1, self.x2 = x
    self.a1 = self.w1*self.x1 + self.w2*self.x2 + self.b1
    self.h1 = self.sigmoid(self.a1)
    self.a2 = self.w3*self.x1 + self.w4*self.x2 + self.b2
    self.h2 = self.sigmoid(self.a2)
    self.a3 = self.w5*self.h1 + self.w6*self.h2 + self.b3
    self.h3 = self.sigmoid(self.a3)
    return self.h3

  def grad(self, x, y):
    #back propagation
    self.forward_pass(x)

    self.dw5 = (self.h3-y) * self.h3*(1-self.h3) * self.h1
    self.dw6 = (self.h3-y) * self.h3*(1-self.h3) * self.h2
    self.db3 = (self.h3-y) * self.h3*(1-self.h3)

    self.dw1 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1) * self.x1
    self.dw2 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1) * self.x2
    self.db1 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1)

    self.dw3 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2) * self.x1
    self.dw4 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2) * self.x2
    self.db2 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2)

  def fit(self, X, Y, epochs=1, learning_rate=1, initialise=True, display_loss=False):

    # initialise w, b
    if initialise:
      self.w1 = np.random.randn()
      self.w2 = np.random.randn()
      self.w3 = np.random.randn()
      self.w4 = np.random.randn()
      self.w5 = np.random.randn()
      self.w6 = np.random.randn()
      self.b1 = 0
      self.b2 = 0
      self.b3 = 0

    if display_loss:
      loss = {}

    for i in tqdm_notebook(range(epochs), total=epochs, unit="epoch"):
      dw1, dw2, dw3, dw4, dw5, dw6, db1, db2, db3 = [0]*9
      for x, y in zip(X, Y):
        self.grad(x, y)
        dw1 += self.dw1
        dw2 += self.dw2
        dw3 += self.dw3
        dw4 += self.dw4
        dw5 += self.dw5
        dw6 += self.dw6
        db1 += self.db1
        db2 += self.db2
        db3 += self.db3

      m = X.shape[1]
      self.w1 -= learning_rate * dw1 / m
      self.w2 -= learning_rate * dw2 / m
      self.w3 -= learning_rate * dw3 / m
      self.w4 -= learning_rate * dw4 / m
      self.w5 -= learning_rate * dw5 / m
      self.w6 -= learning_rate * dw6 / m
      self.b1 -= learning_rate * db1 / m
      self.b2 -= learning_rate * db2 / m
      self.b3 -= learning_rate * db3 / m

      if display_loss:
        Y_pred = self.predict(X)
        loss[i] = mean_squared_error(Y_pred, Y)

    if display_loss:
      plt.plot(loss.values())
      plt.xlabel('Epochs')
      plt.ylabel('Mean Squared Error')
      plt.show()

  def predict(self, X):
    #predicting the results on unseen data
    Y_pred = []
    for x in X:
      y_pred = self.forward_pass(x)
      Y_pred.append(y_pred)
    return np.array(Y_pred)

The data was generated as follows :
data, labels = make_blobs(n_samples=1000, centers=4, n_features=2, random_state=0)
labels_orig = labels
labels = np.mod(labels_orig, 2)
X_train, X_val, Y_train, Y_val = train_test_split(data, labels, stratify=labels, random_state=0)

When I ran the program yesterday, I had gotten a training accuracy of about 98% and a test accuracy of 94%. But when I ran it today, suddenly the accuracy dropped to 60-70%. I tried to scatter plot the result, and it looked like it behaved as if it were a single sigmoid instead of the Feed-Forward Network.
ffn = FirstFFNetwork()
#train the model on the data
ffn.fit(X_train, Y_train, epochs=2000, learning_rate=.01, display_loss=False)
#predictions
Y_pred_train = ffn.predict(X_train)
Y_pred_binarised_train = (Y_pred_train >= 0.5).astype("int").ravel()
Y_pred_val = ffn.predict(X_val)
Y_pred_binarised_val = (Y_pred_val >= 0.5).astype("int").ravel()
accuracy_train_1 = accuracy_score(Y_pred_binarised_train, Y_train)
accuracy_val_1 = accuracy_score(Y_pred_binarised_val, Y_val)
#model performance
print("Training accuracy", round(accuracy_train_1, 2))
print("Validation accuracy", round(accuracy_val_1, 2)

I do not understand how this happened and cannot figure it out.

Could be your initial weights, if you keep more epochs and check the train/val loss pattern, you might be able to find out

It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network to "get UNLUCKY and get knocked into a BAD sectors of parameter space corresponding to a sudden decrease in accuracy -- sometimes it can recover from this quickly, but sometimes not.

In general, lowering your learning rate is a good approach to this kind of problem. Also, setting a learning rate schedule like FactorScheduler can help you achieve more stable convergence by lowering the learning rate every few epochs. In fact, this can sometimes cover up mistakes in picking an initial learning rate that is too high.

you can try using mini-batches.

The error (Entropy) with log functions must be used precisely.

What are some possible reasons why training accuracy would decrease over time?

Check out in this paper a novel way to change the learning rate cyclically.


text:
How to predict the "word" based on the meaning in a document?
What I mean to say is

For example, if I give the meaning of Apple from the dictionary as input to the program, it should give output as Apple.
Or I say My day to day job involves monitoring and managing the resources - the output should be Project management.

The meaning and the word could be a dictionary or it could be custom. I am looking for ideas and tools to go further on this.

While your question has some ambiguities, I try to answer.
From my understanding you want your model to predict the “topic” of a sentence or a description. It’s just a classification problem with a huge possible number of output classes.
The first initial issue is a very short length of documents (sentences). Most of the topic modeling algorithms such as LDA have a statistical approach and do not work very well with very short documents (less than 50 words could be a good definition of a very short document).
The second issue is how do you want to collect enough data to train your model that is supposed to predict the target out of an extremely large number of output classes? Dictionaries are not enough because they offer a single definition for each word. Examples of words in dictionaries don’t help much and they will probably affect your model adversely. How can your model be generalized by a single (or few) example(s) for each class?
So, it’s not possible, but maybe having some innovations can help.
Here is the definition of “apple” in the oxford dictionary: ”a round fruit with shiny red or green skin that is fairly hard and white inside”. There are just two nouns in the definition: "fruit" and "Skin", if we just read the definition without considering these two words, even we, as humans, struggle to guess.
Consider nouns in input data and use them to build up a natural graph. You just consider main classes such as "fruit". If you’re getting some good results, consider other words, adj, adv, ...

Your point of human struggle to decipher word from the dictionary without knowing the word is a very good point. thanks.