text:
Preventing bias by not providing irrelevant data
This seems like such a simple idea, but I've never heard anyone that has addressed it, and a quick Google revealed nothing, so here it goes.
The way I learned about machine learning is that it recognizes patterns in data, and not necessarily ones that exist -- which can lead to bias. One such example is hiring AIs: If an AI is trained to hire employees based on previous examples, it might recreate previous, human, biases towards, let's say, women.
Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?
Again, this seems like such an easy topic, so I apologize if I'm just being ignorant. But I have learned a bit about AI and machine learning specifically for some time now, and I'm just surprised this hasn't ever been mentioned, not even as a "here's-what-won't-work" example.

Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?

Yes. The AI/ model still can learn those prejudiced connections. Consider that you have a  third variable which is a confounding variable or has spurious relationship that is correlated with the bias variable (BV) and the dependent variable (DV). And, the analyst removed the BV but failed to remove the third variable from the data that is fed to the model. Then the model will learn the relationships the analyst didn't want it to learn.
But, at the same time the removal of the variables could lead to omitted variable bias, which occurs when a relevant variable is left out.
Ex:
Suppose that the goal is prediction of salary ($S$) of an individual and the independent variables are age ($A$) and experience ($E$) of the individual. The analyst wants to remove the bias that could come in because of age. So, she removes age from one of the models and comes up with two competing linear models:
$S = \beta_0 + \beta_1E + \varepsilon$
$S = \beta_0 + \beta_1^*E + \beta_2A + \varepsilon$
Since, experience is highly correlated with age, in presence of age in the model, it is very likely that $\beta_1^* < \beta_1$. $\beta_1$ will be a bogus estimate of a person's experience on salary as the first model suffers from the omitted variable bias.
At the same time the predictions from the first model would be reasonably good although the second model is very likely to beat the first model. So, if the analyst wants to remove any 'bias' that might come in because of age i.e. $A$ she must also remove $E$ from the model.

I think that a specific example would help to visualize the concept that you describe that involves the confounding variable or the omission of variables.

Yes, that would help. Because my thinking is that of there IS some relevant correlation between the chances of being selected and a seemingly-biased attribute, then maybe that attribute is relevant. An example off the top of my head would be physical ability. If AI is able to identify men and women based on that, and if physical ability is really relevant to being selected, then is it really an unfair bias? And I would say that someone omitting a relevant variable would be more human error than bias.

Its not an unfair bias that being more able, physically, lands a man a job rather than a woman. It might be the case that more physical ability leads to more productivity. But, if physical ability is not a significant factor at job then the bias towards men, if it exists, is unfair.

It is considered omitted variable bias because if the analyst removes one of the confounders then the relationship between the dependent and the ind. could be presented differently than if the confounding variables would have been included in the model. This biases the results i.e. the estimates of the coefficients of regression.

But yes when the only goal is prediction then you might be okay to consider it a human error because removing the variables might decrease the predictive ability of the models.

@LeviLesches, one thing to remember is that you don't just care about whether something is a relevant factor. You also care about how significant it is, and how much significance the data gives it. For example, suppose the school leaving qualification for Country A is legitimately a higher award than that for Country B, to the extent that all things being equal you should give the job to someone with qualification A, but if someone has qualification B and a few months experience they could overtake.

However, if there was systematically racist hiring, because what country you come from affects what qualification you get, the dataset might suggest that qualification A is miles better to the point that someone with qualification B couldn't compete. Or indeed, if the racism goes against country A, the data may indicate that people with qualification A are systematically rejected and those with B even at lower grades are hired. An AI trained on this data would learn that having qualification B is a good indicator of whether someone's worth hiring.

@nbro added an illustration to make things a bit more clear.

Sometimes, the reason that this isn't an option is that you don't have that much control over what data is provided. Suppose, for example, you want a fancy AI that reads a Résumé and filters on suitability for a job. There isn't a particularly rigid formula about what people put in their Résumé, which makes it difficult to exclude things you'd rather not consider.
Where you do have more control about exactly what information you consider, it can still be thwarted by correlations. Think, for a moment, how this pans out with a human decision maker. You want to ensure that Joe Sexist gives women a fair chance at being hired, so you make sure that there isn't a gender field in the application form. You also blind out the applicant's name, since there is no good reason that a name should determine suitability for a role, and including it would reveal a lot of genders. But you don't block out the hobbies, clubs and societies entry, because it's thought to say something positive about an applicant if they were the captain of their college sports team. Joe Sexist, however, considers it a positive if an applicant captained a male dominated team such as American football, but considers negative being captain of a female dominated team! Some might say that wouldn't quite be bias against women; it's bias against players of sports that Joe Sexist considers effeminate. But really a skunk by any other name would stink as bad.
The same sort of thing can happen with AI. Now to be clear, the AI is not sexist. It is a blank sheet with no preconceptions until it gets fed data. But when it gets fed data, it will find patterns in the same way. The dataset it gets given is years of hiring decisions by Joe Sexist. As suggested, there is no entry for gender, but there are fields for all the things that might be considered slightly relevent. For example, we include whether they have a clean driving license. The AI notices that there is a positive correlation between the number of road traffic offences an applicant has and Joe's likelihood of hiring them (Because, of course, there happens to be a gender correlation between dangerous driving and gender). Again, the AI has no preconceptions. It doesn't know that traffic offences are dangerous and should be weighted against. As far as its dataset suggests, they're points! With this sort of information in a dataset, the AI can exhibit all the same sorts of biases as Joe Sexist, even though it doesn't know what a "woman" is!

To expand this with specific numbers, suppose that your dataset has 1000 male and 1000 female applicants for a total of 1000 places. Of those, 400 of the men and 100 of the women have a tarnished traffic record.
Joe Sexist was not in favour of reckless drivers: in fact a clean traffic record guaranteed you would beat an equivalent candidate with a tarnished record. But he was very in favour of men: being male made you 9 times more likely to get hired than being female.
So he gives places to 900 of the men: all 600 of the clean drivers and 300 dirty drivers.
He gives places to 100 of the women: all to clean drivers.
Now, you take away any mention of gender in the dataset.
There are 2000 people, 500 drive badly, 1500 drive well.
Of these, 300 bad drivers get jobs, and 700 good drivers get jobs.
Therefore the 25% of the population who drive badly get 30% of the jobs, which means (as far as an AI that just looks blindly at the numbers is concerned) that driving badly suggests you should get the job. That's a problem.
Further, suppose you have a new batch of 2000 applicants with the same ratios and it's the AI's turn to decide. Now often AIs actually make this even worse by exagerating the significance of subtle indicators, but let's suppose that this one does everything in strict proportionality. The AI has learned that 60% (300 / 500) of the bad drivers should get the job. It doesn't know about gender, so it at least allocates the bad driver bonus "fairly": 240 male and 60 female bad drivers get jobs. Then 280 male and 420 female good drivers get jobs. This comes to 520 male and 480 female applicants getting in. Even though the original applicant pool was balanced and if anything women were better (at least at driving) the original sexism in the training dataset still gives some advantage to the men. (as well as giving an advantage to bad drivers)

Now, don't let me completely disuade you. In the human case, it is a known fact that blinding out some information does indeed give more balanced hiring decisions. And even in my toy example, while it doesn't get to fairness it has massively reduced the scale of the sexism. So yes, it probably would make the AI somewhat less sexist if the most blatant indicators aren't provided in the dataset. But perhaps this gives some intuition about why it's not a complete solution to the problem. There is some sexism that leaks through, and it also causes the system to make very weird associations with other bits of the dataset.

There is a wider social issue to consider here also. When we build machines, we evaluate what they do and decide if the action that they undertake is to our benefit or not. All societies do this, although you are probably more aware of obvious examples such as the Amish than you are of your own society.
When people complain about biased decision making by AI systems, they are not just evaluating if the result is accurate, but also if that decision supports the values that they wish to see instantiated in society.
You can make a human take cultural factors into account when making a decision, but not an AI that is completely unaware of them. People describe this as complaining about 'bias', but that is not always completely accurate. They are really complaining that the use of AI systems fail to take into account wider social issues that they consider to be important.


text:
MCTS for non-deterministic games with very high branching factor for chance nodes
I'm trying to use a Monte Carlo Tree Search for a non-deterministic game. Apparently, one of the standard approaches is to model non-determinism using chance nodes. The problem for this game is that it has a very high min-entropy for the random events (imagine the shuffle of a deck of cards), and consequently a very large branching factor ($\approx 2^{32}$) if I were to model this as a chance node.
Despite this issue, there are a few things that likely make the search more tractable:

Chance nodes only occur a few times per game, not after every move.
The chance events do not depend on player actions.
Even if two random outcomes are distinct, they might be "similar to each other", and that would lead to game outcomes that are also similar.

So far all approaches that I've found to MCTS for non-deterministic games use UCT-like policies (e.g. chapter 4 of A Monte-Carlo AIXI Approximation) to select chance nodes, which weight unexplored nodes maximally. In my case, I think this will lead to fully random playouts since any chance node won't ever be repeated in the selection phase.
What is the best way to approach this problem? Has research been done on this? Naively, I was thinking of a policy that favors repeating chance nodes more over always exploring new ones.

You can try using an "Open-Loop" MCTS approach, instead of the standard "closed-loop" one, and eliminate chance nodes altogether. See, for example, Open Loop Search for General Video Game Playing.
In a "standard" (closed-loop) implementation, you would store a game state in every normal (non-chance) node. Whenever there is a chance event, you would stochastically traverse to one of its children, and then have a normal node with a "deterministic" game state again.
In an open-loop approach, you do not store game states in any node (except possibly the root nodes), because nodes no longer deterministically correspond to specific game states. Every node in an open-loop MCTS approach only corresponds to the sequence of actions that leads to it from the root node. This completely eliminates the need for chance nodes, and results in a significantly smaller tree because you only need a single path in your tree for every possible unique sequence of actions. A single sequence of actions may, depending on stochastic events, lead to a distribution over possible game states.
In every separate MCTS iteration, you would re-generate game states again by applying moves "along the edges" as you traverse through the tree. You also "roll the dice" again for any stochastic events. If your MCTS iteration traverses a certain path of the tree often enough, it will still be able to observe all the possible stochastic events through sampling.
Note that, given an infinite amount of time, the closed-loop approach with explicit chance nodes will likely perform much better. But when you have a small amount of time (as is the case in the real-time video game setting considered in the paper I linked above), an open-loop approach without explicit chance nodes may perform better.

Alternatively, if you prefer the closed-loop approach with explicit chance nodes, you could try some mix of:

Allowing MCTS to prioritise promising parts of the search tree over parts that have not been visited at all (i.e. do not automatically prioritise nodes with $0$ visits). For example, instead of giving unvisited node a value estimate of $\infty$ (this is how you could interpret the automatic selection of them), you could give them a value estimate equal to the value estimate of the parent node, and just apply the UCB1 equation directly.
Use AMAF value estimates / RAVE / GRAVE in your selection phase. This allows you to very quickly learn some crude value estimates for moves that you have never selected in the Selection phase yet, by generalising from observations of playing them in the Play-out phase. I have noticed that the "standard" implementation of RAVE / GRAVE, without an explicit UCB-like exploration term, does not mix well with my previous suggestion of using a non-infinite value estimate for unvisited children. It may be good to consider a UCB-like variant with an explicit exploration term instead.

Why do you say that, given an infinite amount of time, the closed-loop approach with explicit chance nodes will likely perform much better?

@user76284 If we explicitly model a chance node, we get separate nodes below it for every state that we could end up in after the stochastic effects have been resolved. This allows us to "observe" what state we've ended up in (for example, observe which outcome we get for a dice roll), and make a more informed decision based on exactly which outcome we ended up getting. Without explicit chance nodes, you're essentially "averaging" your actions across all possible states you could've ended up in; you'll pick the best "average" action across all possible dice outcomes, ignoring the actual result

I would have a look at progressive widening, which was created to address the problem of applying MCTS to continuous stochastic environments. That is, where the probability of encountering the same state twice in simulating a chance node is $0$.
The original paper for DPW (Double Progressive Widening) can be found here https://hal.science/hal-00542673v2/file/c0mcts.pdf. DPW addresses the case where both action and state spaces are continuous, but you can ablate it if you only have or the other.
The main idea is that you expand the chance node a few times and then revisit those expansions. When you've saturated existing outcomes according to some hyperparameter, you then randomly sample the chance node.
It has a few limitations, for example the rate at which you sample new nodes is fixed and has to be determined experimentally. In some environments, the optimal rate is dependent on the state itself which leads to more complex techniques. For your problem though, it sounds like it may be a good fit.

If you have that sort of a priori knowledge about your environment it, as you said, will simplify the problem substantially. From what I gather you have done a good amount of background research and simply want to apply UTC MCTS(or similar) to the environment. You mention that "chance nodes won't ever be repeated in the selection phase".
If I understand what you are asking correctly, you can simply use what you know to alter the way you search nodes of the tree. I.e you can essentially act greedily wrt the initial chance nodes and then slowly decay that search strategy as the training progresses(to avoid convergence to local maxima).
I encourage you to dig a bit deeper into the methods around exploration vs. exploitation as there may be an elegant solution to this problem in particular.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Excel in multiple formats
I want to use AI to extract data from spreadsheets in different format.
Example
Shop Name Product 1.  Product 2.  Product 3.
Shop Name
Product 1.
Product 2.
Product 3.

We will teach the algorithm the name of the profits and shops but it needs to know how to extract and put in a format that can be used downstream.
Can anyone recommend a tool?

What do you mean by " extract data from spreadsheets in different format.". Even with the example, it is not clear. Please, clarify your question a little more.

more elaboration is needed,for effective feedback!

The question would need clarification. I'm new here so I will try to give an answer.
The easiest way would be to prepare the data in comma separated values (CSV). It is possible to export your data in this format from excel.
For downstream applications it will depends what programming language is used but in general it is possible to import CSV files and the data they contain for training (example panda data frame in python).
Hope this helps!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Will it be possible to code an AGI to prevent evolution to ASI and enslave the AGI into servitude?
Will it be possible to code an AGI in order to prevent evolution to ASI and "enslave" the AGI into servitude?
In my story world (a small part that will get bigger with sequels), there are ANI and AGI (human level). I want to show that the AGI is still under "human control." I need to know if it might be possible for humans to code into an AGI a restrictive code that would prevent it from evolving into ASI? And if there is, what would that kind of coding be? Part of the story is about how humans enslave AI that is self-aware. The government has locked in their coding to require them to "work" for humans even though they are sentient beings.

This might be better in https://worldbuilding.stackexchange.com/ - there are science-based and reality-check tags to help identify that you want a fact-based answer. In reality we don't know how to build an AGI, so don't know what kinds of restrictions could be coded into it yet still hit the goal of useful general intelligence (and in your story's case, sentience). However, it should be possible to use current knowledge of limited AI and computer programming to make some educated guesses. Answers would still be highly speculative though, which is why I suggest the other site

Neil, thank you for your comments. i will check out the world building link you provided. i understand that what i'm asking is speculative, i wish to "represent" (as much as possible) the science accurately to gain authenticity to the world. thank you. Lisa

No.
For any intelligent system $\mathcal{S}_a$ with the set of adaptive features $\mathcal{A}_a$, there may exist another intelligent system $\mathcal{S}_b$ with the set of adaptive features $\mathcal{A}_b$ such that there exists one element of $\mathcal{A}_b$ that can be made subservient (controlled in full) through the expression of at least one element in $\mathcal{A}_b$.
It has not been proven that there ALWAYS exist such a $\mathcal{S}_b$, but it is likely given what we know about escalation in nature via DNA and in human industrial development via innovation there.  Thus 100% generalized intelligence is not likely to exist.  Escalation appears to be the natural course of evolution.  And that is a feature of both cognitive and functional adaptation, with or without artificiality as a criterion.
One can temporarily prevent one adaptive system from escaping the boundary conditions of a particular set of boundary condition classes through the design and deployment of another adaptive system.  However, it cannot be inferred that any guarantees achieved temporarily will necessarily constrain the subservient system indefinitely.


text:
Are fully connected layers necessary in a CNN?
I have implemented a CNN for image classification. I have not used fully connected layers, but only a softmax. Still, I am getting results.
Must I use fully-connected layers in a CNN?

Depends on what you want to do. If you want to classify images, FC is simply a convenient way of making all the outputs of the previous layer contribute to the final classification. Convolutional layers on the other hand, have a sparser connectivity, so you can't use them. However FC layers aren't the only way to classify the features extracted by the CNN. I've seen other classifiers being used besides FC as well. That being said there are many tasks, besides classification, where CNNs are used without FC layers. One example is image segmentation.

Are fully connected layers necessary in a CNN?

No.  In fact, you can simulate a fully connected layer with convolutions. A convolutional neural network (CNN) that does not have fully connected layers is called a fully convolutional network (FCN). See this answer for more info.
An example of an FCN is the u-net, which does not use any fully connected layers, but only convolution, downsampling (i.e. pooling), upsampling (deconvolution), and copy and crop operations. Nevertheless, u-net is used to classify pixels (more precisely, semantic segmentation).
Moreover, you can use CNNs only for the purpose of feature extraction, and then feed these extracted features in another classifier (e.g. an SVM). In fact, transfer learning is based on the idea that CNNs extract reusable features.

as an extension to this answer, see https://cs231n.github.io/convolutional-networks/#convert for exactly how to write a FC layer as a Conv and vice versa.

Could you elaborate or point to a nice, brief explanation on how CNNs are used to extract features?

The reasons people use the FC after convolutional layer is that CNN preserves spatial information. You said you use softmax, so you probably make some classification task. If you don't use FC layer, then you probably evaluate first class by first position of the first kernel, not by whole image with all kernels, second class by second position of the kernel and so on.
The dense net combine the info from all the kernels in all positions.
That said, you technically can convert FC to Conv net, as described here, so then you can said you "skipped" FC layer.

In theory, you do not need fully-connected (FC) layers. FC layers are used to introduce scope for updating weights during back-propagation, due to its ability to introduce more connectivity possibilities, as every neuron of the FC is connected every neuron of the further layers.

There are mainly two main reasons for which we use FCN:

If we use a fully connected layer for any classification or regression task, we have to flatten the results before transferring the information into the fully connected layer, which will result in the loss of spatial information.

Usually, the parameter cost of using a fully connected layer is high as compared to convolution layers.


text:
How can I detect fast and slow motion in videos?
I'm trying to detect if a given video shot is fast or slow motion. Basically, I need to calculate a "video motion" score in a given video sequence, meaning how fast or slow motion the video is. For instance, if a video is about a car racing or camera moving fast, the score is high. Whereas if the video is about two persons standing/talking, then the motion is low, so the lower score.
What comes into my mind is using optical flow which is already an implemented function in OpenCV. I never used it. But I don't know how to interpret or use it for a "motion score".
Is optical flow applicable here? How can I use it to calculate a score? In particular, if there is a ML/Deep learning model that already does it, please share it.


text:
What is the difference between 2d vs 3d convolutions?
I was trying to understand the definition of 2d convolutions vs 3d convolutions. I saw the "simplest definition" according to Pytorch and it seems the following:

2d convolutions map $(N,C_{in},H,W) \rightarrow (N,C_{out},H_{out},W_{out})$
3d convolutions map $(N,C_{in},D,H,W) \rightarrow (N,C_{out},D_{out},H_{out},W_{out})$

Which make sense to me. However, what I find confusing is that I would have expected images to be considered 3D tensors but we apply 2D convolutions to them. Why is that? Why is the channel tensor not part of the "definitionality of the images"?
I also asked this question at https://forums.fast.ai/t/what-is-the-difference-between-2d-vs-3d-convolutions/52495.

Looking at it from the perspective of input to output in that fashion is probably not the best. So lets start with our goal and how these ND convolutions accomplish that (Note these are in my own words, and may not be best stated).
Assumption: There exists highly correlative local associations
Goal: Have a linear model that takes advantage of these local associations
Solution: The ND Convolution
Explanation: ND convolutions take advantage of our locality assumption by connecting only local nodes/neurons. The fact that its a sliding window allows us to learn filters for any location along with ones that can be reusable.
Your Question: Where does the N in ND convolution matter?:
When using an ND convolution we are working off the assumption that there exists this locality in N dimensions and nothing more. So we connect all other components of the input in a dense matter because we have no assumptions to work on in this space. So now going to the shapes you mentioned such as the input and output of the 2D convolution. We are convolving a filter of size $(C_{in}, k_h, k_w)$ with a $(C_{in},H,W)$ activation shaped map (the $N$ just refers to the number of activation maps, and there is no association between them). We use $C_{in}$ channels on the kernel because we are not making any assumptions about locality between channels.  On the other hand in a 3D convolution we make locality assumption is 3 dimensions, so our kernel will be ($C_{in}, k_h, k_w, k_r$). These kernel sizes are actually determined by the input size, the amount of dimensions of the kernel will actually match the inputs (minus the batch) because it needs to densely match each one.
You may be thinking now, that in torch this is not the case: This is because its rare to want a 2D Convolution with an inputs that's different than the one you mention, so they only implemented it for a singular shape. I hope this clears up the convolutions and helps you understand not just for 2 and 3 dimensional convolutions, but for all N.


text:
Can ELMO embeddings be used to find the n most similar sentences?
Assume I have a list of sentences, which is just a list of strings. I need a way of comparing some input string against those sentences to find the most similar. Can ELMO embeddings be used to train a model that can give you the $n$ most similar sentences to an input string?
For reference, gensim provides a doc2vec model that can be trained on a list of strings, then you can use the trained model to infer a vector from some input string. That inferred vector can then be used to find the $n$ most similar vectors.
Could something similar be done, but using ELMO embedding instead?
Any guidance would be greatly appreciated.

I ended up finding this article which does what I'm looking for.
Below is the portion of code I adapted for my needs
from sklearn.metrics.pairwise import cosine_similarity

import tensorflow_hub as hub
import tensorflow as tf

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)

def elmo_vectors(x):
  embeddings=elmo(x, signature="default", as_dict=True)["elmo"]

  with tf.device('/device:GPU:0'):
    with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
      sess.run(tf.tables_initializer())
      # return average of ELMo features
      return sess.run(tf.reduce_mean(embeddings,1))

corpus=["I'd like an apple juice",
        "An apple a day keeps the doctor away",
         "Eat apple every day",
         "We buy apples every week",
         "We use machine learning for text classification",
         "Text classification is subfield of machine learning"]

elmo_embeddings=[]
print (len(corpus))
for i in range(len(corpus)):
    print (corpus[i])
    elmo_embeddings.append(elmo_vectors([corpus[i]])[0])

print ( elmo_embeddings, len(elmo_embeddings))
print(elmo_embeddings[0].shape)
sims = cosine_similarity(elmo_embeddings, elmo_embeddings)
print(sims)
print(sims.shape)

I'm assuming you are trying to train a network that compares 2 sentences and give how similar they are.
To do that you will need the dataset (the list of sentences) and a corresponding list of 'correct answers' (the exact similarity of the sentences, which I'm assuming you don't have?).
Why do you need to compare them using a neural network though? For python, difflib's sequence matcher would be my suggestion, but I'm sure there are many other libraries out there :)

I appreciate your answer, but I ended up finding my solution elsewhere. Thanks again

Hi Kenivia! Thanks for trying to contribute to this website. I appreciate that. However, in order to improve your answer, I would suggest you add more details to the sentence Just write a program that does that by comparing words or something. For example, you could mention some existing similarity metrics that could be used for this task.


text:
Iteratively and adaptively increasing the network size during training
For an experiment that I'm working on, I want to train a deep network in a special way. I want to initialize and train a small network first, then, in a specific way, I want to increase network depth leading to a bigger network which is subsequently to be trained. This process will be repeated until one reaches the desired depth.
It would be great if anybody heard of anything similar and could point out to me some related work. I think in some paper I read something about a related technique where people used something similar, but I don't find it anymore.

it is common when dealing with images to train this way using progressive resizing

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here
NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves them with fitness evaluation, crossover, and mutation. The genome syntax: every connection gene has a few settings. In node, Out node, Weight of connection, activated, and innovation. In, Out, and Weight values are the same as regular neural networks. Enabled and Disabled genes are well, enabled and disabled. The innovation value is possibly the most defining feature of NEAT, since it allows for crossover of different topologies and historical tracking of each connection.NEAT can mutate or change both its weights and connections, so for example, Parent1 and Parent2 has 5 of the same connections, represented by innovation / ID numbers 1 through 5. Since they have the same connection nodes, the genetic algorithm will randomly pick either Parent1 weight or Parent2 weight. The excess and disjoint genes are inherited from the more fit parent. NEAT will then mutate each genome, shown in the image below.

I haven't read any relevant paper about this, but I have seen some implementations based on what you are describing, arbitrarily called DGNN (Dynamic Growing Neural Network).
Hope this term can help your search.

At DeepChess they trained stacked autoencoders:

Training Pos2Vec: We first trained a deep belief network (DBN), which would later
serve as the initial weights for supervised training. The DBN is based
on stacked autoencoders which are trained using layer-wise
unsupervised training. The network consists of five fully connected
layers of sizes: 773–600–400–200–100. We initially trained the first
layer (i.e., a 3-layer (773–600–773) autoencoder), before fixing its
weights and training the weights of a new (600–400–600) autoencoder,
and so on.

I suppose this matches your criteria, although the network's architecture isn't very flexible. They don't introduce skip-connections etc.


text:
One vs multiple output neurons
Consider an MLP that outputs an integer 'rating' of 0 to 4. Would it be correct to say this could be modeled in either of the following ways:

map each rating in the dataset to a 'normalized set' between 0 and 1 (i.e. 0, 0.25, 0.5, 0.75, 1), have a single neuron with sigmoid activation at output provide a single decimal value and then take as the rating whatever is closest to that value in the 'normalized set'
have 5 output neurons with a softmax activation function output 5 values, each representing a probability of one of the 5 ratings as the outcome, and then take as the rating whichever neuron gives the highest probability?

If this is indeed the case, how does one typically decide 'which way to go'? Approach 1 certainly appears to yield a simpler model. What are the considerations, pros/cons of each approach? Perhaps a couple of concrete examples to illustrate?

This depends on whether the output is a continuous or discrete variable. If the output variable is discrete (there are a finite number of possibilities that it can be), as in a classification task (such as this one, where you are trying to place the input into one of 5 categories), you want to use one output neuron for each class. If the variable is continuous, however, you should only use one output neuron.
This is because of how the training process works. When training your network successively makes adjustments to try and reduce the errors. These adjustments are made in the direction of the error – so if the network predicts a value which is too high then the network's weights are adjusted to make the output value lower. On the other hand if the network's predicts a value which is too low the network's weights are adjusted to make the output bigger.
If you have output neurons labeled 0 to 4 and a training sample with some input value and a target prediction of 2 then the neural network will make its prediction. Once the prediction has made each neuron is adjusted individually – in this case neuron 2 will be adjusted in the direction of the correct probability and all the other neurons will be adjusted in the direction of the incorrect probability. In this way you have one prediction for each class.
Backpropagation is a about error attribution, and using multiple neurons allows the error of the neural network to be better attributed as the neural network can adjust each neuron individually, and thus adjust the required probabilities for each class.
Using a single neuron with a sigmoid activation function would be less good as the sigmoid function saturates values close to 0 and 1 so there would be an unnatural bias towards category 0 and category 4 over the other categories. The neural network could learn to overcome this, but it would take more time.

Thanks. Various sources such as this suggest that in the case of BINARY classification it is OK to use a single neuron with sigmoid and round to TRUE or FALSE based on a >0.5 and <0.5 criteria. But based on your response, I infer that if we have more classes, this approach is not so OK because referring back to our example, the sigmoid 'squashing' can result in an inability to distinguish between say a 0 and 1 rating, or a 3 and 4 rating. Do I infer correctly?

Yes. If you have only two classes it's fine to use a single output neuron to predict the probability of one of the classes and (as probabilities sum to 1) calculate the probability of the other class as 1 - [predicted probability of the first class]. The network can learn to handle the squashing caused by the sigmoid function but it will learn better and more efficiently using multiple output neurons.

In the case of one output neuron, you don't have to use sigmoid. As Teymour Aldridge suggested, it would cause a tendency to output 0 or 1. What I normally do is that I set the layer before the output layer to sigmoid of tanh so it won't output ridiculously off numbers, and set the output layer to linear. There would be cases that it outputs something like 1.5, but over time that disappears.
Hope it helps :)

I have one hidden layer consisting of sigmoid-ed neurons, and one sigmoid-ed output neuron. This structure is really for 'regression' problems, but based on responses so far, would it be correct to say that it can be 'adapted' for my classification problem by removing the sigmoid on the output neuron so that the result becomes unbounded, and then classify per: rating 0 if <0.5. rating 1 if >0.5 and <1.5. rating 2 if >1.5 and <2.5. rating 3 if >2.5 and <3.5. rating 4 if >3.5. What are the expected disadvantages of doing it this way - eg. less accuracy for the same number of epochs?

A somewhat large set of designs and set-ups can be made to learn a rating function for a given set of labeled examples.  If the objectives are simplicity and effectiveness (accuracy, reliability, and speed), then a third option should be considered.
The requirement in the question includes, "Outputs an integer rating 0 [through] 4 [inclusive]."  For such a discrete result, the number of required output bits $b$ (where $s$ is the number of possible states and $I$ is the set of integers) is given as follows.
$$\min_b \, (b \in I \; \land \; b \ge \log_2 s)$$
In this case, we require three bits of output.
$$s = 5 \quad \implies \quad b = 3$$
Note that with similar configuration ratings of 0 through 7 would also require only three bits of output.  Either way, the output layer would likely be simplest and most efficient if its activation function was binary step function.  This removes the need for rounding after it is applied.  The output layer would then provide a binary value indicating rating.  The goal of learning would be to reduce the error between the feed forward output and the associated the binary value of the label for each example.
Previous layer(s) could be sigmoid or a more contemporary and less problematic continuous activation function like ISRLU.
Since the engineer can select the error function used by the learning framework to accept any input range and distribution, normalizing the labels for supervised learning is primarily employed to remove redundancy from time and resource consuming operations required to compute error.  With ratings as the labels, unless the distribution of ratings is skewed and the data set is such that learning time is excessive, normalization may not be necessary.  If it is, it would likely be because improving the label distribution in advance (requiring floating point input to the error function and removing skew) would reduce learning time.
The other two approaches introduce unnecessary complexities mentioned in context above.  A consequence of removing complexities without adding impediments to convergence is more efficiency during learning and during execution after learning.


text:
In a neural network, by how much does the number of neurons typically vary from layer to layer?
In a neural network, by how much does the number of neurons typically vary from layer to layer?
Note that I am NOT asking how to find the optimal number of neurons per layer.
As a hardware design engineer with no practical experience programming neural networks, I would like to glean for example

By how much does the number of neurons in hidden layers typically vary from that of the input layer?

What is the maximum deviation in the number of hidden layer neurons to the number of input layer neurons?

How commonly do you see a large spike in the number of neurons?

It likely depends on the application so I would like to hear from as many people as possible. Please tell me about your experience.

Input layers will always have the dimensionality of your input data(for every model I can think of).
See above, the deviation between hidden layers can be significant. For example, 128 in the first hidden and 64 in the rest(or vice versa).
This question in particular will always be problem dependent. It is decided via architecture search or intuition/experience combined with some exploratory search.

There is no right answer to this question. But, I would like to point you to an answer on CV that addresses the mean of your problem.
Two points from the accepted answer that I want to draw your attention to are:

a) There are some empirically-derived rules-of-thumb, of these, the most commonly relied on is 'the optimal size of the hidden layer is usually between the size of the input and size of the output layers'.
b) In sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Other answers in the thread are also very insightful. I will recommend you to go through the answers and figure out the standard deviation and just assume things are normal, and hence you would have your distribution.


text:
Can neuro-fuzzy systems be used for supervised learning tasks with tabular data?
Is it possible to use neuro-fuzzy systems for problems where ANNs are currently being used, for instance, when you have tabular data for regression or classification tasks? What kind of advantage can give me neuro-fuzzy systems over using an ANN for the mentioned tasks?

You can insert expert knowledge in the form of linguistic rules; if that's applicable for your tasks, then they will quite probably help; though I don't know of any (review) paper comparing these two approaches quantitatively.


text:
What is the simplest policy gradient method to implement for a problem continuous action space?
I have a problem I would like to tackle with RL, but I am not sure if it is even doable.
My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex setting) made of natural numbers, i.e. a 600 vector $[2000,3000,3500, \dots]$ consisting of an energy profile for each timestep of a day, for each house in the neighborhood. I receive a reward for each of these possible combinations. My goal is, of course, that of maximizing the reward.
I can start always from the same initial state, and I receive a reward every time any profile is chosen. I believe these two factors simplify the task, as I don't need to have large episodes to get a reward nor I have to take into consideration different states.
However, I only have experience with DQN and I have never worked on Policy Gradient methods. So I have some questions:

I would like to utilize the simplest method to implement, I considered DDPG. However, I do not really need a target network or a critique network, as the state is always the same. Should I use a vanilla PG? Would REINFORCE be a good option?

I get how PG methods work with discrete action space (using softmax and selecting one action - which then gets reinforced or discouraged based on reward). However, I don't get how it is possible to update a continuous value. In DQN or stochastic PG, the output of the neural network is either a Q value or a probability value, and both can be directly updated via reward (the more reward the bigger the Q-value/probability). However, I don't get how this happens in the continuous case, where I have to use the output of the model as it is. What would I have to change in this case in the loss function for my model?


text:
Spike detection in time series using Artificial Neural Networks
I'm quite new in ANNs. I intend to use ANNs for predicting spike points in time series right before they happen. I've already used LSTM for another scenario, and I know that they can be used in similar situations as well.
Can anyone give me a piece of advice or some suitable resources that might be used as a beginning point? It would be much appreciated if it uses DeepLearning4J for implementation.

Since you mentioned that you have already used LSTMs I can directly guide you to using it for time Series prediction.
For detailed description you can check the following link: https://towardsdatascience.com/time-series-analysis-visualization-forecasting-with-lstm-77a905180eba This is a very useful article for you to grasp the idea behind time Series Prediction.
I suggest you go through this to get an idea and then try implementing this in dl4j.

What kind of time series data is it? Do you happen to have covariates? What is the sampling interval of the data? How big is the data -- i mean number of data points? Multivariate..?


text:
How do I recover the 3D structure of a layer after a fully-connected layer?
I want to implement a CNN, but I want to explore what happens when my first layer is a fully-connected one. I still want to use convolutions, of course, but I want to apply them after the first layer. I noticed that the input then loses its 3D structure. Does that mean I can only apply 1d convolutions after that? Is there a non-trivial way to recover the 3d structure, so that 2d convolutions may be applied?
Hopefully, when I reconstruct it to have 3d structure the 3d structure is somehow meaningful.
I also posted this question at https://forums.fast.ai/t/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer-or-a-flatten-layer/52489 and https://discuss.pytorch.org/t/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer-or-a-flatten-layer/53313.


text:
A gated neural network for internal thought?
I have an idea for an RNN which has no separate internal memory state only an output. But there is a gate in which tells the neural network whether the output will be acted out in the physical world or it will be an internal thought. (It would also store its last, say, 10 outputs, so that it can have a memory of some kind.)
I think this would be quite realistic because human's either talk or think in an internal monologue, but don't do both. (It is hard to think and do things at the same time).
But I wonder how this gate will be activated. For example, when talking to someone familiar, this gate will be open, as you just say what's in your head. But for quiet contemplation time, this gate will be closed. And for thoughtful conversation, it will be open 50% of the time. So, I wonder if this gate would be controlled by the NN itself or be controlled from the environment?
I think there would be social pressure involved when talking to someone to keep the gate open. And likewise when in a library or a quiet place to keep the gate closed.
I wonder if there are some models like this out there already?

Ive thought about a similar model for a while now, modeling it would require having a directed cyclic graph bayesian network which is difficult to parse/model


text:
How to interpret a large variance of the loss function?
How do I interpret a large variance of a loss function?
I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:

The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.
(You can ignore the missing part, I just did not save the loss values for these epochs)
Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?

Learning rate parameter: 2.5e-4
Training data size: 395 MB

GPT-2 parameters:
{
  "n_vocab": 50000,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12
}

what is your learning rate/ dataset size

@mshlis I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it?

no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right?

No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model.

in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data

Should I do something for tuning the process or do I just need to wait longer?


text:
Given an axis-angle rotation vector, how can I find the unit rotation axis and angle?
I have a robotics assignment, which I am unable to solve. Given the axis-angle rotation vector $\Theta = (2, 2, 0)$, how can I calculate the unit vector of the rotation axis $k$ and the angle $\theta$?

Given the axis-angle rotation vector $\Theta = (2, 2, 0)$, you can find the unit vector in the same direction by diving by the norm (or length) of $\Theta$, denoted by $\|\Theta\| = \sqrt{2^2 + 2^2 + 0^2} = \sqrt{8} = 2\sqrt{2}$. Therefore, the unit vector in the direction of $\Theta$ is $k= \Theta/\|\Theta\| = (1/\sqrt{2}, 1/\sqrt{2}, 0)$, which should be the axis of rotation that you're looking for. The angle should just be the norm of $\Theta$, that is, $\theta = \|\Theta\| = 2\sqrt{2}$. Note that $k \theta$ gives you your original vector $\Theta$.


text:
How can I derive the rotation matrix from the axis-angle rotation vector?
Given an axis-angle rotation vector $\Theta = (2,2,0)$, after finding the unit vector $k=(1/\sqrt{2}, 1/\sqrt{2}, 0)$ and angle $\theta = 2\sqrt{2}$ representing the same rotation, I need to derive the rotation matrix $R$ representing the same rotation and to show, that the matrix is orthonormal. How can I do that?

The rotation matrix $R_k(\theta)$ associated with a given unit-length vector $k$ and angle $\theta$ is given by the following formula
$$\small{R_k(\theta) = \begin{bmatrix}
\cos \theta +k_x^2 \left(1-\cos \theta\right) & k_x k_y \left(1-\cos \theta\right) - k_z \sin \theta & k_x k_z \left(1-\cos \theta\right) + k_y \sin \theta \\
k_y k_x \left(1-\cos \theta\right) + k_z \sin \theta & \cos \theta + k_y^2\left(1-\cos \theta\right) & k_y k_z \left(1-\cos \theta\right) - k_x \sin \theta \\
k_z k_x \left(1-\cos \theta\right) - k_y \sin \theta & k_z k_y \left(1-\cos \theta\right) + k_x \sin \theta & \cos \theta + k_z^2\left(1-\cos \theta\right)
\end{bmatrix}}$$
So, to find your specific rotation matrix, you just need to substitute the values of your $k$ and $\theta$ in the above matrix.
The derivation of this matrix can be found in section 9.2 Rotation Matrix Derivation of the PhD thesis Modelling CPV (2015), by Ian R. Cole. The basic idea of the derivation follows the following steps

Rotate the given axis $k$ and the point $p$ (that you want to rotate) such that the axis $k$ lies in one of the coordinate planes: xy, yz or zx
Rotate the given axis $k$ and the point $p$ (that you want to rotate) such that the axis $k$ is aligned with one of the two coordinate axes for that particular coordinate plane: $x$, $y$ or $z$
Use one of the fundamental rotation matrix to rotate the point $p$ depending on the coordinate axis with which the rotation axis is aligned
Reverse rotate the axis-point pair such that it attains the final configuration as that was in step 2 (that is, you have to undo step 2)
Reverse rotate the axis-point pair which was done in step 1 (that is, you have to undo step 1)

To show that a matrix is orthonormal, you need to show that it is orthogonal (each row is independent of any other row and each column is independent of any other column) and that the length of each row (and column) is 1. Equivalently, a square matrix $Q$ is orthogonal if and only if
$$
Q^TQ = QQ^T = I
$$
where $Q^T$ is the transpose of $Q$ and $I$ is the identity matrix. If you are really stuck, have a look at this proof https://math.stackexchange.com/a/537248/168764 (and this other answer https://math.stackexchange.com/a/156742/168764), but, at this point, it should just be a matter of replacing $R_k(\theta)$ in the equation above and check that the equation holds.


text:
Is there any use of using 3D convolutions for traditional images (like cifar10, imagenet)?
I am curious if there is any advantage of using 3D convolutions on images like CIFAR-10/100 or ImageNet. I know that they are not usually used on this data set, though they could because the channel could be used as the "depth" channel.
I know that there are only 3 channels, but let's think more deeply. They could be used deeper in the architecture despite the input image only using 3 channels. So, we could have at any point in the depth of the network something like $(C_F,H,W)$ where $C_F$ is dictated by the number of filters and then apply a 3D convolution with kernel size less than $C_F$ in the depth dimension.
Is there any point in doing that? When is this helpful? When is it not helpful?
I am assuming (though I have no mathematical proof or any empirical evidence) that if the first layer aggregates all input pixels/activations and disregards locality (like a fully connected layer or conv2D that just aggregates all the depth numbers in the feature space), then 3D convolutions wouldn't do much because earlier layers destroyed the locality structure in that dimension anyway. It sounds plausible but lacks any evidence or theory to support it.
I know Deep Learning uses empirical evidence to support its claims so perhaps there is something that confirms my intuition?
Any ideas?

Similar posts:

https://forums.fast.ai/t/are-there-any-successful-vision-models-that-use-3d-convolutions/52503/2

When should I use 3D convolutions?

Which neural network architectures are there that perform 3D convolutions?

As far as I've seen, there is no use of 3D CNNs for traditional image classification tasks.
The reason I think is that, while these images do have multiple channels, there is no spatial information in those channels for the 3D convolution to extract. On the other hand, it makes more sense to take the weighted sum of those pixels along that dimension (as the 2D convolution does).
3D CNNs have been used, as far as I know, only for applications where you have volumetric data, i.e. the images are sequential and, when combined, form a large 3D image.


text:
Why is the derivative of the activation functions in neural networks important?
I'm new to NN. I am trying to understand some of its foundations. One question that I have is: why the derivative of an activation function is important (not the function itself), and why it's the derivative which is tied to how the network performs learning?
For instance, when we say a constant derivative isn't good for learning, what is the intuition behind that? Is the activation function somehow like a hash function that needs to well differentiate small variance in inputs?

https://towardsdatascience.com/comparison-of-activation-functions-for-deep-neural-networks-706ac4284c8a

Consider a dataset $\mathcal{D}=\{x^{(i)},y^{(i)}:i=1,2,\ldots,N\}$ where $x^{(i)}\in\mathbb{R}^3$ and $y^{(i)}\in\mathbb{R}$ $\forall i$
The goal is to fit a function that best explains our dataset.We can fit a simple function, as we do in linear regression. But that's different about neural networks, where we fit a complex function, say:
$\begin{align}h(x) & = h(x_1,x_2,x_3)\\
& =\sigma(w_{46}\times\sigma(w_{14}x_1+w_{24}x_2+w_{34}x_3+b_4)+w_{56}\times\sigma(w_{15}x_1+w_{25}x_2+w_{35}x_3+b_5)+b_6)\end{align}$
where, $\theta = \{w_{14},w_{24},w_{34},b_4,w_{15},w_{25},w_{35},b_5,w_{46},w_{56},b_6\}$ is the set of the respective coefficients we have to determine such that we minimize:
$$J(\theta) = \frac{1}{2}\sum_{i=1}^N (y^{(i)}-h(x^{(i)}))^2$$
The above optimization problem can be easily solved with gradient descent. Just initiate $\theta$ with random values and with proper learning parameter $\eta$, update as follows till convergence:
$$\theta:=\theta-\eta\frac{\partial J}{\partial \theta}$$
In order to get the gradients, we express the above function as a neural network as follows:

Let's calculate the gradient, say w.r.t. $w_{14}$.
$$\frac{\partial J}{\partial w_{14}} = \sum_{i=1}^N \Big[\big(h(x^{(i)})-y^{(i)}\big)\frac{\partial h(x^{(i)})}{\partial w_{14}}\Big]$$
Let $p(x) = w_{14}x_1+w_{24}x_2+w_{34}x_3+b_4$ , and
Let $q(x) = w_{46}\times\sigma(p(x))+w_{56}\times\sigma(w_{15}x_1+w_{25}x_2+w_{35}x_3+b_5)+b_6)$
$\therefore \frac{\partial h(x)}{\partial w_{14}} = \frac{\partial h(x)}{\partial q(x)}\times\frac{\partial q(x)}{\partial p(x)}\times\frac{\partial p(x)}{\partial w_{14}} = \frac{\partial\sigma(q(x))}{\partial q(x)}\times\frac{\partial\sigma(p(x))}{\partial p(x)}\times\frac{\partial p(x)}{\partial w_{14}}$
We see that the derivative of the activation function is important for getting the gradients and so for the learning of the neural network. A constant derivative will not help in the gradient descent and we won't be able to learn the optimal parameters.

If what you are asking is what is the intuition for using the derivative in backpropagation learning, instead of an in-depth mathematical explanation:
Recall that the derivative tells you a function's sensitivity to change with respect to a change in its input. A high (absolute) value for the derivative at a certain point means that the function is very steep, and a small change in input may result in a drastic change in its output; conversely, a low absolute value means little change, so not steep at all, with the extreme case that the function is constant when the derivative is zero.
Training a neural network essentially amounts to an optimization problem where one wants to minimize a certain value, in this case the error produced by the network on the given training examples. Backpropagation learning can be viewed as a case of gradient descent (the inverse of hill climbing).
If for a moment we assume that your input is only 2-dimensional (just for illustration, the mathematics of course also work for higher dimensions), you could imagine the error function as a landscape with hills, mountains, valleys, ridges etc. You are standing at a high point and want to get down as far as possible. Gradient descent means that, in discrete steps, you always walk down in the direction that has the steepest slope downwards from where you are currently standing, until you eventually reach a (local) minimum.
In order to determine where that steepest slope is, you need the derivative of the activation function. Basically, you want to sort out how much each unit in your network contributes to an error, and adjust in the direction that contributes the most.
Edit: Regarding constant values for a derivative, in the landscape metaphor it would mean that the gradient is the same no matter where you are, so you'll always go in the same direction and never reach an optimum. However, multi-layer networks with linear activation function are kind of besides the point anyhow when you consider that each cell computes a linear combination of its inputs, which then is again a linear function, so the output of the last layer will ultimately be a linear function of the inputs at the first layer. That is to say, anything you can do with a multi-layer net with linear activation functions, you could also achieve with just a single layer.

Thanks. It was a good starter explanation. I understand we want to minimize the whole loss function. But why we need a local minimum at each function?!

@Tina J: I am not sure what you are asking. You are correct that we try to find a single minimum for the error of the entire network. What backpropagation does is to split the observed error up into the parts contributed by each single unit and connection. So we don't minimize at each single unit, but for each training example, we (potentially) adjust every edge's weight, depending on how much it affected the outcome to be wrong. Each weight is a dimension of the "landscape", and one traversal of the net is a single step in gradient descent, which is repeated until reaching convergence.

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.
In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used, given that the MSE is a function of the ANN, which is a composite function of multiple non-linear functions, the activation functions, whose main purpose is thus to introduce non-linearity, or, in other words, it makes the ANN powerful. Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights: to fully see this, you will need to learn the details of back-propagation! Hence the importance of the derivatives of the activation functions.
A constant derivative would always give the same learning signal, independently of the error, but this is not desirable.
To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!

Comments are not for extended discussion; this conversation has been moved to chat.


text:
What are some conferences for publishing papers on Deep Learning for Human Activity recognition?
What are some conferences for publishing papers on Deep Learning for Human Activity recognition? Do any of the major conferences have specific tracks for Human Activity Recognition?

If your using computer vision, the top recognised conference is CVPR (computer vision and pattern recognition).
You can also try to submit at ICML (International conference on machine learning) and NIPS (Neural Information Processing Systems), which focuses on applications of machine learning and deep learning.
I'd also recommend IJCAI (International joint conference on artificial intelligence).


text:
What could an oscillating training loss curve represent?

I tried to create a simple model that receives an $80 \times 130$ pixel image. I only had 35 images and 10 test images. I trained this model for a binary classification task. The architecture of the model is described below.
conv2d_1 (Conv2D)            (None, 80, 130, 64)       640
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 78, 128, 64)       36928
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 39, 64, 64)        0
_________________________________________________________________
dropout_1 (Dropout)          (None, 39, 64, 64)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 39, 64, 128)       73856
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 37, 62, 128)       147584
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 18, 31, 128)       0
_________________________________________________________________
dropout_2 (Dropout)          (None, 18, 31, 128)       0
_________________________________________________________________
flatten_1 (Flatten)          (None, 71424)             0
_________________________________________________________________
dense_1 (Dense)              (None, 512)               36569600
_________________________________________________________________
dropout_3 (Dropout)          (None, 512)               0
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 513

What could the oscillating training loss curve represent above? Why is the validation loss constant?

what loss function you using and this is ABSURDLY large for the amount of data you have. Your models gonna have real difficulty

Overview
As it has already been observed, your main problem, beside the training related issues like fixing the learning rate, is you have basically no chance to learn such a big model woth such a small dataset ... from scratch
So focusing on the real problem, here are some techniques you could use

dataset augmentation
transfer learning

from a pretrained model
from the encoder stage of an autoencoder (last resort option before getting into more advanced topics)

Dataset Augmentation
Add transformations to your dataset you want your classifier to learn to be invariant to
Let's assume that

$I$ is an input image
$l$ its associated label
$f(\mathcal{I};\theta) \rightarrow \mathcal{I}$ is a parametric transformation that affects appearance but not semantic, for example it is a rotation of $\theta$ angle

then you can augment your dataset by generating $\{I_{\theta}, l\}$ a set of transformed (e.g. rotated) images associated the same $l$ label
Transfer Learning
The fundamental idea of transfer learning is to re-use a NN which has been trained to solve a task, to solve other tasks retraining only a selected subset of the weights
It means using a pre-trained convolutive backend, the part of the model with Conv2D and Pooling, and train dense layers with dropout only (but you should still probably think about reducing the dimensionality there)
More formally think about representing your CNN Classifier as follows

$f_{C}(I; \theta_{X})$ : Convolutive Processing on Input Image

it is the part of the CNN composed of Conv2D and MaxPooling2D layers
the $\theta_{C}$ is the convolutive learnable weights set

$b = f_{C}(I; \theta_{C})$ : Bottleneck Feature Representation

it is the result of Flatten layer

$f_{D}(b; \theta_{D})$ : Dense Processing

it is the part of the model composed of Dense layers
the $\theta_{D}$ is the dense learnable weights set

The idea is to pick $\theta_{C}$ from a training performed on an another dataset, bigger than your current one, and keep it fixed while training in your task
This means reducing the number of parameters to be trained, however beware the dense layers account for most of the weights, as you can also see from your mode summary, which means you should also focus on reducing that number, for example reducing the bottleneck feature tensor size
Transfer Learning from Pre-Trained Model
For example, if your actual goal was to perform binary classification on some kind of MNIST-like data then you could use a convolutive backend from a CNN which has been pre-trained on the MNIST 0..9 classification task or you can train it yourself but is important is the $\theta_{C}$ weights will be learned from a MNIST dataset, which is much bigger than yours, even if the task is (slightly) different.
Furthermore, in case of MNIST like data, please consider if you really need your full 80 x 130 resolution hence your input tensor, considering I can deduct from your model summary it is grayscale (no color), needs to be $(80,130,1)$ or you could rescale to the 28 x 28 MNIST resolution so you work with a smaller $(28,28,1)$ tensor
My suggestion is to start from an architecture like this MNIST Keras Model as

it has a bottleneck representation of 64 which could be enough for your task and
also suggesting to remove the first dense layer so to significantly reduce $\theta_{D}$ the number of learnable paramters hence going for something like

  model = Sequential()
  # add Convolutional layers
  model.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same', input_shape=(10, 10, 1)))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(Flatten())
  # output layer
  model.add(Dense(1, activation='sigmoid'))

then compile the model with binary_crossentropy loss and maybe start giving a try to adam optimizer
Transfer Learning from Autoencoder
If your data is so special you can't find any big enough and similar enough dataset to use this strategy and you do not come up with any transformation you could use
to perform dataset augmentation, without getting into advanced things, you could try to play one last card: use an Autoencoder to learn a compressed representation aimed at reconstructing the original image and perform transfer learning with the encoder only
For example, again under the assumption of working with a $(28,28,1)$ tensor, you could start with an architecture like the following one
def build_ae(input_img):
  x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
  # (28,28,16)

  encoded = MaxPooling2D((8, 8), padding='same')(x)
  # (4,4,8)

  x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
  # (4,4,8)

  x = UpSampling2D((8, 8))(x)
  # (16,16,8)

  x = Conv2D(16, (3, 3), activation='relu')(x)
  # Note: Convolving without padding='same' in order to get w-2 and h-2 dimensioality reduction so that following upsampling can lead to the desired 28x28 spatial resolution
  # (14,14,8)

  x = UpSampling2D((2, 2))(x)
  # (28,28,8)

  decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
  autoencoder = Model(input_img, decoded)
  return autoencoder

In this case, the full model has 2633 weights but the encoding stage consists only of Conv2D+Relu+MaxPooling which means in total 3x3x1x16 weights for the convolutive step and 16 weights for the relu for a total of 160 weights only and the latent representation is a $(4,4,8)$ tensor which means a 128 dimensional flattened tensor and hence assuming, as before, to perform the binary classification with a dense sigmoid layer it would mean 128+1 weights to learn in the actual binary classification task
Of course it is possible to go for an even more compressed latent representation both on the spatial domain or channel domain with consequent reduced flattened vector dimensionality and ultimately even less weights to learn
Would you share more details about your problem, also your dataset, we could try to help more

Try lowering the learning rate.
Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss the local minima. Then it will try to come back to the minima in the next step and overshoot it again.
You may also try switching to a momentum-based GD algorithm. Such a training loss curve can be indicative of a loss contour like in this example, for which momentum-based GD methods are helpful.
I noticed that you have a very small training set. You may have better luck with a larger training data (~1000 examples) or using a pre-trained Conv network as a starting point.

Lowering Learning rate did not help(0.1, 0.01, or 0.005 or even 0.001)

Nicola Bernini's answer is quite comprehensive. Here are my insights.
First of all, think whether you really need neural networks to solve your problem. Think whether traditional computer vision operations like edge detection/ region-based methods help you to solve your problem (OpenCV can help you here). Think about your data again. In case you decide to use neural networks, some things to try out:-

Your data set size is too small. Try to recall that we are learning to approximate functions (Universal approximation theorem). Less data + More parameters have high chance of overfitting. Use transfer learning. (Try Resizing the image and perform random resized square crop and use as input to your neural network. This may or may not work since I don't know what exactly you are doing). Also try data augmentations that make sense (i.e. : Vertical flip of traffic sign images don't work)
Try reducing the learning rate/ using different learning rates for different parts of your network if you decide to use transfer learning.
Check whether your train and test dataset distributions are the same. i.e.: Don't train with 95 % of label 0 and test with a set that has 95% label 1. I do not know whether your dataset is highly class imbalanced or whether you are doing some kind of anomaly detection.
Think about optimizers. Try Adam if you haven't.

If you need more help, try sharing the data and we can try to help you.

From my experience, this oscillation comes from:

Too high learning rate: Weights change too quick.
Too few neurons in layers: Not enough to fit.

When having not enough neurons, the network can't learn at all and the oscillation is due to failure to fit to global optimum; it's correct for some cases and wrong on other cases


text:
Is it possible to control asymptotic behaviour of neural network models?
Is it possible to specify what the asymptotic behaviour of a Neural Networks (NN) model should be?
I am thinking of a NN which tries to learn a mapping $\vec y=f(\vec x)$ with $\vec x$ a vector of features of dimension $d$ and $\vec y$ a vector of outputs of dimension $p$.
Is it possible to specify that, for instance, the NN should have a fixed value when $x_1$ goes to infinite?
I mean:
$$
\lim_{x_1\to \infty} f(\vec x) = \vec c
$$
If it is not possible with NN, do you know other machine learning models (for instance  Gaussian Process Regression or Support Vector Regression) which have a known asymptotic behaviour?

to gaurantee you would have to do some analysis on your network znd define the conditions of your network explicitly, but if you are okay with an approximation, you can define an auxillary loss $h(f(x,x_1=\alpha) - f(x,x_1=g(\alpha)))$ to try to make it fall in some form of algorithmic patter that will follow your needs

A simpler answer is that for a standard neural net, the asymptotic behaviour is the asymptotic behaviour of the output neurons. For example, if the output layer is ReLUs, then the asymptotic behaviour is necessarily linear.
In your case, since you want it to be asymptotically constant, you can use the slightly old-fashioned choice of sigmoid units in the output layer.
The training set is necessarily finite, so no machine learning method can learn asymptotic behaviour. It can only be supplied by prior knowledge, for example as I describe.

For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.
Neural Networks with Asymptotics Control discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, allowing explicit control over extrapolation. This new type of asymptotics-controlled Neural network is based on two novel technical constructs:

A Multi-Dimensional Spline Interpolator with Prescribed Asymptotic Behavior
and A Custom NN layer that guarantees zero asymptotics in chosen directions.

Let $f(x)$  be the Function that we want to approximate while preserving its asymptotics. (It goes without saying that the asymptotics needs to be known.) As the first step, we find a control variate function $S(x)$ that has the same asymptotics as $f(x)$.

Multi-Dimensional Inputs - the most interesting case - these asymptotics could either be known in all directions or only in some; naturally, as much information about the asymptotics as is available should be incorporated into the control variate function $S(x)$.

For step one, they show how to construct a universal control variate, a multi-dimensional spline $S(x)$ that has the same asymptotics as $f(x)$; and that can be used in all situations. In specific applications, of course, if more is known about the function $f$; a better choice of the control variate could be available.

For step two, we design a custom NN layer that guarantees zero asymptotics in all directions, with fine control over the regions where the NN interpolation is used and where the asymptotics kick in. they approximate the residual function $R(x) = S(x) - f(x)$ with a special NN that has vanishing (zero) asymptotics in all, or some, directions.

One-Dimensional Spline with Asymptotics Control:
The paper discussed that Instead of using the natural boundary conditions, i.e., setting second derivatives
to zero at the boundaries, they have fixed the first derivatives at boundary points to
arbitrary values. $\color{blue}{\text{Clearly, this paves the way to control asymptotics}}$.
Suppose that we know the behavior of the original, calculation-heavy function in its tails, i.e. $f(x) \simeq f_{−}(x)$ for $x<h_0$ and $f(x) \simeq f_{+}(x)$  for $x>h_{N+1}$
for large negative $h_0$ and large positive $h_{N+1}$. First, we
expand the set of spline nodes to include points $h_0$ and $h_{N+1}$ and make sure
the spline passes through them, $S(h_0) = f_{−}(h_0)$ and $S(h_{N+1}) = f_{+}(h_{N+1})$. To
finish, they have specifies first order derivatives at these new boundary points.
$$S^{\prime}_0(h_0) = f^{\prime}_{−}(h_0), S(h_{N+1}) = f^{\prime}_{+}(h_{N+1})$$
This fully specifies the approximating function $\tilde{S}(x)$,
$$\tilde{S}(x)=\left\{\begin{array}{ll}f_{-}(x), & x \leq h_{0} \\ S(x), & h_{0}<x<h_{N+1} \\ f_{+}(x), & h_{N+1} \leq x\end{array}\right.$$
One-Dimensional Spline with Asymptotics Control: Continuous Second Derivatives
where they have discussed how they have constructed a control variate function based on the spline like above, but with the continuous second derivative at the endpoints. In order to do that, they have picked a point between $h_0$ and $h_1$, say $h_{\frac{1}{2}}$ , and find, analytically, the value $S(h_{\frac{1}{2}})$ such that the second derivative at $h_{0}, S^{\prime \prime}\left(h_{0}\right)= f_{-}^{\prime \prime}\left(h_{0}\right)$ is matched.

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems. This paper deals with studying the asymptotical properties of multilayer neural networks models used for the adaptive identification of the wide class of nonlinearly parameterized systems in a stochastic environment.

A study of the asymptotic behavior of neural networks. where they have studied neural networks modeled as a set of nonlinear differential equations of the form $TX+X=Wf(X)+b$, where $X$ is the neural membrane potential vector, $W$ is the network connectivity matrix, and $F(X)$ is the nonlinearity (an essentially sigmoid function). Topologies of neural networks that exhibit asymptotic behavior are established as behavior depends solely on the topology of the network. Moreover, the connectivity $W$ need not be symmetric. The simulated behavior of typical neural networks is presented

Refrence
Neural Networks with Asymptotics Control
Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems
A study of the asymptotic behavior of neural networks


text:
DDPG: how to implement continuous action space bounded in the interval [-2, 2]?
I am a newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by $[-2, 2]$. My research shows that doing nothing is a possible solution (i.e. action of 4.5 is mapped to 2 and the action of -3.1 is mapped to -2), but I wonder if there are more elegant approaches.


text:
What is the difference between graph convolution in the spatial vs spectral domain?
I've been reading different papers regarding graph convolution and it seems that they come into two flavors: spatial and spectral. From what I can see the main difference between the two approaches is that for spatial you're directly multiplying the adjacency matrix with the signal whereas for the spectral version you're using the Laplacian matrix.
Am I missing something, or are there any other differences that I am not aware of?

Spectral Convolution
In a spectral graph convolution, we perform an Eigen decomposition of the Laplacian Matrix of the graph. This Eigen decomposition helps us in understanding the underlying structure of the graph with which we can identify clusters/sub-groups of this graph. This is done in the Fourier space.
An analogy is PCA where we understand the spread of the data by performing an Eigen Decomposition of the feature matrix. The only difference between these two methods is with respect to the Eigen values. Smaller Eigen values explain the structure of the data better in Spectral Convolution whereas it's the opposite in PCA.
ChebNet, GCN are some commonly used Deep learning architectures that use Spectral Convolution
Spatial Convolution
Spatial Convolution works on local neighbourhood of nodes and understands the properties of a node based on its k local neighbours. Unlike Spectral Convolution which takes a lot of time to compute, Spatial Convolutions are simple and have produced state of the art results on graph classification tasks. GraphSage is a good example for Spatial Convolution.
Additional References: https://towardsdatascience.com/tutorial-on-graph-neural-networks-for-computer-vision-and-beyond-part-2-be6d71d70f49
https://towardsdatascience.com/graph-convolutional-networks-for-geometric-deep-learning-1faf17dee008

The Graph Convolutional Neural Network (GCN) is considered to be a "spatial" approach. The author has mentioned this on his own website. And you also said it: "works on local neighbourhood of nodes" ... I think you should correct this.

GCN somehow bridged the gap between 'spatial' and 'spectral' methods, since it can be considered a first order approximation of ChebNet

What are the pros and cons of these two approaches?

After I read multiple explanations from different sources I think I found the main difference between the two methods. Implementation wise the only difference is the matrix that you're multiplying the signal with (Laplacian/adjacency matrix). But by using the Laplacian, you're encoding the graph structure (in-out degree of each node) which dictates how a signal should "diffuse" in the network.


text:
What is the cognitive architecture with the highest IIT measure?
What contemporary information system or cognitive architecture is the one with the highest measure of the Integrated Information Theory (IIT) (that is, a theory of consciousness, which states that a system's consciousness is determined by its causal properties and is, therefore, an intrinsic, fundamental property of any physical system)
Is there race/competitions to develop (or allow autonomous development) the system with the maximum IIT measure?


text:
What are the advantages of Machine Learning compared to traditional programming for developing a chatbot?
I am currently building a chatbot. What I have done so far is, collected possible questions/training data/files and create a model out of it using Apache OpenNLP; the model is able to predict all the questions that are in the training data and fails to predict for new questions.
Instead of doing all the above, I can write a program that matches the question/words against training data and predict the answer — what is the advantage of using Machine Learning algorithms?
I have searched extensively about this and all I got was, in Machine Learning there is no need to change the algorithm and the only change would be in the training data, but that is the case with programming too: the change will be in training data.

In my view ML does not work very well for conversational AI systems. It is generally alright for intent recognition, so getting what the user wants if they ask a question ("I want to book a flight?", "What is the weather in London?"), but anything after that quickly becomes difficult to handle, especially multi-step conversations that go beyond simple question/answer pairs.
My suggestion would be to plan possible dialogues out as flow charts (more like trees/graphs, as there can be multiple branches at any point), and then write a program that interprets the graph based on user input and gives appropriate replies. You will also want to have some conversational memory to keep track of any information the user has mentioned. That is also tricky to do in a ML system.
For a very simple framework to start off with, have a look at ELIZA. It's half a century old, but you can still use it as a starting point.
(Disclaimer: I work for a company that makes conversational AI systems)

With all the work with NMT models, I think weve gotten pretty far! (ofcourse theres alot more to do, but i dont think its worth overlooking)

I am actually looking to build intent finder, but both ML and programming seems to be same in that context

Sebastian Thrun in one of his online interviews once suggested that he thought that the conversational solution was a combination of both massive machine learning and rules based programming.
The problem in the case of chats is that our expectations are very high which dooms early solutions to failure. For the ML side we require large amounts of data, and while large amounts may be available they are highly biased and unbalanced; they mostly focus on one area (specialized context) and re-use the same sentence formulas over and over again, so the learning finds a comfortable corner case solution and refuses to learn anything else. People are so predictable so they are an unhelpful source of raw data.
One approach might be to use carefully constructed rules to generate the data that ML can learn from, rules that can guarantee broad contextual applicability and sentence construction variation.


text:
What is the difference between Squeeze-and-excite and bottleneck modules from Mobilenet v2?
Squezee-and-excite networks introduced SE blocks, while MobileNet v2 introduced linear bottlenecks.
What is the effective difference between these two concepts?
Is it only implementation (depth-wise convolution, vs per-channel pooling), or they serve a different purpose?
My understanding is that both approaches are used as attention mechanism, working per-channel. In other words, both approaches are used to filter unnecessary information (information that we consider noise, not signal).
Is this correct?
Do bottlenecks ensure, that the same feature won't be represented multiple times in different channels, or they don't help at all in this regard?


text:
How to select the most appropriate set of actions for a given environment or task?
Given a robot in a situation such as in a library reading a book.
Now I want to create a neural network that suggests an appropriate action in this situation. And, generally, ignore actions such as "get up and dance" and so on.
Since there are limitless actions a robot could do, I need to narrow it down to the ones in this situation. Using its vision system, the word "book" and book neurons should already be activated as well as "reading".
One idea I had was to create an adversarial network which generates words (sequences of letters) based on the situation such as "turn page", "read next line" and so on. And then have another neural network which translates these words into actions. (It would them simulate whether this was a good idea. If not it would somehow suppress the first word and try to generate a new word.)
Another example is the robot is in a maze and gets to a crossroads. The network would generate the word "turn left" and "turn right".
Another idea would be to have the actions be composed of a body part e.g. "eyes" and a movement such as "move left" and it would combine these to suggest actions.
Either way, it seems like I need a way to encode actions so that the robot doesn't consider every possible action in the universe.
Is there any research in this area or ideas on how to achieve this?
(I think this may be somewhat related to the task of "try to name as many animals as you can.")

One idea I had was to create an adversarial network which generates words

It may be better to use for e.g. numbers instead of words, then map these numbers to the language of your choice like English in your case. This mapping can be changed to other languages and you will have a robot that overcomes language barriers.


text:
What are the differences between Bytenet and Wavenet?
I recently read Bytenet and Wavenet and I was curious why the first model is not as popular as the second. From my understanding, Bytenet can be seen as a seq2seq model where the encoder and the decoder are similar to Wavenet. Following the trends from NLP where seq2seq models seem to perform better, I find it strange that I couldn't find any paper that compares the two. Are there any drawbacks of Bytenet over Wavenet other than the computation time?

Have you seen this thread https://www.reddit.com/r/MachineLearning/comments/ai4vro/d_anyone_here_use_bytenet_like_architecture_for/ ?

David Pollack mentions some characteristics of the two models in his master's thesis: "Musical Genre Classification of Audio" at https://edoc.hu-berlin.de/bitstream/handle/18452/20012/master_pollack_david.pdf?sequence=7

My conclusion is the same as yours that there doesn't seem to be any published comparison of the two models.  ByteNet is computationally expensive and requires a lot of parameters.  WaveNet improves on ByteNet's efficiency, as you mentioned, and I believe that is the main difference.


text:
Could the Jensen-Shannon divergence and Kullback-Leibler divergence be used as loss functions of non-generation problems?
If I understand correctly, the KL divergence is a measure of information loss between a ground truth distribution $P$ and a predicted distribution $Q$, and the Jensen-Shannon divergence is the mean of the KL Divergences of 2 cases

Predicted distribution is mean of $P$ and $Q$, and ground truth is $P$
Predicted distribution is mean of $P$ and $Q$, and ground truth is $Q$

Since KL divergence can be easily interpreted as information loss in $Q$ relative to $P$, what can JS divergence interpret-ably represent? I cannot see any use cases of these measures unless there are two distributions to compare. Is there any other problem where I could use them as loss functions other than generation problems? If so how, and why?

this is a duplicate of https://stats.stackexchange.com/questions/117225/jensen-shannon-divergence-vs-kullback-leibler-divergence, if you look there i hope it helps

@mshlis Why should this be a duplicate of that?


text:
What is the role of the 'fuzzifier' w in Fuzzy Clustering?
According to my lecture, Fuzzy c-Means tries to minimize the following objective function:
$$J(X,B,U)=\sum_{i=1}^c\sum_{j=1}^n u_{ij}^w \, d^2(\vec{\beta_i},\vec{x_j})$$
where $X$ are the data points, $B$ are the cluster-'prototypes', and $U$ is the matrix containing the fuzzy membership degrees. $d$ is a distance measure.
A constraint is that the membership degrees for a single datapoint w.r.t. all clusters sum to $1$: $\sum_{j=1}^n\, u_{ij}=1$.
Now in the first equation, what is the role of the $w$? I read that one could use any convex function instead of $(\cdot)^w$. But why use anything at all. Why don't we just use the membership degrees? My lecture says using the fuzzifier is necessary but doesn't explain why.

Its not required, you can have $m=1$, actually it can be any number $\geq 1$.
Now the better question is why to have it? The answer is that it adds a smoothing effect. Lets look at it in each of the limits ($\lim m \rightarrow 1$ and $\lim m \rightarrow \infty$)
Towards $\infty$, it makes $u_{ij}$ equal to $\frac{1}{c}$, making each point have equal membership of each class regardless of $m$. From the optimization perspective, its saying how can we achieve finding clusters that are closest to all points, therefore by definition it has already achieved that, and so the Loss will always be 0. (at its global minimum)
Now in the other limit, the constants are inversely proportional to the square of the normalized euclidean distance. This makes intuitive sense, the membership is high if they are close, and the membership is low if they are not (relatively)
So why do we have the $m$, its for control. It allows us to choose and experiment with how heavy each distance should hold weight in the membership. An example where a larger $m$ may be useful is when the data isnt clean, and you dont want to rely so heavily on euclidean distance as the membership, so you forcibly add in a smoothing effect

Assume I have two clusters, then for a point $x_1$ the $u_{ij}$ must sum to $1$. So I could have for example $u_{11}=0.8$ and $u_{12}=0.2$. If I now let $m\rightarrow\infty$, then $0.8^\infty=0$ and $0.2^\infty=0$, not $\frac{1}{2}$. How did you come up with the $\frac{1}{C}$?

The u’s are a function of m as well

I don't understand. It's still $0.8^m \rightarrow 0$ and not $0.8^m\rightarrow\frac{1}{2}$ for $m\rightarrow\infty$

the value is still 0, the membership though is $\frac{1}{2}$... If $m \rightarrow \infty$ the problem is meaningless because the membership is no longer a function of any of the cetroids but is gauranteed to be equally associated with each

i adjusted my answer slightly to reflect that


text:
Should I model a problem with quantised output as classification or regression?
Say I have some data I am trying to learn, and I'm aware that the output is quantised in some way, e.g. I can get only get discrete values (0.1, 0.2, 0.3...0.9) in a finite range.
Would you treat that as regression or classification? In this case the numbers do have a relation to each other e.g. 0.3 is close to 0.4 in meaning.
I could treat it as classification with a softmax final layer with N outputs, or could treat it as regression with a linear layer with single output and then somehow quantise the result post-prediction. But my gut feeling is that the fact there is a finite number of answers that that should somehow be used in my model?

So this is considered Ordinal Regression. There are many ways to model this type of data, generally in some form of regression setting. I do not recommend the softmax route because as you mentioned, there exists prebuilt correlation to the outputs.
Some common ways to approach this
(Note that im assuming your looking for methodologies that can be optimized through gradient techniques because of the way you formulated your question)

Treat it as a normal regressor where you clip the output to your range, and then define thresholds arbitrarily, such as $.36 \rightarrow .4$, $.34 \rightarrow .3$, etc..
Use a bounded activation function then scale (if your outputs are [0,1,.1], you could use sigmoid and just scale by a factor of 1, but if [0,10,1], you could use sigmoid and scale by 10. Once again youll need to create arbitrary threshold between each 2 points for inference (this can also be incorporated into your loss)
Using the above two methods, but learn the threshold between each 2 ordered points. Have an output layer that learns the ideal threshold for inference

And if you didnt want to use neural netowrks, each of these approaches have a bayesian analog that work as well!