text:
Is there any flaw to this solution to the One shot prisoner's dilemma
I wrote a solution to the one shot Prisoner's dilemma:

Introduction
My solution applies to a prisoner dilemma involving two people (I have neither sufficient knowledge of the prisoner's dilemma itself, nor sufficient mathematical aptitude/competence to generalise my solution to prisoner's dilemma where number of agents (n) > 2).
Let the two agents involved be A and B, assume A and B are maximally selfish (they care solely about maximising their payoff). If A and B satisfy the following 3 requirements, then whenever A and B are in a prisoner's dilemma together, they would choose to cooperate.
1. A and B are perfectly rational.
2. A and B are sufficiently intelligent, such that they can both simulate each other (the simulations doesn't have to be perfect; it only needs to sufficiently resemble the real agent that it can be used to predict the choice of the real agent).
3. A and B are aware of the above 2 points.
Solution

A and B have the same preference.
(A,B) = (D,C) > (C,C) > (D,D) > (C,D).
(B,A) = (D,C) > (C,C) > (D,D) > (C,D).
My solution relies on A and B predicting each other's behaviour. They use a simulation of the other which guarantees high fidelity predictions.
If A adopts A defect invariant strategy (always defect) i.e committing to defection, then B will simulate this, and being rational B will defect.
Vice versa.
If A adopts a cooperate invariant strategy, then B will simulate this and being rational, B will defect.
Vice versa.
To commit to a choice means to decide to adopt that choice irrespective of all other information. Committing means not taking into account any other information in deciding your choice.
Assuming A commits to a choice. Let the choice A commits to be k.
Then B's strategy would be:

Diagram to Illustrate
h(x) is a function that takes in B's prediction of A's choice as input and outputs the rational response to that choice. As B prefers (D,C) over (C,C) and (D,D) over (C,D), then B would choose to defect whatever choice it predicts for A. Thus, adopting an invariant strategy will cause B to adopt the defect invariant strategy (insofar as B accurately predicts A). Being rational, and preferring (D,C) and (C,C) over (D,D) A would not adopt an invariant strategy.
Vice Versa.
If both of them were to decide to simultaneously adopt invariant strategies, they would adopt the defect invariant strategy. However, as they both prefer (D,C) and (C,C) (in that order) over (D,D) they would both strive to get a better outcome than (D,D) unless no better outcome is possible.
This means that A and B's choices depends on what they predict the other would do.
If A predicts B will defect, A can cooperate or defect.
If A predicts B will cooperate, A can cooperate or defect.
Vice versa.
As A is not committing, A's strategy is either predict(B) (picks what A predicts B will pick) or !predict(B) (picks the opposite of what A predicts B will pick. I.e cooperate if A predicts B will defect, and defect if A predicts B will cooperate).
Vice versa.
To not commit is to decide to base your strategy on your prediction of the choice the opponent adopts. You can either choose the same choice as what you predicted, or choose the opposite of what you predicted (any other choice is adopting an invariant strategy).
If A adopts !predict(B).  A gains an outcome ranked 1 or ranked 4 in its preferences.
If A adopts predict(B)  A gains an outcome ranked 2 or 3 in its preferences.
Vice versa.
We can have:

predict(B) and predict(A)
predict(B) and !predict(A)
!predict(B) and predict(A)
!predict(B) and !predict(A).

Now A's decision is dependent on predict(B).
But B's decision (and thus predict(B)) is dependent on predict(A) (and thus A's decision).
A = f(predict(B) = g(B = f(predict(A) = g(A)) ) ).
f(x) is a function that deterministically returns either x or !x (according to the strategy adopted by the agent implementing it).
g(x) is a function the stochastically return x or !x it returns x with a probability of p. Given that the agents are simulating each other, we can safely assume that p is high (sufficiently close to 1).
Vice versa.
The above assignment is circular and self-referential. If A and/or B tried to simulate it, it leads to a non terminating recursion of the simulations.
A's strategy would be:

Diagram to Illustrate
As such, A and B cannot both decide to base their decision on the decision of the other.
Yet, neither A nor B can decide to commit to an option.
What they can do, is to predispose themselves to an option. To predispose themselves is to decide on their choice independent of the other agent. Assuming I received no further information, what would I do? A predisposition is not a final choice, and should not be mistaking for committing to a choice of action. An agent who predisposes themselves can change their choice based on how they predict the other agent would react to that predisposition.
Assuming A predisposes themselves. Let the predisposition made be q. The assignment becomes:
A = f(predict(B) = g(B = f(predict(A) = g(q)) ) ).

Diagram to Illustrate
Only one of them needs to predispose themselves.
Assuming A predisposes themselves.
If A predisposes themselves to defection, then the only two outcomes are ranked 1 and 3 in their preferences (for A) and 3 and 4 in their preferences (for B).
Upon simulating this, B being rational would choose to defect (resulting in outcome 3). (D,D) is a Nash equilibrium, and as such once they reach there, being rational neither A nor B would change their strategy.
(Note that this outlaws the !predict(A) strategy.
If A predisposes themselves to cooperation, then the two possible outcomes are ranked 2 and 4 in their preferences (for A) and 1 and 2 in their preferences (for B).
Upon simulating this, if B chooses to defect, then B is adopting a defect invariant strategy (which has been outlawed), and A will update and choose to defect, resulting in outcome 3. As B is rational, B will choose the outcome that leads to outcome 2, and B will decide to cooperate.
If B chooses to defect, and A simulates B choosing to defect if A predisposes themselves to cooperation, then A will update and defect, resulting in (D,D). If B chooses to cooperate if A predisposes themselves to cooperation, if A updates and chooses to defect, then B would update and choose to defect resulting in D,D. Thus, once they reach (C,C) they are at a reflective equilibrium (in the sense that if one defects, then the other would also defect, thus no one of them can increase their payoff by changing strategy).
(Thus, B will adopt a predict(A) strategy).
Vice Versa.
Because A is rational, and predisposing to cooperation dominates predisposing to defection (the outcomes outlawed are assumed not to manifest), if A predisposes themself, then A will predispose themself to cooperation.
Vice Versa.
Thus if one agent predisposes theirself, it will be to cooperation, and the resulting outcome would be (C, C) which is ranked second in their preferences.
What if A and B both predispose themselves?
We can have:
1. C & C
2. C & D
3. D & C
4. D & D.
If C & C occurs, the duo will naturally cooperate resulting in (C, C). Remember that we showed above that the strategy adopted is predict(B) (defecting from (C, C) results in D, D).
If C & D occurs, then A being rational will update on B's predisposition to defection and choose defect resulting in (D, D).
If D & C occurs, then B being rational will update on A's predisposition to defection and choose defect resulting in (D, D).
If D & D occurs, the duo will naturally defect resulting in (D, D).
Thus seeing as only predisposition to cooperation yields the best result, at least one of the duo will predispose to cooperation (and the other will either predispose themselves to cooperation or not predispose at all) the resulting outcome is (C, C).
If the two agents can predict each other with sufficient fidelity (explicit simulation is not necessary, only high fidelity predictions are) and are rational, and know of those two facts, then when they engage in the prisoner's dilemma, the outcome is (C, C).
Therefore, a cooperate-cooperate equilibrum can be achieved in a single instance prisoner's dilemma involving two rational agents given that they can predict each other with sufficient fidelity, and know of their rationality and intelligence.
Q.E.D
Thus if two super intelligences faced off against each other in the prisoner's dilemma, they would reach a cooperate-cooperate equilibrium. This solution also applies to two rational robots who have mutual access to the other's source code.

Prisoner's Dilemma With Human Players
In the above section I outlined a strategy to resolve the prisoner's dilemma for two superintelligent AIs or rational bots with mutual access to the other's source code. The strategy is also applicable to humans who know each other well enough to simulate how the other would act in a given scenario. In this section I try to devise a strategy applicable to human players.
Consider two perfectly rational human agents A and B. A and B are maximally selfish and care only about maximising their payoff.
Let:
(D,C) = W
(C,C) = X
(D,D) = Y
(C,D) = Z
The preference is W > X > Y > Z.

A and B have the same preference.
The 3 conditions necessary for the resolution of the prisoner's dilemma in the case of human players are:
1. A and B are perfectly rational.
2. They each know the other's preference.
3. They are aware of the above two facts.
The resolution of the problem in the case of superintelligent AIs relied on their ability to simulate (generate high fidelity predictions of) each other. If the above 3 conditions are met, then A and B can both predict the other with high fidelity.
Consider the problem from A's point of view. B is as rational as A, and A knows B's preference. Thus, to simulate B, A merely needs to simulate themselves with B's preferences. Since A and B are perfectly rational, whatever conclusion A with B's preferences (A*) reaches is the same conclusion B reaches. Thus A* is a high fidelity prediction of B.
Vice versa.
A engages in a prisoner's dilemma with A*. However, as A* as the same preferences as A, A is basically engaging in a prisoner's dilemma with A.
Vice Versa.

An invariant strategy is outlawed by the same logic as in the AI section.
A = f(predict(A*) = g(A* = f(predict(A) = g(A)))
Vice Versa.
The above assignment is self referential, and if it was run as a simulation, there would be an infinite recursion.

Diagram to Illustrate
Thus, either A or A* needs to predispose themselves.
If the predisposition A makes is q, then the assignment becomes:
A = f(predict(A*) = g(A* = f(predict(A) = g(q)))

Diagram to Illustrate
However, as A* is A, then whatever predisposition A makes is the same predisposition A* makes. Both A and A* would predispose themselves. It is necessary for at least one of them to predispose themselves, and the strategy that has the highest probability of ensuring that at least one of them predisposes themselves is each of them individually deciding to predispose themselves. Thus, we enter a situation in which both of them predispose themselves. As A* = A, A's predisposition would be the same as A*'s predisposition.
Vice Versa.
We have either:
1. (C,C)
2. (D,D)
If A predisposes themselves to defection, then we have (D,D). (D,D) is a Nash equilibrium as A and/or A* can only perform worse by unilaterally changing strategy at (D,D). As A and A* are rational, they would predispose themselves to cooperation.
If A and A* predispose themselves to cooperation, If A and/or A* tried to maximise their payoff by defecting, then the other (as they predict each other) would also defect to maximise their payoff. Defecting at (C,C) leads to (D,D). Thus, neither A nor A* would decide to defect at (C,C). (C,C) forms a reflective equilibrium.
Vice Versa.
As B's reasoning process closely reflects A*'s (both being perfect rationalists, and having the same preferences), the two agents would naturally converge at (C,C).

Q.E.D
I think I'll call this process of basing your decision in multi-agent decision problems (that involve at least two agents who are perfectly rational, know each other's preferences and are aware of those two facts from the perspective of one of those agents satisfying the above 3 criteria) by modelling the other agents (who satisfy those 3 criteria) as simulations of yourself with their preferences recursive decision theory (RDT). I think convergence on RDT is natural for any two sufficiently (they may not need to be perfect, if they are equally rational, and rational enough that they try to predict how the other agent(s) would act) rational agents who know each other's preferences and are aware of the above two facts.
If a single one of those criteria is missing, then RDT is not applicable.
If for example, the two agents are not equally rational, then the more rational agent would choose to defect as it strongly dominates cooperation.
Or they did not know the other's preferences, then they would be unable to predict the other's actions by inserting themselves in the place of the other.
Or if they were not aware of the two facts, then they'd both reach the choice to defect, and we would be once again stuck at a (D,D) equilibrium. I'll formalise RDT after learning more about decision theory and game theory (so probably sometime this year (p < 0.2), or next year (p: 0.2 <= p <= 0.8) if my priorities don't change). I'm curious what RDT means for social choice theory though.

If they have access to my source and see a random number generator, how would they react? How would things change if the RNG was uniform, Gaussian, etc? Tell us more about maximizing. Are they maximizing expectation or maximizing magnitude (like a lottery play), maximizing wins?

@BobSalita  They are maximising their payoff.

So are these maximax agents?  (Didn't see any mentions of minimizing downside in a worst-case scenario.  I tend to think of maximax as the "optimistic" strategy, which is exceptionally risky in one-shot, and not generally considered rational. Hofstadter used a new term, superrationality, to describe this strategy.  It's easy to demonstrate how the strictly rational strategy becomes "irrational" in iterated Dilemma with a superrational participant, but it's problematic in one-shot...)

I think of it more as trying to maximise their actual payoff. They want to "win" so to speak. I use that mindset when I think of the rational decision.

This question is re-inventing the analysis for iterated prisoner's dilemma and the co-evolution that can lead to agents playing super-rationally in the one-shot version, which has been studied really extensively.
Dan Ashlock's research career looked at this in great detail from an evolutionary perspective, but it's also been widely studied in other areas of AI. The strategy you describe as superintelligent is called "Tit-for-Tat", is well known to emerge when people play several rounds of the game. It actually mimics how people play the game in experimental settings. It emerges evolutionarily in simulations because any two agents that both implement it will get more reward than selfish agents. Other, more complex strategies can also appear. For example, the fortress family of strategies consists of playing a series of question/response opening moves in the early rounds (unlocking the fortress), and then cooperating for the rest of the game if the other player knows the "password".
Hope this helps!

nice, succinct answer to an interesting but difficult to field question!  Useful links. You may also be interested in: https://math.stackexchange.com/q/1629967/362640

Good point! In fact, I've misidentified the strategy in the question, which is closer to Grim Trigger than Tit-for-Tat. IIRC Grim Trigger emerges evolutionarily before Tit-for-Tat, and is a sort of useful prototype for it. Ashlock's research has a lot more on these kinds of dynamics though.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Extracting multiple softmax values from image
I am currently in the pre-process of starting an image classification and extraction project which needs to output multiple softmax and absolute values from a single image like such:
{
 time: "20:20",
 teams: [
   {
     red: { goals: 2},
     blue: { goals: 1},
   },
   {
     scored_by : [{
      john: 80%,
      kyle: 51%,
      darren: 20%
     }
   ]}
 ]
}

I can create multiple models which are responsible for different task such as reading the time from the image as well as the score and eventually combine both. I would however like to make sure I maximize on efficiency to make sure the process is as fast as possible.
Any pointers in the right direction would be greatly appreciated.
With kind regards,
Dennis

What direction are you talking about, though? Libraries? Architecture of the solution?

Any library, discussion or (code) demo where multiple outputs are extracted without having to chain different neural nets.

I see what you mean. You should add neural networks to your tags, and remove linear regression then.

I don't see any question being asked here. "Any pointers in the right direction would be greatly appreciated." is not a question and it's very vague. So, I closed this post as "needs more details". Feel free to edit your post to fix this issue.

It is not unheard of to share network weights with multiple output layers. I have seen it on DeepMind's Asynchronous Deep Learning paper, and I have also used it here.
The idea is to share all the layers and just have multiple outputs. However, this might decrease the accuracy of your networks, as is the usual performance VS accuracy trade-off.

To optimize this, just calculate the loss of both outputs and sum them when feeding the optimizer, like this
self.target_policy_fast_t = tf.placeholder('float32', [None, a_size], name='target_policy_fast_t')
self.loss_policy_fast = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=self.network.policy_fast_before_softmax, labels=self.loss_policy_fast_t))

self.target_policy_slow_t = tf.placeholder('float32', [None, a_size], name='target_policy_slow_t')
self.loss_policy_slow = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=self.network.policy_slow_before_softmax, labels=self.target_policy_slow_t))

self.loss = tf.reduce_mean(self.loss_policy_fast + self.loss_policy_slow, name='loss')

That's great thank you! Will this mean my training time will increment exponentially per extra hidden layer in order to get the same accuracy? Also please notify me when you are free to share your paper.

The training time has some correlation with the amount of hidden layers and the complexity of the network, but it depends on hardware and several other factors. With libraries like TensorFlow, up to a certain complexity, my laptop takes the same amount of time to learn networks with different architectures. I will update the answer when I can, but it might take a few months.

Updated the answer with a link to my paper.

source: Stack Exchange

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
As a starter: what is the form of training data for image processing
What we are doing in the image processing training.  We are storing some form of data which is going to act as the knowledge or experience of the system.

In which form can the system store it's training data?

For example, with the hand written recognition, we can represent the digits as combinations of curves and straight lines. For every round of training the recognition system stores data.  Is the data typically stored in a flat file (such as txt) or a database?
I have seen in Tesseract OCR that there is a text file that stores the x0,y0,x1,y1. They are the pixel points that represents the square on the training image that has the picture.
I need a efficient form of knowledge for Machine Learning, and would appreciate advice, context, or an explanation of the merits or downsides of different approaches.

I need a form of knowledge that stored in a system. human brain evaluate '7' as 'horizontal line and vertical left crossed line start from right of the horizontal line'. like that machine must have some conceptual data to represent their knowledge.

Welcome to AI!  I took the liberty of editing the question for greater readibility—don't hesitate to let me know if I've misrepresented anything.  It might be helpful if you indicated what language you're planning on working in, and possibly the system architecture you envision.  (Data is data, but the way you store it may be influenced by these factors.)

Your question depends heavily on the method you are using for machine learning. It sounds like you want to extract certain features like "curves and straight lines" from your images and use them as training data. This step of extraction is usually not considered part of the training process but part of pre-processing. During pre-precessing you read your image in, extract certain features or perform some transformation and store the new data as your actual training samples or use them for training right away without intermediate storage. Usually storing this information is a good idea if you want to use the image for training more than once and you want to skip the pre-processing step in future training cycles.
Comparing File Storage and Database Storage
How you store your processed data for training is relatively independent from the application of machine learning and the general principles for data storage apply. Storing the data in a flat file is usually very convenient as disk storage is readily available and the APIs for storing and reading files are part of your programming language and OS. A database adds additional complexity to your architecture but has of course it's benefits, especially if you want to make the training data available to other instances or different learning engines. If you are working with a really huge amount of data, a well structured database can help you organize your data better and provides helpful functions for handling this data efficiently.
Keep in mind that the actual training of your AI takes place initially and once completed, you can roll out your AI without all the training data. So you only need to handle training data for a limited time in your application lifecycle. The speed of training is usually not very important, because it happens once and not during the daily use of the AI. Therefore the speed of file access during training can be neglected.
Conclusion
In conclusion, for most applications, the simpler implementation using just flat files is good enough. If you want to store each sample in an individual file or pack them together on bigger batches and use some meta information to identify the individual samples in the file is more a matter of taste than real technical relevance.
Storing data after training
Judging from your question, you sound like you understand machine learning in general. Just to clarify one potential misunderstanding for beginners - you don't have to store any of the "learned" information as data after training, as each training step just adapts weights and biases in the neural network. The actual training data can be thrown away after successful training, if you don't need it for future training cycles.
Further Information / References
To finish my answer I want to recommend a current course from Stanford University about CNNs and visual recognition (Dated Spring 2017). It is a great source of information for implementing neural networks for image recognition.

asking again. if we input a image as a training input, then that is a direct knowledge of machine. may be my question is not clear

I need a form of knowledge that stored in a system. human brain evaluate '7' as 'horizontal line and vertical left crossed line start from right of the horizontal line'. like that machine must have some conceptual data to represent their knowledge.

With a NN, that is not necessarily the case. The added knowledge manifests itself as small adaptions in the weights and biases of the neurons. There is nothing like a lookup table for the knowledge. When using a CNN there are some "intermediate templates" for each class, which sounds like what you are looking for. Please take a look at the Stanford course I linked for more details. The templates get explained in detail starting in the 5th lecture.

perfect, I'm really considering this as wow fact. that the system will increase it's ability on day my day work.
answer was very clear. thank you.

I think Demento has answered it well, but probably below can add some more understanding to you.
[1] Usually the data stored for image processing relates to the Image Pixel Densities(as per the many courses i visited online), which can be very well maintained using a matrix of pixel density values, corresponding to the resolution of each image.
Then it depends on the image's color spectrum (is it a colored or a gray scale); as per the case of hand written text recognition you don't really require the pixel values for a colored image.
Consider that if you have a colored image then you have to store the RGB values for each pixel, thus tripling the matrix size, and the training could be very well done with just the gray scale values, thus it would be better to convert all the images to gray scale in preprocessing step and also take care to convert the image to gray scale when you actually use the trained network to recognize an image (just a novice mistake!).
As per the storage for the required matrix form of the pixel density values, a flat file storage would suffice, but I would recommend libraries instead (if working in python or other alternatives), like pandas and numpy. For these libraries provide a robust solution for data management and retrieval.
For more details - https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
https://github.com/pandas-dev/pandas
[2] Now I would emphasize on why not to use any DataBase(DB) for the storage of such information.
Firstly the integration of it would only result in an unnecessary overhead to your efforts of training the Network and also when you would want to recognize an image(which would always be a single matrix form of it's Image Pixel Densities), you would require the connection to the same DB and would also need to make an insertion to it first, for the script to be able to extract from there and make recognition(not recomended).
There is a fact we don't really concern ourselves with is the quality of the training image dataset, we do ignore the presence of noise in the dataset(would be minimum if using a preprocessed standard data set such MNIST http://yann.lecun.com/exdb/mnist/), but if creating one's own data set we have to consider noise and rectify accordingly in the preprocessing task itself.
With noise I mean loss or overlap of pixel density information which is usually adds a blurring effect to the images, and can seriously damage the training metrics.
To overcome it, there are approaches to rectify the pixel values with a prior learning methods such as extracting data from obfuscated images,
 see link- https://arxiv.org/pdf/1609.00408.pdf.
With the above case you would require to make updates in the DB for very single record in the preprocessing task even before you start to train the Network, thus it can increase the total requests to the DB in multifolds and thus impeding the whole benefit of using it in the first place.
I suppose the above should give you fair idea, about the kind of data to store and the Data Storage to prefer accordingly.
I worked out the Digit Classification using MNIST data set with obscuration using a NN Pipeline, to refer https://github.com/kchopra456/Digit-Classification-and-Image-Obfuscation-ANN.

thank you for your response, I really admit your justification for 'no storage'.  let consider the image of '1'(digit). I have 3 images in size respectively 234kb,2mb,65kb. I trained my system with this  data. now I am not really wish to store the 234+2048+65 of kb of data. instead of this a vector representation data that emphasize strecture of '1'. that data will be improved by the next trainning.

like us. we know the structure of alphabets by the training of the view we seen in childhood. But we are not having the data about which color and which scale (in books).

is it possible to store that kind of data. is there exists any data structure for that

From my understanding till now, you want to train the network not by pixel densities as a direct classifier but rather using a figurative vector scale(some knowledge about how we draw that character on paper). I'm not sure about the feasibility of that, but i would like to point out something I feel that can get you in a bottleneck.
You mention that you want to use the vector representation achieved after each learning iteration to modify the input data set for the next iteration!
This might not be a good approach because the intermediate data set is guaranteed to have a lot of noise.

Also about the varying sizes of images as Input Data Set, can be handled easily in preprocessing.
You can easily compress the  images to suitable size, because we don't really require high scale images for character classification; as not much detail is required in an image to represent a character.
Go for gray scale images for about 400 pixel size and your network will work miracles!, even though you wan to test on a real world image that would be of much higher resolution; you can just take the image through the preprocessing task and then classify.

No Data Structure of such kind would be available for direct use, you might have to dig for something similar and modify for your needs; or you can use cloud storage for that purpose and use the url and retrieve the image through that before feeding, an overhead but can be used( time required to train is not actually an important parameter! ).


text:
Why are these same neural network architecture giving different results?
I tried the first neural network architecture and the second one, but keeping all other variables constants, I am getting better results with the second architecture. Why are these same neural network architecture giving different results? Or am I making some mistakes?
First one:
def __init__(self, state_size, action_size, seed, hidden_advantage=[512, 512],
             hidden_state_value=[512,512]):
    super(DuelingQNetwork, self).__init__()
    self.seed = torch.manual_seed(seed)
    hidden_layers = [state_size] + hidden_advantage
    self.adv_network = nn.Sequential(nn.Linear(hidden_layers[0], hidden_layers[1]),
                                     nn.ReLU(),
                                     nn.Linear(hidden_layers[1], hidden_layers[2]),
                                     nn.ReLU(),
                                     nn.Linear(hidden_layers[2], action_size))

    hidden_layers = [state_size] + hidden_state_value
    self.val_network = nn.Sequential(nn.Linear(hidden_layers[0], hidden_layers[1]),
                                     nn.ReLU(),
                                     nn.Linear(hidden_layers[1], hidden_layers[2]),
                                     nn.ReLU(),
                                     nn.Linear(hidden_layers[2], 1))
def forward(self, state):
    """Build a network that maps state -> action values."""
    # Perform a feed-forward pass through the networks
    advantage = self.adv_network(state)
    value = self.val_network(state)
    return advantage.sub_(advantage.mean()).add_(value)

Second one:
def __init__(self, state_size, action_size, seed, hidden_advantage=[512, 512],
             hidden_state_value=[512,512]):
    super(DuelingQNetwork, self).__init__()
    self.seed = torch.manual_seed(seed)

    hidden_layers = [state_size] + hidden_advantage
    advantage_layers = OrderedDict()
    for idx, (hl_in, hl_out) in enumerate(zip(hidden_layers[:-1],hidden_layers[1:])):
        advantage_layers['adv_fc_'+str(idx)] = nn.Linear(hl_in, hl_out)
        advantage_layers['adv_activation_'+str(idx)] = nn.ReLU()

    advantage_layers['adv_output'] = nn.Linear(hidden_layers[-1], action_size)

    self.network_advantage = nn.Sequential(advantage_layers)

    value_layers = OrderedDict()
    hidden_layers = [state_size] + hidden_state_value

    # Iterate over the parameters to create the value network
    for idx, (hl_in, hl_out) in enumerate(zip(hidden_layers[:-1],hidden_layers[1:])):
        # Add a linear layer
        value_layers['val_fc_'+str(idx)] = nn.Linear(hl_in, hl_out)
        # Add an activation function
        value_layers['val_activation_'+str(idx)] = nn.ReLU()

    # Create the output layer for the value network
    value_layers['val_output'] = nn.Linear(hidden_layers[-1], 1)

    # Create the value network
    self.network_value = nn.Sequential(value_layers)

def forward(self, state):
    """Build a network that maps state -> action values."""

    # Perform a feed-forward pass through the networks
    advantage = self.network_advantage(state)
    value = self.network_value(state)
    return advantage.sub_(advantage.mean()).add_(value)

I suspect the problem comes from the weight initialization. I suggest to manually initialize your weights like this: def init_weights(m):
if type(m) == nn.Linear:
torch.nn.init.xavier_uniform(m.weight)
m.bias.data.fill_(0.01)
net.apply(init_weights) and then go through the calculations layer by layer.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Applying NNs to 2d image noise reduction?
I'd like to explore possibilities of applying deep learning on image noise reduction problem, more on photographic camera noise. What's a good NN architecture to solve problems like this?
EDIT 25,Nov,2017:
I have a small dataset of clean/noisy reference (~15K 4Kres images) acquired from digital camera. The target is to denoise other images from this camera type but without a reference photo.

You might want to look into autoencoders. What they do is learn the relationship between the input and a simplified version of it, and then try to reconstruct the original version of the input.
The good news about this architecture is that it is not a supervised learning technique, so you can feed it a lot of data without having to construct both input and output.
Hopefully, if you train it right, it will learn that the parts of the image that are not important in the result are the noisy parts that you intend to remove.
Note that autoencoders had already been used to reduce noise in audio and image with great success.
Update: Here's a quick not-so-technical explanation on why they work.
In the general architecture that auto-encoder networks have, you'll usually find an input layer, a certain amount of hidden layers which usually have less nodes than the input layer, and then a certain amount of hidden layers that gradually grow to have the same amount of input layers in the output layer.

Why they learn to reproduce the input in the output layer, the data representation needs to necessarily go through the narrow middle layer -- at that point, you can consider that the input has been "compressed", usually with a lossly approach to have a representation of the data that can later on use to reproduce the input.
(Side note: this compression feature has been tested in replacement of compression algorithms but they are not so good at performing compression as particular algorithms designed for it.)
When reducing the error between input and output, the network then starts forgetting the pieces that don't add as much information to the input as other bits of data in it. For instance, if you always train these images with figures in white backgrounds, it is likely that the encoded representation of the image will not have any information about the white background itself, since it is not required to be recreated.
As such, the most common pieces will start falling apart, and with the right tuning, a general amount of noise will go away too. Uniform noise is easier to remove (for the very same reason as the white background example), but variable noise can be worked around too with the right training.

Thanks for your reply. I have a small dataset that could be used for supervised learning, Any good idea to fit them into auto-encoder framework? I could think of "detaching" the output and input and fine-tune the network with my dataset.

@Wangsta Yup! Autoencoders don't need labels (correct outputs) so it will be easier for you to gather lots of images for training. Also, check out techniques for data augmentation, which basically means altering your data to generate more examples for training. It's particularly easy for images.

Recently I read a bit about GANs. It seems like VAEs are sometimes used as the generator for GANs and the noisy images are often used as the input prior for the generator.

@Wangsta That could work too — NNs are universal approximators, so with the right input, output, and hyper parameters, any architecture should work. It all depends on what information you have and how much computing power / training time you can afford.


text:
Python implementation of Label Ranking Trees
Label Ranking is a classification task in which there is a set of potential labels; given a feature vector, the classifier should decide how to rank the labels from most-relevant to least-relevant. As an example, given an image, the classifier should decide what object is the most prominent, second-most prominent, etc.
There are several label ranking algorithms in Weka; they are based on works by Fürnkranz, Hüllermeier and others. One of the common algorithms is called LRT - Label Ranking Trees. Is there any implementation for this algorithm, or for another ranking algoritm, in Python?


text:
How's the action represented in MuZero for Atari?
MuZero seems to use two different methods to encode actions into planes for Atari games:

For the input action to the representation function, MuZero encodes historical actions as simple bias planes, scaled as $a/18$, where $18$ is the total number of valid actions in Atari.(from the appendix E of the paper)
For the input action to the dynamics function, Muzero encode an action as a one-hot vector, which is tiled appropriately into planes(from the appendix F of the paper)

I'm not so sure about how to make of the term "bias plane".
About the second, my understanding is that, as an example, for action $4$, we first apply one-hot encoding, which gives us a zero vector of length $18$ with one in the $5$-th position(as there are $18$ actions). Then we tile it and get a zero vector of length $36$, with ones in the $5$-th and $23$-rd positions. At last, this vector is reshaped into a $6\times 6$ plane as follows:
$$
0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0\\
0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0
$$

The bias plane is a layer everywhere equal to the constant $a/18$ where $a$ is the action. So each of the 32 frames has three frames for RGB and a fourth frame which is the bias plane making for 128 input layers. This is explained in the Network Architecture section where it mentions that the actions are "broadcast" to the planes.

For this I do not have conclusive evidence, but I think tiling a vector means arranging parallel copies of it into e.g. a grid shape. In other words, the input is 6x6x18, and action 1 is represented as all ones in the first plane with all zeros in the remaining planes. One problem with the way you've depicted is that the input is subject to convolutions, but there is no inherent reason why the fifth position and eleventh actions (which are next to each other vertically) should be included in the same 3x3 filter application, but the fifth and seventh (for example) actions should not.

Hi @expz. Thank you for the explanation. I think you are right, but I'm a little confused about why we use different representations for actions in representation and dynamics functions?

If you were to use the one hot encoding for the representation network input then there would be 18 planes per frame or 576 planes devoted to actions. I assume they wanted to reduce that to something more manageable. However the fact that they use the one hot encoding instead of the more succinct encoding for the dynamics network which has a similar architecture suggests to me that perhaps the one hot encoding performs better when it is practical to use.


text:
Intuition behind $\gamma$-discounted state frequency
At the appendix A of paper "near-optimal representation learning for hierarchical reinforcement learning", the authors express the $\gamma$-discounted state visitation frequency $d$ of policy $\pi$ as
$$
d=(1-\gamma)A_\pi(I-\gamma^cP_\pi^c)^{-1}\mu\tag 1
$$
I've simplifed the notation for easy reading, hoping it does not introduce any error. In the above definition, $P_\pi^c$ the $c$-step transition matrix under the policy $\pi$, i.e., $P_{\pi}^c=P_\pi(s_{c}|s_0)$, $\mu$ a Dirac $\delta$ distribution centered at start state $s_0$ and
$$
A_\pi=I+\sum_{k=1}^{c-1}\gamma^kP_\pi^k\tag 2
$$
They further give the every-$c$-steps $\gamma$-discounted state frequency of $\pi$ as
$$
d^c_\pi=(1-\gamma^c)(I-\gamma^cP_\pi^c)^{-1}\mu\tag 3
$$
To my best knowledge, $A_\pi$ seems to be the unnormalized $\gamma$-discounted state frequency, but I cannot really make sense of the rest.
I'm hoping that someone can shed some light on these definitions.
Update
Thank @Philip Raeisghasem for pointing out the paper CPO. Here's what I've gotten from that.
Applying the sum of the geometric series to Eq.$(2)$, we have
$$
A={(I-\gamma^cP_\pi^c)(I-\gamma P_\pi)^{-1}}\tag4
$$
Plugging Eq.$(4)$ back into Eq.$(1)$, we get the same result as Eq.$(18)$ in the CPO paper:
$$
d=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag 5
$$
where $(1-\gamma)$ normalizes all weights introduced by $\gamma$ so that they are summed to one. However, I'm still confused. Here are the questions I have

Eq.$(5)$ indicates Eq.$(1)$ is the state frequency in the infinite horizon. But I do not understand why we have it in the hierarchical policy. To my best knowledge, policies here are low-level, which means they are only valid in a short horizon ($c$ steps, for example). Computing state frequency in the infinite horizon here seems confusing.
What should I make of $d_\pi^c$ defined in Eq.$(3)$, originally from Eqs.$(26)$ and $(27)$ in the paper? The authors define them as every-$c$-steps $\gamma$-discounted state frequencies of policy $\pi$. But I do not see why it is the case. To me, they are more like the consequence of Eq.$(30)$ in the paper.

Sorry if anyone feels that this update makes this question too broad. This is kept since I'm not so sure whether I can get a satisfactory answer without these questions.
Any partial answer will be sincerely appreciated. Thanks in advance.

See the appendix of Constrained Policy Optimization. I imagine these equations build on that and are a consequence of a geometric series of matrices. I don't have a full proof/explanation, but I look forward to anyone who does.

Thank @PhilipRaeisghasem. I've updated the question according to the resources you provided

I think your update made the question too broad. Try to limit your post to a single question. In this case, removing the update would suffice.

Sorry, I thought I just elaborated my confusion since my updated questions were all related to the same topic. I'm not so sure if I can get a sufficient answer if I remove those questions

Update: I rewrote the first part due to major mistake in the first version
Notice: The notation $P^k$ from Eq.$(20)$ and $(21)$ in the paper does not mean the kth power of some $P$. Instead, $P^k$ should be thought as the $k$ step transition probability of a non-homogeneous Markov chain.

According to the CPO paper, the discounted future state distribution is defined as
$$
d_\pi(s)=(1-\gamma)\sum_{k=0}^\infty \gamma^kProb(s_k=s|\pi,s_0)\mu(s_0)\tag{1}
$$
Consider function form. Let $Prob_{\pi,k}$ denote the $k$ step probability transition operator induced by $\pi$; here $\pi$ can be a hierarchical policy, $k$ can be larger than $c$.
$$
d_\pi=(1-\gamma)\sum_{k=0}^\infty \gamma^kProb_{\pi,k}\mu\tag{2}
$$
Now apply the similar definition as Eq.$(20)$ and $(21)$ in the paper, let $P^k_\pi$ denote the $k$ step transition probability of the non-homogeneous Markov chain induced by the low level policy, with $k$ smaller or equal to $c$.
\begin{align}
d_\pi&=(1-\gamma)\sum_{m=0}^\infty\gamma^{mc}\sum_{k=0}^{c-1}\gamma^kP^k_\pi(P^c_\pi)^m\mu\\
&=(1-\gamma)\sum_{k=0}^{c-1}\gamma^kP^k_\pi(\sum_{m=0}^\infty\gamma^{mc}(P^c_\pi)^m)\mu\\
&=(1-\gamma)A_\pi(I-\gamma^cP^c)^{-1}\mu\tag{3}
\end{align}
which is exactly the form of Eq.$(22)$ and $(23)$ in the paper, with $A_\pi$ defined similar as Eq.$(24)$ and $(25)$.
The "every-$c$-step discounted state frequency" builds on $(3)$, but it lumps the $c$ steps into one "high level" step where the discount factor is $\gamma^c$ and the transition operator is $P_\pi^c$. Starting with $(3)$, replace $c$ with 1, we get the "every one step future state distribution"
$$
d_\pi=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag{4}
$$
Then replace $\gamma$ and $P_\pi$ in $(4)$ with $\gamma^c$ and $P_\pi^c$, we get the "every-$c$-step discounted state frequency", or "every-$c$-step future state distribution"
$$
d_\pi^c=(1-\gamma^c)(I-\gamma^c P_\pi^c)^{-1}\mu\tag{5}
$$
By the way, I read your blogpost on this paper. It's very helpful for me, thank you!

Thanks for answering, your second explanation is great:-). But I'm sorry that I cannot follow your first reasoning exactly. Would you mind explaining how you compute the normalizing term $(1-\gamma^cP_\pi^c)^{-1}$? Furthermore, I'm confused about "the meaning of $(4)$ is the same as $(1)$". Do you suggest state distribution in c steps is the same as that in infinite steps? BTW, I'm glad my blog helps:)

For the derivation of the normalizing term, please see my updated answer. As for my comment "the meaning of $(4)$ is the same as $(1)$", I mean intuitively they roughly refer to the same concept and play the same role as in expressing the discounted value function. But I agree, this comment is misleading, so I deleted it... Hope this update is helpful...

Thanks for your proof. I think I asked the wrong question yesterday. $\tilde d_\pi$ in your update is essentially $d_\pi$ defined in (1) and it is the state distribution in the infinite horizon. Thanks to you, I think I've found what I missed before. I originally thought using $d_\pi$ to express state distribution carried out by low-level policies is inappropriate since the low-level policies usually have a finite horizon. But I did not realize $\pi$ here actually denotes the whole hierarchical policy. That clears all my doubts. Thank you :-)


text:
What is a no-good? When does the no-goods level off?
In section 10.3.2 in Artificial Intelligence: a Modern Approach there is a piece of pseudocode that describes the graph plan algorithm. The graph plan algorithm constructs a planning graph for a problem and extract a solution from it if there is one.
Here is the pseudocode:
function GRAPHPLAN(problem) returns solution or failure
    graph ← INITIAL-PLANNING-GRAPH(problem)
    goals ← CONJUNCTS(problem.GOAL)
    nogoods ← an empty hash table
    for tl=0 to ∞ do
        if goals all non-mutex in St of graph then
            solution ← EXTRACT-SOLUTION(graph, goals, NUMLEVELS(graph), nogoods)
            if solution != failure
                then return solution
        if graph and nogoods have both leveled off
                then return failure
        graph ← EXPAND-GRAPH(graph, problem)

Also a PPT for Graphplan
What confuses me for a long time is when the nogoods levels off? The book describe no-goods as follows:

In the case where EXTRACT-SOLUTION fails to find a solution for a set of goals at a given level, we record the (level,goals) pair as a no-good.

But in that case, every time the first EXTRACT-SOLUTION after EXPAN-GRAPH fails, there will always be at least a new no-good produced: whose level becomes the new level of the planning graph and the goals is always the same--which is the goal of the problem. So when does it level off?
There is another sentence about no-goods in the book:

If the function EXTRACT-SOLUTION fails to find a solution, then there must have been at least one set of goals that were not achievable and were marked as a no-good.

From above excerpt, I learn that goals in no-goods is a set of goals that are not achievable in the level. But, IMHO, that could vary when we chose different actions in EXTRACT-SOLUTION. The example in the book provides a good instance:

Consider an air cargo domain with one plane and n pieces of cargo at airport A, all of which have airport B as their destination. In this version of the problem, only one piece of cargo can fit in the plane at a time. The graph will level off at level 4, reflecting the fact that for any single piece of cargo, we can load it, fly it, and unload it at the destination in three steps. But that does not mean that a solution can be extracted from the graph at level 4; in fact a solution will require 4n − 1 steps: for each piece of cargo we load, fly, and unload, and for all but the last piece we need to fly back to airport A to get the next piece.

In the example above, the set of goals not achievable at level 4 varies based on what actions has been taken.
Could someone help clear up this problem? I know there must be some misunderstanding, welcome to point it out, please. Thanks in advance:)
Ps. I'm a English leaner, I may not present this problem very well, so I'm very sorry for any reason that you may want to down vote it. But I‘m very serious about it and I've try my best to make it as clear as I can. So if you're about to down vote it, please comment below and give me some advice to improve it, I'll be very grateful for that. Thanks again!

I've got a thought about the no-goods. I think (level, mutexes) will do the job, where mutexes is a set of pairs of states that are mutex with each other in the level. It doesn't suffer the problem described above, but I don't know if it's suitable in other conditions I haven't thought of. plz help me check it out.

Kindly,please go through the community guide lines artificial intelligence

Thanks for your comment. May I ask whether you were suggesting I provide my 'answer' along with the question? If that's the case, I want to explain myself. I didn't have any answer about this problem in the first place, I was totally confused by the no-goods defined in the book at that time. That was why I asked it. The 'answer' I provided above was what I worked out later, four days later exactly. And I don't think that was a truly answer, 'cause it doesn't match up with the no-goods described in the book. I provided it in the comment just as a personal opinion..


text:
What is the internal state of a Simple Neural Attentive Meta-Learner(SNAIL)?
In the paper A Simple Neural Attentive Meta-Learner, the authors mentioned right before Section 3.1:

we preserve the internal state of a SNAIL across episode boundaries, which allows it to have memory that spans multiple episodes. The
  observations also contain a binary input that indicates episode termination.

As far as I can understand, SNAIL uses temporal convolutions to aggregate contextual information, from which causal attention learns to distill specific pieces of information. Temporal convolutions does not seems to maintain any internal state, and neither does the attention mechanism they use after this paper. This makes me wonder: "What is the internal state of a SNAIL?"

Here's what I understand, welcome to point out any mistakes.
When starting a new episode(but still in the same task), SNAIL does not clear its batches. Instead, it makes decisions based on the current observation and observation-action pairs from the previous episode. In this way, it keeps knowledge of the previous episode whereby achieving few-shot learning in the test time.


text:
Getting Negative Information Gain (using Entropy as a measure of disorder)
I'm creating a Decision Tree and at the very root level itself, I'm getting negative Information Gain.
As per my knowledge, Information Gain is always > 0....
Any explanation....please....
Please look at the node below.....
    [69+,42-]
     /   \
    /     \
[56+,33-]  [11+,11-]

IG = H([69+, 42-]) - H([56+,33-],[11+,11-])

   = {(-69/111 * lg(69/111) -42/111 * lg(42/111))
      - 89/111 * ( -56/89*lg(56/89) -33/89*lg(33/89))
      - 22/111 * ( -11/22*lg(11/22) -11/22*lg(11/22))}

   = -0.004

It turns out that for every feature, the IG < 0.
What should I do to decide the feature at the root node?
Thanks

It is not possible to have a negative IG, your outcome is negative because of a computational mistake: initially you have 69 positive instances, and after creating 2 children the sum drops to 56+11=67. Set is therefore less pure -> IG is negative.


text:
Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch
I'm training an auto-encoder network with Adam optimizer (with amsgrad=True) and MSE loss for Single channel Audio Source Separation task. Whenever I decay the learning rate by a factor, the network loss jumps abruptly and then decreases until the next decay in learning rate.
I'm using Pytorch for network implementation and training.
Following are my experimental setups:

 Setup-1: NO learning rate decay, and
          Using the same Adam optimizer for all epochs

 Setup-2: NO learning rate decay, and
          Creating a new Adam optimizer with same initial values every epoch

 Setup-3: 0.25 decay in learning rate every 25 epochs, and
          Creating a new Adam optimizer every epoch

 Setup-4: 0.25 decay in learning rate every 25 epochs, and
          NOT creating a new Adam optimizer every time rather
          using PyTorch's "multiStepLR" and "ExponentialLR" decay scheduler
          every 25 epochs

I am getting very surprising results for setups #2, #3, #4 and am unable to reason any explanation for it. Following are my results:
Setup-1 Results:

Here I'm NOT decaying the learning rate and
I'm using the same Adam optimizer. So my results are as expected.
My loss decreases with more epochs.
Below is the loss plot this setup.

Plot-1:

optimizer = torch.optim.Adam(lr=m_lr,amsgrad=True, ...........)

for epoch in range(num_epochs):
    running_loss = 0.0
    for i in range(num_train):
        train_input_tensor = ..........
        train_label_tensor = ..........
        optimizer.zero_grad()
        pred_label_tensor = model(train_input_tensor)
        loss = criterion(pred_label_tensor, train_label_tensor)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    loss_history[m_lr].append(running_loss/num_train)

Setup-2 Results:

Here I'm NOT decaying the learning rate but every epoch I'm creating a new
Adam optimizer with the same initial parameters.
Here also results show similar behavior as Setup-1.

Because at every epoch a new Adam optimizer is created, so the calculated gradients
for each parameter should be lost, but it seems that this doesnot affect the
network learning. Can anyone please help on this?

Plot-2:

for epoch in range(num_epochs):
    optimizer = torch.optim.Adam(lr=m_lr,amsgrad=True, ...........)

    running_loss = 0.0
    for i in range(num_train):
        train_input_tensor = ..........
        train_label_tensor = ..........
        optimizer.zero_grad()
        pred_label_tensor = model(train_input_tensor)
        loss = criterion(pred_label_tensor, train_label_tensor)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    loss_history[m_lr].append(running_loss/num_train)

Setup-3 Results:

As can be seen from the results in below plot,
my loss jumps every time I decay the learning rate. This is a weird behavior.

If it was happening due to the fact that I'm creating a new Adam
optimizer every epoch then, it should have happened in Setup #1, #2 as well.
And if it is happening due to the creation of a new Adam optimizer with a new
learning rate (alpha) every 25 epochs, then the results of Setup #4 below also
denies such correlation.

Plot-3:

decay_rate = 0.25
for epoch in range(num_epochs):
    optimizer = torch.optim.Adam(lr=m_lr,amsgrad=True, ...........)

    if epoch % 25 == 0  and epoch != 0:
        lr *= decay_rate   # decay the learning rate

    running_loss = 0.0
    for i in range(num_train):
        train_input_tensor = ..........
        train_label_tensor = ..........
        optimizer.zero_grad()
        pred_label_tensor = model(train_input_tensor)
        loss = criterion(pred_label_tensor, train_label_tensor)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    loss_history[m_lr].append(running_loss/num_train)

Setup-4 Results:

In this setup, I'm using Pytorch's learning-rate-decay scheduler (multiStepLR)
which decays the learning rate every 25 epochs by 0.25.
Here also, the loss jumps everytime the learning rate is decayed.

As suggested by @Dennis in the comments below, I tried with both ReLU and 1e-02 leakyReLU nonlinearities. But, the results seem to behave similar and loss first decreases, then increases and then saturates at a higher value than what I would achieve without learning rate decay.
Plot-4 shows the results.
Plot-4:

scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer=optimizer, milestones=[25,50,75], gamma=0.25)

scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=optimizer, gamma=0.95)
scheduler = ......... # defined above
optimizer = torch.optim.Adam(lr=m_lr,amsgrad=True, ...........)

for epoch in range(num_epochs):

    scheduler.step()

    running_loss = 0.0
    for i in range(num_train):
        train_input_tensor = ..........
        train_label_tensor = ..........
        optimizer.zero_grad()
        pred_label_tensor = model(train_input_tensor)
        loss = criterion(pred_label_tensor, train_label_tensor)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    loss_history[m_lr].append(running_loss/num_train)

EDITS:

As suggested in the comments and reply below, I've made changes to my code and trained the model. I've added the code and plots for the same.
I tried with various lr_scheduler in PyTorch (multiStepLR, ExponentialLR) and plots for the same are listed in Setup-4 as suggested by @Dennis in comments below.
Trying with leakyReLU as suggested by @Dennis in comments.

Any help.
Thanks

Comments are not for extended discussion; this conversation has been moved to chat.

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise consistently shrinks really should, at worst, just lead to a plateau in your losses (rather than those jumps).
The first thing I observe in your code is that you re-create the optimizer from scratch every epoch. I have not yet worked enough with PyTorch to tell for sure, but doesn't this just destroy the internal state / memory of the optimizer every time? I think you should just create the optimizer once, before the loop through the epochs. If this is indeed a bug in your code, it should also actually still be a bug in the case where you do not use learning rate decay... but maybe you simply get lucky there and don't experience the same negative effects of the bug.
For learning rate decay, I'd recommend using the official API for that, rather than a manual solution. In your particular case, you'll want to instantiate a StepLR scheduler, with:

optimizer = the ADAM optimizer, which you probably should only instantiate once.
step_size = 25
gamma = 0.25

You can then simply call scheduler.step() at the start of every epoch (or maybe at the end? the example in the API link calls it at the start of every epoch).

If, after the changes above, you still experience the issue, it would also be useful to run each of your experiments multiple times and plot average results (or plot lines for all experiments). Your experiments should theoretically be identical during the first 25 epochs, but we still see huge differences between the two figures even during those first 25 epochs in which no learning rate decay occurs (e.g., one figure starts at a loss of ~28K, the other starts at a loss of ~40K). This may simply be due to different random initializations, so it'd be good to average that nondeterminisim out of your plots.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
Weighing connections between two users using machine learning
Given this data set:
User 1: {'Artist': 1, 'Public Figure': 9, 'Film Director': 1, 'Education': 1, 'Musician/Band': 4, 'Musician': 1, 'Community': 1, 'Author': 4, 'Politician': 1, 'TV Show': 2, 'Entrepreneur': 1, 'Journalist': 4, 'Product/Service': 1, 'Defense Company': 1, 'Computer Company': 2, 'Nonprofit Organization': 3, 'Computers & Internet Website': 4, 'Media/News Company': 3, 'Podcast': 1, 'News & Media Website': 2, 'Charity Organization': 1, 'Government Organization': 1, 'Magazine': 1}
User 2: {'Nonprofit Organization': 1, 'Movie': 2, 'Musician/Band': 22, 'Public Figure': 2, 'Entrepreneur': 2, 'TV Show': 2, 'Medical Company': 1, 'American Restaurant': 1, 'Sports & Recreation': 1, 'Media/News Company': 2, 'Sports Team': 1, 'Artist': 3, 'Musician': 1, 'College & University': 1, 'Athlete': 1, 'Music': 1, 'App Page': 1, 'Comedian': 1, 'Interest': 2, 'Product/Service': 1, 'Book Series': 1

I believe I can draw conclusions that the strength of the connection is high or low based on the fact that they like or dislike the same categories of topics. What machine learning technique would be best to apply to these connection types, if I had, say, 500 users with categories and weights for each?
I would like to automatically apply varying weights, if most people have Musician/Band or like Movies, that should be less interesting than when people like "Defense Company" like in User 1.

@DouglasDaseeco ,before all that i would request the OP to go through the community guide lines Artificial Intelligence.No need to rush.

The question might be improved so that the answer can be more specific by defining two things more rigorously.

Strength of the connection
What the automation must infer from that value

Learning, Correlation, Projection, and Maximizing Profitability
Guessing from the comments, it appears that the goal is to profile the user for business purposes.  Two common applications of user profiling are

To present buying options or
To suggest friends in a social network.

Although it may appear that a learning program would serve both of these applications, they are actually two very different problems.
One is to optimize e-commerce profitability through increased sales per session.  The other is to build a social network where there are some similarities and perhaps a few deliberate differences in the pairing of user profiles from which the friend suggestion comes.
The first of the two has to do with the projecting of the potential of a sale by matching the users and relying on the assumption that two people with like interests are more likely to mirror one another in purchasing patterns to some degree that could be projected with statistical analysis.
But approaching the project by immediately jumping to learning algorithms may not produce the best results after a month or a year of development.
One Reasonable Approach for Some Problems of This Type
Depending on the definitions of STRENGTH and AUTOMATICALLY in the question, it may be a simple application of a probability matrix or hypercube that will create a functional application of user profiling based on interests. To create a correlation between users via an ordered list of interests to place in one of the cells of the matrix or hypercube, the Spearman's Rank Order Correlation is a generalized yet effective method for such applications.
Learning Usually Begins With a Guess
First, you must determine what is (preferably in precise mathematical terms) what is a favorable product presentation or favorable potential friend introduction.  After that is determined, one can develop a probability formula to project what would be favorable based on the correlation matrix or hypercube.  Bayes's Theorem may be quite applicable in the development of a probability formula from the constraints and relationships of the application.
Learning Added
Once improved favorablility gained through the application of the correlations  can be demonstrated, it would then be wise to apply a learning algorithm.  The sequence is thus.

Results gained by random selection of options to present the user
Results gained by using the correlation matrix or hypercube
Results gained by iteratively converging on an optimal presentation
Tuning the iterative convergence track change and yet not oscillate

A multi-tier, back propagating Hopfield neural net could be one option for varying parameters in the use of the correlations (or the choice of correlations, since Spearman's is only one correlation model for ordered discrete interests).  Neural networks are only one type of convergent approach too.  Modelling with as a system of differential equations and using extensions of the Taylor Series to evaluate with a defined independent input vector is another.
To see how simple convergence can be and how primitive what appears to be a learning algorithm can become, think about the divide and average method of iteratively determining a square root.


text:
Reasoning with incomplete information
I was wondering if anyone can suggest a good framework for reasoning with incomplete information.
I have found Large Knowledge Collider but it appears dead for some time. Do you possibly have any other suggestions for a maintained project worth checking?
Since many comments are gravitating towards a different direction let me add one approach that I found a potentially good answer to my question - Rough Set Based Decision Trees.
I would hope there is more than only this approach... could you please help me identify them?

Welcome to AI!  (This type of question is a little bit on the line in terms of community guidelines, but I like it and think it is useful.  Decision making in a condition of incomplete information may be said to be the primary concern of game theory.)

What is your original problem? In a way, Reinforcement Learning is (almost) always dealing with incomplete information.

I can see the direction you are heading but RL is not an answer for my problem. I am looking for a usable software / libraries that can be used to derive new knowledge (i.e. claims about relationships between entities / variables) from an existing, provided set of "axioms" with an additional caveat that there is a variability in the degree of trust in those statements (that somehow can be quantified).

I am very thankful to the people who responded with hints and suggestions. However, I think what seems most applicable for my case is Gen - A general-purpose probabilistic programming system with programmable inference from MIT described in the paper "Gen: A General-Purpose Probabilistic Programming System with Programmable Inference" by M. F. Cusumano-Towner et al.
In case you are looking for something along these lines it looks like a very good start for an application in probabilistic programming.

Firstly, before we commence I will recommend that you refer to similar questions on the network i.e https://stackoverflow.com/questions/39386936/machine-learning-with-incomplete-data , https://stats.stackexchange.com/questions/103500/machine-learning-algorithms-to-handle-missing-data
Row Deletion
If a particular row has more than 70% missing values, you can delete the row to handle the null values. This method is advised only when there are enough samples in the data set. The major disadvantage of this method is that it reduces the power of the model because it reduces the sample size.
Replacing With Mean/Median/Mode
We can calculate the mean, median or mode of the feature and replace the missing values with it. Another approach is to approximate it with the deviation of neighbouring values.
Although this approach adds variance to the data set, it yields better results compared to removing rows and columns.
KNN or Random Forest imputation
In this approach, the missing values of an attribute are imputed using existing attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function.
The advantage of this approach is that k-nearest neighbour can predict both qualitative and quantitative attributes. Additionally you do not need to create a prediction model for each attribute with missing data in the dataset.
Predicting the Missing Values
Prediction is one of the more sophisticated methods for handling missing data.
Using the features which do not have missing values, we can predict the null values with the help of a machine learning algorithm.
In this case, we divide our data set into two. One set with no missing values and another set with missing values. The first data set becomes the training data set of the model while the second data set with missing values is the test data set.
We then create a model to predict target variables based on other attributes of the training data set and populate the missing values of the test data set.(Sayali S 2016)
Caret or randomForestSRC packages in R
The R package randomForestSRC can handle missing data for a wide class of analyses i.e. regression, classification, unsupervised and multivariate (Ankur C 2014).
Additionally, the Caret R package can be used to predict missing data.
Reference : https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/

Thank you for your answer but it is on the borderline of what I am asking about. There is a difference between "patching holes"  in the data and proceeding as if the data was fully valid (and this seems to me the approach you are suggesting by enumerating ways to "patch" the data) vs. an approach that to reason (e.g. draw conclusions, i.e. perform logical operations) on the data taking into account that it is not certain that the values are as they are or knowing there is a missing value somewhere among the variables taken into consideration.


text:
How can I make my network treat rotations of the input equally?
I'm attempting to program my own system to run a neural network. To reduce the number of nodes needed, it was suggested to make it treat rotations of the input equally.
My network aims to learn and predict Conway's Game of Life by looking at every square and its surrounding squares in a grid, and giving the output for that square. Its input is a string of 9 bits:

The above is represented as 010 001 111.
There are three other rotations of this shape however, and all of them produce the same output:

My network topology is 9 input nodes and 1 output node for the next state of the centre square in the input. How can I construct the hidden layer(s) so that they take each of these rotations as the same, cutting the number of possible inputs down to a quarter of the original?
Edit:
There is also a flip of each rotation which produces an identical result. Incorporating these will cut my inputs by 1/8th. With the glider, my aim is for all of these inputs to be treated exactly the same. Will this have to be done with pre-processing, or can I incorporate it into the network?

Outstanding question!  I have a similar, impeding issue with my own project and will be very interested to learn about the most efficient techniques to reduce for symmetry.

@DukeZhou I get the impression that it will take a while to get an answer however. I'm prepared to use up my meagre supply of rep to set up a bounty if needed...

Another solution is to pre-process the input, such that all 4 rotations are converted into the same image before being fed to the network.

If I understand well your single output node will be the next status of the square in the middle. You don't need to worry about the number of nodes in the hidden layers while you have sufficient resources to train the model. This problem is very easy to learn for a neural network so no size concerns.
You need to do supervised training that means you need to feed in input data and the matching expected output. You need to be sure that in your training data all 4 rotations are assigned to the same output. This way your network should learn to treat all of these the same way.
You made me curious so I tried myself. My solution could learn 100% correct in about 20 epochs running within a few seconds on my old laptop. I only slightly changed the output to be categorical either [0,1] or [1,0] but that gives the same result that you are looking for. Just for reference here is the code written in python:
from keras.models import Sequential
from keras.layers import Input, Dense
from keras.models import Model
from keras import optimizers
from keras.utils.np_utils import to_categorical
import helper

x_,y_ = helper.fnn_csv_toXY("conway.csv","output",False)
y_binary = to_categorical(y_)

model = Sequential()
model.add(Dense(100, activation='relu', kernel_initializer='glorot_uniform',input_shape =(9,)))
model.add(Dense(20, activation='relu', kernel_initializer='glorot_uniform'))
model.add(Dense(2, activation='softmax'))
adam=optimizers.Adam()
model.compile(optimizer=adam,
              loss='categorical_crossentropy',
              metrics=['acc'])
model.fit(x_, y_binary, epochs=100)

The network I will be using will be simulated by a class written by myself, hence the memory concerns.

If you want reduce memory usage by the network then reducing the number of possible inputs (by rotating) will help to have a smaller network. The easier the learning task is the smaller network needed. In this case pre-processing would be better. Nevertheless NN is for concept learning and to learn the concept of Game of Life you should feed in all patterns. If your aim is to absolutely minimize memory footprint solve the problem on a linear way.

If memory is not a concern, I would rather the network performs this operation for the same reasons you stated. Pre-processing removes some of the task from the network, simplifying it.

Correct, then go for the pre-processing. I think with this the question is answered. You may chose to implement it if you run into issues with memory. Tip: use float for the weights that takes only 32 bits instead of double that takes 64. This will use less memory.

You have identified an optimization in your problem space and desire to bake this into your neural net. I suggest preprocessing: Compose your optimization with a neural net that does a subset of what you want.
In other words, normalize your input by manually coding a rotation algorithm that rotates inputs to capture the equivalence highlighted in your post. Then feed the output of this transformation to your neural net, for training and all other uses. This means you are training the neural net to tackle the sub-problem you identified - rotations are redundant.
Test your normalizer by generating random input, rotating it to all four potential transformations, run the normalizer on each one, then check that they are all equivalent.

To be purist about it, begin with considering the input differently, as a circular array of size four, each item containing a pair of bits, and additionally a center bit:

... 01, 01, 11, 10 ...
0

Throughout the design of the network, continue this circular structure and center point paradigm.


text:
Insight into architecture details of Vision Llama 3.2
I would like to understand the internals of Meta's new multimodal Vision-Text 3.2 models. Not much I could find in online sources or blogs. The code is available and I'm trying to read it, but my understanding is still limited.
It seems to me that the main ingredient is that we have an image and a text

the image is encoded using a pretrained image encoder first, but I'm not sure if it was trained separately or as part of the overall Llama3.2 training
Both "image sequence" and "text sequence" interact in some cross-attention layers, during the image forward pass

I'm not sure how they handle multiple images interleaved with texts, however.
Are there any helpful online sources with details, or would someone educated ellaborate on the details of the architecture and training here?
I would also like to understand what is the advantage of this cross-attention compared to Llava which only uses a projection of the encoded image sequence to the Llama's input embedding space and seems to be conceptually simpler.
I will also appreciate partial answers or relevant remarks.

In image projection models like Llava image features are first passed through an image encoder (e.g., a CNN or ViT) and then projected into the shared text embedding space. Once projected, the image embeddings are treated as if they are a separate sequence of tokens alongside the text tokens. Each sequence (image and text) is processed by the transformer independently, and their interaction is restricted to how the sequences are interleaved and combined at the input stage either during training or inference.
In contrast, cross-attention such as in Llama 3.2 means how each granular token (whether from the text or image) can attend to tokens from the other modality during each layer’s computation, which creates iterative interactions where both modalities can continuously inform and refine the model’s understanding. At lower layers this might be simple matching (e.g., "dog" in text and a "dog" in the image), while at higher layers this might involve more abstract reasoning (e.g., "The dog in the park is barking at the bird"). Also such iterative interactions can learn certain visual features which are only relevant in specific textual contexts, for instance, recognizing a dog in the image may not be helpful unless the context in the text also mentions a dog.
Finally projection model's fixed transformation of an image into a text-based space via feature extraction could lose important spatial or semantic details of the image. In cross-attention models, the model preserves the image's richness using separate image tokens and maintaining their detailed information through the interaction with the text. This is especially beneficial in tasks that require fine-grained understanding, like image captioning, where knowing the exact position of objects and their relationships to text is critical.

Thank you. This part, about Llava "Each sequence (image and text) is processed by the transformer independently" -- are you sure? My understanding was that the image-projections and text-tokens-embeddings are interleaved, as part of one LLM input, and the training comes from backpropagating loss from the "Answer" part in the (Image, Qustion, Answer) triple...

Appreciate your clarification as here by processed independently I mean relative to cross attention layered processing, indeed, in Llava the two sequences of different modality are combined as a single sequence fed into LLM for language modeling loss (typically CE loss). But even with such (Image, Question, Answer) triple supervised learning, there's no layered granular interaction as the flexible cross attention. Hope this clarifies.


text:
Nearest neighbour search in high dimension retrieves certain points too often
We represent some catalogue items (documents, music tracks, videos, whatever) as vectors embedded in R^d and use them to retrieve nearest neighbours to users query. The typical scenario is that users can input any query and the search results are from a fixed user-catalogue.
I noticed experimentally that certain results occur much more frequent among k-nearest neighbours than other results. The majority of documents is never found among nearest neighbours of any other document from the catalogue, while a few are found many times. In other words, the distribution of "how many times a document can be found" is extremely skewed.
It doesn't matter if I use euclidean or cosine similarity or dot product, and it doesn't depend on the catalogue. It seems to depend on the embedding dimension, however.
I tried to reproduce the problem with random data embedded in R^d for various d, and measure the distribution of "how many times a vector is among 10 nearest neighbours of some other point". For low dimensions (up to 10) the distribution is close to normal centered at 10. However, with increasing dimension it's getting very skewed. Hubs (points that occur often in k-NN search) are always those points that are nearer to cluster center.
I asked about this on math stackexchange without any response there. The only comments I got there was "nearest neighbors do not make sense in high dimension, everything is too far from anything else" which is true but not useful.
I found this paper which recommends (in Chapter 5) to mitigate this problem so that any data point that is a hub (occurs often among k-nearest neighbour search) should be punished, in a similarity search, by some discount. This may work, but looks more like a hack than a systematic approach.
So here is my question:

is there some standard method how to aproach it? Maybe try to minimize the embedding dimension, assuming some other metrics are good enough?
are there some other common practices to ensure that customers don't always get the same (even though good) results?

Thanks for any hints or references.

I cannot answer your question why this is happening, hence only a comment. But for high dimensions, I recommend HNSW or Facebook's FAISS. I've used HNSW myself and appreciated how easily you can trade off accuracy (which is pretty high anyways) for runtime by adjusting simple parameters. Their demonstration on github is pretty easy to follow: https://github.com/nmslib/hnswlib
You could also consider combining KNN with TF-IDF, such as here:
https://stackoverflow.com/questions/59057351/knn-for-text-classification-using-tf-idf-scores

The idea of including TF-IDF being that you get rid of "uninteresting" similarities by counterweighing each term-frequency by its inverse document frequency. Finally, perhaps consider using an autoencoder for information retrieval, athough that is certainly trickier.

High-dimensional spaces are very counter-intuitive. https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions/99191#99191


text:
Understanding gumbel-softmax backpropagation in Wav2Vec papers
I'm studying the series of Wav2Vec papers, in particular, the vq-wav2vec and wav2vec 2.0, and have a problem understanding some details about the quantization procedure.
The broader context is this: they use raw audio and first convert it to "features" $z$ via a convolutional network. Then they project any feature $z$ to a "quantized" element $\hat{z}$ from a given finite codebook (or concatenation of finitely many finite codebooks). To find $\hat{z}$, they compute scores $l_j$ for each codebook entry $v_j$, convert these scores to Gumbel-Softmax probabilities $p_j$ (using a formula which is not deterministic, the formula involves random choices of some numbers from some distribution) and then use these probabilities $p_j$ to choose $\hat{z}$. Further stages of the pre-training pipeline are trained to predict $\hat{z}'s$ by either predicting "future" from the "past", or "reconstructing masked segments".
My question is this is about this sentence:

During the forward pass, $i = \text{argmax}_j p_j$ and in the
backward pass, the true gradient of the Gumbel-Softmax outputs is
used.

I have trouble seeing what exactly is happening in the loss function and back-propagation. Could someone please help me to break this down into details?

My mental attempts to make sense out of it (I'm using the notation $\hat{z}$ for quantized vectors, in the second paper they use $q$)
(1) I would say that during the forward pass, in the Gumbel-Softmax, random variables from the Gumbel-distribution $n_j$ are sampled every time (for every training example) to compute the Gumbel-softmax probabilities $p_j$.
(1a) In the back-propagation, these $n_j$'s are kept constant, and $p_j$ is treated as a function of $l_j's$ only.
(2) The loss function has 2 parts here, Contrastive loss and Diversity loss.
(2a) Based on the description, I would say that in the contrastive loss, the "sampled" vectors $\hat{z}_j$ are used, and probabilities never appear (even not in back-propagation of this part of the loss).
(2b) I would believe that in the gradient of the Diversity loss, which only uses probabilities $p_{g,v}$, that here the gradient or the loss actually is used, as this is responsible for maximizing the entropy. This part of the gradient probably does not use the sampled values $\hat{z}_j$.
Is this approximately correct?
If yes, then I still fail to understand what exactly is happening in the vq-wav2vec paper. The sentence

During the forward pass, $i = \text{argmax}_j p_j$ and in the
backward pass, the true gradient of the Gumbel-Softmax outputs is
used.

is there as well, but I cannot see any part of the loss function (in this paper) where the probabilities are explicitly used (such as the diversity loss).

Hello. Although the second question is clearly related to the first one, I would suggest that you ask it in a separate post so that people can focus on 1 question/problem at a time.

Thanks @nbro I only left the second question, as it is kind of clear to me now that the codebook is learned.

Ok, thanks. But can you be more specific about what your question now is? What's confusing about this statement that you want to be clarified? In other words, try to ask a more specific question.

That;s harder but will try :-)

@nbro Can you take a look now pls? I provided more of my thoughts, although the clarity/readability is probably worse than before.


text:
Is my Neural Network program fully connected?
I have the following program for my neural network:
n_steps = 9
n_inputs = 36
n_neurons = 50
n_outputs = 1
n_layers = 2
learning_rate = 0.0001
batch_size =100
n_epochs = 1000#200
train_set_size = 1000
test_set_size = 1000
tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
         for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons])
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:]

I want to know whether my network is fully connected or not?
When I try to see the variables, I see:
multi_layer_cell.weights

The output is:
[<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/kernel:0' shape=(86, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_o_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/kernel:0' shape=(100, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_o_diag:0' shape=(50,) dtype=float32_ref>]

I didn't understood whether each layer is getting the complete inputs or not.
I want to know whether the following figure is correct for the above code:

If this is not then what is the figure for the network? Please let me know.

Yes, it is a feed forward fully connected network.

@PatelSunil Can you please tell me why there is 86,200. Why 200? Why 86? I know 50 is the neurons per layer. Then why it starts with 100, 200? Why 100? I am a bit confused here. Please help me know.

Is figure = code?
No.
Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer.
Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1).

Why size 86?
The input is concatenated with the hidden state:
n_inputs + n_neurons = 36 + 50 = 86.
Why size 100 in the second layer?
The second LSTM layer gets input of size 50 by the first layer (n_neurons) which is concatenated with the hidden state of the second layer (of size n_neurons = 50).
Therefore you get 50 + 50 = 100.
Why size 200?
There are four weight matrices of size $86 \times 50$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($4 \cdot 50$) of size $86 \times 200$ (layer0/kernel).
Why size 50?
The three variables w_f_diag, w_i_diag and w_o_diag are for the peephole connections (fig: dashed lines) and they have the size of n_neurons = 50.

Just one more query. Is the model I am using back propagation and forward propagation? or is it just forward propagation?

The model uses backpropagation for weight adaption during the training process. It (LSTM) is not a feed forward neural network, but a recurrent neural network, as it has recurrent connections (blue arrows in the figure).


text:
Is there a programming training dataset for robot program itself
Where can I find training datasets like the ones provided for linguistic training, but to train a program to program itself.
I want to input this training dataset to a programming script and it should use it to program itself.
What do I need to consider in this kind of artificial intelligence?

I think for what you want to do, you need to focus more on evolutionary algorithms and reinforcement learning related algorithms. This link might help.

I feel reinforcement is tedious work and need to write policy for each game, is there a generic method that can apply to all by write once?

This is an open area of research, but it seems to be considered to be untenable given current methods.
This is relevant, and provides a link to a training set, but doesn't provide much more beyond that.
Specifically, an assumed requisite of this task is meaning comprehension, which we don't seem to be much closer towards given current methods.

On self improving AI
The way it's been done in the past is one AI turns a picture into text an the other turns text into a picture.
The AI generated material is added to known good material. The AI's job then is to guess what is real an what is fake (generated by AI).
At the point when the AI cannot tell in a false positive way both AI are said to be at their limit. If the limit is good enough you're done otherwise go back and code (more nodes or better function or more real data)
The thing to remember is you still need real data. The act of putting the AI together is good only to create more data then you could ever provide.
If that data is bad then you are going to rely on the cost function more then the data. But this allows you to label data real and AI so if the function rewards identifying the AI the AI's find each other's flaws and the better at avoiding flaws the better the AI get. (You still need real data! just less)
For example if two people cannot play ping pong if they play each other they learn slowly and only the cost function enforces their learning.
But have them play a pro (lots of good data) and they get better much faster.

I feel this training data is one to many, if focus on problem to solution style training data , can this be one to one ? Where can find these training data? What is the next step after find frequency of words in all programming script?

The reason say google is good at AI is because of it's ability to label data (reCaptcha).
If you want good a good AI you need good data. you can do it by hand search for what you are looking for but you need data.
It's why Mozilla did this: https://voice.mozilla.org/

I feel reinforcement programming is tedious because it need to write policy for each game, is there a generic method that can apply to all things it learn?


text:
Will/can AI be the worst enemy to humanity?
This is the question which always is being discussed. Will artificial intelligence be the undefeatable enemy for human beings?
Like the Slaughter Bots (Killer Microdrones technology) can kill a human or an living thing in just some seconds. And it neither could be deceived by its target nor it could be stopped by any source. Once it's set to kill the target, it can never be stopped and it identifies its target. It is also the innovation of AI.
See Microdrones technology introduction.

I highly suggest you read Hannu Rajaneimi's Quantum Thief trilogy, William Gibson's Neuromancer trilogy, and Phillip K. Dick's Do Android's Dream of Electric Sheep, Isaac Asimov's I, Robot.
It's a nuanced subject, and no one knows.
The most credible scenarios I've found for the destruction of humanity by automata involves not high-intelligence, but very low intelligence.  (The Grey Goo scenario.)  Dick, a gifted philosopher in his own right, proposes that empathy is a natural function of high intelligence, which provides a credible, increasingly validated ray of light.
But Hawking and Musk are credible imo per their comfort with very large numbers (complexity) and proven world-class abstract thinking capabilities.  It's possibly that strong "hyperpartizan" AI could run amok.  We've already seen issues with high-frequency trading algorithms, and automation is proceeding in weapons tech.
One might say "we are cursed to live in such interesting times."

We are not cursed to live in interesting times,but it's a characteristic of this civilisation.


text:
Is this ML task possible?
What I want to do is from an Internet challenge to transform any given image into the Polish flag using the available filters and crop tool on the iPhone camera app. Here's an example.
There aren't nearly enough of these videos to train a neural network using a labeled dataset, and (while I haven't ruled it out) I don't think automatically inserting a polish flag into an image then adding random filters to it to create my own dataset would work out.
My thinking is that I would feed a neural network the image and it would output a value for each filter & cropping coordinates. Then, I could easily calculate the loss by comparing the resulting picture to a picture of the polish flag. The obvious problem here is that you don't know how each of the neurons in the last layer affects the loss so you can't perform back propagation.
Is my best bet to mathematically calculate the loss (by this I mean as opposed to using high level libraries, which would be difficult but I'm sure it's possible) so I can find the partial derivative of each last layer neuron with respect to the loss function and then backpropagate? Would this even work? Are there any alternatives that you recommend?

I think the best thing to use here is a form of "structured prediction". Our "target" is a sequence of operations. The framework of structured prediction allows us to chain together as many filters as we want.
With a neural network of fixed architecture, you would have to make sure you have enough space for all the filters you might need.

Good idea! I was thinking of something like this but I wasn't sure if it would be practical. Thanks for letting me know what it's called and that it's a viable option. I realized that the order you apply the filters matters: for example, if you use a gradient filter then rotate the image 90 degrees the gradient will be perpendicular to the gradient when you rotate the image first. For this reason, would it be a wise option to train a neural network to make one change at a time (whichever change will get it closer to the end result) and then propagate through it multiple times?

My only concern is that it wouldn't be able to do any long-term planning and I don't know how important that is here.


text:
GAN for specific face attribute modification
A recent paper "MagGAN High Resolution Face Attribute Editing with Mask Guided GAN" published this month (October 2020) describe how an approach has been developed to deal with specific face attribute editing.
The thing is that in this paper and related work (StarGAN, CycleGAN, AttGAN, STGAN ...) seems to tackle the process of adding / editing a face attribute (e.g. adding a hat or mustache...), but not really modifying the face attribute (like eyes, nose, lips ...)
Is there anyway we can make a model that can edit for example the nose type/size or any related works already published?