text:
PPO: dealing with variable episodic length
I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have much larger advantages than actions in smaller episodes simply because of the cascading addition of future rewards/costs. Is there some smart way of dealing with discounted future returns in such scenarios? Thank you.

Generally you can make an agent care less about rewards in the future by decreasing the discount factor $\gamma$.

that this may cause problems with GAE, as actions in large episodes will have much larger advantage

Depending on the particular task you are solving this might not be a problem. If a longer episode gives an agent a higher expected return, then the agent should learn to prolong the episode. Or if the episode length is independent of the agent's actions, the agent should still learn to act optimally with respect to the random distribution of episode length.

The episode length is always predetermined (there are always certain options, but these options are taken changes the cost of the outcome). I wonder if there is a way to incorporate the length of the episode, which I already know, to the training...

You can always include this in your state space -- for example, you could add an integer feature for the known episode length, and an integer feature for the number of timesteps until episode completion.


text:
How to determine if a decision tree is the (globally) optimal tree?
BACKGROUND:  When constructing decision trees, the features are selected at various nodes based on whether it optimally splits the samples at that level (i.e., locally) using some user-chosen metric to judge "optimalness" (e.g., Gini index, Information gain, Entropy).  My understanding is that this is what is called a greedy algorithm, meaning that it optimizes locally but not necessarily globally.  As a consequence, the algorithm does not guarantee that the final tree is going to be the best tree.  Nonetheless, it would be desirable to derive the best tree or, at the very least, know if we have derived the best tree (even if we have no procedure for deriving the best tree).
QUESTION:  How to determine if a decision tree is the (globally) optimal tree?

There are statistical learning guarantees for some decision tree learning methods https://link.springer.com/content/pdf/10.1023/A:1013685603443.pdf

Decision tree for classification is essentially Boolean function approximator which is usually equivalent to a MLP neural network. Thus the usual Gini index, entropy based uncertainty reduction methods to select and order test attributes is akin to optimize MLP parameters based on the same training set. If the size of your training set is relatively small compared to your future test set, they both face the same common overfitting issue. But nonetheless you can always convert to its equivalent MLP form for “optimality”.
If you stick with decision tree method there’s no rigorous mathematical proof for your hoped "globally optimal" tree if your training set is relatively small or incomplete w.r.t your test set, and usually Gini index metric is more efficient to compute in most cases unless in some highly unbalanced data sets where one of the classification category is relatively unlikely and entropy formula can amplify attributes of such rare category. You may further read this 2004 paper summarizing theoretical optimality comparison between Gini index and Information Gain metrics for decision tree induction with the conclusion:

Based on these characterizations we were able to
analyze the frequency of agreement/disagreement of the Gini Index function and the Information Gain function. We found that they disagree only in 2% of all cases, which explains why most previously published empirical results concluded that it is not possible to decide which one of the two tests performs better... Based on the gained deeper insights on the split process we are currently working on a system, which will select the optimal criterion based on a user defined optimality criterion. Preliminary results can be found in [20].

And in the mentioned reference therein about their proposed family of split functions the authors also confirmed:

The tests have shown that the two popular functions are very sensitive to the variation of the training set sizes and therefore the quality of the inferred trees is highly dependent on the training set size. At the same time however, we were able to show that the simplest members of the introduced family of split functions behave in a very predictable way and, furthermore, the created trees were superior to the trees inferred using the Gini Index or the Information Gain based on our evaluation criteria.

One common approach in such case is using the idea of cross validating your existing training set, to divide it into multiple equal-sized subsets. Then for each subset to train your tree based on your chosen metric using the remaining sets and calculate some error rate using the said subset as test set. Finally to average all the error rates and you can get some idea of the degree of your “global optimality” for your specific candidate tree. Of course if you have several similarly performed trees which are all more or less uncorrelated, you can always use ensemble idea to do voting such as random forest.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Latest status on stability of CNN architectures to noise and "clever hans" issues
I was working with CNN architectures for image segmentation a few years ago, and I remember there was a big concern about on the stability of the predicted masks due to the introduction of small amounts of noise. If just a little gaussian noise was added to the image, then the predicted mask might change in important ways--generally around the margins.
Similar issues arose in classification problems, where MRI images were being classified based upon external queues within the images, and not based upon the images themselves--the "clever hans" problem as it is called. For example, sometimes the training images hospital labels that implicitly meant that an image was "positive" for some illness. Hence the CNN was using the label for detection instead of the data contained in the image. Here is a link to a 2022 paper that discusses the continuing challenge of Clever Hans problems in MRI images. There were also issues where the CNN classification of images did not generalize beyond the specific machines that the training images came from.
I was just wondering what the current status of these stability issues was within CNN architectures, or probably across Deep Learning architectures. In particular, are there any new architectural changes to help CNNs or deep learning architectures overcome these issues? Are there newer training methods, etc. For example, do contrastive learning methods improve on the stability of CNN predictions, etc. I am most interested in medical imaging, but input from any topical area is welcome.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
RL - Can RL be applied to problems where the next state is not the next observation?
I'm quite new on the study of reinforcement learning, and Im working on a communication problem with continuous large actions range for my final graduation work. I'm trying to use Gaussian Policy and Police Gradient methods for that implementation. I will try to explain the sequential logic of the task to better understand...
From the current observation (o_t), the agent takes actions reaching a new state (s_t+1). This new state is indeed a function of the previous obersvation (o_t) and the actions (a_t) taken by the agent. However, this next state (s_t+1) is not the next observation that the agent will take into account to take new actions, being only useful for calculating the reward in that iteration for the policy optimization. My question consists of the possibility that RL can be applied to this type of problem, where the state reached by the agent's actions in a given instant of time is not my next observation that the agent will take into account to take the next actions.

Not an expert on RL myself yet, but it seems you are describing a POMDP = Partially Observable MDP (as opposed to a vanilla MDP) which is used a lot in RL. So yes :)

Yes, there are RL algorithms (dreamer v1,2,3) that predict world model to simulate what (h_t+1) given its sampled action and previous state (h_t). It runs simulated world model during critic-actor so it can be much more data efficient. https://arxiv.org/abs/2301.04104v1

You can use mathjax on this site.

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
Can a convolution learn to generate fine details?
I'm trying to get a convolutional autoencoder to reconstruct images of a dataset with crisp details.
I've read in a couple places that convolutional autoencoders "naturally produce blurry images". This makes sense to me when the images are complex and the bottleneck is small : necessiraly, the autoencoder is learning some lossy compression, so you should expect to lose some details. However, I find it more surprising when working with very simple images.
I created a toy dataset composed of images representing circles, which vary only following three (randomly sampled) parameters : position (x,y) and pixel value. I'm trying out a very simple autoencoder, just to test the blurriness/crispness of details. It's just composed of a downsampling, an upsampling, and a convolution :
Code in Pytorch is :
# "Encoder" part (just a spatial downsampling)
nn.AvgPool2d(kernel_size = 2)
# Decoder part
nn.Upsample(scale_factor=2, mode = "bilinear"),
nn.Conv2d(1, 1, 3),

And I train the network on my dataset to reconstruct the original images, with MSE loss. Here is an example of an original image (left), and its reconstructed output at network convergence (right) :

Even though the image is very simple and loss is pretty low, the image is not crisp. I'm guessing this effect becomes even worse when stacking multiple convolutions, downsampling several times in a row, and adding a dense layer as the bottleneck (as in "real" autoencoders).
So I wonder whether part of the bluriness associated with convolutional autoencoders stems from the fact that convolutions are not a great tool to produce crisp details. Is it the case ? Is there another way to upsample images through the decoder (for example, I used to use transposed convolution instead of upsample + conv, and image quality has improved with this change. Is there another similar "step forward" ?) If convolutions are inadequate, how come other generating architectures (modern GANs) are said to be able to produce more crisp images ? Don't those also ultimately rely on convolutions ?

Hi @Soltius, and welcome to AI Stack Exchange! Right now, this post has many questions. If possible, could you please edit your post so that it has a single question? Thanks, and we look forward to having you on this site!


text:
Image classification problem with multiple right classes
I have a use case where the model needs to detect fabricdefects. There are 15+ different kinds of defects. In one image there can be multiple defects present. The straight forward solution for this should be a multilabel model from my understanding. The classification of the data for a multilabel model is extremely tedious and errorprone.
Now I use a multiclass model, which seems to produce ok results. A multiclass model has only one right output class. My goal is to add one image to multiple classes and if the model predicts one of these classes it should result in a correct prediction (or lower loss output).
For example if an image contains defect1 and defect2 the model should look at both these outputs and calculate the loss from the one that has the highest probabilty.
Now my question: "Is it possible to have a model where the samples can have multiple right output classes, but are not penalized by predicting only one right class?"

What if a network learns to predict that all of the images have none of the defects? Assuming that none of the images have all of the defects, some of the predictions will be right for all of the images, and thus perfect. But alas this is an useless model.

This is a good question.  There are definitely good reasons for wanting a loss function that evaluates whether at least one of the classes was picked up by the model.
To do what you are attempting, I believe you will have to write a custom loss function because this is sort of a "special request" kind of thing.  I do not think any built-in loss functions in Tensorflow/Keras can do this, and I'm not really adept with other ML packages.
This how such a custom loss function could potentially work:

Once you have set a threshold for calling positive for each of the
classes, you can convert your probability vector to a prediction
vector of $1's$ and $0's$.

Then, you can do an elementwise check for equality to the ground
truth (y_pred == y_true) and sum over all classes for which the defect is actually present (i.e., include $1 = 1$ but not $0 = 0$ in your tally).

If the sum is $>= 1$, then you know that the model found at least one
defect correctly ($y_{pred} = 1$).  Otherwise, the model did not predict the presence of any defects correctly ($y_{pred} = 0$).

You will have to sum over your original $y_{pred}$ vector, and if the sum $> =1$ (at least $1$ defect present), then the new $y_{pred} = 1$; otherwise, $y_{pred} = 0$.

Now, you have a new binary classification problem:  Predict if defect is present or absent.  You should be able to use binary crossentropy loss or another suitable loss function.

Here is a helpful page to get you started on custom loss functions in Keras.  Secondly, I'm sure you know this already, but it will be important to make sure that your classes are balanced after this "data transformation". Finally, you can set up this problem a little differently such that you have 3 possible outputs:  1) correctly predicts at least 1 defect, 2) correctly predicts the absence of defects, and 3) makes no correct predicts.
Good luck.  Hope that helps.


text:
Training tricks to improve stability of mixed precision
I would love to be able to use automatic mixed precision more extensively in my training, but I find that it is too unstable and often ends in NaNs. Are there any general tricks in training that people here have used to improve stability?
I've seen the following general tips:

plot the gradients and force unstable layers to fp32
bump weight decay in the optimizer
bump epsilon in the optimizer
add/try different normalization layers
force loss calculations to fp32


text:
Confused about NER evaluation
I'm reading from the book Speech and Language Processing by
Dan Jurafsky and James H. Martin and I've stuck for a while trying to understand what the authors mean by the following

The fact that named entity tagging has a segmentation component which
is not present in tasks like text categorization or part-of-speech
tagging causes some problems with evaluation. For example, a system
that labeled Jane but not Jane Villanueva as a person would cause
two errors, a false positive for O and a false negative for I-PER.
In addition, using entities as the unit of response but words as the
unit of training means that there is a mismatch between the training
and test conditions.

In particular, I don't see what's the problem with

For example, a system
that labeled Jane but not Jane Villanueva as a person would cause
two errors, a false positive for O and a false negative for I-PER.

Was is expected to cause one error or something? What if Jane Villaneuva was labeled as B-LOC I-LOC?
Thanks.

You left some critical context out. The author is referring to the following sentence:

[PER Jane Villanueva ] of [ORG United] , a unit of [ORG United Airlines
Holding] , said the fare applies to the [LOC Chicago ] route.

A NER algorithm should label "Jane Villanueva" as a person. The BIO labels would be B-PER for 'Jane' and I-PER for 'Villanueva.' Labeling 'Villanueva' as 'O' is incorrect and produces a false positive. Not labeling 'Villanueva' as I-PER is incorrect and produces a false negative. As Jurafsky explains, "any tokens outside of any span of interest are labeled O."
Labeling 'Jane Villanueva' as B-LOC I-LOC would also be incorrect; therefore, both labels would be false negatives.
The author assumes that the reader knows the BIO tagging labels. For a reference, see this page from Hugging Face. Here is an excerpt:

For those interested, the book is online here.

I still don't understand what's the problem with NER evaluation. Is he referring to that these should actually be counted as 1 error? Furthermore, why don't we say that labelling Jane Villanueva as B-ORG I-ORG causes 4 errors, false negatives relative to B-PER and I-PER and false positives relative to B-ORG I-ORG


text:
How the Critic is used to train the Actor in Actor-Critic network
I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.
But I don't fully understand how the critic output (reward space) can be used to train the actor sub-network.
How can a reward be used to train the actor sub-network to make it choose a better action when the actor sub-network doesn't get the reward as input or output?

As shown in my below answer, the only difference between actor-critic method and REINFORCE with baseline is the former also applies the same state value estimator to the next state of any state transition to form the TD error and thus is a kind way of assessing the latest done action before the state transition to further update actor network parameters, while the latter's state value estimator doesn't assess the action taken at any same iteration during an episode via any of the next state value at all to influence its next action.

Instead of receiving total return in the summation form of (discounted) rewards sampled from the full trajectory as in REINFORCE with baseline to update policy parameters, in actor-critic method to bootstrap, the actor sub-network usually receives a one-step return in the familiar form of temporal difference (TD) error $(R_{t+1} + \gamma \hat V_{\pi_\theta}(S_{t+1}, w) - \hat V_{\pi_\theta}(S_t,w))$ (or n-step) from the critic who evaluates estimated state value function optimized from same TD error.
Obviously from the computational point of view the policy gradient's "advantage" function from the critic's one-step or n-step TD error has much lower variance than that of REINFORCE with baseline from its sampled trajectory's total return, even if the former introduces bias.


text:
Is AI a threat to humanity?
By playing God and creating AI, are humans creating the catalyst of
their own eventual demise?
Why would AI, which is already infinitely smarter than humans do what humans tell it to do?

That AI "already [is] infinitely smarter" is not true. Don't confuse highly specialized algorithms with general artificial intelligence. It does not have sentience, consciousness and can not act on its own volition. The kind of AI you are worried about, if at all possible, is very far down the line and only theoretical.

It is not uncommon for people to be concerned about the potential negative impacts of AI on humanity. However, it is important to remember that AI is a tool that can be used for both good and bad. While it is true that AI has the potential to be vastly more intelligent than humans, it is ultimately up to us as the creators of AI to ensure that it is used in a responsible and ethical manner.
Furthermore, it is worth considering the potential benefits of AI for humanity. For example, AI can be used to improve healthcare, transportation, and education, as well as to address many other pressing global challenges. It is important to recognize that AI is not inherently a threat to humanity, but rather it is a technology that we must carefully manage and control.

There is no current "AI" that is infinitely smarter than humans. Current AI systems make mistakes that humans do not make.
So I would not be worried about AI taking over or being a threat, I am more worried of current uses of AI to undermine societal good, for example, using large language models to massively produce fake news, fake scientific papers, or fake QA answers, etc. This is already happening (see here or here), and in the end the problem is not the AI, but how humans actually use it.

There is a series here: https://www.technologyreview.com/supertopic/ai-colonialism-supertopic/ that talks about AI colonialism, but it is hard to define as its a relatively new concept, so it's best to read the articles on this link.
Then there is AI bias which is kind of like AI discrimination, this goes often unheard of / hardly talked about, especially in terms of recruitment in my opinion.See this article for what I mean: https://www.bbc.co.uk/news/technology-63228466]
I think these two dynamics form a good 'present' basis to hypothesise an eventual AI take over of humanity, this is my own rough opinion that follows to answer "Why would AI, which is already infinitely smarter than humans do what humans tell it to do?" Honestly, I don't think it would eventually, the dynamics of 'super intelligence' most likely would make it ignore us or alternatively not so much enslave us but attend to our every need and manage resources / invent better resources and such to create a type of utopian society I guess politically it would be "Fully Automated Luxury Communism"? but anyway I know this sounds like ramblings but its more my own opinion based on an extensive academic background in AI specifically and computer science and a result of me doing a lot of reading. To look at how a human would 'not' question an AI and do what its told, look at the domestic dog, over time they have evolved a genetic anomaly similar to Williams syndrome in humans. Basically they do what humans tell them to for the most part.I can see a similar thing happening with Humans, but I think it would take too much of a long time to happen I think roughly 6000 years for us to evolve to that point I think possibly to become domesticated under an AI system. So I think realistically, AI will continue to either do what humans tell it to or it will simply see no other choice but to ignore us / span off from us.


text:
How many more years until AI replaces all programming jobs?
I'm curious to know from those who have a clear understanding of both

AI
the software development process

How many years until software engineers, developers and programmers are no longer needed in the workforce because AI can do their jobs sufficiently well? And produce whatever is needed in the market?

I do not think this will ever happen, we do not have true AI (or AGI), current models are just parroting the training set code. they do not really analyze or think about code over human capabilities.
There are also huge ethical and legal issues about AI systems for code, since they were trained on large datasets of open source software, with different licenses, and again models do not output licenses for their code, so you cannot use it commercially.
Not even humans know how to make software properly, there are huge issues in terms of scalability, proper software development practices, and integration with other systems. Current AI models do not know anything about this.
So nobody can tell you how many years left, because it is an event that nobody knows if will ever happen, same for autonomous driving, it is always "two years away".

Having tried out Chat GPT and Github auto pilot recently to help with game engine programming in C++ I was not impressed, it produced terrible code. Honestly, it is hard to say really. It could be a possibility way in the future much like prompting today, that humans create or use an existing logical language like https://www.lojban.org/mw-redir.html lojban here to communicate with machines better, so rather than programming them we bring them into society through communication and such much similar to the likes of bringing up a child, that then AI entity would be added to other entities. Possibly, it could more complicated as well if humans were to be enthused with something like neuralink in the future, would humans need programming also? But overall I agree with the similar answer, it is always two years away or something like that, there's too many questions at the moment and not enough answers to say for sure.


text:
Split dataset into Train/Validation/Test for Object Detection
I have a dataset for Object Detection with YOLO format labels, each imagine can have occurences of different classes and multiple occurences of the same class.
How can the dataset be divided into Training, Validation, and Test sets so that each contains about the same percentage of occurences per class?

You can use the scikit learn train_test_split() by passing the stratify argument with the class value.

This can become a more complicated problem , if there is imbalance in the distribution of classes.


text:
What underlying network is typically meant with ResNET?
When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks or CNNs? Are there applications of ResNets to RNNs or is that pointless because they are conceptually similar (https://arxiv.org/abs/1604.03640v1)?

I think the original implementation was with Deep CNN's, but I think the idea also makes sense for MLPs -- possibly other architectures, but I would need to think more about this.

@DavidIreland
I looked it up and it has indeed been applied for MLPs as well (see e.g. here: https://arxiv.org/abs/2105.03404). Then probably CNNs and MLPs are the most reasonable application areas. Have you had the chance to think about it more or do you agree?

I think that the official paper that introduced residual connections named their network(s) ResNet, so when somebody says ResNet I would assume that they mean an architecture following the original, i.e. convolutional layers with residual connections in between (see here if you want more details about the architecture).
If I want to describe a network architecture with fully-connected layers that uses residual connections I would say exactly this.. fcnn with residual connections.
Regarding RNNs, I think that the design of the LSTM cell greatly resembles the highway networks. In fact I think that the highway networks were inspired by the LSTM (also both are authored by the same guy Jürgen Schmidhuber). For sequential tasks I would say that using gated summation makes more sense than the simple summation that residual connections use. However, I think that there are some ideas to apply residual connections between the layers of a multi-layer rnn.


text:
Why does AlphaZero not use vanilla MCTS?
I understand main difference between AlphaZero and the classic Monte Carlo tree search is the playout (simulation) step is replaced with a neural network prediction which itself is trained from the output of the MCTS. How does this additional complexity improve the performance?
My guess is that classic MCTS would not perform worse than the AlphaZero's hybrid approach on a system with unlimited memory. Since memory is a constraint in the real world, the neural network is a work-around.

I think you are partially right, the constraint is basically that we can't evaluate every position due to computational restraints.
The neural network in Alpha Zero is basically trained to identify which moves should be explored more.


text:
How can an Artificial Intelligence system be ethically trained to generate art?
There have been a lot of popular AI-generating image systems put out recently, with such systems as Midjourney and Dall-E catching attention with how well put-together many of the automatically generated images are.
However, there has been a lot of pushback to these systems. This is largely because apparently the training data they were fed included lots of art that was used without the creators' consent. Sites such as DeviantArt and other image-sharing platforms were apparently scraped for training data without regard for the consent of the original creators or licensing.
Since these images were used as training data without the appropriate licensing, the AI systems that is benefiting from and using that data could be used to generate images that are then used commercially, which would violate such licenses as CC BY-NC-SA, and no attribution or credit has been provided for the original artists.
How can this problem be avoided when training an AI system? How can you ethically compile a comprehensive training data set for AI image generation?


text:
How can I learn about NN architecture?
I have a pretty good understanding of individual neural net layers (fully connected, convolution, pooling, activation, etc) but struggle to construct combinations of them to solve a given problem. I know there are common "off the shelf" architectures for certain problem types but often times they don't do great if your data is considerably different. Is there any "method to the madness" or is it an acquired skill? Any resources would be appreciated!

Does this literature review help?
https://doi.org/10.1016/j.neucom.2016.12.038


text:
Effects of hyperparameters in Q-learning
While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.
Firstly, I noticed that increasing the learning rate increased the variance in total reward from the optimal policy. A low learning rate always gave the same total rewards when I ran the algorithm, but a high learning rate yielded different total rewards. Why is that?
I could see this same behavior when decreasing the discount factor. I once again do not understand why.
Secondly, I saw that increasing the discount factor resulted in longer learning time. I realize why increasing the learning rate increases learning time (number of episodes until convergence) by definition, but not necessarily why this would hold for the discount factor as well.

What's your question here? Are you asking for general guidelines on the behaviour of Q-learning changes as you change these hyper-parameters? Please, rewrite the title so that it's your actual question. Someone below provided an answer that talks about the discount factor, but, actually, you didn't ask any question about the discount factor. You're just describing what you observed or your problem.

Discount factor in (tabular) RL including Q-learning generally acts as a regularization hyperparameter to trade-off optimality with sample efficiency especially for continuous tasks with infinite time horizon. Increasing $\gamma$ close to $1$ will evaluate each of agent's actions based on the convergent sum of almost all of its future rewards which obviously results in longer learning computation time. In practice like $TD(\lambda)$ oftentimes you need to find a balanced discount factor between $0$ and $1$ per above mentioned trade-off.
On the other hand it's a common knowledge in RL with large action space necessarily employing function approximation for control, discounted rewards RL formulation is not even an optimization problem as referenced in papers such as Average reward reinforcement learning: Foundations, algorithms, and empirical results by Mahadevan (1996), and Discounted Reinforcement Learning Is Not an Optimization Problem by Sutton et al (2019).

Discounted reinforcement learning is fundamentally incompatible with function
approximation for control in continuing tasks. It is not an optimization problem in
its usual formulation, so when using function approximation there is no optimal
policy. We substantiate these claims, then go on to address some misconceptions
about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks


text:
Machine learning for arranging 2D points
I have a problem wherein I have 2D points in an image that would be associated with a corresponding label/sequence number. For instance following are 4 such examples:

As you can see all of them have a certain sequence. As if a human were to read them (left to right, top to bottom) but in some cases, it's a little more complicated (see bottom right example above)
As of yet, I am using simple X-Y co-ordinate based sorting to figure out the sequence but it doesn't always work (imagine example top left above, but all the points are arranged at an angle) I'm trying to approach this from a machine learning angle and curious to know how would you do it.
Another challenge is there could be arbitrary number of such points (anywhere between 4 to 16)
We can for instance, use a Neural Network, or an SVM based classifier where the features would be normalized x,y co-ordinates, but I'm wondering if there's a simpler way to do it. Furthermore, I will have to use a lot of augmentations since the output has to be permutation invariant.
I have looked at Geometric Deep Learning based point segmentation methods like PointNet, PointCNN, and so forth but these methods mainly work with point clouds, and would be an overkill for my purpose.

To me it looks like you could cover this change of the labeling strategy (top-left vs. bottom-right) with a clustering method. First, look if there are clear clusters, if yes number each cluster separately, if no clear clusters then just label 'line-by-line'. If you can use a bit more domain knowledge (where do the points come from; where do the labels come from; ...) try to do that as well.

And a small heads up: You wouldn't want the output to be permutation invariant, but permutation equivariant (as you want your labels to permute similar to the points). Invariant would mean that you get the same result for permuted input.

Yes, would want it to be permutation equivariant +1

One solution can be a computational geometric approach. First grid the plane of points with the desired size, denoted by $\epsilon$. Then, row by row, find the points (you can make an appropriate data structure, for example, KD-tree, to have an efficient search). Now, you can decide how many rows of gap are required between two sets of points that should be counted until starting a new line of points. Finally, according to this row index for each point, rank them from 1 to $n$, from left to right of each line (sort all points of each line based on their $x$ coordinate value).
There are two clear advantages to this method:

it is easy to be implemented.
it can be easily employed to work for any number of points.


text:
How to use image feature extraction as input to another model?
I have a dataset with the following info
Image1 x1 x2 x3 y
Image2 x1 x2 x3 y
...
Where x1, x2 & x3 are categorical features. My goal is to extract features from the images and use those features combined with x1 x2 and x3 as an input to another model to classify y.
My question is the following. Assuming I use a pre-trained resnet model, should I retrain the last layer using my images and y's to fit the resnet better to my problem or should I directly use the features extracted from the resnet as an input to my other model without retraining resnet?

Most pre-trained ResNet are pretty capable of returning a rather informative feature representation, even without fine-tuning. This, surprisingly, seems to be the case also for image datasets that do not resemble ImageNet. For example, I've used the feature representation of ResNet18 (the most basic ResNet, if you will) of the MVTec dataset, and the representations were pretty impressive (in terms of how well they were classified later on in my pipeline).
Of course, one can usually expect better results when finetuned to a specific problem, but I would start off with the representation vectors as is, and see how they perform.


text:
Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?
The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:

The cross attention mechanism within the original transformer architecture is implemented in the following way:

The source for the images is this video.
Why are the values in this step coming from the encoder instead of from the decoder? Is this where e.g. the actual language translation happens within a transformer?

The transformer architecture contains a cross attention mechanism which is
enriching the encoder with information from the decoder. The place where this
takes place is visualized in the image below:

I think that you got it the other way round. The encoder passes an enriched input sentence to the decoder. Cross attention helps the decoder to attend to every part of this "enriched" input and produce one output at a time recursively.
Initially, the decoder's first prediction is fixed to  (start of sentence) token. That gets self attended first, then get attended with encoder's output (the "enriched" input) and gives out a prediction from the word vocab list. This word gets appended to the decoder's input and we repeat the process again.

My intuition is that the decoder essentially performs a dictionary look-up where the encoder acts as the dictionary that provides keys and corresponding values. In this way, the decoder can "ask" for relevant information to fulfill its needs.

The idea of the cross-attention layer is to transform the input words to output words. The Decoder provides context of which input words should we pay attention to next based on the already decoded text.
If the values in the cross-attention came from the decoder as you suggest, we would have to have access to the already translated sentence to translate the sentence, which is absurd.
For example, suppose we are going to translate the sentence "में बैडमिंटन खेलने जा रहा हूँ " in Hindi to English : "I am going to play badminton".
And suppose we are have already translated "I am going to play _" and are looking for the next word : "badminton". In this case the decoder self-attention guesses that the next word is going to be a sports name. The query cross-attention linear layer now transforms the input vector to align with the corresponding vector representation in Hindi for sports names. Now amongst the input words the hindi representation of 'badminton' aligns the most with query vector and thus we know that what the next word to transform to English is.
Now what is left, is to do the actual transformation from Hindi representation of badminton to the English representation. The value linear layer handles that part.
The dot product is able to work because the vector representation of words follows a pattern as shown by the papers referred here : https://kawine.github.io/blog/nlp/2019/06/21/word-analogies.html


text:
Is there a neural network method to encode a directed graph?
I want to do a graph classification task. Those graphs are directed, and their edges have features. I knew little about graph representation methods, but I did some research, and find most of the works for graph classification seem to be based on undirected graphs. So is there a way to encode this kind of graph? Or any suggestions to encode this kind of graph?

Most methods generalize naturally to undirected graphs, some of the simple examples are GCN [1], GIN [2] and GAT [3]. There are more sophisticated methods, but depending on the task, often they don't perform drastically different [4].
It really depends on you specific task what works best for you. You might want to try some different architectures and I'd start with the GCN or GIN. You could opt for a nice library that allows benchmarking all the different architectures quite quickly like deep graph library (DGL) or Torch-Geometric.

[1] Kipf and Welling (2017), Semi-Supervised Classification with Graph Convolutional Networks
[2] Xu et al. (2019), How Powerful are Graph Neural Networks?
[3] Veličković et al. (2018), Graph Attention Networks
[4] Shchur et al. (2021), Pitfalls of Graph Neural Network Evaluation

Thanks very much. This answer gives me clear guidance about what to do next.

Nice, I'm glad I could help :)


text:
Fine Tuning Transformer Model for Machine Translation
I am working on the Transformer example demonstrated on TensorFlow's website. https://www.tensorflow.org/text/tutorials/transformer
In this example, Machine Translation model is trained to translate from Portuguese to English. The transformer is coded from scratch and other popular libraries like huggingface are not used.
Let's say I have another dataset which includes pairs of sentences of Portuguese and Finnish and let's say this dataset is fairly small. Since it is a small dataset, I want to use my model trained on Portuguese to English as a PreTrained model for creating the translation model for Portuguese to Finnish.
My question is, what are the key points to consider when using such a PreTrained model and changing ONLY its decoder output structure?

Transfer learning in machine translation is a relatively common technique in machine translation. Mostly, it means fine-tuning pre-trained self-supervised sequence-to-sequence models, such as mBART. It is also often used for low-resource languages to transfer from a related high-resource language, as e.g. most of the participants in the WMT21 low-resource competition did.
What you suggest is very close to a 2018 paper by Kocmi and Bojar. You might use their setup as a starting point. The main challenges addressed in the paper are:

Carefully setting the learning rate schedule to avoid catastrophic forgetting.
Do something about the vocabulary mismatch (Finnish uses a different vocabulary than English).


text:
NLP: Question answer with 2 contexts
Is there a Hugging Face Transformer that takes 2 contexts as input for question answering? For example, I could have transcript of a meeting in first context and agenda of the meeting in the second context. Then ask a question - Did the meeting adhere to the discussion topics given in the agenda? Alternatively, can you suggest how to solve such types of question answering problems that might require multiple contexts?

You could always just try and combine the two contexts right?

yeach I could do that too. But a transformer with 2 separate contexts is a lot cleaner IMHO.


text:
Why are policy gradient methods more effective in high-dimensional action spaces?
David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the implicit policy (e.g., $\epsilon$-greedy) in Q-learning or Sarsa looks for a maximum for $Q_\pi$ in the action space in each time step, which may be infeasible when there are many actions.
However, from my understanding, policy gradient methods that use a neural network to represent the policy choose their actions from the output of a softmax:
$$
\pi(a|s, \theta) = \frac{e^{h(s, a, \theta)}}{\sum_b e^{h(s, b, \theta)}},
$$
where the denominator normalizes between all possible actions $b\in \mathcal{A}$ and $h(s, a, \theta)$ is a numerical preference (notation from the Reinforcement Learning book by Sutton and Barto) that is usually represented by a neural network.
By using a softmax in action preferences, don't we compute $\pi(a|s, \theta)$ for all actions, ending up with the same computational complexity as in value-based methods?

Different but interesting related question

Above softmax in action preferences is used for policy gradient methods with (large) spaces with discrete actions, while for continuous spaces with infinite number of actions Gaussian distribution is often used instead thus you don't need to compute all those potentially large amount of numerical preferences via NN or whatever models.
In the above softmax case sometimes you may simply fit the preferences as linear function of the feature vector $x(s,a)$ which is often much simpler than computing action values via Monte Carlo, TD or DP which all have much higher time complexity. Indeed in these cases policy gradient methods are more effective to learn since policy function is in fact much easier to approximate than action value function or certain types of domain knowledge can be incorporated in policy function as explained in the paper Why Most Decisions Are Easy in Tetris.

most of the problems are easy in the following sense: One can choose well among the available actions without knowing an evaluation function that scores well in the game. This is a consequence of three conditions that are prevalent in the game: simple dominance, cumulative dominance, and noncompensation. These conditions can be exploited to develop faster and more effective learning algorithms. In addition, they allow certain types of domain knowledge to be incorporated with ease into a learning algorithm

Finally if your softmax has to employ action values to compute the preferences such as in most actor-critic methods, then yes of course it has same computational complexity as its corresponding value-based methods during each time step which you cannot avoid. But additionally softmax would allow agent to explore different actions by sampling from the resulting distribution, rather than always select the action with the highest estimated value, thus it can find stochastic optimal policies in problems with heavy function approximations while "deterministic" value-based methods cannot.

It's not the complexity of the function approximator I'm confused about. Rather, at "inference" time, the softmax output vector has the same dimensionality as the number of actions for a given state, so the complexity of choosing an action ends up being the same whether we're using a policy or value-based method. (Apparently that's not the case and policy-based methods are more efficient, I just don't see why.)

See my latest edit if that helps.

Thanks for the reply! So if I read correctly, you don't think policy-based methods are more efficient at choosing an action than value-based methods (setting aside the complexity of learning the preferences themselves)?

The complexity of learning the preferences themselves using action values in softmax as explained above is of same complexity as, say, Q/sarsa-learning. Setting aside this complexity, once you have the (approximated) softmax at hand in each time step, then you just need to sample one function value from it to select an action to proceed to policy parameters update, no need to worry about all the other numerous actions.

Additionally, though with same comupational complexity as value-based apporach in your case, the softmax function has the advantage of being able to smoothly interpolate between exploitation (selecting the action with the highest action-value) and exploration (selecting actions randomly with a probability that is proportional to their action-values). This can be useful for improving the efficiency of learning by allowing the agent to balance exploration and exploitation.

What do you mean by sample one action from the softmax? I think that's where my confusion lies. The "output" of the agent (which we can think of as a neural network) is a probability distribution over actions, which we can think of as a vector of probabilities. Then to select an action for a given state, we do a forward pass through the agent (network) and get a vector of probabilities (or preferences) for each action. To then select an action, we need to search the entire vector looking for which action is assigned the greatest preference. Is this correct?

softmax function just acts like a scalar probability mass function, not a vector of probabilities. It's like a binomial distribution histogram, then to select an action during each time step you just need to randomly sample according to such a probability distribution.

We can think of a probability mass function as a one-to-one mapping from each possible value (in this case, each action) of a random variable to its probability. In that case, we can always know the probability of choosing a particular action, but how can we "sample" (i.e., choose an action) without checking the probabilities of all the actions?

Once you get the (approximate) softmax as probability mass function via whatever means, there're many statistical ways such as inverse transform or Metropolis-Hasting to do sampling according to a specific probability distribution. In practice such as using python we can use random or scipy.stats packages such as random.gauss(5,2) or scipy.stats.binom.rvs(n=10, p=0.5) which hide all the algo complexities.

So the reason that policy gradient methods are more effective for high-dimensional action spaces is that they can use these algorithms for efficient sampling from a PMF, while $\epsilon$-soft policies are stuck with having to look at all the possible actions to choose the one that maximizes value?

Exactly makes sense for me!


text:
How to determine alignment / correlation with comparison based judgments?
I have built an automatic metric (using DL) to estimate some property of text. I want to evaluate its performance by comparing it to human judgments with respect to that property. As far as I know this is usually done using correlation. I found a related data sets. The problem is that the property is not annotated on an absolute scale but with comparisons.
Imagine for example something like this:
Text_A < Text_B < Text_C;
Text_D < Text_B < Text_D;
...

(The text are ranked according to the property. Each Text can occur in multiple compares, but I think not everything is compared with everything.)
So the question is: What is the way to get the correlation? (Or what is the best way to judge the alignment?)
I would be happy about every hint (including suggestions to improve the question)!

Further Information:

I assume such data might originate from using Best-worst scaling (but that's not explicitly stated in the source).
I assume using a rank based correlation fits more than the standard correlation.

My attempts:

An idea is to average the correlations of each set of compares. But I haven't found a way to perform the average in an exact way. Fisher z transformation seems to be only an estimate and as I have a lot of individual correlations with just 3 data points each, I assume the bias would be quite significant. Does anybody know a way to exactly average correlations for sets of equal size?

I was thinking about using accuracy instead of correlation, as it seems easier to aggregate. Just convert everything into binary compares:
Text_A < Text_B < Text_C;
->
Text_A < Text_B; Text_B < Text_C; Text_A < Text_C;
But I am not sure whether this is appropriated. What are advantages of accuracy and correlation, why would one prefer one over the other? What is their relationship?
Using a small R experiment I figured out that accuracy and correlation seem to be closely aligned but not perfect, see code on github. (The experiments did not consider ties in the the simulated ranking data.)

I ended up with converting the compares to an absolute scale using something like ELO scoring from chess (Wikipedia, a paper), than you can just use correlation as usual.


text:
Best way to generate a human face over a face generated by FaceFormer framework?
FaceFormer framework generates a talking face from audio, focusing on the lip and face movement when a person talks. Now from that what would be the best way to generate a human face on top of that?
I tried DeepFaceLive that didn't give good results as that swaps face only.
The first image is given by my FaceFormer (is was a video actually ) and when you use DeepFaceLive (deep fake) to replace it looks like a genie in the bottle.
What would your suggestion be around it ??
Thanks

It is not clear what you mean by "a face generated by FaceFormer framework," as I am not familiar with this specific framework. However, here are some general approaches that you could use to generate a human face:
Use a 3D modeling software: There are many 3D modeling software programs that allow you to create a human face by manipulating geometric shapes and surfaces. These programs often have tools that allow you to adjust the shape of the face, add facial features like eyes and a nose, and apply textures and materials to create a realistic appearance.
Use a face generation tool: There are several tools and frameworks available that allow you to generate a human face using machine learning algorithms. These tools often require you to input a set of parameters or characteristics, such as the shape of the face, the color of the eyes, and the style of the hair, and the tool will generate a face based on those characteristics.
Use a database of pre-generated faces: Another option is to use a database of pre-generated human faces and select one that closely matches the face you want to create. These databases can often be found online and may include a variety of different facial features and styles.
I hope these suggestions are helpful! Let me know if you have any other questions.

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

hello there
Thanks for the reply. Can you please give me an example of the face-generation tools and databases you're talking about? For I cannot find much which is relevant to me :(


text:
How can I formulate a usecase with an additional constraint as a reinforment learning problem?
I am new to the field of reinforcement learning, and I feel a recent use case of mine is highly relevant, but I don't know how to forumate it as a typical reinforcement learning problem.
Let's say I have an actor model. The input to the actor is a state obtained from environment, and the output of the actor is a discrete choice of 3 actions (each action is associated with an action cost). My target is to main a certain of score from enviroment but try our best to achieve the smallest accumulated action cost.
For example, the actions can be A={"use gun": 1000}, B={"use knife": 100}, C={"escape": 10} which basically means if you use gun to fight an enemy, it is supposed to be effective but the cost is 1000, however, if you choose action escape, the cost is low, but it won't harm the enemy.
What I hope to achieve is to :
(1) At least win the game (i.e. open the front door of the castle), and
(2) target the lowest total action cost.
For example, if I can do "use_knife" twice and "escape" twice to open the door, I should not do "use_gun" to open the door.
I am a beginner of RL and I feel normal RL exemplary topic is about to select actions to achieve the highest score, very straightforward. So I was wondering if my above description can be achieved by some standard RL algorithms. Can someone provide some guidance on this? Thanks in advance. I will really appreciate it if you can point me some exemplary repo or papers : )

You could use the cost of each action as a penalty on the reward at each step. So for instance if your agent uses the gun, you could add a reward of -0.1 and using the knife say a reward of -0.01. The positive reward obtained could be 1.0 if the agent is successful at a given step. You will have to play around with these values to see which works best. This is a type of reward shaping.


text:
Does it make sense to build a two-layer neural network with a triangular weight matrix?
I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will  be used?

Triangular weight matrix could be much weaker to approximate complicated functions, unless your computing resources are limited you may try it.

I did some testing on nanoGPT (https://github.com/karpathy/nanoGPT) using triangular weights matrices instead of full one and the result were interesting : using triangular matrices seems to have a very limited impact on the learning (for gpt2).

(black and orange curve are the learning for triangular matrix and the pink one is the full linear one)

I would bet It perform the same as dropout 0.5

My bet (but i'm not sure) that it will not be the case ... dropout randomly choose some activation to shut down. With triangular matrix you have a deterministic behaviour.
Also the advantage of triangular is that you can hope to find better gpu kernel to do your triangular matrix multiplication (but I will not do that, I will not code in cuda :)).

Constraining weights in a neural network, either a hard constraint like your idea, or a soft constraint that adds to the cost function, is often a form of regularisation. By limiting the form of acceptable solutions, regularisation makes it harder, often impossible, to exactly match training data - this is a good thing because what you most often want to learn is a generalisation of the examples that works well with new data. Forcing a machine learning process into a space where it can approach but never reach a perfect solution can make good generalisation.
That means you might use such an idea to combat overfitting. The utility of the idea will vary depending on the problem. There may be no general use for it, because it's never better than other well established approaches. But there could be some problems it is particularly suitable for. You would have to try it and see.
The main problem I see with the approach in general is that it's all or nothing. You cannot easily have an arbitrary fraction of the diagonal mask.


text:
How can I put more weight specific words when doing text classification?

The picture shows the model I want to implement. As you can see, the input will be a single sentence (e.g. I want to massacre those ugly people!). Then, the teacher model will output weights(which will be 0~1 float values) to more important words to classify (e.g. The model will give more weights to words like massacre and ugly for classifying the sentence to be hate-speech). Then, the student model takes those weights from the teacher and the same sentence, and the weights will be a hint for student to know which words it should focus and classifies the sentence.
What I want to know is that how can I make student model to focus several words than others? Just multiplying weights to the word embeddings naively doesn't seem a good solution I think. Please give me some insight. Thank you.

it seems to me like the first model would just learn the equivalent of an attention mask over a sentence. So my question is, why training two models instead of a single one with attention layers? It sound like you're making your life harder this way.

@Edoardo Guerriero Label of first model will be the LIME values which are precomputed so that teacher model can mimic LIME values of one sentence. Would it be more helpful if I just use attention layer in the student model instead of using such LIME value?

if the LIME you're referring to is the algorithm to infer feature importance from an already trained classifier than yes, I would skip that part completely and train a model which include attention layers in its architecture from scratch. First reason: if you have access to the LIME weights of an already trained model then you already have a trained model, I don't see why you should spend time training a model that emulates another one, especially considering that you can't get better performances due to error propagation.

Second reason: LIME is an interpretation algorithm, using it to generate labels is a huge conceptual stretch. But maybe I misunderstood your intention, I think you should at the very least add the LIME part to the question to make it more clear.


text:
What is a heavy node in neural networks?
I was watching a documentary on Netflix about AlphaGo, and at one point (~1:10:16 from the end), one of the programmers uses the term "heavy node," which I assume has to do with neural networks. I did a little bit of research but couldn't find anything on what that term means. The closest I could get was this wikipedia page on Heavy path decomposition: https://en.wikipedia.org/wiki/Heavy_path_decomposition, which seemed like it could be somewhat related, but I wasn't sure how exactly. Has anyone heard of this term being used? Does anyone know what it means?
For context, in the documentary the line is that if it (the network/player) creates something new not in the heavy node, then they don't see it.

I'm not aware of what a heavy node is. I remember watching this movie and that term didn't catch my attention. If you're interested in some of the inner workings of AlphaGo, you could check out their latest paper -
http://www.mooc360.com/agz_unformatted_nature.pdf (note this is AlphaGo Zero - the successor to AlphaGo featured in Netflix)

Yes, firstly you are correct to acknowledge the term node with reference to the neural network.
A neural network is made to mimic the working of a human brain hence the term neural. It comprises of several layers and each layer comprises of several nodes. A network can be a deep neural network, signifying the presence of numerous layers, in which the output of the one, serves as an input for the other.
Further a node in a particular layer has some weights and biases attached with it that are responsible for the computation and producing output.
To your context, a heavy node may refer to a node that holds importance for the network and has got significant weight and bias attached. Henceforth as per the statement if the network does make any significant change to the node with importance in terms of weights and biases, it is noticed else not.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
RL agent's view of state transitions

The above environment is DeepTraffic
Now consider this situation in the above environment, the Red car (we control it with our RL agent) is on the extreme right lane.
During the exploration phase, we take a 'move right' action, which ofcourse will result in the car not moving right,but the other cars will be moving, state changes due to the rules of the environment.
I'm using CNN to solve this, the state representation is the image itself and its a Q-Learning algorithm as described in DQN paper from deepmind.
In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?
and when remembering the state transition (s,a,r,s') should i remember the actual action 'move right'(invalid) or 'do nothing'(correct as per env) ?

If the agent goes right as opposed to donny nothing, does it get a different reward?

'Do Nothing' action has 0 reward on all states. In this situation go-right will also have a 0 reward, since no damage is done.

In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?

Generally the agent does not need to "think" what the next state will be. After taking an action, it should observe the reward and the next state.
The logic for the next state is controlled by the environment. In your case, the environment of the simulator should accept the action choice and then not allow the sideways movement off the road, and present the correct logical next state. It may in addition alter the reward signal if you want to penalise this choice (in comments you say this is not done - that's fine, it's a choice you make on problem set up, what behaviour to reward or penalise).
A more advanced agent might also try to predict what the next state will be, especially if it was using a planning algorithm (e.g. Dyna-Q). Even then there is no reason or rule that it should have to predict an impossible next state - a well-trained agent should in fact predict correctly that moving right is same as "do nothing" when the car is in far right-hand lane. For a value-based approach, like Q-learning the agent may do this simply by scoring the two actions with the same value.
In your case, you do not need to try and second-guess what the agent will predict. Provided your state information is enough for the agent to observe that it is in the far left or far right lane, it should [eventually, after training] correctly predict the value functions and/or optimal actions as it learns the environment. That includes, after a few examples, that it will predict the outcome when moving right when already on right hand side. For that to happen reliably, you do need to check your state representation is adequate, and function approximation hyper-parameters are suited to the problem (the latter part could take some experimentation)
If the state data that the car receives does not contain enough information for the agent to tell that it is in extreme left or right lane, then you will have a problem. This is entirely possible in general, although from the image you show, that does not seem to be an issue.
One possible related issue is if the other cars in the scene all have different velocities. This velocity information is not shown in the state, and could cause your agent problems as it cannot tell the difference between a stationary car in front of it, and one which better matches its own velocity. You could treat this missing data as a partially observable state, and adjust you agent. Or you might be able to do the same as the Deep Mind DQN team for Atari games and use a sequence of e.g. last 4 images as the state, which should then include enough velocity data to make reliable estimates of state/action values.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Data-set values feature scaling: sigmoid vs tanh
As many papers point out, for better learning curve of a NN, it is better for a data-set to be normalized in a way such that values match a Gaussian curve.
Does this process of feature normalization apply only if we use sigmoid function as squashing function? If not what deviation is best for the tanh squashing function?

Yes it applies, and no it shouldn't matter that much between the two activation functions.

So the difference between the two is basically whether the range of data is between <0; 1> or <-1; 1>?


text:
What was the average decision speed pf Alpha Zero in the recent Stockfish match?
The match got a lot of press, and I doubt anyone is surprised that Alpha Zero crushed Stockfish.
See: AlphaZero Destroys Stockfish in 100 Game Match
To me, what's really salient is that "much like humans, AlphaZero searches fewer positions that its predecessors. The paper claims that it looks at "only" 80,000 positions per second, compared to Stockfish's 70 million per second."
For those who remember Matthew Lai's GiraffeChess:

However, it is interesting to note that the way computers play chess is very different from how
  humans play. While both humans and computers search ahead to predict how the game will go on,
  humans are much more selective in which branches of the game tree to explore. Computers, on the
  other hand, rely on brute force to explore as many continuations as possible, even ones that will be
  immediately thrown out by any skilled human. In a sense, the way humans play chess is much more
  computationally efficient - using Garry Kasparov vs Deep Blue as an example, Kasparov could not
  have been searching more than 3-5 positions per second, while Deep Blue, a supercomputer with
  480 custom ”chess processors”, searched about 200 million positions per second 1 to play at
  approximately equal strength (Deep Blue won the 6-game match with 2 wins, 3 draws, and 1 loss).
How can a human searching 3-5 positions per second be as strong as a computer searching 200
  million positions per second? And is it possible to build even stronger chess computers than what
  we have today, by making them more computationally efficient? Those are the questions this
  project investigates.

[Lai was tapped by DeepMind as a researcher last year]
But what I'm interested in at the moment is the decision speed in these matches:
- What was the average time to make a move in the AlphaZero vs. Stockfish match?

@DouglasDaseeco thanks for the vote.  (There is a practical intent--have questions about viability of "respectably weak" gofai that can make adequate-but-less-optimal decisions faster than strong ai in certain partisan, economic contexts.  Purely academic at this point, but I notice decision speed is not always referenced in research, and for certain algorithmic functions such as high-frequency trading, decision speed is a critical factor. On my own project, it involves acquisition of sectors of memory expressed as cells on a gameboard;)

No, ALL computer chess experts were surprised about the outcomes of the match. If you require references, please start a new question.
Your question is simple...

https://arxiv.org/pdf/1712.01815.pdf
... We evaluated the fully trained instances of AlphaZero against Stockfish, Elmo and the previous
version of AlphaGo Zero (trained for 3 days) in chess, shogi and Go respectively, playing
100 game matches at tournament time controls of one minute per move ...

One minute per move. Stockfish would use the one minute if there is more than a single legal move, otherwise it'd move immediately. So the average time for a move for Stockfish is about 57s to 60s.
There is no source code for AlphaZero. However, it's hard to believe the system wouldn't take advantage of the minute it was given. The expected time for AlphaZero should be also 57s to 60s.

Thanks for clarifications (re: Chess experts) and the link to the paper.  I didn't have time to really research, but the question popped into my head so I figured I'd ask it.  As a followup, since your Chess and Chess AI knowledge greatly exceeds mine, can I ask why were the Chess experts surprised?

@DukeZhou If you start a new question, we can discuss further.

https://ai.stackexchange.com/questions/5234/why-were-chess-experts-surprised-by-the-alphazeros-victory-against-stockfish


text:
Why were Chess experts surprised by the AlphaZero's victory against Stockfish?
It was recently brought to my attention that Chess experts took the outcome of this now famous match as something of an upset.
See: Chess’s New Best Player Is A Fearless, Swashbuckling Algorithm
As as a non-expert on Chess and Chess AI, my assumption was that, based on the performance of AlphaGo, and the validation of that type of method in relation to combinatorial games, was that the older AI would have no chance.

Why was AlphaZero's victory surprising?

Good question.
First and foremost is that in Go deepmind had no superhuman opponents to challenge. Go engines were not anywhere near the highest level of the top human players. In chess, however, the engines are 500 ELO points stronger than the top human players. This is a massive difference. The amount of work that has gone into contemporary chess engines is staggering. We are talking about millions of hours in programming, hundreds of thousands of iterations. It is a massive body of knowledge and work. To overcome and surpass all of that in 4 hours is staggering.
Secondly it is not so much the result itself which is surprising to chess masters but instead its how AlphaZero plays chess. It's quite ironic that a system which had no human knowledge or expertise plays the most like we do. Engines are notorious for playing ugly looking moves, those lacking harmony etc. Its hard to explain to a non-chess player but there is such a thing as an "Artificial move" like the contemporary engines come up with often. AlphaZero does not play like this at all. It has a very human-like style where it dominates the opponent's pieces with deep strategic play and stunning position sacrifices. AlphaZero plays the way we aspire to, combining deep positional understanding with the precision of an engines calculation.
Edit
Oh and I forgot to mention something about the result itself. If you are not familiar with computer chess it may not seem staggering but it is.
These days the margins of victory which separate the top contemporary engines are razor thin. In a 100 game match you could expect to see a result like 85 games drawn, 9 victories, and 6 losses to determine the better engine.
AlphaZero 28 wins and 72 draws with zero losses was otherworldly crushing and was completely unthinkable right up to the moment it happened.

Nice answer.  Your point about comparison between AI's re Chess is interesting in terms of the limitation based on Chess' loopiness and the Win/Loss/Draw triad. (Possibly, in the future, we will need finite, intractable games that allow more granular analysis in terms of results.)  I am familiar with the history of Chess engines, and the massive amount of effort and human knowledge that went into them, but the context of the lack of success re: the much more complex 19x19 Go had an opposite ramification to me.

Specifically, my assumption was that if AlphaGo could beat the top humans in the significantly more complex game, it seemed reasonable that it would beat not only the top humans, but the top previous AI's in any other game.

The bit about artifical moves is pretty important, and not something I've seen non-chess players talk about. +1

Four clock hours on an abacus is not four compute-hours on a Cray Y-MP, which is not the same as four clock hours of 100% of all the compute resources of Amazon.  A better question is number of floating point operations (or multiply-adds) (including all parts of the process) to converge at that level.  Probably very cost-prohibitive for an individual practitioner to get and use.

MCTS for chess had been tried in the literature with little success. It was assumed AlphaGo's approach would never work on chess, maybe in Go but not in chess. Suddenly, Google announced the approach was working and it was beating the World's strongest chess program by a very signficiant margin.
Before Google, all chess programmers were taught crafting heuristics in engine programming was a better strategy than machine learning. No matter how you implemented neural networks, it would have never ran faster than a bunch of 64-bit bitboards instructions. AlphaGo was running quite slow, but it played strongest chess.

I see, based on the articles you provide, many levels of surprise in the victory:
Chess is hard game to master and the counter part had the world's best practices, AlphaZero had tabula rasa.
Learning took four hours and AlphaZero lost no match of 100.
Playing style was an alien mix of human and computer like moves, aggressive and some times seeming goofy with sacrifices that have no idea but are actually making future status more strong.
Amount of possiblities taken in account per move was less than counter part, AlphaZero had a mysterious gut feeling or intuition.
The upset feeling came from the amount of training material AlphaZero had built itself with and the time limit, that did not maybe give the traditional machine fair amount of time.

Aaah.  So it stemmed from lack of confidence in the new AI method.  That makes sense.


text:
Keras pattern finding between hash and word
My goal is to build a neural net that can find patterns between a hash and a word on it's own. So that it returns the word of any hash that I will input.
Unfortunatally my skill in the area of neural net isn´t advanced, and I want to use this project to learn more. So I use a German dictionary and encode it via one_hot encoding. Then I generate the sha256 value of every word inside (before I have done this I cleaned the file and wrote every word in another line) it. So I got an big array with the shape of 20000x20000 for the words and another for the hashs.
So then I used the a example of the Keras homepage for binary classification because the one_hot values are represented by ones and zeros.
So if I want to predict a hashs I get these error: Error when checking : expected dense_1_input to have shape (20000,) but got array with shape (1,). So I don't know if this model is working for my problem but I couldn't convert one hash into a size of 20000x20000. (The hash will one_hot encoded for that prediction). So how could I get it to accept different shaped hashs/one hash only?
Is there a way to train the model with each hash after another for example with a for loop?!
EDIT: So I figured out that I can convert a list of characters into a numpy.array with 2 dimensions. So I hot_encoded every character and create a list of them, these list I passed inside the np.array(words,ndim=2). So this I have done for my hashs aswell. Then after I run the code I got this error: ValueError: setting an array element with a sequence So I tried to reshape the array with the .reshape(20000) command but nothing chaged. So what to do with that? EDIT2: I figured out now that the problem is that enhot_encoding generates diffrent sized "arrays" for each word, and if I fill this into a real array and this into a neuronal net it have to return this error. But still the question is: How to convert single words and hashs to a format that I can train a neuronal net with and get usefull output so I can enter any hash and it should return some kind of word(lable). If you need the actual code please inform me and I will upload it`s current state.
Code:
model = Sequential()
model.add(Dense(64, input_shape=20000, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dense(units=64, activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(19957, activation='sigmoid'))
model.compile(loss='binary_crossentropy',optimizer='rmsprop', metrics=['accuracy'])
print("Fitting data...")
model.fit(test_hashs,test_words ,epochs=10,batch_size=128, verbose=1)

train_y=input("Input a hash that is not contained in the training data: ")
#train_x=pd.Series(hashlib.sha256(str.encode(train_y)).hexdigest())
train_y=pd.Series(train_y)
#test_x=pd.get_dummies(train_x)
test_y=pd.get_dummies(train_y)
model.save("first_test")
print(model.evaluate(test_y))
#score=model.evaluate(test_x, test_y, batch_size=128,)
print("Score: "+score)
prediction=model.predict(test_x,verbose=1)
for i in prediction:
    print(i)

I'm not quite sure it's possible. Hash functions are used to map an input to an output in a way that is not reversible.
Many companies store a hash of your password on their servers so in case of a security breach they haven't given the adversaries a long list of passwords.
As far as it goes for finding the exact hash of a word, it seems infeasible.
Edit: Binary classification refers to the possible output being two possible states. A ten dimensional one-hot vector is not binary.

I figured out that it isn't binary classification and I try currently to understand NN but I will not warm up with the predictions...


text:
In number classification using neural network, is training with edge image better than gray image?
i'm trying to identify numbers and letters in license plate. License plate images are taken at different lighting condtion and converted to gray image. My concern with type of data for training is:
Gray Image:

Since they are taken at different lighthing condition, gray image have different pixel intensity for same number. Which means, i have to get many training data for different lighting condition to train.

Edge Image:

They lack enough pixel information since only edge is white while others(background) are black. So i think they will be very weak for translational difference like shearing or shifting.

I want to get some information about which type of image is better for training number in different lighting condition. I wish to use edge image if they don't  differ much since i can prepare edge image right now.

Theoretically you will have no gain in the error ratio if the system preprocess the images with a linear high-pass filter before to send the image to the NN.
Let see a simple 1-dimension case that supports this statement:
Assume inputs are "a", "b" and "c". A node at first hidden layer will receive an input to the activation function equal to s=w1*a+w2*b*w3*c+... being w1, w2, w3, ... the weights for this node.
Now, assume a simple differential case, where inputs will be ... , b'=b-a, c'=c-b, ... being the input to the hidden node s=w1'a'+w2'b'+w3'c'+...=...+w2'(b-a)+w3'(c-b)+...=...+(w2'-w3')b+...
Note both inputs to hidden node are the same if w1=w1'-w2', w2=w2'-w3', ... . So, the NN can itself perform the equivalent of the linear high-pass filtering adjusting the weights of the first hidden layer.
However, changes in learning speed (convergence time) can be expected.
(ps: please, activate tex/latex in this site, urgently ! )


text:
Does augmenting data changes the distribution of augmented data?
When we augment data for training are we also changing the distribution of data and if its a different distribution why do we use it to train a model for original distribution ?

Yes you do change the distribution of your training data if you modify it (for example by augmenting it with rotated versions of images that were in an original training set of images).
This is fine because, typically, our goal of training is not to get a model with high performance on the dataset we happened to collect as training data (e.g. a bunch of natural images). Our goal is to train a model that generalizes well to new data outside of the training data's distribution.
Typically, the training data is only a sample of the distribution we're actually interested in. For example, we'll be interested in making accurate predictions for all natural images in the entire world. That is a distribution that would likely include rotated variants of all images in our training set. So, if we augment our training set by adding such rotated variants, we expect to modify our training data distribution in such a way that it actually gets a little bit closer to the distribution we're interested in (all natural images).


text:
What is easier or more efficient to summarize voice or text? [DP/RN]
If possible consider the relationship between implementation difficulty and accuracy in voice examples or simply chat conversations.
And currently, what are the directions on algorithms like Deep Learning or others to solve this.

Summarizing text is always going to be 'easier or more efficient' than voice simply because voice requires the additional step of converting to text.  That doesn't tell you anything about accuracy.
From an article published on June 1, 2017, Google’s speech recognition is now almost as accurate as humans:
"According to Mary Meeker’s annual Internet Trends Report, Google’s machine learning-backed voice recognition — as of May 2017 — has achieved a 95% word accuracy rate for the English language. That current rate also happens to be the threshold for human accuracy."
If you need this kind of accuracy check out Google's Cloud Speech API.  There is even a speech to text feature on the web page.
Given a speech-to-text conversion accuracy of 95%, voice will be 5% less accurate than text if everything else is equal but it usually isn't.  People generally write better text, such as in documents or emails, than when they speak unless of course they are giving a formal lecture, or talking in a formal meeting.  If one is analyzing text messages, Tweets, or threads found in typical informal forums, you will find very poor quality in grammar, spelling, vocabulary, and punctuation.  The answer to your question will depend on the source of your text.
In another article, dated November 13, 2017, Why 100% Accuracy Is Not Available With Speech Recognition Software Alone, the author gives some reasons, albeit for transcription software which has a special purpose, why there will always be some errors due to:

Speech Patterns and Accents - Regional variations exist, for example English speakers in Boston sound different than Kentucky.  How does the software handle slurred speech or when a person blends their words?
Grammar and Punctuation - speech recognition software doesn't know where a period, comma, or semi-colon belongs
Homonyms and unusual words - "Speech processing software can only recognize words and phrases that it has specifically been trained to recognize."
Ambient Noise, Overlapping Speech, and Number of Speakers

To address your last question about where the technology is going...
Four days ago a paper by Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria entitled Recent Trends in Deep Learning Based
Natural Language Processing was published which gives some of the answers.
From the 'Conclusion' section:
With distributed representation, various deep models have
become the new state-of-the-art methods for NLP problems.
Supervised learning is the most popular practice in recent
deep learning research for NLP. In many real-world scenarios,
however, we have unlabeled data which require advanced
unsupervised or semi-supervised approaches. In cases where
there is lack of labeled data for some particular classes or the
appearance of a new class while testing the model, strategies
like zero-shot learning should be employed. These learning
schemes are still in their developing phase but we expect deep
learning based NLP research to be driven in the direction of
making better use of unlabeled data. We expect such trend to
continue with more and better model designs. We expect to
see more NLP applications that employ reinforcement learning
methods, e.g., dialogue systems. We also expect to see more
research on multimodal learning [167] as, in the real world,
language is often grounded on (or correlated with) other
signals.
Finally, we expect to see more deep learning models whose
internal memory (bottom-up knowledge learned from the data)
is enriched with an external memory (top-down knowledge
inherited from a KB). Coupling symbolic and sub-symbolic AI
will be key for stepping forward in the path from NLP to natural
language understanding. Relying on machine learning, in
fact, is good to make a ‘good guess’ based on past experience,
because sub-symbolic methods encode correlation and their
decision-making process is probabilistic.

Hi, thanks for the explanation this help me so much!

You might want to take the Stanford Online course on YouTube Natural Language Processing with Deep Learning. This course will give you insight into how different kinds of neural networks can be used for different kind of NLP tasks.
In my opinion, you can use Gated Recurrent Units (GRUs) to encode and decode text. Of course, text will be easier because voice data, as it is stored in a computer, is going to be difficult to interpret in the testing phase. Another way is to get most impactful words and then use these to form sentences regarding the original text.
You can also start by looking out for publications related to text summarizers. For example, Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond, will get you started. You can use this as a starting point. In case you need to understand basics regarding the underlying techniques, then you can go through references in this paper and find out useful resources to get you started.

Hi, thanks for the suggestion and the answer


text:
Can a game AI learn the concept of acceleration?
If the game had a variable speed and was essential in evolution/gaining score(IDK AI terminologies). Would the AI be able to figure out when to slow down and speed up?
If it is able to solve the problem or complete the level, will it have an equation to relating acceleration, or perhaps a number on when to speed up and down. What if the game environment was dynamic?
Can you even teach math to an AI?
PS: I'm not sure if I should ask separate question?

This answer mostly assumes you are referring to computer-game-playing bots that learn through experiencing play, such as Deep Mind's DQN as used for playing Atari console games. State of the art for these are typically Reinforcement Learning algorithms, used with neural networks to process input and estimate results of next actions. There are other competitive AI technologies too, and the answer applies generally to most learning or evolving optimisers that would learn through trial and error by playing the game.

If the game had a variable speed and was essential in evolution/gaining score(IDK AI terminologies). Would the AI be able to figure out when to slow down and speed up?

Yes, as long as the game allowed control of something that influenced acceleration, then a learning agent can figure out the consequences of accelerating and braking and use them appropriately within the game.
A well-known toy example that can challenge learning agents is called Mountain Car. In that game, the agent has to learn to accelerate in the correct direction (which is not always towards its objective), in order to escape an area. It is considered challenging because the reward (for escaping) can be significantly delayed compared to the action that best enables it.
One popular proving ground for learning agents is OpenAI gym. This includes several game environments with a physical model of acceleration included, such as Lunar Lander.

If it is able to solve the problem or complete the level, will it have an equation to relating acceleration?

In general, no. The agent will learn to respond to certain stimulus, by taking an action that accelerates or decelerates the game piece that it controls. There will not be any concept like $s = ut + \frac{1}{2} at^2$ encoded in the agent's parameters.

or perhaps a number on when to speed up and down?

Typically the agent will learn which stimuli should be responded to by accelerating. For instance, in a game where the agent's game piece is being chased by an enemy piece and the enemy piece is getting closer, the agent should learn that it will get a better reward if it accelerates away from the enemy.

What if the game environment was dynamic?

Most game environments are dynamic, as in the state changes over time. If you mean would anything change if the game rules themselves varied over time, then this may cause interesting problems for some learning algorithms, but should not change anything about learning use of controls that affect acceleration in a virtual world.

Can you even teach math to an AI?

Generally, no you cannot teach math to the kind of system that plays games, or interacts with real world objects. These kind of learning systems are not yet advanced enough to learn concepts or establish game world logic from interactions. Instead, they work more akin to perception, muscle memory and either inherent or learned reflexes. Exceptions to this will generally have a world model (with the necessary equations) built in or made available to the agent without it needing to learn anything.
However, there are AI systems that use formal logic that can work on mathematical theories. Some have performed interesting feats such as "discovering" prime numbers, given formal definitions of integers and basic arithmetic. An example of this kind of system is the Automated Mathematician.
There is an intermediate possibility: Some learning agents not only learn the value or best policy for certain actions, they also learn to predict what should happen next to the state of the environment. Such an agent would include a model that could observe objects that were accelerating and predict their future positions. In some ways this is a learned concept of acceleration, although it would not be expressed mathematically like Newton's laws of motion, and is more akin to the kind of intuition that allows a person to track, anticipate and catch a thrown ball.

Methods such as Genetic Programming can induce symbolic expressions from observations. Indeed, such methods can be used as an alternative to neural (or other nonsymbolic) approaches when the goal is not merely to find a function that fits the data well, but which also has a chance of giving a human-readable description of the learned function in terms of whatever mathematical functions the user chooses to supply the learning method with.
These methods are even used commercially for knowledge discovery.


text:
How do I statistically evaluate a ML model?
I have a model that predicts sentiment of tweets. Are there any standard procedures to evaluate such a model in terms of its output?
I could sample the output, work out which are correctly predicted by hand, and count true and false positives and negatives but is there a better way?
I know about test and training sets and metrics like AUROC and AUPRC which evaluate the model based on known data, but I am interested in the step afterwards when we don't know the actual values we are predicting. I could use the same metrics, I suppose, but everything would need to be done by hand.

There are a lot of ways to evaluate the performance of an ML model.  You mentioned AUROC and AUPRC.  Generally you start with the confusion matrix and derive metrics such as sensitivity, accuracy, recall, precision, etc.  You can see a good outline of them here.
It seems what you are asking is a shortcut to determining how good your sentiment classification model is but there aren't any without labeled test data.  You either do this by hand or you find a test set in the world, preferably something that is well know and documented and also fits your objectives.  I recommend you read Neil Slater's answer at https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228.  He gives some good advice on the subjectivity of sentiment analysis classification and points out a labeled data set of Tweets which you might be able to use to test your classifier.
I also found this Kaggle competition which has a test set that might be of help to you: Angry Tweets

Thanks! I suppose I'll have to do it by hand then, for my date.


text:
Tuning the b parameter, ANFIS
In order for the generalized bell membership function to retain its defined shape and domain, two restrictions must be placed on the b parameter: 1) b must be positive and 2) b must be an integer. Using backpropagation to tune the membership parameters (a,b and c in the bell), it appears to be possible that the correction to b will break one or both of these restrictions on b. Can someone please explain to me how we can use backpropagation to tune the b parameter (as well as a and c) without violating 1) and 2)?

After re-reading Jang's original (1993?) paper on ANFIS, I learned he recommended simply squaring the b-parameter to deal with the notion that b could be changed to a negative value when using back-propagation. While this solves the negative domain issue, the issue still remains that if b is a non-integer, then the bell function loses its intended shape. I suppose after squaring one could round to the nearest integer, but I suspect that doing so might hinder convergence. Perhaps a consideration is only tuning the a and c parameters using something like a genetic algorithm.


text:
Strong AI vs Singularity - which should happen first?
What is supposed to happen first: Strong AI or Technological Singularity?
Meaning which option is more likely, that the Strong AI that will bring as to the state of technological singularity or achieving technological singularity will allow us to construct strong AI?

Charles Stross, in Accelerando posits that, even in a post superintelligence future, recognizing the advent or nature of the singularity may be fuzzier than people assume.

The definition of "technological singularity" answers the question:

The technological singularity (also, simply, the singularity) is
  the hypothesis that the invention of artificial superintelligence will
  abruptly trigger runaway technological growth, resulting in
  unfathomable changes to human civilization.

(wiki)
note the order of facts is "invention of artificial superintelligence" (AGI) followed by the "unfathomable changes".

Even though I agree, yet there are some scenarios with non-AI singularity.

@Alex: Of course alternative definitions can be done and answers will differ. My answer wants more to present the need of define the concepts before to discuss about them that advocate for an opinion about "singularities". Talk without concept definition is done on TV shows.

Solid answer.  I'm not entirely convinced "singularity" is the right word for the definitions given, which is not to impugn Von Neumann, Goode or anyone else.  Just that the after reading the Causal Angel by mathematical phycist Rajaniemi, who uses game theory to relate intelligence acceleration to the topology of blackholes, I feel like the term "singularity" should be more specific than mere "runaway superintelligence."  i.e. why this term in particular?