text:
How can I use feature extraction in CNN with image segmentation?
I'm just started to learn about meta learning and CNN and in most paper that I've read they mention to have one CNN to feature extraction. These features will help the another network.
I don't know what is feature extraction (I don't know what are those features) but I'm wondering if I can use it on image segmentation.
The idea is to use the first network to feature extraction without doing image classification, and pass those features to the other network.
My question is:
How can I use feature extraction in CNN on image segmentation?

Feature extraction is a way that people use pretrained model to extract information from input data. For example, image segmentation task may use the VGG network or other image classifying network for feature extraction. The output of the last convolution layer is taken. Then, the features are feed into the untrained network to get outputs. The bottom network for image segmentation usually consists of upsampling and convolutional layers. Then output of size of original image is resulted in teh main network. Hope I can help you


text:
Training network with 4 GPUs performance is not exactly 4 times over one GPU why?
Training neural network with 4 GPUs using pyTorch, performance is not even 2 times (btw 1 & 2 times) compare to using one GPU. From Nvidia-smi we see GPU usage is for few milliseconds and next 5-10 seconds looks like data is off-loaded and loaded for new executions (mostly GPU usage is 0%). Is there any way in pyTorch to improve the data upload and offload for the GPU execution.

Using parallel resources optimally is a hard problem even knowing the data flow. You don't give nearly enough information for someone to help with your specific problem, and the general issue is far too broad to answer in this format. Please explain more about the nature of your project, and the data pipelines. Also, as it is a practical matter in a specific library, consider asking in Data Science SE instead, if this work is for a standard ML supervised learning problem. Read their help section to see what makes things on topic there, but you will definitely need to give project details.

Your dataset class probably have a lot of preprocessing code. You should use a dataloader. It will prefetch data from the dataset when the GPU is processing. Also, you can process all the data beforehand and save to a file. Multiple GPU cannot scale as the GPU have to get all data to one GPU to calculate the loss. The performance of 4 GPU is around 3.5x. A large batch size also would help as each GPU will have 1/4 the batch size. a batch size of 64-128 is good for 4 GPU. See the following example code fro CIFAR-10 for multi gpu code. It have dataloaders and dataparallel.
import os
import time
import datetime

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import torch.backends.cudnn as cudnn

import torchvision
import torchvision.transforms as transforms
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader

from model import pyramidnet
import argparse
from tensorboardX import SummaryWriter

parser = argparse.ArgumentParser(description='cifar10 classification models')
parser.add_argument('--lr', default=0.1, help='')
parser.add_argument('--resume', default=None, help='')
parser.add_argument('--batch_size', type=int, default=768, help='')
parser.add_argument('--num_worker', type=int, default=4, help='')
parser.add_argument("--gpu_devices", type=int, nargs='+', default=None, help="")
args = parser.parse_args()

gpu_devices = ','.join([str(id) for id in args.gpu_devices])
os.environ["CUDA_VISIBLE_DEVICES"] = gpu_devices

def main():
    best_acc = 0

    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    print('==> Preparing data..')
    transforms_train = transforms.Compose([
        transforms.RandomCrop(32, padding=4),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

    dataset_train = CIFAR10(root='../data', train=True, download=True,
                            transform=transforms_train)

    train_loader = DataLoader(dataset_train, batch_size=args.batch_size,
                              shuffle=True, num_workers=args.num_worker)

    # there are 10 classes so the dataset name is cifar-10
    classes = ('plane', 'car', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck')

    print('==> Making model..')

    net = pyramidnet()
    net = nn.DataParallel(net)
    net = net.to(device)
    num_params = sum(p.numel() for p in net.parameters() if p.requires_grad)
    print('The number of parameters of model is', num_params)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(net.parameters(), lr=args.lr)
    # optimizer = optim.SGD(net.parameters(), lr=args.lr,
    #                       momentum=0.9, weight_decay=1e-4)

    train(net, criterion, optimizer, train_loader, device)

def train(net, criterion, optimizer, train_loader, device):
    net.train()

    train_loss = 0
    correct = 0
    total = 0

    epoch_start = time.time()
    for batch_idx, (inputs, targets) in enumerate(train_loader):
        start = time.time()

        inputs = inputs.to(device)
        targets = targets.to(device)
        outputs = net(inputs)
        loss = criterion(outputs, targets)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()

        acc = 100 * correct / total

        batch_time = time.time() - start

        if batch_idx % 20 == 0:
            print('Epoch: [{}/{}]| loss: {:.3f} | acc: {:.3f} | batch time: {:.3f}s '.format(
                batch_idx, len(train_loader), train_loss/(batch_idx+1), acc, batch_time))

    elapse_time = time.time() - epoch_start
    elapse_time = datetime.timedelta(seconds=elapse_time)
    print("Training time {}".format(elapse_time))

if __name__=='__main__':
    main()

Source: https://github.com/dnddnjs/pytorch-multigpu/blob/master/data_parallel/train.py
Hope I can help you and have a nice day!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Positioning of batch normalization layer when converting strided convolution to convolution + blurpool
I'm trying to replace the strided convolutions of Keras' MobileNet implementation with the ConvBlurPool operation as defined in the Making Convolutional Networks Shift-Invariant Again paper. In the paper, a ConvBlurPool is implemented as follows:
$$
Relu \circ Conv_{k,s} \rightarrow Subsample_s \circ Blur_m \circ Relu \circ Conv_{k,1}
$$
where k is the convolution's output kernels, s is the stride, m is the blurring kernel size and the subsample+blur is implemented as a strided convolution with a constant kernel.
My issues start when batch normalization enter the picture.
In MobileNet, a conv block is defined as follows (omitting the zero-padding):
$$
Relu \circ BatchNorm \circ Conv_{k,s}
$$
I am leaning towards converting it to:
$$
Subsample_s \circ Blur_m \circ Relu \circ BatchNorm \circ Conv_{k,1}
$$
i.e., putting the BN before the activation as it's normally done. This is not equivalent though, because the first BN operates on the downsampled signal.
Another possibility would be:
$$
BatchNorm \circ Subsample_s \circ Blur_m \circ Relu \circ Conv_{k,1}
$$
with the BN as last operation. This is also not equivalent, because now the BN comes after the ReLu.
Is there any reason to prefer one option over the other? Are there any other options I'm not considering?

After finding the paper authors' Github, I saw that, although they only have a MobileNet V2 model implemented, they choose the Subsample-after-ReLu option (the first one in the question).
Although this doesn't fully answer my question, I'll take "the paper authors do it this way" as enough reason to prefer this over the alternative.

Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can we print weights per iteration in a simple feed forward MLP for an specific class?
im working on a project in which I have to make a multi-layer perceptron with two hidden layers with 3 nodes in each. The target value in my data contains 8 unique values/classes. One of the tasks states "For the most popular class CYT plot weight values per iteration for the last layer (3 weights and bias)".  My question is "does this statement make sense"?  I can access the weights and biases of a layer but I don't get what are  weight values for a specific class and how to access them

A common model used for this kind of classification task is to have one output neuron per class. So, for example, neuron 1 may have a loss function that is related to outputting "1" for examples of class 1, and "0" for examples of other classes. Neuron 2 may be asked to do the same, but for class 2 rather than class 1.
If you use a model of this kind, you can pull the weights for each neuron in the final output layer. It sounds like this is what you are being asked to plot.


text:
Should I use my redundant feature as an auxiliary output or as another input feature?
For example, given a face image, and you want to predict the gender. You also have age information for each person, should you feed the age information as input or should you use it as auxiliary output that the network needs to predict?
How do I know analytically (instead of experimentally) which approach will be better? What is the logic behind this?

For extra input that does not matter, you should not input it to the network.

Feature selection, the process of finding and selecting the most
  useful features in a dataset, is a crucial step of the machine
  learning pipeline. Unnecessary features decrease training speed,
  decrease model interpretability, and, most importantly, decrease
  generalization performance on the test set.

Source: A Feature Selection Tool for Machine Learning in Python
As the source says, unnecessary features decreases accuracy and training speed. Moreover, they have no mapping to the labels, so they won't end up being used. They are unnecessary and adding them will only cause you trouble. Hope this helps you and have a nice day!


text:
Can I use my previous estimate of the state-action values as initialisation in GLIE-Monte Carlo Control?
I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm.
So I repeat n times:

create observations using my previous policy $\pi_{n-1}(s)$
update my state-action values using the observations generated in 1 with the monte-carlo update rule: $Q_n(s_t,a_t)= Q_n(s_t,a_t)+1/N(s_t,a_t)\times(G_t-Q_n(S_t,a_t))$
update my policy to $\pi_{n}$ using epsilon-geedy improvement with $\epsilon=1/(n+1)$.

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.
Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.

Thank you!  However I am not sure whether I understand the term "in place" does this mean that I improve my policy after every Monte-Carlo Update?

I mean that you only have one Q table, and overwrite values as you calculate them. So no need to initialise a new table on each time step


text:
AI natural voice generator
I want to create a solution, which clones my voice. I tried my commercial solutions or implementation of Tacotron. Unfortunately, results not sound natural, generated voice sounds like a robot. Anybody could recommend good alternative?

hi @fuwiak I posted an answer. Hope it can help you and have a nice day!

The reason for robot like speech may be because tacotron uses griffin lim for vocoder, which cannot reproduce sound with perfection, often introducing robot like sound artifects.

A vocoder is a network that transforms a transform a spectrogram image back to speech waveform. Tacotron and many other speech generation neural network uses CNN to generate spectrogram instead of raw waveforms as output. Spectrogram is a lossy representation of raw audio waveform, so a perfect reconstruction of audio waveform is not possible. Griffin-Lim is a vocoder that uses algorithmic way to transform spectrogram to audio waveform, but often introduces a robot-like quality to generated waveforms. A neural network based vocoder can solve the problem. The wavenet vocoder is often used in speech generation as it can transform the spectrogram to audio with little artifects. Many new speech generation models use the wavenet vocoder as the deafult vocoder of the generation model. For a public implementation, this is a good github repository: https://github.com/r9y9/wavenet_vocoder
You can also use the newer tacotron 2 which uses the wavenet vocoder as the default vocoder. You can check it out here: https://github.com/Rayhane-mamah/Tacotron-2

In Tacotron-2, do you for each character in the input-text, how many frames of spectogram are generated during synthesis (inference)? The original paper at https://github.com/Rayhane-mamah/Tacotron-2 refers to sampling the "training dataset" at 1 frame per 12.5ms, which is different than number of frames per second.

Also, how does it associate spectogram frames with a  portion of the input text (e.g. an input character) during training phase, and what is number of frames per character during training? It must need to do that in order to learn using seq-2seq encoder/decoder system.


text:
How exactly does adversarial training help in handling mode-collapse in generative networks?
Of my understanding mode-collapse is when there happen to be multiple classes in the dataset and the generative network converges to only one of these classes and generates images only within this class. On training the model more, the model converges to another class.
In Goodfellows NeurIPS presentation he clearly addressed how training a generative network in an adversarial manner avoids mode-collapse. How exactly do GAN's avoid mode-collapse? and did previous works on generative networks not try to address this?
Apart from the obvious superior performance (generally), is the fact that GAN's address mode-collapse make them far preferred over other ways of training a generative model?

I don't think he said that at all. Going back to the talk you'll see he mentions mode collapse comes from the naivete of using alternating gradient-based optimization steps because then $min_{\phi}max_{\theta}L(G_\phi, D_\theta)$ starts to look a lot like $max_{\theta}min_{\phi}L(G_\phi, D_\theta)$.
This is problematic because in the latter case the generator has an obvious minimum of transforming all generated output into a single-mode that the discriminator has considered acceptable.
Since then a lot of work has been done to deal with this point of failure. Examples include Unrolled GANs (he mentions this one in the talk), where you essentially make the generator optimize what the discriminator will think $K$ steps in the future to ensure the ordering of the $min \ max$ game, and Wasserstein GANs, where you focus on a different metric that still has the same global minimum but allows for side by side training completely eliminating the ordering and failure mode, to begin with. On top of this, other work has been done as well, these are just two important examples.
Regarding how they fare against other generative models, like VAEs, there is no one is better than the other. The recent empirical success of GANs is why they are so popularly used, but we still see others being used in practice as well.


text:
In adversarial machine learning, how does an attacker have access to the test and training dataset in order to poison it?
In the field of adversarial machine learning, machine learning models are vulnerable to attacks both on the test and training data set. However, how does the attacker get access to these datasets? How do these datasets get manipulated/tampered with?

They doesn't. That's why adversarial attack is sham

We can manipulate a model's test data set if the machine learning model takes user input and uses it to resample test data set. The actual training dataset of the ML model does not get manipulated, but if we figure out the ML model through an exploratory attack (sending a lot of inquiries to the ML model to find out its nature), we can generate a training dataset which was built into the original ML model.

They don't have acces to the original training or test dataset. Machine learning environments are build on the premise of a benign environment. The models are trained on real data (real inputs). When someone sends a made up input (fake input) it is very easy to fool the model.
This is used for example in image recognition. Imagine a fotograph of a panda. the model may correctly identify this fotograph as a panda. With knowledge of the model you can now alter some pixels in the fotograph. To the human Eye, the fotographs will appear exactly the same, but the model can be fooled to believe the fotograph is actually of a gibbon.
This is all done after the training of the model and doesn't require the original datasets.
For more info, visit this site:
https://medium.com/@ml.at.berkeley/tricking-neural-networks-create-your-own-adversarial-examples-a61eb7620fd8

In adverserial machine learning, someone (program or human) attempts to fool an existing model with a malicious input.
The best human example would be an optical illusion. The human brain's model for image processing starts outputting wrong information when looking at an optical illusion. So in the end we see wrong colour, shape, etc. In this case, the optical illusion would be considered as the malicious input.
We can trick the human brain’s model through images created with trial and error.
So, if you just have the trained model at hand, you don’t have to know the data it has been trained with. You just need to be able to input a value to the model and get the output.


text:
Interpreting Keras Yolov3 config file
How does one interpret the "min_input_size", "max_input_size" and "anchors" fields in the Yolov3 config file here. In particular, suppose we have the following:
    "min_input_size":       288,

    "max_input_size":       448,

    "anchors":              [55,69, 75,234, 133,240, 136,129, 142,363, 203,290, 228,184, 285,359, 341,260]

Does the min_input_size and max_input_size indicate the maximum number of training images we can have? What do the numbers in the "anchors" field indicate? Are they the coordinates of the anchor boxes? Surprisingly, I have not been able to find a good explanation of many of these  fields within this file.

Hi and welcome to this community! This type of question is off-topic here because you are asking about the meaning of certain parameters in the source code. Have a look at https://ai.stackexchange.com/help/on-topic for more details. This type of question is more suited for Data Science SE.

The min and max input size should be the min and max image size of the input images. The numbers represent pixels in both axis of the image.
The anchors represents the size of the anchor box. Anchor box does not have coordinates, only have size.
Hope I can help you

So "min_input_size":       288 means that the minimum size of of images is 288 x 288?  Also "anchors" =  [55,69, 75,234] means anchors of size 55x69 and 75x234 (using first 4 numbers in "anchors" field)?

Yes. This should be the case.

The min input size are here as the model is a fully convolutional neural network so it can accept a range of image size. However to get a suitable output size only the image size in the range can do it.

I see. So in the example above, there are 9 anchor boxes of varying sizes generated for each grid cell? What is the parameter that controls the number of grid cells usually called? Actually it seems that this parameter is in the YOLO class itself.

First of all, anchor box is not generated, it is just used to choose which output bounding box is good quality. YOLO loss computes IOU of the anchor box to the output box. The number of grid cell called, do you mean output? I assume you mean the output grid. If you mean so, then it is not changeble. It is in the model's architecture. However since it is a CNN if you use a lower image size the grid size will be smaller. Default is 13*13 if I am not mistaken

That's what I meant. So in the above example, a output bounding box would be compared to 9 anchor boxes (e.g. the IOU of the output bounding box and each of the 9 anchor boxes is computed and the one with the highest is the "best" bounding box)?

It's something similar.


text:
Is there has any method to train Tensorflow AI/ML that I focus on detecting background of image more than common objects?
Is there has any method to train Tensorflow AI/ML that I focus on detecting background of image more than common objects?
I'm newbie to ML field, but was assigned to do job that make an application which can detecting on showroom image/places and detecting on the floor, wall then find out what is the material/ceramic/marble/etc. product they are.
Example: This is showroom picture,

the wall and the floor of showroom are using this product material

Is it possible to do something like I described?
How to start with?
If I don't want to install Tensorflow into my computer, is there a service that can make a model to use in the device? (my goal need to use the model in Android device)
What method/type of ML should I approach 'Classification' or 'Object Detection' or other else?

Hi and welcome to this community! Please, ask just one question per post, otherwise, your post could be closed as too broad.

@nbro sorry, but what will you think if I just flood the board with many of questions or just gather them all into one post. So I can ask and clear my minds from self-learn questions as a newbie like me? Hope you understand the blinds people who come into new field of knowledge and try self-learn first but there's ton of technical terms that can't be cleared understand from read them once.

You should ask one question per post, if you are either a newbie or not. Furthermore, these questions is off-topic here.

Trying to address all the questions asked in the end in the same order

Most definitely possible.
I would say its best you approach this with segmentation to start with.
Just use a free GPU runtime notebook service such as Google Colab or Kaggle Kernels. But you would not directly be able to integrate with the device, you'd have to keep moving input and output from your drive (on Colab). There might be a better service for the needs described, but this is the best I know on this.
Your background can be segmented and the segment can work on transforms such as maybe convolutions or affine transforms to be able to get the relevant information regarding the background.

Hope this was helpful!


text:
How to understand the 4 steps of Monte Carlo Tree Search
From many blogs and this one https://web.archive.org/web/20160308070346/http://mcts.ai/about/index.html
We know that the process of MCTS algorithm has 4 steps.

Selection: Starting at root node R, recursively select optimal child nodes until a leaf node L is reached.

What does leaf node L mean here? I thought it should be a node representing the terminal state of the game, or another word which ends the game.
If L is not a terminal node (one end state of the game), how do we decide that the selection step stops on node L? From the terms of general algorithm, a leaf node is the one that does not have any

Expansion: If L is a not a terminal node (i.e. it does not end the game) then create one or more child nodes and select one C.

From this description I realise that obviously my previous thought incorrect.
Then if L is not a terminal node, it implies that L should have children, why not continue finding a child from L at the "Selection" step?
Do we have the children list of L at this step?
From the description of this step itself, when do we create one child node, and when do we need to create more than one child nodes? Based on what rule/policy do we select node C?

Simulation: Run a simulated playout from C until a result is achieved.

Because of the confusion of the 1st question, I totally cannot understand why we need to simulate the game. I thought from the selection step, we can reach the terminal node and the game should be ended on node L in this path. We even do not need to do "Expansion" because node L is the terminal node.

Backpropagation: Update the current move sequence with the simulation result. Fine.

Last question, from where did you get the answer to these questions?
Thank you

Hi and welcome to this community! Try asking just one question per post. You're asking too many questions here! Ask each question in its separate post

@nbro I'm inclined to view this as a four-part question on a single subject, since the "4 steps" imply relation to each other.

@DukeZhou We should really discourage this type of post. The OP is asking too many questions, which both reduces the future usefulness of this post and doesn't facilitate the answerer's life. Essentially, the OP is asking how MCTS works. See https://meta.stackexchange.com/a/39224/287113. I've not read the answer, but it looks like it doesn't address all questions. So, maybe we can reformulate this post so that it is the question to the given answer.

@nbro fair point.  And the questions are nested in the text, which increases the likelihood of various questions being missed.

@wenbo Welcome and I'm glad you got an answer.  I'm going to leave this open b/c of that, but please do restrict to one question per post in the future--we hope to see more of you!

Imagine a game with a very clear first move, such as a game where choosing to go first if you win a coin toss brings a clear and obvious advantage.
In this situation standard MCTS does little exploration down the side of the tree that branches at the win toss > let opponent start step, as the basic simulations of the rest of the game at this split quickly show the large gain you get when always starting when you win the coin toss.
As a result, you would end up with a tree with very little expansion on the side of win the toss > put your opponent in, as every simulation step you do from even the most senior nodes ends with much worse expected outcome values than the alternatives on the other side of the tree where you do the correct move of always playing first.
These nodes on the side of letting your opponent start have huge potential sub trees (as the whole game would still need to be played out if your opponent started), but would have very little searching down them. As a result, on this side of the tree, you would have many leaf nodes with large (but as yet unexplored, outside of the basic, early simulations down that side) sub trees that you could search if you modified the exploration vs exploration algorithm.
As a basic example, take the 0/3 node at the far right on level one of the wiki example below, which would get much less attention than the much more promising 7/10 and 3/8 nodes, despite having potentially many subsequent children it could explore. If you took this node as your L node, you would expand it's children that you have not yet searched, and thus find out more about why this side of the tree is bad and update our now more granular probabilities accordingly, just as it does for the 3/3 node here:

Thank you @Philip for your answer. As you post the diagram here, let me ask my questions in other words based on your diagram. The tree on the most left has 4 levels, there are 2 nodes on the bottom of the tree, 2/3 and 3/3.
How the values 2/3 and 3/3 are calculated?
Whether there are more levels under 2/3 and 3/3 are now shown, what nodes are not shown here? Whether the nodes from 2/3 to the terminal nodes (an end state of the game)  are not shown? Otherwise, how is 2 calculated?.
And why are these nodes not shown?
Why does the step "Selection" stop on node 2/3 and 3/3?

Don't know why I can't edit my last comment. For the question

How the values 2/3 and 3/3 are calculated?
I meant without deeper search under 2/3 and 3/3, how the values of the 2 nodes are obtained. I understand the meaning of 2 and 3.

These would be created using some kind of simulation logic. This could range from just making random moves through to using the current best AI to play itself down until a finish. Those values are what the score has been so far using that simulation logic, and thus represent the best approximation of the current expected value of that branch of the game using that simulation logic.

Say initially, there is only one root node which should be 0/0, then how this node can be filled as 11/21? By simulation logic of step "Simulation"? A simple approach for this step is selecting random action until a finish, can a random action ends with 11/21? I definitely misunderstood something.

At the start of the tree it was 0/0. The first iteration will have picked one of the three possible children (here, nodes now at 7/10, 3/8 and 0/3), and played the game out from there and recorded the result. Say the first child was the node that is now 0/3, we know that game would have been a loss, so the first node would now be 0/1 and the far right node would also be 0/1. As each trial repeats this process updates until we reach 21 trials and you have the graph above at step one.

Also note how at step four above when the simulation is complete the number of trials update all the way up the tree, so that at the end of this step the top node stands at 11/22 instead of 11/21.

we know that game would have been a loss
Thank you. How do we know that? By randomly play the game until a finish? If so, whether it means this value is not reliable for exploitation? and 0/3 means we simulated the game 3 times by randomly selection?

Yes exactly, and that is one of they key trade offs made by MCTS, as playing every permutation of every node in complex games quickly becomes computationally impossible. If you could do that you would have no need for this type of process.

Yes exactly, and that is one of they key trade offs made by MCTS.
Bingo! This helps me a lot.

If this has helped you please mark the question answered by adding the green tick to this answer as per: https://meta.stackexchange.com/questions/106319/in-stack-overflow-how-do-i-put-the-big-green-check-to-the-answer-which-i-like-th

Should I consider the step of exploring from the root node 0/0 to the far right node 0/0 as "Expansion" ?

This video with a worked example also might be helpful: https://www.youtube.com/watch?v=UXW2yZndl7U

I also want to answer my question after watching the video that @Philip post youtube.com/watch?v=UXW2yZndl7U . But still thank @Philip 's answer which is pretty helpful.

The definition of "leaf" node.

The key point is what tree is the host/owner of a "leaf" node to this question. Through "Expansion" step, we are actually creating a tree with MCTS. The tree, the owner of a "leaf" node, should be the one that we are building, not the tree of the game state in our head (or perhaps it is too big to fill in our head, the tree of the game state actually does not exist). Then we can understand that a "leaf" node is the one, which does not have any child, in the tree that we are building.
Once we get the answer of this question, the other questions can be answered automatically.


text:
Classification with deeplearning : clean start vs continue training
I trained some weights to identify apples and oranges (using YOLOv3).
If I want to be able to identify peaches, which approach is usually recommended:

Start clean and train the 3 classes.
Train the peaches over the already-trained weights (with apples and oranges)

Only train with peaches images
Use all available training data (including apples and oranges)

This is what I have found:

If I start clean, it will take longer until I can get a good result, but the detection is usually better.
Every time I add a new class (using 2.2), the detection get worse for the already learned objects, but it takes less time until I can get a good result (however I suspect that apples and oranges become over-fitted?).
I haven't tested 2.1, as I think that it won't be able to re-adjust the weights for the apples and the oranges.

Is the above expected? What is the recommended course of action?

Rule of the thumb: when in doubt start clean

If the task involves only apples, orange and peaches, you should use method 1. As the number of classes is small, the network cannot generalize well to all classes. As a side note, you should start with the pretrained weights of YOLO v3 as some classes of YOLO v3 may be fruits, which can help your model converge faster.
If the number of classes is large, for example a hundred different fruits, you should use method 2.2 . The model should be able to generalize to all fruits and can converage faster as many fruits look the same. This is the case of transfer learning. In original YOLO v3 training, image net weights for dark net is used for the backbone network. It accelerates the training of YOLO v3.
For 2.1, it will not work as gradient descent will not consider the trained weight. The trained weights will be over written by the weights for peaches.
For a recommended method, it depends on your class size. If the model will continue to add more classes, you should perhaps use 2.2 but only start using 2.2 when you have a considerable amount of classes. Hope I can help you.


text:
What is the relationship between robustness and adversarial machine learning?
I have been reading a lot of articles on adversarial machine learning and there are mentions of "best practices for robust machine learning".
A specific example of this would be when there are references to "loss of efficient robust estimation in high dimensions of data" in articles related to adversarial machine learning. Also, IBM has a Github repository named "IBM's Adversarial Robustness Toolbox".
Additionally, there is a field of statistics called 'robust statistics' but there is no clear explanation anywhere about its relation to adversarial machine learning.
I would therefore be grateful if someone could explain what robustness is in the context of Adversarial Machine Learning.

A robust ML model is one that captures patterns that generalize well in the face of the kinds of small changes that humans expect to see in the real world.
A robust model is one that generalizes well from a training set to a test or validation set, but the term also gets used to refer to models that generalize well to, e.g. changes in the lighting of a photograph, the rotation of objects, or the introduction of small amounts of random noise.
Adversarial machine learning is the process of finding examples that break an otherwise reasonable looking model. A simple example of this is that if I give you  a dataset of cat and dog photos, in which cats are always wearing bright red bow ties, your model may learn to associate bow ties with cats. If I then give it a picture of a dog with a bow tie, your model may label it as a cat. Adversarial machine learning also often includes the ability to identify specific pieces of noise that can be added to inputs to confound a model.
Therefore, if a model is robust, it basically means that it is difficult to find adversarial examples for the model. Usually this is because the model has learned some desirable correlations (e.g. cats have a different muzzle shape than dogs), rather than undesirable ones (cats have bow ties; pictures containing cats are 0.025% more blue than those containing dogs; dog pictures have humans in them more often; etc.).
Approaches like GANs try to directly exploit this idea, by training the model on both true data and data designed by an adversary to resemble the true data. In this sense, GANs are an attempt to create a robust discriminator.


text:
Feature extraction timeseries, model compatibility
I've got a timeseries with sensor data (e.g. accelerometer and gyroscope). I now want to extract the activity out of it (e.g. walking, standing, driving, ...). I Followed this Jupyter Notebook. But there are some issues left.

Why do they only pick 500 rows?
What's the point of re-arranging the rows/columns?
When they build their decicion tree learner with the train data, they build it upon extracted features. But how can we then use this tree for new sensor data? Should we extract the features of the new data and pass it as input for the tree? But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)

there's a lot to un-pack in this question.
Why do they only pick 500 rows?
my guess: in order to keep the example running quickly. tsfresh usually takes a while to calculate its features.  note that when they evaluated their model, they took the last 500 samples.
What's the point of re-arranging the rows/columns?
answer: the data frame format that tsfresh requires in order to calculate features is that format. it is a bit of a pain...especially when you need to keep track of an id-column for other data.
When they build their decicion tree learner with the train data, they build it upon extracted features.
answer: yes
But how can we then use this tree for new sensor data?
Should we extract the features of the new data and pass it as input for the tree?
answer: yes
But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)
answer: yes it will. I don't know where your copy-pasted error message came from. When you generate a set of features using tsfresh, you can do it in a couple of different ways. you can generate all of them or you can generate a subset of them---they generated a subset...but then subsetted it once again(using their importance stuff...whether you use the importance methods or other methods to select features you will end up with a bunch of features you need to calculate). If you follow the procedure for generating features based on a pre-determined subset (relevant_features in their case), you will end up with the same # of features.  This needs to be stressed...don't generate tsfeatures that you don't need! as it will take foooorever.


text:
Ideal score of a model on training and cross validation data
The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.
I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:
(1) Train set: 0.99 | Cross-validation set: 0.72
(2) Train set: 0.75 | Cross-validation set: 0.70
(3) Train set: 0.69 | Cross-validation set: 0.69

These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.
My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data.
Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better with unseen data.

Assuming that your cross-validation scores(both on train set and test set) indicate model's prediction performance correctly, you should definitely decide which trained model to use based on your validation accuracy only, regardless your model is overfitted or not.

Hi, thanks for your answer. But I'm not sure if I got it entirely, can be please elaborate a bit more. From my above examples, which one would you go with and why?


text:
Effects of translating RL action probability through non linearity
I am training an RL agent (specifically using the PPO algorithm) on a game environment with 2 possible actions left or right.
The actions can be taken with varying "force"; e.g. go left 17% or go right 69.3%. Currently, I have the agent output 21 actions - 10 for left (in 10% increments), 10 for right in 10% increments and 1 for stay in place (do nothing). In other words, there is a direct 1-1 mapping in 10% increments between the agent output and the force the agent uses to move in the environment.
I am wondering, if instead of outputting 21 possible actions, I change the action space to a binary output and obtain the action probabilities. The probabilities will have the form, say, [70, 30]. That is, go left with 70% probability and go right with 30% probability. Then I take these probabilities and put them through a non-linearity that translates to the actual action force taken; e.g an output of 70% probability to go left, may in fact translate to moving left with 63.8% force.
The non linear translation is not directly observed by the agent but will determine the proceeding state, which is directly observed.
I don't fully understand what the implications of doing this will be. Is there any argument that this would increase performance (rewards) as the agent does not need to learn direct action mappings, rather just a binary probability output?

I don't fully understand what the implications of doing this will be.

Without other matching adjustments, you will break your agent.
The problem is how your new action space gets converted back into gradients to update the agent, after it has acted and needs to learn from the results. The NN component of policy function you are considering is designed to work by balancing a discrete probablility distribution. It learns by increasing the probability of actions (in the binary case, the probability of going left vs going right) that score better than a current baseline level.
When interpreting the result from going 63.8% left, you have to resolve two things - which action did the agent take, and what changes to your parameters will increase the probability of taking that action. Unfortunately neither of these tasks are simple if you combine the action choices in the way you suggest.
Also, you have lost exploration. The combined left/right algorithm will always output a fixed steering amount for each state. Whilst there are algorithms, like DDPG, that can work with this, it is not really possible to adapt PPO to do so.
However, PPO already supports continuous action spaces directly. You can have your network output the mean and standard deviation of a distribution for how to steer, and sample from that. Then the action choice taken will directly relate to the output of the network and you can adjust the policy to make that choice more or less possible depending on results from taking it. If you are using a library implementation of PPO, then this option should be available to you.

Have you considered using a continuous action space? It might be worth looking into. If you aren't familiar with it, here are a few resources for discrete vs continuous action spaces -
Modeling and Planning in Large State and Action Spaces
Deep Reinforcement Learning in Continuous Action Spaces:a Case Study in the Game of Simulated Curling

Can you elaborate your answer so as to avoid rather than being opinionated.


text:
What is the effect of picking action deterministicly at inference with Policy Gradient Methods?
In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution.
When evaluating the policy in an environment, what would be the effect of always picking the action that has the highest probability instead of sampling from the probability distribution?

when evaluating policy you don't need to sample, sampling is useful for inciting exploration during training.

When evaluating the policy in an environment, what would be the effect of always picking the action that has the highest probability instead of sampling from the probability distribution?

Depends what you mean by "evaluating the policy". Unlike in value-based methods, such as Q learning, the policy in gradient methods is not implied by anything else, it is described directly by the probability density function that is being optimised.
Taking the maximum probability item will technically change the policy (unless you are already using deterministic policy gradient), and you would be evaluating a different but related policy to that found by your policy gradient.
However, in a standard MDP environment, and after at least some training, this should be a reasonable process that would give some indication of how well the agent is performing.
In some cases, the nature of the environment means the agent is relying on a stochastic policy. In some partially-observable scenarios it may be better to decide randomly - a simple example is a corridor that needs to be traversed, but where the state features don't give enough information to determine the true direction. A deterministic policy will not be able to traverse the corridor in both directions, but a stochastic policy will get through it both ways, eventually. Another example is in adversarial situations where another agent can learn your agent's policy (the classic version of that being Scissor/Paper/Stone where two ideal opposed agents would learn probability $\frac{1}{3}$ for each action according to Nash equilibrium)
If you don't think you have these special cases, then it should be OK to derive a deterministic policy from your policy gradient agent, and assess that. That's not quite the same as assessing the "learned policy", but is quite sensible to do once you think the agent has converged anyway, since it may still be selecting non-optimal actions at some low probability, and you could get closer to optimal behaviour by removing that.


text:
Are there methods that allow deep networks to learn object categorization in a self-supervised way?
When training a deep network to learn object classification from a set like ImageNet, we minimize the cross entropy between the ground truth and the predicted  categories. This is done in a supervised way. It is my understanding that you can separate categories in an unsupervised way using principal component analysis, but I have never seen that in a deep network. I am curious if this can be done easily in the last case. One possible way to do this would be to minimize a loss that favors categorization into one-hot vectors (this would only guarantee that an image is classified into a single category, rather that the correct category, though). Has this been done, or is there any reason why not?

I did an experiment, took a trained densenet121 and kept the bottom layers. I trained the FC layer to a softmax and then to a lambda layer that normalized the vector. I trained the network with imagenet to make the outputs the most far a away from (1,1,1,1,1...1) as possible, so I would get one hot vectors. I did, but the network trained to a single category (put all in the same hot vector). Then I added a penalty that encourages to make vectors different is their features are also different. it improved a little, a dozen categories instead of one, bout noting close to a thousand, the available number.
I am posting this so nobody wastes his time in a silly idea like this one.


text:
Does adding a constant to all rewards change the set of optimal policies in episodic tasks?
I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?
The answer is Yes - Adding a constant to the reward signal can make longer episodes more or less advantageous (depending on whether the constant is positive or negative).
Can anyone explain why is this so? And why it doesn't change in the case of continuous (non episodic) tasks? I don't see why adding a constant matters - as an optimal policy would still want to get the maximum reward...
Can anyone give an example of this?

Generally we can write for $R_c$ the total reward with added constant $c$ of a policy as
$$
R_c = \sum_{i=0}^K (r_i + c) \gamma^i = \sum_{i=0}^K r_i \gamma^i + \sum_{i=0}^K c \gamma^i
$$
So if we have two policies with the same total reward (without added constant)
$$
\sum_{i=0}^{K_1} r_i^1 \gamma^i = \sum_{i=0}^{K_2} r_i^2 \gamma^i
$$
but with different lengths $K_1 \neq K_2$ the total reward with added constant will be different, because the second term in $R_c$ ( $\sum_{i=0}^K c \gamma^i$ ) will be different.
As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as:
$$
10 + 0 + 0 + 0 = 10
$$
and
$$
0 + 10 = 10
$$
But when we add 100 to every reward:
$$
110 + 100 + 100 + 100 = 410
$$
and
$$
100 + 110 = 210
$$
Thus, now the first one is better.
In the continious case, the episodes always have length $K = \infty$. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in $R_c$ stays the same.

Excellent answer!

Another important effect that distorts the problem and changes its solution is rewards sign changing. Explanation here at time 00:12


text:
Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?
I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position.
The input is any board situation. The output is a score or estimate for the probability to win, or how favorable a given position is. Where 1 = guaranteed to win and 0 = guaranteed to lose.
In any given turn I can then loop over all possible moves for the current player, evaluate the resulting game situations, and pick the one with the highest score.
Hopefully, by letting this neural network play a trillion games vs itself, it can develop a sensible scoring function resulting in strong play.
Question: How do I train such a network ?
In every game, I can keep evaluating and making moves back and forth until one of the AI players wins. In that case the last game situation (right before the winning move) for the winning player should have a target value of 1, and the opposite situation (for the losing player) has a target value of 0.
Note that I don't intend to make the evaluation network two-sided. I encode the game situation as always "my own" pieces vs "the opponent", and then evaluate a score from my own (i.e. the current player's) side or perspective. Then after I pick a move, I flip sides so to speak, so the opponent pieces now become my own and vice versa, and then evaluate scores again (now from the other side's perspective) for the next counter move.
So the input to such a network does explicitly encode black and white pieces, or naughts and crosses (in case of tic-tac-toe) but just my pieces vs their pieces. And then evaluates how favorable the given game situation is for me, always assuming it's my turn.
I can obviously assign a desired score or truth value for the last move in the game (1 for win, 0 for loss) but how do I backpropagate that towards earlier situations in a played game?
Should I somehow distribute the 1 or 0 result back a few steps, with a decaying adjustment factor or learning rate? In a game with 40 turns, it might make sense to consider the last few situations as good or bad (being close to winning or losing) but I guess that shouldn't reflect all the way back to the first few moves in the game.
Or am I completely mistaken with this approach and is this not how it's supposed to be done?

This idea is on the right track! you have pretty much defined alphaZero, where they train a NN to evaluate each move, and then use MC-search for the actual play throughs!

Ok thanks, and do you know if I can find some explanation or further documentation on the training process?

The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay.

Thanks, how do you mean the evaluation of the best next step? For example, if a simulated game of the neural network vs itself proceeds like this. The last move of A (in step 7) should obviously be scored 1 because it wins the game, and the preceding move of B (in step 6) should score 0 because it allowed the opponent to win. But then how further? The preceding step directly before that, move 5 by player A, was actually a very bad move, as player B could have won after that. And in fact player A started the game very badly. So how do I further assign scores?

You should also score it 1(maybe with decay, so 0,99), because it lead to a win. This will make it take this wrong decission more in the future though. Eventually B will learn how to defeat this move and at that point, you wil start scoring step 5 with 0 because it lost that game.

This means that the early steps won't be properly trained at the start of the training sessions, but given enough training games, eventually he will learn how to correctly do the early steps

OK I see, so this means I should rate A's moves 7, 5, 3, 1 with a score of 1, 0.99, 0.98, 0.97 respectively (or 0.99², 0.99³, etc) and B's moves 6, 4, 2 with a score of 0, 0.01, 0.02 respectively (or 1-0.99ⁿ). But one more thing: in this case A and B are actually the same neural network. So do I take the average of all the weight adjustments, i.e. learning rate η times 0.99ⁿ for the moves on the winning side (with n = number of remaining moves until the end of the game for that side) and η(1-0.99ⁿ) for the moves on the losing side. And then backpropagate the resulting average score adjustment?

you just train every move once right? I don't really know why you want to average the weight adjustments?

Ah, OK no I thought you meant some kind of (mini-)batch training and then apply weight adjustments once per game. Although now that I think about it, I guess it actually boils down to the same. So yes I can just adjust the weights for every individual move. Thanks again, much appreciated.

Was this project completed?  I am trying to understand to what extent tree search is required for play, as opposed to something much easier like you describe: loop through every next move, pick best.


text:
Can I shuffle image channel data as a form of data augmentation?
If I want to augment my dataset, is shuffling or permuting the channels (RGB) of an image a sensible augmentation for training a CNN? IIRC, the way convolutions work is that a kernel operates over parts of the image but maintains the order of the kernels.
For example, the kernel has $k \times k$ weights for each channel and the resulting output is the multiplication of the weights and the pixel values of the image and is finally averaged to form a new pixel in the next feature map.
In this case, if we shuffle the channels of the image (GBR, BGR, RBG, GRB, etc.), a CNN that is only trained on the ordering RGB would do poorly on such images. Therefore, is it not sensible to shuffle the channels of the image as a form of data augmentation? Or will this have a regularizing effect on the CNN model?

As a rule of thumb for image data augmentation, look at the augmented images:

Can you correctly classify or measure your target label from the augmented images?
Could something similar to the augmented images appear in the environment where you want to run inferences on previously unseen inputs?

For your suggested augmentation of shuffling the channels, it may pass the first test. However, the second test shows that you are probably taking a step too far.

will this have a regularizing effect on the CNN model?

Yes, but it might not be that useful to have strong cross-channel regularisation.
If there is important information for your task in the separate colour channels, then shuffling the channels makes it harder for the neural network to use that (it is not impossible, the CNN can still learn filters that will trigger most strongly on features that tend to appear in red channel and not blue in your problem for instance).
If there is not important information for your task in the colour information, then you may find it simpler and easier to turn your images into single channel greyscale instead, and use that throughout. Although that is not completely the same, for many image types it will achieve a similar effect (and possible boost to accuracy) for a fraction of the effort.

Understood, I guess the no free lunch theorem applies. I used RGB images to get the point across. On a more general level, if you had layers of image (say cross-sections of an MRI) and these layers was input in no particular order, it would be useful to know if the CNN is "layer-order-invariant"


text:
How can I find what does an specific neuron do in neural network?
How can I know what each neuron does in NN?
Consider the Playground from Tensorflow, there are some hidden layers with some neurons in each. Each of them shows a line(horizontal or vertical or ...). Where these shapes come from. I think they are understandable for nn not a person!

In TensorFlow Playground, the horizontal line show where each class is separated for each neuron. What happens when you take any intermediate neuron to make the decision? You can see the answer by the line provided by that neuron. And this decision is a result of the weighted sum from the decisions of the previous neurons (up to activation).
Take the middle-top neuron in the link you share, which is an almost horizontal line - slightly tilted to the right. This neuron classifies everything above it as a blue, and everything below it as an orange. Hover over the neuron to see a larger picture on the output.
You can also see how this is actually calculated by hovering over the line coming from the neurons in the previous layer to the neuron you are looking at. For the case of the same neuron (center-top), the weight coming from the first input ($x_1$) is 0.091, while from the second one ($x_2$), it is 0.49. The neuron ends up being almost horizontal because the contribution from the horizontal input ($x_2$) is so much larger compared to the vertical one ($x_1$).
Of course you need to take into account the nonlinearity coming from the activation function but the idea presented above is the essence of it. The example uses tanh activation, which behaves very linear in its intermediate region so one can ignore this issue to some extend for this particular case.
Edit: It appears that the values for the weights change at every browser session, so the neuron I describe might look a little different to you. To get the same configuration, simply click on the colored lines between neurons to edit them and use the values above for the connections.

I think serali answered this question well, though I wanted to give some extra reading for those interested.
There are many ways of deciphering what a neuron in a NN is doing. This lecture does a fantastic job at covering some of these methods and is an incredibly interesting watch. This covers more advanced methods of visualising what a model is doing.


text:
In unsupervised learning, what is meant by "finding the probability of an image"?
The specific problem I'm having is with a Fully Visible Belief Network. It is an explicit density model (though I don't know what quantifies something being such) that uses the chain rule to decompose the likelihood of an image x into a product of 1-d distributions.

What is meant by "the likelihood of an image x"? With respect to what? I assume it refers to how common this image would be in the data set it is selected from? Like if you had 1000 images, 800 of which were white and 200 of which were black, the model should ideally output 0.2 for any black image inputted? Obliviously with more complicated clustering like dogs vs cats it'd be a bit different, but that's my intuition. Is that correct?
Also as a side question, that equation looks very wrong. If you have an image of $1048\times720$ pixels, and say every pixel evaluates to have a probability of 0.9, you'd expect the final probability of the image to be 0.9 or 90%. But according to that equation, it's $0.9^{720*1048}$, which is stupidly small, essentially 0. What's going on here?

@DuttaA Ok that makes sense. Do you know if my first assumption about the meaning of "likelihood of an image x" is correct?

In practice, you often work with log probabilities, rather than probabilities. Maybe later I will have a closer look at this question and maybe try to provide an answer.

When you say likelihood, you are invoking several other concepts like events, sample, parameters, model, probability density function (PDF), etc (it would be helpful if you learn more about these concepts). In essence, a likelihood function $l(x|\theta)$ is a PDF that quantifies how likely is that event $x$ happens out of a set of possible events, given the parameters $\theta$ that define your model.
In the specific case of images, the set of possible events is usually one of two options: 1) all the available images in a dataset, or 2) all the existing images. Usually you want to model the likelihood in the option 2), but only having access to a sample of all the possible images. In either case, the likelihood is just the probability that you select one image out of all the possible ones. If you consider only images of $1048\times 720$ pixels, the possible amount of images is $(256\times3)^{1048\times 720}$, where I am assuming that each pixel consists of 3 colors and each color can take 256 values. Since the amount of possible images is so so big, it is very common that the probability of selecting a specific one is very very small. This is a reason why you usually work with log-likelihood (the logarithm of the likelihood) instead of directly using likelihoods. For example, if all your images were equally probable, the likelihood would be in the order of $10^{-{10^7}}$, while the log-likelihood would be around $-10^7$.
To solve your paradox with the probability of images and pixels, consider that instead of pixels you have coins and instead of images you have sequences of coins. Let's say that you have a fair coin, so the probability of tails ($T$) after tossing the coin is 0.5. If you toss a second coin, the probability of having $T$ again is naturally 0.5 as well, but what is the probability that both results where $T$? It is the product (0.25) since the events are independent. Similarly, the probability of the other three sequences $TH$, $HT$ and $HH$ is just 0.25. You can see that since the probability needs to be shared between 4 sequences equally probable, they are less probable with respect to the probabilities of the sequences of length 1. If you toss the coin 3 times, then the probability of all these coins being tails is just $0.5^{3}$. Again, there are now 8 possible sequences, all sharing the same amount of probability. You can see what's going on. Since the amount of possible options becomes large, the probability of each possible sequence of coins becomes small. Clearly, you would never toss a coin 10 times and expect to obtain all $T$, right? Well, exactly the same happens in the case of the pictures.

Thanks for the answer! It makes sense why you would multiply, and use a log. I'm still a bit confused as to why a PDF in this case is being used for a discrete set (while it is massive, the set of all images of defined dimensions is finite). Moreover, normally when you want to find the probability of an event you don't find the exact events probability, but a set of events with some tolerance as an estimate, as an exact event occurring has a likelihood of 0. Is that done in unsupervised learning?

@Recessive, you are right in saying that the likelihood is a prob. mass function if the set of events is discrete. It just seemed more general to refer to a density function. Now, in the case of images, actually it is more common to suppose that the pixel values are continuous and that the likelihood is a PDF. This supposition is made precisely because the set of events is so massive and the probabilities become so insignificant. The numbers I used were mostly to try to give you an intuition.


text:
Prerequisites for Andrew Ng's Machine Learning Course
I am planning to enroll for Andrew Ng's Machine Learning course https://www.coursera.org/learn/machine-learning. I've no background in math. Is it OK if I start the course and learn math as and when required?

This course is focused on machine learning using MATLAB, which is not practical nowadays as it is a programming language used specifically for computing, and cannot display GUI or communicate through the network. The language is powerful but limited in some ways. Nowadays most people use python for machine learning, as it is versatile and can connect to other backend like C++, java, JavaScript easily. The language is also a general language, and unlike MATLAB can do many things not limited to computing.
If you really want to join this course, I would recommend first learning MATLAB language and also learn basic calculus like derivatives. This would greatly help on your learning of the course.
However if you want to here serious about machine learning, I would encourage you to enroll in the Deep Learning specialization also by Andrew Ng on Coursera.
https://www.coursera.org/specializations/deep-learning
This course uses python as the programming language and teaches more modern approaches to deep learning like recurrent neural networks, convolutional neural networks and more. It also talks more about application of neural network. There is also theory, but it also talks about application of a specific algorithm and how it works.
Hope I can help you.

Now I have enrolled and the course is starting from 9th Dec 2019.  As I've liking for the course but no background in calculus and linear algebra, I was just thinking how can I learn the concepts as quickly as possible.

Did you started the deep learning specialization or the Machine learning course that uses MATLAB? I first started deep learning without any calculus experience, but I believe you could learn the basic calculus required for deep learning from Khan Academy easily. Those are not that hard but I strongly encourage you to check it out. Good luck on your course!

I've started Andrew Ng's Machine Learning.

Great! Try learning some basic MATLAB and also calculus to get you started. Also as a quick reminder you can finish and complete the course whenever you want, and late submission to homeworks won't affect your mark whatsoever. Good luck!

No pre-requisites required for Andrew Ng ML course. There are a couple of lectures in which he gives basic idea of Linear algebra. Also you can learn math when required.

I must simply direct you to this excellent blog post on Machine Learning Mastery:
https://machinelearningmastery.com/what-is-holding-you-back-from-your-machine-learning-goals/


text:
Does using the softmax function in Q learning not defeat the purpose of Q learning?
It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that the reward is maximised.
I understand for this to be properly calculated, you must explore all possible game states, and as that is obviously intractable, a neural network is used to approximate this function.
In a normal case, the network is updated based on the MSE of the actual reward received and the networks predicted reward. So a simple network that is meant to chose a direction to move would receive a positive gradient for all state predictions for the entire game and do a normal backprop step from there.
However, to me, it makes intuitive sense to have the final layer of the network be a softmax function for some games. This is because in a lot of cases (like Go for example), only one "move" can be chosen per game state, and as such, only one neuron should be active. It also seems to me that would work well with the gradient update, and the network would learn appropriately.
But the big problem here is, this is no longer Q learning. The network no longer predicts the reward for each possible move, it now predicts which move is likely to give the greatest reward.
Am I wrong in my assumptions about Q learning? Is the softmax function used in Q learning at all?

Just to let you know I have updated my answer as I thought of a way in which softamx distribution is used in Q learning, indirectly.

However, to me, it makes intuitive sense to have the final layer of the network be a softmax function for some games. This is because in a lot of cases (like Go for example), only one "move" can be chosen per game state, and as such, only one neuron should be active.

You are describing a network that approximates as policy function, $\pi(a|s)$, for a discrete set of actions.

It also seems to me that would work well with the gradient update, and the network would learn appropriately.

Yes there are ways to do this, based on the Policy Gradient Theorem. If you read it you will probably discover this is more complex to understand than you first thought, the problem being that the agent is never directly told what the "best" action is in order to simply learn in a supervised manner. Instead, it has to be inferred from rewards observed whilst acting. This is a bit harder to figure out than the Q learning update rules which are just sampling from the Bellman optimality equation.
You can split Reinforcement Learning methods broadly into value-based methods and policy gradient methods. Q learning is a value-based method, whilst REINFORCE is a basic policy gradient method. It is also common to use a value based method within a policy gradient method in order to help estimate likely future return used to drive the polcy gradient updates - this combination is called Actor-Critic where the actor learns a policy function $\pi(a|s)$ and the critic learns a value function e.g. $V(s)$.

But the big problem here is, this is no longer Q learning. The network no longer predicts the reward for each possible move, it now predicts which move is likely to give the greatest reward.

This is true, but it is not a big problem. The main issue is that policy gradient methods are more complex than value based methods. They may or may not be more effective, it depends on the environment you are tryng to create an optimal agent for.

Is the softmax function used in Q learning at all?

I cannot think of any non-contrived environment in which this function would be useful for an action value approximation.
However, it is possible to use a variant of softmax to create a behaviour policy for Q learning. This uses a temperature hyperparameter $T$ to weight the Q values, and provide a probability of selecting an action, as follows
$$\pi(a_i|s) = \frac{e^{Q(s,a_i)/T}}{\sum_j e^{Q(s,a_j)/T}}$$
when $T$ is high all the probabilities of actions will be similar, when it is low even a small difference in $Q(s,a_i)$ will make a big difference to probability of selecting action $a_i$. This is quite a nice distribution for exploring whilst avoiding previously bad decisions. It will tend to focus the agent on exploring differences between similarly high rated actions. The main issue with it is that it introduces hyperparameters for deciding starting $T$, ending $T$ and how to move between them.

This is still Q-learning, remember Q-learning is off-policy value-based.
For Bellman optimality operator  $\mathcal{T}Q=r+max\ Q'$.
If you have enough exploration, it always takes $Q$ to the optimal fixed point.


text:
How can I detect diagram region and extract (crop) it from a research paper
How can I detect diagram region and extract(crop) it from a research paper

your question is pretty general and I'm guessing by the down-vote that others feel the same.  perhaps research a few methods quickly and learn roughly what they mean, make a list of them and pose a question about which of the methods on your list have worked for others. a quick google of "how to detect and crop a region of an image" will get you started.

There are many methods that extract the image from file and you can apply image detection after that but you did not tell what type of file you need to extract(crop). So, your question needs to be clarified.

I will consider that you need to extract(crop) the digram from the pdf research paper. You can use PyPDF2 or PyMuPDF to extract the images from the PDF file and then you can apply machine learning to do recognition and classification of the images. There are different types of machine learning solutions for image classification and you can start with Convolutional Neural Network and you can start here.
For more information for Recognition and classification of figures  see these  and to extract of figures from scholarly documents read this


text:
What makes GAN or VAE better at image generation than NN that directly maps inputs to images
Say a simple neural network's input is a collection of tags (encoded in some way), and the output is an image that corresponds to those tags. Say this network consists of some dense layers and some reverse (transpose) convolution layers.
What is the disadvantage of this network, that directs people to invent fairly complicated things like GANs or VAEs?

The only disadvantage and difference between these generative models and the method you describe, is the input. You describe inputting tags, where as for a GAN, or VAE, the generation segment of the model takes in some representation of a probability distribution. For a GAN, it's mostly random noise, and for a VAE it is some latent space (see nbros answer).
Your described method prevents the network from properly learning fluid generation. If you have a discrete input, the network will attempt to perform a sort of classification on the input, rather then generation, and so when you try to generate a new image, you will most likely get the image equivalent of gibberish.
In fact, that's why a standard auto-encoder (not variational) doesn't work very well for generation. You would think that you could feed in your own custom input into the latent space:
But if you tried this, you would end up telling the network to try and generate something from a latent space it can't interpret.
Hence where the "variational" part of the VAE comes in. This helps the network learn to generate from a continuous distribution, so no matter what input you use, the network will be able to interpret it and give an appropriate output.
As for a GAN, it is simply fed random noise at each training step, so it too generates based on a continuous distribution.
If you were to try and train your method of generation, I would predict that you would find an average of all images that share similar tags (say you have the tags of cat, dog and brown haired, if you input "dog=1, cat=0, brown haired=1" you would get an average of all brown haired dogs), but if you tried to input a combination of tags the network has not seen, as it has not learnt from a continuous distribution, the resultant image would not be anything like what you'd expect from those tags.

Thanks! I guess in the end the main problem with direct mapping is that the network never sees intermediate inputs. So it is unlikely that it will produce high quality images for them.

I will only focus on the VAE because I am more familiar with it, but the explanations may also apply to the GAN and other generative models.
In the case of the VAE, you train a neural network not only to generate images but to represent them compactly in a so-called latent space, so you train the VAE to do dimensionality reduction. More precisely, the VAE attempts to learn a probability distribution with smaller dimensionality than the dimensionality of the training data but that hopefully represents the training data. Consequently, the model is forced to learn the essential features of the probability distribution that generated the training data.
The VAE is a generative model rather than a discriminative model. In the case of the neural network that maps inputs to images, you will not be learning a latent probability distribution (unless you formulate your model in such a way), from which you could sample, but you would just be mapping, in a supervised way (that is, you would need a labelled training dataset) and deterministically, the inputs to the images. In the VAE, there's some form of stochasticity, while training (and testing) the model.

I will probably improve this answer later!


text:
What is the difference between Inductive Learning and Connectionist Learning?
According to what we know about inductive and connectionist learning, what is the difference between them ?
For those who do not know about :
Inductive Learning, like what we have in decision tree and make a decision based on amount of samples
Connectionist Learning, like what we have in artificial neural network

All of the statistical learning is about inductive learning.
What is the difference between inductive learning and connectionist learning?
Inductive learning is about identifying patterns from examples. It is more related to statistics. Connectionist learning is more about finding a common pattern and predicting as well as self-learning(learning from the experience of prediction).
Connectionist learning is where learning occurs by modifying connection strengths based on experience. This is not the case with inductive learning. In inductive learning, we are not modifying things based on experience. Inductive learning just finds common patterns, not self-learning based on experience.

Learning requires both practice and rewards

In inductive learning, we learn the model from raw data (so-called training set), and in the deductive learning, the model is applied to predict the behaviour of new data.

Connectionist Learning is a group of inductive learning and deductive learning.

There several things I don't like about this answer. First, you use the term "self-learning", without providing a definition of this non-standard term. Second, you say that IL is "identifying patterns from examples" and then you say that CL is "more about finding a common pattern and predicting". This distinction is not clear because you're almost saying the same thing. The sentence "In inductive learning, we are not modifying things based on experience" is also wrong. See https://en.wikipedia.org/wiki/Inductive_reasoning. There are other problems.


text:
Are there any strategies that would help me visualize the 'behavior space' and make a novelty function?
In “Abandoning Objectives: Evolution through the Search for Novelty Alone”, it is explained how the novelty search is a function that is domain specific, depending on the differing behaviors that can potentially emerge.
The primary test is a deceptive maze and it seems like they define novelty as a function that is dependent on each actor's ending position as a distance from other actors' ending position.
I am wanting to try implementing this on some tasks. Some simple AI tasks such as playing pong, or recreating MarI/O, or sticking them in an arena as an actor who can move, turn, and shoot (with other actors in the arena with them).
I have a really hard time thinking of how to model the behavior functions for these kinds of instances without making it into an objective.
For pong, I imagine I could determine novelty by the AI's point score, but isn't this basically making the score an objective since it can only go up? For MarI/O, I've seen some implementations that look at the list of unique grid locations that Mario visited in what order, but I didn't come up with that myself.
For the arena example, my first impulse is to have a score based on how long the actor survived and how many other actors the AI eliminated; but again, this can only go up and seems to me like it is defining an objective.
Are there any strategies or ways to think about the problems that would help me better visualize the 'behavior space' and make a better novelty function?

I think that the best approach is to "switch point of view" from the general, objective-oriented, Genetic Algorithm's behaviours.
Usually GAs rely on individualism: the best survives. To do this you have to define what 'best' means and this is done through a fitness function that measures something objective, independent from the individual (i.e. a score, classical example).
If you want to measure novelty, collectivism becomes more important, as the whole population (present and past) must be considered. You have to think about measuring how different the individual X did from 'the rest' (i.e. location never reached before).
Another interesting point is that you should think in a more multidimensional way: a score is uni-dimensional, everyone wants to get higher, that is easy. A location in space is more difficult to achieve and that is why novelty becomes interesting.

To sum up: stay multidimensional, always treat the population as a multi-organism rather than focusing on the single individual. I hope this helps :)


text:
Training methods for bipedal robot
I am looking to train a bipedal robot using unity as a scape with a genetic algorithm. I will import the CAD into unity so the hardware is exact. My questions:

Is Unity physics accurate enough to train a neural network that will perform in the real world?
Should I optimize the network using reinforcement learning in the real world (after trained in scape)?
I am looking to use air muscles for my build. If the physics aren’t exactly right in unity (elasticity, max length, torque) will the bot still perform in the real world?
Are there any other programs that would be better than unity to train a robot inside a scape?
Any other approaches or new ideas on how to train the bot more efficiently would be greatly appreciated.

Not really. Unity physics is just an approximation of an approximation. It has to look more or less real but at the same time the performances are very important, so it has not the realistic level you would hope to "bring things to the real world". There are some physics engine you can install that usually work a bit better. Still, don't expect "real-world level".
Based on 1, yes. If, once you got it to the real world you still give it the chance to optimize, it can only make things better.
This is difficult to say. You should try to use the best possible approximation in Unity, maybe testing multiple scenarios and seeing that they perform more or less in the same way. Then train. Then bring this "approximated model" to the real world and let it train more to adjust to the real physics.
I don't know this. Unity seems the most versatile around (together with other game engines). This versatility is its strength and weakness, as it cannot focus on solving perfectly a single problem. It rather has to aim at generalizing as much as possible. There might be "more specialized" programs around, but I doubt.
Just try different algorithms/methods. Usually with complex problems neuroevolutionary techniques perform better, but the amount of code is hugely higher.


text:
Why does the denoising autoencoder always returns the same output?
I am trying to implement a denoising autoencoder (DAE) to remove noise from 1024-point FFT spectra. I am using two types of spectra: (1) that contain a distinctive high amplitude spectral peak and (2) that contain only noise peaks.
If I understood correctly, I can train the DAE using the corruputed spectra (spectra+noise) and afterwards I can use it the remove noise from new datasets. The problem is that when testing the DAE, it returns the type (1) spectrum mentioned above, regardless of the input. The same case when I apply predict on the training data. This is the code I am using (Python/Tensorflow):
def BuildModel(nInput):
    input_dim = Input(shape = (nInput, ))

    # Encoder Layers
    encoded1 = Dense(896, activation = 'relu')(input_dim)
    encoded2 = Dense(768, activation = 'relu')(encoded1)
    encoded3 = Dense(640, activation = 'relu')(encoded2)
    encoded4 = Dense(512, activation = 'relu')(encoded3)
    encoded5 = Dense(384, activation = 'relu')(encoded4)
    encoded6 = Dense(256, activation = 'relu')(encoded5)
    encoded7 = Dense(encoding_dim, activation = 'relu')(encoded6)

    # Decoder Layers
    decoded1 = Dense(256, activation = 'relu')(encoded7)
    decoded2 = Dense(384, activation = 'relu')(decoded1)
    decoded3 = Dense(512, activation = 'relu')(decoded2)
    decoded4 = Dense(640, activation = 'relu')(decoded3)
    decoded5 = Dense(768, activation = 'relu')(decoded4)
    decoded6 = Dense(896, activation = 'relu')(decoded5)
    decoded7 = Dense(nInput, activation = 'sigmoid')(decoded6)

    # Combine Encoder and Deocoder layers
    autoencoder = Model(inputs = input_dim, outputs = decoded7)

    autoencoder.summary()
    # Compile the Model
    autoencoder.compile(optimizer=OPTIMIZER, loss='binary_crossentropy')
    #autoencoder.compile(loss='mean_squared_error', optimizer = RMSprop())

    return autoencoder

X_train, X_test, y_train, y_test = train_test_split(spectra.iloc[:,0:spectra.shape[1]-1], spectra['Class'], test_size=testDatasetSize, stratify=spectra.Class, random_state=seedValue)

X_train, y_train = shuffle(X_train, y_train, random_state=seedValue)
X_test, y_test = shuffle(X_test, y_test, random_state=seedValue)

X_unseen = X_train.to_numpy()[0:1000,:] # Data not used for training, only for testing
y_unseen = y_train.to_numpy()[0:1000]
X_train = X_train.iloc[1000:]
y_train = y_train.iloc[1000:]

# Scaling
maxVal = max(X_train)
X_train = (X_train/maxVal).to_numpy()
X_test = (X_test/maxVal).to_numpy()
X_unseen = (X_unseen/maxVal)#.to_numpy()

# Corrupted data
noise_factor = 0.01
X_train_noisy = X_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_train.shape)
X_test_noisy = X_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_test.shape)
X_unseen_noisy = X_unseen + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_unseen.shape)

ae = BuildModel(X_train.shape[1])
PrintConsoleLine('Creating model finished')
print('')

history = ae.fit(X_train_noisy, X_train, epochs=NB_EPOCH, batch_size=BATCH_SIZE, validation_data=[X_test_noisy, X_test])
save_model(ae, modelFile, overwrite=True)

# Test
X = X_unseen
X_noisy = X_unseen_noisy
X_denoised = ae.predict(X_noisy) # X_train gives the same result (spectra type (1)) !?!
N = len(X_denoised[0,:])
index = 6
PlotDataSimple(3, np.linspace(0,N-1,N), X[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
PlotDataSimple(4, np.linspace(0,N-1,N), X_noisy[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
PlotDataSimple(5, np.linspace(0,N-1,N), X_denoised[index, :], 'Frequency domain', 'Index', 'Amplitude', None)

Dataset shape: (17000, 65, 65, 1) (files, samples X axis, samples Y axis, class)
Train on 12600 samples, validate on 3400 samples
Epoch 1/3
12600/12600 [==============================] - 26s 2ms/sample - loss: 0.6813 - val_loss: 0.4913
Epoch 2/3
12600/12600 [==============================] - 14s 1ms/sample - loss: 0.1621 - val_loss: 0.0578
Epoch 3/3
12600/12600 [==============================] - 16s 1ms/sample - loss: 0.0230 - val_loss: 0.0169

The results I am getting (Column 1 - Initial signal, Column 2 - Corrupted signal, Column 3 - Denoised signal):

So why does the DAE output the same spectra regardless of the inputs? Am I misunderstanding the DAE principle or is there a problem in my implementation?

You are using Dense layers, try 1d convolution instead.
Have you tried a different activation function such as softmax and instead of Binary cross entropy try MSE loss? Are all your inputs between 0 and 1? Also, I think your noise amplitude is too much in 2nd and 3rd case as compared to the actual signal. Can you try training on different types of spectra separately and check the result, if the DAE is learning anything. Also, try trining for more epochs.
I am not sure and would have commented this but I can't due to low rep, but I am interested in the solution.

Thank you for your suggestions. I have tried them and found that the main issue was that I performed scaling by diving to the max value of ALL the spectra. After performing the scaling individually for each spectrum, changing to MSE loss and using LeakyReLU as the activation function, with 100 training epochs, the solution works better. I will try to implement with Conv1D, but I have a hard time understanding how to implement conv nets. I would be very grateful if you could show/help me how to rewrite the model as a convolutional one.

@CristianM the documentation will help you with the implementation of convnets. Instead of dense, you just use conv1d layers and pass the size of convolutioton filter with it.

If the auto-encoder is converging to the same encoding for different instances, there may be a problem in the loss function. Check the size and shape of the output of the loss function, as it may be getting confused and evaluating the wrong tensors (i.e. you may need to transpose something somewhere).
Basically, assuming you are using an auto-encoder to encode $M$ features of $N$ training instances, your loss function should return $N$ values i.e.,
the size of your loss tensor should be the amount of instances in your training set.


text:
How to explain peak in training history of a convolutional neural network?
I am training a simple convolutional neural network to recognize two types of 1024-point frequency spectra (FFT). This is the model I'm using:
cnn = Sequential()
cnn.add(Conv1D(filters=64, kernel_size=3, activation=LeakyReLU(), input_shape=(nInput,1)))
cnn.add(Conv1D(filters=64, kernel_size=3, activation=LeakyReLU()))
cnn.add(MaxPooling1D(pool_size=2))
cnn.add(Flatten())
cnn.add(Dense(nFinalDense, activation=LeakyReLU()))
cnn.add(Dense(nOutput, activation='sigmoid'))

However I get the following accuracy and loss during training:

Why do I get the large peak in both plots? How can it be explained? Is there a problem with the data I'm using (I mention that I obtain a similar peak when training an autoencoder for denoising using the same data)?

@DuttaA I get it multiple times. I'm using Tensorflow to train the model with default weight initializers for each layer, so it automatically randomizes the weights. I mention that I set the seed values for numpy and tensorflow at the beginning of the script to get a more deterministic output. I've changed the seed value and I get a smaller peak at a different epoch. What do you think it is?

I found that the peak was caused by the data I am using. Specifically, the MinMaxScaler changed the data shape and I resolved the issue by simply dividing to the max value.


text:
How to use LSTM to generate a paragraph
A LSTM model can be trained to generate text sequences by feeding the first word. After feeding the first word, the model will generate a sequence of words (a sentence). Feed the first word to get the second word, feed the first word + the second word to get the third word, and so on.
However, about the next sentence, what should be the next first word? The thing is to generate a paragraph of multiple sentences.

How do you know when the first sentence finishes? If punctuation is included in the "words", then you can just feed that into the model. Or otherwise the last word of the previous sentence.

@datdinhquoc similar to what Oliver just mentioned, but really its how it was trained. This may include just using the last token in the last sentence (probable if punctuation is included) or a seperator token or etc. Usually how it was trained will implicate how inference will work

interesting, use the end-of-sentence punctuation to generate next sentence

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://openai.com/blog/better-language-models/

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens are divided differently. Some model maybe character based models which simply uses each character as input and output. In this case you can treat punctuation as one character and just run the model as normal. For word based model which is commonly used in many systems, we treat punctuation as it's own token. It is commonly called a end of sentence token. There is also a specific token for end of output. This makes the system knows when to finish and stop prediction.
Also, just so you know for language model trying to generate original text, they feed the output as the input of the next data point, but the output they choose is not necessarily the one with the best accuracy. They set a threshold and choose upon that. It can introduce diversity to the language model so taht even though the staring word is the same, the sentence/paragraph will be different and not the same one again and again.
For some state-of-the-art models, you can try GPT-2 as mentioned by @jdleoj23 . This is a character based(actually byte based but basically the same, it treats each unicode symbol individually) model that uses attention and transformers. The advantage of character based system is that even inputs that have spelling errors can be inputted into the model and new words not in the dictionary can be introduced.
However if you want to learn more about how language model works, and not just striving for the best performance, you should try implementing one simple one by yourself. You can try following this article which uses keras to make a language model.
https://machinelearningmastery.com/develop-word-based-neural-language-models-python-keras/
The advantage of making a simple one is taht you can actually understand the encoding process, the tokenization process, the model underneath and others instead of relying on other people's code. The article uses keras Tokenizer but you could try writing your own using regex and simple string processing.
Hope my help is useful for you.


text:
How to make DNN learn multiplication/division?
A single neuron with 2 weights and identity activation can learn addition/subtraction as the 2 weights will converge to 1 and 1 (addition), or 1 and -1 (subtraction).
However, for multiplication and division, it's not that easy. Can a single neuron learn multiplication or division? If not, how many layers of DNN can learn these?

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you only need one hidden layer to approximate the multiplication on a compact, note that you need to apply a non linear activation on the hidden layer to do this.

All functions are bounded on a compact