text:
State of the art and possible development of judging AIs in laws field
I'd like a general explanation of that in AIs that were to mimic judges, prosecutors or lawyers, on very general terms they would act on this way for each case:
A judge AI would give a verdict, having the following input:
All sources of law that have some relationship with the act being judged, knowing the relative importance of each of that sources.
All the facts presented to it, being theorically able to distinguish if some fact were false if there are enough evidence for that.
Prosecutors AI would act into searching the facts that would most help for accusing, matching them with known sources of law so they can make the best accusation.
Lawyers AI would act exactly as prosecutors but for defense of the one being accused.
Obviously AIs are way too far nowadays from something like this, but I find strange that it doesn't even look that this has been done as a proof of concept, the most near thing I find by searching on the Internet is this. I'm not sure of the reach of everything that it's explained there but it looks more of assisting judges, prosecutors and lawyers rather than making what they do. I find this a bit strange, although I'm very far from being an expert on AIs, I think that for very simple cases an AI could even do those things.
In the answer about possible development, I'd like to know about which would be the facts that would make developing an AI like the one I'm mentioning to be possible or not.
PD: I don't find any suitable tag, I ask an admin to put a better one if needed.
Requested edit: My question is about the general state of the art of this field (of AIs being able to somehow mimic a judge, a prosecutor or a lawyer), and how pausible is a development in this way and how it would develop (if this is with current knowledge impossible to even try until a far future, or if we are at least near for stablishing AIs understanding law, which would be the logical way for a type of AIs like them evolving (understing laws, judging, being able to see videos..). I know it's broad so I'm happy with a general explanation.

It would be helpful if you could do another edit and try to clarify the actual question.  My initial answer would be that although Natural Language Processing has made great strides, actual understanding of the language is problematic.  (I can elaborate more in a formal answer if this is the type of information you were looking for.)  I do seem to recall hearing something recently in regard to using AI for sentencing...  PS-Welcome to AI!

There has definitely been work relating to judicial sentencing:

In a new study from the National Bureau of Economic Research, economists and computer scientists trained an algorithm to predict whether defendants were a flight risk from their rap sheet and court records using data from hundreds of thousands of cases in New York City. When tested on over a hundred thousand more cases that it hadn’t seen before, the algorithm proved better at predicting what defendants will do after release than judges.Source: M.I.T. Technology Review: How to Upgrade Judges with Machine Learning

In terms of actually trying cases to deliver a verdict of guilty or not guilty, I think that would be difficult because, although Machine Learning has been effective at outperforming humans in problems where parameters can be clearly defined, we haven't figured out how to tackle the symbol grounding problem. This means that making decisions based on language, which is less precise than mathematics, would be highly problematic. Guilty/Not Guilty is a basic True/False condition, but the rest of the language utilized in law cannot be so precisely translated.


text:
Why do we need weights when training a perceptron as an OR gate?
Without using any of Matlab's neural network tools, I'm writing a program to simulate an OR gate with a perceptron. I have seen many tutorials, but I still can't understand why we need weights to train a perceptron for such a simple purpose.
One way is to program the perceptron with the conditions (0,0)=0. (1,0)=1. (0,1)=1. (1,1)=1. So the two inputs to the perceptron would be either zeroes or ones. I don't see the purpose of weights here. Assuming weights are 1, for the second training example, the output would be 1*1 + 0*1 = 1. For the last example, it would be 1*1 + 1*1 = 2. So an activation function which says if output >= 1, output = 1 else output =0; end should suffice. This would successfully simulate an OR gate. So why do I need to "train" any weights?

Update: I got a hint from the Perceptron Learning example. University of Kurdistan. The table on page 4 is self explanatory, but the question still persists. Why weights? http://eng.uok.ac.ir/esmaili/teaching/spring2012/nn/slides/Perceptron%20Learning%20Example.pdf

Why would you use a perceptron as an OR gate at all?  It is a pedagogical tool to show how a perceptron can map to a specific function.

@antlersoft: It's for a pedagogical purpose. I'm learning how these things work. Hard to believe these techniques model the brain. Looks like it's designed to model the computer more than a brain. Even if the brain performs logical operations.

There are two aspects that you need to consider:

Because it is part of the definition of a perceptron. There are many different mechanisms to implement an OR gate, most of them rather easy. But if you want to implement it based on a perceptron, you have to follow the definition of the perceptron. Otherwise you would just implement an OR gate without using an actual perceptron.
The purpose of an neural network is to initialize it more or less randomly without knowing the correct weights and biases for each neuron (or perceptron in your case). You just know the desired behavior of the network and you train it accordingly. In your example you know the desired output of the OR gate and can train your network to give you this desired output. What's bothering you is the fact that you can easily derive the correct weights without actual training the network just by analyzing it. That's because this is a theoretical example to show that a perceptron is capable to learn the behavior of an OR gate. In a real application you would not benefit from artificially crafting an OR gate based on an perceptron.

This would successfully simulate an OR gate.

Of course. In fact, hardware implementation of an OR gate needs just a few transistors. It may sound surprising, but the best python implementation of OR is a or b.

So why do I need to "train" any weights?

It's one of the simplest functions out there, everyone understands how OR works just from the boolean table. The task shows that a neural network can learn it as well, just from examples and without specification what a logical operation is and how to wire it.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?
When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that they might just be sentences.

Even tutorial sites like this one start talking about the usual parameters, but also say "model_name: This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights". So are the 175 billion "parameters" just neural weights? Why then are they called parameters? GPT3's paper shows that there are only 96 layers, so I'm assuming it's not a very deep network, but extremely fat. Or does it mean that each "parameter" is just a representation of the encoders or decoders?

An excerpt from this website shows tokens:

In this case, there are two additional parameters that can be passed
to gpt2.generate(): truncate and include_prefix. For example, if each
short text begins with a <|startoftext|> token and ends with a
<|endoftext|>, then setting prefix='<|startoftext|>',
truncate=<|endoftext|>', and include_prefix=False, and length is
sufficient, then gpt-2-simple will automatically extract the shortform
texts, even when generating in batches.

So are the parameters various kinds of tokens that are manually created by humans who try to fine-tune the models? Still, 175 billion such fine-tuning parameters is too high for humans to create, so I assume the "parameters" are auto-generated somehow.
The attention-based paper mentions the query-key-value weight matrices as the "parameters". Even if it is these weights, I'd just like to know what kind of a process generates these parameters, who chooses the parameters and specifies the relevance of words? If it's created automatically, how is it done?

Parameters is a synonym for weights, which are what you learn. Batch size, learning rate etc are typically hyper parameters

Thank you David. So now my understanding is that GPT3 has 96 layers and 175 billion nodes (weights or parameters) arranged in various ways as part of the transformer model.

It won’t have 175million nodes, if you think of a simpler neural network then the number of parameters is how many connections there are between nodes. If there was a NN with 1 layer of 2 neurons followed by a layer with 3 neurons, then an output layer with 1 neuron there would be 9 parameters (you could draw this out to see what I mean).

Understood. The weights are vectors which are called the "learned parameters". Thank you very much, David. If you post it as an answer, I'll accept it. Hopefully, this would be of help to anyone else searching for this answer.

Note that @DavidIreland is assuming a fully connected NN for 9 parameters

and no biases, if you’re being pedantic @JordanMorris

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas weights are what the learning algorithm will learn through training.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to calculate multiobjective optimization cost for ordinary problems?
What I did:
Created a population of 2D legged robots in a simulated environment. Found the best motor rotation values to make the robots move rightward, using an objective function with Differential Evolution (could use PSO or GA too), that returned the distance moved rightward. Gradient descent used for improving fitness.
What I want to do:
Add more objectives. To find the best motor rotation, with the least motion possible, with the least jittery motion, without toppling the body upside down and making the least collision impact on the floor.
What I found:

Spent almost two weeks searching for solutions, reading research
papers, going through tutorials on Pareto optimality, installing
libraries and trying the example programs.
Using pairing functions to create a cost function wasn't good
enough.
There are many multi-objective PSO, DE, GA etc., but they seem
to be built for solving some other kind of problem.

Where I need help:

Existing multi objective algorithms seem to use some pre-existing
minimization and maximization functions (Fonseca, Kursawe, OneMax,
DTLZ1, ZDT1, etc.) and it's confusing to understand how I can use my
own maximization and minimization functions with the libraries.
(minimize(motorRotation), maximize(distance),
minimize(collisionImpact), constant(bodyAngle)).
How do I know which is the best Pareto front to choose in a
multi-dimensional space? There seem to be ways of choosing the
top-right Pareto front or the top-left or the bottom-right or
bottom-left. In multi-dimensional space, it'd be even more varied.
Libraries like Platypus, PyGMO, Pymoo etc. just define the problem using
problem = DTLZ2(), instantiate an algorithm algorithm =
NSGAII(problem) and run it algorithm.run(10000), where I assume
10000 is the number of generations. But since I'm using a legged
robot, I can't simply use run(10000). I need to assign motor values
to the robots, wait for the simulator to make the robots in the
population move and then calculate the objective function cost. How
can I achieve this?
Once the pareto optimal values are found, how is it used to create a
cost value that helps me determine the fittest robot in the
population?

I eventually used the keep-efficient function from this answer to calculate the Pareto front and used the k-means function to calculate the centroid of the front. This gave me the approximate knee-point of the front, which is usually the optimal solution. One of the calculations was to maximise the distance moved in x direction (dx) vs. minimising the energy consumed (e), so since the x axis needed positive maximization and the y axis needed minimization, I inverted the y axis, since min(f(y)) = - max(-f(y)). This helped get the pareto front toward the top right side of the graph and both the x and y axes were maximization objectives. The optimal point calculated was the robot that had the best fitness.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why do LLMs need massive distributed training across nodes -- if the models fit in one GPU while batch decreases the variance of gradients?
Why do large language models (LLMs) need massive distributed training across nodes -- if the models fit in one GPU and larger batch only decreases the variance of gradients?
tldr: assuming for models that don't need sharding across nodes, why do we need (massive) distributed training if the models (e.g. CLIP, Chinchilla, even really large GPTs e.g. CLIP fits in a V100 32GB) fit in one GPU and larger batch only decreases the variance of gradients (but not expose ore tokens or param updates)? A larger batch doesn't necessarily mean we train on "more data/tokens" -- or at least that doesn't seem to be wrt SGD like optimizers.

Intuitively, it feels that if we had a larger batch size then we have more tokens to learn about -- but knowing some theory of optimization and what SGD like algorithms actually do -- a larger batch size only actually decreases the variance of gradients. So to me it's not clear why massie distributed training is needed -- at all unless the model is so large that it has to be shared across nodes. In addition, even if the batch was "huge" -- we can only do a single gradient update.
I feel I must be missing something obvious hence the question given how pervasive massive distributed training is.
In addition some toy training curves with V100s & T5's show me there is very little if any benefit in additional GPUs

In addition, it seems from nonGPT we know small batch sizes are sufficient to train (reference https://github.com/karpathy/nanoGPT but I did ask Karpathy directly to confirm  https://github.com/karpathy/nanoGPT/issues/58).
I am missing something obvious, but I wanted to clear this up in my head since it seems to be a foundation thing in training foundation models.
Related to the previous, I've also been unsure about the role of the batch size in training LLMs compared to traditional deep learning. In traditional deep learning when we used epochs to train, a model the larger the batch size the quicker we could go through an epoch -- so the advice I received (e.g. approximate advice by Ruslan Salakhutdinov's at the Simon's institute for deep learning tutorials) was to make the batch size large. Intuitively, the larger the batch size the more data the model sees per iteration. But mathematically this only really improves the variance of the gradient -- which isn't immediately obvious is what we want (I've done experiments and seen papers where noisy gradients lead to better models).
It is clear too the that the larger the context size the better (for everything, but for the sake of this conv it's better for training) -- whenever possible.  But context size is totally different from batch size. So my question is, how does distributed training, especially at the node level help at all if batch size isn't really the helping factor (which might be a wrong assumption)? So the only role for distributed training I see is if the model is to large to fit in 1 node -- since I'm arguing there is no point to make the batch size too large (I'd guess 64-32 is fine due to the CLT).
What am I missing? Empirical answers are fine! Or any answers are fine!

Related:

cross quora: https://www.quora.com/unanswered/Why-do-large-language-models-LLMs-need-massive-distributed-training-across-nodes-if-the-models-fit-in-one-GPU-and-larger-batch-only-decreases-the-variance-of-gradients
cross reddit: https://www.reddit.com/r/learnmachinelearning/comments/113whxu/why_do_llms_need_massive_distributed_training/

I don’t think the problem lies in the gradient or related stuffs - the problem here is the hardware limitation of GPU VRAM.
Sure, CLIP can fit in a single GPU, but what GPU are we talking about? I have done some experiments with CLIP and I am pretty sure that CLIP ViT-B/16 with 224x224 images and batch size above 40, it is very easy to get out-of-memory on RTX 3090.
You can try it on Google Colab. Try with ViT-B/16 and input a random tensor with size 224x244x3, then gradually scale up the batch size to see which threshold it exceeds the VRAM of the given GPU.
So in short, it is more about "we cannot fit it in a single GPU" rather than "It is about training stability".

The large batch size is mostly to speed things up as you point out. You could in principle train these models with batch size 1 and just do gradient accumulation, but then training would take muuuuch longer.
Aside for tokens/seconds, empirically, it has been found that larger batch sizes lead to faster convergence (https://arxiv.org/abs/2212.14034)
The time and efficiency advantages provided by large batch sizes will often necessitate distributed training, as Minh points out, due to the VRAM limitations of individual GPUs. Good luck even just loading (no inference) anything larger than 15B on an A100 without optimization tricks.
It also doesn’t help that transformer’s self-attention mechanism has memory requirements that are quadratic with the input sequence length. When you’re training these models, you tend to pack several examples into a single input sequence such that the maximum length is reached, minimizing padding tokens and maximizing GPU usage (which would otherwise remain underutilized, which is a waste of time/money). The result of that is that you’re training with fully maxed-out sequence lengths, and hence maxing-out memory requirements, once again requiring distributed training typically.


text:
A deep learning algorithm to optimize the outcome
I'm am quite new to deep learning but I think I found just the right real-world situation to start using it. The problem is that I have only used such algorithms to predict outcomes.  For my new project, I need information to feed a machine with to optimize outcomes. Could someone explain briefly how I should proceed? I'm stuck.
Here's the situation:
I have a machine that takes planks of wood with different grades of wood available throughout its length and has to cut it into blocks provided in a cut list. This machine will always choose the highest score it can get from a given plank. The score is obtained by multiplying each block's area by its multiplicator. The algorithm I want to build has to give that machine a multiplicator for each block listed in a cut list. All of the physical output from this machine will be stocked on shelves by a robot until needed. The cutting machine is allowed to downgrade parts of a plank if it helps it reach a higher score.
The value has to act as an incentive for the machine to give me the block I need the most without downgrading too much wood.
OPTIMIZATION GOALS

Make sure each block is in stock by the time it is needed, but not too early without reason
Downgrade as little area of wood as possible (some species are very expensive)

INPUT NODES

Amount of time before this block is needed
Grade of wood for this block
Amount of this block needed
Block's area (Maybe?)

FEEDBACK PROVIDED TO THE ALGORITHM

Amount of time in advance that the block was ready (must be as low as possible)
Area of wood downgraded * number of grades skipped

EXPECTED RETURN DATA

A multiplicator that will give that block an optimal its priority relative to others

INFORMATION I DON'T HAVE BUT COULD GATHER

Mean ratio of each grade for each species of wood

What I've figured out so far is that I may need my feedback to be smashed in only one value in order to make it the output node. The problem is that I can't understand how to make this algorithm to determine a multiplicator. Am I wrong in trying to solve this through deep learning?

I took the liberty of editing the question for more clarity.  Wasn't sure what to do with: "Value per cm² that will give that block an optimal its priority relative to others"  Do you mean giving a block an optimal value in regard to its priority relative to other blocks?

@DukeZhou The machine's algorithm multiplies the value i'm trying to optimise by the total 2D area of the block (width x length) in order to create a "score" and cut each plank to get the maximum score. If I ass the block's area in the algorithm, I could then determine the score and then divide it by the area. All in all, the main focus is in making sure that the highest score matches the highest need so that the machine's goal is the same as ours.

@DukeZhou Sorry if it ain't as clear as it could, English is my second language.

Good info.  You should edit that one line I referenced to make it more clear, and check my edit to make sure I didn't misrepresent any part of your question.

No worries.  It's a really interesting application and I just want to make sure the question is as clear as possible so that you get the answer you are looking for!

@DukeZhou Here. I replaced it with the term multiplicator and gave some extra information about its usage.

Let us continue this discussion in chat.

This sounds very similar to the optimum cloth cutting problem (you can find an example on the last page here: https://www.andrew.cmu.edu/course/15-211/labs/theory2_sol.pdf). This problem can be solved directly with a deterministic algorithm, so why do you think that you need deep learning or any other learning algorithm for yours?

This sounds to me like a classical optimization problem. Have you already solved the problem, or do you want to give it a go using optimization modeling? We need to reformulate the problem and need more detailed data, but then it can be formulated as a mixed integer program and solved using optimization software like GAMS/CPLEX.
BR, Kourosh

Well @ginge, I may be wrong trying to use a learning algorithm for this task, but the reason is that it is not about cut optimizing on my side, another software does it for me. What I need to optimise is a score I give to that software in order to give this machine more or less room for optimisation depending on how quickly I need some parts and how much space I have left to store the parts that are cut earlier.

I feel like I need a learning algorithm because I have many factors to take into account that creates a very large amount of outcomes and I don't see how I could get the the optimal settings without something that detects somme patterns for me and test many variations.

Deep learning models for regression tasks are quite difficult to train so I would suggest not to start with them. Instead I would start with one of the approaches below and maybe try to use deep learning afterwards.
A classical approach to the problem could be to analyze your optimization software and this would probably lead you to some deterministic algorithm.
Different approaches can be to treat your optimization software as a black box - give it a wide variety of inputs, note down the variables you are interested in (execution time, cutting results, etc.) and try to fit some sort or regressor on it.
One option is to follow Kourosh's idea and formulate it as a classical optimization problem.
If you prefer to use machine learning tools than I'd suggest that you start with a simple model like linear regression just to verify that there is any signal in the data that you can use. Afterwards you can look at more powerful algorithms like xgboost, regression trees, etc.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
In a neural network given partial inputs and complete outputs, is it possible to predict remainig inputs
In example, if there is a simple feed-forward neural network with 3 input neurons, 3 hidden neurons, and one output neuron; is it possible to predict a the value of an input neuron given the values and weights for the other two inputs and the output.

Not in general; the output could easily ignore one (or more) of the inputs.

@antlersoft I was assuming that one would have access to the weights as well, but I'll clarify that in the question.

It's important to understand that though neural networks generalize to the whole input space, usually the meaningful input space, from which the training data is taken, is a manifold inside that space.
For example, the image classifier can take any image, including white noise, but it's usually trained on photos, which generally have some nice structure and statistical properties, e.g. smoothness of color, and is expected to work well on photos, not everywhere.
So your question can be reformulated like this: can we learn the data manifold given the neural network and some partial data?
If we ignore deliberately crafted small examples (like ones in which the output just equals to one of features) and consider real world, high-dimensional tasks, we see that the manifold is quite resistant to small perturbations. In high-dimensional spaces a change of one dimension doesn't drastically change the distance to the manifold, it's still very very close. Example: a change of a single pixel is very unlikely to affect the output, hence it would be impossible to predict it. The larger the space is, the less is influence of each individual dimension.
Hope this answers your question.
I'd like to mention here one very promising technique called GAN, which basically does this: it uses one neural network for classification and the other to learn the data manifold. And it works so good that it'd be fair to say that both these networks actually complement each other, not derive from one another.

One can do this to a certain extent but I don't think to the point you are thinking. If the weights are known, you can then find the relative importance of each input to the outputs. From there one could maybe predict or estimate the inputs but I do not think those values would be correct.
For more on this, I encourage you to check out this paper


text:
Can I do image classification with Multi Layers Perceptron (MLP)?
I'm seeking guidence here.
Can I use Multi Layers Perceptron (MLP), e.g regular flat neural networks, for image classification?
Will they perform better than Fisher Faces?
Is it difficult to do image classification with a MLP network?
It's on basic level like classifying objects and not detailed structures and patterns.
Important to me is that the MLP need to be trained with pictures that can have noise in background and different light shadows.

Hi @Daniel, welcome to Artificial Intelligence Stack Exchange, Please see the tour and look around to see how this site works. For your questions, could you please ask one question per post? it will give more focus on each post

I highly recommend for any image classification you look into convolutional neural networks. These should preform much much better than a standard MLP for image classification. Yes while you can do it with MLP, you probably shouldn't. The main issue comes from the input layer, feeding in each pixel as a giant 1D vector might give you too many parameters to train and an overall inferior model...

@PMaynard Thank you. Now I know that MLP is bad practice in image classification. Perhaps I should use Fisherfaces instead?

No use a convolutional neural network, Unless this is an assignment or task that requires it you should almost undoubtedly use a convolutional neural network. Generally for image classification I would say, CNN = Great, MLP = poor/okay, Fisher Faces = Bad.

Multi Layers Perceptron(MLP) can be used for image classification, but it has a lot of deficiency than Convolutional Neural network(CNN). But if you compare MLP and Fisher Faces , the better one is MLP, because Fisher Faces will be increasingly difficult if adding more individuals or classes. You can make a simple MLP model, because it just has 3 layers which are an input layer, hidden layer and output layer, here a source code that you can try:

https://towardsdatascience.com/multi-layer-neural-networks-with-sigmoid-function-deep-learning-for-rookies-2-bf464f09eb7f

if you make a model, it will be based on training data, I think if make data trained composed based on noise in background and different light shadows on your image I think it will have a better performance, but remember if you are using MLP for image classification it can just predict an image on one spot, for example:
"if you train a model with the object in the middle of an image, your model can not predict it when the image is moved to the different spot".
here is pdf to see Fisher Faces performance:

https://core.ac.uk/download/pdf/84801048.pdf

for some reason yes :) ,but i suggest u to use CNN for image Classification, it have better performance

I will convert MPL network to C code. That's why I wonder if MPL can be used for image classification. Else I will go with fisher faces.

Try Deeplearing2C on GitHub. I made it.

Depends, if the faces are centered and have the same background yes. You also need a lot of data.
If they are daily life images, then no. You will have very bad generalization.

let me try to answer your question. yes, you can use multilayer perceptron to image classification. Multilayer Perceptron is topology
the most common of ANN, where perceptrons are connected to form layers. An MLP has
input layer, at least one
hidden layer, and output layer.
Multilayer perceptron is one method
many used. one of them, regards research on classification
human skin based on its color, Khan
(Khan, Hanbury, Stöttinger, & Bais, 2012)
compare the nine methods for
classifications include BayesNet, J48, Multilayer Perceptron (MLP),
Naive Bayes, Random Forest, and SVM.
The results show that the Multilayer Perceptron (MLP)
produce the highest performance after
Random Forest and J48.


text:
What is the lowest possible loss for a language model?
Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence aei, with two occurrences preceding o and one preceding u, i.e., the dataset is:

Input
Output

aei
o

aei
u

aei
o

In this case, the ideal probability distribution for the model's logits for aei would be $\sim 0.66$ for o, $\sim 0.33$ for u, and zero for other letters. In other words, when the model is input with aei, the ideal softmax of the logits would be $\sim 0.66$ for o, $\sim 0.33$ for u, and zero for other letters.
Following this reasoning, the objective is to optimize the model's output for a given input to match the distribution of occurrences in the dataset.
If this reasoning is correct, then we have the following ideal loss (cross-entropy):
$$
L = \frac{- log\left(\frac{2}{3}\right) - log\left(\frac{1}{3}\right) - log\left(\frac{2}{3}\right)}{3} \approx 0.63
$$
Thus, $\sim 0.63$ is the smallest loss we can get with this dataset.
Is my reasoning correct?

Yes.. consider that neural networks are function approximators, and:

In mathematics, a function from a set X to a set Y assigns to each element of X exactly one element of Y

So, given the same X, you can only hope to have a specific Y, and if your dataset is ambiguous (such as in your case), then depending on the loss, you'll get some approximate answer (if derived from the MLE principle, probably the mean answer, but might change if you use MAP or regularization)
So yes, no NN will ever be able to give a 0 loss to such problem (at least formalized as such)


text:
What features should a dataset to predict monthly retail sales for a motorcycle spare parts shop have?
I am making an AI model to predict monthly retail sales of a motor cycle spare parts shop, for that to be possible I have to first create a dataset. The problem I am facing is what features should the dataset have?
I already did some research on some other datasets but still I want to know specifically what features should it have other than Date, Product Name, Quantity, Net amount, Gross amount..?

This is the prototypical problem in AI/ML known as feature-selection. If you do not have too many features, typically one would just use them all(with the exception of features that are known to to be correlated), in this case you would want to perform feature engineering wherever possible as well. One the other hand, if you have lots of features, and some are likely to be useless, you would use a feature-selection technique.
There are many many methods for doing this. One can simply use their intuitive grasp as their problem domain at the simplest level. However, there are also many algorithmic ways of performing it. Some examples of this are a low variance filter, or using a ensemble(classifier or regression depending on the problem) which can then be used to order features according to their derived importance(I personally like this).
A search of "feature selection methods in ML" will yield many potential ways to accomplish your goal.
As far as what to grab if you don't have many, as many as possible wrt domain knowledge.

I would think you would want the brands or model numbers each part fits, and also data on the number of each model or brand sold each year.  E.g. you need to stock more parts for a Honda than for an Indian.
Over time your system should be able to learn what demand patterns are for each part, but to "prime the pump" some sort of information on number of consumers for each part would be helpful.
You would also want to know what the time needed to restock is for each part.


text:
Does it make sense to train an autoencoder using data from different distributions?
Say I have 500 variables and I believe those variables can be shown in a 4-dimensional latent representation which I want to learn.
What I have for training is 100K samples, and those samples are coming mainly from 3 unbalanced groups: 1st group has 1K samples, 2nd group has 49K samples, and 3rd group has 50K samples.
Do you think I can learn a meaningful representation by training a (variational) autoencoder with this data? Is there a reason that requires all samples to come from the same distribution? If not, is there a reason that requires balanced classes?

Can you please create paragraphs in your question body?


text:
Training data for autoencoders
Two practical questions regarding the use of autoencoders. For the size of the NN, let's say I have 50 inputs, so layer sizes go like 50-25-12-6-2-6-12-25-50

how much data I generally need to train it and how does it depend on the size of net?
I'd like to weigh samples differently - e.g. there is a time component to the problem the last samples are most recent, so should weigh more. Is there an easy way to incorporate it?

Each network has a limit that it can learn. To check that limit, you need to try. Especially those kind of simple networks, it can be achieved so quickly. Idea is that, you need to give X data and check the accuracy and increase it to 2X data and check the accuracy. If there is no change in accuracy, it means that your network has reached its limit and you may consider making it deeper and wider.
By nature of the training, the last sample that you bring to training is the result that optimizer will try to converge. Hence, optimizers are already doing what you are searching for. However, if you want to create some weights for them, try considering RNNs. Their nature is suitable for that purpose.


text:
Coding CGAN paper model in Keras
I was coding a CGAN model using Keras along with the paper (https://arxiv.org/pdf/1411.1784.pdf) and I wanted to try and match the models to exactly what the paper says. I knew the models presented in the paper would be primitive but just wanted to replicate those and see. For example the generator model in the paper mentions this:

In the generator net, a noise prior z with dimensionality 100 was
  drawn from a uniform distribution within the unit hypercube. Both z
  and y are mapped to hidden layers with Rectified Linear Unit (ReLu)
  activation [4, 11], with layer sizes 200 and 1000 respectively, before
  both being mapped to second, combined hidden ReLu layer of
  dimensionality 1200. We then have a final sigmoid unit layer as our
  output for generating the 784-dimensional MNIST samples.

So for this I had the code like this:
def build_generator(self):

       model = Sequential()

       model.add(Dense(200, input_dim=self.latent_dim))
       model.add(Activation('relu'))
       model.add(BatchNormalization(momentum=0.8))

       model.add(Dense(1000))
       model.add(Activation('relu'))
       model.add(BatchNormalization(momentum=0.8))

       model.add(Dense(1200, input_dim=self.latent_dim))
       model.add(Activation('relu'))
       model.add(BatchNormalization(momentum=0.8))

       model.add(Dropout(0.5))

       model.add(Dense(np.prod(self.img_shape), activation='sigmoid'))
       model.add(Reshape(self.img_shape))

       model.summary()

       noise = Input(shape=(self.latent_dim,))
       label = Input(shape=(1,), dtype='int32')
       label_embedding = Flatten()(Embedding(self.num_classes, self.latent_dim)(label))

       model_input = multiply([noise, label_embedding])

       img = model(model_input)

       return Model([noise, label], img)

But still I think this not exactly what the paper means. What I understand from the paper is the noise and labels are first fed into two different layers and then combined to one layer.
Does this mean that there should be three separate models inside the generator? Or am I mistaken thinking that? Would like to hear any thoughts on this.


text:
Dense-Sparse-Dense CNN training
I want to implement DSD: Dense-Sparse-Dense training for deep neural networks by Han et al. In short, the paper suggest the following training scheme to improve the network accuracy:

Train as usual till convergence,
prune the network and train with sparsity constraint,
remove the sparsity constraint and let the pruned connections to recover.

My question is about step 2: train with sparsity constraint. The paper mentions training with a binary mask specifying the pruned weights to keep "untouched" so the sparsity constraint is satisfied, however that means implementing a dedicated layer that takes the binary mask as an additional blob and handles it accordingly.
I wonder a simpler approach will give the same result: what if after the pruning step I keep the location of the pruned weights and then use dense training, but after each iteration to zero the originally pruned weights?
The forward path is the same, taking zero weights for the pruned weights anyway.  But would it negatively affect the backward path - since the constraint isn't there, or is it equivalent to the formal training scheme?
Thanks.

Your approach will give a similar result.
The difference with using the constraint is that the constraint allows the network to essentially confirm that it doesn't need the connections before it reallocates them (i.e. if the weight is lessened by the constraint and it results in something not working, then the weight will start increasing).

Thanks for replying. I don't understand your last sentence (if the weight is lessened...) Could you please elaborate?

I think I might have just misunderstood. I assumed that while running the S step, you were planning to add the sparsity constraint to your re-weighting function to cause low values to be less likely to increase, otherwise setting the values to 0 still allows them to grow during the S step. In my last sentence, when I referred to "constraint" I meant that.


text:
How do you encode a chess move in a neural network?
In a neural network for chess (or checkers), the output is a piece or square on the board and an end position.
How would one encode this?
As far as I can see choosing a starting square is 8x8=64 outputs and an ending square is 8x8=64 outputs. So the total number of possible moves is 64x64 4096 outputs. Giving a probability for every possible move.
Is this correct? This seems like an awful lot of outputs!

The number is 4672 from Google.

https://arxiv.org/pdf/1712.01815.pdf

A move in chess may be described in two parts: selecting the piece to move, and then
selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73
stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8
positions identifies the square from which to “pick up” a piece.

4672 might sound a big number but it's nothing to what Google (and many other competitors) has been doing to deep learning for image analysis.

The paper explains the 4672 number: 8 * 8 * (56 + 8 + 9). Can someone help me understand why there are 9 planes for promotion, instead of 12? I understand there are 3 pawn moves: straight, diagonal left, diagonal right. But pawn can be promoted to 4 kinds: knight, rook, bishop, and queen. So that would have been 12. Why queen was not considered in the paper?


text:
Are all breakthroughs in AI and machine learning are due to increase in computational resources?
I have read that all the math responsible for modern day machine learning and AI was already in place in 1900s but we did not have computational resources to implement those algorithms. So, is that true? And if it is, in what areas of machine learning the researchers work? And are all the future breakthroughs will be dependent only on increment of computational resources?

Hey.we are living in the future;the future isn't just about what we can build or engineer basing on Computational Resources. we are making groundbreaking discoveries and achieving remarkable breakthroughs in the field of Artificial Intelligence.So here ML is in!. And we're doing it every week.  We'll cover the biggest ones by breaking down complex research papers and analyzing the implications of their content.Therefore,Grid Computing is playing it big and bigger!!..no more worries about resources anymore..and here comes quantum computing!

High-level answer: Increase in resources has been important in AI, and definitely was a factor with Deep Blue, but Machine Learning is a newer method that seems to produces more optimal results with less resources on problems of greater complexity.
Here is an article on AlphaGo's hardware: "Google reveals the mysterious custom hardware that powers AlphaGo"
Also an interesting analysis on Quora: "What hardware does AlphaGo run on? Is it customized hardware for best performance?"
Still pretty powerful systems, but I think the algorithms are as important as the computing resources, because all the hardware in the world won't help in the algorithms are poor.
Matthew Lai, creator of Giraffe Chess, said:

"In the ensuing two decades [since Deep Blue], both computer hardware and AI research advanced the state-of-art chess-playing computers to the point where even the best humans today have no realistic chance of defeating a modern chess engine running on a smartphone."Source: TechExplore

which suggests that hardware and software are both important parts of the overall equation.

most of the underlying math(like back-propergation)  was discovered a long time ago, and the advances in hardware have only recently made it possible to tackle some problems within  a reasonable amount of time but that is not the only reason. Some of the other things that also made deep learning pick up includes:

The availability of large,  labeled, quality data - deep learning generally requires a lots of data to  train successfully. This kind of data just wasn't there until recently.
Algorithmic improvements(I.e further improvements on the maths) - better weight initialization techniques, the discovery of the ReLu as a replacement for the sigmoid and tach activation which goes a long way towards solving the vanishing gradient problem,  the discovery of neutral network dropout which is found to reduce overfitting, etc.
The discovery of better neutral network architectures.

So, whist the computing power is certainly a big part of it, it certainly isn't the only factor.


text:
How to test if my implementation of back propagation neural Network is correct
I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have noticed during training test of my network:
Specification of the implementation:

A data set containing almost 100000 raw containing (3 variable as input, the sinus of the sum of those three variables as expected output).
The network does have 7 layers, all the layers use the sigmoid activation function

When I run the back propagation training process:

The minimum of costs of the error is found at the fourth iteration (The minimum cost of error is 140, is it normal? I was expecting much less than that)
After the fourth iteration the costs of the error start increasing (I don't know if it is normal or not?)

I would always output the average error, not just the average cost. For the average error of an approximation of sine 140 is way too high, obviously. I would also restrict the input to a certain range. It might be the case that the NN doesn't generalise to bigger inputs. Output your error for the training and the test set. And finally: Try a learning rate so small that almost nothing happens, then slowly increase. It might be that you are jumping around too much in the solution space.

As you progress, you might want to try Residual connections to help with vanishing gradients. Search ResNets. Andrew Ng's Deep Learning ResNet video should be on Youtube. Check it out!

Actually the implementation was correct,
The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.
When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.
Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function

if you change the activation function to relu or elu that would fix the issue

Yet another common solution is residual network, i.e., adding fixed weight connections skipping layers. I like it most as it works without modifying the structure / activation function.

I believe the best way to do this is using numerical gradient. To understand the concept, we need to look the definition of derivatives using limits:

It means that, when you don't know how to derive some formula (or you just don't want to), you can approximate it by computing the output for a small change in input, subtract from the original result (no change), and normalize by this change.
Example:
We know the derivative of f(x) = x^2 is f'(x) = 2x. But, let suppose we don't and we are using x = 3 and h = 0.001 (it tends to zero in fact):
f(3 + 0.001) = (3 + 0.001)^2 = 9,006 (approximately)
f(3) = 3^2 = 9
Thus,
(9,006 - 9) / 0.001 = 6
It's is exactly to f'(3) = 2*3 = 6.
In practice, if you want to know if your backpropagation is correct,

pass a single example (x1) through your network and computes the output (o1). Compute the gradient with respect to x1 (d1).
Then, add a small value to the input (h), pass through the network again, and computes the new output (o2).
Subtract o2 - o1 and divide by h. It must be close enough to d1.

That's it. It's called gradient checking. I hope it helps.

One thing that you should know is that the sigmoid function limits the output to a value between 0 and 1, which means that using a lot of hidden layers will lead quickly to a Vanishing Gradient.
Try to use relu activation function, it has the property to output all the information it gets from the previous layer.


text:
ReLu, Sum and Convolution Layers to Count Pixels of Certain Color
Below is an excerpt in an instructor's manual on ML that is explaining deep neural networks, using cat recognition (what else!) from images as example. On how DL performs this feat, the excerpt said that,

Assume that the first layer returns the number of pixels that are
  brown/black/blue/red, and the second layer finds the most common
  color, and the third layer returns “cat” if previous layer had
  supplied “brown”. [..] Mathematically, this model would be, for the
  first layer, [ sum(r = 255, g=255, b=255), ..., ..., sum(r=255, g=0,
  b=0)] -- this is just a set of appropriately positioned relu functions
  (okay, for r=234, we’d need two relu functions, so two layers, but you
  get the idea). The second layer would be a softmax layer. The third
  layer is simply an identity!

Now I worked with deep nets, but I am not sure how I can structure a DL to do this. ReLu is simply a max(0,x), so how would I filter out pixel vals for example 128,128,128 and sum them up? Wouldn't the convolution layer play a role here too? What would the layout of a simple deep net be that does what is described above?
Thanks,

I can judge from the excerpt only, but I don't see any mention of a convolutional layer there. This doesn't mean that convolution isn't suitable for this task (in fact, it's the best method for image classification so far), it's just seems the instructor proposes this particular network architecture, without convolutions.
Since this architecture is more illustrative and by no means a standard, I can't be sure I fully understand it. But it seems like the first layer outputs a vector of size 4, which contains the counts of four selected colors. The ReLu operation here is a fancy way of filtering extra colors, based on channel value (note that they are close to either 0 or 255 in all four colors). If you select different four colors, this filtering will have to be done differently too, so it's basically hardwired. The second layer is more straightforward.
In general, you shouldn't consider this a "true" deep network, but merely a puzzle to get familiar with ReLu operation.


text:
The ANN is based on cognitrons
I'm trying to understand how to build the ANN on cognitrons, so I have read theory for that topic and found the scheme:

As I got neurons are subdivided in two classes: the exciting and the inhibitory. I have written these classes:
Exciting neuron:
typedef float signal;
typedef std::vector<signal> sigvec;

class inhibitory_neuron
{
public:
    inhibitory_neuron() {}
    ~inhibitory_neuron() {}
    virtual signal call(const sigvec &e_inputs, const sigvec &i_inputs)
    {
        unused(i_inputs);
        signal sum = 0;

        for (auto i = std::begin(e_inputs); i != std::end(e_inputs); ++i) {
            sum += *i;
        }

        return sum;
    }
};

Here e_inputs are c-factors (sum of c is 1). But in the formula:

I can't get it... Whaz OUT_i there?
And below my Exciting neuron class is:
class exciting_neuron : inhibitory_neuron
{
protected:
    std::unique_ptr<sigvec> m_e_weights;
    std::unique_ptr<sigvec> m_i_weights;
public:
    exciting_neuron(sigvec &&e_weights, sigvec &&i_weights)
        : m_e_weights{std::make_unique<sigvec>(std::move(e_weights))},
          m_i_weights{std::make_unique<sigvec>(std::move(i_weights))}
    {

    }
    ~exciting_neuron() {}
    signal call(const sigvec& e_inputs, const sigvec& i_inputs) override
    {
        auto ew = *m_e_weights;
        auto iw = *m_i_weights;
        auto ei = e_inputs;
        auto ii = i_inputs;

        if (ew.size() != iw.size() || iw.size() != ii.size())
        {
            throw std::invalid_argument("Wrong input size.");
        }

        signal e_sum = 0, i_sum = 0;

        for (unsigned int j = 0; j < ew.size(); ++j)
        {
            e_sum += ew[j] * ei[j];
        }

        for (unsigned int j = 0; j < iw.size(); ++j)
        {
            i_sum += iw[j] * ii[j];
        }

        signal n = (1 + e_sum) / (1 + i_sum) - 1;
        return n >= 0 ? n : 0;
    }
};

Here e_inputs are a-factors (exciting) and i_inputs are b-factors (inhibitory).
But how should the ANN structure look? I mean I can't get where should I put an exciting neuron and an inhibitory... Has it some samples or rules of the structure, I couldn't find?

Welcome to AI!  Props for our first question on neocognitrons!


text:
What subjects was ChatGPT trained on the most? Science/history/movies/reddit posts/wikipedia/books/news?
What subjects was ChatGPT trained on the most quantatively?
It was trained on fiction and non-fiction books, wiki, and general web crawling.
A bit of detective work tells me that compared to physics, GPT-3 knows:
12 times more about the color blue,
9 times more about the USA
7 times more about hands
6 times more about sport,
5 times more about covid, purple and china
4 times more about hair, and america
2 times more about tea and coffee
...
It's as familiar with physics as it is with bread, sauce, beer, basketball, cheese, wine,

Specifically, I wanted to know what % of ChatGPT's knowledge is about science, physics, chemistry, fiction, non-fiction, religion, wine etc.
I found this ratio of subject matters on wiki. See also here.
Also for reddit by subject matter, it seams that less than 0.1% of commentators are on science topics.
So I looked on the web, from a general crawl I found this is the ratio of thing ChatGPT knows about (list).
I find that kindof weird and I wonder if there is a better way of finding what volume of words the web, wikipedia and contemporary book shops have, which would be reflected in GPT3.

A LIST OF INTERNET TOPICS BY HITS:  physics 1.9 billion  electron 0.6
billion chemistry 4 bn biology 3  voltage .6 android 13 apples 8 sauce
2 toothpaste .2 president 3.3 cars 13 bn chevrolet .5 jeep .7
volkswagen .6 sugar 4 computer 9 blender .5 microphone 1 b television
3.7 hair 9bn cats 5bn dogs 5 horses 2.5 bn w0od 3.8 fantasy 2.1 harry potter 0.6 fiction 2.7 einstein .3 security 10 bn purple 10 pink 12
violet 1.3
4.2bn sport 12bn music 9bn football 4 basketball 1.6 tea 4.2 coffee 4.2 bread 2bn glass 5.7 soil 1.2 aluminium .5 rap 1.4 blue 25bn country 10bn beer 2 russia 2.7 france 7.5 italy 5.5 USA 18bn america
9.2 england 6.4 africa 7bn japan 8.7 china 10 salad 1.5 keyboard 2.5 forum 6.5 printer 1.2 hands 15bn feet 4.7 box 1bn jeans 4bn depression
4bn religion 3.7 islam 2.5 piano 1.6 guitar 1.5 chair 2.5 atlantic 1.1
robot 1.9 maps 11bn clothes 6.6 roof 1.9 spices 1.6 shoes 3.7 camera
7.7 tomato 1.6 solar 1.8 wheels 2.6 wine 2.8 cheese 2.1 market 6.9 covid 11bn

Did you asked Chatgpt about your question ?

Yes I did, it wasn't trained on text with specific information about it's training data, so the only way to ask it is through some tricks... I think there are also some censored topics, because I had a response like "ChatGPT was not able to load the module that you requested" the session stopped loading new input fields.... added the maths ratio of it's traning sources and their content of i.e. science (Reddit is 0.01% science) so I think it way less than 0.5% sciences.


text:
How can policy gradients be applied in the case of multiple continuous actions?
Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.
When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:
$$L(\theta) = \log(P(a_1)) A,$$
where $A$ is the advantage of rewards, $P(a_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in the Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py.
The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).
Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment?
Is following approach correct for policy gradient loss function?
$$L(\theta) = (\log P(a_1) + \log P(a_2) )*A$$

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.
There are 2 solutions to this problem:
The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and here is a paper which describes this approach with 3 actor-critic agents working together to move a robotic arm.
The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.
This is a more general answer but should help you and others on their related problems.

Jaden thanks for great answer.

I tried multi-agent architecture, but it is not very efficient. Takes much longer to converge.
Now multivariate distribution seems obvious to me too, thank you.

Depending on the application and architecture (if it is a deep net), you can have the agents share low level features and then have them branch off into their own value functions. Additionally, having 1 critic and multiple actors is also a way to increase the architecture.

At the moment I would like to apply your suggestions to TRPO (just policy gradient methods), not actor-critic. I am not very confident in gradient transfer from critic to actor - in many implementations I have seen it looks like that it should not work even though it does converge.

Sorry for this noob question: How is this applied in actor-critic methods (where the actor can perform multiple simultaneous continuous actions), where the actor has the policy function and gets trained by policy gradient method? @JadenTravnik Can you please explain that in the answer under a new heading?


text:
Is there an alternative to the use of target network?
In the context of Deep Q Network, a target network is usually utilized. The target network is a slow changing network with a changing rate as its hyperparameter. This includes both replacement update every $N$ iterations and slowly update every iteration.
Since the rate is hard to fine tune manually, is there an alternative technique that can eliminate the use of target network or at least makes it less susceptible to the changing rate?

I have done some research and would like to share.
Generally to eliminate the use of target network one needs to show that training would be stable under off-policy semi-gradient.
There are two approaches that might work:

Experience reweighting
Constrained optimization

Experience reweighting
Probably the simplest idea is to use importance sampling ratio (Precup 2001) to multiply each sample. This would correct the off-policy sample distribution to be on-policy distribution. It has been shown (Sutton 2016) that on-policy samples lead to stability for semi-gradient. However, this line of work that corrects for sample distribution has high variance and does not work well in practice.
Another line of work aimed to partially correct the distribution only to the extent that would be provably stable is called Emphatic TD (Sutton 2016). The distribution is still mostly off-policy but is proved to be stable under linear function approximation.
Constrained optimization
The general wisdom comes from the fact that updating a value would also alter the target value since they share the same set of parameters. This problem is called over-generalization. To reduce this, Durugkar suggests that constraining the target Q-value after parameter update to be steady helps reduce divergence. Achiam 2019 provides a good overview of the problem, and suggests a way to make sure that the update is non-expansion (which should prevent divergence).
Sidenote
Some works have shown convergence under off-policy samples but only in tabular case. It takes much more to show that it is stable in function approximation (even a linear one).
Works like:

Q($\lambda$) (Harutyunyan 2016), which augments the value function with a correction term
Retrace($\lambda$) (Munos 2016), which extends the former by truncating the importance sampling ratio

are shown to work only in tabular case.

Never tried it, but there couple of approaches which may or may not help:
Distributional DQN ( not C51, other one, cant find ref right now typing from phone) with several output heads, chosen randomly
Multiple agents learning randomly on each other with some regularizer to pervent collapse to same net.
Both approach essentially try to"hide" or smear target.


text:
Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?
Prediction's goal is to get an estimate of a performance of a policy given a specific state.
Control's goal is to improve the policy wrt. the prediction.
The alternation between the two is the basis of reinforcement learning algorithms.
In the paper “Safe and Efficient Off-Policy Reinforcement Learning.” (Munos, 2016), the section 3.1) "Policy evaluation" assumes that the target policy is fixed, while the section 3.2) "Control" extends to where the target policy is a sequence of policies improved by a sequence of increasingly greedy operations.
This suggests that even a proof of convergence is established with a fixed target policy, one cannot immediately imply that of the case where the target policy is a sequence of improving policies.
I wonder why it is the case. If an algorithm converges under a fixed target policy assumption, any policy during the chain of improvement should have no problem with this algorithm as well. With the merit of policy improvement, each policy in sequence is increasingly better hence converging to an optimal policy.
This should be obvious from the policy improvement perspective and should require no further proof at all?

I don't understand what you are asking. What do you mean by "This suggests that even a proof of convergence is established with a fixed target policy". Where does the suggestion come from? Also, you should rephrase this "one cannot immediately imply that of the case where the target policy is a sequence of improving policies.", which is not very clear.

@nbro Actually, my simpler question is what is the need for section 3.2) "control" at all?

@nbro About the suggestion, I imply from the existence of section 3.2. If the proof provided in the section 3.1 (fixed policy case) is suffice, there should be no section 3.2.

It is a mathematical problem. Usually, we use contraction mapping theorem for the proof of convergence. You should apply the Banach fixed point theorem for Bellman's functions.


text:
What makes learned feature detectors specialize in CNN?
It has been shown that it is possible to use unsupervised learning techniques to produce good feature detectors in CNNs. I can't understand what drives specialization of those feature detectors. In publications https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (page 6) they show a very reasonable set of edge and blob detectors with little or no overlap. It goes against intuition - in absence of incentive to specialize, one would expect at least some duplication of learned kernels.

This seems curious to me as well. Could you edit some links to those sources you are mentioning into your question? It would help to look up those papers to give you a better answer.

Added a link to AlexNet paper. Seems like the fact that feature detectors differ is mostly due to random initialization of weights.

You are thinking along the correct line when you consider intuitivity when examining the strange self-specialization aspect of the layers in any deep network that has been tuned, through toil, mental machinations, and late nights, to work.
Do AI experimenters manipulated parameters, topology, and algorithmic hierarchies to force circuit evolution that produces data flows through specialized functions like this one.

Noisy, normalized, cubes (horizontal, vertical, depth) of integers straight from video hardware or the video channel of a multimedia file
Indications of edges
Indications of corners, ends, and bends
Indications of relative angles (in the case of size independent recognition)
Indications of shapes
Indications of 2D topology
Indications of object forms
Indication of scene
Indication of action (when a hypercube is used and frame is another dimension)

Of course, one could design a solution and then tune layers, one by one, to follow the design, but it is not necessary because of the concept of entropy removal, the elimination of information that is redundant or irrelevant.
The principle in information science is this, in general: As redundancy is removed from the information, the number of bits representing the information decreases and the level of abstraction increases.
In minimalist art, a single word an identical twin might say to the other twin as the train rolls away, or a good Pictionary player, given a small window of information passing opportunity, adaptations can be made to pass large amounts of information base on previously agreed upon conventions.
When you think of it this way, it makes sense that by narrowing the bit width of the aggregate data representation from layer to layer that only certain parametric optimizations will provide an end-to-end indication of convergence, given a goal such as fully categorizing objects based on a small number of features.
When Leibniz envisioned a world tied together by mathematical certainty, he was overly optimistic, however he was correct in at least one important respect, mathematics had a reality all its own.  The success of Newtonian mechanics is one early example of a huge reality that has become apparent and has been so widely applicable, deduced from a thought experiment involving a cannon ball and the moon.
There are few demonstrations of discoveries that originally appear to be wild inventions than the amazingly accurate prediction of the previously anomalous orbit of Mercury.  But Einstein was not surprised by his success.  He had deduced general relativity from the challenge of Ernst Mach and the undeniable results from light and gravity experiments.  He discovered how it must be before it was found.
In the same sense, the above sequence of events can be proven mathematically to be the efficient strategy for vision.  The sequence of processes to transform incoming visual signals to collision avoidance control was there in mathematics before there were visual receptors on microbes.  That is why the narrowing of the visual pathways of a Pterodactyl have similarities to those of a shark shark even though their common ancestor may not have been able to see.
The incentive to specialize is the incentive to converge on the objective even after redundancy is stripped by force by limiting the information channel aperture.  Even if the aperture is in multiple dimensions, the same principle applies.


text:
My DQN is stuck and can't see where the problem is
I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it couldn't finish a full episode. I tried two different learning rates (0.0001 and 0.00125) and seems to work better with 0.0001, but the median score never raises above 200.
I'm using a double DQN.
Here is my code and some photos of the graphs I'm getting each session.
Between sessions I'm saving the network weights; I'm updating the target network every 1000 steps.
I can't see if I'm doing something wrong, so any help would be appreciated. I'm using the same CNN construction as the DQN paper.
Here's the action selection function; it uses a batch  of 4  80x80 processed experiences in grayscale to select the action (s_batch means for state batch):
    def action_selection(self, s_batch):
        action_values = self.parallel_model.predict(s_batch)
        best_action = np.argmax(action_values)
        best_action_value = action_values[0, best_action]
        random_value = np.random.random()

        if random_value < AI.epsilon:
            best_action = np.random.randint(0, AI.action_size)
        return best_action, best_action_value

Here is my training function. It uses the past experiences as training; I tried to implement that if it lose any life, it wouldn't get any extra rewards, so in theory, the agent would try to not die:
    def training(self, replay_s_batch, replay_ns_batch):
        Q_values = []
        batch_size = len(AI.replay_s_batch)
        Q_values = np.zeros((batch_size, AI.action_size))

        for m in range(batch_size):

            Q_values[m] = self.parallel_model.predict(AI.replay_s_batch[m].reshape(AI.batch_shape))
            new_Q = self.parallel_target_model.predict(AI.replay_ns_batch[m].reshape(AI.batch_shape))
            Q_values[m, [item[0] for item in AI.replay_a_batch][m]] = AI.replay_r_batch[m]

            if np.all(AI.replay_d_batch[m] == True):
                Q_values[m, [item[0] for item in AI.replay_a_batch][m]] = AI.gamma * np.max(new_Q)

        if lives == 0:
            loss = self.parallel_model.fit(np.asarray(AI.replay_s_batch).reshape(batch_size,80,80,4), Q_values, batch_size=batch_size, verbose=0)

        if AI.epsilon > AI.final_epsilon:
            AI.epsilon -= (AI.initial_epsilon-AI.final_epsilon)/AI.epsilon_decay

replay_s_batch it's a batch of (batch_size) experience replay states (packs of 4 experiences), and replay_ns_batch it's full of 4 next states. The batch size is 32.
And here are some results, after training:

In blue, the loss (I think it's correct; it's near-zero). Red dots are the different match scores (as you can see, it does sometimes really good matches). In green, the median (near 190 in this training, with learning rate = 0.0001)

Here is the last training, with lr = 0.00125; the results are worse (it's median it's about 160). Anyway the line it's almost straight, I don't see any variation in any case.
So anyone can point me to the right direction? I tried a similar approach with pendulum and it worked properly. I know that with Atari games it takes more time but a week or so I think it's enough, and it seems to be stuck.
In case someone need to see another part of my code just tell me.
Edit: With the suggestions provided, I modified the action_selection function. Here it is:
def action_selection(self, s_batch):
    if np.random.rand() < AI.epsilon:
        best_action = env.action_space.sample()
    else:
        action_values = self.parallel_model.predict(s_batch)
        best_action = np.argmax(action_values[0])
    return best_action

To clarify my last edit: with action_values you get the q values; with best_action you get the action which corresponds to the max q value. Should I return that or just the max q value?

just an advice to make your code a bit faster, in action_selection method you are making a pass through ANN and calculating max action, and only after taking into account epsilon for random action choice. During training especially in early episode all that calculation will be done for nothing because you will end up taking a random action anyways. It's better to consider epsilon-greedy action choice first and only if the action isn't random do ANN pass. Also in that method you seem to be returning best action value regardless if action is random or not. Not sure if that changes anything

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were consecutives.
After redoing my experience selection method, my DQN is actually working and has reached about +200 points after 400000 experiences (about 500 episodes; only 2-3 hours or training!). Before I couldn't reach that score after days of training.
I'll let it train to see if there are something I can improve. Thanks to everyone who tried to help me! I let this answer here just in case someone has the same problem as me.


text:
Representation of state space, action space and reward system for Reinforcement Learning problem
I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without crashing into obstacles
I have the following environment:

where the environment is a  matrix. In this the obstacles are represented by 0's and the free spaces are represented with 1s. The position of the agent is given by a label such as 0.8 in the matrix.
The initial internal representation of the environment of the agent will look something like this with the agent position in it .

Every time it explores the environment it keeps updating its own map:

The single state representation is just the matrix containing-

0 for obstacles
1 for unexplored regions
0.8 for position of the agent
0.5 for the places it has visited once
0.2 for the places it has visited more than once

I want the agent to not hit the obstacles and to go around them.
The agent should also not be stuck in one position and try to finish the exploration as quickly as possible.
This is what I plan to do:
In order to prevent the agent from getting stuck in a single place, I want to punish the agent if it visits a single place multiple times. I want to mark the place the agent has visited once as 0.5 and if it has visited it more than once that place will be labelled 0.2
The reason I am marking a place it has visited only once as 0.5 is because if there is a scenario where in the environment there is only one way to go into a region and one way to come out of that region, I don't want to punish this harshly.
Given this problem, I am thinking of using the following reward system-

+1 for every time it takes an action that leads to an unexplored region
-1 for when it takes an action that crashes into an obstacle
0 if it visits the place twice(i.e 0.5 scenario)
-0.75  is it visits a place more than twice

The action space is just-

up
down
left
right

Am i right in approaching the problem this way? Is Reinforcement Learning the solution for this problem?
Is my representation of the state , action, reward system correct?
I am thinking that DQN is not the right way to go because the definition of a terminal state is hard in this problem, what method should I use to solve this problem?


text:
Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?
I've mostly seen (e.g. in  The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-character using a 1-hot encoding.
For example, the text "hello" would be represented like
{h: 1, e: 0, l: 0, o: 0}
{h: 0, e: 1, l: 0, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 0, o: 1}

I was wondering if one could just as well use the ASCII encoding of the text and feed the bits in one by one. So the input "hello" would be input like
0110100001100101011011000110110001101111

Would the RNN have a disproportionately harder time having to figure out how the arbitrary and complex 8-bit ASCII encoding should be used? Or would the ASCII encoding lead to about the same performance as the nicer 1-hot encoding?

My understanding is that the ASCII encoding would not get the best performance or results from the RNN because the ASCII codes for each character are not meaningful; they are arbitrary. If the number of each ASCII code represented something meaningful about the letter, it would work better. But they don't.
The same principles apply as when deciding how to encode any categorical data. If your categories are ordinal (eg. 'First', 'Second' .. or 'Age group 18-24', Age group 25-35' .. or even 'Social Class E', 'Social Class D' ..), then assigning a single numerical value to each class might work well. But in categorical data where there is no meaningful order, one hot encoding will work better.
This is an example of the principle of giving neural networks the most expressive data that we can. In the case of non-ordinal, arbitrary categories, one-hot is more expressive to the next layer of neurons (will stimulate them more distinctly) than using a numerical encoding.

The request for input encoding is that mathematical difference between encoded inputs (numerical subtraction in absolute value of vector norm, etc) must be proportional to logical (real world) dis-similitude between the objects they represent.
Lets say by example the 3 first uppercase letters. In one-hot encoding they will be:
'A' : (1,0,0)
'B' : (0,1,0)
'C' : (0,0,1)

Note that the distance between any two pairs is 0 if the letters are the same and 2 if they differ.
Compare with ASCII in binary as encoding:
'A' : 0100 0001
'B' : 0100 0010
'C' : 0100 0011

Note distance between 'A' and 'B' is 2, but distance between 'A' and 'C' is 1. Thus, we are saying that an 'A' is more similar to a 'C' than a 'B'. Something artificial that will cause network errors.
Equivalent problem appears when using ASCII as integer:
'A' : 65
'B' : 66
'C' : 67

we are saying that distance between 'A' and 'C' is 2, twice the distance between 'A' and 'B'. Again, something that will disturb network performance.


text:
What techniques would allow a model to be trained to produce subjectively pleasant 'artistic' output?
I've only just started looking into Machine Learning, and so far most of the examples I've seen involve starting with a training set of observations, each representing a value for a number of 'features', and using that to train a model - then evaluating the trained model against a second test set of data to allow refinements to the model. The basic operation the model is asked to do is to predict the value of one feature given others.
When talking about art (by which I would include music, poetry, etc), we can no longer assert that things are correct - but perhaps we could isolate a number of subjectively-judged features, such as whether a work is 'mysterious' or 'joyful' to a particular human judge, and then train the model to our judge's taste. However, The problem here would seem to be that having a human being add judgement features to each of our data points sounds like a very slow task, and it might often be impractical to get to the number of 'assessed works' that would allow the model to be trained.
How could the process of training a model to produce art of a certain 'taste' be done in a practicably short space of time?

You will need a large training set to be able to train your machine to produce subjectively pleasant 'artistic' output.
If you can't produce your own dataset with real people, you can try the following:

search for existing datasets on the internet
produce a new dataset using a spider bot that perform mining of artistic output on websites that have voting features
use crowdsourcing


text:
Using a MLP to predict a 12x12 matrix
So, i need to use an MLP to predict a 12x12 matrix composed of floating points.
The matrices are as this one that follows:

Most matrices have this "pattern".
As input, i have 7 floating points, such as these:
"2.0", "0.23", "239.10","3.5", "12,0", "10.6", "0.62".
To simplify the output matrix, i've converted it in a array with the 144 elements of the matrix.
So far i'm using MLPRegressor from Scikit.The problem is, there's absolutely no pattern in the predicted results.The predicted results include negative numbers, numbers really big and no "pattern" for the indexes whatsoever.
Is there a way to adjust these things in the model or the problem is on my dataset?
Thank you very much!
Update:
I wasn't very clear about the problem, so i'll try to explain it better.
About the problem: Predict water distribution in irrigation systems. I already have a program to simulate this, and works with rather good precision. The challenge now was to make an AI model to simulate this, instead of the old methods.
About the inputs: Inputs are one float representing the pressure of the sprinkler used to distribute water, two floats representing the speed and angle of the wind at the time data was collected. The 4 remaining are numbers representing some configuration of the sprinker (these numbers change very little between instances, but they do have impact in the result, which is why i decided to mantain them in the dataset).
About the output: Output is an array of 144 elements, representing the 12*12 matrix. Each element of the matrix contain a number representing the amount of water that was collected in that point. These collectors where evenly spread around the sprinkler. So the position of the matrix matter a lot, since in most cases the first and last lines and columns will have 0 or a close to 0 (but positive) number - this may vary depending especially on the speed and angle of the wind, but also on the sprinkler.
About the dataset: I have available 75 instances. They are all stored in a CSV file, where the 7 inputs and the 144 outputs are, each in one line.

I think that MLP is not a suitable solution in your case and autoencoders are more suitable for your task. However, no guarantees that even AE with the best architecture will work, since it’s not a fact, that your input data contains enough information for such reconstruction. However, if you don’t insist on MPL and can consider using AE instead, I can give you an answer with suggestions how doing it. If you can share your data I even can check it and give you more precise answer . You can find the simple CAE sample here - https://stackoverflow.com/questions/46921246/issue-with-simple-cae

Just updated op with more details. Will do some reading about AE. Thank you!

I think that 75 examples is not enough for good accuracy, therefore it probably can be up to 80-90% for a such small dataset. However, could you share your dataset, so I will be able to test model with it?

The short answer to your question is: you probably do not fully know your data. remember that ML is no magic wand. It needs your understanding of the data and the behavior of it. Although it is approved that neural networks with at most two hidden layers can approximate any model with an acceptable degree precision, setting up the structure of the neural network is still your task; also note that the cleaner your data is, the more acceptable the degree of precision will be for a structured NN.
the following more detailed explanation might also help. understanding your data that I pointed above, means:

the design of your machine learning pipeline is the first and most important thing to take care of. Are you sure you have handled training set and test set appropriately? are the ratios standard? have you performed random sampling well?
the size of the dataset is also a matter of consideration. remember a ratio of n/p>10 (n: the number of instances, p: the number of features) is necessary. So if you have 7 features, you should have more than around 100 instances. check how big is your dataset
the distribution of your dataset is also important. dealing with badly distributed data should be considered if you have an issue here. use plots and histograms to understand the distribution of your dataset over feature axes. There is a high risk of overfitting if your data is distributed badly
also take care of your output distribution. with a bad distribution over your output data (focused outputs in a zone) you loose the power of generalization of your model
check the relations between different features. check if there are any pairs of dependent variables. you can use scatter plot or calculating pairwise covariances. PCA or more complicated methods (like auto-encoders) will be of help if there is any issue here
one possibility is maybe your data doesn't carry any latent information at all. maybe the underlying pattern is so stochastic and nonlinear that it seems to be random. in other words: you can not predict what kind of food your grandma has cooked by looking at results of world cup games, no matter how big your dataset is.

Now about your output matrix:
reshaping your matrix to an array of 144 might be the best or the worst thing to do. it depends on the characteristics of your data. maybe the position of each element in the matrix is also important in predicting the value. so you might want to include the i,j of each element in the input array too. maybe each output also takes effect from neighbors. there are some other options to consider:

you can use 144 different MLP networks, one for each element (a little bit like what you have already done, when reshaping your matrix. since your library constructs a single MLP network that has 144 neurons in the output layer. so it seems like 144 networks that only vary in the output layer, and note that the weight correction step is different too, since the training error for all of the outputs propagate back in the layers simultaneously)
you can create 144 instances out of each instance, each of which has 7+2 inputs and 1 output. the 2 more inputs are the position of the output value (i,j). it might be a poor decision to do that but this method has helped me a couple of times, so test it
Do the second step, but include the neighbor values too. so you'd have for example 7+2+4 inputs (7 initial + 2 i,j + 4 neighbor values)
also consider using RNNs (recurrent neural networks). if you consider the matrix to be a sequence of values which has internal dependency plus the external dependency to those 7 features, then you can use a RNN network like LSTM.

feel free to update your post with more specific experiments and issues.

Updated op with more details about the problem. But i'll do some more tests considering your points, they helped a lot and made me think better about the problem. Thank you very much!


text:
Finding an optimum back propagation algorithm
I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.
So this is what the code is about:
I will have a list of organisms with their features stated in numerical values. I want to write a code that identifies that whether the organism is a cat or a fish or neither based on their characteristics. (For example an organism with a high fur value and 4 legs is more probable to be a cat.)
My idea for the neural network is to have five input nodes(for the five characteristics) and 2 output nodes (one for how cat it is and one for how fish it is). The input nodes are multiplied by a weight value and then all summed together to produce one of the output nodes. This repeats itself for the other output. How much the system got wrong is just the difference between the value of the output nodes and how cat/fish the being actually is.
But how can I use this information to correct the weights of the input nodes? Since the weights are randomly generated they can be in the "wrong" directions to begin with. For example if the subject is a cat then we should be expecting high fur and leg values. But what if the weight for the leg value is negative while the weight for the fur value is positive? Adding or multiplying the weight by the error won't bring us any closer to accurately determining the being. Is my neural network flawed to begin with? Or is there a rule of thumb in choosing back propagation algorithms?
Thanks.

When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a calculus based method which tell you how far you need to adjust the weights in order to get to an optimum value. However, when you have neural networks with many different weights you need to be able to decide which ones to update in order to achieve a more optimal network. This is where stochastic gradient descent comes in. This algorithm is able to take many different gradients and traverse through many dimensional space into a lower error rate. Also, from your diagram of your neural network, I do not see any hidden neurons, which are neurons between the input and output layers which do most of the actual work. While you might be able to train your neural network optimally without hidden neurons, it might improve your accuracy depending upon your dataset. Finally, implementing a back propagation algorithm is a lot of work which is why many people use pre built libraries. Unless you are doing research, I recommend you do the same. I think for a problem like this, pybrain would be perfect, and if you want to use deep neural networks, tensorflow would be good for that.

Thanks a lot! I didnt include hidden neurons because I didn't understand them well. From what I know so far is that they do some math function to the input, which for me is done on the lines (like a function diagram). I wish to understand the fundamentals of neural networks before I approach a library too. The only machine learning code that I have up and running is one that predicts the sum of two numbers.

Update : my code somehow works on multiplication, division and subtraction too, and I have no idea why but am very very happy about it.

See Jian Shin, a hidden layer is a set of neurons just like the input neurons and the output neurons. The hidden layer is the part you actually train in machine learning. All neurons represent a math equation, usually it takes in all of the ouputs from the previous layer, multiplies them by a weight matrix, adds a bias matrix, sums them all up, and applies an activation function used to frame the data for the output.


text:
Sigmoid output layer and Cross-Entropy cost function
I use Sigmoid activation function for neurons at output layer of my Multi-Layer Perceptron also, I use cross-entropy cost function. As I know when activation functions like Tanh is used in output layer it's necessary to divide outputs of output layer neurons by sum of them like what is done for softmax, is such thing necessary for sigmoid activation function?
If it's necessary to normalize outputs of neurons, does it affect derivations?


text:
Can a purely policy convolution neural network based game learn to play better than its opponents?
This question has come from my experiment of building a cnn based tic-tac-toe game that I'm using as a beginner machine learning project. The game works purely on policy networks, more specifically -

During training, at the end of each game, it trains itself on the moves the winner/drawer made for each board position. That is, its training data consists of board positions and the moves made by the winning player on each position.
While playing, it predicts its own moves solely based on that training (that is, it predicts what move would a winning player make with the current board). It doesn't use any type of search or value networks.

I'm seeing that if I train it against a player that predicts the perfect move (using a recursive search) every time, the AI gets good at drawing about 50% games. But if I train it against a player that makes random moves, it doesn't get better at all.
Wouldn't one expect it to learn well (even if slower) regardless of the level of its opponent? Since each game ends in a draw or win for one player, shouldn't it be able to extract features for the winning/drawing strategies even when learning from random players? Or does this behavior mean that the model is not optimal?

Does your current CNN utilize "monte carlo"?

No, no search at all. It's purely policy based - deciding its next based on what winners of its training games have made in similar situations.

I think a problem you may be facing with a random player is that a your game consists of many moves, and a winning player does not make only winning moves. In your example, your moves are random, and the fact that your random player wins against another (random, initially) CNN player is, well, random, too. So I find it intuitive to say that there is not information to gain from observing a random player, and no way to get "better" than it. This would be different in a game with fewer moves, where a good move is more closely related with winning, such as in tic-tac (2x2 tic-tac-toe).

(I am aware that you cannot lose a tic-tac if you make the first move, but you can draw - so by observing a random player, your policy-based CNN should at least be able to learn how to not draw when making the first move.)

Ultimately, in tic-tac-toe and more complex games, I would expect that you would need some way to measure the goodness of an individual position, and by extension, measure the goodness of a move to learn only from good moves.

(I'd leave this in the comments but sadly I can't.)
Here's a great paper (leveraging formal game theory) in which "players" make strategic choices based on each event ("move") and an algorithm calculates the best strategy for all players based on each player's abilities throughout the multi-stage "game". So instead of learning from winning and predicting what the next perfect move the player thinks (based off of learning) will lead to a "win", the player calculates an optimal strategy and moves to win the game itself over multiple moves.
https://users.ece.cmu.edu/~youzhib/paper/bao2017csf.pdf
Note: This paper deals with cyber war games so is heavily tilted towards that domain, but if you take away the game theoretics and concept of strategy over a multi-stage game like ti-tac-toe, you should be able to improve your overall outcomes. That being said, how the model would accommodate a random "player" with random moves would be interesting to see.


text:
Is it possible to create a 'logic system' equivalent to humans?
I've heard of AI that can solve math problems. Is it possible to create a 'logic system' equivalent to humans that can solve mathematics in the so called 'beautiful' manner?  Can AI find beauty in mathematics and solve problems other than using brute force? Can you please provide with examples where work on this is being done?

There is of course a vast amount of work in the area of Automatic Theorem Proving, but most of it is simply concerned with proof, rather than human notions of beauty, elegance, parsimony etc.
There has however been some work in this general area over the years:

Douglas Lenat's famous AM ('Amateur Mathematician').
Douglas Hofstadter's NUMBO program for number sequence extrapolation.
A range of publications by Simon Colton
Shalosh B. Ekhad, the automated proof assistant for Artificial Combinatorics created by Doron Zeilberg and credited by him as a co-author on numerous papers.