text:
How to draw backup diagram when reward is in expectation but next state is iterated?
I am working through Sutton and Barto's RL book. So far in the text, when backup diagrams are drawn, the reward and next state are iterated together (i.e. the equations always have $\sum_{s',r}$), because the text uses the four-place function $p(s',r|s,a)$. Starting from a solid circle (state-action pair), each edge has a reward labeled along the edge and the next state labeled on the open circle. (See page 59 for an example diagram, or see Figure 3.4 here.)
However, exercise 3.29 asks to rewrite the Bellman equations in terms of $p(s'|s,a)$ and $r(s,a)$. This means that the reward is an expected value (i.e. we don't want to iterate over rewards like $\sum_r \cdots (r + \cdots)$), whereas the next states should be iterated (i.e. we want something like $\sum_{s'} p(s'|s,a) (\cdots)$).
I think writing the Bellman equations themselves isn't too difficult; my current guess is that they look like this: $$v_\pi(s) = \sum_a \pi(a|s) \left(r(s,a) + \gamma \sum_{s'} p(s'|s,a) v_\pi(s')\right)$$
$$q_\pi(s,a) = r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'} \pi(a'|s') q_\pi(s',a')$$
My problem instead is that I want to be able to draw the backup diagrams corresponding to these equations. Given the "vocabulary" for backup diagrams given in the book (e.g. solid circle = state-action pair, open circle = state, rewards along the edge, probabilities below nodes, maxing over denoted by an arc), I don't know how to represent the fact that the reward and next state are treated differently. Two ideas that don't seem to work:

If I draw a bunch of edges after the solid circle, that looks like I'm iterating over rewards.
If I come up with a special kind of edge that represents an expected reward, then it looks like only a single next state is being considered.


text:
Neural network to extract correlated columns
I want to use a neural network to find correlated columns in a .csv file and give them as a output. The input .csv file has multiple columns with 0 and 1 ( like Booleans) in it. The file got the assignment from people to interests in it.
Example .csv input:
UserID   History  Math  Physics  Art  Music  ...
User1    0        1     1        0    0      ...
User2    0        0     0        1    1      ...
User3    0        1     1        1    1      ...
User4    1        0     1        1    0      ...
...

The output should be in this case something like: {math,physics}, {art,music}, {history,physics,art} - I exclude here {math,physics,art,music} because in a step afterwards i want to exclude (at least some) which can be created through the combination of others.
At the moment I have a problem that i don´t know which type of neural network could complete this task. How can I solve this problem?
So the important thing, that a column can have more than one column it correlates to - so its not like simple k-means clustering (as far as I understand it).

Did you figure this out? I am trying to do the exact same thing!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How does (or should) AlphaGoZero (which does chess) fare against Deep Blue?
Deep blue is good at chess, but is more "hand-coded" or "top-down".
https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)
AlphaGoZero is "self-taught", and at Go is very much super-human.
https://en.wikipedia.org/wiki/AlphaGo_Zero
How does AlphaGoZero fare when it goes head-to-head with DeepBlue?  Are there indicators like chess ratings?

Deep Blue is pretty primitive for todays best chess engines (e.g. Stockfish). Alpha Zero played Stockfish and beat it more times than it lost if I'm not mistaken. There are plenty of information online so you can reasearch about their games.

Thanks for "stockfish" reference.  I found this article on the subject. https://www.theringer.com/tech/2018/11/8/18069092/chess-alphazero-alphago-go-stockfish-artificial-intelligence-future

@EngrStudent Hi and welcome to AI SE! I just want to let you know that, in case you find an answer to your own question, you can write it below ;)

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Using sigmoid in LSTM network for multi-step forecasting
I'm trying to develop a multistep forecasting model using LSTM  Network. The model takes three times steps as input and predicting two time_steps. both input and output columns are normalised using minmax_scalar within the range of 0 and 1.
Please see the below model architecture
Model Architecture
model = Sequential()
model.add(LSTM(80,input_shape=(3,1),activation='sigmoid',return_sequences=True))
model.add(LSTM(20,activation='sigmoid',return_sequences=False))
model.add(Dense(2))

In this case, using sigmoid as an activation function is it correct?

You should not limit yourself to sigmoid as activation function on the last layer. Usually you're normalizing your dataset, but when you're testing/evaluating the model you're applying the inverse of the scaling transformation to the predictions, so you could easily use tanh which is defined on [-1, 1]

Yes, due the input, output being constrained between zero and one that would be the only viable activation function.

You have a problem in your code, you want to use "sigmoid" in the last layer. Fot the code you are showin you are using linear activation in the last layer.


text:
How to correctly implement self-play with DQN?
I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would improve if the agent trains against itself iteratively. I've seen posts about it, but never detailed implementation notes. My thoughts were to implement the following (agent and opponent are separate networks for now):

Bootstrap agent and opponent with initial weights (either random or trained against CPU, not sure)
Use Annealing Epsilon Greedy strategy for N iterations
After M iterations (M > N), copy agent network's weights to opponent's network
Reset annealing epsilon (i.e. start performing randomly again to explore new opponent)?
Repeat steps 2-4

Would something like this work? Some specific questions are:

Should I "reset" my annealing epsilon strategy every time the opponent is updated? I feel like this is needed because the agent needs sufficient time to explore new strategies for this "new" opponent.
Should the experience replay buffer be cleared out when the opponent is updated? Again, I think this is needed.

Any pointers would be appreciated.


text:
Using AI to enhance customer service
I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can we do the matching?
Generally, how is this problem solved? What sub-topic within AI is suitable for this problems? Classification, recommender systems, ...? Any pointers to open-source projects would be very helpful.

This sounds to me like a use case for a chatbot. You would have different intents reflecting the types of user queries that your system can respond to. The intent matching can be done by pattern matching, machine learning (classification), or a combination of the two (hybrid). You can then use the chatbot to ask clarification questions or elicit more information to identify which live agent would be the best person to take over the call. Essentially each live agent would have a list of intents plus added information (such as geographical area etc) which you then compare against the data from the caller to find the best match.
If this is a voice call you'd need to put an ASR system at the front of the pipeline. Chatbots can usually do live-agent handover to then pass control to a human agent at any time in the conversation.
[Disclaimer: I work for a company that operates in exactly that area and whose system works as described above]

AI can transform the customer experience is by providing personalized content. For example, When you see video recommendation on YouTube, you'll know that it's from AI technology. I recommend you to read this article for knowing how they work: A Sentiment-Enhanced Hybrid Recommender System for Movie Recommendation: A Big Data Analytics Framework (abstract, article).

Welcome to SE:AI!  Could I ask you to provide a quick paragraph on why the fundamentals of a movie recommendation could apply to customer service applications in general?  (We especially like answers that discuss the underlying theory.)


text:
Why does model complexity increase my validation score by a lot?
I learned that when creating neural networks the go to was to overfit and then to regularize. However I am now in a situation where, when I make the model more complex (more layers, more filters, ...) my scores become worse.
I am training a CNN to predict pollution 6 hours in advance. The input I give to my model is the pollution of the past 18 hours.
Can I safely say that because there probably is a lot of noise in this data that, that is the reason when increasing my complexity, my model becomes worse?

It is mostly likely due to overfitting. What's your training loss is like? Also, do you also increase regularization effects, e.g., dropout rate, weight-decay's delta, as you make your model more complex?

Yes it is definetly due to overfitting, but I assume that it is overfitting on noisy data, which makes it even worse. It gives quite good results on the training data, but very bad on the validation data. Even after regularizing the hell out of it, results do not become better than just a simple model with only 2 layers (in comparison to 4).

I see. I mean if the quality of your data is low then I dont think your validation accuracy can increase no matter what you do. But then I might be wrong.

Generally for your situation I think that the noise factor in the data is keeping the validation accuracy low. Another possibility is that you do not have a sufficient data set size to fully capture the probability distribution of the data. How many training samples do you have? Also how did you select the validation data, It should be a random sample of the data set so its distribution matches that of the training set.


text:
Why haven't we solved the problem of bipedal walking?
This has been a mystery to me.
All the walking robots look like idiots now. But we do have a lot of simulation-based results
(Flexible Muscle-Based Locomotion for Bipedal Creatures
), so why can't we just apply the simulation results to a real robot and let it walk, not like an idiot, but like an running ostrich?
With the main loop running at more than 60 fps, I fail to see how possibly the program could fail to stop the robot from losing balance. When I balance a stick on my hand, I could probably only do 5 fps.
We have not only supercomputers connected to the robots, but also reinforcement learning algorithms at our disposal, so what has been the bottleneck in the problem of bipedal walking?

Hi and welcome to AI SE! You say "All the walking robots look like idiots now." Can you please link us to a video/reference that shows that all walking robots walk like idiots, which is possible? Anyway, I assume you're talking about real-world robots, given that you show us a video of simulated robots walking more or less normally.

See the Atlas by Boston Dynamics. He kinda looks like an idiot in some parts of this video: https://www.youtube.com/watch?v=rVlhMGQgDkY :) I think the road from simulation-to-physical is not a trivial one. There are probably some hard challenges to tackle from a mechanical perspective. Otherwise, programming an agent to do bipedal walking in a simulated environment doesn't seem to challenging to me and is probably straightforward via genetic algorithms.


text:
If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?
In the original ResNet paper they talk about using plain identity skip connections when the input and output of a block have the same dimensions.

When the input and output have different dimensions they propose two options:
(A) Use an identity mapping padded with zeros to make up for the extra dimensions
(B) Use a "projection".

which (after some digging around in other people's code) I see as meaning: do a convolution with a 1x1 kernel with trainable weights.
(B) is confusing to me because it seems to ruin the point of ResNet by making the skip connection trainable. Then the main path is not really learning a "residual" relative to an identity transformation. So at this point, I'm no longer sure how to interpret the intent or expected effect of this type of block. And I would think that one should justify doing it in the first places instead of just not putting a skip connection there at all (which in my mind is the status-quo before this paper).
So can anyone help explain away my confusion here?

If it's not, that will make a big difference to the way I think about this. Maybe I'll go double check by looking at more implementations. They keep calling it a "projection" shortcut in the papers, and to me projection means collapsing one dimension of a vector down to zero, so I'm not sure it relates

Here's another implementation in tensorflow. Line 337 defines the projection_shortcut. I'm not too familiar with tf but it looks like a vanilla 1x1 conv to me.

What if someone got it wrong then everyone copied them... :|

I think I'll rewind to before you said anything. I was happy then :P

Well, I found an answer that satisfies me.
The zero-padded identity is not ideal. Suppose we're mapping from 64 channels to 128 channels. Then the zero-padded identity will map to an output where half of the channels are the same as the inputs, and the other half are all zeros. So that means the main path is learning a residual for half of the output channels, and learning a mapping from the ground up for the other half of the channels.
Now, the alternative is to use option (B) which is the single 1x1 convolution. After reading their ResNetV2 paper I realised that they really double down on this concept of maintaining a "clean" alternative path all the way up and down the network. So then the question is, what's "cleaner"?: a block of two convolutions of the form 3x3x64 then 3x3x128, or a shortcut with a 1x1x128 convolution? The shortcut is a tempting choice here. And in fact, in the original paper they show empirically that the convolutional shortcut is better than the identity shortcut. Maybe it might be worth running a test to see if the convolutional shortcut is better than no shortcut at all, but until I decide to run such a test, I'll presume the authors would have mentioned it. The other thing worth noting is that there are only a few places in the whole network where the dimensions need to be increased, so maybe the impact is small.


text:
Human Aggression Detection Community, Competition and dataset
I'm looking for a community or competition website related to human aggression detection using Deep Learning in a video.
Also, I'm looking for a dataset of human aggression activities.
Any suggestions would be appreciated.

Hi and welcome to AI SE! Please, ask just one question per post. If you have 2 questions, ask them in their separate post. Can you clarify why do you need a "community or competition website related to human aggression detection using Deep Learning in a video". I suggest you describe your goal and your specific problem more in detail.

Hey @Keivan I guess you need some dataset for Human Aggression Activity? I guess your work would become even better if you comeup with some kind of detail like Do you consider Aggression Scenes in Movies 'Aggressive according to your definition'? If yes, there are plenty of them, if no, I've some other methodologies to help you :-)

Hi @AbhasKumarSinha, I need not only some dataset for Human Aggression Activity but also methodologies and instructions for implementing it with deep learning (3D CNN, etc.). Aggression Scenes in movies are not very good for me. Can you please share your methodologies. My aim is to detect aggressive behavior between 2 persons or a small group of people in surveillance cameras that have a top view.


text:
MCTS moves with multiple parents
I'd like to develop an MCTS-like (Monte Carlo Tree Search) algorithm for program induction, i.e. learning programs from examples.
My initial plan is for nodes to represent programs and for the search to expand nodes by revising programs. Each edge represents a possible revision.
Many expansions involve a single program: randomly resample a subtree of the program, replace a constant with a variable, etc. It's straightforward to use these with MCTS.
Some expansions, however, generate a program from scratch (e.g. sample a new program). Others use two or more programs to generate a single output program (e.g. crossover in Genetic Programming).
These latter types of moves seem nonstandard for vanilla MCTS, but I'm not deeply familiar with the literature or what technical terms might be most relevant.
How can I model expansions involving $N \ne 1$ nodes? Are there accepted methods for handling these situations?

Approach 1: One way would be to switch from nodes representing programs to nodes representing sets of programs. The root node would represent the empty set ${}$, to which expansions could be applied only if they can generate a program from scratch. The first such expansion would produce some program $p$, so the root would now have child ${p}$. The second expansion would produce $p'$, so the root would now also have child ${p'}$ which would itself also have the child ${p,p'}.
One downside to this approach is that, even assuming reasonable restrictions (e.g. moves can use at most 2 programs, pairs cannot have identical elements, element order doesn't matter), the branching factor will grow combinatorially, because each new program can be combined with all (or most) of the previously generated programs.
Approach 2: Another way would be to switch from nodes representing programs to nodes representing meta-programs describing an initial program and a series of revisions to that program. For example, MCTS might search for expressions in something like this grammar:
Program  -> Empty
          | Program > OneProgramRevision
          | Program Program >> TwoProgramRevision
OneProgramRevision -> AddDatumAsException
                    | SampleStatement
                    | RegenerateSubtree
                    | Delete
                    | ...
TwoProgramRevision -> Concatenate
                    | Crossover
                    | ...

This is just a cartoon: one would need to add details parameterizing each of the various revisions.


text:
What are the main points of the top-down vs bottom-up paradigm in neural networks?
I've been reading some papers on human pose estimation and I'm starting to see the terms top-down and bottom-up crop up a lot. For example in this paper:

Our hourglass module differs from these designs mainly
  in its more symmetric distribution of capacity between bottom-up processing
  (from high resolutions to low resolutions) and top-down processing (from low
  resolutions to high resolutions)

Okay, so what are the main observations or distinctions of top-down vs bottom-up? Why does it make sense to have a paradigm in which we talk about these specifically?


text:
Automated explanation - function results - simple attempt
I used to work as an analyst in a financial project where we had functions $f$ determining the price, and sometimes the inputs $x$ jumped in such a way to produce anomalous results.
We had to report an explanation, and I wish to automate the process.
It's not properly a question of AI, more of information science.
The idea is that once, for a generic non-linear $f$, you can determine the ranking of relevance of $x_i$ in explaining the result, you can generate a full explanation by:

decompose $f$ as a composition of $f_j$, which are intermediate results with a definite meaning in the application domain (in this case, finance)
apply the algorithm using the $f_j$ instead of $x_i$, and then iterate it to explain the $f_j$ in terms of $x_i$

The relevance is quantified by the information gain of each variable. This will be explained for an application in ranking the $x_i$ directly.
We assume to start on a uniform distribution on the $x$ domain, calculate the derived probability density function for $f$, and the information entropy of $f$.
Then we fix the $x_i$ one at a time, for each calculate the new p.d.f. of $f$ conditioned on that $x_i$ and the (lower) information entropy of $f$. The information gain is $IG(x_i)$.
Choose as the first conditioning the $i$ with the largest information gain, then condition of the remaining $i$ with a decreasing order of $IG_i$.
So we could start for example , with $(x_1,x_2,x_3)$, to condition first on $x_2$, then on $(x_2,x_3)$, and then on $(x_2,x_3,x_1)$, getting the percentage contributions as:
$\frac{IG_i}{H_y}$. The successive terms $IG_i$ add up always to the total entropy $H_y$, since conditioning on all variables gives a point and zero entropy.
Any opinion on how to improve this "automated function explanation" is welcome


text:
Data classification model to detect a process in an event log
There are many example in python which has a ready made data set, for example there is T-Shirt pre-trained data and thousands images, within few minutes it will tell how many t-shirt images are there in those folders
but how that data detect.tflite itself is created from scratch step by step, If I wanted to manually write those data in csv how it should be done
Ultimately I have data which says, click event, keyboard event, process exe name, title of the window, and timestamp
I want to detect what exactly that user doing in his computer, who told do to what to him I want the software to tell me in words and diagrams
this is definitely a big software, but data classification part unclear to me
I need some guidance


text:
What does a joint probability density function have to do with Stochastic Optimal Control and Reinforcement Learning?
I stumbled upon a job offer from a company that was looking for someone who was good with Reinforcement Learning (applied to finance) and something in their offer caught my eye. It goes something like this:

We want you to be able to study the price dynamic (of a stock I suppose) and its evolution in order to extract a Joint PDF that will be used in the Optimal Stochastic Control of a Loss Function (or gain)

The thing is I understand what each of these things mean and how they are used separately (from my background in Control theory & dynamical systems) and I worked with fitting Joint PDFs and Copulas before, but I don't understand how a Joint PDF would help with the "Optimal Stochastic Control of a Loss Function" ? Thanks.

Yes but I'm by no means an expert. I'm still trying to learn about it

Yes I'm aware of that but I'm not clear on the in-depth procedures as I'm still just getting started with RL. I was wondering about how would modeling a Joint PDF separately be used in the optimal stochastic control of the Loss Function. The exact procedure I mean

Oh I see. I didn't think they meant the joint PDF of rewards, usually in finance the Joint is computed using the price movement relative to other explanatory variables that's why I was confused. This makes more sense. Thanks a lot for the wealth of information you just gave me ! That'll keep me busy digging through it for days. Thanks also for the book and the chatroom. I'll definitely make sure to join.

Extracting a joint PDF just means that you create a model that models the behavior of several variables combined instead of in isolation.
If these variables aren't independent and your loss functions is influenced by all of them, you obviously have to learn this joint PDF to minimize your loss.
So I don't see this statement as particularly mysterious.

The question is not "how can the joint distribution be useful in general", but "how a Joint PDF would help with the "Optimal Stochastic Control of a Loss Function"", although this answer may also answer the original question, if you are familiar with optimal stochastic control, etc.

@nbro yes that's exactly it. I'm wondering about the procedure by which modeling a joint PDF separately would help with the Optimal Stochastic Control of a Loss Function. I'm just learning RL so I'm sure that's where the confusion comes from.

From where do you get the "separately"?


text:
Is AlphaStar still competing in the Star Craft ladder?
Last year it was announced that Deepmind's Starcraft playing bot AlphaStar was taking on human players in the Starcraft ladder system (some kind of league system as far as I can tell) and that it had reached the Grandmaster level.
Since then I haven't really heard anything anymore about the progress of Alphastar. Given that I don't know anything about Starcraft I was wondering whether somebody has a better clue as to what Alphastar is up to? Is it still playing online? Or when did it stop playing? What was the improvement trajectory during the time it played online? Basically, how did this pan out, as seen from the perspective of the Starcraft community?


text:
Whats the correct loss function to use during deep Q-learning (discrete action space)
After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem.
As I understand, for a task with discrete action space, where there are 4 possible actions (lets say left, right, up, down) my DQN needs to have four outputs. And then argmax of prediction needs to be taken, which will be my predicted action (if argmax(prediction)==2 then I will pick third action, in my case up).
If I use Mean Squared Error loss for my network then the dimension of output needs to be same as dimension of expected target variables. I am calculating target variables using following code:
target = rewards[i] + GAMMA * max(Qs_next_state[i]) which gives me a single number (while predicted output is four dimensional) as I workaround I decided to use custom loss function:
def custom_loss(y_true, y_pred): # where y_true is target, y_pred is output of neural net
    return ((max(y_pred) - y_true)**2) / BATCH_SIZE

But I am not sure if it is correct, from what I have read from tutorials/papers loss functions do not have this max() in them. But how else am I going to end up with same dimensionality between targets and NN outputs. What is the correct approach here?


text:
How do weights changes handles during back-propagation when there are unknown labels
I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this is genomic data and to generate these data would take 8 years of lab work! Nevertheless I have genomic data for samples that have multiple-labels, sex age organ etc. this is a multi-class multi-label problem.
For most classes, ALL labels are complete. For two or three classes, there are unknown labels. an example would be the developmental stage of samples at age x, the developmental stage of sample at age y are known. the developmental stage of samples at age Z are unknown! (generating this data is what would take most time)... I would therefore like to include all this data during training as it is indispensable. I would like to generate the sigmoid probability and assign unknown label 'Z' as belonging to developmental stage 0 or 1 (known classes) based on a threshold (say >= 0.5)... When one-hot encoding the unknown labels simply have no ground truth, 0 for class developing and 0 for not-developing as follows (example of 3 samples shown for class in question):
  [[1., 0.
   [0., 1.
   [0., 0.  ......]]

the first row is known sample 1, second is known sample 2 and 3rd is unknown, and therefore has no ground truth. It is this sample i would like to assign a label of known class 1 or 2 based on the 'highest probability'.. based on reading and discussions, this is the direction i will be taking for this task, as it can be validated in the lab later... so the approach is, include in training and see what the network 'thinks' it is.
My question is: How does back-propagation handle these known and unknown samples with respect to weight updates.
I should note i have trained the network with ~90% validation performance. for all classes for which is there is complete data, the predictions are great. and the same for classes for which there is unknown data. It can accurately classify the samples for which there is known developmental stages... and it does assign a probability value to those samples that have the 'unknown' label (0,0), so i would really like to know how back-prop is handling these samples for the classes where there are unknown ground truth labels.
thank you!


text:
Building a template based NLG system to generate a report from data
I am a newbie to NLP and NLG. I am tasked to develop a system to generate a report based on a given data table. The structure of the report and the flow is predefined. I have researched on several existing python libraries like BERT, SimpleNLG but they don't seem to fit my need.
For example:
input_data(country = 'USA', industry = 'Coal', profit = '4m', trend = 'decline')
output: The coal industry in USA declined by 4m.
The input data array can be different combinations (and dynamic) based on a data table. I would like to know if  there is any python package available, or any resource discussing a practical approach for this.

Welcome to AI Stack Exchange. This seems more like a task for a very simple templating system, rather than anything requiring complex language models. If you don't think that is the case, can you give a few more examples of input and output that capture the range of possible topics and output styles that you need to generate?

I am also looking for a template-based approach like you. Could you please help me with this; since it's been a year you have asked this question, I am skeptical that must have built one.


text:
Object detection: combine many classes into one?
I am trying to train a model that detects logos in documents. Since I am not really interested in what kind of logo there is, but simply if there is a logo, does it make sense to combine all logos into 1 logo class?
Or are "logos" too diverse to group them together (like some logos are round, some are rectangular, some are even text based etc.) and the diversity of features will just make it hard for the neural network to learn? Or doesn't it matter?
(I am currently trying out the YOLOv3 architecture to begin with. Any other suggestions better suited are also welcome)

I think there is no absolute answer for this. Often its kind of trial and error. In general the CNN tries to generalize the problem, so using all logos with different augmentations and ground truths can maybe lead to some feature maps, which are so general that the CNN can find logos.
But if your logos are so various, and embedded in colorful websites, the tasks seems quite difficult, also if they vary in shape and form.Like you said, I think you definitely need an FPN (Feature Pyrimad network) to get the different sizes, scales and so on combined with an RPN (region-proposal network) to find the logos multiple times in the websites (if thats nessecary). For that you can use Mask-RCNN (https://github.com/matterport/Mask_RCNN). You can try to transfer-train it on Imagenet-Backbone for example to reduce training time. Just tried to use Mask-RCNN to segment colored cells from medical images. Worked out quite good.
I used ImageLabel for labeling, worked out quite intiutive and saved all in a JSON file.Imglabel
How large is your dataset and how complex are the logos ? Do you have examples which you can demonstrate?


text:
Where can I find example data for Nonogram solver?
I made CSP nonogram solver and I wanna test it on some bigger data.
Where can I find such data to test my program? I've been looking on the internet but I couldn't find anything.


text:
length independent sequence classification methods
I am looking to do sequence classification using deep learning. The length of my sequences can vary from a few hundred to several tens of thousands of characters. I was wondering what is a good approach for doing this. I had success with splitting a sequence into subsequences a few hundred characters long and using LSTMs, but then one is faced with the task of putting the results of each of those together and it is nontrivial as well. Any help would be appreciated.


text:
Are there known error bounds for TD(0) with a constant learning rate?
Is there any known error bounds for the TD(0) algorithm for the value function after a finite number of iterations?
$$ \Delta_t=\max_{s \in \mathcal{S}}|v_t(s)-v_\pi(s)|$$
$$v_{t+1}(s_t)=v_t(s_t)+\alpha(r+v_t(s_{t+1})-v_t(s_t))$$

The paper "Bias-Variance" Error Bounds for Temporal Difference Updates (2000) by M. Kearns and S. Singh provides error bounds for temporal-difference algorithms, i.e. TD($k$) and TD($\lambda$) (see theorem 1 and theorem 2, respectively). Note that both TD($k$) and TD($\lambda$) include TD($0$) as a special case.

I'm confused, is this for situations where the environment states can be enumerated or does it apply in general? I didn't see a parametrization.

@FourierFlux Do you mean if they assume that the state space is discrete? I need to read this paper again. It was a long time ago that I skimmed through it.


text:
How do I decide which norm to use for placing a constraint on my adversarial perturbation?
I am performing an adversarial machine learning attack on a neural network for network traffic classification. For adding adversarial perturbations in features such as packet interarrival times and packet size, what norm should I use as a constraint? (eg. l1 norm, l2 norm, l-infinity norm, etc.)

You should provide some details about where are you applying exactly the norm to. You're saying you're using it for placing a constraint, but it's not clear, unless someone knows what you're talking about.


text:
How to estimate the accuracy upper limit of any CNN model over a computer vision classification task
We are given a computer vision classification task, that is, a task that asks us to predict the category of an image over $n$ predefined classes (the so-called closed set classification problem).
Question: Is it possible to give an estimate on what is the best accuracy one is likely achieve using an end-to-end CNN model (possibly, using a popular backbone) in this task? Do the performances of state-of-the-arts models on open datasets serve as a good reference? If someone claims that they achieve certain performance with some popular CNN architecture, how do we know s/he is not bragging?
You may or may not have access to the training dataset yet. The testing dataset shall be something close to the real-world production scenario. I know this is too vague, but just assume you have a fair judge.
Background: Product teams sometimes asks engineering teams for quick (and dirty) solutions. Engineering teams want to assess the feasibility before say "Yes we can do $95\%$" and officially launch (and be responsible) the projects.

If the task is similar, you can have an estimate of performance from the state-of-the-art. E.g. if you want to build a facial recognition CNN, you can more or less know what to expect. This assumes that your dataset somewhat follows those that the sota is benchmarked upon. In different tasks you have no chance at all to see what you are expecting.

There is no easy rule for this. You can use transfer learning to select a model that works well on image classification. However the accuracy you achieve will be highly dependent on your training set.  If your training set is "similar" in quantity and quality to what was used  for the accuracy achieved by the transfer learning model in some application you have a reasonable chance of coming close to that accuracy. By similar is mean roughly the same number of images per class. By quality I mean what percentage of the pixels in the images that are occupied by the "region of interest -ROI), same level of noise in the images etc. Also it depends on the nature of the classes. If the classes are widely different (elephants vs trees) the accuracy should be higher than if you are try to classify closely related images (human faces).

Thanks for the answer. What if we assume the classification problems comes from production requirements in a specific domain, so it looks more like a fine-grained classification? (like those in iMaterialist Challenges distinguishing fashion/retail products/furniture) In such cases, "the nature of the classes" is proposing more difficulties. Is it highly likely that we won't get better performance on a 1000-class task than those on ImageNet leaderboard with the same backbone?


text:
What are the current research trends in recognizing narrative similarity?
I am currently working on a term paper on the topic of Narrative Similarity, based on Loizos Michael's work "Similarity of Narratives". I am trying to find the latest trends within this field of study for the literature overview in my assignement. However up untill now I wasn't able to find any new work on this particular subject.
I would appreciate any literature recommendation by anyone out there that has worked on this topic or is currently doing so.
Link to Michael's work:
https://www.researchgate.net/publication/316280937_Similarity_of_Narratives


text:
Is the derivative of the loss wrt a single scalar parameter proportional to the loss?
I am wondering about the correlation between the loss and the derivative of the loss wrt a single scalar parameter, with the same sample. That means: considering a machine learning model with parameters $\theta \in R$, I want to figure out the relationship between $Loss(x)$ and $\frac{\partial Loss(x)}{\partial \theta_i}$, where $i \in \{1,2,3,...,n\}$
Intuitively, I would like to consider that they are in a positive correlation, is it right? If it is right, how can I prove it in a mathematical way?

You need a function relating $x$ and $\theta$, otherwise your situation is not fully described. And typically in supervised learning loss is related to a comparison of transformed $x$ with a ground truth $y$ (although not always required)

For instance, for linear regression, you might have $\hat{y} = \theta \cdot x + b$ and $\mathcal{L}(\hat{y}, y) = \frac{1}{2}(\hat{y} - y)^2$ where $y$ is ground truth. For a concrete example you should [edit] in how $\theta$ is being used with $x$, and maybe the loss function you are using too. If you want a generic description/proof then you still need to specify how the loss function, $x$ and $\theta$ relate before someone could say whether correlation is always positive or not

The derivative $f'(x)$ is correlated with $f(x)$ in a certain sense. In fact, $f'(x)$ is a function of $f$, so we could even say that there's a cause-effect relationship.
The derivative at a specific point $c$ of the domain, i.e. $f'(c)$, can either be negative or positive. If $f'(c) > 0$, then $f(c)$ is increasing (with respect to an increase of $x$). If $f'(c) < 0$, then $f(c)$ decreasing  (with respect to an increase of $x$).
This can easily be seen from an example. Consider $f(x) = x^2$, then $f'(x) = 2x$. Let $c = 2$, then $f'(2) = 4$, so the function is increasing. In fact, $f(1) = 2 \leq f(2) = 4 \leq f(3) = 9$. Similarly, let $c = -1$, then $f'(-1) = -2$, so the function is decreasing. In fact, $\leq f(-2) = 4 \geq f(-1) = 1 \geq  f(0) = 0$ (note that the function is decreasing as we increase $x$!).
Consider a model with only one parameter, then the partial derivative of the loss function with respect to that parameter corresponds to the derivative of the loss function. So, the reasoning above applies to this model. What about a model with more than one parameter? The same thing happens.
If the function decreases, does its derivative also decrease? In general, no, and this can easily be seen from a plot of a function and its derivative. For example, consider a plot of a parabola and its derivative (which is a linear function).

On the left of the y-axis, the parabola is decreasing, but its derivative is increasing, while, on the right of the y-axis, the parabola is increasing and the linear function is still increasing.
This is the same thing with a loss function of an ML model and its partial derivative.

@DuttaA The function that computes $f'$ needs to have $f$ as input. This function is the differential operator. I am not completely sure about the mathematical formalism, but this should give you the idea and info to further investigate it!


text:
Efficient algorithm to obtain near optimal policies for an MDP
Given a discrete, finite Markov Decision Process (MDP) with its usual parameters $(S, A, T, R, \gamma)$, it is possible to obtain the optimal policy $\pi^{*}$ and the optimal value function $V^{*}$ through one of many planning methods (policy iteration, value iteration or solving a linear program).
I am interested in obtaining a random near-optimal policy $\pi$, with the value function associated with the policy given by $V^{\pi}$, such that
$$ \epsilon_1 < ||V^{*} - V^{\pi}||_{\infty} < \epsilon_2$$
I wish to know an efficient way of achieving this goal. A possible approach is to generate random policies and then to use the given MDP model to evaluate these policies and verify that they satisfy the criteria.
If only an upper bound were needed, the idea that near optimal value functions induce near optimal policies could be used, that is, we can show that, if
$$||V - V^{*}||_{\infty} < \epsilon, \quad \epsilon > 0$$
and if $\pi$ is the policy that is greedy with respect to the value function $V$, then
$$ ||V^{\pi} - V^{*}||_{\infty} < \frac{2\gamma\epsilon}{1 - \gamma}$$
So by picking a suitable $\epsilon$ for the given $\gamma$, we can be sure of any upper bound $\epsilon_2$.
However, I would also like that the policy $\pi$ not be "too good", hence the requirement for a lower bound.
Any inputs regarding an efficient solution or reasons for the lack thereof are welcome.

Seems like this will not be possible in the general case. Consider a MDP with a single action. It is not possible find a policy that does worse than taking that one action always, so there is no solution for your query any $\epsilon_1 > 0$.

@mikkola thanks for your comment. It was a nice way of seeing the issue here. From a practical implementation point of view, I ran policy iteration on the inverse MDP (R replaced by -R) to obtain a policy that will be "at least as bad" instead of the "at least as good" set procedure mentioned above (A paper by Singh. et al)


text:
What kind of artificial intelligence is this? A decentralized swarm intelligence where the input and output is split among the agents
I have an AI design for deciding the length of green and red lamps of the traffic. In my design, every crossroads has its own agent. This agent has input the amount of vehicle in each road in a single junction. AI then decide how long is the red lamp and the green lamp in each junction. The fitness function is the average commute time in the city. Each agent may communicate with each other, and give reward or punishment to other AI. What AI algorithm works like this?


text:
Running 10 epochs on the Food-101 dataset
I’m currently working on the Food-101 dataset. I want to train a model that is greater than 85% accuracy for top-1 for the test set, using a ResNet50 or smaller network with a reasonable set of augmentations. I’m running 10 epochs using ResNet34 and I’m currently on the 8th epoch. This is how its doing:
epoch   train_loss  valid_loss  error_rate  time
0   2.526382    1.858536    0.465891    25:21
1   1.981913    1.566125    0.406881    27:21
2   1.748959    1.419548    0.372129    27:16
3   1.611638    1.315319    0.346980    25:16
4   1.568304    1.250232    0.328069    24:43
5   1.438499    1.193816    0.313762    24:26
6   1.378019    1.156924    0.307426    24:30
7   1.331075    1.131671    0.299010    24:26
8   1.314978    1.115857    0.297079    24:24

As you can see, it doesn’t seem like I’m going to do better than 71% accuracy at this point. The dataset size is 101,000. It has 101 different kinds of food and each food has a 1000 images. Training this definitely takes long but what are some things I can do to improve its accuracy?

try using an adjustable learning rate. Keras has a number of callbacks that are useful for this purpose. The ReduceLROnPlateau callback can be used to monitor validation loss and reduce the learning rate by a factor if the validation loss does not decrease after a user specified number of epochs. The ModelCheckpoint callback is useful to monitor the validation loss and save the model with the lowest loss which can then be used to make predictions. Documentation is here.


text:
In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?
I'm creating an RL application for the game Connect Four.
If I tell the algorithm which moves/token positions will receive greater rewards, surely it's not actually learning anything; it's just a basic lookup for the algorithm? "Shall I place the token here, or here? Well, this one receives a greater reward, so I choose this one."
For example, some pseudocode:
function get_reward()
    if 2 in a line
        return 1
    if 3 in a line
        return 2
    if 4 in a line
        return 10
    else
        return -1

foreach columns
    column_reward_i = get_reward(column_i)
    if column_reward_i >= column_rewards
        place_token(column_i)

Maybe you could show a plot of the performance of your algorithm through time.

There is no code at the moment, I'm just trying to work out how, if I'm assigning values for different positions, the algorithm is actually learning anything?

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components and feeding that knowledge into the algorithm, then you place more emphasis on your understanding of the problem, and less on any learning that an agent might do.
Typically in a RL formulation of a simple board game, you would choose rewards or +1 for a win (the goal), 0 for a draw, and -1 for a loss. All non-terminal states would score 0 reward. The point of the RL learning algorithm is that the learning process would assign some nominal value to interim states due to observing play. For value-based RL approaches, such as Q learning or Monte Carlo Control, the algorithm does this more or less directly by "backing up" rewards that it experiences in later states into average value estimates for earlier states.
Most game-playing agents will combine the learning process, which will be imperfect given the limited experience an agent can obtain compared to all possible board states, with a look-ahead search method. Your heuristic scores would also make a reasonable input to a search method too - the difference being you may need to search more deeply using your simple heuristic than if you used a learned heuristic. The simplest heuristic would just be +1 for a win, 0 for everything else, and is still reasonably effective for Connect 4 if you can make it search e.g. 10 moves ahead.
The combination of deep Q learning and negamax search is quite effective in Connect 4. It can make near perfect agents. However, if you actually want a perfect agent, you are better off skipping the self-learning approach and working on optimised look-ahead search with some depth of opening moves stored as data (because search is too expesnive in the early game, even for a simple game like Connect 4).

I actually have a Connect 4 agent in training right now, based on the +1/0/-1 reward scheme plus deep Q learning, thanks to this Kaggle competition:  https://www.kaggle.com/c/connectx - there are plenty of examples and starter code available there

Thank you, Neil. Your first paragraph confirms what I suspected – feeding the algorithm with the 'best' positions/moves means it's not actually learning!
In your second paragraph "choose rewards for a win/lose/draw", are you referring to Monte Carlo methods? As these assign the rewards at the end of the episode/terminal state, right?
The state space of Connect Four would be too large to use basic Q-Learning, correct?
For look-ahead search methods, MiniMax and MCST? Can you recommend any others? Are these both Monte Carlo methods?
The formatting on this is bad, I hope u can follow

@mason7663 That's lots of extra questions and not really a good idea to follow up in comments with so many new directions. You can alsways ask a new question on the site, for any specific thing. Also I do recommend that you take a look at kaggle.com/c/connectx because lots of people had similar questions when that competition started and there's a whole bunch of resources linked


text:
Feature extraction for exponentially damped signals
I am looking into exponentially damped signals where it is a stationary signal (after implementing the Adfuller statistical test) and I would like to look into how can I extract meaningful features out of the signal in order to do pattern recognition with machine learning. Can anyone guide me on where I can find articles/blogs of signal processing techniques and feature extraction of exponentially damped signal?
My situation:
I want to look into features that relate to damping of the signal, I already looked at it in the frequency domain and I found out that from my datasets (considering the first 3 Natural frequencies/modes) the peaks are almost the same (there's like the same deviation by only like [+] or [-] 0.5 from freq. values). Looking into the damping factor, I found out that only the second damping ratio was different but still small deviation around the same ([+] or [-] 0.5). So, I thought that it would be difficult for machine learning to identify the difference between cases. One of my ideas is to look into energy dissipation as it might be related to damping, but I don't know how to approach it or from which domain I need to go in order to get the features.
Side Question:
I have several questions regarding signal processing:

Say I have a signal and would like to extract features from it,
what steps or points that I should know in order to implement signal processing? (As I am using Python).
When I used signaltonoise function online (python) in order to see
the signal-to-noise ratio and I got a positive SNR. However, if I
pass the signal into, for example, a band-pass filter to concentrate on a certain frequency band I get a negative SNR. Why is that?
How can I extract features from STFT? And I also know about wavelet and HHT, what are the uses of both algorithms and how to also extract features from it?

Hi and welcome to AI SE! I haven't fully read your post, but you're asking many questions here. Please, simplify your post so that it becomes more accessible and manageable for readers. If you have multiple questions, I suggest you ask each of them in its separate post. Also, there's Signal Process SE if your problem is mainly related to signals. You may find more help there.


text:
Why can't I Hyper tune my KNNBasic Algorithm?
I've been trying to hyper tuning my KNNBasic algorithm by the help of grid search for recommendation system for movie review data. The problem is that both of my KNNBasicTuned and KNNBasicUntuned shows the same result. Here is my code for KNNTuning..
I have tried the SVD algo tuning and it worked perfectly so all my libraries are working perfectly. However my all libraries are in my github linke : https://github.com/iSarcastic99/KNNBasicTuning
Code of KNNBasicTuning :

# -*- coding: utf-8 -*-
"""
Created on Sat Apr  4 01:25:40 2020

@author: rahulss
"""
#My libraries

from MovieLens import MovieLens
from surprise import KNNBasic
from surprise import NormalPredictor
from Evaluator import Evaluator
from surprise.model_selection import GridSearchCV

import random
import numpy as np

#loading my working data
def LoadMovieLensData():
    ml = MovieLens()
    print("Loading movie ratings...")
    data = ml.loadMovieLensLatestSmall()
    print("\nComputing movie popularity ranks so we can measure novelty later...")
    rankings = ml.getPopularityRanks()
    return (ml, data, rankings)

np.random.seed(0)
random.seed(0)

# Load up common data set for the recommender algorithms
(ml, evaluationData, rankings) = LoadMovieLensData()

print("Searching for best parameters...")
param_grid = {'n_epochs': [10, 30], 'lr_all': [0.005, 0.010],
          'n_factors': [50, 90]}
gs = GridSearchCV(KNNBasic, param_grid, measures=['rmse', 'mae'], cv=3)

gs.fit(evaluationData)

# best RMSE score
print("Best RMSE score attained: ", gs.best_score['rmse'])

# combination of parameters that gave the best RMSE score
print(gs.best_params['rmse'])

# Construct an Evaluator to, you know, evaluate them
evaluator = Evaluator(evaluationData, rankings)

params = gs.best_params['rmse']
KNNBasictuned = KNNBasic(n_epochs = params['n_epochs'], lr_all =  params['lr_all'], n_factors = params['n_factors'])
evaluator.AddAlgorithm(KNNBasictuned, "KNN - Tuned")

KNNBasicUntuned = KNNBasic()
evaluator.AddAlgorithm(KNNBasicUntuned, "KNN - Untuned")

# Evaluating all algorithms
evaluator.Evaluate(False)

evaluator.SampleTopNRecs(ml, testSubject=85, k=10)

Hi and welcome to AI SE! You say "The problem is that both of my KNNBasicTuned and KNNBasicUntuned shows the same result.", but what do you mean by "result"?

@nbro, Thanks for welcoming me. By result, I meant my RMSE and MAE score of both KNNBasicTuned and KNNBasicUntuned are same.

All the parameters you set on the grid search are completely irrelevant for KNN. You probably get the same results cause you're always training the model with default parameters. The only parameters you can fine tune are k, min_k and eventually extra options (like the metric to use to compute the distances). Check the documentation link

@EdoardoGuerriero, First thanks for trying to help. However, as you can see that i've used GridSearchCV and it requires the given parameters above. For more infor pls check the link here     GridSearchCV

@techPirate99 In a grid search you need to specify the parameters you want to tune, but a KNN is not trained using epochs, and it doesn't require a learning rate, it is not a neural network. I'll write an answer in an hour to show you what you can tune.

@EdoardoGuerriero, I completely agree with you that i need something else(other than GridSearchCV) which can help tuning knn and thats what i was hoping to be answered. Waiting for your answer. Thanks again.


text:
Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?
I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again using saver.restore(), the network gives out a reward equal to approximately -1800. Why is this happening, maybe I'm doing something wrong?
My network:
import tensorflow as tf
import numpy as np
import gym

epsiode_steps = 500

# learning rate for actor
lr_a = 0.001

# learning rate for critic
lr_c = 0.002
gamma = 0.9
alpha = 0.01
memory = 10000
batch_size = 32
render = True

class DDPG(object):
    def __init__(self, no_of_actions, no_of_states, a_bound, ):
        self.memory = np.zeros((memory, no_of_states * 2 + no_of_actions + 1), dtype=np.float32)

        # initialize pointer to point to our experience buffer
        self.pointer = 0

        self.sess = tf.Session()

        self.noise_variance = 3.0

        self.no_of_actions, self.no_of_states, self.a_bound = no_of_actions, no_of_states, a_bound,

        self.state = tf.placeholder(tf.float32, [None, no_of_states], 's')
        self.next_state = tf.placeholder(tf.float32, [None, no_of_states], 's_')
        self.reward = tf.placeholder(tf.float32, [None, 1], 'r')

        with tf.variable_scope('Actor'):
            self.a = self.build_actor_network(self.state, scope='eval', trainable=True)
            a_ = self.build_actor_network(self.next_state, scope='target', trainable=False)

        with tf.variable_scope('Critic'):
            q = self.build_crtic_network(self.state, self.a, scope='eval', trainable=True)
            q_ = self.build_crtic_network(self.next_state, a_, scope='target', trainable=False)

        self.ae_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Actor/eval')
        self.at_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Actor/target')

        self.ce_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Critic/eval')
        self.ct_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Critic/target')

        # update target value
        self.soft_replace = [
            [tf.assign(at, (1 - alpha) * at + alpha * ae), tf.assign(ct, (1 - alpha) * ct + alpha * ce)]
            for at, ae, ct, ce in zip(self.at_params, self.ae_params, self.ct_params, self.ce_params)]

        q_target = self.reward + gamma * q_

        td_error = tf.losses.mean_squared_error(labels=(self.reward + gamma * q_), predictions=q)

        self.ctrain = tf.train.AdamOptimizer(lr_c).minimize(td_error, name="adam-ink", var_list=self.ce_params)

        a_loss = - tf.reduce_mean(q)

        # train the actor network with adam optimizer for minimizing the loss
        self.atrain = tf.train.AdamOptimizer(lr_a).minimize(a_loss, var_list=self.ae_params)

        tf.summary.FileWriter("logs2", self.sess.graph)

        # initialize all variables

        self.sess.run(tf.global_variables_initializer())
        self.saver = tf.train.Saver()
        self.saver.restore(self.sess, "Pendulum/nn.ckpt")

    def choose_action(self, s):
        a = self.sess.run(self.a, {self.state: s[np.newaxis, :]})[0]
        a = np.clip(np.random.normal(a, self.noise_variance), -2, 2)

        return a

    def learn(self):
        # soft target replacement
        self.sess.run(self.soft_replace)

        indices = np.random.choice(memory, size=batch_size)
        batch_transition = self.memory[indices, :]
        batch_states = batch_transition[:, :self.no_of_states]
        batch_actions = batch_transition[:, self.no_of_states: self.no_of_states + self.no_of_actions]
        batch_rewards = batch_transition[:, -self.no_of_states - 1: -self.no_of_states]
        batch_next_state = batch_transition[:, -self.no_of_states:]

        self.sess.run(self.atrain, {self.state: batch_states})
        self.sess.run(self.ctrain, {self.state: batch_states, self.a: batch_actions, self.reward: batch_rewards,
                                    self.next_state: batch_next_state})

    # we define a function store_transition which stores all the transition information in the buffer
    def store_transition(self, s, a, r, s_):
        trans = np.hstack((s, a, [r], s_))

        index = self.pointer % memory
        self.memory[index, :] = trans
        self.pointer += 1

        if self.pointer > memory:
            self.noise_variance *= 0.99995
            self.learn()

    # we define the function build_actor_network for builing our actor network and after crtic network
    def build_actor_network(self, s, scope, trainable)
        with tf.variable_scope(scope):
            l1 = tf.layers.dense(s, 30, activation=tf.nn.tanh, name='l1', trainable=trainable)
            a = tf.layers.dense(l1, self.no_of_actions, activation=tf.nn.tanh, name='a', trainable=trainable)
            return tf.multiply(a, self.a_bound, name="scaled_a")

    def build_crtic_network(self, s, a, scope, trainable):
        with tf.variable_scope(scope):
            n_l1 = 30
            w1_s = tf.get_variable('w1_s', [self.no_of_states, n_l1], trainable=trainable)
            w1_a = tf.get_variable('w1_a', [self.no_of_actions, n_l1], trainable=trainable)
            b1 = tf.get_variable('b1', [1, n_l1], trainable=trainable)
            net = tf.nn.tanh(tf.matmul(s, w1_s) + tf.matmul(a, w1_a) + b1)

            q = tf.layers.dense(net, 1, trainable=trainable)
            return q

    def save(self):
        self.saver.save(self.sess, "Pendulum/nn.ckpt")

env = gym.make("Pendulum-v0")
env = env.unwrapped
env.seed(1)

no_of_states = env.observation_space.shape[0]
no_of_actions = env.action_space.shape[0]

a_bound = env.action_space.high
ddpg = DDPG(no_of_actions, no_of_states, a_bound)

total_reward = []

no_of_episodes = 300
# for each episodes
for i in range(no_of_episodes):
    # initialize the environment
    s = env.reset()

    # episodic reward
    ep_reward = 0

    for j in range(epsiode_steps):

        env.render()

        # select action by adding noise through OU process
        a = ddpg.choose_action(s)

        # peform the action and move to the next state s
        s_, r, done, info = env.step(a)

        # store the the transition to our experience buffer
        # sample some minibatch of experience and train the network
        ddpg.store_transition(s, a, r, s_)

        # update current state as next state
        s = s_

        # add episodic rewards
        ep_reward += r

        if int(ep_reward) == 0 and i > 200:
            ddpg.save()
            print("save")
            quit()

        if j == epsiode_steps - 1:
            total_reward.append(ep_reward)
            print('Episode:', i, ' Reward: %i' % int(ep_reward))

            break


text:
What is the most efficient data type to store probabilities?
In ML we often have to store a huge amount of values ranging from 0 to 1, mostly being probabilities. The most common data structure to do so seems to be a floating point? Indeed, the range of floating points is huge. This makes them imprecise in the desired interval and inefficient, right?
This question suggests using the biggest integer value to represent a 1 and the smallest for 0. Also, it points to the Q number format, where all bits can be chosen as fractional, which sounds very efficient.
Why have these data types still not found their ways into numpy, tensorflow etc.? Am I missing something?


text:
Isn't a simulation a great model for model-based reinforcement learning?
Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having a good model of the environment allows to use planning and thus drastically improves the sample efficiency!
Why is the simulation not used for planning in these cases? It is a sampling model of the environment, right? Can't we try multiple actions at each or some states, follow the current policy to look several steps ahead and finally choose the action with the best outcome? Shouldn't this allow us to find better actions more quickly compared to policy gradient updates?
In this case, our environment and the model are kind of identical and this seems to be the problem. Or is the good old curse of dimensionality to blame again? Please help me figure out, what I'm missing.

Hi and welcome to AI SE! What makes you think that simulated environments are or can not be used for planning?

Hi @nbro! Thank you for the warm comment and the helpful question, which I'm happy to answer: afaiu, the most popular deep RL algorithms are model-free. These are sample inefficient as they learn from pure interactions with the environment. An agent with a good model of the environment could use the model to plan ahead. This way the agent would get experience without interacting with the environment and thus increase the sample efficiency. When this is true, why are popular algorithms like PPO still very sample inefficient? I think I'm missing something. Please help : )

I will give one perspective on this from the domain of robotics. You are right that most RL agents are trained in simulation particularly for research papers, because it allows researchers to in theory benchmark their approaches in a common environment. Many of the environments exist strictly as a test bed for new algorithms and are not even physically realizable, e.g. HalfCheetah. You could in theory have a separate simulator say running in another process that you use as your planning model, and the "real" simulator is then your environment. But really that's just a mocked setup for what you really want in the end, which is having a real-world agent in a real-world environment.
What you describe could be very useful, with one important caveat: the simulator needs to in fact be a good model of the real environment. For robotics and many other interesting domains, this is a tall order. Getting a physics simulator that faithfully replicates the real-world environment can be tricky, as one may need accurate friction coefficients, mass and center of mass, restitution coefficients, material properties, contact models, and so on. Oftentimes the simulator is too crude an approximation of the real-world environment to be useful as a planner.
That doesn't mean we're completely hosed though. This paper uses highly parallelized simulators to search for simulation parameters that approximate the real-world well. What's interesting is it's not even necessarily finding the correct real-world values for e.g. friction coefficients and such, but it finds values for parameters that, taken together, produces simulations that match the real-world experience. The better the simulation gets at approximating what's going on in the real world, the more viable it is to use the simulator for task planning. I think with the advent of GPU-optimized physics simulators we will see simulators be a more useful tool even for real-world agents, as you can try many different things in parallel to get a sense of what is the likely outcome of a planned action sequence.

Shouldn't this allow us to find better actions more quickly compared to policy gradient updates?

It depends on the nature of the simulation. If the simulation models a car as a solid body moving with three $(x,y,\theta)$ degrees of freedom in a plane (hopefully, if it doesn't hit anything and propel vertically), the three ordinary differential equations of solid body motion can be solved quite quickly, compared to a simulation used to model the path of least resistance of a ship on wavy sea, where fluid dynamics equations must be solved, that require a huge amount of resources. OK, the response time needed for a ship is much longer, than for a car, yes, but to compute it predictively, one needs a huge amount of computational power.

The question is generalizability. I completely agree though but, ideally the policy found will generalize to more complex environments the model hasn't seen. You could also run a planner on a new scenario but the issue is that it would be too computationally demanding for real time.


text:
Can artificial intelligence classify textual records?
I am a records manager and I am being asked if I recommend Office 365.  I'm having a hard time making a recommendation because I am missing an essential piece of information: can Office 365 replace the manual process of placing records into categories based on organizational function?  It is important that this is done accurately, because the category determines how long the records should be kept before they are irretrievably destroyed.  James Lappin seems to say that yes, Office 365 is underwritten by Project Cortex, and it is capable of doing this.
My sense is that artificial intelligence is not yet capable of determining a conceptual category for records.  For this to be true, a machine would have to replicate a complex human process: reading a free-text, free-form document; identifying the relevant pieces of information in a document, while ignoring others to determine what the document is "about"; then taking the answer of what the document is "about" and matching it to a predefined set of major organizational activity.
Are there any AI experts who can comment on how realistic it is to expect Project Cortex to do this?

AI can categorize documents very accurately.  It is not a new application but in the last year the accuracy of the underlying algorithms such as text classifiers, and language models in general, has significantly improved.  There are applications of language models which now surpass human performance. Microsoft is one of the leaders in this area.  For example, see their Turing Natural Language Generation (T-NLG) language model: Turing-NLG: A 17-billion-parameter language model by Microsoft. The T-NLG was developed in a research group called Project Turing which works on AI tools for Microsoft's products including Office-365.  My guess is this same research group makes the AI for Project Cortex.

Hi Brian - thank you very much for your informative answer!


text:
NaNs after a while in training of PPO
My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following:
ep=   3| t=  144/450000|  0.8 sec| rew=31.3005| ploss=-2.5e-02| vfloss=4.7e+01| kl=1.3e-03| ent=8.5e+00| clipfrac=0.0e+00
ep=   6| t=  288/450000|  1.1 sec| rew=31.2144| ploss=-2.2e-02| vfloss=4.1e+01| kl=1.3e-03| ent=8.5e+00| clipfrac=0.0e+00
ep=   9| t=  432/450000|  1.4 sec| rew=28.2668| ploss=-2.9e-02| vfloss=3.5e+01| kl=1.6e-03| ent=8.5e+00| clipfrac=0.0e+00
ep=  12| t=  576/450000|  1.7 sec| rew=28.2910| ploss=-2.7e-02| vfloss=3.6e+01| kl=1.7e-03| ent=8.5e+00| clipfrac=0.0e+00
ep=  15| t=  720/450000|  2.0 sec| rew=27.4817| ploss=-2.3e-02| vfloss=3.0e+01| kl=1.8e-03| ent=8.5e+00| clipfrac=0.0e+00
ep=  18| t=  864/450000|  2.3 sec| rew=29.8415| ploss=-4.5e-02| vfloss=3.4e+01| kl=4.0e-03| ent=8.5e+00| clipfrac=2.8e-02
ep=  21| t= 1008/450000|  2.6 sec| rew=29.1447| ploss=-2.7e-02| vfloss=2.7e+01| kl=2.0e-03| ent=8.5e+00| clipfrac=6.9e-03
ep=  24| t= 1152/450000|  3.0 sec| rew=30.2001| ploss=-3.5e-02| vfloss=2.8e+01| kl=1.7e-03| ent=8.5e+00| clipfrac=6.9e-03
ep=  27| t= 1296/450000|  3.3 sec| rew=31.4069| ploss=-2.9e-02| vfloss=3.7e+01| kl=3.0e-03| ent=8.5e+00| clipfrac=2.1e-02
ep=  30| t= 1440/450000|  3.6 sec| rew=27.7963| ploss=-4.6e-02| vfloss=2.3e+01| kl=7.3e-03| ent=8.5e+00| clipfrac=1.7e-01
ep=  33| t= 1584/450000|  3.9 sec| rew=30.8561| ploss=-5.9e-02| vfloss=2.5e+01| kl=9.6e-03| ent=8.5e+00| clipfrac=2.6e-01
ep=  36| t= 1728/450000|  4.2 sec| rew=27.3002| ploss=-6.9e-02| vfloss=2.2e+01| kl=1.3e-02| ent=8.5e+00| clipfrac=3.1e-01
ep=  39| t= 1872/450000|  4.5 sec| rew=28.0270| ploss=-5.6e-02| vfloss=2.1e+01| kl=8.9e-03| ent=8.5e+00| clipfrac=2.0e-01
ep=  42| t= 2016/450000|  4.9 sec| rew=28.0624| ploss=-5.8e-02| vfloss=2.0e+01| kl=7.5e-03| ent=8.5e+00| clipfrac=2.4e-01
ep=  45| t= 2160/450000|  5.2 sec| rew=28.6224| ploss=-8.4e-02| vfloss=2.3e+01| kl=7.2e-03| ent=8.5e+00| clipfrac=2.0e-01
ep=  48| t= 2304/450000|  5.5 sec| rew=32.3889| ploss=-4.3e-02| vfloss=2.6e+01| kl=7.1e-03| ent=8.5e+00| clipfrac=2.9e-01
ep=  51| t= 2448/450000|  5.8 sec| rew=31.4241| ploss=-1.0e-01| vfloss=2.7e+01| kl=7.0e-03| ent=8.5e+00| clipfrac=2.1e-01
ep=  54| t= 2592/450000|  6.1 sec| rew=33.4760| ploss=-5.1e-02| vfloss=2.5e+01| kl=7.3e-03| ent=8.5e+00| clipfrac=2.4e-01
ep=  57| t= 2736/450000|  6.4 sec| rew=31.0780| ploss=-8.8e-02| vfloss=2.3e+01| kl=6.9e-03| ent=8.5e+00| clipfrac=3.0e-01
ep=  60| t= 2880/450000|  6.7 sec| rew=34.1286| ploss=-6.9e-02| vfloss=2.7e+01| kl=7.6e-03| ent=8.5e+00| clipfrac=3.1e-01
ep=  63| t= 3024/450000|  7.1 sec| rew=31.0017| ploss=-6.1e-02| vfloss=2.5e+01| kl=1.4e-02| ent=8.5e+00| clipfrac=3.7e-01
ep=  66| t= 3168/450000|  7.4 sec| rew=32.3697| ploss=-1.1e-01| vfloss=2.2e+01| kl=1.2e-02| ent=8.5e+00| clipfrac=4.6e-01
ep=  69| t= 3312/450000|  7.7 sec| rew=31.4455| ploss=-7.4e-02| vfloss=2.4e+01| kl=8.4e-03| ent=8.5e+00| clipfrac=3.7e-01
ep=  72| t= 3456/450000|  8.0 sec| rew=32.1896| ploss=-9.2e-02| vfloss=2.0e+01| kl=1.3e-02| ent=8.4e+00| clipfrac=4.1e-01
ep=  75| t= 3600/450000|  8.3 sec| rew=31.3721| ploss=-9.4e-02| vfloss=2.4e+01| kl=1.4e-02| ent=8.4e+00| clipfrac=4.2e-01
ep=  78| t= 3744/450000|  8.6 sec| rew=35.5718| ploss=-1.0e-01| vfloss=3.0e+01| kl=1.0e-02| ent=8.4e+00| clipfrac=4.6e-01
ep=  81| t= 3888/450000|  9.0 sec| rew=32.2289| ploss=-1.3e-01| vfloss=2.9e+01| kl=1.6e-02| ent=8.4e+00| clipfrac=5.5e-01
ep=  84| t= 4032/450000|  9.3 sec| rew=31.7656| ploss=-1.0e-01| vfloss=2.3e+01| kl=1.3e-02| ent=8.4e+00| clipfrac=4.4e-01
ep=  87| t= 4176/450000|  9.6 sec| rew=35.4555| ploss=-8.8e-02| vfloss=3.3e+01| kl=5.8e-03| ent=8.4e+00| clipfrac=3.1e-01
ep=  90| t= 4320/450000|  9.9 sec| rew=33.2766| ploss=-1.2e-01| vfloss=2.4e+01| kl=1.5e-02| ent=8.4e+00| clipfrac=5.7e-01
ep=  93| t= 4464/450000| 10.2 sec| rew=32.5218| ploss=-1.1e-01| vfloss=2.4e+01| kl=1.8e-02| ent=8.4e+00| clipfrac=5.6e-01
ep=  96| t= 4608/450000| 10.5 sec| rew=34.7137| ploss=-9.7e-02| vfloss=2.5e+01| kl=1.5e-02| ent=8.4e+00| clipfrac=4.3e-01
ep=  99| t= 4752/450000| 10.9 sec| rew=35.3797| ploss=-8.2e-02| vfloss=2.5e+01| kl=1.6e-02| ent=8.4e+00| clipfrac=4.9e-01
ep= 102| t= 4896/450000| 11.2 sec| rew=nan| ploss=nan| vfloss=nan| kl=nan| ent=nan| clipfrac=0.0e+00
C:\Users\ppo.py:154: RuntimeWarning: invalid value encountered in greater
  adv_nrm = (adv_nrm - adv_nrm.mean()) / max(1.e-8,adv_nrm.std()) # standardized advantage function estimate
ep= 105| t= 5040/450000| 11.5 sec| rew=nan| ploss=nan| vfloss=nan| kl=nan| ent=nan| clipfrac=0.0e+00

Any ideas why this is arised?

$\log (0) \cdot 0$ probably, add little constant to argument in $\log$

Sorry but did you figure this out?

@hridayns Sorry for the late response. I remember that I changed the implementation of the PPO algorithm that I was using to the one and it worked out fine.


text:
Is Mean Squared Error Loss function a good loss function for continuous variables $0 < x < 1$
Suppose I am utilising a neural network to predict the next state, $s'$ based on the current $(s, a)$ pairs.
all my neural network inputs are between 0 and 1 and the loss function for this network is defined as the mean squared error of the Difference between the current state and next state. Because the variables are all between 0 and 1, the MSE difference between the actual difference and predicted difference is smaller than the actual difference.
Suppose the difference in next state and current state for $s \in R^2$ is $[0.4,0.5]$ and the neural network outputs a difference of $[0.2,0.4]$. The mean squared loss is therefore 0.05 $(0.2^2 + 0.1^2) = 0.05$ whereas the neural network does not really predict the next state very well due to a difference of $(0.2, 0.1)$.
Although whichever loss function is used does not matter, It was deceiving to think that despite the loss function outputting low values, it is mainly due to the squared term that keeps the value small.
Is Mean Squared Error loss function still a good loss function to be used here ?

Mean is usually done over batch size, not elements of the output vector. Besides, nothing is stopping you from having $n$ outputs nodes and having separate loss function for each element, that would be equal to summing the loss of all elements


text:
Possible approaches to dealing with unbalanced dataset and highly biased deep learning algorithm
I have an extremely unbalanced  video dataset  for a two class video classification problem.All my videos in my current video dataset is $40$ second long with $900p$ resolution.However the dataset is highly unbalanced with $3000$ samples for class A vs $300$ samples to class B. Due to high imbalance, i added the following class weight implementation to my deep learning model.
https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html
However  my model was still heavily biased due to high imbalance in data. I am considering   the following options:

Adding more video data to balance the dataset:  My only concern here is my current dataset is uniform with same duration of videos and similar resolution of around $900p$. Will it matter if I add  very low resolution videos to balance the dataset?
Adding video augmentation to current dataset.

I am looking for any other recommendations that i could make use. Any pros or cons of any of these methods that I should consider to prevent any bias?


text:
How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?
Neural networks typically have $\mathcal{VC}$ dimension that is proportional to their number of parameters and inputs. For example, see the papers Vapnik-Chervonenkis dimension of recurrent neural networks (1998) by Pascal Koirana and Eduardo D. Sontag and VC Dimension of Neural Networks (1998) by Eduardo D. Sontag for more details.
On the other hand, the universal approximation theorem (UAT) tells us that neural networks can approximate any continuous function. See Approximation by Superpositions of a Sigmoidal Function (1989) by G. Cybenko for more details.
Although I realize that the typical UAT only applies to continuous functions, the UAT and the results about the $\mathcal{VC}$ dimension of neural networks seem to be a little bit contradictory, but this is only if you don't know the definition of $\mathcal{VC}$ dimension and the implications of the UAT.
So, how come that neural networks approximate any continuous function, but, at the same time, they usually have a $\mathcal{VC}$ dimension that is only proportional to their number of parameters? What is the relationship between the two?

VC Dimension of Neural Networks establishes VC bounds depending on the number of weights, whereas the UAT refers to a class of neural networks in which the number of weights a particular network can have is not bounded, although it needs to be finite.
I think that we can show, from theorem 2 and the observations below theorem 3 in Approximation by Superpositions of a Sigmoidal Function, that the VC dimension of
$$S=\left\{\sum_{i=1}^N \alpha_i\sigma(y_i^T x + \theta_i) : N\in\mathbb N, \alpha_i, \theta_i \in\mathbb R, y_i\in\mathbb{R}^n \right\}$$
is infinite.
Let $\{(x_i, y_i)\}_{i=1}^k$ be a sample of arbitrary size $k\in\mathbb N$, and let us see that there is a function in $S$ which can correctly classify it, i.e., $S$ shatters $\{x_i\}_{i=1}^k$.
We note $B(x, \varepsilon) := \{ y\in\mathbb{R}^n : d(x,y) < \varepsilon \}$ (this is just standard notation to denote a ball).
First, let $\varepsilon > 0$ be such that $B(x_i, \varepsilon)\cap B(x_j, \varepsilon) = \emptyset$ every time that $i \ne j$.
Now define $D = \cup_{y_i=1} B(x_i, \varepsilon)$. Define $f_{\varepsilon}(x)$ as in the observations below theorem 3 of Cybenko's paper, and use theorem 2 to find a function $G(x)$ in $S$ that classifies correctly all points at least $\varepsilon$ away from the boundary of $D$, i.e., all points in the sample.

If you look at Cybenko's paper, you will see that the number of weights isn't really unbounded or infinite. The theorem holds for a finite but arbitrary $N$, where $N$ is the number of units in the hidden layer. The question is: in the limit, i.e. as $N \to \infty$, will the neural network have VC dimension $\infty$? Can you prove this from Cybenko's theorem?

Sorry, I can't see that it holds for a fixed $N$. In particular, proof of theorem 1 requires that $S$ is a subspace of $\mathcal C(I_n)$, which I don't think holds if $N$ is fixed. However, defining $S$ to be the set of (finite but arbitrarily large) linear combinations of that form, it is trivially a subspace.

I didn't say "fixed", I said "arbitrary".

Maybe we aren't understanding each other. Originally, I meant that the class of functions for which the UAT applies is ${ \sum_{i=1}^N \alpha_i \sigma(y_i^T x + \theta_i) : N\in \mathbb{N}, \alpha_i, \theta_i \in \mathbb{R}, y_i\in \mathbb{R}^n }$.

Ok. But then why do you say that "the UAT refers to a class of neural networks with unbounded number of weights."?

I meant that the number of weights that a function in the class can have is not bounded, i.e., it contains functions with arbitrarily large number of weights. I will edit the answer to be more clear.

Ok, but your answer still doesn't really try to clarify the relationship between the two, apart from saying that in the case of UAT the number of hidden units can be arbitrarily big and that the VC dimension depends on the number of weights. I already knew this. I wanted to understand more. I wanted to see a formal proof that relates the two. Maybe I should have been more explicit, in fact. I realise now that my question wasn't clear enough.

So, let me try to understand what you're trying to show here. You have $S$, which is a set of functions or, more precisely, the hypothesis class of a neural network. You have some labeled dataset of arbitrary size. You want to show that we can find a function in $S$ that approximates the function represented by the labeled dataset (which you denote, at the end of your answer, as $G(x)$, like in Cybenko's paper). So, 1. you define some ball (question 1: intuitively, why do you do this?).

Then 2. you let $\epsilon$ be such that the balls of different inputs do not intersect (question 2: why do you do this?). Then 3. you define the unions of all balls, i.e. $D$ (question 3: why do you do this? I suppose this is related to Cybenko's proof, which I don't recall anymore, but it would be nice if you can provide more details here about these choices in the proof, so that this answer self-contained; ideally, it would be nice if you recalled Cybenko's proofs or theorems, before proceeding to your theorem and proof).

Then 4. you say that we should define $f_\epsilon(x)$ in some way and then find $G(x)$ in $S$ as described theorem 2. Then 5. you claim that $G(x)$ can correctly classify all points $\epsilon$-away from the boundary of $D$, i.e. I suppose inside $D$, which you defined as the union of all balls.

However, how is this related to my original question "how come that neural networks approximate any continuous function, but, at the same time, they usually have a VC dimension that is only proportional to their number of parameters?"? How is your proof intuitively related to this? Is this because $k$ is arbitrary? I don't see how the VC dimension of neural networks is involved in your proof. I don't see how you're showing that the VC dimension of $S$ (the neural network) is infinite here.


text:
Can neural network help me with detecting center coordinates of particles in an image?
I have an image of some nano particles that was taken with Scanning Electron Microscope (SEM) attached here. I want to obtain center points coordinates (x,y) for each particle. Doing it by hand is very tedious. Since I just started to learn Machine Learning and got introduced to Artificial Neural Networks and kinda understand that they they are helpful with image classification, I am curious if I can use these tools to achieve my goal.
I found this article where they discuss kind similar work,,, but I am curious if you have seen anything practical or if you can give me some steps where and how to start, that's really helpful.
Any guidance is appreciated.

This is a very hard problem, you have many overlapping points with objects which aren't completely round. I'm not very knowledgeable on CV but I suspect you will find it very challenging.
I would probably say a handcrafted detection algorithm would probably be easier, something like an edge detector which fit circles to arcs and labeled the points. But it's still going to be nontrivial and maybe impossible to get it working with high accuracy.


text:
What is the best approach to build a self-learning AI chatbot?
I am a novice in AI and I like to build a chatbot to predict diseases using patient narration as input. Initially, I simply want to train my chatbot on 1 disease only. And once this initial milestone is accomplished then I want to train it on other diseases.
I want to know whether I should build and train my model first and then move towards building a chatbot or should i create a chatbot first and then train it on a disease.
Also please justify which approach is better and why?

The former: build and train a model first, and then think about the user interface.
Effectively, a chatbot is a user interface to your model. If you run it 'off-line' on input text and it works, then you have achieved your goal without the added complexity of driving a conversation (which is harder than one might think).
Also, building an 'abstract' chatbot devoid of any content is going to be harder. What should that chatbot talk about?
From my own experience (I work in conversational AI), you might not succeed in building a decent chatbot, especially as a novice. But you might be able to train a model to identify diseases from textual input. So if you do that first, you have at least got something! Especially if that is your main reason for the project in the first place.

First and foremost, thank you for your reply. I am a bit confused. Are you implying that this approach here in  this blog here is not appropriate: https://medium.com/analytics-vidhya/building-a-simple-chatbot-in-python-using-nltk-7c8c8215ac6e and neither this approach here: https://www.upgrad.com/blog/how-to-make-chatbot-in-python/?fbclid=IwAR3EwM1t5pGw36y7AyOFm0wqiYLJtGKCXS81yee9O2aRGrEPHl4xRSuj_0U ? Please elaborate and explain. Once again, thank you.

Those are just introductory articles to demonstrate the concept. You are trying to build a full-blown application, not just a toy demo.


text:
Who first coined the term "artificial general intelligence"?
Similarly to the question Who first coined the term Artificial Intelligence?, who first coined the term "artificial general intelligence"?

According to Ben Goertzel, the first person that probably used the term "artificial general intelligence" (in an article related to artificial intelligence) was Mark Avrum Gubrud in the 1997 article Nanotechnology and International Security. Here's an excerpt from the article.

By advanced artificial general intelligence, I mean AI systems that rival or surpass the human brain in complexity and speed, that can acquire, manipulate and reason with general knowledge, and that are usable in essentially any phase of industrial or military operations where a human intelligence would otherwise be needed. Such systems may be modeled on the human brain, but they do not necessarily have to be, and they do not have to be "conscious" or possess any other competence that is not strictly relevant to their application. What matters is that such systems can be used to replace human brains in tasks ranging from organizing and running a mine or a factory to piloting an airplane, analyzing intelligence data or planning a battle.

Note that the term "AGI" could have been used even before that Mark A. Gubrud's article (as Goertzel also suggested).
In any case, Ben Goertzel help to popularise this term, especially with the book Artificial General Intelligence. AGI was previously known as "strong AI", which goes back to John Searle's Chinese Room argument, although strong AI often refers to an AGI with consciousness, and the definition of AGI doesn't necessarily imply consciousness.
You can read more about the history of the term "AGI" in this Goertzel's blog post.

According to this Wikipedia article

The term "artificial general intelligence" was used as early as 1997, by Mark Gubrud. in a discussion of the implications of fully automated military production and operations. The term was re-introduced and popularized by Shane Legg and Ben Goertzel around 2002.
The research objective is much older, for example Doug Lenat's Cyc project (that began in 1984), and Allen Newell's Soar project are regarded as within the scope of AGI.


text:
How are weights for weighted x-entropy loss on imbalanced data calculated?
I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the advice of the tensorflow docs
and now I am using a weighted cross entropy loss where the weights are calculated as
weight_for_class_a = (1 / samples_for_class_a) * total_number_of_samples/number_of_classes

following the mentioned tutorial.
It works perfectly, but why is there this factor total_number_of_samples/number_of_classes?
The mentioned tutorial says this

[...] helps keep the loss to a similar magnitude.

But I don not understand why. Can someone clarify?

This comes from the fact that you want the same magnitude from the loss. Think of it this way: a non-weighted loss function actually has all its weights to 1 and so over the whole data set, samples are weighted with 1 and the sum of all weights is therefore $N$, if $N$ is the total number of samples.
Now in the case of a weighted loss, we want the weights to also sum to $N$ so that the loss's magnitude is comparable ($i = 1..C$ are your classes, $N_i$ is the number of samples for class $i$):
$$S = \sum_{i=0\ }^{C} \sum_{s_i=1}^{N_i} w_{i} = \sum_{i=0}^{i}\sum_{s_i=1}^{N_i}\frac{1}{N_i} \frac{N}{C} = \frac{N}{C} \sum_{i=0}^{C}\sum_{s_i=1}^{N_i}\frac{1}{N_i} = \frac{N}{C} \sum_{i=0}^{C}N_i\frac{1}{N_i}  = \frac{N}{C} \sum_{i=0}^{C}1 = \frac{N}{C} C = N$$


text:
Action masking for on policy algorithm like PPO
I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the probabilities of the non valid actions to 0, and renormalizing the remaining actions. However, this would be not the predicted policy anymore, and thus the agent wouldn't act on policy. Is there a better way to mask PPO agents actions, or does it simply not constitute a big problem?


text:
How would semantic segmentation work with a non convolutional neural network
Listening to lectures, convolutional neural network seems to be an improvement over a simple neural network, where for example, you take every pixel in the image, flatten it to a vector, and feed it to ANN with a couple layers. Therefore semantic segmentation should be possible to perform on a classic ANN.
I dont understand exactly how a CNN can classify each pixel in the image to do semantic segmentation.
The way semantic segmentation is explained in lectures is the output of the CNN is fed into it backwards, is that the same as with GAN?
If the output of a CNN is a value between 0 and 1 for each class, how exactly can those values be fed back through a CNN backwards to classify each pixel in the image? Through back propagation?
My understanding is that it should be possible to do the same to a regular ANN described above.
Can someone explain why or why not it wouldn't be possible, and how semantic segmentation feeds output back through network to classify each pixel
Thanks,


text:
MCTS RAVE performing badly in Board Game AI
I'm using Monte Carlo Tree Search with UCT selection to try and build an AI player for a complex multiplayer board game. My regular UCT MCTS seems to be working fine, winning with random and basic greedy players or low-depth 'paranoid' alpha-beta variant player, but I've been looking for some methods to improve it and I found RAVE.
"In RAVE, for a given game tree node N, its child nodes Ci store not only the statistics of wins in playouts started in node N but also the statistics of wins in all playouts started in node N and below it, if they contain move i (also when the move was played in the tree, between node N and a playout). This way the contents of tree nodes are influenced not only by moves played immediately in a given position but also by the same moves played later.".
I've found a lot of literature about it and it was supposed to give good results - 70%-80% win rate against basic UCT on a game of TicTacToe3D. I implemented it as a sort of benchmark, a 4x4x4 version, before trying it on my target game. But, however I tried tuning the parameters, I've been getting worse results, the win rate is at best arount 46%.
I've been calculating the node values like this:
visits[i] is a number of visits for child i of parent p that selection is performed on, wins[i] is a number of wins according to UCT, AMAFvisits and AMAFwins are assigned based on the node's source action -> updated after a finished simulation if a sourceAction (the action that changed the game state into this state) was played in the simulation by the player of the MCTS tree root node.
for (int i = 0; i < nChildren; i++) {
    if (visits[i] < 1) {
        value = Double.MAX_VALUE - rnd.nextDouble();
    }
    else if (m[i] < 1) {
        double vUCT = wins[i]/visits[i] + C*Math.sqrt(Math.log(sumVisits)/(visits[i]));
        value = vUCT;
    }
    else {
        double beta = Math.sqrt(k/(3*visits[i] + k));
        double vRAVE = (AMAFscores[i])/(m[i]) + C*Math.sqrt(Math.log(mChildren)/(m[i]));
        double vUCT = (wins[i])/(visits[i])+ C*Math.sqrt(Math.log(sumVisits)/(visits[i]));
        value = beta * vRAVE + (1 - beta) * vUCT;
        value += rnd.nextDouble() * eps;
        /*double beta = Math.sqrt(k/(3*visits[i] + k));
        double vRAVE = (AMAFscores[i])/(m[i]);
        double vUCT = (wins[i])/(visits[i]);
        value = beta * vRAVE + (1 - beta) * vUCT;
        value += C*Math.sqrt(Math.log(sumVisits)/(visits[i]));
        value += rnd.nextDouble() * eps;*/
    }
    if (maxValue <= value) {
        maxValue = value;
        index = i;
    }
}
chosen = tree.getTreeNode(children.get(index));

Here's a paint rendition of my understanding of how RAVE should work -> https://i.sstatic.net/naQXE.jpg.
Am I missing something? Is my implementation wrong? Here's the rest of the code responsible for traversing the tree in a 'rave way': https://www.paste.org/104476. The expand function on tree expands the tree for all actions, and returns a random one which then gets visited, the others are to be visited in other iterations.
I first tested the code on k = 250 like the authors of the benchmark paper https://dke.maastrichtuniversity.nl/m.winands/documents/CIG2016_RAVE.pdf suggested and on 100, 1000 and 10000 iterations, with tree depth 20 or 50. I also experimented with other k values and other params.

No implementation of RAVE in academia that I am aware of implements an exploration factor into the AMAF value (C*Math.sqrt(Math.log(mChildren)/(m[i])) into your implementation). The AMAF value is only used to bias the search towards nodes with more promising moves, and doesn't care about exploring less visited moves. That doesn't mean it isn't advantageous to do so, but I am somewhat doubtful. I would test without that term.


text:
How does a software license apply to pretrained models?
Google provides a lot of pretrained tensorflow models, but I cannot find a license.
I am interested in the tfjs-models. The code is licensed Apache-2.0, but the models are downloaded by the code, so the license of the repository probably does not apply to the models and I am not able to find anywhere a note about the license of the pretrained models.
How should I handle this, especially when I may want to distribute models derived from the pretrained Google models?

I am not sure if a pretrained machine learning model is actually protected by copyrights or not. Copyright protection exists to protect the creators of creative works from having their work "stolen", and I am not sure if training a ML model is an act of creativity.
That said, assuming that a pretrained ML model is actually protected by copyrights, then it is more likely that the model is a derived work of the data set used for training than that it is a derived work of the software that uses the model.
The software reads the model in as data, assuming that the software can be used with many differently trained models. In that case, the software and the model are considered completely independent works in the same way that MS Word and the documents you write with it are independent works for copyright.
Thus, if you want to publish the trained model with a license, I would recommend to use the BSD license that was also used for the training set.

My problem is, that I want to use and distribute a model pretrained by Google as part of a software. I both need to be sure that I am allowed to do this and I need to know how to license the model to my users.

That a model is loaded as data is true, but I still have two problems. First I need to convert it (from tfjs to tensorflow) and second because Google only hosts the tfjs model I would need to distribute the converted model. In addition I would probably distribute it bundled with an application that is licensed with a copyleft license and then I need to be sure that I am allowed to grant the rights from the license.


text:
How can I use GPT-2 to modify seed text of one form into a different form (LENGTH INVARIANT) whilst retaining meaning?
I am currently starting a research project whereby I am trying to convert text of one form into another.
i.e. If I were to write a seed sentance of the form "Scientists have finally achieved the ability to induce dreams of electric sheep in the minds of anaesthetized robots" I would like GPT-2 to convert this into "Robots have finally had dreams of electric sheep whilst being anaesthetized by scientists." or some coherent permutation of the underlying structure whereby the main logic of the text is conveyed albeit roughly.
The current open source implementation of GPT-2 seeks to predict the next word, i.e. the seed text is given "Scientist have finally" and the generated text would be " started being paid enough!"
My first presumption was to use some form of GAN, however it became quickly evident that:

Recent work has shown
  that when both quality and diversity is considered, GAN-generated text
  is substantially worse than language model generations (Caccia et al.,
  2018; Tevet et al., 2018; Semeniuta et al., 2018).

How could I most effectively achieve this? Thanks.