text:
Which algorithm should I choose for lead scoring
I am a c# senior developer and I got a task to try and predict the potential in each new client, or maybe the worth of each customer.
I don't have experience with machine learning, but I played with accord-framework.net and got some nice results on simple task.
My data model for training is:
GeoLocation, // the country of ip when registed. iso code string
Age, // number
DateRegistered, //date time
Email, //string can be broken to vendors as catergorial (gmail, yahoo, microsoft and such)
EmailValidated, //is the email really exists. bool
PhoneNumber, //string
PhoneNumberValidated, // is the phone number really exists
CampaignName, //string (may be categirial)
UserAgent, //string should I make it categorial? (has info about browser, device, verndor, operation system and such, long string)
LandedOnPage, //string first url the customer entered from
RegisteredFromPage, //string url of the page that the user registered from
RefererUrl, //string url the client came to our site from,
NumberOfPurchases, //the amount of times the customer puschase something on our site
CustomerValueUsd, //the total amount of USD the customer spent in our site

The output shoud be CustomerValueUsd
I have a lot of data in the history, so I can back test it.
My questions:

Does it make sense to do this task even though I don't have an experience with machine learning? How complicated is this task considering I'm using a well known framework?
Assuming that I'm taking the task, which algorithm should I choose to perform this kind of task?
How should I build the training data? see my comments, do you think my comments are ok to start with? or maybe I can break the data directly?

I suggest you take an intro to ML course. Your coding background will help a lot understanding how to turn the concepts into code. You are missing the concepts themselves, but they can be learned. One suggestion: https://www.coursera.org/learn/machine-learning

ML is a wide and deep topic to cover.
What you need is "classification" to predict an outcome. At first, you need to convert everything to numbers (check word2vec). Then, pre-process the data (Check what is a normalization for example). Train a classifier by using any ML techniques (i.e. linear classifier, SVM, deep learning, neural classifier). Then you can use that regression function to predict further values. If you need much more detailed way to do that, it is better to buy a 'thick' book that explains all other options.

Does it make sense to do this task even though I don't have an experience with machine learning? How complicated is this task considering I'm using a well known framework?

I think it does make sense, and using an established framework would get you up and running quickly.

Assuming that I'm taking the task, which algorithm should I choose to perform this kind of task?

This is a regression problem, so I would recommend that you don't treat it as a classification task (unless you want to have a binary output like "profitable" / "non-profitable"). In essence, you are trying to identify the correlation between your inputs (previous purchases, location, etc.) and a certain metric (dollar value of the customer). Neural networks are very good at that (Accord seems to support neural networks, so you should be able to use that; TensorFlow with Keras as the interface or Caffe might be other options to consider).

How should I build the training data? see my comments, do you think my comments are ok to start with? or maybe I can break the data directly?

In most cases, you'd want to normalise your data before feeding it to the algorithm (this is particularly important for neural networks). The other thing you should do is consider what features would be relevant to the task. For example, since you want the customer value as the predicted output, the customer's phone number and email address are most probably irrelevant, but number of previous purchases, age and geolocation might be very relevant. Maybe you have other features in your database - total dollar amount of previous purchases, frequency of purchase, number of returned items or refund requests, etc. Remember to keep your model in check by splitting your data into training, validation and test sets (as a rule of thumb, 70% training / 10% validation / 20% test, but that depends on how much data you have).


text:
Relative compute time for each type of layer in a neural network

Hello,
I would like to know whether this picture from the paper: Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability valid?
Questions:
1) Does InnerProduct (Fully connected) layer actually take more time to compute in a neural network than Convolution?

2) Is assessing GLOPs/time a good way of estimating performance of different types of layers in a neural network on any hardware? (Conv, FC etc.)

3) Does anyone know where I can find GFLOPs vs compute time for different types of layers across GPUs/CPUs? (I know DeepBench, any other suggestions would be great too)

Welcome to SE:AI!  (Thanks for cutting out the graphics and including them in the post:)

I think what you suggest requires an end to end revamp of the hardware architectures we have developed. I find it more feasible personally, to emulate these processes. Once we know the mechanism, we can make it an equation. Just a thought. But I agree with you, backpropogation and batching dont seem like 'intelligence'.

The graph seems to show the relative compute time of each component of the learning process, independent of the layer, using the typical meaning of layer in the context of artificial networks.
The data is not unbelievable, in that one would expect a matrix-vector multiplication of an entire layer output would take more time than a matrix-vector multiplication of the size of a convolution kernel. Still, these representations are for a specific test condition and measurement methodology.
Most of the theoretical and practical limits of parallel processing are heavily biased by seventy years of sequential processing as the foundation of computer science study and practice. If an understanding of parallel processing is desired, the von Neumann architecture is not the place to look. At the time of that development, computers could either be made of wood, steel, vacuum tubes, or relays. Developing a computer with 65536 parallel data paths was unrealistic to the extreme. The invention of algorithms was a workaround to funnel all processing through a single CPU. We've been backing out of that mind set for the last thirty years, beginning with FPUs, GPUs, multiple buses, computer clusters, and multiple cores. More parallelism in VLSI form is coming, and soon.
The demonstration of parallel processing comes, as many things do, from biology. Look at farming and grain growth under the sun. Look at visual circuitry coming from the eyes of thousands of species. Consider the orbits of millions of planets and their satellites. The artificial parallelism to examine is the transportation systems and the Internet. Parallel processing is efficient when the process was designed as a parallel process rather than trying to make parallel that which was designed as serial. For instance, the human ear does the physical equivalent of a real time windowing FFT without performing a single multiplication operation.
The bottleneck of the floating point multiplication of an attenuation matrix with a vector (parameterizing the mix of signals from one layer to the next) is a key to why NVidia has done well with re-purposing rendering hardware for artificial network training. It is also one of the many reasons this question is interesting: If digital values are mere estimates, why not return to analog for AI?.
When surfing for GFLOPs vs compute time for different types of layers across GPUs and CPUs, using the term 'operation' instead of 'layer' in the search terms may help. Even though elementary operations are lower level than that of the comparison desired, the term layer is delegated strongly to the higher level meaning in the machine learning literature. Reliable data in this space is thin because much of the speed optimization has already been researched and developed, so whatever the visualization shows is useless unless you work on VLSI development at Intel, Samsung, NVidia, Qualcom, TI, or one of the other silicon giants.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Do Le et al. (2012) train all three autoencoder layers at a time, or just one?
Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).
Their network is built with three autoregressive layers, and six pooling and normalization layers.
In the paper they state,

Optimization: All parameters in our model were
trained jointly with the objective being the sum of the
objectives of the three layers.

Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?
I asked a follow-on question on the advantage of training all layer simultaneously.

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.
This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use uniform weights.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there a non-geometric formulation of machine learning
Note: I am not asking what is geometric formulation of machine learning. I am searching alternative formulations that are not reduced to machine learning on manifolds
Machine learning is usually formulated as a geometric problem (e.g. [1], [2]). The data are represented as points in some manifold where coordinates are features, and the task to find another geometric object related to the data. This object can be a line going through the data in case of regression or a hyper-surface separating subsets of the data in case of classification.
I couldn't find a formulation of machine learning that is not reduced to such geometry.
To me, it looks like geometric formulation is not fundamental. The space in which data are represented is not unique. It depends on the feature set created for the data. For the same data many (potentially infinite) different feature sets can be created, so the same machine learning problem can be represented in different spaces of different dimensions.
For example, I work in chemistry and material science. There is a lot of machine learning here, mapping compounds to points in so called 'chemical space' where most predictions fail. I keep asking in seminars and conferences what is this chemical space, its structure, dimension, topology. It looks like there is no definite answer except 'it depends on the descriptor (feature set, a vector of numbers, corresponding to a compound, etc, graphs are also there)'.
There are thousands of papers about different descriptors, and even multi-volume books devoted solely to descriptors to choose from. According to some descriptors, two molecules are very similar, yet one is benign, another is a poison. According to another descriptor, they're different. That's why I say 'not fundamental' - any feature set says very little about chemical compounds they are supposed to represent.
Another issue is that geometric formulation requires the notion of distance. Different types of distances can be used which gives different, even contradicting predictions. It is often unknown whether distance and other geometric constructs can make sense at all.
Is the distance between graphene and amorphous silicon less or greater than the distance between boron nitride and amorphous carbon? Is there triangle inequality in that space? What is greater, the distance between copper oxide and molybdenum dichalcogenide plus the distance between molybdenum dichalcogenide and silicon carbide or the distance between copper oxide and silicon carbide? I think, I can construct features of these substances and distances between them that the answer can be anything you want.
One way to solve this in chemistry is to continue searching for a better feature set. But then there are materials like VO2 and TiO2 whose electron densities are almost indistinguishable, so any set of features would map them into points that are next to each other, yet one of them is a conductor, while another is an insulator.
So, my question is: Maybe instead of searching for a better feature set that maps data to points in space and designing a distance function that measures similarities between data, I could use some other machine learning paradigm that does not reduce a machine learning problem to a geometric problem?
Is mapping to geometric objects, and getting inference based on their similarity is the only possible paradigm in machine learning? Are there formulations of machine learning alternative to the geometric one?

This question is similar to: What is geometric deep learning?. If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.

@nbro That question is totally different. That question asks "What is geometric deep learning". I am not asking what is geometric formulation, I know what it is. Moreover, in my question I reiterate what it is. I am asking what are alternative formulations if any. The question you link does not discuss alternative formulations

I know you're not asking about a geometric formulation of deep learning. You're asking the opposite. The linked question, however, tells you that geometric deep learning deals with graphs and graphs are non-Euclidean data, which means that many algorithms and models that assume an Euclidean space don't directly work on graphs. The name geometric DL is a bit misleading. (The comment above was generated automatically, btw, when you vote to close a post as a duplicate.)

Your post makes some arguable assumptions, like "geometric formulation is not fundamental." and then you say that the spaces are not unique. I don't see the relation between the two. You can have spaces like $R$, $R^2$, $R^3$, etc., and what's the problem with that? Why would you need to have only one unique space for what you call "geometric formulation" to be fundamental?

Firstly you claim that geometric formulation of machine learning (M)L is not fundamental and the same ML problem can be represented in different spaces of different dimensions. However, the essence of contemporary deep learning is built from two simple principles known as feature learning and local gradient-descent. Many
high-dimensional learning tasks previously thought to be beyond reach, such as computer vision, playing Go, or protein folding, are in fact feasible after adapted learned features capture the appropriate notion of regularity for each task, such as semantic similarity (GPT and BERT large variant pretrained token embeddings have different dimensions and representations, but both share regularity of semantic similarity) and group symmetry (translational and rotational equivariance & invariance of CNNs).
Similarly for the geometric feature of distance, each ML task will have inductive bias for such feature if it's really relevant with domain knowledge. Therefore though features can be represented in many seemingly different ways, for most real world ML problems their predicted or generated data have many intrinsic regularities as well to ensure stability and applicability.
Besides geometric formulation, ML has many other formulations such as probabilistic (Bayesian) and PAC formulation where sample data are drawn from some unknown probability distribution and ML becomes the task of inferring or approximating distributions rather than finding geometric shapes such as Normalizing Flows or the task of updating a prior (latent) distribution to a posterior using Bayes' theorem such as VAEs.
For reinforcement learning problems, sequential decision-theoretic formulation is the standard, and in fact classification and regression ML problems can be viewed as one-shot decision problems as well, not just as geometric lines or hyper-surfaces separating or going through data which is hardly applicable for cost sensitive loss with imbalanced data, for instance. Traditional knowledge base learning problems are usually formulated in purely symbolic rules and logic such as Inductive Logic Programming (ILP) where symbolic rules are learned from samples. Finally there's information-theoretic Minimum Description Length (MDL) formulation of some ML problems where the goal is to find the model that compresses the observed data most effectively.


text:
Why does PCA of the vertices of a hexagon result in principal components of equal length?
I do PCA on the data points placed in the corners of a hexagon, and get the following principal components:

The PCA variance is $0.6$ and is the same for each component. Why is that? Shouldn't it be greater in the horizontal direction than in the vertical direction? The data is between $-1$ and $1$ in the $x$-direction but only between $-\sqrt{3}/2$ and $\sqrt{3}/2$ in the $y$-direction. Why PCA results in the equal length components?
The length of each vector in the picture is the twice the square root of the variance.
UPDATE: added more points, the variances changed to $0.477$ but still they are equal.

UPDATE 2: Added even more points, the variances changed to $0.44$ but still they are equal.

Somewhat related

@RodrigodeAzevedo Yes, added more points, and updated my question. $AA^T$ remains diagonal. I wonder, what is the minimal symmetry at which PCA starts to fail to find the direction of largest variance.

Assuming that the $6$ vertices of the hexagon are on the unit circle,
>>> from sympy import *
>>> A = Matrix([[ 1, Rational(1,2),-Rational(1,2), -1, -Rational(1,2), Rational(1,2)],
                [ 0,     sqrt(3)/2,     sqrt(3)/2,  0,     -sqrt(3)/2,    -sqrt(3)/2]])
>>> A * A.T
Matrix([[3, 0],
        [0, 3]])

Since ${\bf A} {\bf A}^\top - 3 \, {\bf I}_2 = {\bf O}_2$, any two orthogonal directions could be the principal components.

The 6 vertices of the hexagon are on the unit circle but the points on the second picture are not on the unit circle, yet for them $AA^T$ is also diagonal and is Matrix([[5.25, 0],  [0, 5.25]]) with equal length diagonal elements. I wanted to use PCA to find direction of the largest variation of the data. Does it mean that PCA is not always find these directions even if data are not spherically symmetric? Also thanks for editing my question.

@VladislavGladkikh Rotate the point cloud by certain angles, and you obtain a point cloud that looks just like the one you started with, right? In my opinion, in these two cases, there's too much symmetry for a single "direction of the largest variation" to exist. At least in classical PCA. What if you use other norms?


text:
Why there are only three machine learning paradigms: supervised, unsupervised, reinforcement?
I read in books, blogs, and articles that there are three learning paradigms: supervised, unsupervised, and reinforcement.
However, I have never found a proof that this list is exhaustive. Can it be proven that there cannot be other types of learning?
The notions 'supervised' and 'unsupervised' could be complementary to each other, like if $S$ is supervised and $\bar S$ is unsupervised, then $S\cap\bar S=\emptyset$ and $S\cup\bar S=U$ where $U$ is the set of all learning paradigms. However, this would be true only if $S$ and $\bar S$ were mutually exclusive and exhaustive. I don't believe that this is the case. First, there is reinforcement learning which is neither $S$ nor $\bar S$. Then, there is nothing that prevents learning to be not fully supervised or not fully unsupervised, e.g. labels can be not well defined, the teacher may teach only partially, but the system can still improve i.e. learn. And if there is reinforcement learning with is neither supervised nor unsupervised, can there be some other learning which is neither supervised, nor unsupervised, nor reinforcement?
Is it correct to say that in principle there can be many different learning paradigms?

Who says that? Who says there are only three paradigms?

You can formulate RL and unsupervised learning as "some sort of supervised learning"... in the case of UL you have that the target is an handcrafted task, it being similarity learning, next token prediction and what not, and in case of RL, the reward signal is by definition a form of supervision
The reason why we distinguish them it's because they aim at solving very different problems, but this is subjective
There are other types of learning, here are few of them:

semi-supervised learning
weakly-supervised learning
meta learning
self-supervised learning
transfer learning
anomaly detection

as per the others, you can lead these back to SL UL or RL, becuase at the end of the day, everything can be thought as an Energy model of some sort...

I'd would not say that anomaly detection is a type of learning. In fact, you can use unsupervised and supervised methods for anomaly detection. I'd rather say that it's a general task.

@nbro anomaly detection includes dataset which contains both valid and unvalid points, thus it's not just density estimation (indeed, it's also known as outlier detection)


text:
Confidence Maps and Non-Linearity
I am currently trying to improve a CNN architecture that was proposed for generating depth images.
The architecture was originally proposed for autonomous driving and it looks like following :

The idea behind this architecture is to improve the accuracy of depth images by adding confidence maps
to the outputs of different sensors to govern their influence on the result for each pixel. Input is an
RGB image and its corresponding LIDAR data, output is a depth image with the same dimensions.
For example, RGB features are best at discriminating global information such as sky, land, body of water, etc.
while LIDAR features are better at capturing fine-grained, local information. So to decide which features will
have more influence at the final regression result of which pixel, scientists proposed a confidence-guided
architecture where confidence weights for each map are learned during training.
Judging by their test results and how successful their paper was, their idea worked out pretty well for their
problem domain. That's why I would like to employ the idea of confidence weights in my own domain, where I have
multiple sources of features too. I have implemented the same architecture and got promising results: Accuracy
have been improved, but not enough to compete with SOTA.
However, I believe that the architecture above can be improved for my needs. The diversity of the scene structure
in my domain adds some amount of complexity to the image generation problem: Opposed to depth image generation for
autonomous driving where scene structure is somewhat restricted (there is sky, there is road, there are sideways etc),
the images that I need to analyze are sometimes taken from handheld cameras, some other images are areal views.
Here is a typical example of their scenario and some examples related to my domain, respectively.

This means my CNN needs to learn confidence weights for regions in fundamentally different scene setups. Top of
the image can be sky, but can also be populated by people. Bottom of the image can be sea, or can be road. This
brings me to the understanding that there is a non-linear relation between the position of the segments and confidence
weights in my case; and I need to modify the CNN architecture by introducing some additional non-linearity to learn the
confidences for different CNN columns in each situation correctly.
TLDR; I want to improve the CNN architecture above by introducing additional non-linearity, but I do not know how to do it.
I have tried adding another layer to confidence weights (Extended the architecture by duplicating the same weights and activating
with ReLU), but it has decreased the accuracy of the resulting model. Using the confidence weights as-is increases the accuracy, but
not as much as I need.


text:
What is the difference between a non-stationary policy and a state that stores time?
This question is related to What does "stationary" mean in the context of reinforcement learning?, but I have a more specific question to clarify the difference between a non-stationary policy and a state that includes time.
My understanding is that, in general, a non-stationary policy is a policy that doesn't change. My first (probably incorrect) interpretation of that was that it meant that the state shouldn't contain time. For example, in the case of game, we could encode time as the current turn, which increases every time the agent takes an action. However, I think even if we include the turn in the state, the policy is still non-stationary so long as sending the same state (including turn) to the policy produces the same action (in case of a deterministic policy) or the same probability distribution (stochastic policy).
I believe the notion of stationarity assumes an additional implicit background state that counts the number of times we have evaluated the policy, so a more precise way to think about a policy (I'll use a deterministic policy for simplicity) would be:
$$ \pi : \mathbb{N} \times S \rightarrow \mathbb{N} \times A $$
$$ \pi : (i, s_t) \rightarrow (i + 1, s_{t+1}) $$
instead of $\pi : S \rightarrow A$.
So, here is the question: Is it true that a stationary policy must satisfy this condition?
$$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$
In other words, the policy must output the same result no matter when we evaluate it (either the ith or jth time). Even if the state $S$ contains a counter of the turn, the policy would still be non-stationary because for the same state (including turn), no matter how many times you evaluate it, it will return the same thing. Correct?
As a final note, I want to contrast the difference between a state that includes time, with the background state I called $i$ in my definition of $\pi$. For example, when we run an episode of 3 steps, the state $S$ will contain 0, 1, 2, and the background counter of number of the policy $i$ will also be set to 2. Once we reset the environment to evaluate the policy again, the turn, which we store in the state, will go back to 0, but the background number of evaluations won't reset and it will be 3. My understanding is that in this reset is when we could see the non-stationarity of the policy in action. If we get a different result here it's a non-stationary policy, and if we get the same result it's a stationary policy, and such property is independent of whether or not we include the turn in the state. Correct?

So, here's is the question: Is it true that a non-stationary policy must satisfy this condition?
$$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$

With your custom notation (which certainly isn't common, but seems reasonable)... I assume you meant to say that a stationary policy must satisfy that condition, rather than that a non-stationary policy must satisfy that condition. In that case, yeah, that seems correct to me. A stationary policy would satisfy that condition, and a non-stationary one wouldn't.

Wrapping back to the more usual notation, where $\pi(S, A)$ denotes a probability of selecting $A$ in $S$ (which also still covers the case of a deterministic policy, which would simply assign a probability of $1$ to a single action, and $0$ to all others)... I think it's still interesting to consider the case where we decide to "bake" a time-step counter into the state representation.
With this notation, for two different time steps $t$ and $t'$, such that $t \neq t'$, I'd say that a policy $\pi$ is stationary if and only if:
$$\pi(S_t, A_t) = \pi(S_{t'}, A_{t'}) \text{ if } S_t = S_{t'} \wedge A_t = A_{t'}.$$
Note that if we decide to include $t$ in the state-representation, the case that $S_t = S_{t'}$ with $t \neq t'$ will actually never hold within the same episode, states at different time steps will always automatically be different from each other if the time step is one of the "features" encoded in the state. So, within a single episode a policy with a "time-aware" state representation will always automatically be stationary because there cannot be any repeating states. Of course, if you start looking across multiple different episodes, this can change; this is what you're doing when you write:

Once we reset the environment to evaluate the policy again, the turn, which we store in the state, will go back to 0, but the background number of evaluations won't reset and it will be 3.

If you chose to also embed that "episode counter" into the state representation, you would also no longer have any state repetitions at all anymore across different episodes (I don't think doing this would ever be a good idea though).

you're right, that was a typo in the first quote, I meant "is it true that a stationary policy must satisfy this condition?" I have edited the question.

I think you're overthinking it. I've never seen a formalisation of the concept of "stationary policy" (apart from yours). However, in general, "stationary" means that something does not change (over time). In the context of reinforcement learning, you can interpret it in such a way that it is consistent with the context where you find this expression, unless you decide to formalise this expression (like you've tried in this question).
I think it might be useful to differentiate between the learning phase and the "inference" (or "behaviour") phase, even though these two might interleave in the RL context, that is, you might be using a policy (to behave in the real-world) even though you're still attempting to find the best policy (and I am not just referring to on-policy algorithms, but, in general, you might be using a policy to behave in the real-world which is sub-optimal while your learning algorithm, like Q-learning, is attempting to find the best policy).
During the learning phase, the policy will keep changing over time (anyway), because you still haven't found the optimal policy. So, you could call the policy derived from the $Q$ values, during the $Q$-learning algorithm, a non-stationary policy, because it keeps changing (because the $Q$ values also keep changing). However, it is often the case that these are considered different policies associated with different approximations of the $Q$ function.
You could also call a policy that changes in response to changes of the (dynamics of the) environment a non-stationary policy. In this case, you would call such an environment a non-stationary environment (because e.g. its transition model might keep changing over time).
The problem that you describe when you compare the state $i$ with states that encode time only arises because of your definition of state $i$. You can also reset $i$ when you reset the environment.


text:
Do good approximations produce good gradients?
Let’s say I have a neural net doing classification and I’m doing stochastic gradient descent to train it. If I know that my current approximation is a decent approximation, can I conclude that my gradient is a decent approximation of the gradient of the true classifier everywhere?
Specifically, suppose that I have a true loss function, $f$, and an estimation of it, $f_k$. Is it the case that there exists a $c$ (dependent on $f_k$) such that for all $x$ and $\epsilon > 0$ if $|f(x)-f_k(x)|<\epsilon$ then $|\nabla f(x) - \nabla f_k(x)|<c\epsilon$? This isn’t true for general functions, but it may be true for neural nets. If this exact statement isn’t true, is there something along these lines that is? What if we place some restrictions on the NN?
The goal I have in mind is that I’m trying to figure out how to calculate how long I can use a particular sample to estimate the gradient without the error getting too bad. If I am in a context where resampling is costly, it may be worth reusing the same sample many times as long as I’m not making my error too large. My long-term goal is to come up with a bound on how much error I have if I use the same sample $k$ times, which doesn’t seem to be something in the literature as far as I’ve found.

i wonder - would regular gradient descent avoid this problem?

In general $|f(x) - f_k(x)| \leq \epsilon$ doesn't ensure $|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$. And for neural networks there is no reason to believe it will happen either.
You can also look at Sobolev Training Paper (https://arxiv.org/abs/1706.04859). In particular, note that Sobolev training was better than critic training, which indirectly may indicate approximating function may not be the same as approximating gradient and function. In Sobolev training, the network is trained to match gradient and function whereas in critic training network is trained to match function. They produce quite different results which might give us some hints about the above problem.
In general, if two functions are arbitrary close, they might not be close in gradients.
Edit:
(Trying to come up with a negative example)
Consider $f(x) = g(x) + \epsilon \sin (\frac {kx} {\epsilon}) $. $g(x)$ is some neural network. Now, we train some another neural network $h(x)$ to fit $f(x)$ and after training we get $h(x) = g(x)$ ($h(x)$ and $g(x)$ have same weights precisely). However, $\nabla f_x =\nabla g_x + k\cos (\frac {kx} {\epsilon})$ is not arbitarariy close $\nabla g_x$.
I hope this example is enough to prove that a neural network that nicely approximates the function may not nicely approximate the gradients and no such result can be proved mathematically rigorously. However, considering the paper in the discussion, you might think for practical purposes it works. However, if you have both function and grad information available that is expected to work better.

great answer! i wonder - would regular gradient descent avoid this problem?


text:
Which loss function is the brain optimizing in order to learn advanced visual skills without expert/human supervision?
In computer vision is very common to use supervised tasks, where datasets have to be manually annotated by humans. Some examples are object classification (class labels), detection (bounding boxes) and segmentation (pixel-level masks). These datasets are essentially pairs of inputs-outputs which are used to train Convolutional Neural Networks to learn the mapping from inputs to outputs, via gradient descent optimization. But animals don't need anybody to show them bounding boxes or masks on top of things in order for them to learn to detect objects and make sense of the visual world around them. This leads me to think that brains must be performing some sort of self-supervision to train themselves to see.
What does current research say about the learning paradigm used by brains to achieve such an outstanding level of visual competence? Which tasks do brains use to train themselves to be so good at processing visual information and making sense of the visual world around them? Or said in other words: how does the brain manage to train its neural networks without having access to manually annotated datasets like ImageNet, COCO, etc. (i.e. what does the brain use as ground truth, what is the loss function the brain is optimizing)? Finally, can we apply these insights in computer vision?

Update: I posted a related question on Psychology & Neuroscience StackExchange, which I think complements the question I posted here: check it out

Cross-posted: https://psychology.stackexchange.com/q/21965/11209, https://ai.stackexchange.com/q/11666/1794, https://datascience.stackexchange.com/q/48645/8560.  Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted.

Your brain is not a computer (see this or this). Your brain doesn't optimize a loss function.
Also, even if I reluctantly accept this analogy, I feel like the visual system has plenty of "external supervision" from the other senses, etc.

@sfmiller940 if the brain is not optimizing a loss, then what mechanism does the brain use to learn new things? Regarding other senses, can you elaborate more on how they exert supervision on the visual system?

@PabloMessina I don't think anybody knows exactly how the brain learns. But I'm not an expert, so I imagine that neuroscientists (or similar) would know much more about what is and isn't known. As far as supervision, let's consider the senses of touch and hearing. If we see a ball, we don't necessarily know if it's a discrete object but, if we can touch it, then we get additional information about it's shape and relation to other objects. Similarly we can hear sounds and voices that can instruct us directly or indirectly. Other senses seem a form of supervision that guides visual learning.

@sfmiller940 "Other senses seem a form of supervision that guides visual learning", probably, but how?

@PabloMessina That's a question more for neuroscientists than data scientists.

I think you are slightly confusing 2 problems. 1 being classification of meta visual elements and the other being the visual system itself.
Our visual system, when it comes to processing information, has had billions of years of iteration(training), so that at birth(and before), we are already tuned for the processing of visual stimuli, as well as have the mechanisms to decipher objects in our spatial field of view.
These two papers(L1, L2), have a great deal of information about the evolution of our visual system and its processing. The second speculates on the connection of said evolution and the construction of "seeing systems" very interesting.
For further inquiry on this in particular, check out David Marr. He was probably the most influential early computer vision mind. He still is mentioned in many top-down AGI and computer vision research projects to this day.

Do you mean babies are born right off the bat already identifying people, dogs, cats, mountains, water, intuitive physics, predicting object trajectories, etc.? Or do their brains learn this over time instead? If your answer to the last question is yes, then how do brains manage to find training examples to train their neural networks without having access to manually annotated datasets like ImageNet, COCO, etc.?

@PabloMessina while tests on infants are difficult for obvious reasons, yes. We're born with ingrained classification functionality and can identify spatial forms after and likely before we are out of the womb. Physics is evolutionary as well(at least for examples relevant to what our ancestors would of faced, lions, snakes, etc.). Some things get learned over time as well, like object permanence. Trying to compare our visual system to convolutional neural networks is a very rough analogy and usually is only used for instructive reasons. Our visual system is more likely than not convolutional.

Likely not convolutional*

Hey @hisairnessag3, your answer motivated me to ask this question in Psychology & Neuroscience, feel free to answer it if you want


text:
Training an AI to play Starcraft 2 with superhuman level of performance?
I'm interested in working on challenging AI problems, and after reading this article (https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/) by DeepMind and Blizzard, I think that developing a robust AI capable of learning to play Starcraft 2 with superhuman level of performance (without prior knowledge or human hard-coded heuristics) would imply a huge breakthrough in AI research.
Sure I know this is an extremely challenging problem, and by no means I pretend to be the one solving it, but I think it's a challenge worth taking on nonetheless because the complexity of the decision making required is much closer to the real world and so this forces you to come up with much more robust, generalizable AI algorithms that could potentially be applied to other domains.
For instance, an AI that plays Starcraft 2 would have to be able to watch the screen, identify objects, positions, identify units moving and their trajectories, update its current knowledge of the world, make predictions, make decisions, have short term and long term goals, listen to sounds (because the game includes sounds), understand natural language (to read and understand text descriptions appearing in the screen as well), it should probably be endowed also with some sort of attention mechanism to be able to pay attention to certain regions of interest of the screen, etc. So it becomes obvious that at least one would need to know about Computer Vision, Object Recognition, Knowledge Bases, Short Term / Long Term Planning, Audio Recognition, Natural Language Processing, Visual Attention Models, etc. And obviously it would not be enough to just study each area independently, it would also be necessary to come up with ways to integrate everything into a single system.
So, does anybody know good resources with content relevant to this problem? I would appreciate any suggestions of papers, books, blogs, whatever useful resource out there (ideally state-of-the-art) which would be helpful for somebody interested in this problem.
Thanks in advance.

It's not clear to me whether the question was about ways to intergrate ML, CV, knowledge bases, etc., in order to build some kind of general-purpose AI to play Starcraft, or if the question was about resources to get you started with solving Starcraft. It seems that people have tried to answer the second question, but it seems to me that you were interested in the first.

StarCraft II is a real time strategy game that combines fast paced micro actions with the need for high level planning and execution. StarCraft II being a popular game with millions of users it proceeds that defeating top players becomes a meaningful and measurable long term objective in AI research.
Computer games provide a compelling solution to the issue of evaluating and comparing different learning and planning approaches on standardized tasks. They are an important source of challenges for research in AI.
Game playing AI agents i.e. deepmind's Atarinet and DQN alongside Open AI's Dota 2 bot represent the first demonstration of a General Purpose Agent that is able to continually adapt behavior without any human intervention, a major technical step forward in the quest for general AI (Source deepmind blog).
Computer games offer numerous advantages in AI research i.e:

They have clear objective measures of success.
Computer games typically output rich streams of observational data, which are ideal inputs for deep networks.
They are externally defined to be difficult and interesting for a human to play. Therefore they provide an excellent test for intelligence.
Games are designed to be able to run anywhere with the same interface and game dynamics. This enables running many simulations in parallel. Sharing and updating the same table throughout training.
In some cases pools of superb human players exist, making it possible to benchmark against highly skilled humans.

The Starcraft challenge for reinforcement learning, introduces a taxing set of problems because it is a multi-agent problem with multiple players interacting. There is imperfect information due to a partially observed map, it has a large state space, it has delayed credit assignment requiring long term strategies.
Tools
The SC2LE Environment
DeepMind and Blizard games have collaborated to release the SC2LE, which exposes StarCraft II as a research environment.
The SC2LE consists of three sub-components.

A Linux Starcraft II binary.
StarCraft II API which allows programmatic control of StarCraft II. The API can be used to start the game, get observations, take actions and review replays.
PySC25 which is an open source environment written in Python. It includes some mini-games and visualization tools

Open source Open AI RL environments
Universe - Universe is a software platform by Open AI for measuring and training an AI's general intelligence across games, websites and other applications.
Gym - Open AI Gym is a toolkit for developing and comparing reinforcement learning algorithms. It makes no assumptions about the structure of your agent and is compatible with any numerical computation library such as Tensorflow or Theano.
Supervised Classification Approach
Consider this, we could decide to screen capture game sessions from expert players and use it as input to a model. The output could be the direction in which the AI agent could move. This would be a supervised classification approach.
However, this is not an elegant solution because we are training a model not on a static dataset but a dynamic one (game environment). The training data from a game environment is stochastic/continuous meaning any number of events can occur.
Furthermore, humans learn most effectively by interacting with the environment. Not by watching others interact with the environment.
Markov Decision Process
Markov Decision Processes (MDPs) provide a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker i.e. game environments.
Reinforcement Learning with Deep Q-Learning
Q-Learning is a strategy that has been proven to find an optimal action selection policy for any Markov Decision Process (MDP). In Q-Learning we choose an action that maximizes future reward. The further in the future we go, the further the rewards can diverge, we resolve this by adding a discount in future rewards.

Unlike policy gradient methods, which attempt to learn functions which directly map an observation to an action, Q-Learning attempts to learn the value of being in a given state, and taking a specific action there. (Arthur J 2016)

The formula for Q-Learning is:

Where:
R = Reward
s = State
a = Action
Experience during learning is based on (s, a) pairs
One has an array Q and uses experience to update it directly
(Source wikipedia https://en.wikipedia.org/wiki/Markov_decision_process)

One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment.
For further reference, I recommend you look at Siraj Ravals tutorial on Deep Q-Learning
https://www.youtube.com/watch?v=79pmNdyxEGo and source code for the same available here https://github.com/llSourcell/deep_q_learning
Additionally I recommend the following references for more information on computer game playing AI agents.
StarCraft II: A New Challenge for Reinforcement Learning https://arxiv.org/abs/1708.04782
Playing Atari with Deep Reinforcement Learning https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Human-level control through deep reinforcement learning  https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf

Good to have a Starcraft question.  The game has been the subject of growing interest re: AI in recent years, possibly due to its status as king of RTS, which has led to a professional player class no doubt useful for evaluation of AI strength.
Because, last time I checked, Humans Are Still Better Than AI at StarCraft—for Now...
It's highly likely there will soon be an algorithm that can beat humans at the game, probably an extension of DeepMind's Alphas, so the clock is ticking...
I'm personally interested in classical, generalized approaches to strategy game AI, which is archaic from the standpoint of pure strength, but interesting from a game solving perspective.  (Motivations here are from a game product perspective, under the assumption that most humans don't like losing every game with no possibility of ever winning;) The way I'd personally go about it would be to start thinking about how to abstract the game, generalize the map, unit densities, etc., and try to determine though AI testing if there are sound axioms.
For superhuman strength, Deep Learning is clearly the way to go.  The recent results in Go and Chess are just the beginning of the validation of the technique.
Speaking generally, the way I see it, you have a few ways to go: (1) bootstrap existing NN and tweak until it can beat you every time.  But I'm sure many people are already doing this; (2) try to reinvent the wheel and write your own better NN from the ground up.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?
Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:

Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) > sim(Q,Y).

However, examples of trickier cases would be:

Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) > sim(Q,X) (assuming the car shapes are more or less the same).
Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) > sim(Q,Y).

Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?

Are you commenting in the wrong question by any chance? I don't understand why you are talking about time, simultaneity, past, present and future.

@Bobs, I really think you are just trolling. Your comments have nothing to do with my original question. Please stay on-topic.

A simpler way of putting this: The OP has to define "similarity" by some kind of measure. There is no single correct, objective measure. The section labelled "trickier cases" gets closer to it. Consider the two men and woman against plain background. Which is more important: 1) pose of person (is one man standing with back facing camera, woman and man both facing camera) 2) colour or nature of background 3) colour of hair/skin/eye, clothing or some cultural similarity between the people  . . . etc, etc. There is no "true" answer, the goal can be highly subjective, depending on the data set

From your examples I assume you presuppose image recognition in the sense that you don't compare the actual images, but the descriptions of what the images contain.
For comparing images there are various algorithms working on the visual similarity. This can sometimes lead to interesting results, as you probably have seen images on the internet like "dog or muffin". A purely visual approach would find this hard to do.
However, if you do have the description of the image (as there are ways of getting captions from images), then it would just be a text comparison between three sentences: the one describing your query image, and those describing your images X and Y. There are ways of getting at the semantic similarity of sentences. The simplest way (from your examples) would be to look at the overlap in words: [yellow, car]/Q, [red, car]/X, and [yellow, car]/Y obviously has the largest overlap with Q and Y. This is rather simplistic, but it is what your examples suggest you are dealing with.
But, getting a proper description from an image is a hard task, of the calibre that Google and Instagram are still working on; they have large amounts of training data and huge resources they can throw at the problem. So, unless that is your starting point, it will not be easy to achieve.
Overall I do not think it is currently possible to solve this problem with high accuracy.

I would go further and say that it is not even possible to define the problem with high accuracy, although I like the approach of comparing open-ended text annotations, because in theory these will emulate the subjectivity of comparing photos done by people (at least in a given context, such as how they would be labelled when uploaded to the internet).

Thanks for the idea. In fact I do have access to some textual metadata, but unfortunately that would only help me solve the easy cases, e.g. "a car is probably more similar to another car than to a dog", however for the trickier cases textual metadata falls short, which makes me believe I need to resort to some kind of fine-grained visual features to break ties. In fact my goal is to learn an embedding with a Neural Network using a Triplet Loss, but the crucial problem lies in generating accurate triplet samples, including both easy and tricky cases. Hence my question.

One idea I have in mind is to extract a ton of features using different techniques (pre-trained CNNs, handcrafted features, etc.) and somehow figure out a hybrid score that combines the "opinions" of all the individual features. A sort of "mixture of experts". This hybrid score would let me judge which image is the most similar to the query image Q. But I have no idea of how to define such a hybrid score.


text:
Methods to tell if a question can be answered from a paragraph
I'm working on a project related to machine Q&A, using the SQuAD dataset. I've implemented a neural-net solution for finding answers in the provided context paragraph, but the system (obviously) struggles when given questions that are unanswerable from the context. It usually produces answers that are nonsensical and of the wrong entity type.
Is there any existing research in telling whether or not a question is answerable using the info in a context paragraph? Or whether a generated answer is valid? I considered textual entailment but it doesn't seem to be exactly what I'm looking for (though maybe I'm wrong about that?)

That isn't an easy problem since you'd need AI capable of making common sense deductions.
For example:
There was war. Were there any dead people and of what gender and age were they?
Yes and they were mostly young men. Or
John is older than Peter. Ann is younger than Peter.
Is Ann younger than John?
Yes. You need a puzzle solving program, dudd.

Have you solved this problem yet?

This problem is very challenging since you need to evaluate the quality of the candidate answers.
In question answering, there are common steps that you need to follow. To summarise you need first to find the sentence that can answer the question and then compose the final answer.
In the first step you can measure the semantical similarity between Q and A,   this a first filter where you can use several deep learning methods. Also, you can define a threshold to validate if the pair QA are enough related.
In the second step, you must extract the answer, if the answer is a fact you can use KB extraction, or if it is a summary, list you can use other DL methods. There is also the possibility that you must infer the answer, also suitable for DL methods.
I suggest to check this paper:
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Which edges of this tree will be pruned by Alpha-beta pruning?

So I know that 'h' and 'f' will be pruned, but I'm not sure about 'k' and 'l'.
When we visit 'j', technically there is no need for us to visit 'k' and 'l' because there are 2 options:

one or two of them might be higher than 8 ('j')
both of them less than 8

But no matter what, the decision of the max(root) will not change, the max will choose the right side no matter what 'k' and 'l' are, because the right side will either be 8 or 9, which is still higher than 4 (returned value from left side)
so will alpha beta prune 'k' and 'l' or not? if not, then it means alpha beta is not "optimal" overall right? considering it will not prune all the unnecessary paths.

If you prune k and L then you could miss the optimal solution. Assume L=9, if you prune L then the value of the tree is 8. If you don't prune L then the value of the tree is 9. Now I will try and address what I think your actual question is

But no matter what, the decision of the max(root) will not change, the max will choose the right side no matter what 'k' and 'l' are, because the right side will either be 8 or 9, which is still higher than 4 (returned value from left side).

From this sentence it seems like you don't care about the value of the tree, you only want to find the optimal first move based on alpha beta. You are correct in saying that the correct first move will always contain the right most child of the root, but oftentimes that is not the only information we want. Sometimes we want to know the value of the tree as well or what the complete correct path is, but if we had pruned k and L we would not know these.
Edit: I have changed all 'L's to uppercase, because lower case 'L' looks to much like uppercase 'i'

Thanks for the answer, also i think one of my mistakes was i assumed that alpha beta knows that we are in the right most subtree, but the default implementation of alpha beta does not know whether we are in the last tree or not correct? because if we are not in the right most tree, the value of that sub tree is important because we might need to compare it with other trees so in this case we need to check L and K because we might need to compare that value with some other sub tree which we have not checked yet, am i correct?

You are correct in both cases.


text:
Does the output layer in a deep neural network need an activation function?
I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows:

and here is a PyTorch implementation:
class MnistModel(nn.Module):
    """Feedfoward neural network with 1 hidden layer"""
    def __init__(self, in_size, hidden_size, out_size):
        super().__init__()
        # hidden layer
        self.linear1 = nn.Linear(in_size, hidden_size)
        # output layer
        self.linear2 = nn.Linear(hidden_size, out_size)

    def forward(self, xb):
        # Flatten the image tensors
        xb = xb.view(xb.size(0), -1)
        # Get intermediate outputs using hidden layer
        out = self.linear1(xb)
        # Apply activation function
        out = F.relu(out)
        # Get predictions using output layer
        out = self.linear2(out)
        return out

shouldn't the output layer also have activation functions?

All layers (except input) have an activation. If it is not defined explicitly, it will be the linear activation. Depending on your task, you may need the linear activation (e.g., for regression)

@ArayKarjauv Thank you for replying. In the course I'm doing, there is no activation function on the output layer. Do you know why is that?

can you please provide a piece of additional information or an example?

Well, this isn't strictly true.  For example, I have some regression networks where the output layer is designed to not have an activation.  It really depends on what you're doing.

@ArayKarjauv https://jovian.ai/kukuquack/04-feedforward-nn
this is my notebook, in one of the code cells, there are 2 linear layers and one activation function for the first layer, You will find it in the mnistmodel class, forward function.

A neural network layer with no activation function is the same as "linear activation" i.e. $f(x) = x$
This is often used for the output layer in regression problems, where a constrained output like that of sigmoid, hyperbolic tangent or ReLU may not be appropriate.
For an output layer, this is fine, and does not conflict with any theory behind neural networks. It will often be combined with a mean squared error (MSE) loss function that makes the gradient calculation simple at the output layer.
For hidden layers, skipping the activation function can be a problem, since a purely linear layer in the middle of a multi-layer network is redundant - it could be replaced or even removed with no impact. That is because two directly connected linear layers are functionally equivalent to a single linear layer with different parameters, and every hidden layer consists of a linear component plus an activation function. So even one missing activation function on a hidden layer directly connects two linear sub-components, making one of them redundant.
In the case of a classifier, some libraries (notably including PyTorch which you are using) require you to have two variants of your neural network - a trainable version without a final sigmoid or softmax layer, and the "full" version which adds sigmoid or softmax on top (usually just a function that calls the training network and add this last activation). This is done to allow you to use more stable gradient calculations that are based on the logits, the values before applying an activation function.

Thanks for this.  Exactly the point I was making in my comment.

While this is correct, it does not answer the question. The problem considered in the example is multiclass classification with cross-entropy loss. As far as I know, it requires softmax activation. Will it work with linear activation?

@ArayKarjauv Actually yes it will, sort of. It is common in PyTorch to not implement the final softmax, and to use a more stable loss function gradient function on the logits. Then there is also an extra method added to the nn module that runs the softmax when needed. So you will often see an incomplete variant of your network used for training purposes only. I added a para about that

I see. So the network does have softmax activation for the output layer, which is integrated into F.cross_entropy

@ArayKarjauv That's not the specific function I recall using, I would use https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss - but I think they are very similar just with different calling conventions, and likely resolve to the same optimised C code underneath

I'd like to add more information to Neal's answer.
A network implemented in the example does not include activation for the output, because it will be applied during the training:
# ...
def training_step(self, batch):
    images, labels = batch
    out = self(images)
    # apply activation and calculate loss
    loss = F.cross_entropy(out, labels)
    return loss
# ...

The problem considered in the example is multiclass classification, which is usually solved with softmax activation and cross-entropy loss. F.cross_entropy combines log_softmax and nll_loss in a single function, which is numerically more stable as softmax and NLL loss. See this discussion for more details, and here is an article about different implementations.

But why isn't there the activation function on the output layer during calling the forward function? @ArayKarjauv

@OmarZayed That is unnecessary. PyTorch allows additional calculations (e.g., mean, normalization, multiplication, etc.) to be done outside of the model class.

So when is adding an activation function necessary in the output layer?

@OmarZayed when you use the negative log-likelihood loss alone or calculate it manually (in that case you can also apply softmax manually outside the forward method). Please read this article for more information.

so according to my understanding, the output layer shouldn't have an activation function because the only thing we care about is getting the highest value for some label?

@OmarZayed what do you mean with the highest value? You shouldn't apply softmax activation to the output layer, because the F.cross_entropy loss function will do it for you.


text:
Language Model from missing data
I want to learn how a set of operations (my vocabulary) are composed in a dataset of algorithms (corpus).
The algorithms are a sequence of higher level operations which have varying low-level implementations. I am able to map raw code to my vocabulary, but not all of it.
e.g. I observe a lossy description of an algorithm that does something:
X: missing data
Algo 1: BIND3 EXTEND2 X X ROTATE360 X PUSH
Algo 2: X X EXTEND2 ROTATE360

The underlying rotate operation could have very different raw code, but effectively the same function and so it gets mapped to the same operation.
I want to infer what the next operation will be given a sequence of (potentially missing) operations (regions of code I could not map).
i.e. I want a probability distribution over my operations vocabulary.
Any ideas on the best approach here? The standard thing seems to throw out  missing data, but I can still learn in these scenarios. Also, the gaps in the code are non-homogenous--some could do many things, The alternative is to contract the sequences and lose the meaning of the gaps, or to learn an imputation.


text:
Dynamic frames processing with CNN LSTM combination or otherwise
I have a unique implementation where I have to process videos with dynamic frame rates (that is the number of frames is different for each video in a batch). I am stacking all the frames in a single tensor and processing the same from there on. This works fine with Conv2D layer but creating a 2D tensor (batch_size, features) by a flattening operation this has to be fed to a Dense layer. I can't find a suitable way to implement this.
For more information on why it should be like this kindly explore: explore this link. Instead of the MNIST images, I have multiple videos in a single bag, each with a variable number of frames.


text:
Is there any way to classify Document Image without OCR?
I have multiple invoices images which need to classify invoice types such as fright, utility, goods, etc. Is there any way to classify without OCR?

Welcome to AI.SE!
Please consider providing a few examples of the invoice images to clarify your project's requirements.

It is possible to classify invoice scans without passing through an OCR component if they are visually different (they demonstrate different visual features). On the other hand, if the invoices look very similar, then the classifier might not be very accurate.
Another challenge would be the number of images you need to train a deep network for image classification (you may start with pretrained models and only perform the finetuning of you do not have enough images). On the other hand, the combination of pretrained OCR models and NLP-based document classifier may not need that many samples for training (for this specific task).

Thank you for the answer @Borhan, can CNN works for these kind of problems?

To reduce the number of parameters, CNN works very well on image classification tasks.


text:
Decide Number of input Parameters and Output Parameters - ANN
I have to create a Neural Network for regression purpose. Basically, I created a Model which predict next 5 values when we give past 6 values.

I want to make a change in this neural network. For example,
when giving 6 past values I have to predict the next 10 values.
Here, is there any issue of selecting the number of output dimension greater than the input dimension. Which type of parameters arrangement makes the Neural Network achieve good accuracy? do I have to decide the number of input parameters always greater than output parameters?
Thanks in Advance!

This should be possible given the fact that ANNs have the ability to do the feature engineering and feature selection tasks by themselves.
This means that given a lesser number of input parameters, the model will be able to generate and select additional features by itself. You will obviously not be able to understand or model these features manually.
The only thing to keep in mind is that you will need a large dataset and a number of iterations before you are able to achieve a decent accuracy.
For example, there are networks that can generate image from classes.
Give this a read and here is an example where the output layer is larger than the input layer.


text:
How can I build an AI with NLP that read stories
I want to do an NLP project but I don't know if it's doable or not as I have no experience or knowledge in NLP or ML yet.
The idea is as follows: Let's say we have a story (in the text) that has 10 characters. Can we define them, their characteristics, whole sentences they said, and then analyze emotions within those sentences.
After that, is it possible to generate an audio version of the story where: the text, in general, is narrated by one voice, each individual character's sentences are read in a different voice generated specifically for that character. Finally is it possible to make the tones of the characters voices change depending on the emotions detected in their sentences?

This doesn't sound like a machine learning problem - if written well you should be able to use a rule based decision on the sentences. For example if you have  'You never listen to my ideas!' Frank yelled, 'Perhaps that's because you've never had a good one.' Paul sneered back. You've got characters clearly stated and emotions. Now you may have to do a little expansion on this to cover all bases but for a machine learning example you need lots of examples where you know the true answer (as in who is truly speaking) to train your model.

@LioElbammalf it may not be a ML problem, but I think it can be considered AI and on topic here. If you think your comment is good enough to help OP achieve goals, then please turn it into an answer. However, I think significant NLP will be required, as most authors don't restrict themselves to such simply parsed phrasing

@NeilSlater Indeed, I wasn't implying the question wasn't on topic, only that using machine learning may be the wrong tool for the job.

I was wondering if you have managed to develop your project. I would be really interested to see how you have done so, because I am working on something similar.

This is quite an ambitious project, and IMHO well beyond the scope of what a single individual can do (within a reasonable time span) at present.
You need to first analyse the story text to identify the characters. This can already be quite a tricky task, as pronouns and other reference expressions are generally used to make a text less monotonous. If a character is referred to by name, say Jane, then you can assume that a follow-up the young woman will refer to her and not a male character mentioned in the same paragraph. But what about the young scientist? Such expressions can be very opaque, and you'd need a lot of world-knowledge to decode them correctly, as they can refer to any distinctive attribute of the character.
Identifying speech is a bit easier, unless you're talking about indirect speech. Jane was thinking aloud. She wasn't going to be able to do that. It was too hard. -- is that speech or not? Compare to Jane was thinking aloud: "I am not going to be able to do that. It is too hard.", which is the direct speech equivalent. Also, unless you're dealing with a play, most of the text will probably not be speech. For the audio version you will probably only want to deal with direct speech, which is usually (but not always) indicated by quote marks.
Analysing emotions seems to be comparatively easy if you have reached this stage, though if it is just based on keywords in the speech it probably won't be very accurate. If you can assign any descriptive statements to characters, that might be more successful, though by no means trivial.
Generating the text as audio should be straight forward. Most operating systems nowadays have speech synthesis integrated, and you can generally choose different voices, so if your text is marked up properly with which voice should speak which part it would be trivial.
To summarise: The NLP part is the hardest bit of it. As has been mentioned in the comments already, I don't think it's a problem that machine learning can help with, and I would stick to traditional methods of parsing the text into a structural representation and then applying rules to identify the bits you are interested in. The recognition of emotion might be a subtask that is suitable for ML, but in the past I have only applied pattern matching to similar tasks, so I can't really say much about that.
From my own experience in text analysis I would think that you might be able to get decent results with a few simple heuristics, but those will likely fail when it becomes a bit more complicated. A lot hinges on the type of story: children's fairy tales might be easier than War and Peace in that respect.


text:
How to choose our data set wisely?
I have a couple of questions and I was wondering if you could answer them.
I have a bunch of images of the cars, side view only. I want to train the model with those images. My objects of interest are 3 types of trucks that have different trailers. I Rarely see two target object at one image (maybe 1 or 2 in every 1000 images). However, I do see other types of cars that I do not want to detect.
My questions are:

Do you think I should tackle this problem as a detection task or classification task? (for example, should I consider multi-label classification or omit those pictures)
Should I also include other vehicles that I do not want to detect in my training dataset? let say I do not assign bounding box to them but include them in training dataset just to make the system robust.

I trained YOLO with 200 images, sometimes the trained model confused and detected the wrong object that is not in any of classes, this happens when training with 2000 images per class?
Is this due to a small number of dataset or it is because of not including those images with no bounding boxes?
Thank you in advance!

Welcome to the community! Perhaps adding a few links (e.g. YOLO) and including some samples of your images would help readers understand the case better.


text:
Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?
I learn that the Viterbi algorithm used for Hidden Markov Model (HMM) can classify a sequence of hidden states from the corresponding observations; Markov Random Field (MRF) and Conditional Random Field (CRF) can also do it.
Can these algorithms be used to classify a single future state?

Yes this is possible and is exactly what a Markov process aims to accomplish.
The Hidden Markov Model can be considered a simple dynamic Bayesian network. This means that it will generate a probabilistic outcome for a number of states (classes) given a sequence of inputs.
This can be used for a classification task based on a threshold probability to make a confirmed decision.
Let's take a simple example

The above example is a model to determine  how a person will be feeling given some sample data of the previous days.
# Sample data
data = ['Healthy', 'Healthy', 'Fever']

Using this data, it's possible to calculate the probability of each possible state (outcome).
# Probability after 2 Healthy and 1 Fever day
P = 0.6 * 0.7 * 0.3

# Final day state 'Fever'
# Calculating probabilities for each state
Dizzy  = P * 0.6
Cold   = P * 0.3
Normal = P * 0.1

We can calculate the probability of each state and this will be unique for every unique sequence of historical data.

Do you have example code?

Code for this example? Or in general? The issue is that HMMs are not easy to generate. The probabilities, specified in the example are dummies, in a real use case, those values will have to be computed statistically from a large dataset before the model can be used.

code in general is fine. I’ve tried a python package, but I can only do decoding to get a sequence of states, while I want to get the state of a single input.


text:
Generate QA dataset from large text corpus
I have a corpus of a domain data in form of 10-15 books pdf and some articles and my end-goal is to make a question-answering system particular to that domain.
For that, I would need a dataset on Q/A which I can use on top of something like SQuAD(Stanford Question Answering Dataset) for domain-specific knowledge
My stuck point is how to convert this corpus into a usable question-answering dataset.
My current strategy is something AllenAI has been working with. A list of their research papers on it can be found here
As I understand they use a combination of Knowledge Extraction, Natural Language Understanding, and Inference to get the job done. But I cannot find any good practical implementation.
Where can I find a good resource?


text:
What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?
I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.
Here is a screenshot of theorem 3.1.

In the proof of theorem 3.1

they say

By setting the derivative of Equation 1 to 0, we obtain the following expression
...
$$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$

Here's equation 1 from the paper.
$$\hat{Q}^{k+1} \leftarrow \arg \min _{Q} \alpha \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$
My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?
The $\hat{B}^\pi$ is the empirical Bellman operator and is defined as $\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum_{s'} \hat{T}(s' \mid s,a) \mathbb{E}_{a'\sim \pi(a' \mid s')}\hat{Q}_k(s', a')$. Since in offline reinforcement learning, the dataset $\mathcal{D}$ typically does not contain all possible transitions $(s, a, s')$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.

Can you actually format the equations as text, rather than an image? That would make it a lot more accessible.

Yes, please, next time, rather than providing the screenshots, for future reproducibility and easy accessibility, copy and quote the relevant parts of text from the paper, at least, as I've just done to your post. To get the equations in latex automatically, you can use a tool like "Mathpix Snipping Tool" (which is what I used).

I apologize, I will try to copy the relevant parts. And thanks for recommending mathpix snipping tools!

Please can I confirm what happens to the expectations when the derivative is take w.r.t Q?

why can it ignore the expectation.

first we need to transform the distribution of the first term:
$\mathrm{argmin} \space \alpha \mathrm{E}_{s\sim D, a \sim D}[\frac{\mu}{\hat\pi_{\beta}}Q(s,a)] + \frac{1}{2}\mathrm{E}_{s,a \sim D}(Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a))^2   $
then its derivate respect to Q is:
$\alpha\frac{\mu}{\hat \pi ^{\beta}} + Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a) = 0$
then we have
$ Q(s,a) =\hat\beta^{\pi}\hat Q^k(s,a) -\alpha\frac{\mu}{\hat \pi ^{\beta}}$

how did you find the derivative with respect to Q?


text:
Influence of location on a Neural Network trained for parking detection occupancy
I loaded a neural network model trained with Caffe by other people in OpenCV.
The model should detect the presence of a car in a single parking spot outputting the probability of it being free/occupied.
The model was trained with images all belonging to the same parking area, taken at different hours of day and with different light conditions. Images were taken by different cameras but the cameras are all of the same model (raspberry cameras).
I tried to run the model with a few images some of them taken from their dataset and other downloaded from google.
The images taken from their dataset are correctly classified while the ones taken from google are not correctly classified.
My question is: is it possible to deploy a NN model trained with images all coming from a single parking area in another parking area? Is not such a model for parking detection occupancy supposed to generalize independently from the location where training images have been taken?
If you know about an already existing trained model that works good please let me know.

It is unlikely anyone will know of or be willing to search for an existing model - such requests are generally off-topic in Stack Exchange.

That is not my main question and I am not asking for people looking for it. I am just asking that if they know about the existance of such models please let me know.

My question is: is it possible to deploy a NN model trained with images all coming from a single parking area in another parking area? Is not such a model for parking detection occupancy supposed to generalize independently from the location where training images have been taken?

Generalisation in machine learning has limits, and for complex systems such as computer vision, it is very easy to exceed them. This is why the state-of-the-art systems use huge amounts of training data which is often augmented with automated variations, plus trained on for many days.
If you want some factor in your model to generalise well, you should:
a) Train with variations in that factor
b) Test with variations in that factor that have not already been seen in the training set
For the factor you wish to have the model generalise in, then it looks like (a) has not been done, and that you have just done (b) and found that the first attempt has failed an important test that you care about.
I suspect the trained model you have used will generalise well by time of day, type of vehicle and some other details. It does not appear to generalise to a new location because it was trained on only one location.
If you have limited training data, you may want to look at approaches to fine-tuning against more general image classifiers (such as the many ImageNet-trained classifiers, which are available to download). However, even then you should train with a few varieties of parking space, or at the very least with the spaces that you care about (if you do the latter then it probably won't generalise to other spaces, but you won't care).

It may also be worth seeing what you can do to transform the input that you are working with to have similar view over the new parking area that the training images have. Using a transform to align the outline of the new area with the old one, and possibly some colour correction could help a lot.


text:
What is meant by "generate the data" in describing the difference between on-policy and off-policy?
From the book:
Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. "
following is stated:
"On-policy methods attempt to evaluate or improve the policy that is used to make decisions, whereas off-policy methods evaluate or improve a policy different from that used to generate the data."
Looking at off policy:

and on-policy:

What is meant by "generate the data"? I'm confused as to what 'data' means in this context.
Does "generate the data" translate to the actions generated by the policy ? or Does "generate the data" translate to the Q data state action mappings?

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.
In both the SARSA and Q learning pseudocode from the book, there is a behaviour policy that selects the next action to take. Other than the initial start state, this policy drives the observations that the learning process must handle - every action is directly selected by it, and due to how MDPs work, every next state and reward are indirectly influenced by it. It is this behaviour policy therefore that must "generate the data".
You can see from the pseudocode that both algorithms describe this behaviour policy as "policy derived from Q (e.g. $\epsilon$-greedy)".
The difference in the two algorithms is in which target polcy is being used:

In SARSA, the update is based on the Bellman equation for the same policy that generated the most recent set of data for S, A, R, S' and A' - all of these values will be directly or indirectly caused by the behaviour policy. So the whole equation uses the same data that was generated. Another way to put that is behaviour policy and target policy are the same.

In Q learning, the $\text{max}_a Q(S',a)$ from the Bellman optimality equation removes the need to use A' in the update. Effectively it is a local search for actions to find one that is potentially better than A', and runs the update as if that one was the one that was taken. This revision of A' is what makes the off-learning evaluate a different target policy to the behaviour policy that generated the rest of the data used in the update.


text:
Is it required that taking an action updates the state?
For some environments taking an action may not update the environment state. For example, a trading RL agent may take an action to buy shares s. The state at time t which is the time of investing is represented as the interval of 5 previous prices of s. At t+1 the share price has changed but it may not be as a result of the action taken. Does this affect RL learning, if so how ? Is it required that state is updated as a result of taking actions for agent learning to occur ?
In gaming environments it is clear how actions affect the environment. Can some rules of RL breakdown if no "noticeable" environment change takes place as a result of actions ?
Update:
"actions influence the state transitions", is my understanding correct:
If transitioning to a new state is governed by epsilon greedy and epsilon is set to .1 then with .1 probability the agent will choose an action from the q table which has max reward reward for the given state. Otherwise the agent randomly chooses and performs an action then updates the q table with discounted reward received from the environment for the given action.
I've not explicitly modeled an MDP and just defined the environment and let the agent determine best actions over multiple episodes of choosing either a random action or the best action for the given state, the selection is governed by epsilon greedy.
But perhaps I've not understood something fundamental in RL. I'm ignoring MDP in large part as I'm not modeling the environment explicitly. I don't set the probabilities of moving from each state to other states.

A very vague question. What's the objective?
Reinforcement Learning (RL) typically uses the Markov Decision Process framework, which is a sequential decision making framework. In this framework, actions influence the state transitions. In other words, RL deals with controlling (via actions) a Markov chain. The objective in RL is figure out how to take actions in an optimal (in some sense) way!
If, in the application you mentioned, the actions don't influence the state transitions and the objective is to predict states, RL is not required. It's just a regression/ time-series problem.

I guess one thing that also might be useful to OP is that you could have an MDP where you take an action that does 'nothing' - i.e. taking action $a$ in state $s$ will just transition the agent trivially back into state $s$ with probability 1 and receiving reward 0.

@math_pile thanks, please see question update.

@blue-sky in your edit your understanding of $\epsilon$-greedy is the wrong way around, with probability 0.9 the agent would take the greedy option (the action that maxes the Q value) and with probability 0.1 act randomly.

@blue-sky: $\epsilon-$ greedy is as David pointed out. Coming back to your question, it is still not clear what your objective is. From what I understand, you don't need RL. If there are actions, and the actions might or might not affect the state, then perhaps Bandits framework might be useful (https://en.wikipedia.org/wiki/Multi-armed_bandit)

It seems to me that you are confusing two things, State of the agent and State of the environment.
Think about a robot learning to walk on a rugged terrain.
The actions of the robots don't change the terrain at all ! The robot can include in his own state, part of the environment state, the topology of the terrain in front of him for instance.
In an MDP,  an agent may or may not modify the environment state, in your case, if we make the assumption that you don't manipulate the market, you can consider that you don't modify the environment state.
Your issue might be in the way you designed your agent, relying only on a part of the state environment, and not really having any state of his own, such as a portfolio for instance.
If you're only interested in the way the state environment changes, then the above answer is right, the RL framework may not be a good fit.
Finally, you don't have to model explicitly the environment, that's the beauty of model free RL ! Q-learning, that you mentioned, is model-free.


text:
Looping over Sarsa algorithm for better Q values
Let's say an RL trading system places trades based on pricing data.
Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.
The state is a sequence of prices, and the action is either buy, hold, sell.
Instead of "Loop for each episode" as per the Sarsa algorithm :

I add an additional outer loop. Now instead of just looping for each episode we have:
for 1 to N:
   "Loop for each episode"

Manually set N or exit out of the loop on convergence.
Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.

do you have a fixed data set or are you able to generate data from the environment? if it is the latter, then yes your approach is correct

@David Ireland the dataset is fixed as for a given time interval the price values do not change. I'm confused by your comment 'generate data' as the model utilises data from the environment, the source of the environment pricing data is a trading exchange. So I can generate data by reading the pricing data but I'm not sure if I'm interpreting correctly 'generate data from the environment' .

The way I understood your post was that you have an episode that is 24 steps long. Each step you get a series of prices and you have to decide whether to hold/sell/buy. If so, then are you able to interact with an environment arbitrarily, e.g. can you run simulations of this? or is your data set fixed?

@DavidIreland yes can run simulations where the reward of choosing buy/hold/sell for each step is observed. The dataset is fixed in that the number of prices for the environment is immutable. The episode length is mutable, for example If I set the episode length from 1 to two hours then there are 12 episodes available and the number of price attributes for each state will double.

Okay, then it seems like it should work by following the table. However, is your state space continuous? If so, then you may want to use Deep Q-Learning.

@DavidIreland the state space is continuous. I'm considering Deep Q-Learning but I'm also considering quantizing the data or using an auto encoder to try and ensure states are revisited during training. Do you want to provide an answer to this question or I'll collate your comments and provide an answer.

I will write an answer for DQN.

Given that your state space is continuous, then I would recommend using Deep Q-Learning. As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.


text:
What is the relation between semi-supervised and self-supervised visual representation learning?
What's the differences between semi-supervised learning and self-supervised visual representation learning, and how they are connected?

Both semi-supervised and self-supervised methods are similar in the sense that the goal is to learn with fewer labels per class. The way both formulate this is quite different:

Self-Supervised Learning:

This line of work aims to learn image representations without requiring human-annotated labels and then use those learned representations on some downstream tasks. For example, you could take millions of unlabeled images, randomly rotate them by either 0, 90, 180 or 270 degrees and then train a model to predict the rotation angle. Once the model is trained, you can use transfer learning to fine-tune this model on a downstream task like cat/dog classification just like how you finetune ImageNet pretrained models. You can view an overview of the methods and also look at contrastive learning methods that are currently giving state-of-the-art results such as SimCLR and PIRL.

Semi-supervised Learning

Different from self-supervised learning, semi-supervised learning aims to use both labeled and unlabeled data at the same time to improve the performance of a supervised model. An example of this is FixMatch paper where you train your model on labeled images. Then, for your unlabeled images, you apply augmentations to create two images for each unlabeled image. Now, we want to ensure that the model predicts the same label for both the augmentations of the unlabeled images. This can be incorporated into the loss as a cross-entropy loss.

Semi-supervised learning
Semi-supervised learning is the collection of machine learning techniques where there are two datasets: a labelled one and an unlabelled one.
There are two main problems that can be solved using semi-supervised learning:

transductive learning (i.e. label the given unlabelled data) and
inductive learning (generalization) (i.e. find a function that maps inputs to outputs, like classification).

Self-supervised learning
Self-supervised learning (SSL) is a machine learning approach where the supervisory signal is automatically generated. More precisely, SSL can either refer to

learn data representations (i.e. learn to represent the data) by solving a so-called pretext (or auxiliary) task, in a self-supervised fashion, i.e. you automatically generate the supervised signal from the unlabelled

automatically label an unlabelled dataset by exploiting data coming from different sensors (this is the usual definition of SLL in the context or robotics)

What is the relationship between the two?
SSL (for data representation) can be considered a semi-supervised learning approach, if you fine-tune the learned data representations with a labeled dataset to solve supervised learning problem (i.e. the so-called downstream task), which you will probably do otherwise the data representations are pretty useless. Read this answer for other details.

The previous answer has given a good insight into the difference between two areas. I would like to give more examples.
Semi-Supervised Learning work with improving the data set by adding up new examples. There are iterative systems where we train a model on a given dataset and improve the model further after deploying it on the real world by adding interactions of the real world and their outcomes to further train the system.
Self-Supervised Learning is becoming a very hot topic these days. It has the ability to understand the underline properties of a given dataset with some kind of a supervisory signal (Not exactly a label). self-Attention introduced in Transformers is a modern day popular self-supervised learning. Also, check this tweet from Yann Lecun tweet