Calculate similarity on boolean data I am trying to implement simple recommender system and I am trying to understand different approaches to achieve my goal.

My dataset consists of users and items that they bought. I have information about what items user bought and descriptions of these items in form of titles.

At first I though I could use user based collaborative filtering approach but I am stuck at this. I am not quite sure how to calculate similarity for boolean data.

When I have data like this for example

   1  2  3  4
A  0  1  0  1
B  0  1  0  1
C  1  0  1  1
D  0  1  0  0
E  0  0  1  1

And I want to recommend items for user E, so how should I calculate similarity in this case? I chosen for example cosine similarity from scikit learn module in python. But I am not quite sure what should be considered as input. From what I read it should be only vectors of items that two users for which similarity is calculated have in common.

So for example if I wanted to compute similarity between user E and C what should be my input? Because if I input only values that they have in common it does not make sense right? Beacuse input will be [1, 1] and [1, 1] and for that similarity is 1.

Then I tried to input the whole vector like this:

from sklearn.metrics.pairwise import cosine_similarity
from numpy import array, reshape

c = array([1, 0, 1, 1])
e = array([0, 0, 1, 1])

result = cosine_similarity(c.reshape(1, -1), e.reshape(1, -1))

>>> result is 0.81649658

And this approach I think makes more sense but I am not sure if it is acceptable based on what I learned about this type of recommendation.

You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, 1 means the item is present, 0 otherwise. For example: for an universe of items banana, orange and apple. the set banana, orange will be represented by (1, 1, 0). The Jaccard Index is the intersection of the sets over union the sets so for a set (1, 1) it's value is 1.

Cosine similarity is for real-valued vectors, but whether cosine similarity is better than the Jaccard index and vice versa depends on the application. You should do a test on your data and verify which is better, for a discussion see this question


Should the output of regression models, like SVR, be normalized? I have a regression problem which I solved using SVR. Accidentally, I normalized my output along with the inputs by removing the mean and dividing by standard deviation from each feature.

Surprisingly, the Rsquare score increased by 10%.

How can one explain the impact of output normalization for svm regression?

In regression problems it is customary to normalize the output too, because the scale of output and input features may differ. After getting the result of the SVR model, you have to add the mean to the result and multiply that by the standard deviation, if you have done that during normalizing.

  How can one explain the impact of output normalization for svm regression?

If you normalize your data, you will have a cost function which is well behaved. means that you can find local parts more easily. The reason is that you have to construct the output using the input features in regression problems. It is difficult to make large values with small normalized features but with small numbers, making a normalized output is easier and can be learned faster.


Is there a link between Training, Test errors based on k fold CV and not doing CV? I am using Matlab to train a feedforward NN using Cross validation (CV) approach. My understanding of CV approach is the following. (Please correct me where wrong)

Let X be the entire dataset with Y as the label set. Split X into 90/10 ratio to get: [Xtrain,Xtest] using holdout approach by calling the cvpartition(Y,'Holdout',0.1,'Stratify',true)
Apply CV on Xtrain: For every fold I calculated the accuracy. At the end of the CV loop I have an average accuracy score. Let accCV denote this variable. Inside the CV loop xtrain is further split into [xtrain_cv,xtrain_val].
After CV loop, I reinitialze the weights and re-train a new model using Xtrain. Then I get a training accuracy which I denote by the variable accTrain.
Using the model obtained in Step3 I test for evaluating the model's purpose and consider this to be the generalization performance that is the performance when we have an unseen future data, Xtest. I call this accuracy as accTest.

Question1: Is it possible that accCV will be less than the accuracy over the train set Xtrain when not using the CV approach? That is I call the NN model over Xtrain only once and record the accuracy and denote it by variable accTrain, then is it possible that accCV ~ accTrain?. Or intuitively, accCV should be close to the accuracy when not using CV approach since the dataset is the same which is Xtrain. If this is the case, then why use CV when outside the CV we do not reuse the model that was created inside the CV? What does it tell us?

Question2: If accCV < accTest but the accuracy on the entire dataset Xtrain without using CV is close to that of accTest (accTrain ~ accTest) are we doing something wrong? What is the best case scenario? Is it accCV ~ accTest?

It is expected that accCV < accTrain: the former is the accuracy on the test folds (averaged over all the splits), so represents models' scores on unseen-to-them data.  Similarly, you would expect accTrain > accTest.

There are two main reasons to evaluate a model, whether by k-fold cross-validation or simple train/test split: for hyperparameter optimization / model selection, or to estimate future performance.  (N.B., k-fold should generally give better predictions than simple train/test split.)  If you make any decisions based on the scores, then they no longer represent unbiased estimates of future performance.  If, in your setup, you make no decisions based on step 2, then you should expect accCV ~ accTest, and there's no real reason to include that step.  If you do make decisions based on step 2, you may expect accCV > accTest, though the gap is probably substantially smaller than the gap in accTrain > accTest.  You may see discrepancies here, due to natural variation in the datasets, or perhaps data leakage.


Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0? By definition, Relu is max(0,f(x)). Then its gradient is defined as:
1 if x > 0 and 0 if x < 0.

Wouldn't this mean the gradient is always 0 (vanishes) when x < 0? Then why do we say Relu doesn't suffer from the gradient vanish problem?

You're mostly correct!  ReLU does have a problem with the gradient vanishing, but only on one side, so we call it something else: the 'dying ReLU problem'.  See this stack overflow response for more information: What is the "dying ReLU" problem in neural networks?

It's a small semantic difference.  Lots of functions (tanh and logistic/sigmoid) have derivatives very close to zero when you're outside the standard operating range.  This is the 'vanishing gradient' issue.  The worse you get, the harder it is to get back into the good zone.  ReLU doesn't get worse the farther you are in the positive direction, so no vanishing gradient problem (on that side).  This asymmetry might be enough to justify calling it something different, but the ideas are quite similar.


SPC vs Autoencoders in anomaly detection Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?

Control charts are widely used still (whether people know it or not). However it is limited to cases where it is possible to define meaningful and stable targets and bounds for a metric. The classical example is in manufacturing, when the intent is to produce items which has a certain property at a certain value. Like each cereal box produced should be 510 grams, 500 grams of net contents and 10 grams for the box. Being outside for example +-5% of this would be considered an anomaly.

However when there are no such clear metric, or targets, more advanced tools can be useful. For example consider you run a web service which serves webpages to users. Different time of day have different usage patterns. Different users make different kinds of request. Different kinds of requests have different impact on the system resources and thus the ability for your system to serve other (unrelated) requests. System performance will vary quite a lot, but a lot of it due to factors outside your control. And often it is not individual metrics which are the clue to "something" is wrong, but combinations of them.
This makes it hard define single-value metrics, and to set control-chart targets and limits. But an autoencoder can learn time-based patterns, the user-dependent patterns, request-type dependent patterns, and their relationships. And give an overall score for how anomalous the overall situation is.


Text comparison: spot the differences I would like to know what would be the best approach to compare two texts and see the differences between them.

For example:

Sent_1=“This toolset is a set of macros for performing a number of modelling tasks.”

Sent_2=“This tool is a set of macros which help performing a certain number of tasks.”

I do not mind the context/meaning at the moment, but I would like to know what it would be the best approach to spot differences (looking at each word, antecedent and subsequent to it) and see how accurate it is.

You could look at string similarity measures and TFIDF (usually with cosine). If you want a measure which works at both levels of words and sentences, there are more advanced options such as SoftTFIDF.


is gradient descent also used during feed forward propagation in Neural Network? To the best of my understanding , weights are updated during back propagation only using gradient descent and no gradient descent is used during feed forward propagation. is it correct ?

The feed-forward pass computes the outputs of each layer, and hence the output of the network, given a current state, i.e. the values for all the parameters for this pass or iteration. The parameters are the weights and biases of the neurons.
Once the feed-forward pass has completed the next step is to update all the parameters in order that in the next feed-forward pass the cost function is smaller. To update the parameters, the parameters are moved in the opposite direction of the gradients of the cost function w.r.t them. This is called Gradient Descent Algorithm, and basically, taking into account that the gradients indicates the direction of steepest increase of the cost function, if the parameters are moved towards the opposite direction, the next step the cost function is supposed to be smaller.
The problem with neural networks is that we don't know the gradients of the cost function because the number of the parameters are massive (billions) and the cost function themselves are quite complex (not convex, with pathological curvatures...) To overpass this issue, backpropagation has invented as a means of efficiently compute the gradients of the cost function (it uses the chain rule and propagates the errors from the lat layer to the first one)

To the best of my understanding , weights are updated during back propagation only using gradient descent and no gradient descent is used during feed forward propagation. is it correct ?

Yes and no...

Yes if you think of gradient descent as only the updating stage of the parameters
No if you understand Gradient Descent as a whole, in which you need the results obtained in the forward pass to compute the values of the gradients in the backwards pass

I'm personally prone to the second option, in which Gradient Descent comprises two stages: a gradient computation and parameter updating


How to prepare colored images for neural networks? I have seen many examples online regarding the MNIST dataset, but it's all in black and white. In that case, a 2D array can be constructed where the values at each array element represent the intensity of the corresponding pixel. However, what if I want to do colored images? What's the best way to represent the RGB data?

There's a very brief discussion of it here, which I quote below. However, I still don't get how the RGB data should be organized. Additionally, is there some OpenCV library/command we should use to preprocess the colored images?

  the feature detectors in the second convolutional-pooling layer have
  access to all the features from the previous layer, but only within
  their particular local receptive field*

  *This issue would have arisen in the first layer if the input images were in color. In that case we'd have 3 input features for each pixel,
  corresponding to red, green and blue channels in the input image. So
  we'd allow the feature detectors to have access to all color
  information, but only within a given local receptive field.

Your R,G, and B pixel values can be broken into 3 separate channels (and in most cases this is done for you). These channels are treated no differently than feature maps in higher levels of the network. Convolution extends naturally to more than 2 dimensions.

Imagine the greyscale, single-channel example. Say you have N feature maps to learn in the first layer. Then the output of this layer (and therefore the input to the second layer) will be comprised of N channels, each of which is the result of convolving a feature map with each window in your image. Having 3 channels in your first layer is no different.

This tutorial does a nice job on convolution in general.

http://deeplearning.net/tutorial/lenet.html


Building a Citation Network to Analyze in R I am working on network analysis project at one of our Nation's Service Academies, and I need a little help.

As a starting point, we are looking at a citation network that we build by using the keyword, "Network Analysis", for peer reviewed articles only in Web of Science. We took the top 50 results, and looked at the articles to which those cited and built a citation network.  It outputs the information in Bibtex, Tab-Delimited, HTML, Plain Text, End Note, and a few others.

Here is my question:

What is the best way to create a citation network?

We have these formats, but it is not usable yet.  Ideally, we would have a matrix with article names on the x and y, and binary numbers for the
    connections.

I don't understand if you want to build a network of authors who cite each other, or a network of papers who cite other papers (which would be a much sparser network because the coauthorship relationships won't show up as edges).

I would follow a strategy similar to this one (off the top of my head):

Assign ids to your papers
Build 2 csv files : papers.csv, citations.csv
read in 2-col csv file "paper.csv" as a two-column data frame:  col1: paper_id, col2: title
read in 2-col csv file "citations.csv" as a two-column data frame:  col1: paper_id, col2: cites_id,

With R's igraph package, you can construct a  network pg (for papers_graph)
with (pseudocode)

pg <- igraph::graph_from_data_frame(citations)

Then you assign "vertex attributes" to the nodes in the network:

pg <- set_vertex_attribute(pg, "title", value= papers)
# same as:  V(pg)$title <- papers$title

(and possibly many other attributes)

Then you can use igraph's many functions (~200) to analyze the network.

For visualizations, you plot the ids, and use a diagram type which gives you the title when you move the mouse over / click on the node symbol (which is labeled only with the id to save screen space). You can use other design elements such as coloring the nodes by release year for instance).


Which classification algorithms to try for classifying text data into 300 categories I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category.
I want to classify that into 300 categories. Some categories are independent while some are somewhat related. Distribution of data among categories is not uniform either i.e some of the categories(around 40 of them) have less data about 2-3 rows.

I am attaching log probablity of each class/categories. (OR distribution of classes) here.

In general, a decent starting point for problems like these is Naive Bayes (NB) classification using a simple bag of words model. Here are some slides describing NB as applied to natural language processing. There's nothing especially fancy about this approach, but it's pretty easy to implement and will give you a starting point to expand from.

Once you've found some initial results assuming independence among your features and your output labels, you'll probably have a better sense of where the model is weak.  From that point forward you can apply some feature engineering (maybe TF-IDF) as well as some post processing to deal with samples that get assigned to related categories.


Split a list of values into columns of a dataframe? I am new to python and stuck at a particular problem involving dataframes.

The image has a sample column, however the data is not consistent. There are also some floats and NAN. I need these to be split across columns. That is each unique value becomes a column in the df.

Any insights?

It looks like you're trying to "featurize" the genre column.

df = pandas.Series([('Adventure', 'Drama', 'Fantasy'), ('Comedy', 'Family'), ('Drama', 'Comedy', 'Romance'), (['Drama']),
                    (['Documentary']), ('Adventure', 'Biography', 'Drama', 'Thriller')]).apply(frozenset).to_frame(name='genre')
for genre in frozenset.union(*df.genre):
    df[genre] = df.apply(lambda _: int(genre in _.genre), axis=1)

The output:

| row | genre                                   | Romance | Documentary | Thriller | Biography | Family | Drama | Comedy | Adventure | Fantasy |
|-----|-----------------------------------------|---------|-------------|----------|-----------|--------|-------|--------|-----------|---------|
| 0   | (Drama, Adventure, Fantasy)             | 0       | 0           | 0        | 0         | 0      | 1     | 0      | 1         | 1       |
| 1   | (Comedy, Family)                        | 0       | 0           | 0        | 0         | 1      | 0     | 1      | 0         | 0       |
| 2   | (Drama, Comedy, Romance)                | 1       | 0           | 0        | 0         | 0      | 1     | 1      | 0         | 0       |
| 3   | (Drama)                                 | 0       | 0           | 0        | 0         | 0      | 1     | 0      | 0         | 0       |
| 4   | (Documentary)                           | 0       | 1           | 0        | 0         | 0      | 0     | 0      | 0         | 0       |
| 5   | (Drama, Biography, Adventure, Thriller) | 0       | 0           | 1        | 1         | 0      | 1     | 0      | 1         | 0       |


Which algorithms or methods can be used to detect an outlier from this data set? Suppose I have a data set : Amount of money (100, 50, 150, 200, 35, 60 ,50, 20, 500). I have Googled the web looking for techniques that can be used to find a possible outlier in this data set but I ended up confused.

My question is: Which algorithms, techniques or methods can be used to detect possible outlier in this data set?

PS:Consider that the data does not follow a normal distribution. Thanks.

You can use BoxPlot for outlier analysis. I would show you how to do that in Python:

Consider your data as an array:

a = [100, 50, 150, 200, 35, 60 ,50, 20, 500]

Now, use seaborn to plot the boxplot:

import seaborn as sn
sn.boxplot(a)

So, you would get a plot which looks somewhat like this:

Seems like 500 is the only outlier to me.  But, it all depends on the analysis and the tolerance level of the analyst or the statistician and also the problem statement.

You can have a look at one of my answers on the CrossValidated SE for more tests.

And there are several nice questions on outliers and the algorithms and techniques for detecting them.

My personal favourite is the Mahalanobis distance technique.


Does traning the Word2Vec model multiple times affect `min_count` parameter? In Word2Vec, If I train a set of sentences multiple times with change in order (as it increases the vector representations), will the frequency of a word get changed due to it.?

For example, if I have the word "deer" in my corpus 4 times and If I set the min_count to be 5, does training the model 3 times repeatedly count "deer" with frequency 12 and will be included in the model ?

If it knows it is the same corpus then how it is possible to differentiate, if I retrain the model with a new corpus.

The question has been answered in google groups by Gordon mohr.

Normally there's one read of the corpus to build the vocabulary (which includes initializing the model based on the learned vocabulary size), then any number of extra passes for training. It's only after the one vocabulary-learning scan that word counts are looked at (and compared to min_count for trimming).
If you supply a corpus (as a restartable iterator) as one of the arguments to the initial creation of the Word2Vec model, all these steps are done automatically: one read of the corpus (through the build_vocab() method) to collect words/counts, then one or more passes (as controlled by the 'iter' parameter and done through the train() method) for training. Still, only the count for the single pass over the supplied corpus matters for frequency decisions.
If you don't supply a corpus at model-initialization, you can then call build_vocab(…) and train(…) yourself. It's only what's passed to build_vocab() that matters for retained frequency counts (and the estimate of corpus size). You can then call train(…) in other ways, or repeatedly – it just keeps using the vocabulary from the one earlier build_vocab(…) call.
(Note that train(…) does try to reuse the single-pass corpus size, remembered from the vocab-scanning pass, to give accurate progress-estimates and schedule the decay of the training-rate alpha. So if you give a different-sized corpus to train(…), you should also use its other optional parameters to give it a hint of the size.)


How to handle a valuable feature that is missing on 99\% of the samples in the data set? Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this feature in building a prediction model? Any ideas would be greatly appreciated!

Its best to remove such a variable. Reasons are following:

Artificial imputation can add bias and result cannot be justified because 99% data for the particular variable was artificially created.
The variables/features that you choose for building the predictive model should have low correlation with the target/outcome variable/feature. Because, variable that are highly correlated with the target/outcome variable reduces the model predictive performance.
You want a model which has low bias and low variance.


Machine Learning in Spark I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get some snippet in python to find the probability of labels.

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model  then  model.transform(test) contains a node of probability, for more details please check the below code, going to show you the probability node and others useful nodes also for iris dataset.

Partition dataset randomly into Training and Test sets. Set seed for reproducibility

(trainingData, testData) = irisdf.randomSplit([0.7, 0.3], seed = 100)

trainingData.cache()
testData.cache()

print trainingData.count()
print testData.count()

Output:

103
47

Next, we will use the VectorAssembler() to merge our feature columns into a single vector column, which we will be passing into our Naive Bayes model. Again, we will not transform the dataset just yet as we will be passing the VectorAssembler into our ML Pipeline.

from pyspark.ml.feature import VectorAssembler
vecAssembler = VectorAssembler(inputCols=["SepalLength", "SepalWidth", "PetalLength", "PetalWidth"], outputCol="features")

For iris dataset, it has three classes namely setosa, versicolor and virginica. So let's create a Multiclass Naive Bayes Classifier using pysaprk library ml.

from pyspark.ml.classification import NaiveBayes
from pyspark.ml import Pipeline

# Train a NaiveBayes model
nb = NaiveBayes(smoothing=1.0, modelType="multinomial")

# Chain labelIndexer, vecAssembler and NBmodel in a pipeline
pipeline = Pipeline(stages=[labelIndexer, vecAssembler, nb])

# Run stages in pipeline and train model
model = pipeline.fit(trainingData)

Analyse the created mode model, from which we can make predictions.

predictions = model.transform(testData)
# Display what results we can view
predictions.printSchema()

Output

root
 |-- SepalLength: double (nullable = true)
 |-- SepalWidth: double (nullable = true)
 |-- PetalLength: double (nullable = true)
 |-- PetalWidth: double (nullable = true)
 |-- Species: string (nullable = true)
 |-- label: double (nullable = true)
 |-- features: vector (nullable = true)
 |-- rawPrediction: vector (nullable = true)
 |-- probability: vector (nullable = true)
 |-- prediction: double (nullable = true)

You can also select a particular node to view for some dataset as:

# DISPLAY Selected nodes only
display(predictions.select("label", "prediction", "probability"))

Above will show you in tabular formate.

Reference:

spark
Models using pipeline
https://mike.seddon.ca/natural-language-processing-with-apache-spark-ml-and-amazon-reviews-part-1/
https://stackoverflow.com/questions/31028806/how-to-create-correct-data-frame-for-classification-in-spark-ml


Probability of an event How should I go about solving the following problem?

If you randomly type a 6-digit number on a note, what is the
probability that you can see the same number if you flip your note
upside down? How would you explain your answer to a 6-year-old?

The only piece I have been able to calculate is that there are 9x10x10x10x10x10 numbers.
Is this correct? and/or what is the correct way to get the answer to this problem?
Thanks a lot for your help!

For the number to be same when seen upside down:

It can only contain either 0,1,8,6,9.

For first and last digit :
For it to be a 6 digit number first digit and the last digit cannot be 0.
If first digit is 6 then last should be 9 and vice-versa.
If first digit is other than 6 or 9 (ie. 1,8) then they both should have same digit.

For second and fifth digit / For third and fourth digit :
If first digit is 6 then last should be 9 and vice-versa.
If first digit is other than 6 or 9 (ie. 1,8) then they both should have same digit.

Total Number of 6 digit numbers = 9 * 10 * 10 * 10 * 10 * 10  ([9] as first digit cannot be 0)
Numerator = 4 * 5 * 5 ([4] as first and last digit cant be 0)
So,
Probability should be = Numerator / Total number of 6  digit numbers


Figure out relative importance of entity attributes I'm trying to understand how various aspects of a movie contribute to its gross revenue. I want to rank a movie's attributes in that sense - the attributes that most strongly determine the revenue are ranked higher.
Let $A_1,\ldots,A_n$ be a list of attributes of a movie and let the possible values of $A_i$ be $a_{i1},a_{i2},\ldots$. Many of these attributes (like primary genre) are categorical and some of them (like rating) are continuous.
Approach 1: Consider $A_1$: I can form groups of movies having the same value of $A_1$, e.g. all movies with $A_1=a_{12}$ form a group. The other attributes in a group can vary freely. I can then calculate the mean of the revenues of all movies within a group, and then take the variance of means of all groups.
This will give me the "variation in average revenue as we change $A_1$ values". If this variation is high, that means changing $A_1$ significantly affects the average revenue - so $A_1$ should be a highly ranked attribute.
Approach 2: Again consider $A_1$: fix the values of all other attributes $A_2,\ldots,A_n$ and look at movies with the same values for $A_2,\ldots,A_n$ but different values of $A_1$. Find the variance in revenues of such movies - call it "$A_1$ variance". The attributes $A_i$ with highest "$A_i$ variance" will be ranked higher.
Approach 3: Train some ML model (not sure which one) with revenue as target variable and attributes as features. Then look at feature importances to get attribute importances.
A few queries:

What assumptions do I need to check for approach 1? e.g. minimum size of a group, distribution of other attribute values within a group, etc.
Are there any potential flaws or gotchas in approaches 1 and 2?
What approach would you prefer out of the three?
What ML model should be used for approach 3? I'm confused because there are plenty of regression models: Linear regression, GB regression, random forest regressor, etc.

Approaches 1 & 2
1. Noise
If we adopt a frequentist view, we can consider the attributes of a movie to have an underlying probability distribution (or, if you'd rather be Bayesian, you can choose to believe they come from a certain distribution). For a single attribute, its values are a sequence of random variables drawn from the same distribution. We can model our data for one attribute by considering a data model of the general form $Y = X + \epsilon$, where $Y$ is a random variable representing one measurement, $X$ is a random variable coming from the "true" underlying distribution and $\epsilon$ is a noise term.
For some variables, the noise term is negligible (e.g. the runtime of a movie is actually not noisy at all, it's a real, objective, fixed value). For others, noise may follow a normal distribution, or another distribution with mean 0. In that case, over many measurements, the law of large numbers dictates that the mean of your $Y$s will converge to the expectation of $X$. Here comes the first warning: if you have noisy data and few of them, your data might not mean much.
Finally, you might be so unlucky that you have noise $\epsilon$ coming from a distribution with a mean which is not $0$, in which case you are likely to have systematically biased data.
In both approaches, you partition your observations into groups/classes (in 1 based on the values of one attribute, in 2 on the values of all other attributes). This can lead to class imbalance, in which case you might end up with some classes with many observations (and presumably little noise) and some classes with few members (maybe only one!) and a possibly a lot of noise.
Furthermore, if you look at variation only, and not correlation, then you might end up with high variance (even comparatively, if one attribute is very noisy perhaps due to previously mentioned class imbalance) and low correlation, which would erroneously lead you to deduce that an attribute has good explanatory power.
2. Multicollinearity
Approach 1 opens you up to the problem of collinearity: two predictor variables might be strongly linked, in which case it's not easy to say which one is really a valuable predictor and which one can be considered just an epiphenomenon. For instance, assume fantasy as a genre is very popular, but people dislike overly long movies. Fantasy movies would then be very long and have a large revenue, but it would be wrong to conclude that the long runtime is responsible for their large revenue. Even worse: while this example used two attributes which you probably have access to, a hidden, unobserved (i.e. latent) attribute might actually be causally responsible for the phenomenon and for the correlation between the two observed attributes.
In approach 2, you're doing something akin to statistical matching, so you at least stand a better chance of avoiding multicollinearity and confounding variables.
Personal recommendation
I would personally start with some exploratory data analysis (EDA). Plot your data! Use heatmaps, small multiples, violin plots, the works, anything that might give you insight into its structure.
Next, start small, go big if needed. Try measures of "linear" correlation, first Pearson correlation. For categorical variables, you can use Spearman rank correlation or Kendall rank correlation. You can also use measures of rank correlation (e.g. those mentioned previously) to estimate non-linear correlation. You can also use SHAP values, Q-Q plots, partial plots.
You can then move on to maybe a simple decision tree (or random forest), or a linear regression if you encode your categorical variables. If that gives you good accuracy, then you can examine what your model has learned. You can use stepwise linear regression, where you either add or subtract attributes one by one and examine the change in predictive power.
You can also try dimensionality reduction, for instance do PCA or ICA then look at which attributes carry a lot of weight in the first few PCA/ICA components. You can also try other types of matrix factorization.
If accuracy is poor with simple models, you can try more complex models, such as neural networks, then use interpretability models designed for those, such as LIME, which builds a locally accurate linear surrogate of a deep network.


Mean error (not squared) in scikit-learn cross_val_score I need to know if the values generated by each fold of cross_val_score have a distribution which is centered on zero.  Something as simple as the median or mean of y_true - y_predicted would suffice.  All I see in the available options are absolute and squared.  I've looked into make scorer but can't see how to code the simple mean error and then call it as the scoring argument in cross_val_score.

from sklearn.datasets import load_diabetes
from sklearn.metrics import make_scorer
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

def mean_error(y, y_pred):
    # assuming y and y_pred are numpy arrays
    return np.mean(y_pred - y)

X, y = load_diabetes(return_X_y=True)
mean_error_scorer = make_scorer(mean_error, greater_is_better=False)

regr = LinearRegression()
cross_val_score(regr, X, y, scoring=mean_error_scorer)


How can I find to starting point of skewed data in python? I have a list like this,

import random
import seaborn as sns

years = []

for i in range(1000):
    if i % 100 == 0:
        val = random.randint(1900, 2000)
    else:
        val = random.randint(2000, 2021)

    years.append(val)

sns.distplot(years);

Here is output graph,

As you can see, there is a density after 2000. There is not much data before this point. My question is how can I find this point in skewed data? Is there a formula that gives this? Any idea? Thanks in advance.

Try looking at ways to find outliers, such as Tukey's fences or the modified Thompson's tau.


Keras binary classification model is not learning anything. How to improve my model? I am trying to train a binary image classifier on a an imbalanced dataset of images which are really small in dimensions(the largest dimension is 40*70). I am augmenting images by grayscale, rescaling and resizing to 48*48. My keras model is not learning at all. Can you suggest some technique using which I can improve the model or data?
I figured out the problem. It was my data. Without knowingly I was doing single class classification while doing binary classification. Any advice what approach to be taken?

I wrote few notes on your implementation:

Classification on imbalanced data can be a problem. How imbalanced is your dataset? You can try to compensate by training your model by feeding balanced mini batches of data. You artificially build each batch so that classes are more or less equally represented, so that your classifier can learn from all of them.
I would change the last layers. I suggest you to use softmax activation at the output layer, and a crossentropy loss function that is specific for this task. The number of output nodes must be equal to the number of classes.
Adam optimizer is commonly referred to as the best algorithm for gradient descent.
MAE is a metric for regression tasks, it doesn't work for classification. You can use 'accuracy' instead.


Neural Net Backprop Weight updating Pseudo code help please Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the error using the gamma variables in the code.
The problem is I get very small change in the average error after the first iteration. I'm using initial weights randomly between -0.1 and 0.1 and learning rate of 0.03. Inputs are roughly between -0.001 and 0.001. Target output is between -0.5 and 0.5. I just want to check if the code is correct. I've tried higher values for learning rate and initial weights and the error actually goes up and settles around 1.0. Now is the code wrong or do I just need to fine tune the initial weights and learning rate? Many thanks for your replies!

ERROR = 0.5 * MathPow( (TARGET - output), 2 );

double gamma = ERROR * ( 1 - ( output * output ) );

// HIDDEN-OUTPUT LAYER UPDATE...
for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
{
       double gamma_hi = gamma  * HIDDEN_OUTPUT_WEIGHTS[ i ];

       HIDDEN_GAMMAS[ i ] = gamma_hi;

       double hi_weight_delta = LEARNING_RATE * gamma_hi  * HIDDEN_OUTPUTS[ i ];

       HIDDEN_OUTPUT_WEIGHTS[ i ] += hi_weight_delta;
}

// INPUT-HIDDEN LAYER UPDATE

for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
{
   for ( int j=0; j<INPUT_LAYER_SIZE; j++ )
   {
        double gamma_ij = HIDDEN_GAMMAS[ i ] * INPUT_HIDDEN_WEIGHTS[ i ][ j ];

        double ij_weight_delta = LEARNING_RATE * gamma_ij * INPUT_LAYER_INPUTS[ j ];

        INPUT_HIDDEN_WEIGHTS[ i ][ j ] += ij_weight_delta;
    }
}

// OUTPUT CODE:

// Calculate value of each hidden...
   for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
   {
      double hi = 0.0;

      for ( int j=0; j<INPUT_LAYER_SIZE; j++ )
      {
         hi += INPUT_LAYER_INPUTS[ i ]* HIDDEN_OUTPUT_WEIGHTS[ i ][ j ];
      }

      HIDDEN_OUTPUTS[ i ] = TANH( hi );
   }

   double output = 0.0;

   // Calculate value of the ouput
   for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
   {

      output += HIDDEN_OUTPUTS[ i ] * HIDDEN_OUTPUT_WEIGHTS[ i ];
   }

   output = TANH( output );

It looks like you're adding the delta to the weights instead of subtracting it. Gradient descent is given by the following calculation run for some iterations:

$$
x^{current} = x^{previous} - \alpha \frac{dy}{dx}
$$

where $\alpha$ is the learning rate. We subtract the value because derivatives go in the direction of steepest ascent. So the following lines:

HIDDEN_OUTPUT_WEIGHTS[ i ] += hi_weight_delta;
INPUT_HIDDEN_WEIGHTS[ i ][ j ] += ij_weight_delta;

should be

HIDDEN_OUTPUT_WEIGHTS[ i ] -= hi_weight_delta;
INPUT_HIDDEN_WEIGHTS[ i ][ j ] -= ij_weight_delta;

Additionally, I'm not sure what the architecture of your network is, but it doesn't look like you have bias units- consider adding them so that your neurons can have non-zero centered activations. Not having them may impact your network's ability to learn.


Why this single layer network does'nt work I am trying following code (modified from https://www.kdnuggets.com/2017/10/seven-steps-deep-learning-keras.html ):

def single_layer(input_shape, nb_classes):
    print("input shape:", input_shape)
    print("print nb_classes:", nb_classes)

    from keras.models import Sequential
    from keras.layers import Dense, Activation

    model = Sequential()
    model.add(Dense(nb_classes, input_shape=input_shape, activation='softmax'))

    model.compile(optimizer='sgd', loss='categorical_crossentropy')
    model.summary()
    return model

However, when I try to fit this model with an X_train of dimensions 64,64,3 and 17 classes, following is the output with error:

input shape: (64, 64, 3)
print nb_classes: 17
Using TensorFlow backend.
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_1 (Dense)              (None, 64, 64, 17)        68
=================================================================
Total params: 68
Trainable params: 68
Non-trainable params: 0
_________________________________________________________________
Traceback (most recent call last):
....
....
  File "/home/abcd/.local/lib/python3.5/site-packages/keras/engine/training.py", line 950, in fit
    batch_size=batch_size)
  File "/home/abcd/.local/lib/python3.5/site-packages/keras/engine/training.py", line 787, in _standardize_user_data
    exception_prefix='target')
  File "/home/abcd/.local/lib/python3.5/site-packages/keras/engine/training_utils.py", line 127, in standardize_input_data
    'with shape ' + str(data_shape))
ValueError: Error when checking target: expected dense_1 to have 4 dimensions, but got array with shape (10396, 17)

Why this code is not working and how should it be modified to make it work?

I can't find the code in the link you posted to get more background information on what kind of data source it is that you are using.

However, a data source that is $60\times60\times3$ is described as being high dimensional. You have a total of 10,800 input features which need to be mapped down to 17 different classes. This is actually a very complex task and will not be successful using a single layer network. Such a simple network will not have enough parameters to capture the non-linearities between features in your input space.

That being said, if you insist on using a single layer network with a data of size $60\times60\times3$ with 17 input classes the code is as follows.

Let's first create some artificial data of the same dimension as your data

import numpy as np

n = 1000
x_train = np.zeros((n,64,64,3))
y_train = np.zeros((n,))
for i in range(n):
    x_train[i,:,:,:] = np.random.random((64,64,3))
    y_train[i] = np.random.randint(0,17)

x_train = x_train.reshape(n,64,64,3,)

n = 100
x_test = np.zeros((n,64,64,3))
y_test = np.zeros((n,))
for i in range(n):
    x_test[i,:,:,:] = np.random.random((64,64,3))
    y_test[i] = np.random.randint(0,17)

x_test = x_test.reshape(n,64,64,3,)

print('Training data: ', x_train.shape)
print('Training labels: ', y_train.shape)
print('Testing data: ', x_test.shape)
print('Testing labels: ', y_test.shape)

  (1000, 64, 64, 3)
  (1000,)
  (100, 64, 64, 3)
  (100,)

For a classification task we should convert our outputs to categorical vectors. Where we use one-hot encoding to identify the correct class.

import keras

# The known number of output classes.
num_classes = 17

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

We then build the model.

from __future__ import print_function
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.models import model_from_json
from keras import backend as K

input_shape = (64,64,3,)

model = Sequential()
model.add(Flatten(input_shape=input_shape))
model.add(Dense(17, activation='softmax'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

You can see the summary of the model by using

model.summary()

Then we can train this model using

batch_size = 128
epochs = 10
model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

This code works. However, the data is completely random thus the model cannot learn anything. However, even if your data is easily distinguishable, as I said above, I do not expect this model to be complex enough to distinguish such a large input space from 17 different possible classes.

If you post the data source you are using we can design a model to get a good result.


R aggregate() with dates I am working on a data set that has multiple traffic speed measurements per day. My data is from the city of chicago, and it is taken every minute for about six months. I wanted to consolidate this data into days only, so this is what I did:

traffic <- read.csv("path.csv",header=TRUE)
traffic2 <- aggregate(SPEED~DATE, data=traffic, FUN=MEAN)

this was perfect because it took all of my data and averaged it by date. For example, my original data looked something like this:

DATE        SPEED
12/31/2012   22
12/31/2012   25
12/31/2012   23
...

and the final looked like this:

DATE        SPEED
10/1/2012    22
10/2/2012    23
10/3/2012    22
...

The only problem, is my data is supposed to start at 9/1/2012. I plotted this data, and it turns out the data goes from 10/1/2012-12/31/2012 and then 9/1/2012-9/30/2012.

What in the world is going on here?

I am going to agree with @user1683454's comment. After importing, your DATE column is of either character, or factor class (depending on your settings for stringsAsFactors). Therefore, I think that you can solve this issue in at least several ways, as follows:

1) Convert data to correct type during import. To do this, just use the following options of read.csv(): stringsAsFactors (or as.is) and colClasses. By default, you can specify conversion to Date or POSIXct classes. If you need a non-standard format, you have two options. First, if you have a single Date column, you can use as.Date.character() to pass the desired format to colClasses. Second, if you have multiple Date columns, you can write a function for that and pass it to colClasses via setAs(). Both options are discussed here: https://stackoverflow.com/questions/13022299/specify-date-format-for-colclasses-argument-in-read-table-read-csv.

2) Convert data to correct format after import. Thus, after calling read.csv(), you would have to execute the following code: dateColumn <- as.Date(dateColumn, "%m/%d/%Y") or dateColumn <- strptime(dateColumn, "%m/%d/%Y") (adjust the format to whatever Date format you need).


Neural network approach to the cocktail party effect Imagine you have 2 people at 2 different microphones but in the same room. Each microphone is going to pick up some sound from the other person. Is there a good neural network based approach to isolating the signals so that the sound from each microphone only captures 1 person?

I remember hearing a solution to this a few years back, but Im not sure if I remember that correctly

I ask because a similar problem was mentioned to me today. During EEG brain wave data collection, each electrode can pick up signal from multiple sources in the brain. In that world they try to isolate the sources and reduce the "noise" from other brain areas, and its common to use ICA for such a task. The problem with ICA is that the post-processing stage is very time consuming, so I'm wondering if theres a better ANN/DNN approach that could solve the problem more efficiently, or maybe with better accuracy

Take a look at this.

No DNN, but math, if different channels are available.

DNN were used for single channel input, but have to be trained to the signals you want to separate.


How do you integrate nlp into an existing website? Let's say I have a db driven app which has two tables: people and organizations. I run my documents through a named entity recognition program. Now - what do I do with this additional ner information?

Does it go in my existing db tables?
Do I have to rewrite my existing tables to accommodate the ner data?
Do I make new tables for the ner data?
Do I connect them to my older tables with foreign keys?
Do I need new functions that pull response data from these new ner objects instead of, or with, my original ones?
Are these new entities "objects" in the traditional OOP sense?
But they are different objects from the ones in my original db tables, so do they replace the old, non-ner objects?
Does it matter what kind of datastore the new ner objects go into in order to be fully utilized? rmdbs? key/value? document based?

The only thing I’ve been able to find online about this is a Medium post that pickles the model - it says nothing about any other kind of datastore - and the Flask website just displays the information by accessing it through an api.

A search here on datascience.stackexchange.com got this:

  We couldn't find anything for integrate with existing website

And now I get a warning that my question "appears to be subjective and is likely to be closed". I'm sure that's machine learning at work. Granted, the step by step details might be different if your site is php vs python, but the issue is a major one I assume a lot of people are going to run into sooner if not later. My site is in Django.

How do you integrate the product of nlp into an existing, complex website with many apps and functions?

The answer is it depends on you architecture.

Some things to consider:

The tags are attributes of words in a (presumably) text field in a database. So Boyce Codd would prescribe putting it in a seperate table referencing the primary key (and possibly the position in the text)
The tags are just a function over your data, so storing it at all should be for caching purposes. Which you might not want to do in your database, but at the presentation level, or perhaps in a separate database, or schema.
The tags are in fact the product of an approximation of a function over your data. There is no such thing as a perfect NER-tagger. This means that you might want to switch the function later buy something more suited.

It could make sense to extract the information on display time from a microservice (perhaps hosted by you), and cache the resulting presentation for better performance.


How can I use octave function minimizers in c++? I have downloaded and built octave library and it works fine . But I cannot call function minimizers like fminunc() , fmingc() etc to minimize my functions for performing logistic regression or using it in neural networks . Can these functions be accessed from c++ ? If yes, then how ?

You can use a native library in C++ for optimization. Check out this one :
dlib


Linear kernel in SVM performing much worse than RBF or Poly When trying to train a SVM on some Kaggle data, I have encountered a situation where the linear kernel fails to give any results.

This doesn't make sense to me because the RBF kernel works just fine, and my understanding is that a linear kernel is a strictly simpler version that doesn't map the data into higher dimensions.  In fact, all my research suggests that not only is a linear kernel possible, but that in most cases, it should be faster to converge (with the trade-off that the results might not be as accurate).

However, this has not proved to be the case.  While the RBF kernel was able to produce a result during cross-validation after ~6 minutes, the linear kernel just sat there with no output after ~6 hours.  After this point, I just force quit the training and moved onto testing other SVM parameters.

A breakdown of some facts:

Roughly 60,000 examples, around 120 features (Sometimes cutting down to 30 features helps, and linear kernel will produce a result after many hours)
I'm using SKLearn GridSearchCV to perform my training
I have tested this used SVC(kernel='linear') as well as LinearSVC(), both with the same outcome
This issue has come up with multiple data sets across different competitions

My current hypothesis is that the training stops possibly because the data is not linearly separable.  I might be missing something else obvious though.  Any guidance is appreciated, thanks!

This depends very much on your dataset: There is data, which can not be separated using a linear kernel. It is impossible, and you won't get any reasonable result. Take for example the simple XOR function. The plot of the XOR looks like this:

No matter how hard you try, you will never be able to separate this using a linear SVM, or any linear method.

Here's some short MATLAB code which illustrates the problem:

x = [0 0; 0 1; 1 0; 1 1];    % training points
y = [0; 1; 1; 0];            % target: XOR
linearSvm = fitcsvm(x,y);    % train with linear kernel
linearSvm.predict(x)

>> ans =
        0
        0
        0
        0

The SVM has no chance of finding a solution. Now if we use a kernel, we create a mapping to a higher-dimensional space. The kernel trick allows us to implicitly do this mapping, by calculating a kernel function, which returns the result of a scalar product in this high-dimensional space, without needing to map the input values to this space. The RBF kernel is special, as it corresponds to an infinite-dimensional space.
Now if we specify a RBF kernel and run the same example again, then:

gaussSvm = fitcsvm(x,y,'KernelFunction','rbf');  % RBF kernel
gaussSvm.predict(x)

>> ans =
        0
        1
        1
        0

the SVM easily finds the correct result. The same result (in this XOR case) is also found when using a polynomial kernel. There are many more examples, where data simply can't be separated linearly, for example:

Of course, if your data is almost linearly separable, then you can (and probably will) get good results with a linear kernel. Using soft-margin SVMs, which allow for a few misclassifications, often leads to good results, as they can ignore outliers.

If the linear kernel gives no result, you will have to accept that your data is not (well) linearly separable. You'll either have to switch to a kernel, or allow for more misclassifications in the soft-margin SVM.


Algorithm for gesture classification in a wearable I'm fairly new to machine learning so please pardon me if I mess up terms. I'm working on a wearables project that involves hand gesture detection, looking for any pointers on potential algorithms that would be applicable.

Basically there will be sensors that pick up tendon movement at the wrist to provide input, so for simplicity I'm assuming that would mean 5 features (one for each finger), each with values ranging from say 0-100.

There are also 5 classes that the gesture could be classified as. The user would do each type of gesture several times to provide training data, and then for each gesture class I'd need an algorithm to determine the best model for it based on that training data.

After initial training, whenever the user does a gesture it would input the sensor readings into each model to see which one is the closest match.
Additionally, since this is a wearable anything that optimizes for low memory and low processing power would be best.

If anyone has any input on either the type of models or the algorithm with which to generate them that I should look into I'd appreciate it!

Edit: Here is some sample data from 4 sensors during 3 different gestures. These are basically just pressure readings from sensors lying along tendons on the wrist.

It might be really useful to provide a bit more details regarding the actual data, produced by your sensor. Can you provide a simple example for any simple gesture?

Anyway, let's try to address the problem in the following way:

sensor can be represented as a function f(t)
range of the function will be {1,2,3,..100}

So we can draw something like the following (can we?):

You can extract various high-level features out of that data, but I would suggest to go with inflection points as the most straightforward option:

Those points, basically, define a pattern. I'll use a figure from the different answer to illustrate the idea. For example, the naive pattern below is used to identify a circle.

In your case situation is a bit simpler, because sensor output is the only data you can work with. Naturally, you can combine multiple sensors in order to define a bit more complex interaction scenarios:

Also you can take into account f'(t) in order to distinguish between slow and fast movements, etc.

Update

Thanks for the data. It does look like these three gestures looks pretty much the same from the data perspective (see visualization below).

It really looks like several features are required to identify the gesture.


Does it make sense to label a dataset manually? I would like to implement a classifier on a dataset which does not have a label. I've written a script which labels each row of the source file by some specific values like "IF H > 45 && T <= 89 THEN label = True".

Does it make sense to do something like this? I mean if I'm able to label the dataset by the script I've wrote, why do I need the classifier? There are any arguments why it makes sense to still use a classifier for it?

The chicken egg dilemma

What came first: the labeled data or the machine learning model?

If you have labeled data, then you can train a machine learning model.
If you have a trained machine learning model, then you can label data.

Precision vs. Recall

Suppose you are in case 2 (you have a model). Then how is the precision and recall of your model? Unless both are $100\%$, you might still want to label data.

                       recall
                 100%     |     < 100%
          --------------------------------------
          Your model is   | You can confidently
p   100%  perfect. You    | trust the labels you
r         don't need more | get. But your model
e         data.           | might miss some.
c         --------------------------------------
i         You get         | You get some
s  < 100% labels, but you | labels, but you
i         still need to   | still need to
o         confirm that    | confirm that
n         they are correct| they are correct

Most people will be at the lower-right quadrant (precision and recall will be $\lt100\%$). The goal will be to get as close as possible to $100\%$. Therefore, the model will output labels which can be the basis for manual labeling.

Your case

If you can label all rows with some rules and you are creating these rules, then, in fact, you are already manually labeling data. Further, you could use a decision tree later on, to model all your rules in order to deal with new data points that come your way.


Error in Keras Fitting I am having the problem with the following python code which uses the library keras, and I have not been able to find a solution online:

       def model_build():
        model_mag=Sequential()
        model_mag.add(Dense(64, activation='relu', input_dim=5))
        model_mag.add(Dropout(0.5))
        model_mag.add(Dense(64, activation='relu'))
        model_mag.add(Dropout(0.5))
        model_mag.add(Dense(64, activation='relu'))
        model_mag.add(Dense(1))
        model_mag.compile(loss=losses.mean_squared_error,optimizer='sgd')
        return model_mag
       clf = KerasRegressor(build_fn=model_build(),nb_epoch=100)
       # here is the error
       clf.fit(X_train,y_train, nb_epoch=20)

I know that X_train and y_train are well defined above and I can run other classifier methods just fine with them so they are not the problem. It keeps throwing me the following error in the fit line and I can't figure out why:

TypeError: call() takes exactly 2 arguments (1 given)

I am giving two arguments though.

You have put nb_epoch in fit.

You have to remove nb_epoch from fit, and let it only in KerasRegressor.


Max 75% accuracy! help! i am trying to train a deep neural network to figure out that if there is a 1 and 0 present in the first two columns of X that the output is 1 otherwise its 0. but im only getting a 75% accuracy on the model!

import numpy as np
import tflearn

X = [[0, 0, 1],
    [0, 1, 1],
    [1, 0, 1],
    [1, 1, 1]]

Y = [[0, 1],
    [1, 1],
    [1, 0],
    [0, 1]]

Xtest = np.array([[1, 1, 1],
                  [0, 1, 1],
                  [1, 0, 1],
                  [0, 1, 1]])

# Build neural network
net = tflearn.input_data(shape=[None, 3])
net = tflearn.fully_connected(net, 32, activation='sigmoid')
net = tflearn.fully_connected(net, 32, activation='sigmoid')
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net, optimizer='adam')

# Define model
model = tflearn.DNN(net)
# Start training (apply gradient descent algorithm)
model.fit(X, Y, n_epoch=5000, batch_size=16, show_metric=True)

pred = model.predict(Xtest)
for i in range(4):
    print(pred[i][0])

The output should be: [0, 1, 1, 1]

Training Step: 4999  | total loss: 0.50493 | time: 0.004s
| Adam | epoch: 4999 | loss: 0.50493 - acc: 0.7813 -- iter: 4/4
--
    0.01631585881114006
    0.4872587323188782
    0.9684665203094482
    0.019177177920937538

It looks like you are training this as a multiclass classifier, to represent a binary choice. In which case, your Y value is wrong:

Y = [[0, 1],
    [1, 1],
    [1, 0],
    [0, 1]]

Here your second label is not self-consistent, and thus it is impossible to predict using a softmax output layer (where the sum of all outputs must equal 1). The best it can do is [0.5, 0.5] to match that label and you can see actually it got close to that in your test.

You want this instead:

Y = [[0, 1],
    [1, 0],
    [1, 0],
    [0, 1]]

A few asides . . .

Your example inputs all have the same third column (=1). This is redundant data, and you could drop it.
Your network is more complex than it needs to be for this task. A single hidden layer with only a few neurons in it should be sufficient.
For this specific task you could have chosen a single output neuron using a sigmoid activation (and need only one column in Y).


Watermark detection in Python I have a lot of images and I would like to be able to classify them into two groups: one containing images with watermarks and one containing images without any watermark.

There are about 40 different watermarks. I created "fake" watermarked images to train a CNN and it worked very well on the "fake" validation set but not on the real images. Plus it was a long shot because I would have needed to train a model for each watermark (and I don't have the original watermark) or train a big model.

I quit the watermark approach to try and find text. So I tried OpenCV text detection and it really wasnt working since the text is crooked and not that different from the background.

Is there an easy solution I missed? Any idea is welcolme. I am kinda new to machine learning :)

Interesting question! Maybe the pretrained models in Keras can help. Either by means of transfer learning, so that you might have to label only a small number of images by hand to retrain the higher layers. Or by using them for feature extraction and see if a certain keyword appears frequently for watermarked images.

Or just upload the pictures some place that does not allow watermarks and see if they get flagged ;-)


Is there any logic to adding a threshold to see if two variables are related? I have two variables $X$ and $Y$ given as tuples of $(x, y)$, and I want to see if there is a relationship between the two variables. I can do so by finding the correlation coefficient.

However, I found that by selecting an arbitrary subset of the data (e.g. $(x, y) | x > k$ ), I can get a higher correlation coefficient and a stronger result. Is doing so mathematically sound? I have no a priori reason to believe that certain data points are "more important" than others, to put it simply.

I think you may have a sample size effect on your correlation coefficient. You may find the following references useful:

At what sample size do correlations stabilize?
The effects of sample size and variability on the correlation coefficient.
And also this post in stats.stackexchange demonstrating the effect of subsampling on correlation coefficient.


How to clusterize points lying on the sphere I want to cluster protein conformations by dihedrals angles.
My point is an n-dimensional vector, where is n - number of dihedral angles.
I think I can't use Euclidean distance for distance metric because of the distance between +179 degrees and -179 degrees is 2 degree, not 358.

Can you suggest some clustering algorithms and distance metrics for such tasks?
Maybe some algorithms from geo points clusterization.

You have circular variables.
One way to deal with them is to create two variables for each:
sin(alpha_1), cos(alpha_1), sin(alpha_2), cos(alpha_2), ...
Then you can use regular clustering algorithms.

There exists custom algorithms for circular data.
Examples: Unsupervised clustering of multivariate circular data


Struggling to understand GCNNs (Graph Convolutional Neural Networks) Although I've worked with CNN's for over a year, I am struggling to understand how GCNNs work paper on their simplification. I've read several papers, and I find myself out of my depth when they talk about Chebyshev polynomials or Fourier spaces.

The descriptions talk about using an adjacency matrix as input, and perhaps my primary confusion is how I can supply such a matrix to a convolutional neural network (if that is what is in fact what is done). I can't just convolve over the matrix as if it were an image because spatial similarity in the matrix (i.e. rows/cols that are near to each other) doesn't signify actual closeness between nodes in the graph. The question is therefore, how does an adjacency matrix, fit into the framework of CNNs which uses the spatial filters, and the logistic operations (non-linear regressions) in the layered stack/graph.

As this GCNN is less mature than CNNs, which have more explanatory material, some further explanation material as to how these methods differ would be helpful. Are there simple situations in which the usage of GCNN can be used to derive results specific to this method/framework?

The name "Graph Convolutional Neural Network" is a bit misleading, as no "traditional" convolutions (like in the context of CNNs) take place at all. You are correct that it doesn't really make sense to perform convolutions on the adjacency matrix of a graph. The thing that GCNNs have in common with CNNs is that there is a concept of a "local neighbourhood" which is exploited.

In CNNs, this neighbourhood consists of the surrounding pixels of each pixel. A linear transformation (discrete convolution) is applied to the pixels in the neighbourhood, before being passed through a nonlinearity.

In GCNNs, however, the neighbourhood for each node is given by the set of other nodes that share edges with the node. The representations of each node are averaged with the representations of the nodes in their neighbourhoods before having a linear transformation applied and subsequent nonlinearity. The method for performing the local averaging can be done using some matrix arithmetical tricks with the adjacency matrix, which is perhaps what you got confused about.

In the simplified version, they remove the nonlinearity step which vastly simplifies the computation of the hidden representations and the linear transformation step.


Bag-of-words model : Boolean vs. TF-IDF When I design a document classifier using traditional feature engineering, I would prefer (to Boolean model) tf-idf model to represent a document into a vector because intuitively Boolean model loses information of how important each word is for classifying a document into certain class.

I mean using Boolean model for representing a document as a vector is to give it a less meaningful position in n-dimensional vector space than tf-idf-based feature extraction when each dimension represents a term, by using discrete value rather than a continuous value, since discrete(0 or 1) value is made to ignore the difference of weight of each term although parameter tuning process may optimize coefficient of each term when using linear combination for document classification.

Am I justified in my thinking that using Boolean feature for bag-of-words model to extract feature vector from a document is not a good choice for the above-mentioned reason?

I already know the recent approach like representation learning and dimensional reduction like word embedding or BERT language model. My question is limited to some traditional feature extraction from document data.

Your reasoning is correct: for most tasks related to information retrieval and/or document classification based on the semantics of the documents, it's recommended to take into account the importance of the terms (both inside the document and across all documents, hence TF and IDF).

However TF-IDF is not necessarily always the best choice:

There are some classification tasks which are not based on the semantics of the document. For example if the goal is to classify documents by writing style (e.g. find documents by the same author) then the topic doesn't matter and therefore IDF is not relevant.
In the case of a very small dataset and/or very short documents, using TF-IDF scores can lead to overfitting. In such cases using boolean values might perform better because it makes the job of the model easier.


Hive / Impala best practice code structuring Coming from a DWH-background I am used to putting subqueries almost everywhere in my queries. On a Hadoop project (with Hive version 1.1.0 on Cloudera), I noticed we can forego subqueries in some cases.
It made me wonder if there are similar SQL-dialect specific differences between what is used in Hadoop SQL and what you would use in a DWH-setting. So I would like to extend this question so that people can mention what they noticed as differences between Hadoop and DWH in when structuring their queries. I noticed there was very little reference to this topic for Hadoop.
There are some normal principles that apply to Hadoop like mentioned here: https://streever.atlassian.net/wiki/spaces/HADOOP/pages/3211279/Hive+SQL+Best+Practices
It would be nice to get a few of your best practices for working with Hadoop. E.g. You write your queries as neutral as possible so that it works in both Hive and Impala avoiding using language-specific functions such as left (Impala only)
The example I came across was that a group by-query worked in two different ways. In classic RDBMS I know that only one of both would work.
Here is an example (columns are displayed in different order then in group by):
Classic SQL (Oracle, SQL Server, etc)
select t2.b, t2.a

from (select t1.a,t1.b from table1 as t1

group by t1.a, t1.b) t2

Hive SQL
select t1.b, t1.a

from table1 as t1

group by t1.a, t1.b

Notice that everything is combined in 1 single query Hadoop. The classic SQL snippet first does the group by in a subquery before the data is displayed in the right order.
I would be curious to know if you came across other subtleties like this one.
I know, based on my experience, that it is good practice to respect the following guidelines while coding:

Keep your code clean (hence avoid sub-queries if you can do without)

Consider what impact changes to settings might have. E.g. yarn.nodemanager.resource.memory-mb=24576 is great to use, but what happens if you are not allowed to change the node memory size. In an automated job it is not necessarily good practice. Look at the general Hadoop settings, e.g. file size.

Avoid using functions that are specific to a program. E.g. select left("Hello world", 3) is useful in Impala but in Hive it has to be rewritten to select left("Hello world", 1,4). This can lead to problems when later on down the line it will run in a different program.

The biggest help to me was learning about how MapReduce works on a distributed environment. I don't have exact rules of thumb for you, but I might be able to help.

In Hadoop/Hive, the hive.execution.engine definitely seems to matter. Make sure understand the differences and try changing it for certain problems.
Subqueries can actually HELP performance, especially with really wide tables, MapReduce as your execution engine, and Hive. For example, if you only want 2 or 3 columns from a 2nd table, its better to:

SELECT T1.*, T2.col1, T2.col2
FROM T1
LEFT OUTER JOIN (SELECT key, col1, col2 FROM T2) T2
ON T1.key = T2.key;

This is all due to how the execution engine works under-the-covers, so learning about that and the differences of the execution engines will really help.

Temp tables are your friend - prior to hadoop, I avoided staging data in temporary tables a lot and wrote more complex SQL to avoid it. But with Hadoop and big data, I've actually found it much faster to create tables and then subsequently joining to them helps in some cases with massive data.

Learn how the table's underlying data is stored. STORED AS TEXTFILE is a lot different than data stored as parquet

Impala vs Hive also has performance implications. I've found to find Hive more stable, but slower.

How the data is stored (skewed or not, bucketed or not, etc) makes a huge different. Keep an eye out for queries where you have 100 reducers and 99 of them finish super fast and 1 of them takes forever.

Like I mentioned, spend time learning how the execution engines do the work in a distributed environment. Once you understand how the execution happens behind the scenes, you can start to pick up on other subtleties like why this:
select count(distinct user)
from table;

can be a low slower than this:
select count(*) from
(select user
 from table
 group by user) q;


Neural Network for Multiple Dependent Outputs I have a dataset with approx 6 input features and 5 output values to be predicted. I am trying to understand what kind of neural network would be most suitable to assign probability across multiple dependent outputs, with the motivation to maximise the value of a single feature.
For example, suppose the inputs are:
company size
number of employees
turnover
average salary
country
years of operation

And the outputs to be predicted are:
% budget allocated to marketing
% budget allocated to sales
% budget allocated to R&D
% budget allocated to training
% budget allocated to shareholders

The training dataset also contains the column:
profit

Which is what I want to optimise against (i.e. the primary motivation of the model).
For a given set of inputs (company size, number of employees, turnover, average salary, country, years of operation), I want to be able to predict values for the 5 outputs which are most likely to achieve the highest 'profit'. The sum of the 5 outputs must equal 100%.
In other words, there is a finite budget to be allocated, and I want to create a model to predict the best budget allocation to maximise profit.
What neural network would be most suitable for this purpose? I have looked into multi-output regression, however I imagine this would assign independent probabilities for each of the 5 outputs (i.e. they won't add up to 100%).
Is it possible for probability to instead be assined to each output dependent of the other outputs (i.e. so they add up to 100%)? If so, is there a name for this type of approach?

In many cases, at the end of a neural network you could find a Softmax layer, which outputs probabilities, so they add up to 1. It seems exactly what you're looking for. Please find more information about softmax function here or here.
Hope it helps.


What kind of statistical analyses can I do with my data? I'm trying to analyze human intentions in clicking google ad word keywords.

In this dataset I have the usual adword details, for example

CTR = Clicks / Impressions
CPC = Cost / Clicks
CPA = Cost / Converted Clicks
ROI = Total Conversion Value / Cost
CVR = Converted Clicks / Clicks
Weighted Avg Pos = WAP / Impressions
and Ad position placement.

Basically, I can imagine a relationship between Keywords and CTR. I can do some psychological analysis, like opinion mining and emotion detection, like a vice. Still, I need to show different types of analysis.

So what kind of advanced statistical analysis methods can be applied to this data set? My data looks like this:

  Keyword |Average Position|    Average CPC|    Clicks| CTR|    Cost|   Impressions

  android app developers    ,1  ,0.5,   21.79,  8.7%,   10.99,  250

This is what your post looks like to me:

"I have these data, what can I do with them?"

The answer is "A LOT". What do you want to do exactly? what is the goal?

From the topic, I assume your goal is to find the relations between "Keyword", "Average Position", "Clicks", "Cost", ... with "Impressions". If this is true, then you have multiple predictor variables and a target variable (Impressions) in which you would like to know the relation. This can be done through multiple statistical techniques such as:

Pearson Correlation
Spearman Correlation
Kendall Correlation
Mutual Information
Simple regression (and checking importance of each predictor variable)
RReliefF algorithm etc.


What is meaning of different symbols in image? I am learning about Deep Generative Models, tutorials all over the places use symbols and no one actually explaining the meaning of it. Can you please suggest the meaning of these terms?

In general in probabilities the symbol $\sim$ means "follows the distribution ..", for example:

$$X \sim N(3,7)$$

would mean that the random variable $X$ follows a normal distribution with mean 3 and std. dev. 7. In your example the symbol is used to define the notations apparently: the training data follows the distribution of the real data denoted  $p_{data}$ and the generated samples (whatever these are) follow the distribution calculated by the model denoted $p_{model}$.

The point seems to be about the distinction between the real distribution of the data and the distribution obtained by the model: naturally one wants the latter to represent the former as accurately as possible, that's the job of the estimation algorithm.

An alternative very common notation would be $p(x)$ for the true (i.e. data) distribution and $\hat{p}(x)$ for the estimated (i.e. model) distribution.


Evaluating Language Model on specific topic I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in the form of either a single keyword(e.g. 'Computers') or a bag-of-words(e.g. 'Computers', 'Linux', 'Server'...).
For example given a LM, how relative are the outputs of the model to the topic specified by the word Computers?
What I have already tried: Generating a large enough number of sentences from the LM and taking the average cosine similarity between these sentences and the target topic(or every word in that topic we have more than one) as described here . I am not sure if this is a valid way to go and furthermore the cosine similarity between sentences yields poor results in many cases.
Thanks in advance for any help.

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that.
However, you should take into account that it is possible that your model generates very similar sentences always. For that reason, people normally evaluate not only quality but also diversity. For this, you can compute the "self BLEU" of your generated sentences, that is, you compute the BLEU of each generated sentence against the rest of the generated sentences. You can find a script for that here.
You should also take into account that you can trade quality for diversity and vice versa, by setting different values of the output softmax temperature. For this, you may want to evaluate your model at different values of the temperature, to understand different generation regimes, plotting them as a curve, like this:

You can find more about this kind of evaluation in this article, where I took the figure from.
You can find alternatives to the quality-diversity evaluation here.


HMMLearn Predict Next Observed Event From my understanding you can use the transition matrix to predict the probability of going from the last predicted hidden state(state t), to the t+1 hidden state. My confusion is how in code format do I go from the hidden state predicted at time t+1 to the predicted observed state at time t+1.

For each state of continuous HMM, an emission probability distribution is defined. It can be a Gaussian distribution or a Gaussian Mixture Model (GMM). After computing the state variable, you can get the emission probability distribution of observed variables using corresponding state distribution. Note that there is no deterministic value for observed variables and just a probability distribution conditioned on the predicted state variable. In hmmlearn, you can get the means, covars and priors of the GMM using means_, covars_ and weights_ attributes of the model, respectively.


How do I create a new column to an existing data frame which is the product of other columns? I have data estimating what kind of food is used on a daily basis at school to feed kids.

Pizza   Apples  Lemonade  Lemon   Banana  Milk  Soup
 12        3       10       1      9      10     6
 14        5       8        0      13     11     7
 ....................................................

I want to create a new variable Fruits_total which is a sum of variables Apples, Lemon, and Banana, and add it to an existing data frame.

  df['Fruits_total'] = df.Apples + df.Lemon + df.Banana

However, it doesn't work. Where is my mistake?

Use the below code

df['Fruits_total'] = df[['Apples','Lemon','Banana']].sum(axis=1)

axis = 1, shows that you are summing up the rows. This is what you missed in your code.


Aggregate categorical feature by the target Having a list of triplets {X1,X2,Y} such as :

{pennsylvania, fever , malaria}
{pennsylvania, headache  , malaria}
{arizona, ketone smell  , flu}
{new york, fever  , cancer}
{ohio, hand pain  , trauma}

i have thousands of samples with states , symptoms and diagnoses.
i need to aggregate the states to reduce the dimension into smaller set of states using the symptoms and the diagnose. Any idea for that?

i started using Kmode algorithm for clustering the data into several clusters but i'm not sure if it makes any sense doing that

Interesting problem...

If I understand correctly you'd like to obtain clusters of states which have similar patterns/proportions of symptom+diagnosis, right?

If yes, I would suggest you reorganize the data so that one instance represents a state, with its features being the frequency of each pair (symptom, diagnosis). Based on this representation you could cluster states which have similar prevalence for a pair.

The disadvantage of this idea is that it considers pairs (symptom, diagnosis) as distinct even if only one of the two is different. Of course the same process could be done by considering only symptoms or only diagnosis. There are probably better approaches but this simple one might already provide some insight.


Can someone explain to me the structure of a plain Recurrent Neural Network? I have seen pictures of RNNs and LTSMs, and they usually look like this:

Here the task is to take a sentence and make a prediction of some sort.  What are each of the green squares?  Are each of them layers, or does each green square have it's own set of layers and neurons?
Additionally, in such a scenario, the input data is the entire sequence of words.  When dealing with time series data, would the input be the entire data set since the entire dataset is a sequence?  If you have 1million observations, you would need to construct 1million green squares?

The architecture of a RNN is called recurrent because it applies the same function at each step. So all the cells on the graph actually represent the same computation, but not the same state. Each green square in your figure represent the computation.
$$
s^{(t)} = f(s^{(t-1)}, x^{(t)}, \theta)
$$
Where $f$ is the function of the RNN, $\theta$ are parameters, $s^{(t)}$ is the RNN state at step $t$ and $x^{(t)}$ is the input at step $t$ in the input sequence.
What you see represented in the figure is actually what is called the unfolded representation, that is the same RNN cell applied to the input sequence one input vector at a time.
I recommend you to read the chapter 10 on RNN Deep Learning book. The following figure is from this book and summarize the idea.


Convolution and Cross Correlation in CNN What would be the intuition behind using the convolution and cross correlation operation inside Convolutional Neural Networks? I am interested in putting together the theory from Digital Image Processing where these 2x operations are defined, and CNNs. Could anyone help me connect the dots?

A neural network takes every input into a neuron and then through an activation function the neuron produces an output. When applying this method to images we can quickly see that this is not an ideal solution. The similarity of a pixel is much more related to a neighboring pixel.

The convolution operation in deep learning was used for this exact purpose. It is better to focus on the neighborhood of inputs before considering the correlation of that pixel with those on the other side of the image. So we can instead apply a mask that will mix the neighborhood of pixels.

Before we get into some theory, it is important to note that in CNNs although we call it a convolution, it is actually cross-correlation. It is a technicality, but in a CNN we do not flip the filter as is required in typical convolutions. However except for this flip, both operations are identical.

Discrete convolutions

From the wikipedia page the convolution is described as

$(f * g)[n] = \sum_{m=-\inf}^{\inf} f[m]g[n-m]$

For example assuming $a$ is the function $f$ and $b$ is the convolution function $g$,

To solve this we can use the equation first we flip the function $b$ vertically, due to the $-m$ that appears in the equation. Then we will calculate the summation for each value of $n$. Whilst changing $n$, the original function does not move, however the convolution function is shifted accordingly. Starting at $n=0$,

$c[0] = \sum_m a[m]b[-m] = 0 * 0.25 + 0 * 0.5 + 1 * 1 + 0.5 * 0 + 1 * 0 + 1 * 0 = 1$

$c[1] = \sum_m a[m]b[-m] = 0 * 0.25 + 1 * 0.5 + 0.5 * 1 + 1 * 0 + 1 * 0 = 1$

$c[2] = \sum_m a[m]b[-m] = 1 * 0.25 + 0.5 * 0.5 + 1 * 1 + 1 * 0 + 1 * 0 = 1.5$

$c[3] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0.25 + 1 * 0.5 + 1 * 1 = 1.625$

$c[4] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0.25 + 1 * 0.5 + 0 * 1 = 0.75$

$c[5] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0 + 1 * 0.25 + 0 * 0.5 * 0 * 1 = 0.25$

As you can see that is exactly what we get on the plot $c[n]$. So we shifted around the function $b[n]$ over the function $a[n]$.

2D Discrete Convolution

For example, if we have the matrix in green

with the convolution filter

Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. Very much like the wikipedia page shows, this kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$.

You will notice that there is no flip of the kernel $g$ like we did for the explicit computation of the convolution above. This is a matter of notation. This should be called cross-correlation, it is not a true convolution. However, computationally this difference does not affect the performance of the algorithm because the kernel is being trained such that its weights are best suited for the operation, thus adding the flip operation would simply make the algorithm learn the weights in different cells of the kernel to accommodate the flip. So we can omit the flip.


Which service could I use to train my networks? My laptop's Intel i7 3630QM 2.4GHZ, 8Gb RAM and GXForce 670M are clearly not sufficient...

By reading some papers, I've written an SRGAN with Python Keras. At runtime there is no error but training only 2 images (324*324) with 1 epoch, batch size of 2 lasts too much...perhaps more than 1 hour of training...

Which service could I use to train my deep neural networks, see the console outputs, eventually see plots, and download the trained model?
Are there such paid services? Free services?

The idea would be that I query the compute server from my laptop.

You can use AWS EC2, or the GCP Compute Engine, or the GCP Machine Learning Engine. You pay based on how many CPU/GPUs you employ and on training times.


Package/function to convert the word to one hot vector in Python NLP Is there a package or function in NLP which can be used to convert the word into an one hot vector.

Thank you.

There are several ways to convert words to one hot encoded vectors. Since I do not know the data structure you store your data. I assume it is going to be a list

  from keras.preprocessing.text import Tokenizer
  samples = ['The', 'dog','mouse','elephant']
  tokenizer = Tokenizer(num_words=len(samples))

This builds the word index

  tokenizer.fit_on_texts(samples)

one hot representation

  one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')

By changing the mode from binary to 'tfidf' or 'count', you can make a matrix of any type, apart from one hot.

You can achieve the same result using other packages like sklearn. But it does involve a bit more lines of code.


Representing state in Q-Learning I have a fairly simple game in which I wish to use Q-learning to train an agent, but I have some questions regarding state representation. I'm new to RL so bare with me:

If you have a game where you and your enemy each have 3 players, each player has health and a location on a 10 x 10 grid. My understanding is that you would need to enumerate all possible states for this game in order to successfully implement Q-learning. How would this be done? If we forget about health for a second and we look at position alone, we have 6 players, each of which could be in any of the 100 locations so our state space for position alone would be 100^6. This seems like an exceptionally large matrix to be working with, and we haven'y even added a parameter for health. What am I missing? is this the point at which you look for alternative state representations?

Some guidance or links to relevant articles would be greatly appreciated. Thanks!

If we forget about health for a second and we look at position alone, we have 6 players, each of which could be in any of the 100 locations so our state space for position alone would be 100^6.

Yes, that is correct, adding health in, and assuming it was an integer from 1 to 20, then you would have $20^6 \times 100^6$ discrete states.

  My understanding is that you would need to enumerate all possible states for this game in order to successfully implement Q-learning. How would this be done?

For the enumerated case with enough resources, there is a theoretical guarantee that Q learning (and other RL methods) will converge on the optimal result. However, in this case - and many others - there are clearly too many states to calculate or store values for in detail. So the answer is that you don't do that. Instead you need look into ways to calculate approximate calculating action values which you hope will also generalise to previously unseen state/action pairs, at the very least to unseen states and actions that are relevant in real games.

There are many ways to structure the state and action representations to help with approximation in RL. Sutton & Barto's Reinforcement Learning: An Introduction dedicates the whole of the second (of three) sections to approximation in general, and chapter 9 part 5 reviews some popular approaches to representations that work well with simpler linear function approximation.

This kind of approximation is also the main driver behind "Deep" Q-learning (DQN), which is essentially Q-learning with a neural network approximating the action value table. There are some details to add to get this to work well - but plenty of resources available to help with this on the internet: Tutorials, examples and fully coded agents ready to set up and solve problems.

So I would recommend moving to DQN when the state/action space gets too large for a simple tabular approach, and where it is not clear what other simpler approximation could get the job done more efficiently. If you do that, then the state and action representation becomes what will work as inputs to a neural network. There are some rules to follow for that. First, all inputs should be on a suitable scale. Neural networks like to work with inputs with mean 0, standard deviation 1 over the sample population. Often you don't know that for a RL problem, so you can go to the next best thing - scaling to fit in a range such as 0..1 or -1..1

I have also written an answer covering approaches to choices in state representation and feature engineering for another problem  Reinforcement learning: easily learnable state representation which might help.


adjust output from normalization? I've trained a neural network network that given a minmax normalized input, provides a minmax normalized output.

this might be late, but is it possible from a minmax normalized output to create the actual output, given you know the actual min and max value?

so an unormalized output?

Yes, just rearrange the formula for finding the normalised value. Where $x_i$ is the original attribute, and $z_i$ is the normalised value:

$$
\begin{align}
  z_i &= \frac{x_i - min(x)}{max(x) - min(x)} \\
  x_i &= z_i(max(x) - min(x)) + min(x)
\end{align}
$$


Terminology to distinguish between ML methods and optimization methods (PSO, ACO..) I am currently writing a scientific thesis which consists of two parts. In the first part I am building ML models with neural networks, support vectors etc. and the second part is about finding global minima for optimization objectives with optimization methods like Particle Swarm Optimization, Ant Colony Optimization, Simulated Annealing etc.
When writing about the parts in the introductory part, I would write machine learning methods when referring to the first part and metaheuristic optimization methods when referring to the second part.
Is it possible to distinguish them like this?

In a wider context every machine learning method can be re-cast as some type of optimisation problem. For example for Neural Networks the associated optimisation problem is "find the weights which minimise some loss function of the data given an architecture". This is solved using back-propagation (which is a layered gradient-descent method) and when minima of the loss function are found we say the systen has "learned".
On the other hand there is nothing stopping us from re-casting optimisation problems as "learning" problems
.
A possible differentiation (based on what you mention in the question) is between analytic methods and non-analytic methods.
A. Analytic Methods: Gradient-based, Primal-Dual etc... (eg NNs, SVMs)
B. Non-Analytic Methods: Particle systems, Genetic/Evolutionary systems, Simulation methods, Stochastic methods, etc..
The basic differrence is both whether the form of the objective function is known and if analytic methods (eg gradient-descent vs particle systems) are used to find optima.
Again, there is nothing, in principle, stopping us from using these non-analytic optimisation methods to do machine learning.


Analyzing time series association I am pretty new for time series analysis and I would like to share one of my research questions.

Here is the graph:

Apparently, both 1 and 2 are time series data. 1 represents monthly humidity and 2 represents a morbidity rate.

I would like to analyze if the morbidity rate (2) is associated with seasonal humidity (1), what method is the best in this scenario? Also, from the visualization, it seems like the morbidity peak is about 3 months after the peak of humidity.

Can anyone give me some suggestions of this question?

Thank you for any replies!

Here's some ways what I would try:

Shift humidity for n days, which makes Pearson correlation coefficient minimize.
Extract frequencies using Fourier Transform etc.
Do power spectral density estimation using Welch's method etc.


Pass 2 different kinds of X training data to ML model simultaneously I'm trying to classify if a book is fiction/nonfiction based on title and summary.

This is 2 distinct types of information - is there a way to segment title and summary before feeding it to a model, rather than concatenating the information?

For example:

Title: "such a long journey"

Summary: "it is bombay in 1971, the year india went to..."

Label: "fiction" (where fiction =1)

Current procedure:

What I've been doing until now is concatenating the information, so the above becomes,

example = "such a long journey it is bombay in 1971, the year india went to..."
label = 1

Then the usual setup, something like,

X.append(example)
y.append(label)
...
X = lemmatize(X)
...
X_train, X_test, y_train, y_test = split_data(X,y)

vectorizer = TfidfVectorizer(...)
X_train = vectorizer.fit_transform(X_train)
X_test = vectorizer.transform(X_test)

classifier.fit(X_train, y_train)
y_predict = classifier.predict(X_test)

But feeding the data concatenated feels intuitively wrong. Is there a better way to do this?

If for some reason its possible with a library other than sklearn (keras, tensorflow) I'd be also open to hearing about that.

UPDATE

Going from,

X = ['two'],['two'],['four'],['two'],['four'],['four']]
y = ['human','human','dog','human','dog','dog']

to,

X = [['two','hello'],['two','hello'],['four','bark'],['two','hi'],['four','bark'],['four','woof']]
y = ['human','human','dog','human','dog','dog']

causes errors to be thrown.

'list' object has no attribute 'lower' is X is a list, and 'numpy.ndarray' object has no attribute 'lower' if X is an array.

The error is thrown when I call,

X_train = vectorizer.fit_transform(X_train)

Is it possible to pass in a vector of features?

You could just apply two independent vectorization steps on your input X (one vectorizer for description and another for summary) and then concatenate the obtained feature matrices into a single feature matrix.

Doing it this way you will have features such as description_"such", description_"long", ..., summary_"bombay", summary_"1971", ..., so any model you apply will be able to:

Use all the features from the description and summary altogether
Give a different weight to desctiption tokens and summary tokens