Is there a way to recognize which of these scatter plots is "better"? I doubt better is the correct adjective, apologies for that. What I mean is this:
I have a set of files (1200~) each paired with a scatterplot image. I need to find a way to classify which data files will have scatter plots that a person would classify as "clearly separated data"(again, not the right words), and which ones as "not clearly separated".
For example:

The first and second scatter plots are examples of data that would end classified as "good" or "clear", while the third one would be classified as "unclear". Are there metrics or distances that could be applied on the data to predict wheter the plots generated by that file would be of either the first kind or the second?

Thanks for taking the time to read this. I'm new to this, and english is not my first language.

You could:

Label manually few of them (say 100-150), then train a simple model to classify data. A small Random Forest could do the job well.
Train a super basic model on the each dataset used to produce each scatterplot. Something like a Linear Classifier. If the Classifier doesn't make mistakes, you have "clearly separated data", if it makes mistakes then it's likely to be the opposite.


How to solve this ? Value Error : operands could not be broadcast together with shapes (0,) (100,522) I am trying to apply a PSO algorithm to train a neural network applied to "User Identification From Walking Activity Data Set" problem. Can be found here.

So i extrated the dataset, and calculated average value and standard deviation between aceleration x,y and z at every 5 seconds for each person that would be the input of my neural network. After this i try to optimize the weights and bias of the neural network with Pyswarms functions, but i get an error which i dont know where it is coming from.

The Code :

import numpy as np
import statistics as st
import pyswarms as ps
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier

#EXTRACTING THE DATASET AND CREATING INPUTS AND TARGETS/LABELS
def load_input():
  data = np.empty((0,3),float)
  for x in range(1,23):
        fich = np.loadtxt("%d.csv" %x,delimiter=",");
        aux = []
        value = 5.0;#5 seconds for each person
        for y in range(0,len(fich)):
            if fich[y][0] <= value :
                aux.append(fich[y,1])
                aux.append(fich[y,2])
                aux.append(fich[y,3])
            elif fich[y,0] > value or y == len(fich):
                data = np.append(data,np.asfarray([[np.mean(aux),np.std(aux),x-1]]), axis =0);
                y = y-1;
                aux[:] = []
                value = value + 5;
  return data

#OPTIMIZING WEIGHTS AND BIAS
def forward_prop(params):
 n_inputs = 2
 n_hidden = 20
 n_classes = 22

 W1 = params[0:40].reshape((n_inputs,n_hidden))
 b1 = params[40:60].reshape((n_hidden,))
 W2 = params[60:500].reshape((n_hidden,n_classes))
 b2 = params[500:522].reshape((n_classes,))

 z1 = X.dot(W1) + b1 # Pre-activation in Layer 1
 a1 = np.tanh(z1) # Activation in Layer 1
 z2 = a1.dot(W2) + b2 # Pre-activation in Layer 2
 logits = z2 # Logits for Layer 2

 exp_scores = np.exp(logits)
 probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)
 # Compute for the negative log likelihood
 N = 923 # Number of samples
 corect_logprobs = -np.log(probs[range(N), Y])
 loss = np.sum(corect_logprobs) / N
 return loss

def f(x):
 """Higher-level method to do forward_prop in the
 whole swarm.
 Inputs
 ------
 x: numpy.ndarray of shape (n_particles, dimensions)
 The swarm that will perform the search
 Returns
 -------
 numpy.ndarray of shape (n_particles, )
 The computed loss for each particle
 """
 n_particles = x.shape[0]
 j = [forward_prop(x[i]) for i in range(n_particles)]
 return np.array(j)

##### MAIN FUNCTION #####
data = load_input();
X = data[:,[0,1]];
Y = data[:,2].astype(int)
# Initialize swarm
options = {'c1': 0.5, 'c2': 0.3, 'w':0.9}
# Call instance of PSO
dimensions = (2 * 20) + (20 * 22) + 20 + 22
optimizer = ps.single.GlobalBestPSO(n_particles=100,
dimensions=dimensions,options=options)
# Perform optimization
cost, pos = optimizer.optimize(f, print_step=100, iters=1000, verbose=3)

Examples of Inputs and target matrixes :

This should be on stackoverflow, with a precise description of the error (line, traceback...), but the error seems to be here:

np.empty((0,3),float)

And then not appending the data as expected.

If you are doing this, use a list, and then at the end, transform your data in a numpy array:

data = []

for...
    data.append(new array)
data = np.array(data) # add the Fortran flags if you need to

There would be just one memory copy instead of O(n-squared) with np.append.

Also the shape for X or Y (?) seems to be 2D ((100,522)), where as you only have 1D. Check your data (which we don't have, so impossible for me to tell you exactly where to look for) so that you can ensure that the dimensions are correct.


Is it always better to using stacked LSTM than single LSTM? I am currently studying LSTM and RNNs.

I came across several concepts like Multidimensional LSTM and Stacked LSTM.

I have used Stacked LSTM and it gives me a better performance than single LSTM. As per my understanding, if I increase the depth of LSTM, the number of hidden units also increases. It means overfitting, right? Then why am I getting better results?

[Note: I have used BatchNorm and Dropout after every stack of LSTM ]

From https://machinelearningmastery.com/stacked-long-short-term-memory-networks/:

"Stacking LSTM hidden layers makes the model deeper, more accurately earning the description as a deep learning technique ... The additional hidden layers are understood to recombine the learned representation from prior layers and create new representations at high levels of abstraction. For example, from lines to shapes to objects ... A sufficiently large single hidden layer Multilayer Perceptron can be used to approximate most functions. Increasing the depth of the network provides an alternate solution that requires fewer neurons and trains faster. Ultimately, adding depth it is a type of representational optimization."

Increasing the number of layers/hidden units in a neural network doesn't necessarily result in overfitting. Too few will result in low training and test accuracies; too many will result in high training accuracy but low test accuracy (overfitting). Somewhere in the middle there will be the right amount of hidden layers and units for the problem. Some complex problems like NLP require a number of stacked hidden LSTM layers http://ruder.io/deep-learning-nlp-best-practices/.

This thread might be useful: https://ai.stackexchange.com/questions/3156/how-to-select-number-of-hidden-layers-and-number-of-memory-cells-in-an-lstm


Product classification based on Description of a product in eCommerce website I am searching for an eCommerce dataset which should have a product category and description for my academic research in Machine learning. Please suggest any public dataset available.

INPUT: UNIONBAY Men's Survivor Iv Relaxed Fit Cargo Pant-Reg and Big and Tall Sizes

OUTPUT: Clothing

http://jmcauley.ucsd.edu/data/amazon/

Who else but Amazon!

(the exact link you want I believe) -
http://jmcauley.ucsd.edu/data/amazon/amazon_readme.txt


non query-based document ranking We have ~500 biomedical documents each of some 1-2 MB. We want to use a non query-based method to rank the documents in order of their unique content score. I'm calling it "unique content" because our researchers want to know from which document to start reading. All the documents are of the same topic, in the biomedical world we know that there is always a lot of content overlap. So all we want to do is to arrange the documents in the order of their unique content.

Most Information Retrieval literature suggest query-based ranking which does not fit our need.

You could use Topic Modeling as described in this paper:
http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf

They performed Topic Modeling on abstracts of patents (limited to 150 words).  They identified papers as "novel" if they were the first to introduce a topic, and measured degree of novelty by how many papers in the following year used the same topic. (Read the paper for details).

I suggest that you follow their lead and only process paper abstracts.  Processing the body of each paper might reveal some novelty that the abstract does not, but you also run the risk of having much more noise in your topic model (i.e. extraneous topics, extraneous words).

While you say that all 500 papers are on the same "topic", it's probably safer to say that they are all on the same "theme" or in the same "sub-category" of Bio-medicine. Topic modeling permits decomposition of the "theme" into "topics".

The good news is that there are plenty of good packages/libraries for Topic Modeling.  You still have to do preprocessing, but you don't have to code the algorithms yourself.  See this page for many resources:
http://www.cs.princeton.edu/~blei/topicmodeling.html


When I add regularization like L1,L2 , do I need more epochs to properly train my model? When I add regularization techniques in my model like L1 or L2 do i need more epochs to properly converge my model.
for r in (None,"L1","L2"):
        for max_iter in (30,45,60):
            classifier=SGDClassifier(loss="log",penalty=r,max_iter=max_iter,learning_rate="constant",eta0=0.01,random_state=42)
            print("max_iter={}".format(max_iter))
            classifier.fit(X_train,Y_train)
            acc=classifier.score(X_test,Y_test)
            print("accuracy when r={} is {}".format(r,acc*100))

When r = None:

max_iter = 30/45 it says ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
max_iter = 60 no warning.

When r = L1:

max_iter= 30 same warning.
max_iter = 45/60 no warning.

When r= L2:

max_iter = 30/45/60 same warning

Does it matter or this is random?

The convergence time is sensitive to the data you have and a random seed. Specifically, the convergence time is linear in expectation in all three cases. SGDClassifier uses the stochastic gradient descent for optimization. Since L1 loss is only subdifferential, the L1 penalty causes the algorithm to converge noticeably slower.
Comparing with or without the L2 penalty, it is not clear what algorithm is faster. The loss function is differential. The L2 penalty may be faster in the underdetermined case. In the example below, I consider the gradient descent instead of the stochastic linear descent and regular regression to simplify the argument. Say, we aim to solve y = Xb + e, where we observe y and X only. We set the loss function to be f(b) = 0.5||y - Xb||^2. Without regularization, the solution is sol1 =(X^TX)^{-1}X^Ty and with L2 regularization, the solution is sol2 = (X^TX + lambda I)^{-1}X^Ty. In the latter case, we can guarantee that the matrix to invert is not close to singular, and, therefore, the faster convergence is expected.
In short, on average, I would expect the following number of iterations requires from smallest to largest ON AVERAGE:

L2 penalty
No penalty (potentially, with a close tie with L2 penalty)
L1 penalty

You observe the opposite order. It should be very specific to your data or random seed.


Keyword/phrase extraction from Text using Deep Learning libraries Perhaps this is too broad, but I am looking for references on how to use deep learning in a text summarization task.

I have already implemented text summarization using standard word-frequency approaches and sentence-ranking, but I'd like to explore the possibility of using deep learning techniques for this task. I have also gone through some implementations given on wildml.com using Convolutional Neural Networks (CNN) for sentiment analysis; I'd like to know how one could use libraries such as TensorFlow or Theano for text summarization and keyword extraction. Its been about a week since I started experimenting with Neural nets, and I am really excited to see how the performance of these libraries compares to my previous approaches to this problem.

I am particularly looking for some interesting papers and github projects related to text summarization using these frameworks. Can anyone provide me with some references?

The Google Research Blog should be helpful in the context of TensorFlow.

In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text summarization.

The 2014 paper by Sutskever et al titled Sequence to Sequence Learning with Neural Networks could be a meaningful start on your journey as it turns out that for shorter texts, summarization can be learned end-to-end with a deep learning technique.

Lastly, here is a great Github repository demonstrating text summarization while making use of TensorFlow.


Sentiment Analysis Using Neural Networks with Python I'm working on a task for an assignment for school. The course doesn't have a textbook and instead uses the tutorial on DataCamp. I've struggled severely using DataCamp to learn (been through it multiple times) and instead have found that written tutorials help me learn much better and quicker (at least with machine learning models). I usually use Medium but haven't found anything useful on there, so far.
I was curious if there's a tutorial that might help me with the following task.
You will build a neural network designed to learn word usage and context using NLP techniques. You will provide visualizations and a report, as well as build your network in an interactive development environment.

Here's how to create such a network. Convert your text into numbers using some tokenizer (Tensorflow's Keras has one). Break it off into equal blocks of some size x. Feed these blocks into an Embedding layer followed by LSTM or GRU (or their bi-directional variants) and finally have a three way classification (positive, neutral or negative) or for regression just a single sigmoid.
One trick though if you have class imbalance, train with full data in the beginning (some initial epochs). Then, for the further iterations, drop the neutral data, just train on positive and negative examples (since, you mostly have a lot of neutral data in these cases).


How would one impute missing values for a Discrete variable? How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.

It depends, if you have the distribution of that feature, you can take the marginal distribution over that feature which its interpretation is to use the expected value of that feature. If you don't have the distribution you can take the mean of the sample in hand of those samples which have value for that feature and add the mean for those which don't have. Another solution is to separate the data of each class and find the mean of the feature of those data samples having the value and putting the mean for each entry of the corresponding class which does not have value in that entry.


After plotting out the density of the dependent varaible, flux, I would recommend exploring a Generalized Linear Model.

flux appears to be the output of a gamma distribution or perhaps a log-normal distribution. I would recommend exploring a GLM, especially with the lme4 package in R. GLMs are convenient transformations of the prototypical linear regression, and thus are easily interpreted. They are flexible and well-founded in theory as well.

Is there any work which might suggest a data generating process for your problem? You might ask yourself things you know about the process: is it true that x values are always greater than 0? Is there an upper bound?


Principal components analysis with compositional data Another beginner question: I'm trying to do PCA on compositional data. In other words, all the variables in the group add up to 100%.

I've since learned on this forum that compositional data poses a problem for linear regression. Will compositional data also pose a unique problem when doing PCA?

I'm in the same situation as you.  PCA seems to have some difficulties for compositional data.  I've listed some papers below that have been helpful to me.  I'm still trying to understand everything, but these papers have a good overview I think.  Or googling "Compositional Data" or "Compositional Data Principal Component Analysis" returns a lot of other papers that are freely available and informative.

Principal component analysis of compositional data J. Aitchinson

A Concise Guide to Compositional Data Analysis J. Aitchinson

Principal components analysis for compositional data with outliers P. Filzmoser, et al.


How to convert 200 bag of word features to several continuous features I am using one text dataset for regression task. The baseline regression method (name concealed) that I employ will work well if all the input features are continuous.  However, the text dataset that I am considering has 200 bag-of-word features that are binary (0, or 1), i.e the first feature will be 1 if the associated word appearing on the text and 0 otherwise. I am just curious how can I convert those 200 binary to one or several continuous features such that my regressor can work with.
Thanks

Latent dirichlet allocation (LDA) would be a good one to try, as it is tailored for bag-of-words representations. LDA is a topic model, which basically means that it assigns $n$ topics, and figures out the words that associate with each topic. The output is a vector of $n$ continuous values where each value indicates membership to the corresponding topic. You should be able to find an implementation of LDA for whatever language you're using easily (but don't confuse it with linear discriminant analysis, also commonly referred to as LDA).

You could also try principal components analysis or singular value decompositions, which are other standard dimensionality reduction techniques.


Calculation of Output in LSTM Many-to-One Architecture I'm new to Recurrent Neural Network but I want to train my data with LSTM but I'm having a trouble to understand LSTM Many-to-One architecture.
Suppose the size of my data is time_step x num_features say 2 x 2 and I have to use many-to-one LSTM architecture because I want to do classification. So in the last time_step I have to add dense (a) which contain sigmoid activation function to predict the sequence class which is 0 or 1.
My questions are,

When I compute a, do I need to include all the hidden state (h1 and h2) or just the last hidden state h2?
If I just include the h2, how do I calculate the derivatives of the loss function (cross entropy) w.r.t h1?

The derivation for h1 is highly recommended. Thank you :)

While training, a set of training examples will be provided in a batch. At end of each batch, weights for all layers are updated (Dense and LSTM).

https://adventuresinmachinelearning.com/keras-lstm-tutorial/


What is generative and discriminative model? How are they used in Natural Language Processing? This question asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? How are generative and discriminative models used in NLP?

Let's say you are predicting the topic of a document given its words.

A generative model describes how likely each topic is, and how likely words are given the topic. This is how it says documents are actually "generated" by the world -- a topic arises according to some distribution, words arise because of the topic, you have a document. Classifying documents of words W into topic T is a matter of maximizing the joint likelihood: P(T,W) = P(W|T)P(T)

A discriminative model operates by only describing how likely a topic is given the words. It says nothing about how likely the words or topic are by themselves. The task is to model P(T|W) directly and find the T that maximizes this. These approaches do not care about P(T) or P(W) directly.


Automatic question categorization when we know important words in each category I am currently working on a question categorization problem where I automatically want to assign a category to the question. The question set I have is unlabelled. The categories for the problem are well defined e.g. Business, Technology, Sports etc. I also have a specific set of words which are important for each of the category (which have been taken from the dataset) and which define the respective categories. Using these information, which learning algorithm can I use wherein I can specify these categories and the important words for that category in context of the unlabelled data that I have?

Repsonses are highly valued.
Thanks in advance.

You said you have a bunch of keywords for each category extracted from the data. It is better to explain how did you get them as your data does not contain labels i.e. how do you know the word $w_i$ is a representative of category business?

You might have assigned them based on semantics which is not bad but does not give you a level of confidence that they are really the representatives of category. The whole point about clustering is to find this representative words via the algorithm. An example is the famous concept of Stop Words. Everyone thinks that words such as $The$, $is$, $an$ and the rest of common lists are stop words but empirically you always get better classification result including them! That's because in NLP the only reference to indicate a stop word is the data itself. So be careful about it.

Assuming your words are really representatives (however I would rely on LDA results), I would design a score for similarity of each question to each category representation list. It could be the category which has the maximum number of representatives in the sentence. It most probably falls in overfitting problem however you will not see that (your problem is unsupervised) but if you insist using them it could be a starting point.

In the direction of this idea, another one could be to extract the very extreme words for each category (e.g. "Manchester united" and "Neymar" are more probable to represent sport) then do the same clustering with only extreme words and based on found categories find more representatives and feed them to the algorithm again (iterative manner) and continue till some stopping criterion is satisfied. This looks like above but the difference is that the least input (first set of words) are fed manually and the rest will be found through algorithm.

You also may have a look at embedding algorithm such as word2vec or doc2vec and use them with some similarity measure (cosine similarity for example).


In a list, find the numbers where they are bigger than the next numbers I'm using Rstudio. I have a list like below, I want to find all the numbers that are bigger than the next numbers (i.e. numbers on their right hand, I make them bold).

7  7  9  9  9 10 10 1 51 51 53 58 58 58 58 59 59
62 64 67  0  0  3  3 12 12 12  1  4  4  5  5  9 21 25 25 27
27 29 33 33  49 50  0  0  0  3  4  4  8  8  8 21 21 21 21 21 24 25 25 26 27

Thanks anyone who kindly offers an idea!

I think the following is the easiest and fastest way:

Create a new vector with the "next neighbors"
Subtract the "neighbors" vector from the original vector
Determine which entries in the new vector are positive

I've actually never used R, but here's how I think it would work:

original_list <- c(7, 7, 9, 9, 9, 10, 10, 1, 51, 51)
# extract the "right neighbors" of the original list elements
neighbors <- original_list[-1]
# fill the missing position after the last element
neighbors <- append(neighbors, 0)
# compute the difference between the original vector and the right-neighbors
diff <- original_list - neighbors
# find the indices of positive elements
idx <- which(diff > 0)

# your output: elements which have a smaller right-neighbor
output <- original_list[idx]


Will stochastic gradient descent converge for multivariate linear regression I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1).
My guess is yes, based on the fact that stochastic gradient descent will converge for most well-behaved, convex functions, but the multivariate aspect is throwing me off, as well as the batching.
What is a good way to think about this problem, from both the multidimensional aspects as well as the hyperparameters?

For an MSE Loss, you will have a convex surface

Gradient Descent on a convex function is guaranteed to converge if the learning rate is not too high. The only that can happen is a high enough learning rate which diverges the learning

Impact of one record per learning step - Light on computing but a zigzag gradient

Working on a single instance at a time makes the algorithm much faster because it has very little data to manipulate at every iteration.
On the other hand, due to its stochastic (i.e., random) nature, this algorithm is much less regular than Batch Gradient Descent: instead of gently decreasing until it reaches the minimum, the cost function will bounce up and down, decreasing only on average.
$\hspace{2cm}$[Ref] - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow,by Aurelien Geron

Impact of multi-variate gradient -
Multi-variate means, now you have multiple participants to decide the resultant Gradient. Everything will be the same and the learning will follow the net Gradient path.
One thing that will change is that with a high dimension, we have a long and flat surface in most of the places in the Loss space which can slow down the learning if the learning rate is not scheduled. Convergence is still guaranteed if LR is not too high.
For Intuition -
For more than 3-D, you will have to trust on the extrapolation and vizualize in your own way.
$\hspace{3cm}$
$\hspace{3cm}$Individual record Gradient and Overall gradient - 1 Feature
$\hspace{3cm}$
$\hspace{6cm}$A Loss surface in 3-D - 2 Features
$\hspace{3cm}$[3-D plot credit] - https://www.monroecc.edu/faculty/paulseeburger/calcnsf/CalcPlot3D/


Reducing noisy data from non normal distribution of data with std deviation? I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks

First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .

Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?

Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.

If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.


Solutions for Labelling Training Data for Binary Classification Problems I have a huge dataset for which I am trying to use an 80-20 (Holdout method) approach to train and test my model. However, the dataset I have been given has 6m rows. The objective is to train+test+validate the model before using live data traffic for real-time predictions.
The expected result here is "It's not corrupted with 97% accuracy" which is implementation details and output of some Jupyter notebook etc.
My Question is - Is there any alternatives than manually labelling such a big dataset?
By manually labelling - I mean a human (or a group) going through all the 6m rows(!). Also, not all input strings have identical contents so it's hard to just push it through some script/csv and automate it. But I am trying to understand if this is the ONLY way.

Ofcourse not. Here is a simple possible solution.
Do unsupervised learning. If you do it good and efficiently you will only see these two groups in your data (binary classification). And your silhuette score will be high. Hence you can automatically than label these groups/clusters.


Resources on Data Science for Football / Soccer? I am searching for resources on data science projects which used football/soccer data.

There's this website called Sportsdatamart which allows you to
download CSV and XLS file formats for a number of Football series
like the England Premier League, the Italian Serie A, the German
Bundesliga and the Spanish La Liga.
On Football Data UK, you can find EXCEL and CSV data files to
use for quantitative testing of betting systems in spreadsheet
applications. League tables, head2head statistics and information on
goalscrores, first scorers and top scorers could be accessed through
the Livescore service.
Soccer Wiki is a collaborative database and anyone can create
and edit data. This community driven database contains information
on players, clubs, stadiums, managers, referees, leagues and other
data related to the world of soccer.
If you are looking for data sets centered on European Football,
check this out.


Tutorials on topic models and LDA I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with some real examples.

If you're working in R, Carson Sievert's tutorial on using LDA to model topics in movie reviews is an excellent starting point:

http://cpsievert.github.io/LDAvis/reviews/reviews.html

This tutorial makes use of LDAvis, an interactive visualization of topic and word distributions that can really aid intuition.

Also, although not short, David M. Blei's lectures on topic models are a great resource for understanding the meaning behind the parameters: http://videolectures.net/mlss09uk_blei_tm/


How can you build a model that extracts data out from receipts? I'm trying to build a model that is capable of identifying information on receipts and invoices.

I have used google cloud vision api for text extraction from the receipt but the problem is it just returns all the text from a receipt. I am looking to build a model that returns only a certain fields such as total price, brought alcohol etc.. from a receipt.

I could parse the text to extract by hard coding things but it's not optimal I think. Is there any way to build something for this use case. I need just something to go on.

Comment --- Output:

'Berghotel\nGrosse Scheidegg\n3818 Grindelwald\nFamilie R. Müller\nRech. Nr. 4572\nBar\n30.07.2007/13:29:17\nTisch 7/01\nNM\n#ರ\n2xLatte Macchiato à 4.50 CHF\n1xGloki\nà 5.00 CHF\n1xSchweinschnitzel à 22.00 CHF\n1xChässpätzli à 18.50 CHF\n#ರ #ರ #1ರ\n5.00\n22.00\n18.50\nTotal:\nCHF\n54.50\nIncl. 7.6% MwSt\n54.50 CHF:\n3.85\nEntspricht in Euro 36.33 EUR\nEs bediente Sie: Ursula\nMwSt Nr. : 430 234\nTel.: 033 853 67 16\nFax.: 033 853 67 19\nE-mail: grossescheidegg@bluewin.ch\n'

The simplest pipeline would be to do the following:

OCR
Named Entity Extraction
Entity Disambiguation

OCR

This is basically transforming your receipts into plain text.

If you have scans (pictures) of the receipts, then you need a method that can deal with images. For example, you could use tesseract.

Named Entity Extraction

This is detecting the parts of interest in the text. This includes detecting dates, prices, currency, locations, names, and so on. Once you detect a named entity, for example: price=2.59, you still don't know if that is the unit cost, or total cost after tax, or total cost before tax, but that will come in the next step. All you know that this is an area of interest.

There are tools that are able to detect most named entities that you might want to use. For example: spaCy. The cool thing about spaCy is that you can train it to recognize more named entities that you might want.

Entity disambiguation

This is identifying the purpose of the named entity that you extracted. For example, given price=2.59, is it the unit cost, or total cost after tax, or total cost before tax.

This is where rules might come in handy, because it will be difficult to teach a machine learning model to identify them. You can have rules like:

highest value = total cost after tax
unity quantity × unit price = total cost before tax
total cost before tax × 1.21 = total cost after tax
and so on...


Does Gradient Boosting detect non-linear relationships? I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn.

My questions are:

1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2, y increases as x approaches negative infinity and positive infinity. What if the graph looks like y=sin(x)?

2) Is the algorithm able to detect interactions/relationships among the features? Specifically, should I add features that are the sums/differences of the raw features to the training set?

GB method works by minimizing a loss function and by splitting each node in a fashion that produces high pure leaves. there is no population formula being estimated and therefore you can estimate all types of relations between the target and the features.
However I wouldn't put in the model correlated variables as:

  For gradient boosted trees, there's generally no strong need to check
  for multicollinearity because of its robustness. But practically
  speaking, you still should do some basic checks. For example, if you
  discover that two variables are 100% the same, then of course there's
  no point in keeping both. Even if it's 98% correlated, it's usually
  okay to drop one variable without degrading the overall model.
  Source: Quora


transfer learning with sentiment analysis? The question is how good and what are some things to keep in mind when sentiment analysis models are tested on different datasets than they are trained on.

Say the task is to perform sentiment analysis on product reviews (unlabeled datset) - to classify positive, negative or neural. Because the data is unlabelled, a model can be trained (perhaps using logistic regression or NN) on a similar labeled dataset (say movie reviews, or product reviews) and tested on the original unlabelled dataset.

Will something like this work? Because the words of product names that occur in the unlabelled dataset will not be words that the model was exposed to during training, during test time will these words possibly throw off the model?

I can not fully answer your questions, but would like to offer a couple of my thoughts here:
1) Transfer learning for sentiment analysis can be hard given that knowledge learned from one topic may not be not broad or general enough to perform well on the target or downstream tasks. For example, I have recently trained a neural network along with Word2Vec embedding using Twitter airline customer review data and get an prediction accuracy of 77%. However, when I use the same Word2Vec and neural network to classify some general customer review data, I got an prediction accuracy of only 35%.

2) Transfer learning in natural language processing is a hot topic and many researchers are working on it recently. 2018 sees some breakthroughs in transfer learning, i.e., Google Universal Sentence Encoder, BERT algorithm etc. I can not give you a comprehensive list here since I'm also learning. I would suggest you dive into some blog articles or even the original research articles to get a better understanding.

May it helps.


Which learning tasks do brains use to train themselves to see? In computer vision is very common to use supervised tasks, where datasets have to be manually annotated by humans. Some examples are object classification (class labels), detection (bounding boxes) and segmentation (pixel-level masks). These datasets are essentially pairs of inputs-outputs which are used to train Convolutional Neural Networks to learn the mapping from inputs to outputs, via gradient descent optimization. But animals don't need anybody to show them bounding boxes or masks on top of things in order for them to learn to detect objects and make sense of the visual world around them. This leads me to think that brains must be performing some sort of self-supervision to train themselves to see.

What does current research say about the learning paradigm used by brains to achieve such an outstanding level of visual competence? Which tasks do brains use to train themselves to be so good at processing visual information and making sense of the visual world around them? Or said in other words: how does the brain manage to train its neural networks without having access to manually annotated datasets like ImageNet, COCO, etc. (i.e. how does the brain manage to generate its own training examples)? Finally, can we apply these insights in computer vision?

Maybe this paper will give you an overview about (and entrance to) this topic from biological side. It is a review about the state of the art in human brain development (and its implications for clinical treatment).

The table of contents include for example

Stage 1: the first year, early maturation of vision and the structure of V1 neurobiology
Stage 2: preschool children have high variability in V1 development (1–4 years)
Stage 3: experience-dependent visual development in school aged children (5–11 years)

(V1 means "visual cortex")
Where this points all handle 3 categories: visual milestones (i.e. contrast sensitivity, contour integration), anatomical milestones (i.e. morphology) and neurobiological milestones (i.e. synapsis, but also a lot of genetics).

And as second : Maybe you could ask this question on bioinformatics.SE too, because this connection between biological example and computational reproduction is one of their fields.


What can be done with highly correlated variables (>.95 and <-.95) I hope we can remove the highly correlated variables based on the feature importance may be with PCA etc.

Is there anything we can do with highly correlated variables/

Thanks in advance !

An alternative to the one provided by @Kasra is dimensionality reduction. It's another way of solving your multicollinearity problems, while avoiding deleting variables more or less arbitrarily.

You can use simpler, linear techniques such as PCA, or more complex non-linear techniques such as Autoencoders. t-SNE is a non-linear technique that is typically used for visualization, I do not recommend to use it for a Training set.


How do I access orange on Ubuntu 16.04 I have used pip to install orange and the installation seemed fine.  There was not Orange icon on my ubuntu 16.04 desktop and tried orange3 in the command line, it didn't work, and tried python3 -m orange and python3 -m orange 3 and neither worked as well.

How can I use orange?

From the Orange3 Pypi documentation :

To start Orange GUI from the command line, assuming it was successfully installed, run:
orange-canvas # or python3 -m Orange.canvas


target variable prediction among possible answers I have a dataset on which I would like to apply a Machine Learning algorithm for multi-class classification. An example of my target variable (in string format, will be later OneHotEncode-d):
|       target      |
+-------------------+
|  (Mike, Michael)  |
| (Anne, Annabelle) |
|    (Joe, Joseph)  |
|    (Mark, )       |
|    (Aaron, )      |
+-------------------+

I want to have a model that, for example, predicts Joseph but does not consider it wrong because it is one of the two possible answers.
Is this a multi-class or multi-label problem? Can the two definitions go alongside or is it one or the other? I am a bit confused, though I propend towards the latter.
In case my guess was correct, would it be sufficient to just pick one of the models that supports multi-label classification, fit() it and score() it?

I want to have a model that, for example, predicts Joseph but does not consider it wrong because it is one of the two possible answers.

It seems that you are confusing prediction and evaluation: whether an answer is correct or not is a matter of evaluation, and the evaluation method doesn't have to follow the same rules as the predicting system.
For example the system can be designed to always predict a single name, and the evaluation measure can be based on whether the predicted name belongs to a set of correct answers or not. This would technically be simple multiclass classification, since there's a single class predicted. However you would have to implement your own evaluation method, because the standard one will consider correct only if the prediction is exactly the true class.
You could also decide that the system can predict any combination of names, that would be multi-label classification. But here again you could decide any specific evaluation method, for instance count a prediction as correct as long as it contains at least one of the true answers, or contains only names which belong to the true answer, etc.
Note that in theory any multi-label problem can be transformed into a multiclass problem simply by considering any possible set of classes as a single class (e.g. "Mike+Michael" as one class). In general it's a matter of number of values, since if every possible combination can happen in the data there would be $2^N$ classes as a result. I don't think this is relevant in your case, since this would defeat the idea of a flexible evaluation method.


Choosing between prepadding and postpadding for variable length sequence data in a CNN Convolutional Neural Networks (CNN) assume the same sized data. Often sequence data is variable length (e.g., natural language). Thus sequence data is padded to make sure all data is all the same length.

Prepadding is adding zeros to the beginning of shorter sequences. Postpadding is adding zeros to the ending of shorter sequences.

What is the empirical reason to choose prepadding or postpadding?

I don't think that it makes a difference. As long as (1) your sequence is long enough to handle a reasonable amount of your sentence/paragraph/whatever and (2) you are consistent in they way you pad between your training cycles and your prediction cycles, then which padding direction you use should be a moot point.


Is classifier able to say there's no-such-case? I am a starter in ML and I need some help...
The problem
Assume that I have a classifier which can classify left hand / right hand well.
I am curious whether it can decide whether there's a hand in the image?
Because if it is not the case, I have to collect additional data that are labeled as "no-hand". But there's so many things that are "no-hand" such like dogs, cats, flies... I don't think it is possible to collect all those possible "no-hand" data..
My intuition
Would it be reasonable that we use some activation functions like [sigmoid] as the final outputs for the classifier(neural network) and we consider the outputs for both left/right classes as the scores that classifier think how far the input is from the left/right hands classes.
Then if the outputs are below some threshold for both left hand and right hand we can say there's no hand in this image input.

Taking your questions one after the other:

  Assume that I have a classifier which can classify left hand / right
  hand well. I am curious whether it can decide whether there's a hand
  in the image?

No. It cannot. A model which is trained on hand/no-hand can do that. Not this. At most, it shall predict one of the either (left/hand) with a very low confidence score. But, it definitely is not a nice idea to use it for a hand/no-hand identification.

  But there's so many things that are "no-hand" such like dogs, cats,
  flies... I don't think it is possible to collect all those possible
  "no-hand" data

You are trying to do build a binary hand.no-hand classification model, not a hand/cat/dog/... style multi-label classification model. So, you just need to have a bunch of images with a hand in it, and some more which doesn't have one.

  Would it be reasonable that we use some activation functions like
  [sigmoid] as the final outputs for the classifier(neural network) and
  we consider the outputs for both left/right classes as the scores that
  classifier think how far the input is from the left/right hands
  classes.

Possible, but not advisable.


Can we use a model that overfits? I am on a binary classification problem with the AUC metrics. I did a random split 70%, 30% for training and test sets.
My first attempts using random forest with default hyper-parameters gave me auc 0.85 on test set and 0.96 on training set.
So, the model overfits. But the score of 0.85 is good enough for my business.
I also did a 5-folds cross validation with the same model and same hyper-parameters and the test set results were consistently something between 0.84 and 0.86

My question is: can I believe on the score 0.85 and use this model in production?

Yes, if your 0.85 AUC is good enough for your use case this is a good enough model. The performance on the training set indicates how well your model knows the training set. This we don't really care about, it's just what the model tries to optimize. The performance on the test set is an indication on how well your model generalizes. This is what we care about, and your model gets to around 0.85 as an estimate for your generalization. Differences between training and testing are the norm and in this case it could be that you might get a better performance by adding stronger regularization but if 0.85 is good enough, go for it!


How to create user and item profile in an item to item collaborative filtering? (Non-rating case) I want to build a recommender system for a coupons website which should do the following: Given the past purchase behaviour of a user, recommend coupons which the user is likely to buy. The data does not have any ratings for coupons by the user. It tells if a user bought a certain coupon or not, which category did it belong to, what is the gender and location of the user who bought these coupons etc.? I am implementing an item to item collaborative filtering. Based on what I have learnt through videos available on this topic, the following is my approach:

Build an item profile of all the coupons with attributes such as category of the coupon, location the coupon is available in and the days on which the coupon can be redeemed.
Build a user profile of all the registered users with attributes such as age, gender, and location.

Now I am making an item and user matrix, where I will have Coupons on the rows and Users in the columns. I will fill 0 and 1 depending on if a coupon C1 was bought by a user U1 in the past or not. In the same dataset, for the coupons for which I want to predict, I will calculate a score based on cosine similarity between items.

My question is if I am using just 0s and 1s (coupon bought or not) to recommend coupons, what is the use of the item and user profile? How I can incorporate the intelligence in the recommendation system that if a user has bought more coupons from X category in the past than recommend him new Y coupons from the same category?  I read online about using random forests etc. to calculate "weights" of the attributes in recommendation systems, where do they fit in this scenario?

P.S. I am doing everything in R.

Many thanks in advance for any advice/suggestions.

You can use your item and user profiles to generate a prediciton function (i.e. a function that will predict how relevant a coupon will be for a user, also known as a representation for your items and users). Typical functions used for this purpose are Dot Product and Cosine Similarity. This step ensures the intelligence in the recommendations will incorporate categories information.

Once your scores are predicted, you must compare them to your historical interactions by using a loss function (some examples are Root-mean-square-error and Kullback-Leibler divergence). This step will produce an error that you can use to inform your algorithm for learning (e.g. adjust the weights in your representation functions for items and users).

For more details I strongly suggest you to check out the great slides compiled by James Kirk, which provide a framework to unify the research that has been done on the topic.


Best strategy for extracting specific structured data from unstructured sentences Given a list of sentences like this:

4 to 5 hours over a period of 16 weeks
1st session: 2.0-2.5 hours & 2nd session: 1.5-2.0 hours
Approximately 5-6 visits over the course of 5 months. Visit 1, 3, 5: about 1.5 hours. Visit 2, 4: short
15 visits over a period of approximately 74 weeks.
You will come to the organization about 12 times, over a period of a little more than three years. Each visit will take from 3-6 hours.

What tools/strategy should I use if I want to let the model spit out the following data for the above sentences:

Number of sessions
Total duration(h)
Total timespan(w)

Unknown
4-5
16

2
3.5-4.5
Unknown

5-6
4.5
20

15
Unknown
74

12
36-72
156

I'm a ML beginner and wondered if this is achievable with Tensorflow or GPT? For further learning on my own: what is the specific terminology I should google for? Is this NER, text extraction or more like text classification?

The task is a specific case of  NER (technically NER is a sequence labeling task, a special case of classification).
I think you would have two main options:

Apply a pre-trained NER model: most deal with time entities but not always very accurately, and it wouldn't be specifically adapted to your data so you wouldn't obtain the distinction between the three types of values. Advantage: no need for training data.
Train your own NER model: that's the ideal scenario in terms of performance, assuming you have (or can have) a good amount of annotated data for training.


Replace entire columns in pandas dataframe I would like to replace entire columns of a pandas dataframe with other columns, for example:

and I would like to replace the columns A and B. What I did is the following:

df['A']=dataFrame['opcodes'].values
df['B']=dataFrame['opt'].values

or also

df['A']=dataFrame['opcodes']
df['B']=dataFrame['opt']

but it does not work. In particular I get the following error:

KeyError: 'opcodes'

in both cases. Can anyone help me? Thank's in advance.

[EDIT] The original dataframe is the following:

on which I did the following modification:

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()  [0] for i in x])

now I would like to define an other dataframe on which only the 'opt' column and a column with the values 'opcodes' appear.

You are getting a KeyError because 'opcodes' doesn't exist in dataFrame. Can you show us the contents of dataFrame not just df?

EDIT:

You don't need to create an empty DataFrame and then apply the data across. Simply do:

df = dataFrame[["opt", "opcodes"]].copy()


What does num_negatives mean? Does anyone know what does the num_negatives mean? What is it for?
def get_train_samples(train_mat, num_negatives):
    user_input, item_input, labels = [], [], []
    num_user, num_item = train_mat.shape
    for (u, i) in train_mat.keys():
        user_input.append(u)
        item_input.append(i)
        labels.append(1)
        # negative instances
        for t in range(num_negatives):
            j = np.random.randint(num_item)
            while (u, j) in train_mat.keys():
                j = np.random.randint(num_item)
            user_input.append(u)
            item_input.append(j)
            labels.append(0)
    return user_input, item_input, labels

This occurs here:
user_input, item_input, labels = get_train_samples(train_mat, NUM_NEGATIVES)

And here:
hist = model.fit([np.array(user_input), np.array(item_input)], np.array(labels),
                 epochs=EPOCHS, verbose=VERBOSE, shuffle=True, batch_size = BATCH_SIZE,
                 validation_data=([np.array(val_user_input), np.array(val_item_input)]),
                  callbacks=CALLBACKS)

From the comment specified by the programmer it is clear that he intends to generate negative samples
def get_train_samples(train_mat, num_negatives):
    user_input, item_input, labels = [], [], []
    num_user, num_item = train_mat.shape
    for (u, i) in train_mat.keys():
        user_input.append(u)
        item_input.append(i)
        labels.append(1)
        # negative instances   <============================  FROM THIS COMMENT LINE
        for t in range(num_negatives):
            j = np.random.randint(num_item)
            while (u, j) in train_mat.keys():
                j = np.random.randint(num_item)
            user_input.append(u)
            item_input.append(j)
            labels.append(0)
    return user_input, item_input, labels


Implementing a model for a language to another I have a dataset of sentences of language X and Y (2 columns, for example, "abc def lang" ==> "xyz pqrt mno uages"). I want to have a output as a table that translates word by word (abc => xyz, def => pqrt mno, lang => uages).
I think we should use deep learning but idk how to implement and what to use. Can you give me some ideas?
Thanks all

In case you want some state-of-the-art technique, you can implement a neural machine translation model with attention, as fully described in this course with Google members libraries like Trax.
In case of short texts, you can still use sequence-to-sequence models without attention, but the attention mechanism prevents you from suffering the vanishing gradient problem on longer sentences.
As a very brief schema of the type of model, find the image below (from the same given source):


How to predict next year's gross revenue given this year's data? I have the following dataset (.csv format) which contains:

100 columns: $\textbf{Period}$ (in years, e.g. 2017, 2018, ..., 2028), $\textbf{Gross Revenue}$, $\textbf{Region}$ (e.g. APAC, NEMEA, etc), 97 other columns as $\textbf{factors}$ (e.g. # of customers, # of goods produced, etc).
1,000,000 rows

where for $\textbf{Period}$, only 2017 is actual data, and 2018, ..., 2028 are $\textbf{planned}$, in the sense that data from these years are what the company expects to commit to and receive in these years in the future.

Now, I want to predict the gross sales for each $\textbf{Region}$ in 2018.

I tried using regression with stepwise, but the linear model that resulted from the stepwise had NA values for a majority of the predictors, and also the probability arising from the t-test.

I've consulted a few of my pals whom are working in the field of data science & analytics, and they all told me that it is impossible to predict for 2018..

Is it possible to do so given the state of the data? Some insight on this will be deeply appreciated!

Predicting next year's revenue from previous years' revenues is Time Series Forecasting.
You will need data from many previous years to do this. One year is clearly not enough.
Just think about it this way: Can you predict tomorrow stock's price given only today price?


Keras: Softmax output into embedding layer I'm trying to build an encoder-decoder network in Keras to generate a sentence of a particular style. As my problem is unsupervised i.e. I don't have the ground truths for the generated sentences, I use a classifier to help during training. I pass the decoder's output into the classifier to tell me what style the decoded sentence is.

The decoder outputs a softmax distribution which I was intending to feed straight into the classifier but I realised that it has an embedding layer, which in Keras, accepts only integer sequences not softmax / one-hot. Does anybody know a way to remedy this?

Thanks

You are describing a variation of Generative adversarial networks (GANs). The Generator, the part of the model that creates new examples, needs to output complete sentences. The Discriminator, the classifier, takes those complete sentences as input.

In particular, you are describing a DiscoGAN which discovers cross-domain relations given unpaired data. DiscoGANs have been applied to the style of images.


Feature Scaling of Training Set and Test Set Suppose I want to use the Gradient Descent algorithm.
I have a training set and a test set and I want to do the feature scaling with mean normalization.

Should I use the same mean and variance for equivalent features in the 2 sets? Why?

You do feature scaling for accelerating learning process. Features may have different scales. One maybe from 1 to 10 and one may be from -100 to 1000. Using normalization, you make the scale of them the same as each other, helps accelerate the learning process. You should find the mean and variance for each feature separately on your training data. then during training and testing each feature should be reduced by the corresponding mean and be divided by the corresponding standard deviation. So yes, for each feature during testing and training you have to provide same values for mean and std which are obtained using training data. I suggest you taking a look at here.


Periodically executing a scraping script with Python Here is my idea and my early work.

My target

Fetch 1-hour resolution air pollution data from China's goverment continuously.
The website's data which collected from the monitor sites over the country update per hour .

My Code

Now, I can grab the useful information for a single hour. Here is my code:

Input the website links for different pollution(co,no2,pm10, etc)

html_co = urllib.urlopen("http://www.pm25.in/api/querys/co.json?city=beijing&token=5j1znBVAsnSf5xQyNQyq").read().decode('utf-8')
html_no2 = urllib.urlopen("http://www.pm25.in/api/querys/no2.json?city=beijing&token=5j1znBVAsnSf5xQyNQyq").read().decode('utf-8')
html_pm10 = urllib.urlopen("http://www.pm25.in/api/querys/pm10.json?city=beijing&token=5j1znBVAsnSf5xQyNQyq").read().decode('utf-8')

Get the content of the html doc.

soup_co = BeautifulSoup(html_co)
soup_no2 = BeautifulSoup(html_no2)
soup_pm10 = BeautifulSoup(html_pm10)

Extract the useful information from the whole content.

l = soup_co.p.get_text()
co= json.loads(l)
l = soup_no2.p.get_text()
no2= json.loads(l)
l = soup_pm10.p.get_text()
pm10= json.loads(l)

Tight the raw data into neat Pandas.Dataframe.

data = {"time":[],"station":[],"code":[],"co":[],"no2":[],"pm10":[]}
for i in range(0,len(pm10)-1,1):
    ## 'station' is the monitor station's name in Chinese
    data["station"].append(co[i]["position_name"])
    data["time"].append(co[i]["time_point"])
    data["co"].append(co[i]["co"])
    ## 'code' is the monitor station's index
    data["code"].append(co[i]["station_code"])
    data["no2"].append(no2[i]["no2"])
    data["pm10"].append(pm10[i]["pm10"])

My result

Some pre-explanation

Ignore the chinese character in the table.
I only grab one city(Beijing here)'s data, and index from 0-11 notify that Beijing has 12 monitor site.
The columns 'co'/'NO2'/'PM10' represent the concentration of these air pollutants.

My problem

Now, I can grab the web data manually according my code above. But, I want to achieve the working flow below hourly  automatically.

Hour i

Execute the code
(1) Grab the data for Hour i's air pollutants data from website;
(2) Save the data into .csv based on the true date(like 20160101.csv)

After one hour.

Execute the code
(1) Grab the data for Hour i+1's air pollutants data from website;
(2) Save the data into .csv based on the true date.
  if it's the same day liken to hour i --> same .csv(like 2016-01-01.csv)
  if the present day has past --> creat a new .csv(like 2016-01-02.csv)

I havn't done these kind of stuff before. Can somebody offer me some advice?
So, I can get an useful data scraping tool run in the background and I don't have to worry about it.

If you are on a Unix-based OS, use a scheduler like cron to run your script periodically. cron is the most common one, and you can find many tutorials for it. I guess this is the part you are unfamiliar with.

When it executes your script, make it check the time like so: fname=datetime.datetime.now().strftime('%Y-%m-%d-%H.csv'). Open the appropriate file in append mode (with open(fname, 'a') as csv_file) ; if it does not exist it will be created. That's all there's to it!


How to score arbitrary sentences based on how likely they are to occur in the real world? According to article about LSTM here, I know that:

  it allows us to score arbitrary sentences based on how likely they are to occur in the real world. This gives us a measure of grammatical and semantic correctness. Such models are typically used as part of Machine Translation systems.

But, it seems that this article doesn't point out how to compute the score with LSTM.

Any way to compute the score?

Typically you would use a perplexity value. For example, if your LSTM model is word-based and you have a sentence $[x_1, x_2 . . . x_N]$, and your model predicts the words that appear in that sentence with probability $p(x_i|x_0..x_{i-1})$ (where $x_0$ is a "start token" or whatever you use to start your RNN prediction sequence). Then you might quote a per-word perplexity for that sentence under your model as

$- \frac{1}{N}\sum_{i=1}^N log_2(p(x_i|x_0..x_{i-1}))$

Using an LSTM to predict consecutive words, it is practical to construct an array of probabilities $[p_1, p_2 . . . p_N]$ by running the network on the sentence and noting the probabilities for the correct matching word - i.e. $p_i = $ the predicted probability of the correct class $x_i$ at each step, which simplifies the expression:

$- \frac{1}{N}\sum_{i=1}^N log_2(p_i)$


How can I group by elements in a column in pandas? I have a dataset that looks like this:

I need to transform it so it looks like this:

Meaning I need to show the balance by balance groups and time gaps.
Now the time gaps are elements of a column. I tried to use the following;
b = a.pivot_table(values='GAP', index=a.index, columns='BALANCE_GROUP', aggfunc='first')

With different parameters but it outputs something really weird. I also tried to use the groupby function which works fine, but it won't allow me to set the elements in the gap column as seperate columns...
b = a.groupby(['BALANCE_GROUP','GAP']).sum()

and I got something like this:

If I understand what you need, I think it is this:
b = a.pivot_table(values='TOTAL_BALANCE_EUR', index=['NSFR_GROUP', 'BALANCE_GROUP'], columns='GAP', aggfunc='sum')
b

It's easier for others to help you if you make the data available to others. Just make a tiny dataframe with 10 rows for instance. Also, you can make the code a bit easier to read by enclosing it in three backticks: ``` or using the code sample button when you write your post.


Riskscore creation on Numerical Data I am working to create a Risk score on data where I have variables -
Invested_amount, Profit Amount, Age of Account in days, Total Trading Transactions, Profit per Transaction & Investment per transaction. Basically, I want to derive a method to calculate the Risk score, where a person's profit is more (most of the time he is winning). Here the score should be higher so that I can classify them as High Risk (Mostly wins and create a big profit), medium Risk, and Low-Risk customers (Always makes Loss).
Basically, my problem is to assign a Risk score to each customer, the higher the score more the customer is Risky. Once this score is derived then using this score we will segment the customers into three classes.
Below are the variables currently I am using…
|Variable|               Definition|
|username|               UserId to identify the records|
|Total_Freq_transaction| Total Number of trades done|
|Total_Freq_win|         Total number of trade won|
|Account_age_days|       Account activated date - Last trade date |
|Invest_Amount|          Total Amount invested in trade|
|total_profit|           Profit made on trade (pay out amount - Investment Amount)|
|Trade_per_day|          Total Number of trades done / Account age|
|Win_prob%|              Wining probability from total number of trades|

Any help would be greatly appreciated.

There are traditional ways of creating a risk scorecard using linear regression techniques. It's a whole topic in itself. A good book for beginners to read in-depth about it would be this.

For sure, you can also tackle it as three-class classification problem if you have labelled data for that. Or consider it as binary classification problem and create three buckets out of class probability output by the classifier.


For very simple linear regression can we quantify the prediction accuracy hit between using one hot encoding and simple numerical mapping? Suppose I had a simple linear regression model that had the following input or X variable:
[North]
[East]
[West]
[South]
[North, East]
...
[North, East, West, South]

and I decided to numerate them like:
[North] - 0
[East] - 1
[West] - 2
[South] - 3
[North, East] - 4
...
[North, East, West, South] - 15

I had someone take a look at my model and tell me to use One Hot Encoder or One Hot Binary Encoder instead of assigning inputs like this.
My question is from a linear regression perspective what is the advantage for using OHE over my simple numerical mapping? If we can quantify an accuracy loss would it be substantial? If I had 10 model variables that I had to map like this would the loss be more substantial?
I want to know what sacrifice I'm making not using OHE

The issue with numerical encoding in this context is you are enforcing that your input variable X is ordinal when it's likely not. This is telling your model that the order in which you encode your inputs are either increasing or decreasing monotonically with your target. Let's say you encoded your data like this:
[North] - 0
[East] - 1
[West] - 2
[South] - 3
[North, East] - 4
...
[North, East, West, South] - 15

If you train a linear regression model with this encoding you are telling your model that [North] either indicates a higher or lower target than [North, East, West, South], and that [East], [West], [South], and [North, East] are somewhere in between. What if [West] typically has a lower target than either [North] or [North, East, West, South]? In this case you would be enforcing some constraint on your model which is not true.
To test this you could split your input data in a 70/30 train-test-split and evaluate how your numerical encoding performs against one-hot-encoding on the test set. Another alternative would be to look into Target Encoding - this would allow you to keep a small feature space (an issue with one-hot-encoding) while attempting to keep your input and target increasing/decreasing monotonically. I would recommend testing all three encoding methods to figure out the ideal solution for your given problem.


Find multiple categories of promises in texts I have conversations a customer with an agent (without punctuation). There are phrases of several categories of promises that an agent gave to a customer (call back, make an appointment, etc.). It has been done manually. Altogether 12 categories. Now I'm thinking of creating an algorithm for this. I am thinking to do this task in two steps.

In the first step, I need to create an algorithm that can find an
end and a beginning of all promises. This algorithm has to insert a
start tag and an end tag.
The second step is to create a classifier that would label a promise
to the necessary categories.

As I understand, the second step is well known and this is called text classification. But for the first step, I could not find any articles and github repositories. But I think it is an important NLP task and there must be information on this. Maybe are there approaches that solve two steps at the same time?

Update

Just sample an agent's transcript (in reality it is more difficult):

hi my name is ben how can i help you yes good what about i can help probably yes sir do you have a problem with internet connection i see let do you need a help at place okay i see so what i can do i can arrange appointment with technical will it be good for you great can i help you with something else you okey okey to have a great day you too

Promise here is

i can arrange appointment with technical

That would be a sequence labeling task, the most common type is Named Entity Recognition, you'll find many examples about it but you can train a custom system with your data. The traditional method is Conditional Random Fields, there are a good few libraries available.

Side note: usually a single CRF model is used to do both detecting and labeling at once (your steps 1 and 2).


Finding parameters with extreme values (classification with scikit-learn) I am currently working with the forest cover type prediction from Kaggle, using classification models with scikit-learn. My main purpose is learning about the different models, so I don't pretend to discuss about which one is better.

When working with logistic regression, I wonder if I need the 'penalty' parameter (where I can choose L1 or L2 regularization). Based on what I found, these regularization terms are useful to avoid over-fitting, specially when the parameter values are extreme (by extreme I understand the range of some parameter values are very large compared to other parameters, Correct me if I am wrong. In this case, wouldn't it be enough to apply log-scale or normalization to these values?).

The main questions are: as the number of parameters is large, are there visualization techniques and tools in scikit-learn which can help me to find parameters with extreme values? is there any statistical function/tool which returns how extreme the values of parameters are?

If by "parameters" you mean features (called "Data Fields" at Kaggle), then, yes, you can log-scale those. To visualize them you can just use histograms.
To do it for all features in python, for example, you can put your data in pandas DataFrame (let us call it "data") and then use data.hist()
This has nothing to do with the regularization in any model.

If by "parameters" you mean the coefficients obtained after fitting the logistic regression, then one uses regularization. This has, however, is not directly related to log-transform. How you list/visualize your coefficients depends on the programming tool you use for logistic regression (or other model)


Why gradient boosting uses sampling without replacement? In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger corrected me).  Here there are the sampling techniques implemented for Catboost

My questions:

Why is Gradient Boosting sampling done without replacement?
Why would it be worst to sample with replacement?
Are there any sampling techniques used in GB that are with replacement?

I quote a paper for SGB:

  Stochastic Gradient Boosting is a randomized version of standard Gradient Boosting algorithm... adding randomness into the tree building procedure by using a subsampling of the full dataset. For each
  iteration of the boosting process, the sampling algorithm of SGB selects random s·N objects without replacement and uniformly

Why is Gradient Boosting sampling done without replacement?

Your first question seems to suggest that the base classifier will always have a subsampling mechanism but this is not necessarily true.

Notice that in the Catboost documentation it is mentioned "Stochastic Gradient Boosting" not "Gradient Boosting". Stochastic Gradient Boosting is a variation of Gradient Boosting that is precisely based on building the boosting process using a subset of the data at each iteration. Therefore each base model does not see the whole train data, it sees only a subset (or minibatch) of the data.

You might want to do this for 2 main reasons: Faster train time, Regularization effect.

Why would it be worst to sample with replacement?

As long as you do sampling in your base classifier the  speed benefit should be quite similar. If your base classifier does not do any sampling then the Stochastich Gradient Boosting algorithm will converge much faster. Just like a Mini Batch version of a Neural network converges much faster than a Full Batch version.

Are there any sampling techniques used in GB that are with replacement?

Standard Gradient Boosting uses all the data to build the gradient at each step of the boosting process.

If you do full batch learning but your base classifier has a "subsample" parameter then you are essentially doing sampling with replacement at every step of the boosting process.


Unusually High BLEU score on a NMT model This is the project on Neural Machine Translation on the English/Irish language pair. I have been spending the past month or so trying to train a good baseline to do 'experimentation' on. I have a corpus of ~850k sentences (unfortunately Irish is very limiting). When I trained it and evaluated it with BLEU, I got a score of 65.02, which is obviously absurdly incorrect. These were my Fairseq-train settings:

!CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin-full_corp/MayNMT \
  --lr 5e-4 --lr-scheduler inverse_sqrt --optimizer adam\
  --clip-norm 0.1 --dropout 0.2 --max-tokens 4096 \
  --arch transformer --save-dir checkpoints/full-tran

I know not everyone uses Fairseq in NLP, but I hope the arguements are self-explanatory.

I deduplicated the dataset (converted to a Python set() which only takes unique entries) so I don't think the issue is the dev/valid and test sets contain duplicate entries, but I'm not sure what else causes this. Some suggest overfitting may be a cause, but I feel that would only affect the BLEU if the dev set shared training entries.
I've tried to find the problem myself, but there aren't many places that cover NMT, let alone BLEU.

According to recent publications, it is not impossible to get BLEU scores as high as yours for English→Irish. Nevertheless, without any other knowledge, they certainly seem too high.

From the command line arguments, there does not seem to be any evident problem.

The most probable explanation is, as you already pointed out, a data leakage between validation/test and training. Note that, while you removed exact duplicates, you may be getting partial matches that go unnoticed. You may want to look into different similarity metrics. The most straightforward is the Jaccard Similarity.


Extracting Useful features from large convolutional layers I have been training a convolutional neural network on emotion detection. Now, I would like to extract features for my data to train an LSTM layer. In my case, the top convolutional layers in the network has the following dimensions: [None, 4, 4, 512] and [None, 4, 4, 1024]. Therefore, this will give a total of 8192 and 16384 dimensional vectors. This is too large to train an LSTM layer. Therefore, I would like to know what is the best possible way to reduce the dimensionality of this vector? In other words, should I apply global average pooling to the conv layer after obtaining the activation or any other dimensionality reduction technique? In this case, my features will be a vector of 512 or 1024 dimensions, which makes sense.

Any help is much appreciated!!

Applying a pooling layer following a convolution is a standard way to reduce the size of the input matrix and get the invariant features. You might also want to consider adding a dense layer with a smaller number of output neurons.


Conv bias or not with Instance Normalization? It is well known that Conv layers that are followed by BatchNorm ones should not have bias due to BatchNorm having a bias term.

Using InstanceNorm however, the statistics are instance-specific rather than batch-specific yet there are still are two learnable parameters $\gamma$ and $\beta$, where $\beta$ is a learnable bias.

Naturally, Conv layers followed by InstanceNorm layers should also not use bias.

Though some well-renowned implementations on github insist on using Conv bias when followed by InstanceNorm.

What's your take on this?

Bias should not be used before InstanceNorm exactly as before BatchNorm. Whether you set affine to True or False, these layers first normalize the input, so any added bias is discarded.


Is a Python Webscraper a Wrapper? Would a simply Python Script to extract Information from a Webpage (e.g with use of beautiful soup for example) be called a Wrapper (see https://en.wikipedia.org/wiki/Wrapper_(data_mining))?

It might be, or it might not. It depends on how it performs.

The key to a wrapper in the more general sense is that it provides for the interoperability of two distinct classes of object which are not inherently able to work together.

In the specific case you are talking about, simply creating a script to scrape the web is not necessarily a wrapper. Once you format the xml into a recognized format such as JSON, pandas dataframe or  in a SQL schema that your program can operate over, it becomes a wrapper.

The wrapper 'repackages' the data to work in your current system's class transparently so that you do not see the underlying nature of the incompatible structure.


What are common problems around HADOOP storage? I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity.  What questions should I ask?

Data age,
Data size?
Housekeeping schedule?
How do we identify the different types of compression used by different applications?
How can we identify where the duplicate data sources are?
Are jobs designated for edge nodes only on edge nodes?

You can ask following questions:

Hadoop Cluster Size (No. of nodes/machines allocated for data storage)
Hardware configuration (storage capacity) for nodes/machines
Data Block Replication factor for fault tolerance
Data Compression technique used for file storage
Data Archiving process


Why training a Restricted Boltzmann Machine corresponds to having a good reconstruction of training data? Many tutorials suggest that after training a RBM, one can have a good reconstruction of training data just like an autoencoder.
An example tutorial.
But the training process of RBM is essentially to maximize the likelihood of the training data. We usually use some technique like CD-K or PCD, so it seems that we can only say that a trained RBM has high probability to generate data which is like training data (digits if we use MNIST), but not correspond to reconstruction. Or are these two things just equivalent in some way?
Hinton said that it is not a good idea to use reconstruction error for monitoring the progress of training, this is why I have this question.

1) I believe this comes from general property of Maximum Likelihood estimation, namely, that it is equivalent to minimization of KL-divergence between the model distribution and the (empirical) data distribution.

Proof: Suppose we have data $X=\{\mathbf{x}_1 \ldots \mathbf{x}_N\}$ drawn independently from unknown data generating process. To approximate its distribution we consider family of distributions $p_{\text{model}}(\;\cdot\;;\theta)$ parameterized by parameters $\theta$. Then
$$
\theta_{MLE}
=\text{arg}\max_{\theta}p_{\text{model}}(X;\theta)
=\text{arg}\max_{\theta}\prod_{n=1}^N p_{\text{model}}(\mathbf{x}_n;\theta)
=\\
=\text{arg}\max_{\theta}\sum_{n=1}^N \log p_{\text{model}}(\mathbf{x}_n;\theta)
=\text{arg}\max_{\theta}\underbrace{\left(\frac{1}{N}\sum_{n=1}^N \log p_{\text{model}}(\mathbf{x}_n;\theta)\right)}_{  \mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log p_{\text{model}}(\mathbf{x};\theta) \right]},
$$
where $p_{\text{data}}(\mathbf{x})=\frac{1}{N}\sum_{n=1}^N \delta_{\mathbf{x}-\mathbf{x}_n}$ is (empirical) data distribution. Now by definition
$$
D_{KL}(p_{\text{data}} \| p_{\text{model}}(\;\cdot\;;\theta))=\mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log p_{\text{data}}(\mathbf{x})-\log p_{\text{model}}(\mathbf{x};\theta) \right]
$$
Observe that the first term is independent of $\theta$ thus minimizing KL-divergence is equivalent to minimizing only the second term, which is a quantity with minus sign of what Maximum Likelihood maximizes. $\blacksquare$

Now if training data has high probability according to the RBM $\Leftrightarrow$ model distribution of RBM is close to empirical data distribution, then taking into account that RBM is trained using MCMC-based algorithm (CD-k/PCD-k), if we assume that state of Markov Chain particles are close to their stationary distribution, then feeding training example into RBM shouldn't change it too much after reconstruction. Therefore the low reconstruction error is expected in this case, and typically this happens in practice.

2) But as you correctly pointed out, even though it is convenient to measure reconstruction error during the training process, it is a poor measure of what actually is going on inside RBM during learning, since this is not the function RBM is aim to optimize (RBM maximizes probability of visible units $p(\mathbf{v})$ over all training data, which is equivalent to minimizing resp. KL-divergence).

Unfortunately, measuring (log-)likelihood directly is intractable in this model, but there exist better proxies to the true likelihood, such as pseudo-likelihood (more here or in the Hinton's practical guide you mentioned).


How to calculate the silhouette coefficient? Calculate the silhouette coefficient of point Pi from the above image.

To apply the given formula, how to know which is a(i) and b(i)?

a(i) : the average distance between 'i' and all other data within the same cluster (source)

b(i) :  the lowest average distance of 'i' to all points in any other cluster, of which 'i' is not a member (source)

So, from the question, a(i) will be 24 as point 'Pi' belongs to cluster A and b(i) will be 48 as it is the least average distance that 'Pi' has from any other cluster than A (to which it belongs).

So, as a(i) < b(i),  silhouette coefficient s(i) = 1 - 24/48 = 0.5


is this a good case for NOSQL? I'm currently facing a project that I could solve with a relational database in a relatively painful way. Having heard so much about NOSQL, I'm wondering if there is not a more appropriate way of tackling it:

Suppose we are tracking a group of animals in a forest (n ~ 500) and would like to keep a record of a set of observations (this is a fictional scenario).

We would like to store the following information in a database:

a unique identifier for each animal
a description of the animal with structured fields: Species, Genus, Family, ...
a free text field with additional information
each time-point at which it was detected close to a reference point
a picture of the animal
an indication whether two given animals are siblings

And:

there might be additional features appearing later as more data comes in

We would like to be able to execute the following types of queries:

return all the animals spotted between in a given time interval
return all the animals of a given Species or Family
perform a text search on the free text field

Which particular database system would you recommend ? Is there any tutorial / examples that I could use as a starting point ?

Three tables: animal, observation, and sibling. The observation has an animal_id column which links to the animal table, and the sibling table has animal_1_id and animal_2_id columns that indicates two animals are siblings for each row.

Even with 5000 animals and 100000 observations I don't think query time will be a problem for something like PostgreSQL for most reasonable queries (obviously you can construct unreasonable queries but you can do that in any system).

So I don't see how this is "relatively painful". Relative to what? The only complexity is the sibling table. In NOSQL you might store the full list of siblings in the record for each animal, but then when you add a sibling relationship you have to add it to both sibling's animal records. With the relational table approach I've outlined, it only exists once, but at the expense of having to test against both columns to find an animal's siblings.

I'd use PostgreSQL, and that gives you the option of using PostGIS if you have location data - this is a geospatial extension to PostgreSQL that lets you do spatial queries (point in polygon, points near a point etc) which might be something for you.

I really don't think the properties of NOSQL databases are a problem here for you - you aren't changing your schema every ten minutes, you probably do care that your database is ACID-compliant, and you don't need something web-scale.

http://www.mongodb-is-web-scale.com/ [warning: strong language]


What are the advantages / disadvantages of off-policy RL vs on-policy RL? There are various algorithms for reinforcment learning (RL). One way to group them is by "off-policy" and "on-policy". I've heard that SARSA is on-policy, while Q-Learning is off-policy.

I think they work as follows:

My questions are:

How exactly is "on-policy RL" and "off-policy RL" defined?
What are the advantages / disadvantages of both?

This was answered in cross-validated and stackoverflow:

The reason that Q-learning is off-policy is that it updates its Q-values using the Q-value of the next state $s′$ and the greedy action $a′$. In other words, it estimates the return (total discounted future reward) for state-action pairs assuming a greedy policy were followed despite the fact that it's not following a greedy policy.
The reason that SARSA is on-policy is that it updates its Q-values using the Q-value of the next state $s′$ and the current policy's action $a′′$. It estimates the return for state-action pairs assuming the current policy continues to be followed.

These slides offer some insight on pros and cons of each one:

On-policy methods:

attempt to evaluate or improve the policy that is used to make decisions,
often use soft action choice, i.e. $\pi(s,a) >0,  \forall a$,
commit to always exploring and try to find the best policy that still explores,
may become trapped in local minima.

Off-policy methods:

evaluate one policy while following another, e.g. tries to evaluate the greedy policy while following a more exploratory scheme,
the policy used for behaviour should be soft,
policies may not be sufficiently similar,
may be slower (only the part after the last exploration is reliable), but remains more flexible if alternative routes appear.

For reference, these are the formulations of Q-learning and SARSA from Sutton and Barto seminal book:

Q-learning:

SARSA:

P.S.: I referenced and quoted the original answer from a different stackexchange site, as indicated in this meta question.


How to build a network that detects common object in images? I have a group of images. All contain some common object (let's assume all at the same size and no other alterations). I want to train a CNN that will learn the filter for the common object and gives me back a boolean flag of whether it exists in a given image or not.

(for simplicity, I assume the "object" is smaller than 5x5 pixels)

I thought of something like:

A Conv layer with 1 filter of 5x5
A MaxPooling layer with pool_size = input size (meaning - returns a single value: the maximal activation of the filter)
Output max activation (or a variation of it)

Some questions:

Would you output the raw max activation and use a threshold to decide? sigmoid of it? or force the network to output "object" and "no_object" classes and do softmax?
Quite related - what loss function?
Is it a problem to have a large MaxPooling layer (one that reduces all activation to a single one)? What happens to the gradient on backprop?
It seems like it would just go randomly to one of the activations and I'm not sure that it can "learn" the object that way..

I'm wondering how to look at this problem as it's not really a 2-class classification problem. Even if I can come up with images that doesn't contain the object, there is nothing to "learn" from this class except of the fact that it doesn't contain something (which is a valuable information here, I must agree).

I've tried playing with it on simulated small black-white images and a cross "object" that I randomly added. Couldn't get the filter to learn the cross pattern although on some tries it got the horizontal line..

Any idea would help! Thanks!

You probably just need a deeper network to learn the cross test example. Convolutional layers need to have multiple feature channels and be stacked in order to learn anything more complex that edge/corner counting. Look at example networks for e.g. MNIST and you will usually see typically 2 or 3 CNN layers, with multiple channels (e.g. 32 or 64 channels, each of which is associated with previous layer channels amount of NxN kernels) plus 1 or 2 fully connected layers before classifier.

For a binary choice, yes train on object present/not present classes. There is no major difference between a single sigmoid output or softmax with two outputs. Use cross-entropy loss function, multiclass version if you use softmax.

I suggest just adapt an MNIST example network to your problem by changing input shape and number of outputs. So that would use softmax and multiclass cross-entropy loss.

  Is it a problem to have a large MaxPooling layer

It could work in some situations, but by having a single layer, single channel CNN, you have made the network too simple to classify even moderately complex shapes. Reducing the activations from this layer to just the max is going to limit the network to essentially classifying by strongest matching 5x5 template to the filter in the image. With enough training examples it should be able to create such a filter, but it would not cope very well with noise, rotation or any partial matches.

You might get something more effective with a large pooling layer over say a 2-deep multi-channel CNN, and then a small fully-connected layer over those max channel outputs. It would still be an unusual choice of architecture, but I suspect it could learn problems like your cross test.


Update the saved model after training Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text = chunk['body']
  label = chunk['user_id']
  print(text.shape, label.shape)

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3)
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

For example, if the first chunk had labels 1 and 2, and the second chunk 3 and 4, will the final model be able to predict just 3 and 4? Or 1 and 2 as well, given the testing data has all the labels. Any help?

UPDATE
The chunk is used to get text from the csv. I have updated my code.

Short answer : a model will not be able to predict attribute it has not seen during the training process. From what I understand from your code, each model will erase the previous one.


How to filter Items in Recommender Systems? I have a Recommender System which recommends Articles based on Similarity from 3 Features, "Page-Title, Article Content, Tags". But some of the Articles are NSFW(Related to Adult Topics). I want to keep these Recommendations separate from the normal ones? Any idea about how do I go ahead with it,
I was thinking about keeping the adult articles separate and using another metric to recommend them, but I do not feel its right as it would create a whole different Recommender System.

Is there a reason you cannot filter NSFW articles as a post-processing step? For instance, if you need to recommend 10 articles, you find the top 50 recommendations and then apply the filter if it is needed and truncate down to top 10?
In my experience, it is pretty common to apply business rules before/after recommendations to account for things like this. Grabbing an example article on this topic for reference:
https://buildingrecommenders.wordpress.com/2015/11/10/the-components-of-a-recommender-system/


Will a larger data set in order to support your subset? Is it possible to utilize/create a larger data set with like data in order to be able to improve the Ml model optimization and/or ML model feature options that you can utilize on your subset of data? If so what should one take into account when building this data? i.e. when sourcing additional data is there mistakes to avoid and/or how many multiples of your subset data should you ideally add?

This is a common approach to address class-imbalance. You'll generally see it referred to as "upsampling" or "over-sampling". In addition to simply reusing existing data, you can "augment" your data by transforming it or adding noise, or generate synthetic data. A good demonstration of data augmentation is the winning solution to the kaggle Galaxy Zoo contest. A popular technique for generating new data is SMOTE. Keep in mind: if you change the class proportions in your dataset, your effectively changing the prior for those classes. Depending on your choice of algorithm and what you are trying to accomplish, you may want/need to apply a probability adjustment to the model's output.

Another technique you can use is to just pick a different decision threshold. For example, if you're performing a logistic regression and don't care about performing inference on parameters, duplicating the data for one class won't actually change the slope of the decision boundary, it will just shift its location. You could equivalently use the decision boundary learned on the unmodified data and then pick a cutoff other than $P(X=1) \ge 0.5$. A good heuristic is to use the threshold which maximizes Youden's J, which can be trivially calculated from a ROC curve. If you have a specific model evaluation metric you are concerned with, choose your threshold relative to that.


normalize usage data I'm looking at identifying trends within my data, particularly wondering if usage of my app on new sign ups is increasing over a week to week basis.  As we are constantly improving our product, I'd love to be able to identify a correlation between app usage & new feature releases.

The data I have is split into weekly cohorts.  Each week I have the number of new sign ups, and how many of those accounts are still active (e.g. users are logging into the system).

Let's say I have the following data

Week 1 - 10 new trials, 3 accounts active

Week 2 - 15 new trials, 5 accounts active

week 3 - 4 new trials, 3 accounts active

week 4 - 20 new trials, 12 accounts active

week 5 - 17 new trials, 9 accounts active

In my current approach of analysis, week 3 looks amazing because 75% of accounts are still active ... In reality though, the number of new trials is extremely small compared to other weeks.  As a result, I don't feel like I'm accurately comparing apples to apples in a week to week comparison.

Is there a way I can normalize the weekly data so that I am performing accurate analysis - or is percentage based really the best way to look at this data?

I am pretty new to this, so any help is much appreciated.

Cheers

With this low number of sign ups I doubt you can profit from advanced statistics at this stage. Just chart your numbers in excel along the timeline and judge visually.


How to handle missing data data in dependent variable? I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k records in the dependent variable that have missing values.
- How do I solve this problem? Should I be excluding these records?
- Is imputing the missing values a good method of solving this problem? Wouldn't that give me inaccurate values as the dependent variable is dependent on a number of variables? I'm not sure.

Can anyone please help?

The questions you're asking are empirical questions. The only answer anyone can give is to try all of them and see which works better.

You have three options:

Impute data
Throw away data
Use a classifier that can handle missing data, e.g. xgboost. See this answer. xgboost is a powerful classifier. So, if you're not tuning very hard for performance, xgboost is a great way to get a good v0.

Some other points:

The pattern of missing values is important, and can influence the choice of algorithm.
If your dataset is noisy, imputing may only amplify the noise. If you can afford to drop those 2k rows, try that, or train both with and without that data and see if the combination performs better.

Regarding software, there are many options:

Scikit Learn has some imputation functions
MICE, Multiple Imputation through Chained Equations works well for random data. Available in fancyimpute, and also in statsmodels.

You will find many resources if you search


Given that two images are similar, how do I find and classify the difference between them? We have used a derivative of the P-hash algorithm to determine similar images when the user selects an image.

This allows the customer to check out very similar looking photos together. It has also helped us cluster pictures.

When the user selects an image, we would like to show him/her how each of the similar images is different from the selected image, by highlighting the difference, and by classifying the difference.

(eg: this is the hard cover version of the book; The font is different on this cover; original dress in black: same dress in blue, green and yellow )

I'm unsure what kind of annotated training data would be useful for Tensorflow. Could someone please point us in the right direction?

My problem is that we have successfully been able to classify single pictures and even do a bit of object detection by drawing bounded boxes, but we are unable to do pairwise difference identification.

In order for you to do pairwise difference, you need to do the following:

Generate Ground truth data by drawing bounding boxes on elements you need comparison for and label them i.e if you want to compare text on the hard cover, you need to mark the text region and add a label to it like font type.
Once you have ground truth data, you can then use object detection networks like different variants of Yolo to identify multi class objects in each document.

This should provide you a starting point to solving the problem at hand.


Tutorial for restricted Boltzmann machine using PyTorch or Tensorflow? I am trying to find a tutorial on training Restricted Boltzmann machines on some dataset (e.g. MNIST), using either PyTorch or Tensorflow. The few I found are outdated.

Can you recommend any?

Try to look for implementations of deep belief networks.

https://github.com/albertbup/deep-belief-network
https://github.com/JosephGatto/Deep-Belief-Networks-Tensorflow
https://medium.com/analytics-army/deep-belief-networks-an-introduction-1d52bb867a25
https://skymind.ai/wiki/restricted-boltzmann-machine
https://www.csrc.ac.cn/upload/file/20170703/1499052743888438.pdf