Loss function in GradientBoostingRegressor I was looking at the Scikit-Learn documentation for GradientBoostingRegressor.
Here it says that we can use 'ls' as a loss function which is least squares regression. But i am confused since least squares regression is a method to minimize the SSE loss function.
So shouldn't they mention SSE here?

Note that the algorithm is called Gradient Boostign Regressor.
The idea is that you boost decision trees minimizing the gradient. This gradient is a loss function that can take more forms.
The algorithm aggregates each decision tree in the error of the previously fitted and predicted decision tree. There you have your desired loss function.
This parameter is regarding to that.


Web page data extraction using machine learning I would like to extract some specific information from web pages. Web pages contain person profiles, and I want to extract information such as name, email, research interested-areas.
Structure of each page is different from one another. How can I extract such information using machine learning? What kind of a method, features I can use?

Or can I use NLP for such task?

If I understand your question properly, this seems to be a scraping problem which you can do using Beautifulsoup in python.


Data structure design for supporting arbitrary number of columns in table or database I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data.

The task I'm currently struggling is: user needs to enter data column by column, which would consist of a number n of ID's , a number m of attributes and a number k of classes, with the conditions that n, m > 0 and k >= 0. Data is heterogeneous, so indexes can be both numeric or text, and the same goes for attributes and classes. I'm supposing there will be no null in the data for simplicity.

I'm currently thinking on:

1) Creating a table with more than enough columns (all with null values), so that I can work using only the non-null columns (which will be got from user input). However this would limit the size of the datasets people could input.

2) Resorting to create an specialized data structure on a programming language, do all the work there and finally, create a table dynamically  to store the result data there.

3) Using a database specialized for this kind of data (maybe a document-based DB).

4) Create a data structure on the RDBMS itself (I'm using PostgreSQL), let's say a variable size array, so that I can create the table directly from the user input, using only 3 variable arrays (one for indexes, one for attributes and one for classes). However, I keep in mind that attributes and indexes could be of different types, so the array would have to support heterogeneous data type and I don't know if this is possible on a RDBMS or SQL.

I've been looking for information on information but got no result until now. Any guidances to a package, language library, extension or paper, thesis, technical report with relevant information would be appreciated.
Also, personal experiences with doing something similar could be useful.

I've done something like you're describing using mongoDB--I think you'll best use your time using some sort of NoSQL approach, rather than creating a specialized one-off solution. If you're using Python, I've had excellent experiences using PyMongo to handle reads and writes from within my code.

I would strongly caution you against adopting your approach #1. This could break very easily in the future, and there are databases designed to handle your exact problem!


Correlating company entities between different data sources I have two datasets with information about companies and my task is to correlate (match) companies from dataset A to companies in dataset B. Datasets are from different sources.

The columns in both datasets include fields such as company_name, country, state, city, address, zip.  All companies are in the US.

The problem is that even though we have the company_name on both sides - the names in A aren't equal to the names in B. So for example on A you might have Google and on B you might have Google Inc. Another example is Amazon and Amazon LLC. etc, there are many different variations to that.
These aren't typos, but just different representations of the same entity, one is more common and the other is more formal.
The addresses themselves aren't always the same as well. Probably b/c a company might have more than one address. (at least large companies do)

What is the best approach to correlate (match) these entities b/w these two data sources?
There are about 500k companies in each dataset.

A few ideas come to mind:

Soundex function on the company name (tried it, not great)
Levenshtein distance b/w names of each two potential matches (didn't try it yet, but it is O(n*m))
Levenshtein b/w the concatenated values of company names state, city, etc (also O(n*m))
Geocode the address and build a function that takes into account the Levenshtein distance as well as geographical distance.
Clean up the "INC", "LLC" and all other extensions and run any of 1-4.

What's your take? Any other suggestions?

Thanks!

First of all I would clean up INC, LLC, BV etcetera from both the sources. After this there are a few options. Since Levenshtein is a metric you can use metric trees to search your space more efficiently (about O(n*log(m))). This will still be very slow so there are approximations available, for example the cosine similarity on bi-grams of the names. You can do this using matrix multiplication which is both very efficient and easily distributable. Instead of taking the highest similarity you could take the top-n and do further analysis on these, for example the real Levenshtein distance. The fact that you have additional information could be useful, you could add this to your similarity function in some way but this will be guess work. Most of these ideas I got from a PyData meetup that was recorded, a speaker from ING (a big bank) discusses the exact problem you have albeit on a bigger set with less additional information:

https://www.youtube.com/watch?v=4ohTsblxOJs


Can CBOW model only accept fixed number of words? I have a question about CBOW prediction. Suppose my job is to use 3 surrounding words w(t-3), w(t-2), w(t-1)as input to predict one target word w(t). Once the model is trained and I want to predict a missing word after a sentence. Does this model only work for a sentence with four words which the first three are known and the last is unknown? If I have a sentence in 10 words. The first nine words are known, can I use 9 words as input to predict the last missing word in that sentence?

That's a good question. The answer is yes and no.

No because the input layer of the CBOW model expects a fixed number of words. So you'll either always input 3 words or 9 words.

Yes because you can however set the sequence length as 9 words and provide just 3 words as context while the remaining 6 words can just be zero vectors. Remember in a CBOW each word is represented by a vector.


How to reach continue training in xgboost I read the paper but found nothing talking about how to implement incremental learning.

Can someone share some basic or deep knowledge? not in coding way.

I know how to write code snippet to train incrementally.

When new data comes in, how to train incrementally if I use XGBRegressor? Reserve the old trees and train new data with new trees?

I found nothing talking about this in detail

In the fit method, parameter xgb_model can be specified to continue training an old model.

https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor.fit

Edit: Yes, as I understand it, the old trees are preserved in this method.  The new trees will fit the residuals of whatever data you pass: you could continue with only the new data, which will first be run through the existing trees and have their residuals fit by new trees; or continue with all the data, which will do the above but also use the old data when splitting (probably better, so that the model doesn't lose its fit to the old data, but maybe your use case cares about the new data substantially more?).  In some cases, it may be better to just retrain from scratch (if the existing trees do a poor job on the new data).


Possible to correct an actual cell state in LSTM via gradient? Why in LSTM we calculate gradient w.r.t weights, but not w.r.t the cell state?

Is it theoretically possible to correct the contents of the cell state, and what would it result in?

I understand that weights are like a "set of skills", so that network can respond correctly to the input, even gazillions of iterations later. The cell-state is an understanding of what's going on in the past, up to the start of the current minibatch.

So why not to correct the value stored in the cell state? It would be very useful if we carry the cell state forward, between minibatches.

https://stackoverflow.com/a/44183738/9007125]

  Generally it is recommended to reset state after each epoch, as the state may grow for too long and become unstable. However in my experience with small size datasets (20,000- 40,000 samples) resetting or not resetting the state after an epoch does not make much of a difference to the end result. For bigger datasets it may make a difference.

Is it theoretically possible to correct the contents of the cell state, and what would it result in?

Yes it is. Using back propagation, it is possible to get the gradient of any value that affects a well-defined output. For training data, that includes all current cell outputs - in fact these are necessary to calculate as an interim step in order to get the gradients of the weights. Once you have the gradients of a cost or error function, then you can perform a step of gradient descent in order to discover a value that would result in a lower error for given training data.

In usual training scenarios you do not alter neuron outputs after they have been calculated using feed-forward, because these are not parameters of the model. Typical reasons to alter neuron values (or even the input) are in order to view what the ideal state might be in a given scenario. If your state can be visualised through some decoder - maybe even another neural network - then this could allow you to see the difference between actual internal state, and a potentially better one. That could be useful in error analysis for example.

  So why not to correct the value stored in the cell state?

That's because in usual training scenarios, you are creating a network that predicts an output value. You can calculate the right corrections for training data, but not when predicting, because the whole point of predicting is to estimate a label that you do not already have. As such, you want to alter your function parameters, and not interim values.

  It would be very useful if we carry the cell state forward, between minibatches.

Only during training. In a prediction scenario you usually have no way of calculating the necessary gradients. What you don't want is to train a system that then requires using error values and gradients that you do not have in production.

In some scenarios, such as an online system predicting next item $x_{t+1}$ in a sequence, where you could immediately train based on error after you observed the next item and before you continued the prediction sequence for $x_{t+2}$, you could possibly use the approach. I am not sure whether it would help performance, but in principle it could. If it did help, you'd have to compare the improvement versus other simpler changes such as different hyper-parameters on a network that didn't correct internal state using gradients.

In summary, it is possible your idea would work quite well in an online system with near-immediate feedback. In that case you could think of a set of weights as being "rules to update a belief state from data", and the output of hidden layer neurons as being "a current belief state". When errors occur, it does appear to make sense to update both the rules that led to the error and the current belief that resulted from earlier faulty rules. It is perhaps worth an experiment or two. The main caveat is that the two update processes (for weights and LSTM layer state) would interact and/or adapt to each other, so it may not lead to measurably different performance than just adding more LSTM cells to the layer.


Alternative Hunspell dictionary for stemming I am using Hunspell to spellcheck and stem the words in my documents to reduct dimensionality. For spellchecking Hunspell works great with the default en_US dictionary by SCOWL (and friends), but not so for stemming.

The reason is that the dictionary is pretty inconsistent. (e.g. some words have their plurals in the dictionary, but some others doesn't, so it's inconsistent if the stem of a word in plural will be in plural or not).

Hunspell is definitely able to stem english words consistently, but it needs an other dictionary that is more "correct" morphologically.

What alternative dictionaries can I use to improve the quality of stemming English text with Hunspell?
If there are no such dictionaries, what other tools can I use for stemming? EDIT: note that I'm using python, which can be hooked up with most libraries and command-line tools easily, but R or Matlab specific tools might not fit in my current ecosystem that well.

I can't comment on your question (my reputation isn't high enough), but I've got a few clarifying questions to ask: Why do you want to stem words?  Text classification?  Sentiment analysis?  Something else?

I've used both the 'tm' package and the 'RTextTools' package for stemming when doing text classification.  Both have some built-in functions for stemming.

Whether they would be useful to you or not depends on why you want to stem words ...

FWIW, here are some links related to those packages:

tm: https://cran.r-project.org/web/packages/tm/index.html
RTextTools: http://www.rtexttools.com/documentation.html


Expanding Standard deviation The problem I face is of a language barrier. I have a set of samples consisting of the following data:

Sample number    Organism Count
1                5
2                8
3                5
4                7
5                13
...              ...

I need to find the number of samples required to make an accurate estimate of the organism within a square meter of ground. This is for a school assignment. I don't need an answer to that. What I need to do is find the standard deviations between the set like follows:

(sample 1) = Standard deviation
(sample 1, sample 2) = standard deviation
(sample 1, sample 2, sample 3) = standard deviation

In a rolling way like that. The only information I can find about rolling standard deviation is with a specific window of samples. What I need is to grow from the first one. What is this called? I'm writing a program in python to calculate and display it in a graph. I want to know the name so I can prevent reinventing the wheel in a likely less performant way (if there is a name to this specific question). It is likely that the libraries I'm working with already provide a method to calculate what I'm asking.

So the problem is:
I need to find the standard deviation of a 'growing set' and I want to know if that has a specific name, so I can look for it in the documentation of the python library I'm using.

Feel free to change the name of the question to make it better fit, since I didn't know the name I couldn't state it in the title.

If you are using Python, you can use pandas. It comes with an expanding standard deviation function.

import pandas as pd
import numpy as np

# Generate some random data
df = pd.DataFrame(np.random.randn(100))

# Calculate expanding standard deviation
exp_std = pd.expanding_std(df, min_periods=2)

# Print results
print exp_std


Feature extraction in machine learning I am a bit confused by reading A survey on object detection in remote sensing. They state that machine learning-based object detection consists of three essential parts - feature extraction, feature fusion + dimension reduction, and classifier training. Then, they list the feature extraction methods:

Histogram of Oriented Gradients,
Bag of Words,
Texture Features,
and more.

Later in the section, they list approaches to the classifier training, e.g.:

SVM,
AdaBoost,
k-nearest-neighbor,
neural networks.

This does not align with my understanding of things. My experience is mostly with CNNs. When I use convolutional neural networks for object detection, they consist of a feature extractor and a classifier, don't they? And the feature extractor are the convolutional (and/or other) layers. I've never come across an algorithm/technique that uses e.g. Bag of Words to extract features and then feeds this to a neural net. I can imagine e.g. Bag of Words and k-nearest-neighbors or SVM. But I thought that deep neural nets' main advantage is that we don't have to do the prior feature engineering.
Question: Does the article (the part I mention here) accurately describe the state of object detection machine learning models? Is the article just outdated (it's from 2016)? Or am I missing something

I don't know much but maybe my answer would helps you. For machine learning, we have to do feature engineering to extract the important features from the data. Here we have images and the methods above to extract features of each object and image, then classify them using one of machine learning algorithms.
In CNNs, you do all these together and that's why the the CNNs object detection approach is better than ML approaches.
If you look at any of famous object detection models, you will see a CNN model such as ResNet or VGG which are used for feature extraction as a first part of the model.
Also, you can use the model for feature extractions and then apply ML algorithms to classify such as KNN, SVM, etc.


How to give a 3D Tensor as input to LSTM I'm having X_train of shape (1400, 64, 35) and y_train of shape (1400,). I want to give X_train as input to LSTM layer and also want to find the Average (using GlobalAveragePooling Layer) of the Output of LSTM at each time step and give it as input to a Dense Layer. For this problem how to connect the layers and build a sequential model?

I'm using Tensorflow.Keras API's

LSTM takes as input 3 dimension tensors (batch_size,time_step,input). So before adding a LSTM() layer you need to either use Flatten() or TimeDistributed(Flatten()) layer.


Gradient Boosted Decision Trees How to Find Prediction of Each Tree? I'm doing a project. I have a classification problem that I should solve using gradient boosted decision trees. What I want to do is create a matrix that gives prediction of each decision tree for each sample. For example if I have 100 samples and 100 trees, I should have 100x100 matrix. i, j th entry gives the prediction of jth tree for ith sample.

I'm using sklearn and problem is I can't get prediction by each tree.

So far I tried:

newgb=gb.estimators_[0][0].fit(X_train, y_train)
print(newgb.score(X_train, y_train))

where gb is already a fitted model. What I understood from documentation of sklearn

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor.staged_predict

  .estimators_

should return (number-of-trees x 1) matrix, each entry contains a tree that used by our model. By gb.estimators_[0][0] I tried to access to the first tree, and predict it with score. What I get as output is:

[0.12048193 0.95       0.95       0.95       0.95       0.95
 0.95       0.95       0.95       0.95       0.12048193 0.95
 0.95       0.95       0.12048193 0.12048193 0.12048193 0.12048193
...]

None of them are 1 or 0, like it should be(it is binary classification) and values repeat themselves like 0.95 and 0.12. I didn't use any likelihood function either so

  .score()

supposed to give me only 1's and 0's.

I don't know how to get predictions for each individual tree. I don't know what I do wrong either.

Sklearn's GradientBoostingClassifier is not implemented using trees of DecisionTreeClassifiers. It uses regressors for both classification and regression. You can read it here:

  GB builds an additive model in a forward stage-wise fashion; it allows
  for the optimization of arbitrary differentiable loss functions. In
  each stage n_classes_ regression trees are fit on the negative
  gradient of the binomial or multinomial deviance loss function. Binary
  classification is a special case where only a single regression tree
  is induced.

This means it will not be as simple as calling predict on the tree estimators. I previously suggested it could be implemented using sklearn's private methods, but as BenReiniger pointed out sklearn has already implemented this for us in the method staged_predict.


Suitable methodology to assign job titles to categories I am trying to find an automated way to assign job titles to categories.
My input data consists of job titles that are already assigned to a category and I want to be able to assign new job titles to one of those categories that fits best according to the already categorized ones.
The titles are not standardized. So one title could be "Marketing manager" that is already assigned to category A and a new title that should be assigned could be "Head of marketing". Therefore a simple 1:1 mapping of title to category will not be possible.
I appreciate any hints into the right direction.
EDIT:
The prerequisites have now changed so that I have specific job codes rather than varying job titles. So the dataset looks something like this:
category, jobCode, seniority, location
1, 1, CXO, us
1, 4, intern, us
2, 3, manager, uk
2, 4, intern, us

So the task is now to provide a function that uses the components jobCode, seniority and location to output the most suitable category based on the existing assignments.
I used word2vec to assign job titles to the jobCodes. Is this something I should expand on. E.g. just concatenate jobcode seniority and location and treat them as an input for word2veb?
I suppose there as a much more straight forward and suitable algorithm for this kind of problem.

This is regular text classification, but with very little text (only the job title). You could start with a simple one-hot encoding over the words in the job title, then apply your favourite algorithm (e.g. Naive Bayes, Decision Trees, etc). It will probably work better with some form of normalization of the words (at least using the lemma in order to mach variants of the same word).
Word embeddings are probably a good choice too, but my guess is that this would help only if you have a quite high amount of examples.


How to remove features from a sklearn pipeline after it has already been fitted? Background:
I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to fit an estimator that performs automatic feature-selection, for example, lasso regression. Thus, I begin my workflow with n features, but ultimately only m (where m < n) of them will be used to make predictions for the final model. A simple example is provided below:
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Use Wisconsin breast cancer dataset as an example
X, y = load_breast_cancer(return_X_y = True)

# Produce train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)

# Pipeline preprocesses then applies lasso regression
pipe = Pipeline(steps = [
    ('preprocessing', StandardScaler()),
    ('lasso', LogisticRegression(penalty = 'l1', solver = 'saga', C = 0.4))
]
)

# Fit the pipeline
pipe.fit(X_train, y_train)

# View the coefficients
pipe.named_steps['lasso'].coef_

print('Count of non-zero coefficients: ',
      sum(pipe.named_steps['lasso'].coef_[0] != 0))
print('Count of zero coefficients: ',
      sum(pipe.named_steps['lasso'].coef_[0] == 0))

Count of non-zero coefficients:  20

Count of zero coefficients:  10

Objective:
I am now looking to apply my model to a new set of observations to produce predictions for them, but it seems computationally wasteful to run all of the n features through the pipeline, as only m of them will ultimately contribute to the final prediction. The preprocessing steps are not theoretically required on the other n-m features once the model has been fit. Of course this example uses a toy dataset, but on larger datasets the excessive processing could be material.
# Predictions require all features to exist in the X dataset
pipe.predict_proba(X_test)

# Passing a subset of the features causes an error - and understandably so since the pipeline
# lacks the context needed to determine which features are included
pipe.predict_proba(X_test[:,np.where(pipe.named_steps['lasso'].coef_[0] != 0)[0]])

ValueError: X has 20 features, but StandardScaler is expecting 30 features as input.

Is there a solution that would allow me to 'pare down' an already-fitted pipeline object so that it can operate on only the m features that are theoretically needed to make my predictions going forward?

Standard Scalar trained on 30 features so it expects 30 features only. One simple hack you can do is, you can create a new Standard Scalar and train with those 20 features, and replace your pipeline Standard Scalar with the new one.
For the LogisticRegression, get the non zero weights and set those weights to the new model with 20 features without any training like below.
model.coef_ = numpy_array_of_non_zero_coef
model.intercept_ = numpy_array_of_intercept


Why don't the dimensions in this linear regression equation match up? I'm going through an article on linear regression, and they give the following formula for computing estimates:

The convention is that all vectors are column vectors. So if n is the number of samples and m is the number of parameters, hw(x) has dimensions n x 1, wT has dimensions 1 x m, and x has dimensions n x m. The problem is that these dimensions don't work out for matrix multiplication.

Where have I gone wrong here?

My guess is that $x$ here refers to a single datapoint with shape $m$x$1$. So $h_w(x)$ and $w^Tx$ both have shape $1$. In the case of multiple datapoints given as a matrix $X$ of shape $n$x$m$, the equation becomes:

$h_w(X) = Xw$


Number of epochs in Gensim Word2Vec implementation There's an iter parameter in the gensim Word2Vec implementation
class gensim.models.word2vec.Word2Vec(sentences=None, size=100,
alpha=0.025, window=5, min_count=5, max_vocab_size=None, sample=0,
seed=1, workers=1, min_alpha=0.0001, sg=1, hs=1, negative=0,
cbow_mean=0, hashfxn=<built-in function hash>, **iter=1**, null_word=0,
trim_rule=None, sorted_vocab=1)

that specifies the number of epochs, i.e.:
iter = number of iterations (epochs) over the corpus.

Does anyone know whether that helps in improving the model over the corpus?
Is there any reason why the iter is set to 1 by default? Is there not much effect in increasing the no. of epochs?
Is there any scientific/empirical evaluation of how to set the no. of epochs?
Unlike classification/regression task, the grid search method wouldn't really work since the vectors are generated in an unsupervised manner and the objective function is simply by either hierarchical softmax or negative sampling.
Is there an early stopping mechanism to cut short the no. of epochs once vectors converges? And can the hierarchical softmax or negative sampling objective converge?

Increasing the number of epochs usually benefits the quality of the word representations. In experiments I have performed where the goal was to use the word embeddings as features for text classification setting the epochs to 15 instead of 5, increased the performance.


Whole time column in csv file convert into UTC (epoch) using python I have a datset with time and column. I want to plot a graph with time and value. I tried many methods but didn't come proper graph. Because I have a time series. Then I thought I will convert time into UTC then try to plot it. But I didn't have any idea how to convert whole column into UTC . Can anyone help me to solve this error?

time.strftime("%Y-%m-%d %H:%M:%S",
          time.gmtime(time.mktime(time.strptime("2008-09-17 14:04:00",
                                                "%Y-%m-%d %H:%M:%S"))))

This code is for only one time . If I want to convert whole column then What should I have to do?

def convertTime(s):
tm = ((datetime.datetime.strptime(s, '%d/%m/%Y %H:%M'))-datetime.datetime(1970,1,1)).total_seconds()*1000
return

I wrote this code. But it gave me an error.
my csv file

my code is :

 condition = ""
date_time  = []
x1 = []
x2 = []
x3 = []
def convertTime(s):
  tm = time.strptime(s, " %d/%m/%Y %H:%M")
  return datetime.datetime(tm.tm_year,tm.tm_mon, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec)
with open('data43.csv','r') as csv_file:
csv_data = csv.reader(csv_file, delimiter=',')
 row_num = 0
for row in csv_data:
 if(row_num == 0):
  condition = row[0]
elif(row_num > 1): #Data starts here
  if(row[0] != ''):
    date_time.append(convertTime(row[0]))
  if(row[1] != ''):
    x1.append(int(row[1]))
  if(row[2] != ''):
    x2.append(int(row[2]))
  if(row[3] != ''):
    x3.append(int(row[3]))
row_num = row_num + 1
fig1 = plt.figure(1)
ax = fig1.add_subplot(2,1,1)
ax.plot(date_time,x1)
ax.stem(date_time,x2,'C1--','C1o',linefmt=None, markerfmt=None, basefmt=None)
ax.stem(date_time,x3,'C2--','C2o',linefmt=None, markerfmt=None, basefmt=None)
ax.legend()
ax.xaxis_date()
ax.get_xaxis().set_major_formatter(DateFormatter('%d/%m/%Y %H:%M:%S'))
plt.xlabel('t')
plt.ylabel('k')
leg = plt.legend( loc = 'upper right')
plt.draw() # Draw the figure so you can find the positon of the legend.
bb = leg.get_bbox_to_anchor().inverse_transformed(ax.transAxes)
xOffset = 0.3
bb.x0 += xOffset
bb.x1 += xOffset
leg.set_bbox_to_anchor(bb, transform = ax.transAxes)
plt.rcParams["figure.figsize"] = [20,20]
ax.plot(style='.-')
plt.show()

One of the easiest ways to solve the problem of plotting your time-series graph is by using pandas. You can use this code

data = pd.read_csv('YOUR_FILENAME_HERE.csv')
data['datetime'] = data['date'] + ' ' + data['time']
data['datetime'] = pd.to_datetime(data['datetime'])
data.index = data['date_time']
data.drop(['date', 'time', 'datatime'], inplace = True, axis = 1)
data['x1'].plot()


Classifying text documents using linear/incremental topics I'm attempting to classify text documents using a few different dimensions. I'm trying to create arbitrary topics to classify such as size and relevance, which are linear or gradual in nature. For example:

size: tiny, small, medium, large, huge.
relevance: bad, ok, good, excellent, awesome

I am training the classifier by hand. For example, this document represents a 'small' thing, this other document is discussing a 'large' thing. When I try multi-label or multi-class SVM for this it does not work well and it also logically doesn't make sense.

Which model should I use that would help me predict this linear type of data? I use scikit-learn presently with a tfidf vector of the words.

If you want these output dimensions to be continuous, simply convert your size and relevance metrics to real-valued targets.  Then you can perform regression instead of classification, using any of a variety of models.  You could even attempt to train a multi target neural net to predict all of these outputs at once.

Additionally, you might consider first using  a topic model such as LDA as your feature space.

Based on the values, it sounds like the "relevance" might be a variable best captured by techniques from sentiment analysis.


How to deal with features that have widely different dimensionalities Say I'm building a neural network to fit some classifier of some sort. To make things concrete, let's take the example of predicting housing prices using features of houses.

What should I do if one or two of my features consist of many more numbers than the other features, or even all other features combined? For example, say I have a few housing features: size in sqft, age, median income of location. These are 3 numbers. And then I have another feature, height of the roof for each square foot of the house (it's a bit contrived for this example of course) for which I would have actually "size in sqft"-numbers for this feature. So now my feature vector looks like this:

X = [1500sqft, 34 years, $54,000, 10ft, 10.1ft, 10.3ft...1497 more numbers here...]

It seems that if I just naively put this into a neural net that the first 3 features would essentially be ignored since they only account for 3/1503 features. But they might actually be important. One try might be to simply average the "height of roof" feature over all of its elements to get an "average height of the roof" feature. That makes sense for this example, but what if sometimes I don't want to take this average?
Are there any industry practices on what I might try if I ran into a problem like this?

One solution is using the same technique that is used in sequence learning. In that technique we use from Recurrent Neural Network and LSTM module.

  we can train recurrent neural networks to learn very specific outputs for an arbitrary sequence of inputs, which is very powerful.

To know more about this, you can follow this link.


Is there any single disadvantage to use GPU in deep learning? In most cases I frequently heard that to make a deep learning experiment, it is highly recommended to use GPU. It makes the computation blazingly faster than CPU, and sounds like a magical tool (although I can't use it since I use MacBook...).

However, is there any single disadvantage to use GPU, possibly except the higher monetary cost? For example the higher likelihood of getting crashed, etc...?

Transportation of data to and from the GPU. Data goes into GPU through the bus, allocated into the memory, processed and sends back the results through the same channel. Whether it is noticeable or not it takes time.

Unlike CPU which features sequential execution, GPU features parallel executions and its blazingly fast.

One main disadvantage of GPUs is that it's not built for deep learning instead its originally designed to implement graphics pipelines. As deep learning also align the same kind of computation(matrix multiplications), GPUs were used for deep learning.

More at Are there any disadvantages of using GPU in deep learning?

GPU's are very fast at applying the same instruction to multiple data points (SIMD). However, if you add branching (an if), the two branches will be serialized (first the if on all data that takes it, then the else on all data that doesn't - so still quite parallel, but not entirely). Also, if you have very few data points, the overhead of uploading to the GPU and downloading the result will likely dominate the overall execution time - it might be cheaper to just execute on the CPU instead.


Pytorch doing a cross entropy loss when the predictions already have probabilities So, normally categorical cross-entropy could be applied using a cross-entropy loss function in PyTorch or by combing a logsoftmax with the negative log likelyhood function such as follows:

m  = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
pred   = torch.tensor([[-1,0,3,0,9,0,-7,0,5]], requires_grad=True, dtype=torch.float)
target = torch.tensor([4])
output = loss(m(pred), target)
print(output)

The thing is.  What if the data at the output is already in a state with the probabilities where the variable pred already has the probabilities.  Where the data is presented like the following:

pred = torch.tensor([[.25,0,0,0,.5,0,0,.25,0]], requires_grad=True, dtype=torch.float)

How could the cross-entropy then be completed in PyTorch?

You can implement categorical cross entropy pretty easily yourself. It is calculated as

$$
\text{cross-entropy} = -\frac{1}{n} \sum_{i=0}^{n} \sum_{j=0}^m \mathbf{y}_{ij} \log \hat{\mathbf{y}}_{ij}
$$

where $n$ is the number of samples in your batch, $m$ is the number of classes, $\mathbf{y}_i$ is the one-hot target for example $i$, $\mathbf{\hat{y}}_i$ is the predicted probability distribution, and $\mathbf{y}_{ij}$ refers to the $j$-th element of this array.

In PyTorch:

def categorical_cross_entropy(y_pred, y_true):
    y_pred = torch.clamp(y_pred, 1e-9, 1 - 1e-9)
    return -(y_true * torch.log(y_pred)).sum(dim=1).mean()

You can then use categorical_cross_entropy just as you would NLLLoss in the training of a model. The reason that we have the torch.clamp line is to ensure that we have no zero elements, which will cause torch.log to produce nan or inf.

One difference you'll have to make in your code is that this version expects a one-hot target rather than an integer target. You can easily convert your current target list like so:

one_hot_targets = torch.eye(NUM_CLASSES)[targets]

where targets is a torch.tensor with integer values and NUM_CLASSES is the number of output classes that you have.


Smoothing or averaging plots to better represent trends and their variations My picture looks like this. It shows some percent variation but, due to fluctuations, curves are not very smooth at all.

I would like to draw a smoother image, without altering too much the data (it is okay if the graph is not a perfect representation of the data).
I am using matplotlib to draw the data, and my code looks like this:

x = df["Value"].shift(period)+1
y = 100*df["Value"].diff(period)/(period*x)

Basically, it draws the variation from the total over a period. What is the suggested way to alter the points (possibly drawing them alongside) the original data, so that the graphs do not look so jagged?

I tried just drawing every other third point, doing:

w = x[2::3]
z = y[2::3]

but I am sure there are better ways to do this. Again, this is for a nice representation only and I am mostly interested in showing a trend or any variation of it, not in showing the exact data points.

You could take averages piecewise. Ideally, is like sliding a window of size k on the trend, taking the average of these points (it could be 3 or 5, for example).

Alternatively, you can use exponential smoothing functions, such as savgol filter available in scipy, LOESS smoothing, or Holt-Winters smoothing. The statsmodels.tsa submodule made many smoothing techniques available.


How would you teach multiplications to a neural network? Let's say you train a neural network on Input = 1/Output = 2; Input = 4/Output = 8.

How would you train a machine to recognize that it needs to multiply the input by 2, from scratch?

You need to limit the network model to one which naturally would generalise your problem.

For a simple multiplication, this would be a single linear neuron. You can also train bias so that the network represents y = Ax + B.

If you take that very simple network give it just 2 or 3 examples to learn from, and train it using least squares error, it will quickly converge to the function you want. A single layer linear network like this can generalise to y = Ax + B where x, y and B are vectors, and A is a matrix - i.e. it can solve simultaneous linear equations provided you supply enough examples to do so (usually number of examples equals number of dimensions of the vectors).

This is actually really trivial, and if you train such a simple network using any modern NN library, it will be very fast to learn your linear function. However, in the general case of having a specific non-linear function and wanting the network to learn it in a general sense, it is not possible. The network can be made to learn to generalise reasonably well near where you have provided example inputs and outputs, but will output incorrect results when you give it an x value far away from the training examples.

It is important to note that the network does not learn the math formula for your function. It finds the best approximation given its own internal model. But by picking a network where the internal model is a good match to the function you want to learn, it is going to generalise well to it, and may be able to get it exactly if there is no noise in the examples and enough of them.


How to extract entities from text using existing ontologies? I am working on a entity extraction task and I am using Stanford CoreNLP NER. Here, I want to detect entities of type "Animal", "Building", "Imagery", etc., which are not covered in Stanford CoreNLP entity extraction method. Also, I want to detect organization entities which are not included in CoreNLP entity model. I have the data about the organizations with me.

How can I do this?

I think you are looking for Spacy, Polyglot and AllenNLP to find your NERs.


Should the baseline comparison be done on the training or the test set results? I have a classification problem where I want to find out whether feature engineering has improved my final model. Cross-validation is used evaluate the impact of the feature engineering steps, so there is no validation set (only train/test). In short, my situation entails the following:

Collect data
Train baseline model
Feature engineering
Train final model
Compare final model against baseline (question)

Comparing the baseline and final models, I assume, can be done by running both models on the test set, subsequently evaluating the differences in their results (if any). However, I wonder if it is useful to compare the models using the training set as well/instead. It would be great if someone could elaborate on this issue.

You definitely want the comparison to be based on the test set:

Evaluating on the training set doesn't make sense for all the usual reasons.
Especially in the case of different features, comparing the performance on the training set could be badly misleading: if one of the model overfits, its performance on the training set will appear better but its real performance (on the test set) is likely worse.

Note that it might make sense to study what happens on the training set (e.g. to measure overfitting), but that cannot be the real evaluation used for comparing the models.


Dealing with no data I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with missing value techniques. Also cannot give some number because even 0 and -1 are valid values for temperature. How to deal with such data?

I am thinking of putting very big value for such columns which is not possible as temperature. Please suggest.

From your question it seems that you are easily able to identify the periods assets were down for maintenance and simply wonder how to best code that in the data?

If that is the case I would simply add a new column of information marking assets as either 1 - active or 0 - down for each period. If need be you could even code it into active/down/maintenance, etc.

This would be additional valuable data for your prediction model (maybe assets are more or less likely to go down after a maintenance period). Additional it would allow you to simply mark the sensor data as missing NA because you will be able to differentiate between a broken sensor and an asset on maintenance from your other data.

As @Tasty213 suggested any good classification algorithm might have been able to do this as well but it seems you are simply able to add this data yourself.


Applying a keras model working with greyscale images to RGB images I followed this basic classification TensorFlow tutorial using the Fashion MNIST dataset.
The training set contains 60,000 28x28 pixels greyscale images, split into 10 classes (trouser, pullover, shoe, etc...).
The tutorial uses a simple model:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

This model reaches 91% accuracy after 10 epochs.

I am now practicing with another dataset called CIFAR-10, which consists of 50,000 32*32 pixels RGB images, also split into 10 classes (frog, horse, boat, etc...).

Considering that both the Fashion MNIST and CIFAR-10 datasets are pretty similar in terms of number of images and image size and that they have the same number of classes, I naively tried training a similar model, simply adjusting the input shape:

  model = keras.Sequential([
     keras.layers.Flatten(input_shape=(32, 32, 3)),
     keras.layers.Dense(128, activation='relu'),
     keras.layers.Dense(10)
  ])

Alas, after 10 epochs, the model reaches an accuracy of 45%. What am I doing wrong?

I am aware that I have thrice as many samples in an RGB image than in a grayscale image, so I tried increasing the number of epochs as well as the size of the intermediate dense layer, but to no avail.

Below is my full code:

import tensorflow as tf
import IPython.display as display
from PIL import Image
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import pdb
import pathlib
import os
from tensorflow.keras import layers #Needed to make the model
from tensorflow.keras import datasets, layers, models

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

IMG_HEIGHT = 32
IMG_WIDTH = 32

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

train_images = train_images / 255.0
test_images = test_images / 255.0

def make_model():
      model = keras.Sequential([
         keras.layers.Flatten(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
         keras.layers.Dense(512, activation='relu'),
         keras.layers.Dense(10)
      ])
      model.compile(optimizer='adam',
                   loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                   metrics=['accuracy'])
      return model

model=make_model()
history = model.fit(train_images, train_labels, epochs=10)

Your model is not sufficiently complex to adequately classify the CIFAR 10 data set. CIFAR-10 is considerably more complex than the  Fashion-MNIST data set and therefore you need a more complex model.You can add more hidden layers to your model to achieve this. You should also add DROPOUT layers to prevent over fitting. Perhaps the easiest solution is to use transfer learning. I would recommend using the MobileNet CNN model if you want to try transfer learning. Documentation for that can be found here. Since CIFAR-10 has 50,000 sample images I do not think you will need data augmentation. First try a more complex model without augmentation and see what accuracy you achieve. If it is not adequate then use the keras ImageData Generator to provide data augmentation. Documentation for that is here.


Using DONUT algorithm with keras I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows:

timestamp, values, labels = ...
# If there is no label, simply use all zeros.
labels = np.zeros_like(values, dtype=np.int32)

# Complete the timestamp, and obtain the missing point indicators.
timestamp, missing, (values, labels) = \
    complete_timestamp(timestamp, (values, labels))

however, when I do so, I get this error:

ValueError: The shape of ``arrays[0]`` does not agree with the shape of `timestamp` ((109577, 11) vs (109577,))

which does not make sense to me as I can't think of a reason timestamp would be an 11 dim array. When I pass the values as the timestamp arg, I get "timestamp must be a 1D array"

Very confused, hopefully, someone can shed some light.

Here are the checks in the code:

if len(timestamp.shape) != 1:
    raise ValueError('`timestamp` must be a 1-D array')

has_arrays = arrays is not None
arrays = [np.asarray(array) for array in (arrays or ())]
for i, array in enumerate(arrays):
    if array.shape != timestamp.shape:
        raise ValueError('The shape of ``arrays[{}]`` does not agree with '
                         'the shape of `timestamp` ({} vs {})'.
                         format(i, array.shape, timestamp.shape))

As well as the repo itself:
https://github.com/haowen-xu/donut

How does your timestamp look like? Apparently there are too many dimensions.

When using pandas DataFrames you could pass the .index (in case it's no multiindex) or just np.arange(len(<your_data>)) as timestamps.


CNN can't predict images outside the dataset I am using celeba dataset to train my CNN face landmark detection model.
Here is my model
class LandmarkModel:
    def __init__(self,inp_shape):
        self.model = models.Sequential()
        self.model.add(layers.Conv2D(16, (3, 3), activation='relu', input_shape=inp_shape))#l1
        self.model.add(layers.Conv2D(32,(3, 3), activation='relu'))
        self.model.add(layers.MaxPooling2D((2, 2)))
        self.model.add(layers.Conv2D(64,(3, 3), activation='relu'))
        self.model.add(layers.Flatten())
        self.model.add(layers.Dense(512))
        self.model.add(layers.Dense(10))

    def getModel(self):
        return self.model

I have trained my model for around 5k-6k images with loss of 0.1.
When I use image from dataset that is outside of training sample I get correct prediction.
But when I use my own clicked images predictions are completely off. I have clicked photos exactly like in dataset.
I have also tried with downloaded celeb images still wrong predictions.
What is the reason of this behavior?

That could a lot of reasons for that. For example, images are sometimes represented with numbers in range 0-1 and sometimes in range 0-255 and it's very easy to mix these ranges for in-dataset / external as the model would fail silently without any warnings.
In general, if the same model gives you different results, then images are not exactly the same.
I suggest you take the same image from your dataset and internet and compare the raw numbers on the input of the model. If results different, then input tensors should be different and you can check your preprocessing pipeline step by step to find the point of discrepancy.
The debugging tools like ipdb are usually very useful in such situations.


Convey time lag information to a linear regression model I am using a simple linear regression to predict the number of units an item has moved and price of the item is one of the input parameters.

For a few items, the older prices are not relevant and hence this results in incorrect predictions. The definition of old price varies from item to item.
Is there a way to make the linear regression know that recent prices are more relevant?

Linear regressions have fixed number of variables. An alternative could be to run a separate regression for each item group, and add a different number of time lags for each. If you can't do that, I suggest you to find an optimal number of time lags to be chosen for this general model. You can take a look at ACF, and specifically PACF plots to understand how much memory (time dependency) your time series have.

In theory, adding a more time lags should't result in a problem: if the n-th time lag is not relevant to explain your y, then it's parameter will show no statistical significance.


Machine learning model to predict the best candidate Problem: I would like to build a machine learning model that can predict the best candidate from any given set. What could be a good architecture for such a model?

Given: I have several training examples, each of which consists of:
- a set of candidates.
- a descriptor for the set as a whole.
- a label that tells which one of those candidates is the best in that set.

Details:
- I will have around 10K such sets.
- The number of candidates in every set may be different (may vary roughly from 10 to 100)
- Every set is unordered.
- The descriptor of each set is currently a fixed length one-hot vector. I'm open to add more features to it though.
- Each candidate is represented by a fixed length feature vector. (However in future, the number of features describing each candidate may also differ for every candidate).

What I tried but didn't work:
One approach I tried was a simple MLP that takes one candidate as input and outputs whether or not the candidate is the best. But since this MLP wouldn't know which set the candidate belongs to, it fails in situations where a candidate is the best in one set but the same candidate is not the best in another set.

To get into some more specifics, in my current problem, each candidate is a 2D polygon with a fixed number of line segments. Labelling on the training examples is being done manually to pick the most good looking polygon in a given set of polylines. Each polygon is described by an array of (x,y) coordinates.

One problem I face is that I don't have a natural starting point for a polygon to begin it's array of (x,y) coordinates from. Currently I'm choosing the starting point to be the one with the minimum value of x+y and going counterclockwise from there.

Currently each 2D polygon has the same number of segments. But I would soon need to support polygons with varying number of segments.

In future, I would like to extend this ML model to 3D polyhedrons too, but I don't know how to even build a feature vector to describe for 3D polyhedron yet. I guess that's a problem for another day.

I think it would make more sense to train a model to grade (regression) each candidate, them from candidates of a particular set you can use the best candidate from its "grade".

Also, you should try changing the information from raw cloud of points to more meaningful geometric form irfomation:

Number of vertices/segments
Segments length  mean and variance
Skewness
Size and direction of major and minor axis
Center position
Moments of area (first,second,third...)

Update

To apply a regression model you will need to generate grades for the training set and that might be a challenge. For that I will propose a few heuristics to generate this:

Since you have about 10k samples, you could assign to every candidate the probability of been the best candidate in any set (for example, if he is the best candidate in 10 sets you can give him a grade $\frac{10}{10,000}$.

You could try clustering the samples and assigning a grade to every cluster as the probability of a candidate in that cluster been the best candidate by $\frac{N_{best}}{N_{cluster}}$, where $N_{best}$ is the number of best candidates in any set in that cluster and $N_{cluster}$ is the number of candidates in that cluster
You can assign to each best candidate a grade like $1$ if he is best candidate in every set it appears and $1-e^{-\alpha N}$ for every $N$ times it appears in a set without been the best candidate. You will have to tune the decay rate $\alpha$ like any hyperparameter.


Proper masking in the transformer model Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short questions and would appreciate if you could clarify those:

Are the attention scores the only place (besides the loss) where masks are needed or should the input be masked out as well?

I am asking because is see implementations where a linear layer for the query, key and values is with bias=False is used.

Is the reason for setting bias=False to have zeros preserved in the output of the layers or is there a different explanation?
Should a padding_idx be used when learning word embeddings in order to zero out the padded tokens?

I will take as reference fairseq's implementation of the Transformer model. With this assumption:

In the transformer, masks are used for two purposes:

Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask.
Self-attention causality: in the multi-head attention blocks used in the decoder, this mask is used to force predictions to only attend to the tokens at previous positions, so that the model can be used autoregressively at inference time. This corresponds to parameter attn_mask.

The weight mask, which is the combination of the padding and causal masks, is used to know which positions to fill with $-\infty$ before computing the softmax, which will be zero after it.
You don't need to preserve any zeros in the output, as the attention blocks take care of that (see answer (1)). In the original Transformer article, the attention works without bias, but the bias does not change performance. Actually, in fairseq the bias are used by default.
Yes, padding_idx is certainly used to zero out padded tokens.


Incorporate luck in statistical modelling I was wondering if it is possible (and if yes, how is it done) to incorporate a luck component in statistical models.

So let’s assume I’d perform a regression on the goal difference between two teams in order to model the outcome of a sports match. How could I also model a random luck component that would make the underdog draw or win in certain rare cases?

And what if I took it a step further and tried to model the match outcome using neural networks? Is is possible to include luck in this case?

Use a probabilistic model. For example, if the probability of one team beating the other is $σ(w⋅(team1−team2))$, where $σ(⋅)$ is the logistic function, then there's always a probability that the weaker team will win.

If you wanted to use a neural network you'd use it to replace the logistic function.

Answer copied from comments.


How to handle fixed values for variables in pre-processing I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are real-valued but all same values. When doing standardization of the variables so that each is a zero mean and variance 1, these variables give NaN values. Therefore, is it ok to exclude such variables (irrespective of being categorical or real-valued) that contain constant values from the normalization/standardization step? These variables are important as features hence I cannot delete them. Is there any other way to handle such variables?

By definition, if these columns or features contain a constant value and yet the output variables change, then they are not influencing the output and likely can be ignored.
A more formal test is to determine how much of the variance between a model that uses that feature is attributable to that feature.
A simple example to illustrate this principle is to look up examples of PCA. In those examples, the technique tries and identifies feature that drive the most variance.


General approach on time series for customer retention/churn in retail I have a time series of data in the following form:

| purchase_date    |    cutomer_id  |   num_purchases | churned |
   2018-10-31            id1              39             0
   2018-11-31            id1              0              0
   2019-01-31            id1              6              1
   2019-03-31            id2              300            0
   2018-04-31            id2               2             1
    ...

I grouped the data by month and summed num_purchases by month. The churned column for user id1 for example represents in which month customer churned. So id1 in my case churned in January. Before this, to label who has churned or not, we sampled customers based on 2 months of inactivity  period from the churn date. I need to predict if a user is going to churn in a 2 months from now. I am not sure what is the best approach for this.

Q1: Should I be grouping customers like I am doing, on a monthly basis or I have to group them on a 2-month basis since that is how they were labeled?
Q2: Also, how do I model this? Do I keep customer_id as a feature of the model or not? Is the gap in dates for each customer relevant and how should I deal with it (if)? The dates repeat for different users, should I create index out of a date but it won't be unique or should I create index out of customer_id?
Q3: If I need to predict whether the user is going to churn by the end of the year for example or in the next 6 months, would that change how I group/arrange my date and model this?

I plan to add more features to this dataframe (both categorical and numerical).

OK, let me try to dissect your questions:

  Q1: Should I be grouping customers like I am doing, on a monthly basis
  or I have to group them on a 2-month basis since that is how they were
  labeled?

This depends a bit on what your goal is and what kind of features you want to use. I think, with the limited things I know from your question, I would go for a daily prognosis for the next two months (so if you're doing daily you can forecast a period of two months daily data for each day). Reason being that otherwise you might loose some inactivity "features" that are implicitly part of your data. As an alternative you can group your data and create features that help you introduce the information anyway. For example: activity_in_last_5d, activity_in_last_10d, etc.

  Q2: Also, how do I model this? Do I keep customer_id as a feature of
  the model or not?

Customer_id doesn't seem to be a feature to me but rather the key for identifying which features to feed into your model.

  Is the gap in dates for each customer relevant and
  how should I deal with it (if)?

You are basically asking about imputing your data (see this for more: https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779). Gaps can cause problems in your modeling. Some models (for example ARIMA for time series) won't work at all if you have gaps that aren't handled. Looking at your use case, I think taking the last known value for a gap should work fine since a gap means your customer didn't churn on that day.

  The dates repeat for different users, should I create index out of a date but it won't be unique or should I create index out of customer_id?

Probably go for indexing the customer_id and creating features based on the date! Maybe there are some seasonality features interesting as well (for example how many people churned on the same day last year?)

  Q3: If I need to predict whether the user is going to churn by the end
  of the year for example or in the next 6 months, would that change how
  I group/arrange my date and model this?

You would have to change the size of the period you are predicting for. Changing features, grouping, etc. might be useful as well but that really depends on what features you are choosing aka are important and how your model is going to work.


Evaluating the performance of a machine learned recommendation system I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes from the set of possible employers $E = \{{e_1,...,e_m\}}$.

From this, I have trained a machine learning model. This model then takes some $r_i$ from the test set, and assigns a probability to each member of $E$. The results are then ranked, from highest probability to lowest probability.

E.g. $P(e_2|r_i)=0.56, P(e_{52}|r_i)=0.29, P(e_{29}|r_i)=0.14,...etc.$

The resume, $r_i$ belongs to some individual, so this ranking is used to inform the individual as to what companies the model believes are most likely to hire them, given the details of what their resume contains (their skills, past employers, education, personal summary). In this case, company $e_2$ is most likely, followed by $e_{52}$ and so on.

My question is, how do you evaluate the performance of this recommendation system? Where the information need of the user is to learn what companies their resume matches to the best.

My own ideas

My understanding from information retrieval is that we need to determine some measure of relevance. From this, it's possible to use some measure like mean average precision to measure performance. Determining relevance seems like the tricky part. For instance $e_2$ has a high probability, but is it actually relevant? Maybe $r_i$ is based on aeronautical engineering, but $e_2$ is a food store, which is clearly not relevant. My current idea is to take each $r_i$ in the training set belonging to the same label $e_j$, and then compute a single TF-IDF vector which is the average of the TF-IDF vectors belonging to each $r_i$ labelled as $e_j$.

E.g. (an unrealistic example) Suppose $r_2$ and $r_9$ are labelled as $e_4$. Now suppose $r_2$ has TF-IDF vector $[0.2, 0.1, 0.5, 0.2]$ and $r_9$ has TF-IDF vector $[0.22, 0.12, 0.44, 0.22]$. Then the average of these is $[0.21, 0.11, 0.47, 0.21]$. Repeating this process for all $e_j\in E$ results in $m$ of these vectors. From this, it's possible to compute the cosine similarity between some $e_i$ and $e_j$.

Returning to the first example, we can take the true label of $r_i$, and then find the cosine similarity between this label and each member of $E$. Then we set some threshold and evaluate whether $\text{cosineSim}(\text{true label}, e_j) < \text{some threshold}$. If the cosine similarity is above the threshold, then $e_j$ is relevant, otherwise, $e_j$ is not relevant.

I'm not sure if this is a sensible/valid approach (I wonder if it defeats the point of the machine learning, since I may as well just use the cosine similarity? That said, I cannot forgo the machine learning component in this project).

Maybe this is an over complication, and something like top k accuracy would be fine. I.e. is the true label in the top k suggestions?

I'm not sure, I'm interested to have some more informed perspective.

To the extent possible you should try to evaluate based on your data rather than some ad-hoc measure. As you rightly noticed, there is a real risk that the ad-hoc measure would just confirm the predictions of the model, since it uses a somewhat similar method.

I would suggest that you split your data between a training set and test set (or even better use cross-validation), and indeed use top-K accuracy (or something similar) to evaluate on the test set. That would be the safe option for a proper evaluation, and then you could try to see if your ad-hoc measure correlates with it: if it does, then you have evidence that in the future it can be be used instead of a test set.

Side note: your instances don't contain any negative evidence such as resumes rejected by an employer. In case you could obtain this kind of data, it could probably improve the predictions.


Gaussian Process for Classification: How to do predictions using MCMC methods Problem

I was reading about Gaussian Processes for regression in the "Gaussian Processes for Classification" textbook and in a few other online resources. Everywhere I look people seem to avoid talking about one would go about doing this. Can anyone provide a simple answer to this?

Mathematics and Context

$X\in\mathbb{R}^{n\times d}$ is a matrix whose rows ${\bf{x}}_i$ are the $n$ training observations living in $d$-dimensions.
${\bf{y}}$ is an $n$-dimensional vector containing training labels $0$ and $1$ for each training input.
${\bf{f}}$ is an n-dimensional vector whose elements $f_i={\bf{x}}_i^\top {\bf{w}}$ are the so-called linear predictors.
${\bf{x}}_*$ is a testing input.

Formulation of Gaussian Process for Classification

Inference is performed in two steps:

Compute
$$
p(f_*\mid X, {\bf{y}}, {\bf{x}}_*)=\int p(f_*\mid X, {\bf{x}}_*, {\bf{f}})p({\bf{f}}\mid X, {\bf{y}})d {\bf{f}}
$$
Squish the value using the sigmoid function to find the class probability.
$$
\overline{\pi}_* = p(y_*=1\mid X, {\bf{y}}, {\bf{x}}_*) = \int \sigma(f_*)p(f_*\mid X, {\bf{y}}, {\bf{x}}_*)d f_*
$$

What I don't understand

How does one go about solving this using sampling methods? My idea is that the first integral might be similar to an expectation so maybe we can do something like this.

Get samples ${\bf{f}}_1, \ldots, {\bf{f}}_N$ from $p({\bf{f}}\mid X, {\bf{y}})$ and then approximate the first integral like this
$$
\mathbb{E}_{p({\bf{f}}\mid X, {\bf{y}})}\left[p(f_*\mid X, {\bf{x}}_*, {\bf{f}})\right] \approx \frac{1}{N}\sum_{i=1}^N p(f_*\mid X, {\bf{x}}_*, {\bf{f}}_i)
$$
but then how do I compute $p(f_*\mid X, {\bf{x}}_*, {\bf{f}}_i)$ ?

In step 1, it's no need to use MCMC, instead, we can compute the posterior with some assumptions in closed form, it's $N(E(f_*),Var(f_*))$. In the second step, we use approximation method to compute the probs by sampling from the posterior distribution.


Will one hot encoding / unbalanced columns cause bias to Clustering Analysis? I'm wondering if having too many columns about one certain feature is gonna cause bias to the clustering analysis.

For example, if my dataset has columns = ['incoming calls', 'outgoing calls', 'missing calls', 'age'], and if I run clustering algorithms such as K-means or Mixture Model, will the clustering results be biased since it splits datasets mainly based on calls?

Another example is if I have two categorical columns: color ('red','blue','green'), and shape ('circle','square'), after one hot encoding, color will expand into three columns and shape will expand into two. If I cluster on the one-hot encoded dataset, will color have a larger weight than shape in terms of splitting the data?

To answer your question we need to understand what the aim of the clustering analysis that you are doing. Some of goal's of clustering analysis are:

Outlier Detection,
Pattern Detection,
Grouping Data together, etc

Now depending on the type of data, we can choose the algorithm that best fits the data at hand. If you have only numerical features, then you can go for KMeans. If there is a only categorical features we can choose K-Prototypes algorithm and if there is a mix of both categorical and numerical, we can use the K-Medoids algorithm.

To answer your question specifically, we need to choose an appropriate clustering algorithm while using one-hot encoded data to avoid the issues that you are worried about. Since one-hot encoding is intended for categorical features, we have to go with an algorithm that is specifically designed for that. For example K-Prototypes or K-Medoids.

I would recommend you to understand the exact business problem that you are trying to solve. Correlate that to the data at hand and then use an appropriate mentioned algorithm to solve the problem that you are facing.

Regards,


Why do we need the hyperparameters beta and alpha in LDA? I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind:

First: Why do we need to add alpha and gamma every time we sample the equation below? What if we delete the alpha and gamma from the equation? Would it still be possible to get the result?

Second: In LDA, we randomly assign a topic to every word in the document. Then, we try to optimize the topic by observing the data. Where is the part which is related to posterior inference in the equation above?

LDA is a bayesian model. The equation that you gave is the posterior distribution of the model. The alpha and beta parameters come from the fact that the dirichlet distribution, (a generalization of the beta distribution) takes these as parameters in the prior distribution. So to answer your first question, will the formula above work without the alpha and gamma, yes, you would implicitly be assuming these parameters from the prior distribution are both equal to zero. You will still get a result, but the prior you are using might be problematic on a conceptual level (maybe it would cause some numerical problems too, I'm not sure?)

To answer question 2, I want to point out that in LDA we use the conjugate prior to simplify the calculation of the optimal parameter values. That is what you are seeing in the above equation. What you are seeing is the posterior value is proportional to the left hand side. Adding the "c" terms allows you to calculate the posterior.


German Chatbot or conversational AI I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set!

So does it make sense to use google translator API to translate the English dataset to German and then train the model on it?

Any idea where I can find German datasets or solve this issue?

To start with chatbot first of all you to decide which type of chatbot are you trying to build.

Order placing bot
Chitchat bot

Both required different approach to solve the problem.

Order Placing Bot required multiple model for different task like intent identification, named entity recognition, state machine.

Chitchat bot required only 2 person conversation dataset which is available easily on kaggle.com

But if you are looking for specific language dataset then it difficult to find it in both type of bots. For that either you use any translation api which you to pay for it or use web scrapping techniques to do same task at free of cost.


Why we should not feed LDA with tfidf Can someone explain why we can not feed LDA topic model with TFIDF?  What is wrong with this approach conceptually?

Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915

Direct quote:

  In fact, Blei (who developed LDA), points out in the introduction of the paper of 2003 (entitled "Latent Dirichlet Allocation") that LDA addresses the shortcomings of the TF-IDF model and leaves this approach behind. LSA is compeltely algebraic and generally (but not necessarily) uses a TF-IDF matrix, while LDA is a probabilistic model that tries to estimate probability distributions for topics in documents and words in topics. The weighting of TF-IDF is not necessary for this.

That sums it up on the high level. It would be interesting to understand more technically, why the model would perform more poorly if TF-IDF is used. Actually, there is another reply in the SO link which claims that LDA can be improved with TF-IDF.


My neural network 'learns' to estimate each class equally I've implement a neural network. It's a fully connected back propogation network. I'm training it on this data set;

Iris Data Set

I'm using that data just as a test to check if my implimentation is working.

My neural network minimises it's cost function by giving an estimate of each class equally. In this case (Iris) there are three classes and after around 1000 iterations my network output is something like this;

0.3563204029871746
0.34553220619664271
0.36660774391053058

If I run it for 2000 iterations I get this;

0.33748041999368034
0.33555095097164173
0.33929502380142113

The network is almost certainly approaching 1/3 estimate for each class. This is obviously not what is desired, but I can't figure out where my algorithm is going wrong. I can post all my code/pseudocode if it would be useful, but I'm hoping someone has seen this before and can tell me where I'm missing something.

If it matters I'm using a logistic activation function.

Edit: Pseudocode

Ok here's my code;

**Sub DataInput**
Load in the iris text file
split file into lines
redim x(#lines)
redim y(#lines)
for each line
  datain = split line into columns (comma as separator)
  redim x(line #)(3)
  x(0) = datain(0)/7.9
  x(1) = datain(1)/4.4
  x(2).... etc.

  redim y(line #)(2)
  y(0) = 0
  y(1) = 0
  y(2) = 0

  select case datain(5) 'selects which type of flower
    case is flower1
      y(0) = 1
    case is flower2
      y(1) = 1
    case is flower3
      y(2) = 1
  end select
next line
**end sub**

**sub initialise**
'sets up array sizes for all required values
'sets all weights to random
**end sub**

**sub Train**
Do until cost function is below threshold
set DR = 0 for all l, i, j 'DR is the cumulative error value from each training example at each weight (l = layer, i = neuron in next layer, j = neuron in current layer)

for i = 0 to number of training examples
  call sub ForwardPropogate(i)
  call sub BackwardPropogate(i)
end for

for l = 0 to number of layers -1
  for i = 0 to number of neuron in next layer
    for j = 1 to number of neurons in this layer
       CapD(l)(i)(j) = 1/numExamples * DR(l)(i)(j) + lambda*theta(l)(i)(j)
    end for
    CapD(l)(i)(0) = 1/numExamples * DR(l)(i)(0) + lambda*theta(l)(i)(0)
  end for
end for

for l = 0 to number of layers -1
  theta(l) = theta(l) - alpha*CapD(l)
end for
loop

**end sub**

**Sub ForwardPropogate**(byval exNum as int) ' exNum is the training example being propagated

a(0) = x(exNum) ' a(n) is the activations of all neurons in layer n
a(0) = leadingBias(a(0)) ' leading bias adds a 1 to the start of the array for the bias neuron
for i = 1 to number of layers -1
  z(i) = theta(i-1) * a(i-1)
  a(i) = g(z(i)) ' g is the activation function
  a(i) = leadingBias(a(i))
end for
z(number of layers) = theta(number of layers -1) * a(number of layers -1)
a(number of layers) = g(z(number of layers))

**end sub**

**Sub BackwardPropogate**(byval exNum as int)
delta(number of layers) = a(number of layers) - y(exNum)
for i = number of layers - 1 to 1 step -1
  delta(i) = [transpose(theta(i)) * delta(i+1)] .*g'(a(i))    g' is the derivative of the activation function
end for
for i = 0 to number of layers -1
  DR(i) = DR(i) +delta(i+1) * transpose(a(i))
end for

**end sub**

Some things you can try, to help debug your network:

Implemented a gradient check, to make sure you are calculating the gradients correctly.

Experiment with different learning rates, and different initializations of the initial weights.

Make sure to use randomization / symmetry breaking, or some suitable standard method, for initializing the weights.

Try starting with a single layer, and make sure your model can learn (you can solve some simple learning tasks) with a single-layer network.  Then try two layers.

Try a ReLU activation function.


What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels? For example I have apple and pear pictures. What I am trying to do is to predict if a picture is an apple or pear picture and AT THE SAME TIME predicting whether the fruit is big and/or yellow.

Thus predicting the fruit type (apple/pear) is multi-class problem, one vs all. Predicting whether the fruit big and/or yellow is multi-label problem.

If label ordering is [apple, pear, big, yellow] then a “big yellow apple” picture should have the label [1,0,1,1]. First two part is mutually exclusive, one-hot, however last two are not.

So, what loss function should I use for this problem?

The labels of data are not mutually exclusive so you can't say this is a one vs. all problem, because more than one entry may be one in the output vector. Moreover, if in the seen there should be an apple or pear this can be considered as an exhaustive problem which means one of them should happen for each input.

My opinion is that for this problem you don't have to make a new cost function. For the mutually exclusive part, as you have truly stated that, and for the second part of your vector, the well known cross-entropy cost function will perform fine. I guess the problem is something else. For problems with mutually exclusive classes, we use soft max layer as the last layer for neural nets while for cases which classes are not mutually exclusive, you can use sigmoid as the activation function. In your case that you have combination of them I suggest you an alternative approach:

Change your mutually exclusive part to a binary output, means if the corresponding entry is less than half, you can understand that it is e.g. an apple otherwise, it is the other class. and for the rest, just keep the output vector as it is. Finally use sigmoid activation function as the last layer if you are using neural nets.


Where can I find free spatio-temporal dataset for download? Where can I find free spatio-temporal dataset for download so that I can play with it in R ?

If you have R and the spacetime package then you are only data(package="spacetime") away from a list of space-time data sets bundled with the package:

Data sets in package ‘spacetime’:

DE_NUTS1 (air)          Air quality data, rural background PM10 in
                        Germany, daily averages 1998-2009
fires                   Northern Los Angeles County Fires
rural (air)             Air quality data, rural background PM10 in
                        Germany, daily averages 1998-2009

then for example:

> data(fires)
> str(fires)
'data.frame':   313 obs. of  3 variables:
 $ Time: int  5863 5870 6017 6018 6034 6060 6176 6364 6366 6372 ...
 $ X   : num  63.9 64.3 64.1 64 64.4 ...
 $ Y   : num  19.4 20.1 19.7 19.8 20.3 ...


In generative adversarial models (GANs), why should we solve min-max problem and not max-min? I know that in GANs model, there is min-max game between generator and discriminator which discriminator tries to maximize the loss function and the goal of generator is to minimize it. But why we write the loss function as min-max problem and not max-min? As I know, the loss function is not convex, so there is difference between these approaches.

You can use either a min-max or a max-min game, it doesn't matter. Look, this is the "usual" loss function (i. e. if you use the original loss proposed by Goodfellow et al. in their paper on GANs, though I use a slightly different notation):
$$\min_{G}\max_{D}\mathcal L_{\text{GAN}}\left(D, G \right) = \mathbb E_{x \sim p_{\text{data}\left( x\right) }}\left[ \log \left\{ D\left( x \right) \right\} \right] + \mathbb E_{z\sim p_{z}\left( z\right)}\left[ \log\left\{ 1 - D\left( G\left( z\right) \right) \right\}\right].$$
Now, you could also write this as:
$$\max_{G}\min_{D}\mathcal L_{\text{GAN}}\left(D, G \right) = \mathbb E_{x \sim p_{\text{data}\left( x\right) }}\left[ \log \left\{ 1 - D\left( x \right) \right\} \right] + \mathbb E_{z\sim p_{z}\left( z\right)}\left[ \log\left\{ D\left( G\left( z\right) \right) \right\}\right].$$
I hope that clarifies your doubt.


Is it better to augment both training and validation sets or just training set? Is it better to augment data both training and validation sets or just the training set in order to achieve the best accuracy possible on a convolutional neural network? why?

Is it better to augment data both training and validation sets or just
the training set in order to achieve the best accuracy possible on a
convolutional neural network?

Augmentation is always a good thing provided it represents the actual distribution of data. Augmenting your training set may give you enough samples to train high variance low bias models [as an increase in training set samples, which if are general enough, prevents overfitting]
Validation set augmentation gives you more samples to validate on so it's always good.
But there may be instances where the augmented data closely resembles the initial training set samples. This increases the number of certain samples data points compared to other samples. This may cause the model to overfit due to the biased nature of the dataset. So its always wise to make sure that the samples in your augmented set are not mere copies of your data. For eg: In images, instead of just rotating them, you can add in a random noise.


Hyperparameter Tuning Time Series in Production I have a time series data that handled using GDBT to predict the next value. I always use previous 30 days data to train daily, but overtime the data to predict and train is increased because the number of combination things increased.

My question is, how often do we need to hypertune our model? and what eval number combination considered to be enough for hyperparameter tuning? 50? 100? or just 10 is enough?

right now I do it  daily, but it getting costly and costly, previously it just aroung 10 minutes, but now about more than hour. The system need to do other things hourly so this hypertuning model will be a deadlock for the system.

Some ideas:

Number of previous observations to use: depends on the process you are modelling. If the target is related in some way to many of the previous values, you may need to use more data in your tuning process. On the other hand, if the target is largely unrelated to previous observations you can use fewer. You will need to check the relationship - lag plots and auto-correlation plots may be useful.
Frequency of parameter tuning: again, depends on your model and process. It lay be a good idea to monitor the prediction accuracy of your model, and to only perform more parameter tuning if the model accuracy drops below a certain threshold.


How to handle data collecting bias in machine model training In many ML problems we collect data and train models using the collected data. Using recommendation as an example, data collected could be biased for various reasons:

presentation bias. For example, on web one may have pagination; for mobile one may need to scroll down to see items beyond a few.
bias from existing recommendation system: usually data is collected on a running production recommendation system, this introduces bias. For example, due to personalization, a male consumer may have little chance to see female products, etc.

What are approaches to handle this when training a model?

In the world of Big data, the data is already there and there are more than enough ML models currently but constantly there are improvements and applications, the main reason for that is how the features are defined from the raw data.

In your case, introduce a new feature as penalty for the bias present in the data. For ex. in the presentation bias, high penalty is given to the top items and low penalty is given to the items farther down. similarly, introduce weight feature with less weight for men products when most of the products are male products and vice versa.

ML still needs human intuition in the form of features ;-)


Clasification - ROC Curve with very high number of false negatives I have the following curve as the result of a azure machine learning classification experiment.

This obviously shows a lot of false negatives, but I'm struggling to interpret this, as my data has a high number of negatives and a low number of positives (I'm doing a binary classification).
I am interpreting this as saying my data is almost entirely made up of positive values, with the majority being identified as negative. I believe that the false negatives on the chart are actually true negatives. Am I reading the chart correctly?

Notice how the precision is very high and all of the other metrics are very low.  Now look at the class balance of your problem:

$$TP+FN=Actual Positive=31,245$$
$$TN+FP=Actual Negative=508$$

So your data is heavily skewed toward positives.  To have gotten a model that is producing this poorly, I think you may have provided the model with the precision as the cross validation metric.  The precision is a very bad cross validation metric in this case since it will result in poor accuracy and poor recall.  accuracy is also not a good metric as your model could classify everything as positive and get an accuracy of:

$$AC=\frac{31,245}{31,763}=.984$$

For cases like this where the classes are grossly weighted toward one value, I suggest using the F1-score as your cross validation metric.  The F1-score is the harmonic mean of precision and recall and hence balances these two factors nicely.  Wikipedia actually has a very nice explanation of classification metrics here and this paper is top notch if you even need to understand multi-class metrics and confusion matrices.

Hope this helps!


How to select best feature set and not ranking for tree based models? I am currently using feature selection approaches like filter, wrapper, embedded etc.

All these methods give different set of features and I rank them based on their frequency of occurrence in other feature selection approach.

Ex: If Age occurs in all 6 feature selection algorithm, then they will have a rank as 6. If gender occurs only in 2 feature selection algorithm, its rank would be 2.

So I arrange them in descending order and choose the features which has occurred in at least 3-4 feature selection algorithm.

But where I am trying to seek your help is

1) Is there any systematic way where I get only a subset of features which returns best output? I thought genetic algorithm for feature selection will return a feature subset which will give high output. But unfortunately it lists all features with their importance. I don't wish to define a threshold myself to select few. Example is currently Xgboost produces an F1-score of 81 with around 27-28 features but when I manually play around, it gives around 81.9 or 82.1 with just 6-7 features.  So I would like to have a way to select this 6-7 features automatically/systematically without human intervention

2) Is there any algorithm, like RFE but which does exhaustive search like genetic algorithm and finally returns us the best feature set? Mix of RFE and genetic algorithm is what I need

3) I wish to have an exhaustive search in feature space and finally provide me the best feature set which will provide best f1-SCORE which is my objective.

I am looking for a systematic way/algorithmic way to do this rather than me deciding to pick features which occurs at least in 3 feat selection algorithms

Hope my question is clear. Can you help me with this?

I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If anyone knows a python implementation, please let me know!


when to use dot product and when to use the common product In neural networks? I wanted to know when to use dot product and when to not , I also don't know when we must transpose an array and why should we , could someone help me to understand this ?
If you could give me the derivative of cost function with respect to each Weigh (W1, W2, W3) using notations bellow to compare it with the results I found I'll appreciate that !

Neuron-level operations:

First, let's describe the output of each neuron in the network individually.

For the first hidden layer:

$$ A_2^1 = f_2(Z_2^1) = f_2(X \cdot W_1^1)$$
$$ A_2^2 = f_2(Z_2^2) = f_2(X \cdot W_1^2)$$
$$ A_2^3 = f_2(Z_2^3) = f_2(X \cdot W_1^3)$$

Let me explain the notation a bit: $A_2^1$ is the output of the first neuron of the second layer. $f_2$ is the activation function of the second layer. $W_1^2$ is line connecting the second neuron of the weights you denote $W1$ in the figure.

Second hidden layer:

$$ A_3^1 = f_3(Z_3^1) = f_3(A_2^1 \cdot W_2^{11} + A_2^2 \cdot W_2^{21} +  A_2^3 \cdot W_2^{31})$$
$$ A_3^2 = f_3(Z_3^2) = f_3(A_2^1 \cdot W_2^{12} + A_2^2 \cdot W_2^{22} +  A_2^3 \cdot W_2^{32})$$
$$ A_3^3 = f_3(Z_3^3) = f_3(A_2^1 \cdot W_2^{13} + A_2^2 \cdot W_2^{23} +  A_2^3 \cdot W_2^{33})$$

Again $A_3^1$ is the output of the first neuron of the third layer and $f_3$ is the activation function of the the third layer. This time $W_2^{12}$ corresponds to the weight $W2$ that connects the first neuron of the previous layer to the second neuron of the current layer.

For the output layer:

$$ \hat y = A_4 = f_4(Z_4) = f_4(A_3^1 \cdot W_3^1 + A_3^2 \cdot W_3^2 + A_3^3 \cdot W_3^3)$$

The notation follows the rules above.

Note that all the above operations are between simple numbers, meaning that all $\cdot$ symbols you saw above are common multiplications! Because this way of representing Neural Networks isn't effective, instead of describing each neuron individually, we usually describe the operations of layers with vectors or arrays. This is where your question lies (about the dot product and the transposing).

Layer-level operations:

First hidden layer:

$$ A_2 = f_2(Z_2) = f_2(X \cdot W_1) $$

In the above equation $W_1$, $Z_2$ and $A_2$ are all $1 \times 3$ arrays. While $X$ remains a simple number (you can think of it as a $1 \times 1$ array).

Second hidden layer:

$$ A_3 = f_3(Z_3) = f_3(A_2 \cdot W_2) $$

Again $A_3$, $Z_3$ and $A_2$ are $1 \times 3$ arrays. This time $W_2$ is a $3 \times 3$ array.

Output layer:

$$ A_4 = f_4(Z_4) = f_4(A_3 \cdot W_3) $$

Here, $A_4$ and $A_4$ are numbers and $W_3$ is a $3 \times 1$ array.

Every $\cdot$ in the above is a basic matrix multiplication. Because this example wasn't the best to answer your question, I'm going to change it a bit. Imagine if the second hidden layer had $4$ neurons instead of $3$. In order for us to get from a $1 \times 3$ array to a $1 \times 4$ one we would need a weight array with a shape of $3 \times 4$.

$$
 \begin{pmatrix}
  A_2^1 & A_2^2 & A_2^3
 \end{pmatrix}
\cdot
 \begin{pmatrix}
  W_2^{11} & W_2^{12} & W_2^{13} & W_2^{14}\\
  W_2^{21} & W_2^{22} & W_2^{23} & W_2^{24}\\
  W_2^{31} & W_2^{32} & W_2^{33} & W_2^{34}
 \end{pmatrix}
=
 \begin{pmatrix}
  Z_3^1 & Z_3^2 & Z_3^3 & Z_3^4
 \end{pmatrix}
$$

This is summarized as $A_2 \cdot W_2 = Z_3$. Because in matrix multiplication the order of the two matrices counts, we need to make sure that the operation is possible. If they are in a reverse order we need to transpose the weight array. This would result in the following equation: $W_2^T \cdot A_2 = Z_3$.


Skip NA in Mean function within Pandas agg function Here's my working code:
new_df = old_df.groupby('store', as_index = False).agg({'reviewScore': 'mean'})

I would like to add an argument to the mean function: skipna=True
I can't figure out the syntax to make it happen.
Do I need to learn how to use lambda functions?

NaN values are skipped automatically. For example if we have the following table:
  store  reviewScore
0     a          4.0
1     b          3.0
2     a          3.0
3     a          NaN
4     b          5.0

groupby and find mean for each group:
df.groupby('store', as_index = False).agg({'reviewScore': 'mean'})

what is equivalent to:
df.groupby('store', as_index = False).mean()

Output:
  store  reviewScore
0     a          3.5
1     b          4.0

To use arguments in aggregation functions you can utilize a lambda function:
df.groupby('store', as_index = False).agg({'reviewScore': lambda x: x.mean(skipna=True)})

or
df.groupby('store', as_index = False).agg(lambda x: x.mean(skipna=True))

The result should be the same.


Avoid hardware limitation while competing in Kaggle? I've learned machine learning via textbooks and examples, which don't delve into the engineering challenges of working with "big-ish" data like Kaggle's.

As a specific example, I'm working on the New York taxi trip challenge. It's a regression task for ~ 2 million rows and 20 columns.

My 4GB-RAM laptop can barely do EDA with pandas and matplotlib in Jupyter Notebook. However, when I try to build a random forest with 1000 trees, it hangs (e.g. Kernel restart error in Jupyter Notebook).

To combat this, I set up a 16GB-RAM desktop. I then ssh in, start a browser-less Jupyter Notebook kernel, and connect my local Notebook to that kernel. However, I still max out that machine.

At this point, I'm guessing that I need to run my model training code as a script.

Will this help with my machine hanging?
What's the workflow to store the model results and use it later for prediction? Do you use a Makefile to keep this reproducible?
Doing this also sacrifices the interactivity of Jupyter Notebook -- is there a workflow that maintains the interactivity?

My current toolkit is RStudio, Jupyter Notebook, Emacs, but am willing to pick up new things.

Yes - a Python script will have less overhead than a Jupyter Notebook
Pickle is the standard way to store a scikit-learn model, see model persistence documentation.
The two primary ways to scale Jupyter Notebooks is vertical (rent a bigger machine on a cloud service provider) or horizontal (spin-up a cluster).


how to apply similarity algorithm(or comparision) of over one million vectors with other one million vectors? How can I apply similarity algorithm (or comparison) of over one million vectors with another one million vectors?

I am following this pyimage search tutorial but don't know how to scale up the algorithm when I have millions of images to compare. Like comparing similarity of over 1 million images with other 1 million images. It stops with an out of memory error at 100000.

You might want to take a look at this project FAISS.

  Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.

It has nice wrappers for you to use from Python. Check the wiki for examples on how you can integrate this in your application.

It is built with scalibilty in mind, supporting indexing and comparisons of millions of vectors and it can be run either on CPU or GPU.


Build a binary classifier with only positive and unlabeled data I have 2 datasets, one with positive instances of what I would like to detect, and one with unlabeled instances. What methods can I use ?

As an example, suppose we want to understand detect spam email based on a few structured email characteristics. We have one dataset of 10000 spam emails, and one dataset of 100000 emails for which we don't know whether they are spam or not.

How can we tackle this problem (without labeling manually any of the unlabeled data) ?

What can we do if we have additional information about the proportion of spam in the unlabeled data (i.e. what if we estimate that between 20-40% of the 100000 unlabeled emails are spam) ?

My suggestion would be to attempt to build some kind of clustering on your unlabeled data that somewhat approximates a labelled dataset. The rationale is more or less as follows:

You have some feature vector for representing your documents
Based on that feature vector, you can come up with a number of different clusterings, with either fuzzy, rough, or class-based clustering methods
Knowing what a positive example looks like, you can quickly evaluate the overall similarity of a cluster to your positive cluster
Knowing that there should really only be two clusters, you can adjust the hyperparameters on your clustering method so that the above two metrics are closer and closer to satisfaction
With the two clusters, you have what is likely a close approximation of a labelled dataset, which you can then use as a silver-standard corpus of sorts to actually train your model

Hope that makes sense, if you're specifically looking for clustering algorithms, a few that I personally enjoy that might be good in this scenario are FLAME and tsne. Alternately, looking at the spectacular gensim library in python will get you a long way toward the clustering you're looking for.

Hope that helps and makes sense, leave a comment if you've got any questions.