How to use Keras predict_generator() for segmentation output? Below is the code I'm using for segmentation mask prediction after using fit_generator(...) on model named m :-

test_datagen = ImageDataGenerator(rescale=1./255)

test_generator.reset()

test_generator =test_datagen.flow_from_directory('result/test', class_mode=None,seed=1, color_mode="grayscale",target_size=(256,256),batch_size=1)

results = m.predict_generator(test_generator,steps=17, verbose=1)

It runs without any errors but how do I visualise the predicted segmentation masks from results?

Okay, I checked that m.predict_generator returns a numpy array so I just plotted it using plt.imshow(). Obviously I'd to do some slicing and squeeze on it.

Sorry if it was too trivial


Predicting next action to take to reach a final state Does anyone know of an algorithm that could be used to determine the next action to take to reach a desired state when trained on time-series data?

For example, a robot starts at a certain state, then takes an action to get to another state. This occurs continuously for many iterations (imagine the robot is randomly exploring a room). If the robot is at a specific starting state, and I desire the robot to end up in a different state, is there an algorithm that could recommend the best next action (or set of next actions) to take to reach that final desired state?

One approach I've tried is to use a neural network with the current state and the next state being the input and the action to get from the current state to the next state being the output. The network would know for a single state how to get to a next desired state that is one action away. The issue is, what if the desired state is many actions away?

The problem you've described can be formalized as a Markov decision process, the foundational problem of reinforcement learning. In broad strokes, reinforcement learning is concerned with how agents (robot) in a given environment (room) out to take actions (movements from one state to another) to maximize some notion of reward.

Formalizing your problem requires defining a few parts of an MDP model:

Set of states $S$
Set of actions $A$
A reward function $R(s)$, defining the reward of arriving in a given state. In your case, a simple scheme of $R(s) = \mathbb{1}(s = s_{goal})$ is one option.
A transition function $T(s,a,s')$ giving the probability of winding up in state $s'$ having taken action $a$ from state $s$.(Note that you can model deterministic transitions by returning a value of $1$ for a single $s'$.)

If the problem goes on infinitely, you'll also need a discount factor $\gamma$.

In reinforcement learning, the term optimal policy describes a function that returns the best action to take from a given state. I.e., this gives the recommendation you're looking for.

If you know all of the model components described above—or can at least derive ones that match your problem—you can use a variety of planning algorithms to find the optimal policy, e.g. value iteration or policy iteration.

If you don't know the rewards and transitions—say, for example, that the robot is seeking a sensor attached to a charging station and you don't know where it is or the size of the enclosing room—you'll likely need to explore algorithms that observe action-outcome pairs and try to learn optimal policy from these learning episodes.

A full description of these is beyond the scope of your question, but a good place to start is Sutton & Barto's Reinforcement Learning: An Introduction. An html version is freely available. Another resource is RL udacity course produced by Georgia Tech.

In your example, you may also want to research potential-based reward functions. Very loosely, one potential might be the robot's distance to goal state, and the reward would be based on changes to this potential value. (This is described in a paper by Ng, Harada and Russell, and in unit 6 of the GA Tech course mentioned above.)


Validation accuracy is always close to training accuracy I am trying to tune the hyperparameters of a LSTM I have to do time series forecasting.  I have noticed that my validation accuracy is always very close to my training accuracy.  I am not sure whether or not this is good or bad or what it implies in general?

At the moment I have kept all the hyperparameters the same and have only varied the number of units in the LSTM layer from [1, 5, 26].

I expected 26 units to give me good results and have since added units 1 and 5 to help me investigate.  I was also expecting to see that my validation accuracies are worse than my training accuracies but this does not seem to be the case.  My validation accuracies track the training accuracies very well.  Is this something to be concerned about?

The plot below shows the average loss, average training and validation accuracies.

You can see from the plot that each of the units training and validation accuracies stay very close together.  Why is this?  Does this mean that in general my model should generalise well on unseen data as there isn't much of a discrepancy?

Obviously the accuracies of the model are not very good yet in general and it requires some tuning but to do the tuning process I was more expecting to see some differences between the training and validation data and not that they would stay largely similar to each other.

EDIT:

Further information as requested.  The hyperparameters used for each the model can be seen in the legend of the plots. Here is an updated plot which shows a greater variation in accuracy from varying the number of neurons and batch size:

From the plot it seems as if the model accuracy converges the quickest with a batch size of 1 and 39 neurons.  However it is characteristic amongst all that the training and validation accuracies track each other closely.  I hadn't expected that varying two hyperparameters independently would lead to a consistent result like that.

In this problem I am working with, the model is to provide forecasts for one time series.  I am using it as a 'toy' problem to try and learn what kind of models work well for my particular problem.

I have 315 data points in total for the time series.  I have left the last 52 out as a hold out test set which I haven't ever looked at to this point. 52 points as I am making weekly predictions. I then left the next last 52 points out to use as a validation set.  Due to the time dependency I am unable to use a validation method such as KFold cross validation.  Therefore I am using something called rolling origin analysis which is explained well here (see under predictive performance).  this simulates well how the model would be used in practice and my accuracies are measurements using a modified sMAPE formula for a multi-step forecast.

Essentially what this means is that I train (in this case) 27 separate models.  I have a forecast horizon of 26, so what I do is take my train data and take the first 26 points of my validation set.

I train my first model on the train data (which is 131 samples) and this is validated against 1 sample.

I train the next model which uses the same training data except it is shifted one along so the very first point from train is dropped and the first point from the validation set is appended to the end of the train set.  The validation set is then also shifted along by one.  As a quick example (numbers represent indices of the time series):

Model1 train: [1, 2, 3, 4, 5], val: [6, 7, 8]

Model2 train: [2, 3, 4, 5, 6], val: [7, 8, 9]

The average accuracy shown in the plots is therefore the average of the accuracies that each model computed against its own validation set.

So each model is trained on the same amount of data and each model is completely fitted anew, that is Model1 doesn't share any information with Model2.

The accuracies are computed in this way because I do not have much data and this is the best way I have found that allows me to get more than two validation sets and more than two test sets.

The fact that the training accuracy and the validation accuracy are close it is nothing to be concerned about. As you mention, it means your model are generalizing well.

The thing to be worried about is the low training accuracy. It seems that your model is under-fitting the data but with low variance. This is the typical scenario of a model with high bias and low variance.

The bias is the error (whatever you measure this error) in your training data. It tells you about how well the model is forecasting with seen data. High bias means poor performance (under-fitting). The variance is the difference in performance between the training data and the validation data. If the differences are high (far better performance in training than in validation) the model has high variance (over-fitting).

Typically, decreasing the bias implies increasing the variance (bias-variance trade-off) then, probably, as you get better results in training, the validationa ccuracy may decrease. But, if the model is well designed, you may end up with low bias and low variance.


MAD vs RMSE vs MAE vs MSLE vs R²: When to use which? In regression problems, you can use various different metrics to check how well your model is doing:

Mean Absolute Deviation (MAD): In $[0, \infty)$, the smaller the better
Root Mean Squared Error (RMSE): In $[0, \infty)$, the smaller the better
Median Absolute Error (MAE): In $[0, \infty)$, the smaller the better
Mean Squared Log Error (MSLE): In $[0, \infty)$, the smaller the better
R², coefficient of determination: In $(-\infty, 1]$ not necessarily the bigger the better

Are there any strong reasons not to use one or the other?

Well actually these can give you different insights into your models errors.
If $y$ is your target, $p$ your prediction and $e = p - y$ the errors:

Mean Error: $ME = mean(e)$

In (-∞,∞), the closer to 0 the better.
Measures additive bias in the error. Unbiased estimates should have the same mean as your target thus ME should be close to 0, if it's positive your predictions overestimate the target, if it's negative they underestimate.

Root Mean Squared Error: $RMSE = \sqrt{mean(e^2)}$.

In [0,∞), the smaller the better.
Measures the mean square magnitude of errors. Root square is taken to make the units of the error be the same as the units of the target. This measure gives more weight to large deviations such as outliers, since large differences squared become larger and small (smaller than 1) differences squared become smaller.

Mean Absolute Error: $MAE = mean(|e|)$.

In [0,∞), the smaller the better.
Measures the absolute magnitude of errors and it's units are the same as the units of the target. Makes for more easily interprectable errors and gives less weight to outliers. However a model with good $MAE$ can have punctually very high errors.

(Root) Mean Squared Log Error: $MSLE= mean((log(p+1)-log(y+1))^2)$.

In [0,∞), the smaller the better.
This is useful when dealing with right skewed targets, since taking the log transform makes the target more normally distributed. In practice it's usually achieved by changing the target to $\hat{y}=log(y+1)$ and then predicting as $y=e^\hat{y}-1$

Median Absolute Deviation: $MAD = median(e - median(e))$.

In [0,∞), the smaller the better.
This is a spread metric similar to standard deviation but meant to be more robust to outliers. Instead of taking means of squares as the sd, MAD takes medians of absolutes making it more robust.

R², coefficient of determination:

In (−∞,1] the closer to 1 the better
Is a measure of the ratio of variability that your model can capture vs the natural variability in the target variable.

In practice I usually use a combination of $ME$, $R^2$ and: $RMSE$ if there are no outliers in the data, $MAE$ if I have a large dataset and there may be outliers, $RMLSE$ if the target is right skewed.

This link offers a very nice overview on the matter:
http://www.cawcr.gov.au/projects/verification/#Methods_for_foreasts_of_continuous_variables


K-means clustering on the data frame having only one column unsup_df is a DataFrame which has only one column: review.

I want to form 2 clusters of the reviews. One positive and one negative.

from sklearn.cluster import KMeans

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(unsup_df)
num_clusters = 2
km = KMeans(n_clusters=num_clusters)
km.fit(tfidf_matrix)
clusters = km.labels_.tolist()

The above piece of code is throwing an error:

  ValueError: n_samples=1 should be >= n_clusters=2

on the line km.fit(tfidf_matrix)

Here is how to fit k-means to single dimensional text data in Pandas:

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer

df = pd.DataFrame({"corpus": ["I am Sam. Sam-I-am",
                              "That Sam-I-am! That Sam-I-am! I do not like that Sam-I-am",
                              "Do you like green eggs and ham?",
                              "I do not like them, Sam-I-am. I do not like green eggs and ham"]})

x = TfidfVectorizer().fit_transform(df.corpus)
km = KMeans(n_clusters=2).fit(x)
km.labels_.tolist() # Results in a list similar to this: [0, 0, 1, 1]


How do I compare traditional classifiers performance with my proposed method? I want to compare my proposed method with traditional machine learning classifiers like Multilayer Perceptron(MLP) and SVM to check the classification accuracy.How do I compare different classifiers?. I want to draw a comparison between these three terms

1).Proposed method (Based on DNN)
2).Shallow neural networks
3).SVM

What about doing cross validation on your training set? Once you have the different train/test splits I would start by printing the accuracy (number of correct predictions / total predictions) and the confussion matrix for each method.

If you are using python sklearn.metrics offers a wide variety of useful functions.


Doc2Vec Input from Paragraphs I understand how doc2vec works, but I am unclear the best practice on feeding in data.

Suppose we have a document with multiple sentences

['I really love football. Peyton Manning was a great player']

If we feed this into the algorithm as is, the window for 'Peyton' could include ['love','football','Manning','was']

This doesn't make intuitive sense to me, however, since the words come from different sentences.

Any suggestions?

The aim of Doc2Vec is to produce document level embeddings, thus even if words are sentence-separated if you include them in the same document it has to be considered part of the same semantic source for word similarities. If you don't want such a behavior, you might want to separate your documents in a sentence-level and maybe aggregate the embeddings later in different groups to form your initial paragraphs (depending on your plans)?


About confidence/prediction intervals: parametric methods VS non-parametric (via bootstrap) methods About the methodology to find confidence and/or prediction intervals in, let's say, a regression problem, I know 2 main options:

Checking normality in the estimates/predictions distribution, and applying well known Gaussian alike methods to find those intervals if the distribution is gaussian
Applying non-parametric methodologies like bootstraping, so we do not need to assume/check/care whether our distribution is normal

With this in mind, I would basically always go for the second one because:

it is meant to be generic, as it does not assume any kind of distribution
it feels more like experimental as you can freely run as many iterations as you want (well, if it is computionally feasible)

The only drawback I could see is the computational cost, but it could be parallelized...

Can anyone give another point/advice?

The simplest type of bootstrapped confidence intervals, percentile intervals (described in simple terms here) are by no means the only way of creating bootstrapped CIs. You should think of bootstrapped confidence intervals as a family of techniques - there are some examples listed here along with some good information around the assumptions underlying the use of bootstrapped statistics.

How many original samples you need in order for bootstrapped confidence intervals to converge varies. Like the parametric approach, this convergence will arrive at the wrong value to the extent that your sample population is not representative of the global population you are estimating. A rule of thumb is to use parametric approaches if the sample population is n < 30, but in principle bootstrap could sometimes work with smaller samples (see this thread). If you have 30 samples or more, standard practice is to draw >1000 bootstrap samples, but the important thing is that you see the estimates converging.

If you want to use bootstrap CIs you should be really explicit about the approach you have used whenever you present your analysis, because different ways of producing them can lead to different intervals.


Sensor fusion using recurrent neural network: obtaining a smoothed output I am trying to use a recurrent neural network to perform sensor fusion for an inertial measurement unit. IMUs are commonly used in conjunction with a Kalman filter (KF), which performs both fusion of accelerometer and gyroscope data along with 'smoothing', which results in a filtered, final output of orientation (and position if desired). Out of curiosity, I am trying to replicate this functionality using a neural network containing LSTM units: i.e., get the network to compute the hidden representation that Kalman filters estimate using training data. In order to get the network to replicate the working of a Kalman filter fusion algorithm, my first step was to train it on inputs of raw sensor data and the output coming from a KF.

The network takes 6x10 + 6 = 66 inputs (3 velocities from gyro, 3 accelerations from accelerometer from the present instant along with those from the past 10 instants of time) and aims to provide 4 outputs (quaternion representation of orientation). The network structure is

Input -> LSTM units (128) -> Dense layer (128) -> Dense layer (64) -> Output(4)

My initial training data consisted of noisy sensor data from the gyro and accelerometer in the input and smoothed KF orientation data as output, from which I was expecting the network to attempt to learn the nonlinear mapping between the sensors and the orientation. Yet, although the output I am training on is clean, there is a lot of noise in the network's predictions, even on the training data. For example:

Kalman filter output (used as output in training data):

Corresponding predictions from neural network:

The noise I am seeing in the predictions is not unnatural, because the input sensor data is very noisy as well, and the amount of noise in the output corresponds to that of the inputs: but is it not right to expect the neural network to 'learn' the smoothing as well, along with the fusion? Is there any way to reduce the noise and get the network to perform closer to the way the KF does?

To remove the noise I'd suggest you utilize Fourier Transformation to get the frequency spectrum of the signal. Then you can apply a threshold to remove high-frequency signal components which are mainly noises.

As I understood you are using python. The code below creates an artificial noisy signal and removes it using FFT (Fast Fourier Transformation) algorithm.

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0, 10, 0.01)

y = np.sin(x)
y_noisy = y + np.random.uniform(low=0, high=0.5, size=len(y))

fft_values = np.fft.fft(y_noisy)

mean_value = np.mean(np.abs(fft_values))

threshold = 10 * mean_value  # Fine-tune this

fft_values[np.abs(fft_values) < threshold] = 0

filtered_samples = np.fft.ifft(fft_values)

plt.plot(x, filtered_samples)
plt.show()

Good Luck!


Why is logistic regression not sigmoidal? The blue dots are the raw data and the line is my logistic regression. The line is quite straight and not sigmoidal as I would have expected. I suspect there is something wrong in my gradient descent equation but I don't understand the maths well enough to find the mistake.

This is the code I used to generate the logistic regression:

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from scipy import stats

def logit(z):
    return (1/(1+np.exp(-z)))

def cost(data, weights):
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    cost = np.sum(y_val*scores - np.log(1 + np.exp(scores)))
    return cost

def gradient_descent(data, weights, learning_rate):
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    weights -= (learning_rate/m)*np.dot(np.transpose(x_val),(logit(scores)-y_val))
    return weights

dataset = pd.read_csv("/Users/An/Desktop/data/glass.csv")
dataset.sort_values('Al', inplace=True) # sort by ascending "Al" values
dataset.head()
dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})
ones = np.ones(len(dataset))
data = np.stack((ones,dataset["Al"], dataset["binary"]), axis=-1)

weights = np.random.rand(2)

def log_reg(data, weights):
    for i in range(9000):
        weights = gradient_descent(data, weights, 0.000001)
        loss = cost(data, weights)
    return weights

w = log_reg(data, weights)

if would suggest you implement the following formulas:

Sigmoid activation function

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

Output (prediction) formula

$$\hat{y} = \sigma(w_1 x_1 + w_2 x_2 + b)$$

Error function

$$\text{Error}(y, \hat{y}) = - y \log(\hat{y}) - (1-y) \log(1-\hat{y})$$

The function that updates the weights

$$ w_i \longrightarrow w_i + \alpha (y - \hat{y}) x_i$$

$$ b \longrightarrow b + \alpha (y - \hat{y})$$

If you want the corresponding Python code for the above just let me know. I also have it but I guess it would be more useful for you (from the point of view of learning) to implement it yourself. In any case, if you want it, I can update my answer with the Python code.

Source code:

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def output_formula(features, weights, bias):
    return sigmoid(np.dot(features, weights) + bias)

def error_formula(y, output):
    return - y*np.log(output) - (1 - y) * np.log(1-output)

def update_weights(x, y, weights, bias, learnrate):
    output = output_formula(x, weights, bias)
    d_error = y - output
    weights += learnrate * d_error * x
    bias += learnrate * d_error
    return weights, bias


My NN has less than 1% accuracy rate, what's wrong with my model/dataset? Data info

X_train.shape

(20160, 44)

y_train.shape

(20160,)

#first row of the dataset
X_train.values[0]

array([93., 87., 85., 79., 78., 74., 64., 60., 60., 56., 51., 41., 37.,
       36.,  0., 36.,  0., 31.,  0.,  0., 32.,  0.,  0., 31.,  0., 23.,
       22., 17.,  0.,  0., 14.,  0., 12.,  8.,  5.,  1.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.])

y_train.values[0]

106.0

Neural Network

n_cols = len(X_train.columns)
es = EarlyStopping(patience = 5)
adam = Adam(lr = 0.0001)
model = Sequential()
model.add(Dense(n_cols, activation = 'relu', input_shape = (n_cols,)))
model.add(Dense(50, activation = 'relu'))
model.add(Dense(50, activation = 'relu'))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(optimizer = adam, loss = 'mean_squared_error', metrics = ['accuracy'])

history = model.fit(X_train, y_train, batch_size = X_train.shape[0], epochs = 100, validation_data = (X_test, y_test), callbacks = [es], verbose = 0)

The code above gives me a maximum accuracy of around 0.0025, I've also tried normalising X_train but same results.

A linear regression with the exact same data returns an R2 of 0.97

You note that y_train.values[0] returns 106.0 - is that correct? Sigmoid activation binds the output to between 0 and 1, so it's not possible for it to predict a 106.

Try changing the activation for your final layer to relu too.

You should also switch to a more regression-appropriate metric like mean_squared_error or R2. Accuracy is more of a classification thing.


Small dataset in Time series I have soccer data with a time series index. 30 seconds interval. So, 194 rows for 90+ minutes per game. I have 1500 games.
The dataframe has the following information.

Home/Away:

•   Goal Total.

•   Shots Total.

•   Dangerous Attack Total and.

•   XG(Expected goal per game) given at the start of the match which needs to be adjusted per section.

I am required to build a time series model to predict time t+1 (say Goal score).

Given that for each variable the graph shows the highest spike in the later stage of the time, Is there any machine learning algorithms that can learn with such a small dataset?

Sample Data

g = pd.DataFrame({'TimeSlot':[0, 30000, 60000, 90000, 120000, 150000, 180000, 210000, 240000, 270000, 300000],
    'xG_A':[1.5]*11, 'xG_B':[1.2]*11, 'A_DAT':[0,1,2,3,4,4,5,5,6,6,7],
    'B_DAT':[0,0,1,1,2,2,2,3,3,3,4],
    'A_ST':[0,0,1,1,2,2,2,3,3,3,3], 'B_ST':[0,0,0,1,1,1,1,2,2,2,2],
    'A_GT':[0,0,0,0,0,0,1,1,1,1,2], 'B_GT':[0,0,0,1,1,1,1,1,1,1,1]})
g.set_index('TimeSlot', inplace=True)

The assumption is only one Goal/Shot can be scored/taken per Timeslot. So the same data per period will look like...

f= pd.DataFrame({'TimeSlot':[0, 30000, 60000, 90000, 120000, 150000, 180000, 210000, 240000, 270000, 300000],
    'xG_A':[1.5]*11, 'xG_B':[1.2]*11, 'A_DATP':[0,1,1,1,1,0,1,0,1,0,1],
    'B_DAP':[0,0,1,0,1,0,0,1,0,0,1],
    'A_STP':[0,0,1,0,1,0,0,1,0,0,0], 'B_STP':[0,0,0,1,0,0,0,1,0,0,0],
    'A_GTP':[0,0,0,0,0,0,1,0,0,0,1], 'B_GTP':[0,0,0,1,0,0,0,0,0,0,0]})
f.set_index('TimeSlot', inplace=True)

***ARIMA(2,1,0) with train, test = X[0:size], X[size:len(X)]
gives MSE:0.323 for g and MSE: 0.291 for f.***

Questions

Which dataframe should I use?

I would really appreciate if someone starts off with a few python codes...

This is a typical application of the ARIMA model. By choosing the proper order (p,q and d in the wikipedia page), this model can be trained effectively with relatively small amount of data.


Combining Linear Regression and Time Series Does anyone know of a predictive model that can combine the linear regression model and time series model?

I have some data about some products. The data has two parts, some attributes about the products themselves, and some time series data of month-by-month sales amounts for each product. We want to predict future month-by-month sales amount for each product. with the products attributes data, we can build a regression model, or an svm model, to predict sales amounts. With the time series sales amount, we can build a time series model to predict sales amount, but is there any predictive model that can combine the regression, svm, and time series models to do the prediction?

You can use time as an additional attribute of the data. If you are having monthly data then your time feature could have 12 different values from 0 to 11. If you are having quarterly data your time attribute could have 4 different value from 0 to 3. You can refer Forecasting: Principles and Practice for more clarification. It's a good book.


Inference order in BERT masking task In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order?
For example:

The dog walked in the park.

Can be masked as

The [mask] walked in [mask] park.

In what order (if any) are these tokens predicted?
If you have any further reading on the topic, I'd appreciate it as well.

All the tokens are inferred at once, independently from one another.


Can bidirectional RNN use variable sequence length? A bidirectional RNN consists of two RNNs, one for the forward and another for the backward sequential directions, which outcome is concatenated at each time step. Would this configuration restrict the model to always use a fixed sequence length? Or would it still work as the unidirectional RNN, which can be applied to any sequence length?

This question was raised because the bidirectional architecture merges the output of both forward and backward RNNs at each time step. Thus, if the sequence length was 4, the outputs of both forward and backward RNNs would merge this way: 1th forward with 4th backward, 2nd forward with 3rd backward,... 4th forward with 1st backward. However, if a different sequence length was used this merge order would be modified:

Let's say the network was trained with sequence length 4, but at test time a sequence length of 5 was used. Merge would be: 1th forward with 5th backward, 2nd forward with 4th backward... 5th forward with 1th backward. Would this shift in merge order negatively affect the bidirectional RNN performance?

The short answer is no, a bidirectional architecture will still take in a variable sequence length. To understand why, you should understand how padding works.

For example, let's say you are implementing a bidirectional LSTM-RNN in tensorflow on variable length time series data for multiple subjects. The input is a 3D array with shape: [n_subjects, [n_features, [n_timesteps...] ...] ...] so to ensure that the array has consistent dimensions, you pad the other subject's features up to the length of the subject with features measured for the longest period of time.

Let's say subject 1 has one feature with values = [22,20,19,21,33,22,44,21,19,26,27] measured at times = [0,1,2,3,4,5,6,7,8,9,10]. subject 2 has one feature with values = [21,12,22,30,13,42,20] measured at times = [0,1,2,3,4,5,6]. You would pad features for Subject 2 by extending the array so that the padded_values = [21,12,22,30,13,42,20,0,0,0,0] at times = [0,1,2,3,4,5,6,7,8,9,10], then do the same thing for every subsequent subject.

This means the number of timesteps for each subject can be variable, and the merge you refer to occurs with the dimension for that particular subject.

Below is an example of a bidirectional LSTM-RNN architecture for a model that predicts sleep stages for different subjects using biometric features measured over variable lengths of time.


Can map-reduce algorithms written for MongoDB be ported to Hadoop later? In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select from for implementing the required analyses:

One approach is to extract the data from MongoDB to a Hadoop cluster and do the analysis completely in Hadoop platform. However, this requires considerable investment on preparing the platform (software and hardware) and educating the team to work with Hadoop and write map-reduce tasks for it.
Another approach is to just put our effort on designing the map-reduce algorithms, and run the algorithms on MongoDB map-reduce functionalities. This way, we can create an initial prototype of final system that can generate the reports. I know that the MongoDB's map-reduce functionalities are much slower compared to Hadoop, but currently the data is not that big that makes this a bottleneck yet, at least not for the next six months.

The question is, using the second approach and writing the algorithms for MongoDB, can them be later ported to Hadoop with little needed modification and algorithm redesign? MongoDB just supports JavaScript but programming language differences are easy to handle. However, is there any fundamental differences in the map-reduce model of MongoDB and Hadoop that may force us to redesign algorithms substantially for porting to Hadoop?

You can use map reduce algorithms in Hadoop without programming them in Java. It is called streaming and works like Linux piping. If you believe that you can port your functions to read and write to terminal, it should work nicely. Here is example blog post which shows how to use map reduce functions written in Python in Hadoop.


How can I predict rank of a team based on list of team members and past placement? I just started with ML, so this could potentially be a pretty stupid question, plz forgive me.
Here's the gist of the problem:
I have a list in json format like this:
[
  {
    // Final rank of the team after all the matches are done
    "rank": 7,
    // List of team members
    "characters": [
      { "char_id": "my_char_1", "level": 2, "item_ids": [1, 2, 3] }
      ...
    ]
  }
]

I want to predict the rank for a given "team" of characters without looking at the competitors.
Basically, I want to go from this:
[
  { "char_id": "my_char_1", "level": 2, "item_ids": [1, 2, 3]  }
  ...
]

to a single output neuron with a value from 1 to 100.
I parsed the source data into the first json structure, but I'm currently stuck at trying to give Tensorflow pairs of data (team + past rank), ideally in the form of two arrays where elements at a given index belong together.
TLDR:
Assuming this is possible at all (and not a lack of information or something), how do I feed Tensorflow this example data (list of team members -> past rank) properly to predict the future rank of a completely new team?

I've found this question which is somewhat similar:
Predicting outcome of MOBA team games
Based on this answer, I'm assuming I'll have to normalize the data, so instead of an array I would need a map like this:
{
  "my_char_1": null,
  "my_char_2": {
    "level": 2,
    "item_ids": {
      "1": true,
      "2": false,
      "3": true
      ...
    }
  },
  ...
}

However, I struggle with the act of passing the data itself, not so much how to prepare the json structure. Basically, take whatever json structure would be optimal and assume it already exists.

This sounds like an interesting problem. Based on what you're trying to achieve, I think you should consider a more classical approach first, rather than diving straight into a neural network. I would consider something like an ordered logistic regression model. This is an extension of logistic regression, handling two or more ordered outcomes.
Since you have your data in a json file, I'd first convert this to a csv. Then load the csv into Python to estimate/train any models. You can then use the data to estimate an ordered logistic regression model. See the Python documentation here.
Your response will be rank (1-10). You can use the characters and items used as predictors. You could also look at interaction terms between the characters and predictors - this is because certain item and character combinations impact the outcome differently. One concern is that this model might become quickly over parameterised. Next you could consider a neural network.
In the neural network, your input nodes will be the characters and items. You'll need to perform a one hot encoding on the rank before you can use it to train the network. Note that you should use a cross entropy loss function, with a softmax activation function on the output layer. It's important to note that this doesn't achieve exactly the same thing as an ordered logistic regression (we're ignoring the fact the outcome is ordered). See this question about setting up a neural network mimic an ordered logistic regression.


How R performs the operations like Inf + Inf, Inf * Inf, Inf - Inf and Inf / Inf? In R When we do some operation like Inf + Inf, Inf * Inf gives Inf back, whereas similar operation  Inf - Inf and Inf / Infgives results as NaN, how R actually handles these type of concepts

Inf is a reserved word in R, so it is handled at interpreter level. That is, similar to FALSE & TRUE $\rightarrow$ FALSE and FALSE | TRUE $\rightarrow$ TRUE, when interpreter is faced with Inf ? Inf, it decides what to do based on the operation ?. You can think of a set of if-then rules under the hood that resolve the expression to Inf for + and *, and to NaN for - and /, no complicated inference is involved.


Sentence Segmentation for Hindi I only found stanza that has sentence segmentation for hindi. All other available tools do word tokenization but not sentence segmentation. Could someone suggest any other available tools for sentence segmentation for Hindi

I'm surprised that there is no other tool, especially since usually when there are word tokenizers available there are also sentence tokenizers (the two tasks are often performed at the same time).
In case you need to train your own tokenizer, you could use the Universal Dependencies Hindi corpus.


Categorical and non-categorical data in the same column I have a unique dataset that has many columns and most columns contain both categorical and non-categorical data.  For example, let's say that one column is attribute_1 and for observations that have data for attribute_1 the value can be between 100 and 1000.  If an observation does not have data for attribute_1 then they are given a value between -4 and -1, where the value describes why they don't have data for this attribute.

How can I encode the categorical part of the columns while also applying feature scaling to the non-categorical part of the column?  Would it make sense to split the column into two where one is the categorical and another column just for the non-categorical?

If you provide more information about the detail of why there is no data for some data points that would make it easier.

That being said, I would split it into three columns as follows:

col_1: it includes 0 and 1 >>> 1 for those samples that have a value of -4 in the original column. For the rest of the samples: 0

col_2: it includes 0 and 1 >>> 1 for those samples that have a value of -1 in the original column. For the rest of the samples: 0

col_3: the real values of the original columns. If the value is -4 or -1, there are two options as follows:
1. Use 0 for those samples
2. If they are missing values replace them using one of the methodologies used for replacing the missing values.


Technique to make NN (specifically autoencoders) more robust by artifcally making input data more sparse I believe some month ago I have read somewhere that autoencoders can respond better to sparse input data when trained with such. For example when the training data is not sparse, but many features of the test data are zero. Apparently randomly setting some features of the training data to zero can help to perform the model better on the test set. I desperatly tried to find the book/paper where I read it again, but I was not able to and I cannot remember the name for this. Does somebody know what this technique is called?

I've finally found it. The term I was looking for was 'masking corruption' and is mentioned in 'Practical Recommendations for Gradient-Based Training of Deep Architectures' by Yoshua Bengio.


What is the difference between public test and private test on Kaggle I am trying to use the datasets from a competition held on Kaggle in which the dataset contains fer2013.csv file having columns emotion, pixels, and usage where usage contains 3 values i.e "training", "privatetest" and "publictest". So I am not getting what is publictest and privatetest. It would be appreciated someone could explain to me what it is?

"publictest" data is used to calculate the score for the public leaderboard while competition is running.

After the competition is finished, competitors will be ranked on the data which was marked as "privatedataset".


How do i iterate functions until convergence in R? I am looking to iterate until convergence but I am not sure how i should code it.
A simple similar example is something like
$X_0=5 ,  Y_0=3$
for $n=1,2..$
$P_n=X_{n-1}-Y_{n-1}$
$Y_n=3P_n$
$X_n=P_n+Y_n$
and end when it converges.
I have seen that for the repeat loop it loops indefinitely with an exit condition but I am unsure how i can word the iterations  within R and how to word the exit condition for convergence. Any references or help would be appreciated.

Generally you want to check that the change in your value of interest becomes very small, eg $abs(X_n - X_{n-1}) < eps$ where eps is a small number like 0.0001.


How to load numerous files from google drive into colab I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch.

Therefore I have first mounted my Google drive using:

from google.colab import drive
drive.mount('/content/gdrive')

Next I have unzipped the image file using:

!unzip -uq "/content/gdrive/My Drive/path.zip" -d "/content/gdrive/My Drive/path/"

Counting how many files are located in the directory using:

len(os.listdir(path-to-train-images))

I only find 13k images (whereas I should find 30k). According to the output of unzip, the files appear to be unzipped correctly.

Also, I found there are some issues with loading in many files from a google directory: https://github.com/googlecolab/colabtools/issues/510.

Does anyone know where I am going wrong? Or whether there is a workaround?

One possible option would be operate directly on the zip files using zipfile.ZipFile.
Counting the number of items in a zip file:
from contextlib import closing
from zipfile import ZipFile

with closing("/content/gdrive/My Drive/path.zip") as zip_file:
    count = len(zip_file.infolist())


Clustering with sets as values I have gathered a large amount of qualitative data and am now looking to cluster it so as to make sense of it. For this, I am using Biolab's Orange.

In my data, specific values may co-occur in a given feature, or they may not. I am wondering how I could cluster the data (either in Orange or other software), where values that co-occur would be seen as two values, rather than one string.

To make matters clearer, imagine I have a feature X, with the possible values A and B. The values can occur in the following way: A, B, A and B.

Question: how can I cluster my data without "A and B" being treated as a separate string, but rather "A" and "B" co-occurring?

It would appear that you have a data processing task. You could use two columns 'A Occurrence' and 'B Occurrence'. If that value occurs for the record that index would contain a 1. If that value does not occur for that record the index would contain a 0.

Example:

X       | A Occurrence | B Occurrence
A       | 1            | 0
A and B | 1            | 1


How to compare LDA and TF-IDF? I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked better for me. I'd like to evaluate both but I can'f find any useful validation or metric to compare these two. How is it possible to compare these two with a useful metric?

If you have the ground truth value of the documents (their topics) all you gotta do is pick a metric and compare results. For classification problems, as yours, a common metric would be f1_score; reference: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html


What algorithms can be used to derive matching rules between known matches in datasets? Lets say I have two datasets with different column names except for a unique ID key

Table 1 CSV

first_name,middle_name,last_name,uno,id
John,D,Smith,1,1
John,C,Smith,1,2
John,B,Doe,1,3
Suzy,C,Q,1,4

Table 2 CSV

fname,mname,lname,one,id
John,D,Smith,1,1
John,C,Smith,1,2
John,B,Doe,1,3
Suzy,C,Q,1,4

John D Smith is user ID #1 and is in both tables.

Is there a pre-built algorithm, package or tool that can do the following.

Join across tables where id is the same
For known matches, try to identify what rules could have been used to match the two records together.
Test hypothesis, like "fname and first_name are the same, is that enough to produce the target 'id' variable? Let me check other data. No. What about fname + lname?
Test if assertions hold true against other known matches.

End output would be

table 1 (first_name, middle_name, lastname) are the best join against

table 2 (fname, mname, lname)

What you're trying to do is called "entity resolution" or "record linkage" (you can do a more thorough search in google). A typical approach is to treat it as classification, using as rows each combination of entries from one table matched against the other, label being whether they are a match or not; and any features that you can think about such as edit distance between the two, number of characters in common, whether they start and end with the same letter, largest common subsequence, number of characters in each, etc.

(a more computationally efficient approach is to treat it as one-class classification).


Bad classification performance of logistic regression on imbalanced data in testing as compared to training I am trying to fit a logistic regression model to an imbalanced dataset (0.5/99.5) with high dimensionality(about 15k). I used random forest to select top 200 important features. Observations are around 120K.
When I fit a logistic regression model on based dataset (using Smote for over sampling) , on training f1, recall and precision are good. But on testing, precision score and f1 are bad. I assume it makes sense because in training there were a lot more of the minority case while in reality/testing there is only very small percentage. So the algorithm is still looking for more minority cases, which caused the high false positive.
I was wondering what kind of methods could I try to improve the performance?
I am currently trying different sampling method for imbalanced dataset, also plan to try PCA.

I suspect the reason is that the class balance in your test set is different from the class balance in your training set.  That will throw everything off.  The fundamental assumption made by statistical machine learning methods (including logistic regression) is that the distribution of data in the test set matches the distribution of data in the training set.  SMOTE can throw that off.

It sounds like you have used SMOTE to augment the training set by adding additional synthetic positive instances (i.e., oversampling the minority class) -- but you haven't added any negative instances.  So, the class balance in the training set might have shifted from 0.5%/99.5% to something like (say) 10%/90%, while the class balance in the test set remains 0.5%/99.5%.  That's bad; it will cause the classifier to over-predict positive instances.  For some classifiers, it's not a major problem, but I expect that logistic regression might be more sensitive to this mismatch between training distribution and test distribution.

Here are two candidate solutions for the problem that you can try:

Stop using SMOTE.  Ensure the training set has the same distribution as the test set.  SMOTE might actually be unnecessary in your situation.
Continue to augment the training set using SMOTE as you're currently doing, and compensate for the train/test mismatch by shifting the threshold for classification.  Logistic regression produces an estimated probability that a particular instance is from the positive class.  Typically, you then compare that probability to the threshold 0.5 and use that to classify it as positive or negative.  You can adjust the threshold to correct for that: replace $0.5$ with $0.5/k$, where $k$ is the ratio of positives in your training set after augmentation to positive before (e.g., if augmentation shifted the training set from 0.5%/99.5% to 10%/90%, then $k=10/0.5=20$); or you can use cross-validation to find a suitable threshold that maximizes the F1 score (or some other metric).

Incidentally, I recommend you make sure to use regularization with your logistic regression model, and use cross-validation to select the regularization hyper-parameter.  There's nothing wrong with 15K features if you have 120K instances in your training set, but you might want to regularize it strongly (choose a large regularization parameter) to avoid overfitting.

Finally, understand that dealing with severe class imbalance such as you have is just hard.  Fortunately, there are many techniques available.  Do some reading and research (including on Stats.SE) and you should be able to find other methods you could try, if these don't work well enough.


Distance between users I want to compute the "distance" between users in order to return the top n similar users, for any given user.
For each user a have a bunch of features.

This is close to a recommendation system, however I don't have ratings and I can't assign each user to different products, hence I'll need to compare "how similar" Users are.

Do you have any suggestion for the algorithms/approaches for this kind of problem?
I find a lot of resources based on recommendation system (having rating and items), but that's different from what I want to achieve

No need for algorithms, or recommendation systems.  You have:

  For each user a have a bunch of features.

As long as they're numeric, or can be made numeric (e.g. aggregating the values or one-hot-encoding them), you already have distances.  What you may not have is the proper variance across the feature space, i.e. features are scaled in different orders of magnitude.

If you know the exact weight of the features in relation to user similarity you may try to tune (scale) the features by hand. Otherwise you can simply make every feature have mean 0 and standard deviation 1.  In other words, per feature subtract the mean from all points and divide by the current standard deviation. (sklearn has a StandardScaler that does exactly that.)

In the scaled dataset, from any point (user), you can just calculate euclidean distance to any other point.  And the closest the points the more similar the pair of user will be.  i.e. top $N$ similar users to a user are just the $N$ closest points.

Plain euclidean distance works in many cases.  If euclidean distance does not work for the problem at hand, then you can explore more complex possibilities: starting from manhattan distance, through minkowski distance (combination of euclidean and manhattan distances).


Running Jupyter notebook on remote server with local files I would like to train my model faster and that's why would like to create a standalone remote machine with GPU support. The solution I am looking for is to have the data locally on my machine and to use the resources from the remote machine. Is this possible?
I am checking my project for example locally on my laptop. When I use some remote Jupiter notebook, then it needs the data also to be on the remote machine. But after the model is trained I would like to commit it in git from my local machine. How to make this?

What sort of virtual machine are you using?
If you are using Google colab for example you can connect it to your drive and add data from your drive.


Time series forecast for everyday for till a distant future I have time series data for every single day from last 5 years with seasonal variation and a general increase in trend. This is what my data looks like:

And I am trying to predict for every single day for 4-5 years in future. Approaches I have used currently:

LSTM, GRU (but they are extremely prone to overfitting ad I am trying successive predictions which results in massive error accumulation and often flat over time for smaller lookbacks and sine wave driven patterns for long lookbacks.
SARIMAX and other seasonal forecast models but they tend to scale very bad for a large period of predictions. Mostly they eventually result in a curve that follows trend(increase/decrease) but values are almost linear or unrealistically close to each other.

What I am currently thinking of:
Separate same day of each year and fit a model to those values and then have successive predictions for the same day for subsequent years. Thus having 365 different models and then concatenating the resultant values to have the whole year and future years predictions. This way I'll perfectly preserve the seasonality and yet accumulation errors will be minimized. So my question is:
Is this a good idea (and if it is, is there a method to do it feasibly without assessing each of the 365 models) or I should try any other approach?

Have you tried the good old Fourier Transform?
If you can take the load of math, you can try the Wavelet Transform, but first I would recommend a simple Fourier Transform.

A Guide


How to create a language translator from scratch? I want to create a translator which can translate English, Korean and Tamil sentences into English sentence, I tried googletrans but is there any way to create something better than that using DL and NLP techniques?

Google translate itself uses Deep learning to translate sentences which can be seen here.

You can translate sentences across languages for which you need two things :

A large dataset which has pairs of translations ( like English-French ). You can find such a dataset from here.
A sequence-to-sequence RNN model. They have Encoder-Decoder architecture which encodes the source sentence into a thought vector and then decode it to form the translation. This image may be helpful.

The picture shows the Neural Machine Translation adopted by Google.

Also, TensorFlow and Keras help in creating such deep learning models. You can refer to this blog from the author of Keras. These models ( seq2seq ) have gained popularity in text summarization, NMT, and other sequence-to-sequence tasks.

If you wish to create a full-scale production ready language translator then you will need a GPU as these models are heavy to train as well as a very large corpus of text.


How to fill missing numeric if any value in a subset is missing, all other columns with the same subset are missing There is a clear pattern that show for two separate subsets (set of columns); If one value is missing in a column, values of other columns in the same subset are missing for any row.

Here is a visualization of missing data

My tries up until now, I used ycimpute library to learn from other values, and applied Iterforest.

I noted, score of Logistic regression is so weak (0.6) and thought Iterforest might not learn enough or anyway, except from outer subset which might not be enough? for example the subset with 11 columns might learn from the other columns but not from within it's members, and the same goes for the subset with four columns.

This bar plot show better quantity of missings

So of course, dealing with missings is better than dropping rows because It would affect my prediction which does contain the same missings quantity relatively.

Any better way to deal with these ?

[EDIT]
The nullity pattern is confirmed:

I usually use mice for missing data imputation. It relies on chained equations and performs very well. It also has a random forest method, but typing methods(mice) you will find a list of available options,  that can overcome the issues encountered by standard approaches such as logistic regression techniques.
Another option is using the na.impute option available in caret package.


Topic models for Relevance Prediction Suppose I have data in the form of  Query/Document Pairs, along with corresponding relevance scores (or class labels). Is there a way to use topic modeling to devise a model so that later given a query and a document, we can predict its relevance score?

Certainly. A part of topic modeling output is typically some vector for each document describing it as a distribution over the topics in your model. You can use these as continuous-valued predictors for a classifier trained with the labels you have. I've done this before, but, depending on the domain characteristics of your problem, it may not be the best approach, but it is certainly a testable one!


SKLEARN Metrics report "Number of classes, 28, does not match size of target_names, 35. Try specifying the labels parameter" What's the proper way to define the labels or target names of classification_report?
I have the report like this:
print(metrics.classification_report(twenty_test.target[:n_samples], predicted, target_names=twenty_test.target_names))

I noticed i get the error like in the title or similar once the twenty_test.target[:n_samples] (or possibly also predicted array) is different length than twenty_test.target_names...
It doesn't work just out of box if i simply provide all target names (also those not included in the "target" or "predicted" array. (If i provide it as "target_names" param).
But if I provided it as the "labels" param as hinted i simply see 0 filled report.
Btw this is the output of unique values of "target" array:
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 25}

This is the output of predicted array (unique values):
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 25, 27, 30, 31, 32}

I tried to get the longer list of these and build the list of the longest value like this:
longer = max(max(set(predicted)), max(set(twenty_test.target[:n_samples])))+1
print(metrics.classification_report(twenty_test.target[:n_samples], predicted, target_names=twenty_test.target_names[:longer])

Doesn't work. I would expect all target names also those not include should work out of box.

I tried to fix it like this, not sure if it is correct.
from sklearn.utils.multiclass import unique_labels
target_names_filtered = [twenty_test.target_names[i] for i in unique_labels(twenty_test.target[:n_samples], predicted)]

then i used target_names_filtered as the target_names param.


Why Gaussian mixture model uses Expectation maximization instead of Gradient descent? Why Gaussian mixture model uses Expectation maximization instead of Gradient descent?
What other models uses Expectation maximization to find best optimal parameters instead of using gradient descent?

Not all the parameters (e.g., the assignment parameters) for a Gaussian mixture model are smoothly differentiable, thus can not be fit with gradient descent.
Other use cases for the expectation–maximization (EM) algorithm are:

Clustering
Latent variable estimation
Missing data estimation


What expresses if/how two variables are dependent on each other? The accepted answer to question Why is a correlation matrix symmetric? includes this:

  The correlation matrix is a measure of linearity. It does not express how two variables are dependent on each other.

My question is : What is there that is related to if/how two variables are dependent on each other?

Edit 1 : When asking this question, not considering anything related to correlation, only want to know if there are ways to see if two or more variables have dependency on each other or not, irrespective high or no correlation between them.

I used the correlation tag because I could not think of what tag to use, please feel free to edit/correct the tag.

I think what the clarification has in mind is that correlation does not imply causation, that is, that two variables are mathematically correlated, does not mean they are dependent on each other.

Here a wikipedia article if you'd like to expand on reading.

Here is a page full of examples where two variables, although highly correlated, are very clearly independent one from the other. For example: Per capita consumption of chicken vs Total US crude oil imports, or Per capita consumption of mozzarella cheese vs Civil eng. doctorates awarded.

Having shared my understanding of the clarification, now going back to your question:

  What is there that is related to if/how two variables are dependent on each other?

I think what does show if two variables are dependent on each other is not correlation alone, but a further analysis based on common sense.

Hope this helps.


Kmeans on mixed dataset with high level for categ My retail dataset contains 3 numeric attributes and two categorical attributes  Time and ID with 50,000 records. Both categorical attributes have more than 20 thousand levels and their format is 1/11/2011 11:54 and 1TD10051 respectively.

How do I do kmeans on these dataset? Converting categorical to binary will give very sparse dataset?

How to proceed?

There are plenty of methods, variations of k-means for the case of mixed dataset: k-modes, k-protoypes etc.

It has been discussed already.


RNN: why Wx + Uh instead of W[x,h] Traditionally, a state for RNN is computed as
$$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$

For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single matrix times a concatenated vector:

$$W_m[x, h_{t-1}]$$

where $[...]$ is concatenation.

In other words, we would end up with a long vector like $\{x_1, x_2, x_3, h_{1,t-1}, h_{2,t-1}, h_{3,t-1} \}$ multiplied by $W_m$.

It seems like the second approach would have a significantly larger matrix, which has more elements than $W$ and $U$ combined.

Does that mean $W$ and $U$ are a simplification, what do we lose by using them, and adding up the results?

Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often.
In practice, both approaches are actually used in production and hence are equivalent. It's just a matter of preference.

Tensorflow

For example, Tensorflow is often optimized for performance. Here's how basic RNN cell is implemented there (tensorflow/python/ops/rnn_cell_impl.py):

def call(self, inputs, state):
    """Most basic RNN: output = new_state = act(W * input + U * state + B)."""

    gate_inputs = math_ops.matmul(
        array_ops.concat([inputs, state], 1), self._kernel)
    gate_inputs = nn_ops.bias_add(gate_inputs, self._bias)
    output = self._activation(gate_inputs)
    return output, output

A single matrix multiplication is more efficient, so it's applied, even though the comment describes the expanded formula with two matrices.

Keras

On the other hand, Keras often chooses simplicity and clarity over performance. Here's its implementation (keras/layers/recurrent.py):

def call(self, inputs, states, training=None):
    prev_output = states[0]

    ...

    if dp_mask is not None:
        h = K.dot(inputs * dp_mask, self.kernel)
    else:
        h = K.dot(inputs, self.kernel)
    if self.bias is not None:
        h = K.bias_add(h, self.bias)

    ...
    output = h + K.dot(prev_output, self.recurrent_kernel)
    if self.activation is not None:
        output = self.activation(output)

The class thus makes its it easy to access two matrices separately (self.kernel and self.recurrent_kernel).

Pytorch

Pytorch approach is closer to keras. Moreover, not only they use separate kernel matrices, they also have two bias vectors, and these four arrays are accessible for the client code.

Despite these differences all three libraries are functionally equivalent.


Python for data analytics What are some data analytic package & feature in python which helps do data analytic?

You're looking for this answer: https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists


Universal function approximation with fixed values (as vector or matrix) I was thinking about way to represent/approximate universal function and came up with the idea that a plain fixed numbers could be used to represent pretty much any function on a fixed interval.

I wonder how such approach called. And if it's used in logistic regression, genetic algorithms etc.?

The good thing is that it's universal, you can extend it to multiple dimensions and you don't have to make any assumption about function if it should be polynomial or trigonometric etc.

I would like to know more about it. But can't find much.

By "plain fixed numbers" do you mean a piecewise-constant function?  Sure, those are universal approximators (with some natural hypotheses*).  Indeed, decision trees generate a special kind of piecewise-constant function, and they are universal approximators*.

*I've actually been unable to find references for these, and am not sure how to work out reasonable hypotheses.  Using the examples $\sin(x)$ and $\sin(1/x)$ for $x>0$ indicates that we need a compact domain, and surely some sort of continuity is required.  That's probably enough, but my analysis classes are too long forgotten to know how to produce a proof; you might need some fancier stuff like Stone-Weierstrass.  If you take Lipschitz continuity as a hypothesis, it seems like there'd be an easy(/ier) direct argument.


Why normalize function has a different result on a matrix vs single value? I have a matrix like:

B=[ 1.5035; 1.5728; 1.6485;  1.5369;  1.5467;  1.572; 1.5374;  1.787;  1.5825; 1.6905];

Using normalize function like  normalize(B,'range') has this result:

ans =  0
  0.24444
  0.51146
  0.11781
  0.15238
  0.24162
  0.11958
        1
  0.27866
  0.65961

But when I use it for a single value like:

normalize(B(2,:),'range')

the result is 0 , but the result for row number 2 in ans is  0.24444.
Why its different and how can I fix it?

The normalize function works by calculating z-score for the given data. Now z-score is given by:
$$z_{i} = \frac{x_{i} - mean(X)}{s.d.(X)}$$
where $X$ is your original vector/matrix.

Now, when you give a vector as input it first calculates the mean and standard deviation for the row/column. Then the output you ask for is calculated the above formula. So, if you only ask for a scalar it only shows one value. You can see in your example the normalized value(or z-score) is same for that particular position i.e. $0.24444$.

But when you give only a single value as input the z-score is 0 by definition. For more on it's working and use take a look at the docs.


Input shape in a multivariate RNN So I've seen this:
Keras LSTM with 1D time series
And this: Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have many, many, many accountIDs, and 40 or more features associated with them for the start of each week since 2017.
I'm attempting to predict expected profit next week (next time step) for a given accountID with a given history (the idea is to modify strategy features, pass them both to the RNN, and see which one has the higher expected profit once the model is trained).

The suggestion provided in the second one I linked is that I make a sub-model for each accountID it seems, but I have many of them and new ones come in all the time.

Should I strip out accountID, but still pass its whole history in as a sequence, and then just append it back after prediction?

What if I only have a few weeks of data for some accounts, and years for others?

The dataset would look roughly like this: (weekstart, accountID, x, y, z, p, q, ...), and each accountID would have many tuples like that, but they wouldn't all have the same number of tuples.

For reference, I'm familiar with CNNs and standard neural networks, but this is my first attempt at using an RNN.

Ok... Take a step back. When you're dealing with RNNs your dataset should have a shape that looks like (nb_samples, nb_timesteps, nb_features)
Translating this to your use case means that each account is a sample (what you'll iterate when doing mini-batching), each week is a timestep (what your rnn will iterate over) and your features are.... Features.

Now... Don't use accountID as a feature. That's just an ID with no prediction power. Also try to add some seasonality features, like month or something.
Does that help you out?


How to operate dataset from paperswithcode.com? I'm trying to use IG-3.5B-17k dataset from paperswithcode.com, but I can't figure out how to do that exactly.
How to do it? I guess I need to use this site API? Am I even able to use this dataset?

This is specified in the dataset description in paperswithcode.com (emphasis mine):

IG-3.5B-17k is an internal Facebook AI Research dataset for training image classification models. It consists of hashtags for up to 3.5 billion public Instagram images.

So the dataset is not public.
In the paper, however, the authors argue that the images and their hashtags are "visible" in Instagram:

Our datasets have two nice properties: public visibility and simplicity. By using publicly accessible images, the data used in our experiments is
visible to everyone. To see what it looks like, the images are browsable by hashtag at https://www.instagram.com/explore/tags/ followed by a specific hashtag;
for example https://www.instagram.com/explore/tags/brownbear shows images tagged with #brownbear. Our data is also taken from the “wild”, essentially as-is, with minimal effort to sanitize it. This makes the dataset construction process particularly simple and transparent.

I understand that they are saying "you can go and query Instagram yourself" to see the images in the dataset, but I don't think this is actually practical or even allowed in their terms of service.


impact of old reviews on new reviews I am working on e-commerce, how to rank smartphones in the same category. I want to calculate a weighted average of sentiment score from reviews posted by buyers.

Weights will be based on how many days before review was posted, as people might give more importance to recent reviews than old reviews. Now to find weight for previous day review, i want to understand impact of previous reviews on current review. i.e. impact of t-1, t-2, t-3 on t.

I want to know for how many days should i aggregate review sentiment?

Welcome to DataScience.SE. Let's see if I've interpreted your question correctly:

  i want to understand impact of previous reviews on current review

Keep track of which reviews a reviewer has read (randomly withhold some if you have to), and use this information together the rating of the new review to create a regressor to estimate the score of a new review given the age and score of selected previous reviews.

You can do similar regressions with the topic and sentiment, if you use embeddings.


Date and time format in R I have date and time in one column of the form
 2004/12/15 9:30 , how can I format it  to date, so that I can plot it on x axis. Also posxlt is not working for me.

as.POSIXct formats dates so as to include the time, while as.Date formats so as to include just the year, month, and date, i.e. yyyy-mm-dd.

Assuming you have a column x with the dates and times included as you mentioned, i.e. 2004/12/15 9:30, then you can simply apply:

date<-as.Date(x)

Now, you can check if it is in date format through using:

str(Date)

You should find that the times are no longer visible and only the dates are displayed.


Can you interpret probabilistically the output of a Support Vector Machine? I am trying to build a binary classification system using different classification algorithms like random forests, support vector machines, AdaBoost. I want to use the output of these classifiers to visualize a score. For example, when using random forests, I would like to use the probability of a sample belonging to class A to build a score from 0 to 100. Given that random forests output a probability (from 0 to 1) using it as the score is intuitive (I would just multiply it by 100). However, given that SVMs output a classification but not a probabilistic output (i.e. distance to the hyperplane, but not probability), would it be legitimate to use the distance to the hyperplane as some sort of "pseudo probability"?  I would, for example, do max-min scale on the distance to the hyperplane for all samples so all distances are scaled from 0 to 1.

I want to be sure that I can use the distance to the hyperplane as a pseudo-probability and that this pseudo-probability is comparable to the probability to belong to a given class outputted by the random forest. For example, that a sample with a probability of .80 of belonging to class A is the same than another sample of a (min-max transformed) probability of belonging to class A according to the SVM.

One standard way to obtain a "probability" out of a SVM is to use Platt scaling.  See, e.g., this Wikipedia page and this question on Stats.SE.  Platt scaling involves fitting a logistic regression model to predict the "probability", based on the distance to the hyperplane.


How to explain MAE/MSE at each node of decision tree for regression in sklearn python? If the mean value at any node is 60 and MSE = 169 so RMSE is 13. Can I conclude that the error at my node is 60 +-13 i.e my values in this particular sample split ranges from 60-13 to 60+13. If not , what is mae/mse at each split and how do I interpret this ?

Can I conclude that the error at my node is 60 +-13 i.e my values in this particular sample split ranges from 60-13 to 60+13.

No you cannot, because the actual error values depend on the data. For example you might have 1 instance with error 41.11 and 9 instances with error 0:
$$MSE=\frac{41.11^2+0^2+...+0^2}{10}=169$$
This example shows that the only guarantee you have for an individual absolute error is that it's lower or equal than $\sqrt{MSE \times n}$.
The MAE (mean absolute error) is easier to interpret: it's literally the mean of the absolute value of the error.


TextInformationRetrieval content based I need to know how to avoid spam document(file with repeated keywords) weighting while ranking the top k documents.

This type of spam is called keyword stuffing and it is a widely used SEO technique.
There might be different approaches to deal with this.
One is to use pre-trained classifier that will provide a "spam" score to each document. One such example is Waterloo spam classifier, covered in paper:

"Efficient and Effective Spam Filtering and Re-ranking for Large Web
Datasets"

, by: Gordon V. Cormack, Mark D. Smucker and Charles L. A. Clarke.
In addition, several predictors of document (specifically, Web document) quality were proposed. One of the most effective ones is the entropy of the unigram language model of a document. Specifically, when there is a large amount of repeated keywords in a document, the entropy of the language model is unusually low, since these keywords have unusually high probabilities. Another signal of document quality is the percent of stopwords in it, since natural language has more stopwords than a spam document.
More on Web documents quality measures can be found in: "Quality-Biased Ranking of Web Documents", by Michael Bendersky, W. Bruce Croft,Yanlei Diao.


How to find optimal sample weights in binary classification/regression problem I'm training a model (NN) that gets some data as input and outputs a single value in the range of $[0, 1]$. Right now, the average of the outputs in my dataset is around 0.5, but I know that future data will largely consist of 0.0s, and thus there will eventually be a strong data imbalance towards 0.0. I want the training procedure to be future-proof and scalable, so I'm trying to find a way to automatically rebalance the dataset. My library (Keras) supports sample weights in training, which seems like a straightforward way to do this without losing any information.

Basically, I think what I'm looking for is a function $w(y_i) \rightarrow w_i$ that given a training example $y_i$ gives me a weight $w_i$, so that the weighted average of all training examples $Y$ with the weights $W$ is $0.5$. I know there are many configurations of weights that have this property, but of course the weights should be as close to 1 as possible and definitely $> 0$. I also realize that this is not possible for cases where e.g. all numbers are the same, or all are $< 0.5$. But let's assume my data is diverse enough.

I'm sure I'm not the first one to think of this, but I can't find any solutions / best practices for this case. I guess I could treat it as a little optimization problem of its own, but I hope there is something simpler.

There are a few things which are unclear so I am going to have to make some assumptions.

You say it is binary classification/regression. Are you trying to find the probability of each binary class?

When you say "future data", do you mean future training data or future test data? If it is test data then you do not need to worry at all about his effect. If it is future training data then yes you may have a problem. You do not want to do model building on a training set that is different from the training set you would actually use in the productionized model. I would suggest you produce a number of validation plots for this case.

If you really are doing binary classification. Most models in python have a method (eg predict_proba() ) that will produce a probability and will let you decide on class based on that probability. The default being 50%. For this reason I do not think that you want to have the weighted average be 0.5 but the "probability" represent the true probability. Please refer to this tutorial http://scikit-learn.org/stable/modules/calibration.html

As to your original question. You want the weights to be $M/N_i$ where $N_i$ is the number of samples in each class $i$ and $M$ the total number of samples. This is a way to do some sort of pseudo-upsampleing and have the classes be weighted by the number of samples.


Extract relevant features from time series data I have a time series data set from a sensor and the task is to predict the time before a failure event is occurred. The data set has one feature and has almost 20 million rows. This is a regression problem.

I tried polynomial features, auto correlation, rolling statistics and expanding statistics. The only one that seemed to improve my model was expanding sum. What are some relevant features to be extracted from this data?

My model is a Linear Regression model, the data set was scaled and currently only two features improved my model. The sensor data and the expanding sum. Any other suggestions to tackle this problem other than using deep learning?

Update: For clarification I added the plots for both the input and output.

Sensor reading plot - Input

Time to failure plot - Output

I think the first question you should ask before you start going off on a deep learning model is, can you tell when the failure is going to occur just by looking at a plot of your data? If you can't, then no model will help you deduce when a failure will occur.

You shouldn't overlook some basic models also such as exponential or poisson distribution models that should model your current problem well.

Lastly, since it's a time series for rare events, some ways to gain more insight might include time between failures, identifying thresholds of the data value triggering a failure, splicing all the data into intervals and the number of failures, etc.. You should also check if the failures follow a pattern.


Correlation based Feature Selection vs Feature Engineering I'm a bit confused about the superiority of Feature Selection over Feature Engineering or vice versa.

Let's say I just want to get the best possible performance on a couple of models like a neural network, something tree-based and a Naive Bayes Classifier.

Before starting any training I looked at my features and engineered some additional (hopefully) even more expressive features. I did this from a domain expert point of view. For instance I added a new ratio feature C = A / B because I think this will be a very expressive information for the model.

Furthermore I added several features measuring basically the same thing but in different ways. Lets say a feature D measuring the the length of any text including empty lines and an additional feature E measures the length of any text excluding empty lines.

So this leads to very many features in my data set with also very high correlation / multi-collinearity. (of course D and E are very high correlated and A, B and C are also high correlated / multi-collinear.

So any kind of correlated based feature selection (between the features) would remove a lot of the engineered features, but could the removal provide the model any kind of better discriminative power with just less information? What helps the model more, keeping all features or removing correlated ones?

What helps the model more, keeping all features or removing correlated ones?

There is some theory about it but in the end Machine Learning is try and error. You should give it a try with all features and then doing a feature selection to see if you are able to improve your model. What works for some models doesn´t necessarily have to work for the rest of the models.

If you want to select which features help your model you could do otherwise, instead of having all and removing, starting with one and adding features and only keeping them in case that it boosts the performance of the model. There are cases when you add a feature and the performance of the model drops.

There are a lot of ways in which we can think of feature selection, but most feature selection methods can be divided into three major buckets. From this source

Filter based: We specify some metric and based on that filter features. An
example of such a metric could be correlation/chi-square.
Wrapper-based: Wrapper methods consider the selection of a set of features as a search problem. Example: Recursive Feature Elimination
Embedded: Embedded methods use algorithms that have built-in feature selection methods. For instance, Lasso and RF have their own feature selection methods.


What are some key difference in functionality between ProM and Apromore? I would like to know what are some use cases where one would use ProM and where one would use Apromore.
I have traces and events from logs that I would like to use process mining on (mapping a business process from these informations, filter to isolate types of cases, estimating average time for each event, estimating the percentage of cases that go through a branch, visually, etc.), but I have a hard time choosing a tool. Also, the two seem able to interact; therefore, I have trouble understanding the key features of both.
ProM seems to be a more desktop application, while Apromore web-based, distributed via a Docker image, but functionality-wise, what are some differences?

Below are three key differences.

Apromore is a server-side collaborative tool. It is typically installed on a server (on-premise or on the cloud). Users connect to Apromore via the Web browser. Users can upload process models and event logs, organize them into folders, enrich them with log filters and dashboard designs, and share them with other users. Apromore comes with a process model editor, so you can edit the process models you discover from your event logs, beautify them, and export them in BPMN format (so you can share them with other tools). Meanwhile, ProM is a desktop tool. You typically install it in your laptop. You use it to analyze an event log locally, but you cannot use it in a collaborative manner with say 3-4 team members. You cannot save your analysis (e.g. custom dashboards, filters) in ProM.
A second difference is that Apromore supports BPMN natively. The process discovery algorithm in Apromore is Split Miner, which discovers BPMN models by design. Traditionally, ProM is designed to support an academic notation called Petri nets, which is used in research studies but generally not used by process improvement practitioners.
Finally, Apromore supports both process mining use cases (discovery, conformance), but also process performance monitoring use cases via customizable process performance dashboards.


normalizing data and avoiding dividing by zero I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly detection algorithm and see what happens.

I would like to normalize the data for the autoencoder so my values are either between 0,1 or -1,-1 because my output activation function will either be a sigmoid or tanh. This way my algorithm can train and the input will be in the same range as the output values of the NN.

However, when I normalized with

x(i)-xmean/(xmax-xmin)

I ended up dividing by 0 in several features of the data which gave NaN. Is is possible to normalize my data so it is between -1,1 or 0,1 while avoiding dividing by 0 for my data?

While you could do this manually, Python also has a handy little function called MinMaxScaler, which will automatically apply max-min normalization to scale data between 0 and 1.

Assume we have an array of 200 values for variables s and t:

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

mu, sigma = 20, 10 # mean and standard deviation
s = np.random.normal(mu, sigma, 200)
t = np.random.normal(mu, sigma, 200)

Reshape your variables if necessary:

s=np.reshape(s,(-1,1))
t=np.reshape(t,(-1,1))

Now, you can see that we are forming two new variables, snew and tnew, which we are scaling using MinMaxScaler.

scaler = MinMaxScaler()
print(scaler.fit(s))
print(scaler.fit(s))
snew=scaler.transform(s)
tnew=scaler.transform(t)

Here is a sample of our new variables:

>>> snew
array([[0.24896606],
       [0.63121206],
       [0.60448469],
       .......
       [0.49044733],
       [0.28131596],
       [0.32909155]

>>> tnew
array([[0.91224005],
       [0.74540598],
       [0.3938718 ],
       .......
       [0.75749275],
       [0.80709325],
       [0.19440844]


What type of regression should I use I have a dataset that gives data on infertitlity and causes. The dataset is mainly 0,1 to represent "yes" and "no". However, some fields have "Sometimes", "Often" which would be represented by -1 or 2. I've only learnt how to do categorical data i.e. 1,0 and Numberical data. So my question is since there are more options other than 1 and 0, which type of regression do I use? Logistic regression or Linear regression?

Using  Logistic regression or Linear regression depend on the dependent variable(DV). Based on your question, I believe that your DV will be infertitlity(Yes/No) so you should use logistic regression because linear regression is for the continuous variables(e.g:exam score) and logistic regression is for categorical variables(e.g.L Yes/NO)


creating a csv file from two csv files I have two csv files:

sub_compiler.to_csv('sub_compiler.csv')
sub_compiler.head()

and

sub_opt = pd.read_csv('sub_opt.csv')
sub_opt.head()

and what I would like to do is to create a csv file where I have something of the form

compiler, opt

How could I do this? I need to do this to make a submission.

Thank's in advance.

[EDIT] Thank you for the answers. Now I have obtained :

don't understand why I have the columns called unnamed.

My objective is to get a csv file that contains only the columns compiler and opt. How could I do this? Thank's again.

[EDIT 2] I have solved my problem but if I open the csv file manually, so just click on the file, I have the following:

s it contains only 11 lines, but it contains 3000 lines in jupyter. Is it there something wrong?

[EDIT 3] I tried to to the following:

import pandas as pd

test = pd.read_csv('1495927.csv')
test

and I have:

where 1495927.csv is the csv file I have created and that is in the image where I have 11 elements.

sub_compiler.merge(sub_opt, how='inner', on='instruction')

This is how to do table join on your table.

Another option if it is already aligned and you just want to concatenate you can do

pd.concat([sub_compiler,sub_opt[['opt']]], axis=1)

This is naive column-wise concatenate.

Finally just proceed with to_csv.


What is a benchmark model? I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing.

A section of the report that I have to write is "benchmark models" and I have no idea what that means. I googled the term and it doesn't seem to be something well defined in data mining. Any idea what that means?

Thanks!

Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which you can benchmark, you might compare to the best naive solution (guessing the mean, guessing the majority class, etc) or a very simple model (a simple regression, K Nearest Neighbors).  If the field is well studied, you should probably benchmark against the current published state of the art (and possibly against human performance when relevant).


Comparing Clustering Over Time I've recently conducted a k-prototypes R routine on some mixed data.
In particular, the data is health data concerning a certain public health intervention, with categorical variables for health scores and numerical demographic data such as age
The utility scores were measured at different points in time, with one sample at week 1, and one sample at week 10.
At the moment, I have only conducted clustering analysis on one of these samples. However, I wondered if there is a recognised routine for making inference on the clusters between week 1, and week 10.
All feedback would be appreciated. I recognise that it is not quite longitudinal clustering, but more comparing two different clustering states.
Thank you

There are many options. One option could be to find clusters at the first time point and define a change "metric" at the second time point. A possible change metric is if the data points belong to the same cluster at the second time point.
You mention a variation of k-means. Since k-means is an iterative algorithm, this cluster re-assignment is what happens between each training step. The difference with this method is you are allowing the data points to move between training steps and measuring how many cluster assignments change.


Target feature in training set or not? If I analyse a random forest in python with scikit I do:

target = "time"

dataIn = data[features + [target]]

# Splitting data into x % training data and Hyperparameter.TestSize % test data
X, y = dataIn[dataIn.columns.difference([target])], dataIn[target]

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y,
                                                                    test_size=TestSize,
                                                                    random_state=42)

If I do a gradient boosted random forest in R I do:

#split data into training and test data sets
trainIndex = createDataPartition(daten$target, p=0.9, list=FALSE, times=1)
train = daten[trainIndex,]
test = daten[-trainIndex,]
rownames(train) <- NULL
rownames(test) <- NULL

gbmTraining <- gbm(train$target~., data=train, distribution = "gaussian", verbose = TRUE,
                   n.trees = 100,
                   cv.folds = 4,
                   n.cores = NULL,
                   shrinkage = 0.3,
                   interaction.depth = 7,
                   n.minobsinnode = 15)

As you see in the scikit implementation I have the target not in the data set. There will be created a new dataframe that has only the target feature as column.

In the gbm R implementation instead the dataframe train contains my target and I communicate to it with train$target.

Therefore I want to ask if you split the data in train and test parts in machine learning algorithms in general (I mean I am comparing two different models), do the target has to be a column in the train data set or do the target has to be a dataframe for it's own?

Whether you need two different data sets depends on the arguments needed in function you are using. Some model fitting functions need a formula and a data argument. Some functions need a xand a y argument. Sometimes you have both options.

For example a linear model (lm) takes a formula and a data argument. Since you can only enter 1 data frame in the data argument, your dependent and independent variables have to be in one and the same dataframe. This is the same as in your example with the gbm function.

However for example a random forest (randomForestpackage) takes the arguments x and y. Therefore, you can have two dataframes (one with y data, one with x data).

Conclusion: it always depends on the function and package that you are using and the only way to find out is to read in the documentation which of those two ways you should be using.


What is the use of function build in custom layers in tensorflow keras? I am trying to build my own custom keras layer following the documentation at
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer
In the custom layers we need three functions call, build and init I am not able to understand where the function build gets called and what is its use.
I checked the source of Dense layer (https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/layers/core.py#L1159-L1190) in keras which is also built on top of the Layer class of tensorflow.keras and the same thing is observed there as well. I cant find any call to the function build.
Can someone please explain me what build function is for ? I mean I understand that add_weights is called inside build and it is important but where is build being called

build is called by the __call__ function which is implemented in the parent Layer class.
From the TF docs:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer
"build(self, input_shape): This method can be used to create weights that depend on the shape(s) of the input(s), using add_weight(). __call__() will automatically build the layer (if it has not been built yet) by calling build()."


Consider ratings as sentiment labels? Beginner here!

I have a dataset, with reviews of a product as text, ratings for the product.

My previous motive was to use Naive Bayes classifier for sentiment analysis. But my data doesn't have the variables( sentiment) required - negative/positive.

Shall I use the ratings (1-5) and encode it as positive and negative?
Or using the lexicon-based methods is more valid?

That would depend on the exact goal of the task and the specifics of the dataset, but in general I would say that it's always better to use the information specifically provided with the data if it's relevant for the task. In this case the rating for the product is indeed very likely to reflect the sentiment of the text, so I would go with it. Notice that you could also do both and compare the cases where the predicted sentiment differs from the one derived from the rating.

Given that the ratings are provided as 1-5 scores I would also consider the option of treating the task as a regression problem, instead of the standard binary classification setting.


How exactly does adding a new unit work in Cascade Correlation? I've just read The Cascade-Correlation Learning Architecture by Scott E. Fahlman and Christian Lebiere.

I think I've got the overall concept (or at least the "cascade" part - a 4min YouTube video how I think it works):

Start with a minimal network with input and output units only
Learn those weights with standard algorithms (e.g. gradient descent - they seem to use another training objective which I don't quite understand, so it is gradient ascent in the paper)
When the network doesn't improve, add a single new hidden unit. This unit gets input from all input nodes and all hidden nodes which were added before. Its output goes to all output nodes only.
Repeat step 3

However, I don't understand the details of step 3: The input weights to hidden units are frozen (indicated by boxes in the paper). When exactly do they get frozen? Are they just initialized by random and never learned at all?

I also don't understand this paragraph:

  To create a new hidden unit, we begin with a candidate unit that receives
  trainable input connections from all of the network's external inputs and from
  all pre-existing hidden units. The output of this candidate unit is not yet
  connected to the active network. We run a number of passes over the examples of
  the training set, adjusting the candidate unit's input weights after each pass.
  The goal of this adjustment is to maximize $S$, the sum over all output units $o$
  of the magnitude of the correlation (or, more precisely, the covariance)
  between $V$, the candidate unit's value, and Eo, the residual output error
  observed at unit o. We define S as

  $$S = \sum_{o} | \sum_p (V_p - \bar V) (E_{p,o} - \bar{E_o}) |$$

  where $o$ is the network output at which the error is measured and p is the
  training pattern. The quantities $\bar V$ and $\bar{E_o}$ are the values of
  $V$ and $E_o$ averaged over all patterns.

What is an "residual output error"? Is $V_p$ simply the activation of the unit given the pattern $p$? What does the term $S$ mean and why do we want to maximize it?

I've been reading up on cascade correlation quite a bit recently and made a python implementation https://github.com/DanielSlater/CascadeCorrelation (though it still needs a bit of cleaning up/extra work and has a bunch of me mucking around with using Particle Swarm Optimization for selecting candidates, definitely not production ready).

To try and explain step 3.

Start by creating a number of candidate hidden nodes with random weights. These have incoming connection from all existing hidden nodes and input nodes.
We then use that equation $$S=\sum_o \left | \sum_p(V_p - \overline{V})(E_{p,o} - \overline{E_o}) \right |$$ to train the candidate nodes.
The residual output error is the difference between the output of the network and the target value(think sum of squared error without the square).
$S$ is the correlation between the activation of our candidate node and the the residual error.
$V_p$ is the activation of the candidate node given input $p$.
After a bit of backprop training against $S$ we choose our best candidate. This becomes a new hidden node.
This is when it's weights are frozen. That is to say after random initialization then back-prop training, then selecting the best one.


How to Identify Repeating Data Entries when the Repeated Entries are Spelled or Constructed Differently I have a dataset of entries and a variable for the owner of the entry. Some of these people occur more than once. However, the names are sometimes written differently. I want to eventually be able to aggregate the other data to the single owner. These are the names of business owners so sometimes it's a singular name, sometimes it's more than one name, and sometimes it's just the company name. Here's an example of some of the styles of names in the data:

DOE JOHN
DOE JOHN J
DOE, JOHN
DOE, JOHN + JANE
DOE JOHN + JANE
JOHN DOE J ETAL % JOHN J DOE
COMPANY CO

I've never done anything like this before. How could I go about identifying some of the same people? Is there a way to create an index to identify the similarity between these groups? Most of the ones I've seen are for longer text. Is there an index well suited for this?
I apologize if this is too basic a question. I'm new to doing things like this and I'm not sure if I know exactly what to search for. I'm most comfortable with Stata and R but I've used Python before and I could eventually figure out how to do something with that.

For R: Have a look and the stringr package. I would use for example the str_detect() function as follows: str_detect(column_of_different_names,"DOE|company_name"). This will return TRUE for each string that includes "DOE" or the company name in "company_name".