Binary classification with time-series features I have the following time-series features: Diastolic Blood Pressure, Systolic Blood Pressure, Heart Rate, RR variability and Arterial Blood Pressure. Each of these clinical parameters was measured for 900 seconds during a surgical procedure and after the surgery, the patient was assessed for acute kidney injury: 1(yes) or 0(no).

My training data kind of looks like this: (see below for screenshot)

Patient 1 Time(s) Features AKI

Patient 2 Time(s) Features AKI

and so on.

What approach would I take to utilize this data for the binary classification task?

I would plot the measurements time $t$ and the corresponding measurement 5 to 10 samples for each category. Try to detect some patterns. Possible patterns are the trend (Is the cure growing? Is it linearly? Exponentially?), frequencies of oscillations (Does one category have oscillations with higher magnitude or frequencies? You can use Fast Fourier Transform for this) self similarity of signal (autocorrelation) Then look at means, median, standard deviation, skewness and kurtosis of your signals.

After having extracted all these features I would try to calculate the correlations of your features with the target variable. Then you can eliminate variables that are not very highly correlated with your target variable. In the next step, I would look at the correlations between your features and eliminate the variables that are highly correlated by eliminating the one variable which less correlated with the target variable. Then I would use some classical binary classifiers like discriminant analysis or logistic regression.

If you see that this method will not lead to sufficient results then you should try more sophisticated methods like neural networks/decision trees for the features that you extracted.


2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd I'm trying out a Kaggle competition, which puts me in the unusual position of being able to get feedback on my models' "true" performance (you can submit several predictions per day and they give you your score on a performance metric -- in this case MSE -- each time).

As such, I've been observing a situation I can't explain -- my cross-validation results for my first stage Random Forest model, constructed with sklearn in Python, are consistently accurate within 1 percentage point of the true MSE.

For example, my first  stage model will give a CV MSE of about .22 and the "ground truth" (actual) MSE will turn out to be about .23.

OTOH, my 2nd stage Random Forest, which is a different type of RF constructed in R, gives way "better" (smaller) training and cross validation MSE errors, like 0.05, but consistently turns out to have a much higher true MSE, like .23. In the 2nd stage Random Forest I have the predictions from the first stage and a few additional engineered features as regressors.

Is there a statistical explanation for this?

For example is it the result of some sort of recursion from stacking the models? If so, given that most winners of these Machine Learning contests are stacking many models, what's the appropriate way to do so that would avoid this statistical problem?

To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the first RF and some other additional features. The error on CV training data is very low and on test data are very high.

Now we will analyze the performance of the second RF without considering how it was built. Usually if you fit data very well on training set and very poor on test set it means you simply overfit data. This happens when your model learns too much from the data. The point is that in your training data you have two kinds of information. The first one is the real structure of the data, and the second one is the noise around that data, assuming that you have enough data and your model is able to handle the useful structure. The ideal case happens when you learn only the useful structure and when leave out the noise.When that happens you will have close errors in training and test data, which means that what your model learnt is at least almost only useful data. When you learn too much, which means you learn useful and noise together, your training error decreases because the results have the noise structure which is in training data, and on test data the errors grows since the noise from test data does not look like the noise from training data.

Now the first learner looks like it does not over fit. This of course does not always mean that you can't do better. It only means that what your learn is a structure which is also found in test data and thus has predictive power. So is understandable that you aim to improve the performance of the first predictor. One way would be to try to improve that predictor, and you can search for that by studying how that model learns. For example you can study if that model improves performance with more data by training repeatedly on samples of increased size and see if stabilize itself or not. There are many approaches. Another approach would be ensembles and a prominent one is stacking.

But you did not understood the idea of stacking. The whole point of stacking can be expressed in the following statement: explore the model space by building various different model families and in the end get the best from all of them. So stacking starts with the assumption that since we do not know which would be an ideal model for our data, we should try many imperfect ones. Some of them will behave well in some regions of space, some others in other regions of space, but none of them will be able to handle all the space at high performance. The key point is to have models from different families. Sometimes the same algorithm can produce very different models, and sometimes not. For example two SVMs with very different kernels can produce totally different surfaces. Sometimes even for the same kernel an SVM can produce very different results for very different complexity parameter. Often for RF this is not the case, but it is not impossible to get different surfaces with different parameters on RF. So the conclusion is that stacking aims to explore different ways of learning.

What you have done is to apply stacking on a single base learner by adding new features. First of all is that in general it does not work. The predictions from the first learner has a string structure, it comes from averaging on regions. So, this predictions compared with additional variables from the second model are more powerful. This means that every wrong thing learned in the first RF becomes very powerful on the second RF, because it uses the same averaging over regions procedure for learning. So the errors from the first learner amplifies in the second learner. I do not know about the additional features, but in this conditions it looks like it is very easy to overfit.

Turning back to what's in a data set: this data set has a structured content which you have to learn and an unstructured content (noise) around that. Your purpose of learning is to learn only the structure. The first RF is able to distinguish the structure from noise simply because the noise is not altered. The second RF is not able anymore to make that distinction, since the predictions of the first RF are already smooth in regions, which means it has a powerful structure, at least from the point of view of the second RF, which learns in the same way.

A last idea is that it looks like what you have done is somehow similar with a boosting procedure. The first learner gives you a structure which is used as a baseline by the second learner. But there is a big difference than in boosting. While in boosting you have a base line which you aim to improve, you improve that baseline by looking again at the original features, which have the noise unstructured. In your case at the second iteration you do not have again that chance and everything is structured. Thus errors amplifies.

My 2 cents: try multiple learners, eventually using additional features in those learners, try different models for base learners, start with very simple (low complexity or small degrees of freedom for) the stacking learner, like a penalized logistic regression and more further to more complex ones in a controlled fashion.


Trying to get Orange connected to Postgres Database I recently came across Orange, and am very keen to try it for our company's purposes (around 20 potential users, perhaps more). I have installed Orange (latest version) and am trying to connect to our database, as it has a large (multi GB) database of telemetry collected from the sawmill.

However I am unable to connect. I have installed psycopg2 as instructed, but when I try to connect, I get a message saying "Could not open Extension file" ... Apparently a tsm_system_time.control file is missing. And it indeed is not in the folder as directed by psycopg2, I have looked.

I have an IT specialist friend, who tried installing Orange on his laptop (he also has a postgres database on it) and he got the same message, except he apparently was missing a quantile.control file. He didn't proceed further, but I am, wondering

What are all these "missing" .control files, and
Why does psycopg2 need them if they are not pertinent to database operation (everything else works in the database, including reporting etc.)
How do I resolve the problem? No help forthcoming online in terms of missing .control files.

I'm running on Windows 7 (Database is a local server running Windows 2012 Server)

To install tsm_system_time you need the contrib package installed too, see instructions on that page (or this page).

These are various extensions that are not needed for basic database operations but are useful for certain applications, this is apparently such a use case. The control file is like a script that tells Postgres how to install each extension when it encounters the CREATE EXTENSION command.


cosine_similarity returns matrix instead of single value I am using below code to compute cosine similarity between the 2 vectors. It returns a matrix instead of a single value 0.8660254.

  [[ 1.         0.8660254]

  [ 0.8660254  1.       ]]

from sklearn.metrics.pairwise import cosine_similarity
vec1 = [1,1,0,1,1]
vec2 = [0,1,0,1,1]
print(cosine_similarity([vec1, vec2]))

Based on the documentation cosine_similarity(X, Y=None, dense_output=True) returns an array with shape (n_samples_X, n_samples_Y). Your mistake is that you are passing [vec1, vec2] as the first input to the method. Also your vectors should be numpy arrays:

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
vec1 = np.array([[1,1,0,1,1]])
vec2 = np.array([[0,1,0,1,1]])
#print(cosine_similarity([vec1, vec2]))
print(cosine_similarity(vec1, vec2))

  X : ndarray or sparse array, shape: (n_samples_X, n_features)
      Input data.

So you have to specify the dimension.

np.array([1, 2]).shape

has funny shape:

  (2, )


LSTM predicting expected value and standard deviation with one-hot encoding I am implementing LSTM to a time series prediction which requires predicting both the expected value (i.e. the mean of the prediction), and the standard deviation (i.e. the interval that the future value should fall into).

I have considered that one-hot encoding can return the possibility of the prediction of each category, which can then be used to both find the mean and the S.D. of the prediction, for example if the output of the LSTM is

Category  | Possibility
[0,5)     |    0.1
[5,10)    |    0.2
[10,15)   |    0.4
[15,20)   |    0.3

I then conclude that the expected value of the prediction is
(2.5x0.1 + 7.5x0.2 + 12.5x0.4 + 17.5x0.3 = 12)

and the S.D. is find in the similar manner.

My question is, is this method a valid method to find the mean and the S.D. of the prediction, why / why not?

The standard deviation and mean of a categorical variable is not meaningful. It looks like the original data are from a range of [0, 20), and the space has been discretized. Now, instead of ranges, you have ordered categories 1 through 4. The individual numbers within each category have lost their meaning. For more details, see the accepted answer here


One attribute includes another attribute I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, but always zero when "Voice mail plan" is no. When removing "Voice mail plan" from the dataset the accuracy of classifiers is lowered, so how can we inform the classifier that No is impeded in zero voice calls

The two features "voice mail calls" and "voice mail plan" are related but they are not linearly correlated. "Voice mail plan" still contains some information that is not available from other features. Why do you want to remove "Voice mail plan" in first place? If you need to decrease your number of features, you can try dimension reduction (linear or non-linear), this way you make sure most of the variance of your feature set is considered for building your model.


[Keras][LSTM] error due to shape mismatch I have following data. Where I have 2 samples. Each sample I have 3 time steps each with 2 features. I intend to have 2 batches (to updates weights after every sample)

X=np.array([[[0.54, 0.3], [0.11, 0.2], [0.37, 0.81]],[[0.55, 0.4], [0.12, 0.3], [0.38, 0.9]]])
X=X.reshape(2,3,2)
y = np.array([[[0.2],[0.3],[0.4]],[[0.3],[0.4],[0.5]]])
y = y.reshape(2,3)

Following is my Keras LSTM code

model = Sequential()
model.add(LSTM(50, activation='relu', batch_input_shape=(2, 3, 2)))
# model.add(LSTM(50, activation='relu', batch_input_shape=(1, n_steps, n_features)))
# model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

model.fit(X, y, epochs=10000, verbose=0)

Which gives me following error. According to my understanding it is due to the size mismatch but cannot figure out the issue in my sizes. Appreciate your input

ValueError: Error when checking target: expected dense_1 to have shape (1,) but got array with shape (3,)

LSTM gives 2 outputs:-

Sequences
States.

Now as you are working with Keras, by default the LSTM layer is designed to return states over Sequence (Which can to altered by turning return_sequence argument to True in keras layer).

Looking at the shape of your output, (2,3) is incorrect for the states and is actually looks like it should be for the sequence. This is the reason why your code is throwing an error.


Dimensionality and Manifold A commonly heard sentence in unsupervised Machine learning is

  High dimensional inputs typically live on or near a low dimensional
  manifold

What is a dimension? What is a manifold? What is the difference?

Can you give an example to describe both?

Manifold from Wikipedia:

  In mathematics, a manifold is a topological space that resembles
  Euclidean space near each point. More precisely, each point of an
  n-dimensional manifold has a neighbourhood that is homeomorphic to the
  Euclidean space of dimension n.

Dimension from Wikipedia:

  In physics and mathematics, the dimension of a mathematical space (or
  object) is informally defined as the minimum number of coordinates
  needed to specify any point within it.

What does the Wikipedia even mean in layman terms? It sounds like some bizarre definition like most machine learning definition?

They are both spaces, so what's the difference between a Euclidean space (i.e. Manifold) and a dimension space (i.e. feature-based)?

What is a dimension?

To put it simply, if you have a tabular data set with m rows and n columns, then the dimensionality of your data is n:

  What is a manifold?

The simplest example is our planet Earth. For us it looks flat, but it really is a sphere. So it's sort of a 2d manifold embedded in the 3d space.

  What is the difference?

To answer this question, consider another example of a manifold:

This is so-called "swiss roll". The data points are in 3d, but they all lie on 2d manifold, so the dimensionality of the manifold is 2, while the dimensionality of the input space is 3.

There are many techniques to "unwrap" these manifolds. One of them is called Locally Linear Embedding, and this is how it would do that:

Here's a scikit-learn snippet for doing that:

from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_neighbors=k, n_components=2)
X_lle = lle.fit_transform(data)
plt.scatter(X_lle[:, 0], X_lle[:, 1], c=color)
plt.show()


Overfitting CNN models I tried to develop a number of CNN architectures to train on a 1000-point subset of the "cat-dog" Kaggle training set (meaning, by the way, that all 1000 data points were labeled). I used a 700-150-150 train-validate-test split, and used the following code on an Xception architecture:

pre_xception_model = keras.applications.Xception(include_top=False, weights='imagenet',
                                                 input_tensor=None, input_shape=(224,224,3), pooling=None, classes=2)
for layer in pre_xception_model.layers:
    layer.trainable = False
dropout = Dropout(0.5)(pre_xception_model.output)
flatten = Flatten()(dropout)
output = Dense(2, activation='softmax')(flatten)
xception_model = Model(pre_xception_model.input, output)

xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='categorical_crossentropy', metrics=['accuracy'])

aug = ImageDataGenerator(rotation_range=20, zoom_range=0.15,
    width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,
    horizontal_flip=True, fill_mode="nearest")
batches = 20

xception_model.fit_generator(aug.flow(X_train, y_train, batch_size = batches), steps_per_epoch = len(X_train) // batches,
                          validation_data = (X_valid, y_valid), validation_steps = 4, epochs = 10, verbose = 1)

xception_prob = xception_model.predict(X_test, verbose=1)
xception_predict = xception_prob.argmax(axis=-1)
cm_xception = confusion_matrix(y_test[:,1], xception_predict)
plot_confusion_matrix(cm_xception, cm_plot_labels, title='Confusion Matrix')

print(f'\nAccuracy = {(cm_xception[0,0]+cm_xception[1,1])/150}\n')

This produced , and the .predict() produced . I'm at a loss as to explain the overfitting; can someone here help me as to where I went wrong on this?

Correct me if I'm wrong, but it seems you have a binary classification problem - the image either contains a cat or a dog.

If that's the case, then:

output = Dense(1, activation='sigmoid')(flatten)

and

xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='binary_crossentropy', metrics=['binary_accuracy'])

This thread is a great resource on the difference between the two questions.

Let me know if it works.

EDIT: Further explanation

Although we normally talk about “binary classification”, the way the outcome is usually modeled is as a Bernoulli random variable, conditioned on the input data. So:

$$P(y = 1|\mathbf{x}) = p, \ 0\leq p\leq1$$

A Bernoulli random variable takes on values between 0 and 1. So that’s what our network should produce. One idea might be to just clip all values of $wth+b$ outside that interval. But if we do this, the gradient in these regions will be 0: The network cannot learn.

A better way is to squish the complete incoming interval into the range (0,1), using the sigmoid function:

$$\sigma(x) = \frac{1}{1 + e^{(-x)}}$$

But the sigmoid function saturates when its input gets very large, or very small. Is this problematic? It depends. In the end, what we care about is if the cost function saturates. If we follow the general principle of maximum likelihood/cross entropy, the loss will be:

$$ - log P (y|\mathbf{x}) $$

where the $log$ undoes the $exp$ in the sigmoid.

In Keras, the corresponding loss function is binary_crossentropy. For a single item, the loss will be:

$−log(p)$ when the ground truth is 1
$−log(1−p)$ when the ground truth is 0

Here, you can see that when for an individual example, the network predicts the wrong class and is highly confident about it, this example will contribute very strongly to the loss. Therefore, you only need to have one output node, because binary classification is "symmetrical". By that, I mean that if we say that an item is of class 0 with probability p, we are also referring to the fact that this same item is of class 1 with probability 1 - p.


GridSearchCV Decrease performance RF Can Gridsearchcv params perform worst than default RF?
RF with default values performs rmse_train=4886,r^2_train=0.84, rmse_test=11008,r^2_test=0.22.
RF after GridSearchCV tuning performs worst on train set (rmse_train=9104,r^2_train=0.45, rmse_test=11091,r^2_test=0.21). This is the code (my first ML algorithm implementation)
#data
features = pd.read_csv("dati_nn.csv")
labels = np.array(features['Cost_damage']) #regression problem
features = features.drop('Cost_damage', axis=1)
features = np.array(features)
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=0.2, random_state = 123)

#RF default
rf = RandomForestRegressor(random_state = 123)
rf.fit(train_features, train_labels)
pred_train = rf.predict(train_features)
predictions = rf.predict(test_features)
# compute rmse and r_score
rmse_train = np.sqrt(metrics.mean_squared_error(train_labels, pred_train))
rmse_test = np.sqrt(metrics.mean_squared_error(test_labels, predictions))
r2_train = r2_score(train_labels, pred_train)
r2_test = r2_score(test_labels, predictions)

#GridSearch
param_grid = {
    'bootstrap': [True],
    'max_depth': [20, 30],
    'max_features': ['auto', 'sqrt'],
    'min_samples_leaf': [1, 2, 3],
    'min_samples_split': [5, 8, 10, 12],
    'n_estimators': [500, 600, 700, 800, 1000]
}
rf = RandomForestRegressor(random_state = 123)
grid_search = GridSearchCV(estimator = rf,
                           param_grid = param_grid,
                           scoring = 'neg_mean_squared_error',
                           cv = 10,
                           n_jobs = -1,
                           verbose = 2
                          )
grid_search.fit(train_features, train_labels)

best_grid = grid_search.best_estimator_
pred_train = best_grid.predict(train_features)
predictions = best_grid.predict(test_features)
#rmse and r2_score as above

In my opinion, the code is correct but I don't understand why rmse grows.
I searched for many days on the internet to find an answer...no success. I tried to use RandomizedSearchCV, same problem.
Any suggest? Is my workflow correct?
Thanks in advance!!

These are different for a valid reason
The standalone model is overfitting. You can see that with r2score of 84 Vs 22.
The reason for that is, the standalone model goes to full depth and hence overfit the train data and badly fit the test data.

max_depthint, default=None
The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

While with CV, you have parm for max_depth. Hence bad but Truth.


How to adjust the hyperparameters of MLP classifier to get more perfect performance I am just getting touch with Multi-layer Perceptron. And, I got this accuracy when classifying the DEAP data with MLP. However, I have no idea how to adjust the hyperparameters for improving the result.

Here is the detail of my code and result:

.

from sklearn.neural_network import MLPClassifier

import numpy as np
import scipy.io
x_vals = data['all_data'][:,0:320]

y_vals_new = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])
y_vals_Arousal = np.array([3 if each=='Pas'  else 4 if each =='Neu' else 5 for each in data['all_data'][:,321]])

DEAP_x_train = x_vals[:-256]        #using 80% of whole data for training
DEAP_x_test = x_vals[-256:]         #using 20% of whole data for testing
DEAP_y_train = y_vals_new[:-256]     ##Valence
DEAP_y_test = y_vals_new[-256:]
DEAP_y_train_A = y_vals_Arousal[:-256]   ### Arousal
DEAP_y_test_A = y_vals_Arousal[-256:]

mlp = MLPClassifier(solver='adam', activation='relu',alpha=1e-4,hidden_layer_sizes=(50,50,50), random_state=1,max_iter=11,verbose=10,learning_rate_init=.1)

mlp.fit(DEAP_x_train, DEAP_y_train)

print (mlp.score(DEAP_x_test,DEAP_y_test))
print (mlp.n_layers_)
print (mlp.n_iter_)
print (mlp.loss_)

If you are using SKlearn, you can use their hyper-parameter optimization tools.

For example, you can use:

GridSearchCV
RandomizedSearchCV

If you use GridSearchCV, you can do the following:

1) Choose your classifier

from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(max_iter=100)

2) Define a hyper-parameter space to search. (All the values that you want to try out.)

parameter_space = {
    'hidden_layer_sizes': [(50,50,50), (50,100,50), (100,)],
    'activation': ['tanh', 'relu'],
    'solver': ['sgd', 'adam'],
    'alpha': [0.0001, 0.05],
    'learning_rate': ['constant','adaptive'],
}

Note: the max_iter=100 that you defined on the initializer is not in the grid. So, that number will be constant, while the ones in the grid will be searched.

3) Run the search:

from sklearn.model_selection import GridSearchCV

clf = GridSearchCV(mlp, parameter_space, n_jobs=-1, cv=3)
clf.fit(DEAP_x_train, DEAP_y_train)

Note: the parameter n_jobs is to define how many CPU cores from your computer to use (-1 is for all the cores available). The cv is the number of splits for cross-validation.

4) See the best results:

# Best paramete set
print('Best parameters found:\n', clf.best_params_)

# All results
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))

5) Now you can use the clf to make new predictions. For example, check the performance on your test set.

y_true, y_pred = DEAP_y_test , clf.predict(DEAP_x_test)

from sklearn.metrics import classification_report
print('Results on the test set:')
print(classification_report(y_true, y_pred))


Can ReLU replace a Sigmoid Activation Function in Neural Network I'm new into this and I'm trying to replace the sigmoid activation function in the following simple NN with ReLU. Can I do that? I've tried replacing the sigmoid function, but it's not working. The output should be the AND gate(if input (0,0)-> output 0).

import numpy as np

 # sigmoid function
def nonlin(x, deriv=False):
   if(deriv == True):
       return x*(1-x)
   return 1/(1+np.exp(-x))

# input dataset
X = np.array([[0, 0],
          [0, 1],
          [1, 0],
          [1, 1]])

# output dataset
y = np.array([[0, 0, 0, 1]]).T

# seed random numbers to make calculation
# deterministic (just a good practice)
np.random.seed(1)

# initialize weights randomly with mean 0
syn0 = 2*np.random.random((2, 1)) - 1

for iter in xrange(10000):
    # forward propagation
    l0 = X
    l1 = nonlin(np.dot(l0,syn0))

    # how much did we miss?
    l1_error = y - l1

    l1_delta = l1_error * nonlin(l1, True)

    syn0 += np.dot(l0.T,l1_delta)

To answer your question directly, yes you can replace the sigmoid activation function with the ReLU activation function.

I think the real question is "should you?"  This is a much harder question to answer.

The sigmoid function returns a number between 0 and 1, but the path of the function is never constant, decelerating as it approaches 0 and 1, and peaking at 0.5 when the input value is 0.

On the other hand, the ReLU function returns a 0 for input values less than 0, while input values above 0, the function returns a value between 0 and 1.

So, sigmoid will only "theoretically" return a 0 at the lowest most extreme value of the input $x$. For the mean input value of $x$, the sigmoid returns 0.5, and for the max value of $x$ will "theoretically" return a value of 1.  Based on a normally distributed variable, you would not expect to see a 0 or 1 very frequently.

With the ReLU function, there is nothing theoretical about the possibility of the function returning a 0.  It will return a 0 for approximately half the input values.

One other point, the activation function can absolutely change the behavior of a neural network, but a poorly chosen activation function may not necessarily prevent a NN from converging, but it will probably require a different model architecture, and probably make it harder to train.

At this point, the community appears to rely on heuristics and empirical assessment to choose one function over another.  There are certain patterns that seem to be accepted by the community as a general rule of thumb in certain domains. There are also, some patterns that are pretty well accepted as the only valid pattern. For example, when using the softmax activation function to determine the output, it is generally accepted that the previous hidden layer must employ the sigmoid function.

My guess is that most practitioners start with a sigmoid activation function unless there is domain knowledge that implies that it would be inappropriate, but I expect many different architectures are evaluated during model selection.


Regression coefficient(s) when explanatory & response variables are time series I have multiple VMs (Virtual Machines) on one physical server (hardware machine). I have CPU utilization information (every 5 minute) for all the virtual machines and the physical server. Now I want to understand how each VM is contributing to the overall load of the physical server. I am assuming that this is a time series problem. The CPU utilization of each VM as well the physical server are separate time series. One way to approach this problem is to apply linear regression considering CPU utilization of each VM as explanatory varaibles and CPU utilization of physical server as response variable.

  cpu_physical_server(t) = Function(cpu_vm_one(t) + cpu_vm_two(t) + cpu_vm_three(t) + .....)

As per my understanding, I can't directly apply regression modelling (lm() or glm()) as these are all time series (Link).

The question is how to calculate the regression coefficients when both explanatory and response variables are time series? What kind of correction(s) should I apply?

You should also have a look at the Cross Validated forum on StackExchange a lot questions asked there are related to time series analysis.

What you are looking for seems like a "Dynamic Regression or "Intervention Analysis" https://www.otexts.org/fpp/9/1 (with examples in R)

An here https://onlinecourses.science.psu.edu/stat510/node/72 you will find a procedure which does an ordinary linear regression but investigates the autocorrelated errors caused by the time series structure of your data (also with an example in R)


Sentiment Analysis Datasets I am looking for sentiment analysis data, mostly customer product review.

I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date data and I am willing to pay.

Is there any vendor that sells aggregated sentiment data from retailers or e-commerce. Or is there any other avenues we can get the review data.

There is an Amazon Customer Review dataset which you can read more about here - the data is stored in S3 and freely available for anyone to use.

It contains over 160 million customer reviews (the README suggests 130 million, but more data must have been added since that was written) made available by Amazon - you can access them in either TSV or Parquet format.

There is also a blog post on using this data set in sentiment analysis here, which links to another post giving more technical detail. In summary, the blog looks at how sentiment correlates with the star rating in the reviews over time, across product categories, etc.


Why my perceptron doesn't train well and produces bad results on test data? I am a newbie in Machine learning and I am writing a small code for Perceptron. This is the first time I am writing code in Python.
I have four training data points (X). As they are used for supervised learning so, each data point has its corresponding correct output pair (D). I have implemented SGD and used generalized Delta rule (wij ← wij + α δixj). I have trained my perceptron 10,000 times (epochs= 10,000).
Although everything looks fine to me, I don't get the right results when I test it with test values. I need some suggestions so that I can improve my results on test data. P.S. How can I improve this code?

Code

import numpy as np

def sigmoid(x):
return 1 / (1 + np.exp(-x))

def Delta_SGD(W, X, D):
 N = 4
 for x in range(N):

    v1 = np.dot(X[x][0], W[0])
    v2 = np.dot(X[x][1], W[1])
    v3 = np.dot(X[x][2], W[2])
    #weighted sum
    V = v1+v2+v3

    #output of neuron
    y = sigmoid(V)

    #error
    e = D[x] - y

    #derivative of sigmoid(y)
    delta = (y)*(1-y)*e

    #Delta rule
    DW = alpha*delta*X[x]

    #updated weights
    W[0] = W[0] + DW[0]
    W[1] = W[1] + DW[1]
    W[2] = W[2] + DW[2]

return W

#input data points
X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])

#Correct output pairs
D = np.array([[0,0,1,1]]).T

#learning rate
alpha = 0.9

#random weights
W =  2*np.random.random((3,1)) - 1

#10000 epochs
for epoch in range(10000):
 W = Delta_SGD(W, X, D)
 print(epoch)

#Final weights after all epochs
print("Final weights are \n", W)

#testing network
N = 4
for x in range(N):

 v1 = np.dot(X[x][0], W[0])
 v2 = np.dot(X[x][1], W[1])
 v3 = np.dot(X[x][2], W[2])

 V = v1+v2+v3
 y = sigmoid(V)
 print("output of neuron is \n ", y)

Two points about the whole thing

You did not test yet. The point behind the training process is to make machine able to learn from the data conditioned on the ability of generalizing this to predicting samples which it has not seen before. Otherwise, the good training is actually overfitting. So here you trained on X and you need to create new samples and check the result to really call it testing (this is an introductory explanation).
Machine Learning is about features a lot! Playing with features and cleaning, modifying and filtering them is a key point. In your example, the last dimension of your 3d data is always 1. Does it distinguish anything? (in my course you get a complete explanation of this in the Lecture 2). So that feature (dimenstion, element of the vector) can/should be removed. To better understanding, imagine your 3d spread of the data. The z axis is always 1 which means the topology of points is what you see in x-y plain. So use only that one.


How do I get the feature importace for a MLPClassifier? I use the MLPClassifier from scikit learn. I have about 20 features. Is there a scikit method to get the feature importance? I found

clf.feature_importances_

but it seems that it only exists for decision trees.

The short answer is that there is not a method in scikit-learn to obtain MLP feature importance - you're coming up against the classic problem of interpreting how model weights contribute towards classification decisions.

However, there are a couple of great python libraries out there that aim to address this problem - LIME, ELI5 and Yellowbrick:

LIME (or Local Interpretable Model-agnostic Explanations, blog post here, arxiv paper here) which "explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction", or basically to explain model outputs by approximating the results of a classifier with a model that can be explained. It's on pypi and they have examples on their github page - very straightforward to get started.

ELI5 (or Explain Like I'm 5) is a "Python library which allows to visualize and debug various Machine Learning models using unified API". Although not all scikit-learn integration is present when using ELI5 on an MLP, Permutation Importance is a method that "...provides a way to compute feature importances for any black-box estimator by measuring how score decreases when a feature is not available", which saves you from trying to implement it yourself.

Yellowbrick is "a suite of visual diagnostic tools called “Visualizers” that extend the Scikit-Learn API to allow human steering of the model selection process" and it's designed to feel familiar to scikit-learn users. Compared to the other two libraries here it doesn't offer as much in the way for diagnosing feature importance, but it's still worth mentioning for more general use cases.


drop columns and rows in one line in pandas I want to drop a range of rows and columns of a dataframe, I did it as follow:

df.drop(df.columns[3:], axis=1, inplace=True)
df.drop(df.index[3:], axis=0, inplace=True)

Can I do the two processes in one method instead of two?
Or is there any more sufficient way to accomplish this?

Why don't you do this:

df = df.iloc[:3, :3]

It returns the same thing as your code.


Predict real world data after modelling with scaled features I trained and test a model with scaled features.
Now, I want to predict a single real world sample. If I have one sample alone, I can't scale it to fit into the model like I did with the test data.
I imagine it's a pretty common situation.
What are the steps I need to do to predict this single sample?

I would say it depends on your training set. If you had a sufficiently large (and representative) training set, then you could say that the mean and the variance of the training set are the estimates of the population mean, thus, use these values also for the standardization of the one single testing sample. If it is not the case, then the whole idea of prediction is not really good.
Generally, it is wrong to use different transformations for training and testing sets separately. You can read about it more here


Error using image: Error when checking input: expected dense_56_input to have 2 dimensions, but got array with shape I am trying to train a Sequential model using simple flow_from_directory() but i am getting this error , I have tried using lesser layers but the error dose not go away.

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
train_directory,
target_size = (28,28),
class_mode = 'categorical')

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Test'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
validation_directory,
target_size = (28,28),
class_mode = 'categorical'
)

model = Sequential()

model.add(Dense(128, input_shape = (784,)))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'])

model.fit_generator(training_generator,epochs=10)

Here is the error:

File "C:\Users\Ankit\.spyder-py3\temp.py", line 31, in <module>
    model.fit_generator(training_generator,epochs=10)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training.py", line 1732, in fit_generator
    initial_epoch=initial_epoch)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training_generator.py", line 220, in fit_generator
    reset_metrics=False)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training.py", line 1508, in train_on_batch
    class_weight=class_weight)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training.py", line 579, in _standardize_user_data
    exception_prefix='input')

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training_utils.py", line 135, in standardize_input_data
    'with shape ' + str(data_shape))

  ValueError: Error when checking input: expected dense_56_input to have 2 dimensions, but got array with shape (32, 28, 28, 3)

The error occurs due to mismatch in the input shape. In the model you have specified it as input_shape = (784,) but the actual images that the model is getting as input are of different size. Just put the input shape as input_shape = (32, 32, 3) and you are good to go. Take a look here for more details on how to specify the input_shape.

Also, you'll have to use convolutional layers to process images or you can use a flatten layer before using dense layers!


What happens when scikit-learn does a Lasso Model? I have started an MLS course. As a beginner and non-mathematician it has been hard.

I am trying to understand the exercise about Lasso Models.
I have done Lasso models on R-cran, but this is my first time with Python.

I have a dataset of cars with seven variables. The exercise consists on doing a Lasso Model to predict the gasoline consumption of the cars, the dependent variable, so $x$ is a table with the rest of variables and $y$ is the consumption.

Then, if I launch Lasso on scikitlearn:

modelLasso = Lasso(alpha=0.1).fit(x, y)

I do not understand the result. Has it generated a prediction on $y$ of every row in the table?
If so, how can I access to the array of predictions of the model and how do I use the model to predict the consumption given new $x$ values?

Lasso() created a Lasso regressor object. The .fit() argument makes the regressor "learn" from your data. The parameters are tuned in order to "fit" the training data.

Once the regressor object is fit, you can use it to .predict() on test data.


From where should I start Machine Learning? I want to know how to start from scratch for Machine Learning. Also which language is best for implementing its algorithms or developing future applications based on it. Thanks!

You can start learning machine learning by attending the lectures of Andrew Ng at Coursera. This course helps you to develop basics as well as advanced concepts in machine learning.
Following is the link to the course:
Machine Learning by Andrew Ng
Start learning Python for implementing algorithms related to machine learning.


Classification of substrings? What is the appropriate method to find n-grams/sub-phrases/parts-of-sequences that are referring to a specific topic or belong to a certain category?

For instance:

Imagine a topic of "transfer of ownership" and some example sentences:

A >change of ownership< occurs when a title is transferred from one person or entity to another
Making a >change in business ownership< is a lengthy and complex process
In this respect, the >transfer of legal ownership< is not the relevant feature for determining the treatment of repo-like operations

What I am looking for is a method to identify sentences and ideally parts of sentences which refer to a specific theme / topic.

What NLP methods are appropriate?

What is the appropriate method to find n-grams/sub-phrases/parts-of-sequences that are referring to a specific topic or belong to a certain category?

An important question to solve this problem would be: what is the range of input topics? Is the topic selected among a predefined closed list? Can it be any search query?
A similar question might be asked about the target documents and/or terms: can they be processed so that any candidate term is extracted in advance, and the task only consists in identifying the right terms for the particular topic?

Assuming the most open variant of the question (i.e. nothing is available beforehand), I think that one would need:

a terminology extraction system which extracts any candidate term from the text (preferably specific to the data to be processed).
a third-party resource in order to calculate a semantic representation (typically a vector) of any possible term including the input topic query, so that the topic can be matched/compared against any term.


Is it scientifically correct to derive conclusions unrelated to hypothesis from A/B test data Consider a software A/B test with the hypothesis that "the addition of feature F is predicted to increase metric X".

At the end of the test, the data doesn't show any significant change in X, but it does show a significant increase in Y - something that wasn't expected or even considered at the beginning of the experiment.

At this point, is it scientifically valid to say that F increases Y, or should a new A/B test be designed and executed?

It looks analagous to drug testing, where reporting of side effects during drug trials is obviously very important - i.e. the increase in Y seems analagous to a side effect. And some famous drugs have begun their lives as research into a side effect. Viagra is probably the most famous case, being a spinoff from a drug developed as angina medication. So in your write-up on your experiment you should definitely report the apparent effect on Y.
However, if the effect on Y is commercially important, then you still need to go back and do an experiment around a hypothesis that references the increase in Y to validate the existence of the effect properly.


Find similar observations in two datasets I have two datasets A and B. What I would like to do is for each observation in A, I would like to find 5 observations from B that are closest and match to A.

How should I start?

Thank you for your help!

Look at "unsupervised nearest neighbor" algorithm. This algorithm needs records to be first expressed as vectors so that "distance" between two point so that it makes sense to talk about distance between two points. For each point in data A, you can look for K nearest neighbors from data B, after expressing all observations in a common vector space.

You will have to handle categorical columns correctly (say using one-hot encoding), as there's no concept of (direct) distance between categorical data.

Python scikit-learn library has a good implementation of this algorithm. Reading the API documentation is a good place to start.


How to choose tools for web dashboard? I am working in a small sales company. Started with making relatively easy, small, static reports in Excel, but then it was required to make a lot of highly-customizable and adaptable reports, that are really troublesome to do in Excel. I do a lot of VBA, but that still doesn't meet my needs. Our ERP has no BI solution. Excel can't keep up with amount of data used (millions of lines).

I am only accustomed to JS and HTML and have some experience working with
relational databases (MySQL, MariaDB in particular).

Basically, what I need help with is deciding what tools would be appropriate for my  needs considering I am not in a rush and would like to develop some skills along the way.

I would like to create a dashboard that would retrieve data from a local DB and process it. I would also need the dashboard to be dynamic - be able to change some inputs and run calculations accordingly. I am interested in packages that are suitable for mathematical analysis. Reports would include both tables and graphs.

I've read about R and Python packages, but still am not sure which one would be the most appropriate.

In the future I guess I would like to distribute it to other colleagues (run it from a server; have different accounts with different access level etc.).

Will highly appreciate any tips!

There are quite a few tools that serve this purpose (E.g.: Tableau , QlikView are expensive but feature rich; Kibana / Dash have Open Source versions).

You can start with Kibana. It will take a week to learn it. Advantages are :

Part of ElasticSearch stack, it is easy to ingest data (Say that you need to update data from ERP every hour / day /week etc.)  'Logstash' component isused for ingesting / updating data
Widely used, so tutorials / documentation are easy to look up
You can customize dashboards with JS / html
As usage grows, you can add features (Such as access control, periodic email of reports as PDF / Excel etc)

Some tutorials / Documentation  :

https://en.wikipedia.org/wiki/Kibana
https://www.elastic.co/products/kibana

https://www.youtube.com/watch?v=gQ1c1uILyKI


Does XGBoost handle multicollinearity by itself? I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, for example: a variable (feature) $A$ also appears as $\frac{B}{A}$ and $\frac{C}{A}$.

My questions are :

How (If?) do Boosted Decision Trees handle multicollinearity?
How would the existence of multicollinearity affect prediction if it is not handled?

From what I understand, the model is learning more than one tree and the final prediction is based on something like a "weighted sum" of the individual predictions. So if this is correct, then Boosted Decision Trees should be able to handle co-dependence between variables.

Also, on a related note - how does the variable importance object in XGBoost work?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features.

Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. However, its a good practice to remove any redundant features from any dataset used for training, irrespective of the model's  algorithm. In your case since you're deriving new features, you could use this approach, evaluate each feature's importance and retain only the best features for your final model.

The importance matrix of an xgboost model is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees. The second column is the Gain metric which implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.


Implemented early stopping but came across the error SGDClassifier: Not fitted error in sklearn Below is the simpler implementation of early stopping which i came across the book and wanted to try it.

# Implement SGD Classifier

sgd_clf =   SGDClassifier(random_state=42,
                          warm_start=True,
                          n_iter=1,
                          learning_rate='constant',
                          eta0=0.0005)

minimum_val_error = float('inf')
best_epoch = None
best_model = None

for epoch in range(1000):
  sgd_clf.fit(X_train_scaled,y_train)
  predictions = sgd_clf.predict(X_val_scaled)
  error = mean_squared_error(y_val,predictions)
  if error < minimum_val_error:
    minimum_val_error = error
    best_epoch = epoch
    best_model = clone(sgd_clf)

Once the above snippet is executed, best model and best epoch are stored in variable best_model and best_epoch.So, to test the best_model, i ran the below statement.

y_test_predictions = best_model.predict(X_test)

But then i came across the error This SGDClassifier instance is not fitted yet

Any hints on how to solve this, would be greatly helpful. Thanks

It is because clone will only copy the estimator with the same parameters, but not with the attached data. So it results a new estimator that has not been fit on the data. Hence, you couldn't use it to make prediction.

Instead of clone, you can use either pickle or joblib.

1. pickle

import pickle
...

for epoch in range(1000):
    ...
    if error < minimum_val_error:
        best_model = pickle.dumps(sgd_clf)

Later if you want to use the stored model:

sgd_clf2 = pickle.loads(best_model)
y_test_predictions = sgd_clf2.predict(X_test)

2. joblib

You can also use joblib, and store the model to the disk.

from sklearn.externals import joblib
...

joblib.dump(sgd_clf, 'filename.joblib')

To use the stored model

clf = joblib.load('filename.joblib')


Building predictive model with low correlated data I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or approaches to tackle such data and thus build a predictive model out of it. To give you a little glimpse of the dataset, here are a few columns from the

+----+----------+-----------+---------+
| ID | OAG CODE | CONSIGNEE | SHIPPER |
+----+----------+-----------+---------+
| 11 | 665      | 1001      | 20100   |
| 11 | 665      | 1006      | 20105   |
| 13 | 667      | 1023      | 20110   |
| 13 | 669      | 1015      | 20104   |
| 13 | 669      | 1006      | 20105   |
+----+----------+-----------+---------+

I want to perform EDA over the data and also build a predictive model from the same. Please point out some of the standard techniques and methods for tackling such a problem.

really hard answer for u question cause there is to little information. Try to make EDA and attach it to question. Cause EDA define the model

Anywhere, for low corellated data, try to use k-NN. If u use Python: scikit-learn have implementation. Also u can try decision trees.

Sorry, but i'm also new in DS and can be wrong:)


is it a good idea to take the derivative or integral of some features and add them as new features in machine learning? I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column and the dataset is a time series dataset so they are monotonic. would it make sense to calculate the derivative or integral and add it as new feature ?

as an example let's say I have speed and acceleration as features, would it make sense to add the jerk (which is the derivative of the acceleration) and the snap(which is the derivative of the jerk) as new features ? also maybe the integral of the speed which would give the displacement I think?

the goal is let's say 2 features are not enough and we want to produce more features, is it wise to add the derivative or integral as a new feature? or is it a bad idea?

I also want to know whether the correlation between the derivative and integral according to timestamp and the feature that I derivated from would be high if I do this and is it bad or good if I make new features that correlate with others in my dataset

While performing feature engineering , transforming the existing features is recommended step instead of adding transformed values of the existing feature into the dataset.
Since, if we add these kind of data elements they will be highly correlated with each other and it effects the model performance.
If you have less features and cannot create a model, ask for additional data.


Which algorithm can i use for predicting length of stay in coming year based on historical claims data? I have two years historical health claims data of one thousand members. Based on this two years data, I have to predict length of stay in hospital in 3rd year for all members. here is the data sample.

Year MembID x1 x2 x3 x4 x5 x6 x7 LengthOfStay

2010 1 6 35 0 3 0 0 4 1

2010 1 8 35 0 5 0 0 3 0

2009 1 5 35 0 5 0 0 3 3

2009 1 3 35 0 8 2 0 8 0

2010 2 6 30 0 3 3 2 4 0

2010 2 8 30 0 5 0 0 3 0

2009 2 5 30 0 5 0 0 3 0

2010 2 5 30 1 5 0 2 2 0

2009 3 5 55 1 5 1 2 2 0

2010 3 10 55 1 5 0 2 2 0

2010 3 5 55 1 5 1 2 2 0

2009 3 10 55 1 5 0 0 2 0

2010 4 5 24 1 5 0 0 2 0

2009 4 3 24 1 8 0 0 2 0

2009 5 10 65 1 5 1 2 4 5

2009 5 5 65 1 5 0 2 3 0

2010 5 6 65 1 3 0 0 4 1

2010 5 4 65 1 5 0 0 4 0
2010 6 10 44 1 5 1 2 4 5
2011--- i expect------ 1

I did the classification with randomforest. How can I proceed further for prediction on 2011?

If you want to do prediction using 2011 features, the answer is yes, you can do that.

However, as you don't want to use these features, the answer might be no.

Without using 2011 features, your dataset will have only 2 samples(2009 and 2010) under the assumption that every memberID is different. Prediction from two samples is neither reliable nor feasible.


Is there any method to determine which clustering algorithm to use on a particular dataset? I'm having a hard time getting kmeans to cluster data effectively. It fails to segment data well even for a simple attribute with 5 categories. I'm aware of DBSCAN, Hierarchical Clustering and GMM. However, just wanted to know if there's any way (visual or otherwise) to narrow down the clustering algorithm which might work on the dataset in question, before I start to write the code for each of these algorithms.

Thanks in advance.

No.

Clustering is an explorative technique, it is subjective what is good, and the best clusters are those that are "interpretable but unexpected", a property that you cannot quantity with statistics. So it is a trial-and-error task.

Furthermore, data preparation is much more important than the choice of clustering algorithm. On badly prepared data, none will work.

Last but not least, categoricial data is a huge problem. It lacks detail for most clustering approaches - treating this as binary variables is much too coarse and tends to produce bad solutions (such as tiny "clusters" and trivial splits on a single variable). This is likely a problem of the data, not the algorithm. Similar issues can be seen with integer attributes or any other attribute that has only very few discrete levels (including Likert-like-scale questionnaires). Methods such as k-modes exist for categoricial data, but often don't produce better results either...


What inference can we draw from the frequency distribution of thresholds? I've the probability scores of positive class of two models. The frequency distribution of those probability scores(thresholds) are like this

Model #1

Model #2

It's a binary classification problem. And the thresholds obtained are from the test data. The stats in the confusion matrix are nearly same for two models. Can we say the Model #2 is doing well in separating classes? What other conclusion we can draw from the frequency of the thresholds.

I'd say the model 1 is performing really well. If you can use different colors while plotting for positive and negative classes, then you should be able to see the difference. When you are trying to do binary classification, the distributions of negative and positive classes should be dipping in the mid region where as Model 2 is the opposite.


Value extraction from a python dataframe [ problem statement specific ] This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that post: Coding Problem - Extracting values from a column and forming a new dataframe [edited]

While extracting the weather parameters from the 'hourly' column entries, It only extracts the first value of a particular weather parameter. Each entry in the 'hourly' column is a list of weather parameters. Each parameter has 24 different values denoting the weather parameter value for 24 hours in a particular day. So, the code I used gives me 31 different entries in the new data frame while it should have been 31*24 entries for the month of January. Can you guys please help me out ?

Notebook link (edited) is provided here.

data set link is presented here.

[code]

import json
from pprint import pprint
import pandas as pd

with open(r'C:\Users\Murali\Desktop\Downloads\2016-1.json') as f:
    s = f.read()
jansixteen = json.loads(s)
#print(jansixteen)

a  =jansixteen['data']['weather']
b = ((pd.DataFrame(a)))
b.head()

def extract_col_as_df(df, column_name):
    data = [datum[0] for datum in df[column_name]]
    df = pd.DataFrame(data = data)
    return df
df_hourly = extract_col_as_df(b, 'hourly')
    print(df_hourly)

df_hourly.drop(['FeelsLikeF', 'HeatIndexF','HeatIndexC','weatherIconUrl','WindGustMiles','weatherDesc','weatherCode'], axis=1)

The problem with your code is that it considers the numerics as column names ie., 0,1,2,3, etc., upto 31 rows and 24 hours column  but u must loop through each of the column name value which is stored as dictionary so u need two looping:

This loops and creates a new list which stores each and every attribute of the json
This adds the attributes to the DataFrame
now perform your operation at this dataframe you might get an answer

Google drive shared location of edited ipython notebook


Layer notation for convolutional neural networks When reading about convolutional neural networks (CNNs), I often come across a special notation used in the community and in scientific papers, describing the architecture of the network in terms of layers. However, I was not able to find a paper or resource describing this notation in detail.
Could someone explain to me the details or point to where it is described or "standardized"?

Examples:

input−100C3−MP2−200C2−MP2−300C2−MP2−400C2−MP2−500C2−output
(source)
input−(300nC2−300nC2−MP2)_5−C2−C1−output
(source)

A good guess seems that xCy are convolution layers (x is number of filters? y is one side of square kernel?). MPz is max-pooling layer (pool size z×z?).

But instead of guessing, I would love to have a reference (which I could possibly also reference in a paper).

One paper referenced by the first paper you linked to is here. It explains in section 3 (experiments) the following notation:

  2x48x48-100C5-MP2-100C5-MP2-100C4-MP2-300N-100N-6N represents a net
  with:

  2 input images of size 48x48
  a convolutional layer with 100 maps and 5x5 filters
  a max-pooling layer over non-overlapping regions of size 2x2
  a convolutional layer with 100 maps and 4x4 filters
  a max-pooling layer over non overlapping regions of size 2x2
  a fully connected layer with 300 hidden units,
  a fully connected layer with 100 hidden units
  a fully connected layer with 6 neurons (one per class)

From this, the answer to your question is:

100C3 means a convolutional layer with 100 maps and 3x3 filters
MP2 means a max-pooling layer with non overlapping regions of size 2x2
200C2 means a convolutional layer with 200 maps and 2x2 filters
etc with the "C" layers (means convolutional, preceding integer is the number of features maps; final integer is the filter size)

According to the second paper you linked, the subscript _5 indicates five pairs of 300nC2−300nC2−MP2 connected layers (see section 3), and the n indicates "the number of filters in the nth convolutional layer is [300]n".  According to the accompanying model diagram (figure 3 in the linked paper), the C2 and C1 layers produce 1x1 output, meaning a scalar value. This would mean C2 is a convolutional layer with 1 map and a 2x2 filter and C1 is a convolutional layer with 1 map and 1x1 filter (though I don't fully understand what this adds).


How do I perform K-Means clustering of the Olivetti Dataset This Question pertains to Matrix Factorization and the full question is given below.
Provide for k-means clustering of the Olivetti dataset the following visualizations:

A scatter plot of the r = 2-dimensional representation of the faces in the latent space, together with the faces which represent the two dimensions/features.
A visualization of the faces which define the features of the latent space for r = 5.
The reconstruction of the faces with indices i ∈ {0, 10, 20} when using a rank of r ∈ {5, 25, 50, 100}

To load the Olivetti Dataset run the following code:

from sklearn.datasets import fetch_olivetti_faces
faces = fetch_olivetti_faces()

And print(faces.DESCR) provides a description of the data set.

When I try to K-Means clustering on the model I receive

**TypeError: float() argument must be a string or a number, not 'Bunch'**

Thank you for your time and consideration

I have a couple of observations about this:

Before you run clustering, I'd run some dimensionality reduction model. Pixel data are too noisy to be fed as they are into a model. With dimensionality reduction you get both noise reduction and lower computational costs.
Images usually come as 2D or 3D arrays (depending whether they are in grayscale or coloured). On the other side, k-Means and other sklearn models require your whole dataset as a 2D array. You should process each image and turn it into a 1D array that can be fed as a single observation.

Could you share your clustering code, just to make sure?


Using One Hot Encoding VS Label Encoder for Football Match Outcomes I am working on a project to predict football matches using bookmakers' odds. The 6 columns I have are Home Team, Away Team, Home Win Odds, Draw Odds, Away Win Odds, Outcome.

The possible football match outcomes are Away Team Win (A), Draw (D) and Home Team Win (H). I have been using label encoders to represent the variables (0, 1, 2 respectively) but it occurred to me that the outcomes may not be ordinal (since there is no order/rank/preferences to it).

Questions

In this case, should I continue with label encoders or should I use one hot encoding? Or it depends on the model used (like Tree based or linear etc)?
If I should change to one hot encoding, how should I ensure that my prediction vector outputs two 0s and one 1 (since it has to be either A, D or H) because sometimes it gave me a vector of three 0s?
I have used one hot encoding for the football teams (20 teams each for both Home and Away). Since a team must play against a unique team, there exists one 1 and nineteen 0s in the first 20 columns and one 1 and nineteen 0s in the next 20 columns. Does this relationship/restriction across the columns affect the models in any way?
Should I apply dimensionality reduction techniques like PCA to it since I now have 20+20+3=43 features as compared to the 5 features previously? I have read that if a tree model is built, the 3 numeric features (bookies' odds) will be rarely used. Or can I just increase the max_features for RandomForestClassifier?

Thanks in advance!

I would expect some ordering, since the expected outcome is probably depending on the team's strengths. However, there may also be a tendency to/away from draws if teams are equally strong. I would guess that the one-hot encoding is the better choice if you have sufficient data to train your model.
Your prediction vector would typically give e.g. [0.1 0.7 0.2] as probabilities, and you select the highest probability as the outcome. Either by hand or in a (final) layer in the model.
It shouldn't. Typical models are dimension agnostic. E.g. a fully connected first layer loses all positional/dimensional bias. For trees similar arguments hold, however, I don't know how well tree models would perform on such sparse data.
The bookies' odds should highly correlate to the expected outcomes, e.g. bookies will likely have odds that reflect relative strength. In such a case, the odds could more or less 'vanish' in the model.
With such sparse data, you should definitely tune your parameters for forest models carefully.

Just some added notes: If you want to improve on your model, you could also include a few calculated parameters, e.g. current rank, win/draw ratio for both teams (in general, or just the respective home/away versions), most recent (home/away) results. Or even travel distance if you have the data.


Is Python suitable for big data I read in this post Is the R language suitable for Big Data that big data constitutes 5TB, and while it does a good job of providing information about the feasibility of working with this type of data in R it provides very little information about Python. I was wondering if Python can work with this much data as well.

To clarify, I feel like the original question references by OP probably isn't be best for a SO-type format, but I will certainly represent python in this particular case.

Let me just start by saying that regardless of your data size, python shouldn't be your limiting factor. In fact, there are just a couple main issues that you're going to run into dealing with large datasets:

Reading data into memory - This is by far the most common issue faced in the world of big data. Basically, you can't read in more data than you have memory (RAM) for. The best way to fix this is by making atomic operations on your data instead of trying to read everything in at once.
Storing data - This is actually just another form of the earlier issue, by the time to get up to about 1TB, you start having to look elsewhere for storage. AWS S3 is the most common resource, and python has the fantastic boto library to facilitate leading with large pieces of data.
Network latency - Moving data around between different services is going to be your bottleneck. There's not a huge amount you can do to fix this, other than trying to pick co-located resources and plugging into the wall.


Which algorithm to use for efficient resource assignment? I am a starter in ML. So pardon me if the question is naive.

We have a Project Management tool where users can create a ticket and assign it to others. This is just one part of the tool but we are trying to automate assignment of ticket to people. Based on the history, i am trying to suggest who should the creator assign the ticket to so that the ticket is closed without any delay, minimal risk etc.. While i have the features/variables ready, I am having a hard time trying to find which algorithm or method to use.

Appreciate the help. Thx.

For the most part, this probably doesn't need a ML solution. Instead it'll probably be based on assigning rules based on business logic. For example, If the Customer is from a certain country, assign +5 points. Then the tickets are assigned out based on that.

This way, it captures a lot of information and goals you're wanting to achieve as business.

If you wanted to go down the ML route, my hunch is a model such as RandomForest or a tree based solution would be helpful as they're relatively explainable. At least as a starting point.

Uber Engineering team talk about how they prioritize their tickets here, which is an interesting read.


Does using your test set ultimately burn your data set in case of failure? Given a data set I want to train a machine learning algorithm on. The data is split into training, validation, and test data.
I now successfully trained my algorithm to work well with the training data and validating using the validation set is also promising. However, when applying the test data the model underperformes.
I am now stuck with two options:

Throw everything away and start new with the same data set. This however has been likened to p-hacking.
Throw the data set away as it is now burned. This could kill my project or be really expensive as I need to recollect data, this might even be impossible.

Is my data set ultimately burned when applying the trained model unsuccessfully on my test set?
Related Bonus: Is there some form of Bonferroni Correction I could apply to keep reusing the data set in case I would have burned the data set?

I would try one more time to split the data & train a new algo. This is similar to p-hacking as you've stated so try to avoid doing this more than once but if the alternative is the project stops there doesn't seem to be much more you can do.
This time when you split the data I would try stratifying it by some feature (the target? Some category of the input variables?). This ensures the training/validation/testing sets have the same proportion of some criteria which might help your performance be a little more consistent when comparing the validation set to the testing set.


How can I improve my regression model? I am seeking theoretical suggestions more than else. This is my first actual practical work, and I am kinda stuck now that I have done a few things: I don't know in which direction to look in order to improve my model.

What kind of data I have

I have a dataset of information about a population. Each entry is a percentage of people belonging to a certain category (my features) over the whole number of inhabitants. I have 52 observation and 35 features, so a 52x35 input matrix. So my target is a 1D-vector of 52 elements, and these are percentages too.

       male  female  students  employees  freelancers  unemployed  catholics...
city1  ##%  ##%     ##%     ##%...
city2
city3
...
city52

and my target vector is the results of the winning candidate of the last presidential election in each city.

What I did

I thought that, having such a small dataset, it would be better to stick to simple models, staying away from polynomial models etc. in order to avoid overfitting. Some engineering work to do on the features, too.

Feature selection

I removed redundant information and I agglomerated variables, for example by summing up "irrelevant" percentage (compared to other values of the same conceptual categories) and filing them under new "other" features.

Model selection

As I said, I decided to keep it simple, start with some simple linear regression and then checking how other regression models performed compared to linreg.

Dataset was splitted into train $(30%)$ and test set. For each model I selected the related parameter which maximized the score. I then did Random Feature Elimination (with cross validation) for each model, using the parameters selected in the previous step, leaving just a few most relevant features (since I know I have few observations compared to the number of features), and took this as final output.

I focused mainly on linear and ridge regression, with the latter performing better. My best R2 is around 0.7xx.

I see a bit of $overfitting$, but no clear path in the residuals. The target points with the worst predictions actually make sense in the context of the work (they are all belonging to a certain category).

After this I tried with some trees and randomforest regressors, but they performed worse than ridge regression.

What to do next

Well, this is where I lose my path. I think I did the right things until now, considering the kind of data I have (but please feel free to correct me), but.. is that it? Am I done or do I have something more to do, or maybe do again something in a different way? When can I call my self satisfied?

This is my very first ML work so any suggestion is very welcome.

I have a similar dataset atm and have been doing some research, not yet finished.

What needs to be kept in mind is the need for generalization given that there is such few samples and many dimensions.

The linear regression model is not robust enough given the dataset, it does not handle a high number of dimensions, compared to samples, well. Even if you could tune in order to get a good result, you are still probably overfitting.

Stepwise feature/model selectors such as Cp-statistics does not seem to do well with so many dimensions vs samples.

Ridge, Lasso, and Elastic net seems to be the way to go for feature selection. These penalized linear regression models handles the high number of dimensions vs samples much better. The linear regression is penalized in order to generalize better and not overfit, creating a less complex solution.
R: GLMNET - Penalized regression
Depending on alpha, you may receive a different feature set, coefficient that are not zero. E.g Lasso is biased for not returning features that are very correlated with each other while ridge might return a set of variables/dimensions/predictors/features that are correlated with each other.

K-Fold Cross-validation could be used in order to check the features selected are selected over and over again. This could be used in order to make sure that you don't overfit.

I hope this gives you a few ideas.


Intra-cluster similarity metric I have some observations belonging to groups and I would like to compute the similarity of them within different groups in order to tell which observations, within specific groups, have similar characteristics or not.

Which metrics can be useful to do such things and under which conditions?

I know that there are many metrics to compute the similarities between individuals in a group, but there is no threshold to tell whether individuals in a cluster share similar patterns or not.

I would like to know which metrics can be useful to do such things

Since you are clustering your data, why not use the same metric that you used during clustering for comparing your data inside your clusters? I assume this should technically give you exactly what you need.

  there is no threshold to tell whether individuals in a cluster share similar pattern or not

This hardly depends on your data and what you want to do with your similarity. If you just want to retrieve the most similar point, you can do it pretty straight forward.

If you, on the other hand, want to remove outliers using this measure, you can surely also do so, but you might as well just use an algorithm that deals with outliers in the first place.


On fitting a Poisson distribution to make sense of data Hi guys I am working with a regular network which has the shape of a square grid and contains 100x100=10000 nodes. The edges (links) between these nodes simply follow the shape of a chess table: each node which is not placed in the corner or along the boundary has 4 connections only, all of them involving its nearest neighbors. Accordingly, the nodes on the boundary have 3 connections only, while the nodes in the four corners have 2.

Now, this brings me to my problem. If you plot such data you will figure out you have 4 nodes with links=2, 98*4 nodes with links=3, and 10000-(98*4)-4 with links=4. This translates into three tuples to plot: (x,y)=(2,4),(3,392),(4,9604). The resulting histogram is heavily skewed toward the right hand side:

My question is: what kind of distribution do you think would fit this dataset? I was thinking of a Poisson distribution (the x-axis values are discrete and not continuous) skewed to the right. I will appreciate any kind of help/guidance. Thank you!

Unlike other graphs, the degree distribution is a function of N. Specifically, the number of nodes with k=2 is constant (4), the  number of nodes with k=3 grows linearly with the length of one side $4l-4$, while the number of nodes with k=4 grows exponentially with the length of a side, $l^2-4l-4$. In the large $l$ (or $N$) limit, the degree distribution is 4.

You can get the exact distribution by normalizing this piecewise defined function (divide by $l^2=N$).


Detecting cats visually by means of anomaly detection I have a hobby project which I am contemplating committing to as a way of increasing my so far limited experience of machine learning. I have taken and completed the Coursera MOOC on the topic. My question is with regards to the feasibility of the project.

The task is the following:

Neighboring cats are from time to time visiting my garden, which I dislike since they tend to defecate on my lawn. I would like to have a warning system that alerts me when there's a cat present so that I may go chase it off using my super soaker. For simplicity's sake, say that I only care about a cat with black and white coloring.

I have setup a raspberry pi with camera module that can capture video and/or pictures of a part of the garden.

Sample image:

My first idea was to train a classifier to identify cat or cat-like objects, but after realizing that I will be unable to obtain a large enough number of positive samples, I have abandoned that in favor of anomaly detection.

I estimate that if I captured a photo every second of the day, I would end up with maybe five photos containing cats (out of about 60,000 with sunlight) per day.

Is this feasible using anomaly detection? If so, what features would you suggest? My ideas so far would be to simply count the number of pixels with that has certain colors; do some kind of blob detection/image segmenting (which I do not know how do to, and would thus like to avoid) and perform the same color analysis on them.

You could simplify your problem significantly by using a motion/change detection approach. For example, you could compare each image/frame with one from an early time (e.g., a minute earlier), then only consider pixels that have changed since the earlier time. You could then extract the rectangular region of change and use that as the basis for your classification or anomaly detection.

Taking this type of approach can significantly simplify your classifier and reduce your false target rate because you can ignore anything that is not roughly the size of a cat (e.g., a person or bird). You would then use the extracted change regions that were not filtered out to form the training set for your classifier (or anomaly detector).

Just be sure to get your false target rate sufficiently low before mounting a laser turret to your feline intrusion detection system.


What does the term 'Facet' in Seaborn FacetGrid imply? I am a newbie to data science. I have a very basic understanding of Seaborn. My understanding is that it is used to plot grids. That said, I intend to have a better understanding of 'FacetGrid' and the term 'facet'.

  FacetGrid is a multi-plot grid for plotting conditional relationships.
  FacetGrid object takes a DataFrame as input and the names of the
  variables that will form the row, column, or hue dimensions of the
  grid. The variables should be categorical and the data at each level
  of the variable will be used for a facet along that axis.

The confusion is in the line 'the data at each level of the variable will be used for a facet along that axis'. Here, what does it mean by facet?

I am adding a code example where FacetGrid is used(Here FacetGrid is used to plot a 2D scatter plot):

sns.set_style('whitegrid')
sns.FacetGrid(haberman, hue ='Survival Status', size = 8) \
.map(plt.scatter, 'Age of Patient', 'Positive Axillary Nodes') \
.add_legend()
plt.show()

Can someone give a better explanation for a better understanding?

According to the facetGrid documentation :

This class maps a dataset onto multiple axes arrayed in a grid of rows and columns that correspond to levels of variables in the dataset. The plots it produces are often called “lattice”, “trellis”, or “small-multiple” graphics.

Here levels mean  « categories ». The main goal is to not display the full set of data, but multiple subsets of data that fit conditions on categorical variables.
Definition of facet found here :

facet : a particular aspect or feature of something.

In the following example, the vertical axis (column of graphs) represents the facet/feature/attribute Smoker/Not-Smoker and the horizontal axis (line of graphs) represents the attribute Time. So each graph is the conditional AND between the two attributes :

Smoker AND Lunch
Smoker AND Dinner
Not Smoker AND Lunch
Not Smoker AND Dinner

(source: pydata.org)


what is the complexity of a bidirectional recurrent neural network? In particular, what is the complexity of a bi-directional recurrent neural network taking into account the variants of LSTM and GRU as well for training?
I am hoping if I can get links to some additional research papers which talk or have mentioned the computational complexity of these methods in their works. I have been searching, but haven't come across anything meaningful till now

The computational complexity of simple single-layer recurrent networks, either vanilla RNNs, LSTMs or GRUs is linear with the length of the input sequence, both at training time and inference time, so $O(n)$, where $n$ is the length of the input sequence. This is because in order to get the last time step output, you need to compute all the previous ones.
This is assuming that there is a single output. If there are multiple output time steps, then it is linear on the sum of both input and output lengths.
Take into account that, inside LSTMs and GRUS there are internal steps that account for a multiplication by a constant in the complexity.
You can complicate the network architecture in many different ways (more layers, skip connections, etc), and this can affect its computational complexity. Here you can find an in-depth study of the computational complexity of different architectural variations.


Extract Information from PDF using DL We are having this requirement of extracting information from a credit history document. Usually it is a PDF and a computer generated document.
Because these PDFs are generated by different sources, the layout of the document will be different for each source. The columnn header labels will also be different.
Presently, there are 4 sources which are generating this document, but going forward, it will be from many sources. From each of these documents, we will need to extract information such as lender name, lending amount, outstanding balance etc;

I need to know what are the steps and practical approach involved in extracting the data I want such as lender name, amount, balance etc;

Do we have an established Machine Learning / Deep Learning approach that can be implemented here? Just getting to know the basics of ML/DL, therefore need a direction please

The task is doable, but time consuming and not easy. This is how I would plan the work:

Write a PDF scraper that navigates the document and converts all the informations it contains into some standardized textual format.
Label the words/elements of this text that correspond to the informations that you are looking for ("lender name", "lending amount", "outstanding balance", etc.).
Run some NLP model, such as an RNN classifier or CRF, to extract information from text.

Steps 1 and 2 are very time consuming (2 more than 1), but it's certainly doable. It will be a lot of work, especially labeling observations to create the Training set, but a nice thing to put on your CV.


Dropout on inputs instead on outputs = DropConnect? Is dropping out parts of the Input vector better than dropping out parts of the Output vector?

The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring pieces of the input means some of the further neurons will be able to see this neuron.

Is masking the input pretty much the same as masking the weights (aka DropConnect) and thus gives a higher quality regularization?

It depends on the type of input pattern but to make a decision, I suggest not to. There are different reasons for that. First of all, you are damaging your input signal. I don't know whether you are familiar with the information theory or not but the signal to noise ratio will be too small and if you do so, you will be left with a signal which is far from your real signal.

Moreover, it is also not good to add dropout in the convolutional layers because they are feature extractors and they are significant features for classification problems. If you miss them, means that you are losing information more than usual. Consider the point that your input to the network is already resized to a smaller shape than its original shape, for instance, the input shape of typical CNNs is 224 * 224 while the original shape may be ten times bigger or even more  for each direction.

You may have seen that in the Lenet-5 the authors have used a data-augmentation technique that changes the colors of the inputs with different distributions. The point there is that they have not changed the locality of the input signal. Moreover, the signal to noise ratio also is not a too small number due to the fact that they have not set the input features to zeros. They just have changed them slightly.

Finally, the last layer should not employ drop-out. Because the output size has to have specified characteristics, sum to one. Dense layers due to having a large number of weights, and consequently a large number of activations, are good points for exploiting drop-out.


How is complex number handled in NN? I am currently working with a dataset with complex numbers, and wanted to know how neural network handles situation as such.

What kind of activation should one use, what form of normalisation would sense and so on.

The theory of backpropagation should work with complex numbers, it doesn't make any assumptions about all numbers being real. However, most NN libraries would not be able to calculate with them. Your weights and biases would also need to be complex. I have no idea if you could reasonably expect convergence using gradient descent - many activation functions have periodic output and numerically unstable regions when calculating with complex numbers, which makes me think it would be risky.

The simplest thing to do would be to split each complex number up into its components at the input, representing as a 2-dimensional vector of real numbers - either using cartesian or polar approach. Neural networks should cope just fine with splitting features up like this.


What's the Seminal Book on data science? Can people provide the best textbook they have used for learning data science?  I know this is somewhat difficult because you have no basis of comparison but if you can just give your opinion on a book and why, I'd greatly appreciate it.

As already pointed out by the first comment, data science is a very broad field and not sharply defined.
But I think since you like to get started with data science and you're looking for resources, I share some of my recommendations with you (in no particular order).

Very applied, less academical, with lots of code examples: Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron
The Deep learning bible, more academical, great overview: Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville
A classic for Machine learning, covers a lot of topics, very educational with great explanations: Pattern Recognition and Machine Learning by Christopher M. Bishop
If you're more into statistics: An Introduction to Statistical Learning: with Applications in R by James, Witten, Hastie & Tibshirani
Machine Learning: A Probabilistic Perspective by Kevin P. Murphy
The classic RL reference: Reinforcement Learning by Richard S. Sutton & Andrew G. Barto

This is only a partial list focusing more on theory and classic data science algorithms. Keep in mind there are many many more books, e.g. I didn't mentioned any resources for Big Data, Data Analytics, Data Mining etc. whatever one would throw into the data science bucket. Further it's possible that this list might be outdated in a few years.


Extracting specific portions of text from poorly formatted document? I have a corpus of text files (free books) which are poorly formatted.  The goal is to extract a particular chapter (say chapter 2) from the raw text with all weird formatting removed.  Some documents are literal copy and pastes from pdfs, so that the text contents from a chapter would have page numbers interspersed throughout, etc., and I.e., it would be impossible to create a good regex rule.

I've manually gone through a number of documents and stripped out all unwanted formatting to arrive at the desired, clean section of text.  Using this as the training set, what would be a good model to set up for this task?

To clarify, the input would be a (very long) string, and the output would also be a string.

EDIT:

Please note that this particular example can be very easily handled by a series of regex rules, but it's only for illustration purposes only.  Imagine if each document had their own minor idiosyncrasies making it very difficult to come up with a general series of regex rules to apply to all docs in the corpus.

Sample input:

Book Title ABC

Table of Contents

Preface ................................ Pg 1
Chapter 1 .............................. Pg 2
Chapter 2 .............................. Pg 3
Chapter 3 .............................. Pg 4

Preface

<p> Contents for the preface sit here.  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse scelerisque turpis ac commodo porttitor. Suspendisse iaculis lacinia ante, non suscipit purus venenatis et. </p>

- 1 -

Leftover Contents from Preface

Chapter 1

Praesent at justo sit amet nulla porttitor hendrerit. Etiam pretium sem ac feugiat posuere. Nulla mi risus, mollis sit amet ligula egestas, lacinia dapibus tellus. Proin iaculis ligula ultricies nunc elementum maximus. Nulla accumsan libero at nunc scelerisque, ut fermentum lacus euismod. Nulla imperdiet pharetra laoreet. Quisque id mollis diam. Vestibulum interdum orci at lectus ullamcorper pharetra. Curabitur laoreet mollis pharetra.

- 2 -

Chapter 2

Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.

- 3 -

Leftover contents from Chapter 2.  Vivamus pretium mauris at metus egestas, eget luctus ligula auctor.

Chapter 3

Maecenas vel dapibus lorem. Sed eget justo sit amet libero aliquam maximus a quis augue. Nunc consequat, urna quis elementum condimentum, quam arcu consectetur tellus, vehicula gravida est purus id nisl.

- 4 -

Nulla eget molestie velit, pharetra euismod magna. Vestibulum ultricies justo vitae massa vulputate, et venenatis orci fringilla.

Index

Sample Output:

Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.
Leftover contents from Chapter 2.  Vivamus pretium mauris at metus egestas, eget luctus ligula auctor.

If I understand the problem, you have text that is pre-sliced into some unit of interest (your chapter), that requires pre-processing to convert into a form suitable for further work. You've done some of this pre-processing by hand and want to use it as a training set for a model to infer the rules that should be applied to accomplish the same result on raw text.

Because I don't know what you intend to do with the scrubbed text, I'm going to assume that it's destined for natural language processing to extract some semantic content. Here's what you'd normally do.

Convert unicode encoding to UTF8 to get rid of the diacritical marks and other characters that aren't part of your target language
Tokenize the text into a list of strings of words and punctuation marks
Strip (or optionally convert to words if short) numbers
Strip excess white space
Optionally strip stop words (such as the, of, them)
Lemminize (remove inflections, such as by changing plurals to singular)

This gets you to the point where you're ready to do linguistic analysis (which, as I say, I only assume is your goal).

Fortunately, there are packages, such as NLTK in Python that do this for you without the need for regex.

Writing a model to infer that from a training set of clean text might not be the best way to go, if getting to clean is your goal, rather than extending the frontiers of machine learning.

Could you post a MWE (Minimal Working Example) of a chunk of raw text and it's corresponding manually scrubbed version. It may turn out to be a quicker means to your end to apply a combination of off-the-shelf tools to get to where you want to be.


What is done first, cross validation or grid search? When I have the data set to train a model with SVM, which procedure is performed first, cross validation or grid search? I have read this in a couple of books but I don't know in what order all this should be done. If cross-validation is first performed, what hyperparameters do I use there if I have not found the optimal values provided by the grid search? In addition, throughout this procedure, where should the confusion matrix be calculated?

Thanks in advance

Well, grid search involves finding best hyperparameters. Best according to what data set? a held out validation set. If that's what you mean by cross validation, then they necessarily happen simultaneously.

It doesn't really make sense to do something called cross validation before testing hyperparams - indeed, what would you be evaluating?

CV as in k-fold cross validation can also happen within each model fitting process in the search, to produce a better estimate of the loss (and its variance, which is useful in more sophisticated tuning procedures). I think this is less usual but valid.

It's possible to use CV when fitting the final model after hyperparameter search. It might give you a better estimate of the loss, or confusion matrix, as you compute many of them. But each model you fit isn't using all available data. I think it's probably more conventional to take the best model's parameters and loss / confusion matrix, from the fitting process, as an estimate of generalization, and then refit the final model on all data. This means no CV at that stage.


Which machine learning model should I learn for this problem? I'm working in python. Would like to practice some machine learning, and I've always been curious about an analog to the problem below...

A collection of 3 letters are drawn randomly from the 26 letters of the alphabet. None, any, or all of the letters can be discarded and replaced with an equal number of not-yet-drawn letters. This discarding and replacing happens 3 times (or more generally n times). After all the replacements have taken place a point value is awarded to the resulting 3-letter-word. Some words have high point values, and others less so. I'm interested in the optimum replacement strategy. For example, say only the words "FOX" and "THE" reward any value. All other 3 letter combos are worthless. I want the machine to learn the correct replacement strategy while holding "FOW" with one replacement remaining. In this simple case, the strategy is to replace the W only, and attempt to draw the X. This strategy is superior to replacing all 3 letters in an attempt to make "THE," since drawing "THE" only gets points 1/('count of remaining letters' choose 3) times, whereas replacing the W gets points 1/('count of remaining letters'); the instance where an X is drawn.

Can anyone point me in the right direction?

You should use a Markov reward model to model your problem. All the possible words are the different states of your chain. The replacement process corresponds to the transitions of your Markov chain.

After defining all the properties of your chain (states, transitions, rewards, ...), you can train your model and get the best strategy for each current word.