Classify sentences containing typos into groups My data is a list of sentences, where each sentence contains between 1 and 4 words. These sentences are typed in manually so some of them contain typos and some additional words such as GmbH, GER etc.

However, I do know the set of valid sentences. As an example we assume this valid set is given by {Hello human, Horse, Hello bird} and the data (where some sentences contain typos and extra words) is given by

Hello human
Horse
Hello human GmbH
Hello human GmbH, GER
Horse GmbH
Horse
Hello humn
Hell humn
Hello human
Hello bird

I would like to give each sentence above an ID 1, 2 or 3 where 1 is for Hello world, 2 is Horse and 3 is Hello bird. But due to the typos and extra words such as GmbH, GER I cannot make a simple comparison between strings.

Is there a numerical technique within NLP or a related field that I can use to achieve this task?

If I understand correctly, you're looking for string similarity. There are several techniques available, the most simple is "edit distance" (aka levenshtein distance), which is the count of the minimum insertion/deletion/substitution/transposition operations needed to get from one string to the other.

For your particular task, I suspect "jaro-winkler similarity" would be better. JW is similar to ED, but was specifically designed for "entity resolution" (i.e. "record linkage"), which it looks like is what you're trying to accomplish. You can see a short demonstration of how this would work here


Is ARIMA appropriate for time series prediction involving a mix of explanatory and independent variables? I have a table with the following columns:

Date(Month,Year), Sold_Past_Month, Quantity_Available, Quantity_Shipping_In, Missed_Sales, Quantity_Needed

Quantity_Needed is the dependent variable that is numerical.

I want to predict Quantity_Needed and train a model using the columns mentioned above; however, I also want Quantity_Needed to train the model.

I have data from the past 5 years -> so I would have roughly 60 rows of data for one item.

Is it possible to use ARIMA for this?

If it is, what should I do next to build my model?

The correct name of what you are looking for is: ARIMAX, the ARIMA model works without covariables, the X (exogenous variables) include the possibility of having other explanatory variables.


Can linear regression or any other models be used to judge if Y increases as X increases? I have a database like:

Site    X    Y
S1      1    1.5
S1      1    1.3
S1      2    1.7
S1      1    1.1
S1      4    5.9
S2      3    4.0
S2      2    2.5
S2      4    9.1
S2      4    9.2
S2      1    2.3

I need to find if $Y$ increases as $X$ increases for every site. In other words, Bigger $X$ corresponds to bigger $Y$.

I know linear regression might suit this problem. But please take a look at the following graph:

Figure 1 is not what I want because small $X$ corresponds to big $Y$. However, Figure 2 is what I want. When I use a linear regression model and RMSE as a measure, it cannot tell the difference between Figure 1 and Figure 2.

Another thing about my database is: $X$ are like levels, which are the same to all the sites. But the $Y\text{'s}$ of every sites are different. For example, for one site, $X=1$ and $Y=20$ means $20$ is a small value because it corresponds the lowest level of $X$. But for another site, $Y=15$ and $X=6$ means $15$ is the highest value because $X$ is the highest level.

So, my problem is: for every site, I need to use a linear model or any other algorithms to judge if $Y$ increases as $X$ does. Then, I need to use a measure to select some sites.

I don’t think linear model is a good idea for your problem, because it can capture only linear pattern. Furthermore, if I understand you correctly the results for different sites will not be comparable due to different scales/ slopes.

Instead I suggest using the Spearman rank correlation coefficient.


How do you evaluate ML model already deployed in production? so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production.

But because people change, data and many other factors change as well. And performance of our model eventually will decrease. So then it needs to be replaced with the new, better model.

What are the common techniques, model stability tests, model performance tests, metrics after deployment? How to decide when to replace current model with newer one?

After the models are deployed in production, I'd monitor the following:

(1) The same metric you used to evaluate the performance of your model, in some cases it is accuracy, or it could be precision, recall, RMSE. I'd plot a daily time series charting the metric and see that it is still performing above a satisfactory threshold. There might be seasonality within the calender, the model performs well around certain months and not so well in other months. I'd compare the performance against the test/validation sets of the same months to account for seasonality.

(2) Apart from looking at the performance of the model, especially if one is working with shared computing resources, I'd also recommend keeping a close tab on the data aggregation runtime, model runtime, success rates of models runs over the past period.


Is this a good practice of feature engineering? I have a practical question about feature engineering... say I want to predict house prices by using logistic regression and used a bunch of features including zip code. Then by checking the feature importance, I realize zip is a pretty good feature, so I decided to add some more features based on zip - for example, I go to census bureau and get the average income, population, number of schools, and number of hospitals of each zip. With these four new features, I find the model performances better now. So I add even more zip-related features... And this cycle goes on and on. Eventually the model will be dominated by these zip-related features, right?

My questions:

Does it make sense doing these in the first place?
If yes, how do I know when is a good time to stop this cycle?
If not, why not?

1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is amenable to learning, but you can always construct features from it that are. The feature you are adding are based on one feature. This is common. However, your learner, logistic regression, is sensitive to multi-collinearity. You need to be careful on which feature and how many features to add. Otherwise, you model may overfit.

2) Since you are using a logistic regression, you can always use AIC or perform a statistical significance test, like chi-square test (testing the goodness of fit) before adding new structure, to decide whether the distribution of the response really is different with and without this structure. This is particularly useful when your data is scarce. Another way is to add penalty term to your model. For example, logistic lasso regression.

3) Keep adding new features is not always a good idea. Be careful with the curse of high-dimensionality. When adding new feature, you are actually adding a new dimension on your data.  Naively, one might think that gathering
more features never hurts, since at worst they provide
no new information about the class. But in fact their benefits
may be outweighed by the curse of dimensionality. I hope Useful things to know about machine learning session6 could help.


Linear regression on probabilistic data I have a dataset which is probabilistic in nature. For instance, if a record has d features, then values of $x_1$, ..., $x_d$ are such that their sum adds to 1. In other words, for each $x_i$ data record (row)

$$P(x_{i,1}) + P(x_{i,2}) + ...+ P(x_{i,d}) = 1$$

I want to apply Linear Regression Model to predict a variable $y$.

My questions are:

can Linear Regression work for such datasets? What are other models I can use?
Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).
As I use Linear Regression all independent variables
($x_{i,0}$, $x_{i,d}$)'s are < 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $\mathbf{w}$ for
$\hat{y}_i = \mathbf{w} \cdot \mathbf{x}_i$ calculated would be very large)?

Ideally, I am applying gradient descent for a convex optimization problem. It can be that $x_0...x_d$ belong to some unknown distribution and in my case, I know that distribution. But if there is an alternate model them pls suggest.

The issue with linear regression in your case is not whether it can be used - it can nearly always be used to build a predictive regression model - it is whether the resulting model is useful, usually whether it is accurate enough for the intended use.

The accuracy and utility of your model does not depend only on the nature of $\mathbf{x}$. Instead it depends on the true nature of the $\mathbf{x} \rightarrow y$ relationship. Your measurements and model are an approximation of that relationship. Linear models are fast and stable to compute, but can be limited if the true relationship being approximated is non-linear.

Here are some basic thoughts/feedback on your questions:

  Can Linear Regression work for such datasets?

Yes it can work. Will it be good enough? You will know after testing.

  What are other models I can use?

Almost any regression model class could be applied to this problem. For a quick comparison, to see if a non-linear model will make more accurate predictions for you, then you could try an easy-to-apply model from an existing library, such as XGBoost (which has a stand-alone command-line version).

  Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).

For a linear model, your intuition is correct that one feature is redundant, as it is itself a linear combination of the other features, so there is no impact to removing it. You could remove any single column. You should not need to though, and some model classes might work better if left in. As always, experiment if you are not sure, and use hold-out test data to verify your ideas (or even better a separate cross-validation set before final test stage)

  As I use Linear Regression all independent variables ($x_{i,0}$, $x_{i,d}$)'s are < 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $W$ for $\hat{y}_i = W\mathbf{x}_i$ calculated would be very large)?

For linear regression this scaling is not very important. However, there are a few caveats:

If you measure success by an error metric (such as mean square error), then to compare different ideas fairly, you should report the test error in the same way - e.g. you should scale back the $log(y)$-based predictions and calculate error on the same $(y - \hat{y})^2$ each time.
For some model classes, they could work better if the target variable is within a certain range. You should check documentation for this.
How useful the mapping is depends on the true nature of the relationship between $\mathbf{x}$ and $y$. Specifically for a linear model, you have completely changed the line you are trying to fit - it could as a result be a lot better or lot worse, but that is not dependent just on $\mathbf{x}$. So again you will have to give it a try and test the results.


Determining which entries in input vector have the greatest effect on output vector I'm looking for a solution in Python. I'm doing a project where I've built a densely packed neural network which takes a 48-dimensional input vector and the last layer is a 2-dimensional linear output. I'm looking for an algorithm that can determine how much each scalar in the input vector contributes to the output vector.

Specifically, my input vector is 24 consecutive pairs of position and momentum. For example the input would be $(x_1,v_1,x_2,v_2,...,x_{24},v_{24})$ and my output is the next position and momentum pair $(x_{25},v_{25})$. I've hypothesized that the more recent (higher index) pair should impact the output more. Is there any technique that I could use to quantitatively verify this?

I'm not very experienced in the interpretation of neural networks but I have two suggestions.
1: Alter the values of the input, one feature at a time, and watch how the output varies.
2: Use another model with more interpretability. You could do this just for interpretation.


Are more classes more favorable than a single combined class? Imagine the following scenario.

Train a classifier that classifies an object into one of these n+m classes:

class 1..n     -> triggers an action A depending on the predicted class p
class n+1..n+m -> triggers an action X independent on the prediction

Used is the CalibratedClassifierCV of a LinearSVC of scikit-learn. So, the prediction is based on the maximum predict_proba value.

Question

Because action X is independent on the prediction, I would very like to combine the m classes (n+1, n+2, ..., n+m) into a single class (to improve training speed, classification speed, memory consumption etc.)

Would that harm the classification result?

So what you are interested in (in terms of prediction) is $n+1$ classes, where the last class is $m=1$. This would be my default model. In other words, I don‘t see why you should have $m$ additional outcomes if these outcomes are irrelevant for your task.

For the quality of a prediction, features $X$ matter the most. I‘m not aware that the classes are relevant unlike your outcome (=classes) have an order.

However, ultimately you need to test which model performs best. So by the parsimony principle, start with a simple model and gradually see if adding complexity delivers a better result (by a predefined measure such as accuracy).


Predict_proba for Binary classifier in Tensorflow I'm working in binary classifier problem, where I have used Tensorflow low level API's.
Last layer wrapped with Sigmoidal Function and just returning a  single value.

For my prediction I just set a standard threshold value of 0.5 and hence if it's >0.5, I'm predicting as 1, otherwise 0.

I'm also looking for a confidence level of each prediction. Is it possible to get the same with this architecture, or does it need some modification?

Since a neural net that ends with a sigmoid activation outputs probabilities, you can take the output of the network as is.

If you're referring to scikit-learn's predict_proba, it is equivalent to taking the sigmoid-activated output of the model in tensorflow. In fact that's exactly what scikit-learn does.

E.g. if tour model outputs $0.8$ for class $1$, you would classify this as $1$ (since $0.8 > 0.5$), with a probability of $0.8$. S


gradient descent diverges extremely I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value.

I have done exactly like in the manual and for some reason my predictor coefficients are going to infinity, even though it worked for another case.

Why, in this case, can it not predict a simple 1.4 value?

clear all;
n=10000;
t=1.4;
sigma_R = t*0.001;
min_value_t = t-sigma_R;
max_value_t = t+sigma_R;
y_data = min_value_t + (max_value_t - min_value_t) * rand(n,1);
x_data=[1:10000]';

m=0
c=0
L=0.0001
epochs=1000 %iterations

for i=1:epochs
   y_pred=m.*x_data+c;
   D_m=(-2/n)*sum(x_data.*(y_data-y_pred));
   D_c=(-2/n)*sum((y_data-y_pred));
   m=m-L*D_m;
   c=c-L*D_c;
end
plot(x_data,y_data,'.')
hold on;
grid;
plot(x_data,y_pred)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%

question update:
Hello , I have tried to write down your code in the Matlab language i am more femiliar.
my feature matrices of the form NX2 [1,X_data] is called Xmat.
i followed every step in converting the code, and i get in both Theta NAN.
Where did i go wrong?

$
%%start Matlab code
n=1000;
t=1.4;
sigma_R = t*0.001;
min_value_t = t-sigma_R;
max_value_t = t+sigma_R;
y_data = min_value_t + (max_value_t - min_value_t) * rand(n,1);
x_data=[1:1000];
L=0.0001; %learning rate
%plot(x_data,y_data);
itter=1000;

theta_0=0;
theta_1=0;
theta=[theta_0;theta_1];

itter=1000;
for i=1:itter
onss=ones(1,1000);
x_mat=[onss;x_data]';
pred=x_mat*theta;
residuals = (pred-y_data);
  for k=1:2 %start theta loop
  partial=2*dot(residuals,x_mat(:,k));
  theta(k)=theta(k)-L*partial;
  end%end theta loop
end % end itteration loop
%%end matlab code
$

Although this answer shows code with Python, the theory is exactly the same.
Two advices when running a linear regression via gradient descent from scratch:

use matrix notation for your features data (i.e. input data) so you can apply it to n-dimensional datasets (not just 1-D as in this case)
standardize your feature matrix data (although in this case you have actually 1-D, but it is a good practice which always works)

Your data plotted with matplotlib:

import numpy as np
from numpy.random import randn, seed

x_data = 20 * randn(1000) + 100
y_data = x_data + (10 * randn(1000) + 50)
n=1000
t=1.4
sigma_R = t*0.001
min_value_t = t-sigma_R
max_value_t = t+sigma_R
y_data = min_value_t + (max_value_t - min_value_t) * np.random.rand(n,1)
x_data=np.array(range(1000))

pyplot.scatter(x_data, y_data)
pyplot.show()

Build your dataset to train the model:

final_df_DICT = {'X': x_data}
import pandas as pd
H = pd.DataFrame(final_df_DICT)
feature_matrix = np.zeros(n*2)
feature_matrix.shape = (n, 2)
feature_matrix[:,0] = 1
feature_matrix[:,1] = H['X']
#standardize features
feature_matrix = (feature_matrix - feature_matrix.mean()) / feature_matrix.std()
target_data = y_data.reshape(len(y_data), )

Look at the feature matrix, having the first column filled with 1's and the second column having the actual x_data:

Model training with your hiperparameters:

w = [0, 0]
L=0.0001
epochs=1000
iteration = 0
cost=[]
while iteration < epochs:
    pred = np.dot(feature_matrix, w)
    residuals = pred-target_data
    #we calculate the gradient for the 2 coeffs with the scalar product
    for i in range(len(w)):
        partial = 2*np.dot(residuals, feature_matrix[:, i])
        w[i] = w[i] - L*partial

    iteration += 1
    computed_cost = np.sum(np.power((pred - target_data), 2)) / n

    cost.append(computed_cost)

print('coef: {}'.format(w))
print('cost: {}'.format(cost[-1]))

Result:

  coef: [-1.80963253e+00 -6.15189807e-06] cost: 6.466287828899486e-07

Let's plot the fitted regression model predictions over the original dataset (we trained on the whole dataset, not taking into account any validation set in this case...):

my_predictions = np.dot(feature_matrix, w)
pyplot.scatter(feature_matrix[:, 1], target_data)
pyplot.scatter(feature_matrix[:, 1], my_predictions, color='r')

pyplot.show()


Sentence similarity using Doc2vec I have a list of 50k sentences  such as : 'bone is making noise', 'nose is leaking' ,'eyelid is down' etc..

I'm trying to use Doc2Vec to find the most similar sentence from the 50k given a new sentence.

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(data)]

max_epochs = 100
vec_size = 20
alpha = 0.025

model = Doc2Vec(size=vec_size,
                alpha=alpha,
                min_alpha=0.025,
                min_count=1,
                dm =0)

model.build_vocab(tagged_data)

for epoch in range(max_epochs):
    print('iteration {0}'.format(epoch))
    model.train(tagged_data,
                total_examples=model.corpus_count,
                epochs=model.iter)
    # decrease the learning rate
    model.alpha -= 0.0002
    # fix the learning rate, no decay
    model.min_alpha = model.alpha

test_data = word_tokenize("The nose is leaking blood after head injury".lower())
v1 = model.infer_vector(test_data)
#print("V1_infer", v1)

similar_doc = model.docvecs.most_similar(positive=[model.infer_vector(test_data)],topn=3)

for i in range(0,len(similar_doc)):
    print(tagged_data[int(similar_doc[i][0])],similar_doc[i][1])

Such that for the sentence "The nose is leaking blood after head injury" i would like to get the sentence with the highest similarity score ( i guess that it will bring sentences with words like leak or even synonyms like dripping?) . But the sentence i get back are unrelated and change each iteration of model.infer_vector(test_data)

Any idea about what is wrong?

Doc2Vec (and words vectors) need significant amount of data to learn useful vector representation.  50k sentences is not sufficient for this.   To overcome this, you can feed word vectors as initial weights in Embedding Layer of network.
For example, code from following question :
How to implement LSTM using Doc2Vec vectors?
model_doc2vec = Sequential()
model_doc2vec.add(Embedding(voacabulary_dim, 100, input_length=longest_document, weights=[training_weights], trainable=False))
model_doc2vec.add(LSTM(units=10, dropout=0.25, recurrent_dropout=0.25, return_sequences=True))
model_doc2vec.add(Flatten())
model_doc2vec.add(Dense(3, activation='softmax'))
model_doc2vec.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Output of "Flatten" layer will be vector representation of a sentence / document.
Article with example code.


How to Keep Missing Values in Ordinal Logistic Regression I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars).

One of my predictor variables is also ordinal but there are some missing values where the viewer skipped a question because it wasn’t applicable due to skip logic from a prior question or because they missed it.
What’s the best way to indicate a value is “missing” and/or “not applicable” while also retaining the ordinal nature of this predictor variable for everyone else? I don’t think I should delete this viewer or try to impute the value.

I get an error if I leave the NaN. I thought about dummy coding so I have something like question5_never, question5_sometimes, question5_always, question5_na, question5_missing, but I not sure.

Computationally, there is just no way to use the data with the values missing. You do still have a whole three options, though:

Drop the rows. This is "listwise deletion." Like you said, you don't want to do that - and it can cause biased results, as the observations with missing values may be ones that speak against some theory, or (in more general terms) represent some consistent data-generating pattern.
Drop the column. This will eliminate any bias due to the above, at the cost of inefficiency. It may also be a non-starter if the effect of this variable is what you're concerned with anyway.
Impute. You can fill in the missing values with your best guesses (i.e. the maximum-likelihood point-predictions of an imputation model - or even just the "null model", i.e. the global column means) and treat them like real values, though this is bad practice because it doesn't reflect the inherent uncertainty about imputed vs. actually observed values. Standard practice is to ensure your imputation model produces a best-fit distribution for each observation, generate 5 versions of your dataset with 5 sets of random draws from those observation-specific distributions, fit the same model to each of those datasets, and average the results.

Note, though, that if the model is an arbitrarily complicated machine-learning algorithm (i.e. a universal approximator like a neural net), then you can just fill in the missing values in each column with any "free" value - one that's not actually present in the data, and won't be - and the model will learn to treat those values not just differently but appropriately. It will effectively do its own imputation, in fact. But still, you can't have the actual inputs as missing values.


Is there a network analysis tool built into Orange? I am interested in a canned or built-in network analysis tool.  Wondering if this is possible with Orange.

This Networks add-on for Orange data mining suite should help you. If you are open to using other solutions, I would recommend networkx Python library.


how to choose classifer is the best way to create the most accurate classifier to train a bunch of classifying algorithms like ANN, SVM, KNN, etc, and test different parameters to get optimal parameters for each classifier, and see which classifier has the least testing error?

Or is it better to use ensemble method and choose the "majority" decision of different kinds of trained classifiers?

It's usually not that clear cut; there's typically not one universally best approach.

Having said that, there are some prototyped ensemble approaches that are supposed to always be better than their underlying component algorithms, notably Erin LeDell's binary ensemble classifier for H2O. However, even in those cases you still need to optimize the first stage algorithms for the ensemble to be universally better.

Thus if you're willing to spend a lot of extra time, let's say 2 weeks for an ensemble instead of the 1 week it might take you for your single-stage algorithm, then it's possible (especially for binary classification) to find an ensemble that will definitely be better than your single-stage classifier.

However this is rarely the case and the way you framed the question implies there's a choice between

building 1 really good single-stage model, selected from many candidate models (and by the way be sure to avoid overfitting while making those selections)  and
throwing an ensemble at the problem without completing #1 above for each component of the ensemble (or completing #1 but not also optimizing the 2nd stage of the ensemble)

If that's the decision then -- while there's no 1 universally right answer -- I'd say that in the vast majority of the cases it's better to stick with #1.


Sequence forecasting in keras not possible for variable-length sequence forecasts I'm using keras for multiple-step ahead time series forecasting of a univariate time series of type float. Judging by the results I got, the approach works
works perfectly well. There is, however, an important detail in the training process that baffles me:

keras requires the sequence length of the input sequences (X matrix) to be equal to the forecasting horizon (y matrix). That means, for example,
that keras needs input sequences of length 20 in order to forecast the next 20 time steps. My goal is to be able to forecast as many time steps as I specify, given the last 20 time steps. With the below code, this is not possible:

    import numpy as np
    from keras.layers.core import Dense, Activation, Dropout, TimeDistributedDense
    from keras.layers.recurrent import LSTM
    from keras.models import Sequential
    from keras.optimizers import RMSprop
    np.random.seed(1234)

    model = Sequential()
    layers = [1, 20, 40, 1]

    model.add(LSTM(
        input_dim=layers[0],
        output_dim=layers[1],
        return_sequences=True))
    model.add(Dropout(0.3))

    model.add(LSTM(
        layers[2],
        return_sequences=True))
    model.add(Dropout(0.3))

    model.add(TimeDistributedDense(
        output_dim=layers[3]))
    model.add(Activation("linear"))

    rms = RMSprop(lr=0.001)
    model.compile(loss="mse", optimizer=rms)
    model.fit(
        X_train, y_train,
        batch_size=512, nb_epoch=100, validation_split=0.1)

I get a dimension error whenever the sequence lengths of the sequences in X_train and in y_train differ from one another.

What am I doing wrong and how can I fix it? Why are my results still pretty good?

The question is related to this.

No, it is not possible with Vanilla LSTM. It's not a Kera problem, but the fundamental structure of Vanilla LSTM.

I think you have a bit of misunderstanding of how LSTM works. You are assuming given $x_0$...$ x_{19}$ the LSTM is predicting $x_{20}$...$x_{39}$. This is NOT the case. In LSTM if your input sequence is $x_0$...$ x_{19}$ then your output sequence is $x_1$...$ x_{20}$.

To achieve what you want, you can use something like Seq2Seq model.


LinkedIn web scraping I recently discovered a new R package for connecting to the LinkedIn API. Unfortunately the LinkedIn API seems pretty limited to begin with; for example, you can only get basic data on companies, and this is detached from data on individuals. I'd like to get data on all employees of a given company, which you can do manually on the site but is not possible through the API.

import.io would be perfect if it recognised the LinkedIn pagination (see end of page).

Does anyone know any web scraping tools or techniques applicable to the current format of the LinkedIn site, or ways of bending the API to carry out more flexible analysis? Preferably in R or web based, but certainly open to other approaches.

Beautiful Soup is specifically designed for web crawling and scraping, but is written for python and not R


Inputting (a lot of )data into a dataframe one row at a time I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this:

     pixels
0    [[23,4,54...],[54,6,7...],[........]]
1    [[65,54,255,55,...],[43,54,6...],[......]]
...
...
...
7000 [[........]]

For each row in the Series, I want to take these 2D arrays, flatten them to 1D, take the values and assign them to the columns of one row in data frame. Each row will have 30x30 = 900pixels each, storing the values of each pixel. Like this:

    pixel1    pixel2    pixel3...    pixel900
0       23         4        54             77
1       65        54       255             33
...
...
...

I'm using an elaborate function that extracts one row from the series at a time, flattens the array, converts it to a Series again, and appends it to a dataframe. It takes sooo long. I'm sure there must be a faster way. I'm using this code:

def prep_pixels(X):
    # X is a series
    df = pd.DataFrame()
    for i in range(len(X.index)): #iterate through the whole series
        df = df.append(pd.Series(X[i].flatten()), ignore_index=True)
    return df

EDIT:
Upon request from a user, I will provide code with how I ended up in this rut in the first place :D

#reading the files
filepath = 'dataset.pickle'
data_np = pd.read_pickle(filepath)
print(data_np[0])

Output:

 [array([[255, 248, 253, 255, 251, 253, 254, 236, 220, 217, 191, 145, 139,
        185, 216, 227, 252, 251, 254, 248, 251, 236, 221, 222, 213, 175,
        120,  75,  74, 209],
       [255, 253, 254, 253, 252, 254, 223, 146,  87,  75,  58,  30,  27,
         58,  86, 116, 157, 168, 164, 165, 167, 136,  96,  71,  59,  49,
         21,   9,  27, 144],
       [255, 255, 255, 248, 252, 255, 202,  88,  15,  16,  14,  11,  11,
         12,  12,  20,  40,  46,  38,  43,  40,  25,  21,  19,  17,  35,
         53,  58,  64, 124],
    ... 30 rows of 30 pixels
    ...
    ... last row coming up ...
    [255, 255, 254, 254, 253, 252, 253, 254, 255, 255, 254, 252, 249,
    249, 251, 213, 126, 178, 231, 252, 248, 250, 254, 254, 252, 253,
    255, 255, 255, 255]], dtype=uint8), 'क']

The last symbol in this list is the character that this image represents. It's the 'label'. It's supervised learning using CNNs. Anyway, I need them to be in the other format I described to be able to work with them. This is how I'm handling this data:

data = pd.DataFrame(data_np, columns=['pixels','labels'])
def prep_pixels(X):
    df = pd.DataFrame()
    for i in range(len(X.index)): #iterate through whole series
        df = df.append(pd.Series(X[i].ravel()), ignore_index=True)
    return df

X = prep_pixels(data['pixels'])
y = data['labels']

EDIT: a user suggested that I use a mutable datatype to do this procedure. They said that it might speed things up because the computation does not need to make copies of data. I used some nested for loops and it cut the time to half (1 min 22 sec instead of 3 min). I still feel like its pathetic, given that my dataset has just 7000, 30x30 pixel images. Or, maybe I'm just new to data wrangling.

Here is the code I used. Please let me know if you have any other suggestions:

filepath = 'dataset.pickle'
data_np = pd.read_pickle(filepath)

df = pd.DataFrame()
for row in range(IMG_ROW):
    for col in range(IMG_COL):
        pixel=[]
        for img in range(len(data_np)):
            pixel.append(data_np[img][0][row][col])
        columns = pd.Series(pixel, name=col)
        df = pd.concat([df, columns], ignore_index=True, axis=1)

I'm getting timeit results of about 1/4 of the time using:

flatX = X.apply(lambda x: x.flatten())
pd.DataFrame(item for item in flatX)

See also
https://stackoverflow.com/questions/45901018/convert-pandas-series-of-lists-to-dataframe
for some possibly better options for the second line.

(Regarding my earlier comment, I don't get any real savings by just dataframe-ing at the end.)


Python and data analytics how to get the unique number of orders per seller using groupby? I am totally new to data analytics and python.

I took a good course on Machine learning. But I didn't really worked with real life data.

I managed to get data from a store who wants to create a recommendation system. I convinced them, that I will learn and get experience, and you will get benefited with a recommendation system, plus some insights on your previous orders.

I've got a json file having the following structure (I transformed it into excel):

Each order_id is connected through one seller_id. And it may contain multiple product_id.

With that I mean, that a user could have orderd from the seller_id=1, 2 products and the data would be like this:

{
    order_id: 12,
    seller_id: 1,
    product_id: 3,
    ...
},
{
    order_id: 12,
    seller_id: 1,
    product_id: 89,
    ...
}

The main things I need to export from these data is the following:

Number of unique orders per seller;
Products that are being ordered the most per seller;
Products that are being ordered the most in general;
Sum of the quantity field for each product to check how many times has been order;
Number of orders for each customer;

And other things that I may come up with later.

I started with the number 1, wherre I need to get the unique number of orders per seller. I tried the following:

data.groupby('order_id')['seller_name'].value_counts().sort_values(ascending=False)

But I didn't get the right result. A seller having 110 unique orders, but using the line above, it shows he is having more than 500 orders.

Here is a snippet of the data available:

Even though, seller_id=1 having 4 rows, but the actual number of orders is uniquely 2, despite the fact that the quantities of items within these 2 orders are 7.

The same for seller_id=5, he is having 4 rows, but they are only for one single order.

So how can I get the unique number of orders of each seller as a start? I think I will figure the other points out once I get to do the first one.

Based on the information available above I think the solution below should work

# Number of unique orders per seller
a = pd.pivot_table(df, index = ['seller_id'], values = ['order_id'],
                   aggfunc = {'order_id' : pd.Series.nunique})

# Products that are being ordered the most per seller
# Products that are being ordered the most in general
b = pd.pivot_table(df, index = ['seller_id', 'name'], values = ['product_id'],
                   aggfunc = 'count')

# Sum of the quantity field for each product to check how many times has been order
c = pd.pivot_table(df, index = ['name'], values = ['product_id', 'quantity'],
                   aggfunc = {'product_id': 'count', 'quantity': 'sum'})

# Number of orders for each customer
d = pd.crosstab(df.customer_id, df.order_id)


How to reduce pipeline footprint in scikit-learn? I had a post on stackoverflow. As it is related to scikit-learn, I am hoping that I can obtain some assistance from data scientists in this forum. https://stackoverflow.com/questions/38640815/python-running-out-of-memory

In general, however, I would like to seek opinions on the reduction of memory use when one uses scikit-learn, as this can be a daily problem to deal with as a data scientist. Thank you very much.

I don't know if it is still an open question, anyway.

What is painful when using TFidf, it is the size of the ouput sparse matrix. II think it is your problem here. In order to pass this, you should limit the amount of word you want to treat. I didn't try with your case but you should try CountVectorizer before TfidfTransformer or directly TfidfVectorizer.
In CountVectorizer and TfidfVectorizer from sklearn, you can specify the minimum and maximum number of document any term can be found in (min_df and max_df).
This will reduce the size of the output A.

Hope it helps a bit


Desicision tree classification with a "false" attribute This is a pretty specific problem but I think it can help me understand better the whole concept of the subject.

A doctor in the hospital is in charge of 20 medical students. For every patient, the doctor consults with the students rather the patient has a problem. Each student can either say "yes", "no" or "Don't know".

The doctor builds a decision tree based on the answers of the students in period of two months. It is known that one of the students that answered the questions is an imposter, and all of his answers are based and opposite of another student the he picked. When the "real" student answered "yes", the imposter answered "no" and vice versa. When the "real" one answered "I don't know" so did the imposter.
Also it is known that every real student is giving a decisive answer - "yes" or "no" for at least 60% of the cases.

Tasks:

Give an algorithm to find the imposter.
A tree was built again without the imposter's answers. Prove it is the same as the old tree.

I understand that the information gain of the imposter and the real student he based his answers on is the same. So the algorithm needs to find groups of students with the same information gain. Within those groups, find two students with answered all reversed. The one who is wrong more than 60% of the time will be the imposter.

Regarding the second task I'm not sure why it is true. Thinking of a case of a patient where all the students answered "I don't know" except for one student who answered "yes" and the imposter who answered "no". Isn't it possible that the new tree will classify this patient as "yes" instead of "no", now when the imposter is gone?

Maybe I'm not interpreting the scenario correctly, but task 1 seems impossible and task 2 easier than you've made it sound.

I'm reading it as a sample for each patient, with one feature for each student, possible values yes/no/dunno.

In that case, at any split of the decision tree the imposter and the chosen real student will split the data in the same fashion.  If either of them is chosen as the split criteria, then it's random which of the two is used, but if we remove the imposter first, then the real student will be used; but there is not difference in the way the samples are split.

As for finding the imposter, all you can do AFAICT is look for pairs who give exactly opposite responses.  But if there are multiple such pairs, I don't see how to pick one, and even when you have a pair of opposites, I don't see any way to determine which is real and which is imposter.  The 60% figure seems to just indicate that nobody answers "I don't know" too often.  I guess we can make an additional assumption that the students are better-than-random, so that the imposter is worse?


Transformer-XL architecture I am a bit perplex from the transformer-XL architecture that is claimed to solve the issue of context fragmantation. I probably understood it wrong but it looks like all the transformer-XL is doing, is being 2 times larger than the normal one, and that you translate at each iterations. Plus a fancy new way of doing positional encoding.

Am I interpreting it wrong ?

I think your interpretation does not focus on the right places.

First, let's understand where the predecessor idea of Transformer XL in LSTM Language Models: Truncated Back-Propagation Through Time (TBPTT). In LSTM Language Models, the training data text is concatenated into a single very long sequence. From this sequence, we create training batches in a way that for batch $k$, the text sequence at position $i$ is the continuation of the sequence at the same position in batch $k-1$. When training the LSTM, the last hidden states generated at batch $k-1$ were used as initial hidden states when processing batch $k$. The gradients were not propagated through batches, only the values of the hidden states. This allowed the LSTM to be able to profit from unlimited-length context.

The same idea cannot be directly applied to Transformers, as there is not a single hidden state, but as many hidden states as the number of tokens in the sequence times the number of layers.

TransformerXL solves that by incorporating all those hidden states from the previous batch into the computation of the new batch, without propagating the gradients (see figure below from the original article).

This, however, poses a problem in how the positional information in managed in the Transformer, as the positional embeddings encode absolute positions. Therefore, the authors reformulated them to be relative.

Therefore, the main advantage of Transformer XL over vanilla Transformer LMs is that it can profit from unlimited-length context.

Note that Transformer XL is "only" a Language Model, it does not translate, just computes probabilities of the next token based on the previous ones.


Data Preprocessing, how separate background from image to detect animals? Hello I'm looking for some advice making a animal Classifier from trap cams, I applied SIFT to detect points of interested but it turns out that most of the points were in the background and less in the animals. My intention is to recognize animals from a dataset, any ideas that you can bring me to achieve what I'm trying to do?

Images are something like this and the keypoints :

Any other options or suggestions? to change the way I'm going now.

You need to do some Background Subtraction on the images. If you have the Background image without the animal, you can simply subtract it from the current image to get just the animal.
Once you have just the animal, you can apply SIFT or CNNs or whatever.
This is called frame differencing.

If you don't have the background image, you can try methods like this provided by opencv

Basically what you are looking for is background subtraction/foreground detection.

Hope this helps.

image source: http://docs.opencv.org/3.3.0/d1/dc5/tutorial_background_subtraction.html


Suggestion of dataset I am implementing my own deep network, but I am not so good at calculus so my network only works for binary data in the moment. I have been searching for big tabular datasets that are for binary classification. I could not find it, so I want to know if you guys know any dataset with this characteristics, any suggestions?

Not sure how big the data you need but this link has data for binary classification. https://www.kaggle.com/tags/binary-classification


Good books or resources for better understanding mathematical notation used in Machine Learning / Deep Learning Papers Can anybody recommend any books or resources that would be good for better understanding the mathematical notation that is often seen in ML/DL papers? Preferably with examples of how the notation can be "transcribed" into tensorflow code. I did Linear Algebra and Vector calculus in uni a long time ago but it is a long way back and I would like to get back up to speed with it.
This question I think is not what I am looking for because I would like to see an emphasis on how one can transcribe the math into code.

For a pragmatic introduction to Deep Learning with many Tensorflow code examples I'd recommend
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition
or
Deep Learning with Python.
The latter is from 2017 and therefore a bit outdated but a new edition is expected to be published later this year (the author has just finished a draft version).
Both books explain, support or define all concepts with code examples. While there is no literal translation from math to code, I do think they provide the required understanding by directly using code as a notation.


How to analyse the results of cross-validation do determine overfitting I performed k-fold CV and measured the resulting average error (RMSE) for each fold. This was done with 5 folds, and 4 of the measurements gave similar errors (between 10% and 12%), but one of the tests has given a 4% error.
What can be concluded in regards to overfitting in this experiment?
Is the model overfitted because it works much better in one of the situations than in the others?
Thanks.

In short, k-fold CV is not about over-fitting. Your samples can never be ideally identical, so you can only conclude that your error is mean±std.
If your model training process is iterative, then you can detect overfitting by checking test score over the course of training.
If you're making hyper-parameter search with k-fold CV, perhaps with many steps, then you can eventually find out that holdout score is much worse than avg. test score. That would be an overfitting too.


Site for implementing a research papers without code Before a couple of months, I found a web page for implementing papers which are published without code. Someone posts a paper on it and then community competes to implement that paper.

But I don't remember the name of the page and I cannot find it anymore. Any help?

Follow Kristian Kersting, a former professor of mine at University of Bonn who was involved in that event. That was actually a workshop for (if I remember right) NIPS.

UPDATE: well … I found it :D


Features impacting individual predictions after training a classification model I know my question might look odd but I just wanted to get some insights. Every prediction model will give us predictions for validation data set and it also can give/rank features based on their weight on predictions. I am looking for one step after that. Lets say we are dealing with employee resignation and we know the most important features are pay, latest promotion, job satisfaction and external job opportunities around that employees residence. Now, is there any way that we can say which of these features is the most important feature for an individual prediction? May be for one employee, Job satisfaction is the reason behind the leave but based on our feature importance, we need to focus on pay first. I just wanted to know if there is anyway that we can do some post processing on each individual predictions?

Thanks

Depending on what model you are using, the way things are multiplied and added together at prediction time slightly varies, but in principal it's all the same.  You train a model that tries to learn the relationship of the feature vectors you give it to the associated targets.  At prediction time, the same mathematical operations are being applied to every input you are trying to make a prediction for.  So, the weights you learned during the training phase represent the feature importance across the entire training set, in which you hope generalizes to unseen "test" data.  If you suspect that there are other reasons behind why an employee decided to resign outside of what the model weights are telling you, this would indicate that your model does not have enough features to separate the two classes of "not resign" vs. "resign".  Essentially, during testing time, the feature importance is static and the exact same across any input you give.  If feature1 is the most important for input 1, its the most important for every input.  This is what the model learned.


Neural Network for Multiple Float Output I have a dataset containing 40 input columns and 12 output columns (float values).

I supposed it's a regression problem and I am wondering of how to choose the ideal architecture of the Neural Network i didn't find examples of neural Network regression with multiple output
What is the right way to do it?

Just use an output layer with 12 neurons instead of 1. Qualitatively there is no difference. For regression the output activations should be linear, and you have a few choices for cost: RMSE, MAE, or Huber Loss to name a few.


Numeric variables in Decision trees If we have numeric variable, decision trees will use < and > comparisons as splitting criteria. Lets consider this case : If our target variable is 1 for even numeric value, and 0 for odd numeric value. How to deal with this type of variables? How to even identify these type of variables if we have large number of variables? Is there any specific names for these type of variables?

I would call this bad feature engineering, I'm afraid: as the designer of a ML system, one is supposed to analyze their data and find the best way to make the ML system perform as well as possible. In this case by adding a simple feature x % 2 for every instance the decision tree can perform perfectly.

[added] Even in the case of a more complex pattern, if there are such "clusters" of numerical values then there must be a logical explanation why this happens, i.e. something which depends on the task that an expert in this problem can analyze and understand. In most real cases this implies that there are some hidden/intermediate variables, and designing the system so that it represents these variables is key. In other words, the numeric variable is not directly semantically relevant for predicting the response variable, because the assumption when using numeric values is that their order matters (here the numeric value behaves more like a categorical variable).


Clustering Crime Data which has {latitute, longitude, crime-type} tuples I have a data set which has thousands of rows of {latitute, longitude, crime-type} tuples.

Sample Data:

41.757366519   -87.642992854   THEFT
41.910469677   -87.585822373   ROBBERY
41.751270452   -87.690708662   BURGLARY
41.757366519   -87.642992854   THEFT
41.757366519   -87.642992854   THEFT
..             ..              ..
..             ..              ..

I am trying to cluster these based upon the crime types.

For example, if in any  region, THEFT has a high frequency of occurrence, based on the data set, it should show up as a cluster. I have tried clustering using the lat-long data only, and that does not seem to have any meaning for this crime dataset.

I am fairly new to data mining, and gradually figuring my way out.

How can I cluster the data using the latitude and longitude values based such that the clusters are related to each other through the crime-type? Is there any tool available that can use the lat-long data and cluster them on the crime-type basis? Otherwise, I can even write a script once I understand how this can be done.

Also, has anyone had any previous experience in crime-data-mining? In what other ways can I find interesting patterns from a crime data-set?

Thanks a lot!

There is no meaningful way to combine "type" with a distance in meters.

My suggestion is that you:

split the data set by type.
cluster each type, with DBSCAN, haversine distance, and the same minpts/eps values for each crime type
compare the resulting clusters for similarity and differences


Is it ok to fill a pandas dataframe with NaN values? Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values.
Names  Age
Lisa   25
Jack   NaN
Tom    32

Later on I want to work with this dataframe, will I get any problems if I have NaN values in it?

It depends on what problems you are afraid of:

regarding "technical" issues, it should be ok having NaNs in your dataframe and, afterwards, applying the pd.isna(column_name) per attribute to get a boolean mask to find those unknown values per column, more info here
in case your problem is not knowing the actual values, one option is imputation, but not all types of variables are easily imputable (neither it makes sense to impute all of them)


In feature selection, I came across a situation where NaN were filled by median of the column values Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?

There is no rule that only mean or median should be used. Based on the situation sometimes mean is better and sometimes median. In fact there are occasions when mode would be better.
These are not the only techniques to fill NaNs. There are several other imputation methods. If you are starting out, then an excellent hands-on training material would be the Titanic data set which contains a number of NaNs in the 'Age' feature. You can try your hands at finding out which is the best way to impute missing data there. You can refer to: https://www.kaggle.com/c/titanic/discussion/157929 - Missing Ages on the Titanic - Few perspectives from basic to the advanced for some of the advanced strategies (specific to the Titanic scenario)


What are 2D dimensionality reduction algorithms good for? It seems to me that t-SNE and other dimensionality reduction algorithms which reduce the dimensionality to two dimensions are mainly used to get an impression of the dataset. If done well, they look nice (e.g. like this), but I wonder if this is better than just showing random images / grouping them by class on a grid.

I would like to get an answer to the following aspects:

How is t-SNE* better than just taking a random (probably stratified) sample of the data?
How is t-SNE* better than just fitting a neural network with a 2-neuron bottleneck to the data and then taking the (normalized) value of the 2 neurons for an embedding?
Does t-SNE* give any guarantees?
Is t-SNE* good for the construction of classifiers? I mean: If you already have a classifier which is much better than random / guessing the most frequent class, does t-SNE help you to make better classifiers? How?
There are many dimensionality reduction algorithms. How does one compare the non-linear ones? When is one algorithm better than others? Especially: Are they better than bottleneck features of neural networks?

*: You could probably answer this for other dimensionality reduction algorithms as well, but t-SNE seems to be the most popular one.

Please note: I do see the advantage of a reduced dimensionality for compression / easier optimization / faster inference. However, the reduction to 2 dimensions seems to be only for visualization. Hence my question if one can see more in those embeddings than a visually pleasing image of the dataset.

There are many questions there, I'll try to address each in turn.

How is t-SNE* better than just taking a random (probably stratified) sample of the data?

If your goal is to provide a visual overview of the data then clearly a stratified sample is not going to do that -- the samples each still live in high dimensional space, so are no easier to visualise, and if you are interested in how the data relate to each other then looking through a sample item by item does not necessarily help to build a reasonable mental model of those relationships. So sampling and visualisation with t-SNE (or similar approaches) are both attacking similar underlying problems: "What does my data look like"; they provide very different views on that, highlighting different aspects however, and I would say are entirely complementary. Why not both!

How is t-SNE* better than just fitting a neural network with a 2-neuron bottleneck to the data and then taking the (normalized) value of the 2 neurons for an embedding?

This is harder, because I can't point you to explicit studies demonstrating otherwise, but t-SNE generally does a better job of providing a visual representation that is meaningful to users. It is worth noting that t-SNE was co-created by Geoff Hinton, who I am sure knows a great deal about neural networks and their potential uses and benefits. If Geoff felt t-SNE was worth using then one can reasonably presume it has some merits over other NN approaches. Now, with that in mind one can probably interpret t-SNE as a NN with 2-neuron bottleneck; I haven't tried to write that out in detail, so don't quote me on it. Finally, NNs aren't necessarily the answer to all problems; t-SNE is an algorithm for manifold learning specifically designed for low dimensional embedding, while an autoencoder with suitable bottleneck may happen to give similar results, but certainly isn't necessarily clearly suited to the task.

Does t-SNE* give any guarantees?

Guarantees of what? Providing a visual representation of the data? Yes it will do that. Guarantee that the Kullback-Liebler divergence of inferred distributions for the high dimensional and low dimensional representations has been minimised? Up to the quality of the optimisation, yes. Guarantee that local structure is preserved? Up to a suitable choice of perplexity for the dataset, yes. Guarantee that the low dimensional representation is a fully faithful representation of the global structure of the data? No, no guarantees there. It really depends on what you're looking for.

Is t-SNE* good for the construction of classifiers? I mean: If you already have a classifier which is much better than random / guessing the most frequent class, does t-SNE help you to make better classifiers? How?

Not really -- not any more than what you potentially expect from any dimensionality reduction technique, and as you note forcing things to 2 dimensions constrains that a lot. In principle, since t-SNE focuses on preserving local structure and sacrifices global structure to do so, if your classification relies heavily on local structure then t-SNE could perform better than other dimension reduction techniques that seek to preserve global structure and thus have poorer local structure representation. In the case of unsupervised (density) clustering it is certainly true that local structure is more important than global structure so potentially t-SNE can help. However compression to 2 dimensions may be rather extreme.

There are many dimensionality reduction algorithms. How does one compare the non-linear ones? When is one algorithm better than others? Especially: Are they better than bottleneck features of neural networks?

If your interest is in visualisation (as is the case with t-SNE) then unfortunately comparisons are inevitably fairly subjective. This is much like the case of clustering. Certainly there are many different measures of clustering quality, but often that comes down to "what do you mean by a cluster?", and for each measure there is a clustering algorithm that optimises for that measure. Likewise there are measures for how well an embedding has succeeded, but ultimately it depends on what you mean by a successful embedding (is preserving large scale relationships critical? is preserving local structure more important? How do you weigh one against the other?), and for any given measure there is an algorithm that optimises that measure. For a subjective evaluation the standard approach seems to be to embed labelled data and then view the result colored by label -- no, that is never going to clearly demonstrate superiority, but it can hint at what seems to be working. In practice t-SNE seems to fair well at this (hence its popularity). Are any of these better than bottlenecked autoencoders? I'm not sure why one assumes that a bottlenecked autoencoder is by default the superior option, but certainly in my experience whenever I've tried several options on data I've found t-SNE to provide better intuition about the dataset. Anecdotes are not data of course, so by all means try the options yourself.

I hope this covers most of it, although perhaps these were not quite the sort of answers you wanted.


How to use SimpleImputer Class to replace missing values with mean values using Python? This is my code

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

#Importing Dataset
dataset = pd.read_csv('C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 1 - Data Preprocessing/Data.csv')
print(dataset)
X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, 3].values
#Missing Data

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(missing_values= np.nan, strategy='mean')
X.fit[:, 1:3] = imputer.fit_transform(X[:, 1:3])
print(X)

My data set:

Country   Age   Salary Purchased
0   France  44.0  72000.0        No
1    Spain  27.0  48000.0       Yes
2  Germany  30.0  54000.0        No
3    Spain  38.0  61000.0        No
4  Germany  40.0      NaN       Yes
5   France  35.0  58000.0       Yes
6    Spain   NaN  52000.0        No
7   France  48.0  79000.0       Yes
8  Germany  50.0  83000.0        No
9   France  37.0  67000.0       Yes

Error Message:

File "C:/Users/Rupali Singh/PycharmProjects/Machine_Learning/data_preprocessing_Template.py", line 15, in <module>
    X.fit[:, 1:3] = imputer.fit_transform(X[:, 1:3])
AttributeError: 'numpy.ndarray' object has no attribute 'fit'

Your error is due to using Simple Imputer's fit and fit_transform on a numpy array. Here's how i used it on a Dataframe

imr = Imputer(missing_values='NaN', strategy='median', axis=0)
imr = imr.fit(data[['age']])
data['age'] = imr.transform(data[['age']]).ravel()

  X.fit = impute.fit_transform().. this is wrong. you can't assign a value to a X.fit() just simply because .fit() is an imputer function, you can't use the method fit() on a numpy array, hence your error!

Use x[:, 1:3] = imputer.fit_transform(x[:, 1:3]) instead

Hope this helps!


How does keras calculate accuracy for multi label classification? I am using this code for a multilabel problem classification.

from __future__ import print_function

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import LSTM
from keras.models import Model
from keras.datasets import imdb

max_features = len(vocabDic)
maxlen = 500  # cut texts after this number of words (among top max_features most common words)
batch_size = 32

train_set = sequence.pad_sequences(train_set, maxlen=maxlen)
test_set = sequence.pad_sequences(test_set, maxlen=maxlen)

print('train_set shape:', train_set.shape)
print('test_set shape:', test_set.shape)

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(90, activation='sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
model.fit(train_set, train_labels,
          batch_size=batch_size,
          epochs=15,
          validation_data=(test_set, test_labels))
score, acc = model.evaluate(test_set, test_labels,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

My problem is there are 90 classes and the accuracy is too high from the second epoch. I suspect keras is computing something incorrectly.
Any clues?

Edit: I calculated the total recall of the model. It is barely 5%( 5 times better than random). Is it a normal behavior for such a problem?

Why are you using binary_crossentropy?  You should be using categorical_crossentropy.  However, if you insist on using binary_crossentropy change your metric to metrics=['binary_accuracy', 'categorical_accuracy'] (this will display both accuracies).


When combined correlation of features decreases I'm building a machine learning model in Python to predict soccer player values. I'm trying to predict a "player_value" column containing the value of a specific player. Consider a sample of the columns (features) I'm using.
---------------------------------
appearances | goals | goals_per_game
------------|-------|---------------
    20      |   2   |     0.1
    60      |   20  |     0.33
    54      |   30  |     0.55
    43      |   15  |     0.34
    30      |   17  |     0.56

I thought that the correct way to use those columns would be creating a goals per game statistic (goals divided by appearances), since a player can have more goals than another player, but with less matches played.
After that, the correlation of the refered columns with the column that I'm trying to predict (player value) decreased. The correlation of the "goals" and the "appearances" columns with the player value column was about 45% each, while the new "goals_per_game" column has a correlation around 18%.
Should I use the columns "appearances" and "goals_per_game" columns individually and not use the "goals_per_game" column? Is my analysis wrong and it does not makes sense to use a "goals_per_game" metric since the player value is higher when using those features individually?

The only metric you can use to assess the added value of each feature, is the accuracy of your predictive classifier. You can simply include all three feature variables and build your classifier from those. Then you can remove each of the $3$ features and build the three possible classifiers from each of the remaining $2$ features (for example, 'appearances' and 'goals', 'appearances' and 'goals_per_game', and so forth). This analysis gives you the added value of each of the three features to your predictive accuracy. Note the apparent dependency between 'goals' and 'goals_per_game'.
This strategy to feature selection is called sequential backward search. If you have an excess of $10$ features, more advanced algorithms like floating search and MCMC are likely to yield better performing subsets of features. In your case with solely $3$ feature variables sequential backward search should work fine.


fluctuating values for validation set only My model's structure is
                                  Output
                                    ^
                                    |
                             ----------------
                             | Dense Network |
                             ----------------
                                   /\
                                   ||
                                   ||
                                   ||
   |--------------------|          ||          | ----------------------|
   | RNN on features    | ========>||<======== |  Dense Network on non |
   | changing with time |        [concat]      |  time series data     |
   |--------------------|                      |-----------------------|

These are the Training and Validation set metric outputs of my model. Why are the values fluctuating so much for the validation set?. Any ideas?
[![4 Graphs. Loss, Accuracy, Precision, Recall][1]][1]
Update :
As suggested in comments I have tried increasing the validation set size Now the size ratio is
49.6%-50.4%
Also I have made the model very simple by Using fewer layers. The new graph obtained looks like this
[![4 Graphs. Loss, Accuracy, Precision, Recall on simpler model][2]][2]
Is this acceptable as 'okay-fluctuating'?

Thanks for updating the post, this level of fluctuation in the validation set is a lot less dramatic than before and appears to be similar to regular fluctuation I have seen in my experience. Kudos that you have also managed to prevent the model from overfitting.


Transforming AutoEncoders I've just read Geoff Hinton's paper on transforming autoencoders

  Hinton, Krizhevsky and Wang: Transforming Auto-encoders. In Artificial Neural Networks and Machine Learning, 2011.

and would quite like to play around with something like this. But having read it I couldn't get enough detail from the paper on how I might actually implement it.

Does anyone know how the mapping between input pixels to capsules should work?
What exactly should be happening in the recognition units?
How it should be trained? Is it just standard back prop between every connection?

Even better would be a link to some source code for this or something similar.

I've put together some example tensorflow code to help explain (the full, working code is in this gist). This code implements the capsule network from the first part of section 2 in the paper you linked:

N_REC_UNITS = 10
N_GEN_UNITS = 20
N_CAPSULES = 30

# input placeholders
img_input_flat = tf.placeholder(tf.float32, shape=(None, 784))
d_xy = tf.placeholder(tf.float32, shape=(None, 2))

# translate the image according to d_xy
img_input = tf.reshape(img_input_flat, (-1, 28, 28, 1))
trans_img = image.translate(img_input, d_xy)
flat_img = tf.layers.flatten(trans_img)

capsule_img_list = []

# build several capsules and store the generated output in a list
for i in range(N_CAPSULES):
    # hidden recognition layer
    h_rec = tf.layers.dense(flat_img, N_REC_UNITS, activation=tf.nn.relu)
    # inferred xy values
    xy = tf.layers.dense(h_rec, 2) + d_xy
    # inferred probability of feature
    p = tf.layers.dense(h_rec, 1, activation=tf.nn.sigmoid)
    # hidden generative layer
    h_gen = tf.layers.dense(xy, N_GEN_UNITS, activation=tf.nn.relu)
    # the flattened generated image
    cap_img = p*tf.layers.dense(h_gen, 784, activation=tf.nn.relu)

    capsule_img_list.append(cap_img)

# combine the generated images
gen_img_stack = tf.stack(capsule_img_list, axis=1)
gen_img = tf.reduce_sum(gen_img_stack, axis=1)

  Does anyone know how the mapping between input pixels to capsules should work?

This depends on the network structure. For the first experiment in that paper (and the code above), each capsule has a receptive field that includes the entire input image. That's the simplest arrangement. In that case, it's a fully-connected layer between the input image and the first hidden layer in each capsule.

Alternatively, the capsule receptive fields can be arranged more like CNN kernels with strides, as in the later experiments in that paper.

  What exactly should be happening in the recognition units?

The recognition units are an internal representation that each capsule has. Each capsule uses this internal representation to calculate p, the probability that the capsule's feature is present, and xy, the inferred translation values. Figure 2 in that paper is a check to make sure the network is learning to use xy correctly (it is).

  How it should be trained? Is it just standard back prop between every connection?

Specifically, you should train it as an autoencoder, using a loss that enforces similarity between the generated output and the original. Mean square error works well here. Aside from that, yes, you'll need to propagate the gradient descent with backprop.

loss = tf.losses.mean_squared_error(img_input_flat, gen_img)
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)


Scraping mixed elements and passing to SQL I'm running webscrapes via python which are retrieving data from csv's hosted on the web.  I'd like to pass the data into a MSSQL database.  An issue I have is the mixed elements/data types in the csv.  Here is an example of the data

Item    Val1    Val2
A       100      200
B       101      201
C       Null    -2/2(%)
D       Null    2019-Nov-18

I would like to import all of this data into the db, but the critical data is in the "Val2" column.  It contains numbers (formatted as numbers and general), dates, and strings with numbers as above.

I would appreciate your thoughts on two things:

After I get this data into python, whats the best way/package to
use to send it to SQL
Do I need to set up the SQL table in a certain way to handle the mixed elements?  Should I filter those different values out with python and put them somewhere else?

For 1, I've looked at things like pd.to_sql and iterating over rows and sending via pyodbc. Data won't be particularly large to begin with. I'll be setting up a lot of scrapes so just want some ideas on the "best" way to do this transfer so I don't have to re-design them all in the future

Thanks I know this is open ended but would love to hear any thoughts on the "right" way to do this

One way would be to declare that column as a character type. I think this would be a poor solution as it forces all interpretation and processing of the data back onto the application. Better would be to declare Val2 as datatype sql_variant. At least then the DBMS has a fighting chance of interpreting the value correctly.

Since each of those datatypes represents a different attribute with unique semantics it would be better to separate them into their own columns. SQL Server has the TRY_PARSE function which allows you to discover the datatype of each value and assign it to its proper column.


Hindsight experience replay: strategy for sampling goals The authors of Hindsight Experience Replay list out several strategies for sampling a set of additional goals $G$ in Section 4.5:

final - corresponds with final state of environment,
future — replay with k random states which come from the same episode as the transition being replayed and were observed after it,
episode — replay with k random states coming from the same episode as the transition
being replayed,
random — replay with k random states encountered so far in the whole training procedure.

My interpretation of the future method is then that we can only select k random states if the current transition being replayed has already happened in the episode, so this is at minimum the second time we've seen the current transition. This seems very unlikely if working in an environment with a large state space (especially with continuous features). Am I missing something obvious in the interpretation of how to implement this strategy?

I’ve been there for quite a while until i read the implementation of thr algorithm in OpenAI Baselines library, you can find it on Github.
Basically as of what i understood from the algorithm the K factor mainly adjusts the percentage of sampled data (being replayed for the update step in Q weights) that if we sample 1 transition without HER we sample another K with HER. This can be implemenated in the prescribed fashion if we sampled a batch of data and replayed K/1+k with HER by simply changing their goals by other states observed at any time in the future


How to run R scripts without closing X11 I would like to run an R script using a single command (e.g. bat file or shortcut).

This R script asks the user to choose a file and then plots information about that file. All is done via dialog boxes.

I don't want the user to go inside R - because they don't know it at all.

So, I was using r cmd and other similar stuffs, but as soon as the plots are displayed, R exits and closes the plots.

What can I do?

Thanks for your help.

This looks like a similar kind of a problem.

Solutions: (taken from above source)

Just sleep via  Sys.sleep(10) which would wait ten seconds.
Wait for user input via readLines(stdin()) or something like that [untested]
Use the tcltk package which comes with R and is available on all platforms to pop up a window the user has to click to make the click disappear. That solution has been posted a few times over the years on r-help.

2nd option is better to use for the user.

P.S. Since I did not come up with the answer myself, I tried to put it in comment but my reputation is too low for that.


Input 0 is incompatible with layer conv2d_2: expected ndim=4, found ndim=3 I get this error in Tensor flow, What does it mean and how can I fix it? import pickle

import keras

from keras.models import Sequential

from keras.layers import Dense, Dropout, Flatten, Activation

from keras.layers import Conv2D, MaxPooling2D

from keras.utils import to_categorical

import numpy as np

X = pickle.load(open('cancer_image_features.pickle','rb'))

y = pickle.load(open('cancer_image_lables.pickle','rb'))

X = X/255.0

y = to_categorical(y)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=X.shape[1:]))

model.add(Conv2D(64, (3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, kernel_size=(3, 3),activation='relu'))

model.add(Conv2D(64, (3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Activation("softmax"))
model.add (Flatten())

model.add(Dense(3))

model.compile ( loss = 'binary_crossentropy', optimizer = 'adam' , metrics = ['accuracy'])

model.fit(X,y, batch_size = 512, epochs = 10,  validation_split= 0.3)

The error is likely to come from this line:
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=X.shape[1:]))

Input_shape should be a 4dim vectors as stated in the keras doc:

Input shape
4D tensor with shape: (batch, channels, rows,
cols) if data_format is "channels_first" or 4D tensor with
shape: (batch, rows, cols,
channels) if data_format is "channels_last".

You may have to reshape your data, as stated here:
https://stackoverflow.com/q/43895750/8119313


Avoid reloading DataFrame between different python kernels Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks?

I'd be looking for something, which is conceptually similar to MATLAB's persistent variables. There it is possible to call a custom function / library from multiple individual editors (notebooks), and have that external function cache some result (or large table).

Mostly I would like to avoid reloading a heavily used table (which is loaded through a custom library that is is called from the notebooks ), since reading it takes around 2-3 minute whenever I start a new analysis.

If it's important for your use cases, you could try switching to Apache Zeppelin.
As all Spark notebooks there share the same Spark context, same Python running environment.
https://zeppelin.apache.org/

So what you're asking happens natively in Zeppelin. Or to be complete, it is an option to share the same Spark context / same Python envrionment between all Spark notebooks (they're called 'notes' in Zeppelin):

So you can choose to share context Globally (default Zeppelin's behavior), Per Note (the only possible Jupyter's behavior), or Per User.

If you can't / don't want to switch to Zeppelin, look at other options of sharing common dataframes between your notebooks using:

Apache Arrow
Feather

ps. You can't import ipynb files to Zeppelin currently as of now (it has its own notebook format stored as a json file), until https://issues.apache.org/jira/browse/ZEPPELIN-1793 is implemented; although it's not that hard to convert them manually in most cases.


why my batch size doesn't change it happens quite a lot for me that I declared batch_size=64 or any number and end up watching Keras training my net on whole data instead of the batch. then I edit it many times and finally finished with the very first code working.
for example this simple code:
data_train=pd.read_csv("train.csv")
y=data_train["label"]
dt=data_train.drop("label", axis=1)
inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(500, activation="relu", name="dense_1")(inputs)
x = layers.Dense(500, activation="relu", name="dense_2")(x)
x = layers.Dense(300, activation="relu", name="dense_3")(x)
x = layers.Dense(64, activation="relu", name="dense_4")(x)
outputs = layers.Dense(10, activation="softmax", name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)
opt = keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=opt, loss="sparse_categorical_crossentropy", metrics=["accuracy"])
X_train, X_test, y_train, y_test = train_test_split(dt, y, test_size=0.2)
X_train, Xvalid, y_train, yvalid = train_test_split(X_train, y_train, test_size=0.2)
history=model.fit(X_train,y_train, batch_size=64 ,epochs=30,validation_data=(Xvalid,yvalid))

and it train itself with :
26880/26880 [==============================] - 3s 108us/sample - loss: 0.4717 - acc: 0.8718 - val_loss: 0.4149 - val_acc: 0.8847
how I can fix it?
Thanks

Using model.fit() will always train on the whole dataset for n epochs. The batch_size argument denotes how many samples are used to calculate the gradient and updates the parameters. If you want to train the model on just a batch of data (i.e. a subset of your total dataset) simply pass the subset of data to model.fit().


Invariance Property of Vowpal Wabbit Updates - Explaination One of the discussed nice aspects of the procedure that Vowpal Wabbit uses for updates to sgd
pdf is so-called weight invariance, described in the linked as:

"Among these updates we mainly focus on a novel
set of updates that satisfies an additional invariance
property: for all importance weights of h, the update
is equivalent to two updates with importance weight
h/2. We call these updates importance invariant."

What does this mean and why is it useful?

Often different data samples have different weighting ( eg the costs of misclassification error  for one group of data  is higher than for other classes).
Most error metrics are of the form $\sum_i e_i$ where e_i is the loss ( eg squared error) on data point $i$. Therefore weightings of the form $\sum_i w_i e_i$ are equivalent to duplicating the data w_i times (eg for w_i integer).

One simple case is if you have repeated data - rather than keeping all the duplicated data points, you just "weight" your one repeated sample by the number of instances.

Now whilst this is easy to do in a batch setting, it is hard in vowpal wabbits online big data setting: given that you have a large data set, you do not just want to represent the data n times to deal with the weighting ( because it increases your computational load). Similarly, just multiplying the gradient vector by the weighting - which is correct in batch gradient descent - will cause big problems for stochastic/online gradient descent: essentially you shoot off in one direction ( think of large integer weights) then you shoot off in the other - causing significant instability. SGD essentially relies on all the errors to be of roughly the same order ( so that the learning rate can be set appropriately). So what they propose is to ensure that the update for training sample x_i with weight n is equivalent to presenting training sample x_i n times consecutively.

The idea being that presenting it consecutively reduces the problem because the error gradient (for that single example $x_i$) reduces for each consecutive presentation and update (as you get closer & closer to the minimum for that specific example). In other words the consecutive updates provides a kind of feedback control.

To me it sounds like you would still have instabilities (you get to zero error on x_i, then you get to zero error on x_i+1,...). the learning rate will need to be adjusted to take into account the size of the weights.


Improve results using user input I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the user input)
n-result are retrieved but the closest expressions are not necessarily the most relevant one.

For example, by typing : hospital machine
The top results will be "dialysis machine", "medical machine", ... but I'll also find expressions like "building machine", "office machine"
A user will most likely choose medicine related machine.

Is there a way to optimize my ranking system based on the user input while keeping this similarity between vectors of the expression ?

Understanding similarity between two phrases has two aspects

How similar are the unique tokens in the phrases ?
How much should the individual tokens contribute to the overall phrase similarity?

To answer 1, you can use vector similarity which can give you high similarity for tokens similar in meaning. To answer 2, you should look at giving importance/weights to the tokens. You can use a measure like tf-idf. While comparing hospital machine and building machine, machine being a frequent word in your corpus should get a lower score and hence would contribute lesser to the overall similarity. Most of the similarity would be then determined by the similarity between hospital and building which would solve your issue.


What is LSTM, BiLSTM and when to use them? I am very new to Deep learning and I am particularly interested in knowing what are LSTM and BiLSTM and when to use them (major application areas). Why are LSTM and BILSTM more popular than RNN?

Can we use these deep learning architectures in unsupervised problems?

RNN architectures like LSTM and BiLSTM are used in occasions where the learning problem is sequential, e.g. you have a video and you want to know what is that all about or you want an agent to read a line of document for you which is an image of text and is not in text format. I highly encourage you take a look at here.

LSTMs and their bidirectional variants are popular because they have tried to learn how and when to forget and when not to using gates in their architecture. In previous RNN architectures, vanishing gradients was a big problem and caused those nets not to learn so much.

Using Bidirectional LSTMs, you feed the learning algorithm with the original data once from beginning to the end and once from end to beginning. There are debates here but it usually learns faster than one-directional approach although it depends on the task.

Yes, you can use them in unsupervised learning too depending on your task. take a look at here and here.


What is the relationship between MDP and RL? What is the relationship between Markov Decision Processes and Reinforcement Learning?

Could we say RL and DP are two types of MDP?

What is the relationship between Markov Decision Processes and Reinforcement Learning?

In Reinforcement Learning (RL), the problem to resolve is described as a Markov Decision Process (MDP). Theoretical results in RL rely on the MDP description being a correct match to the problem. If your problem is well described as a MDP, then RL may be a good framework to use to find solutions. That does not mean you need to fully describe the MDP (all the transition probabilities), just that you expect an MDP model could be made or discovered.

Conversely, if you cannot map your problem onto a MDP, then the theory behind RL makes no guarantees of any useful result.

One key factor that affects how well RL will work is that the states should have the Markov property - that the value of the current state is enough knowledge to fix immediate transition probabilities and immediate rewards following an action choice. Again you don't need to know in advance what those are, just that this relationship is expected to be reliable and stable. If it is not reliable, you may have a POMDP. If it is not stable, you may have a non-stationary problem. In either case, if the difference from a more strictly defined MDP is small enough, you may still get away with using RL techniques or need to adapt them slightly.

  Could we say RL and DP are two types of MDP?

I'm assuming by "DP" you mean Dynamic Programming, with two variants seen in Reinforcement Learning: Policy Iteration and Value Iteration.

In which case, the answer to your question is "No". I would say the following relationships are correct:

DP is one type of RL. More specifically, it is a value-based, model-based, bootstrapping and off-policy algorithm. All of those traits can vary.

Probably the "opposite" of DP is REINFORCE which is policy-gradient, model-free, does not bootstrap, and is on-policy. Both DP and REINFORCE methods are considered to be Reinforcement Learning methods.

DP requires that you fully describe the MDP, with known transition probabilities and reward distributions, which the DP algorithm uses. That's what makes it model-based.
The general relationship between RL and MDP is that RL is a framework for solving problems that can be expressed as MDPs.


Is there any way to collect categorical features quickly in Julia DataFrames? I'm using Julia 0.6.3 with Dataframes.jl

I was wondering if there was any way to get categorial features easily in Julia?

For large datasets it can be impossible to enter everything by hand.

My workaround is to rely on strings and usually low cardinality but it's not fool-proof.

My workaround so far :

cat_cols = []
for col in cols
    if contains(string(typeof(X_train[col])),"String") == true
        push!(cat_cols,col)
    end
end

But it seems kind of ugly and I don't catch label encoded values because they are integers.

I could also try to rely on low unique counts but then sparse features would be taken in aswell.

Any idea? Thanks!

I think you can use the eltypes function in DataFrames.

categorical_indices = eltypes(X_train) .== String
categorical_columns = names(X_train)[categorical_indices]

This should provide a vector of Symbols for each of the categorical columns.


Training data for multi-category classification algorithm I am putting together a multi-category classification algorithm. Since it's NLP, the training data is very simple with one column for labels and another column for text. However, because it's NLP, some training records can fit multiple categories. Should I iterate through & have multiple entries (same data, different labels) or should each piece of data only have 1 category assigned?

No, it is perfectly possible to train on multiple categories. What you need, though, is an exhaustive list of these categories (in supervised learning, that is).

Suppose you are trying to associate sentences with topics, and you have a list of possible topics topics = ['sports', 'soccer', 'politics'].
It sounds like your data look something like this:

sentence                       | topics
-------------------------------|----------------------------------
'Barack Obama loves soccer'    | ['politics', 'sports', 'soccer']
'The parliament is important'  | ['politics']
'Soccer is fun'                | ['sports', 'soccer']

Then you need to one-hot encode the topics:

X = [['Barack Obama loves soccer'], ['The parliament is important'], ['Soccer is fun']]

Y = [[1, 1, 1], [1, 0, 0], [0, 1, 1]]

And then you train a neural network to predict not one but three (= number of topics) values.


"Binary Encoding" in "Decision Tree" / "Random Forest" Algorithms Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities?
Some facts about my dataset:

My dataset has ~170,000 rows
One of the categoric variables has 1,700 unique values.
Another one has 3,000 unique values.
Note that it is not practically possible to group the values of those variables into more aggregate levels.

As a domain expert,I am sure those categorical columns with high cardinalities are strong candidates as predictors.
On the other hand, binary encoding surely decreases model's interpretability.
Else than interpretability, after binary encoding, is it just alright to build a decision tree / random forest model on the newly formed dataset with new variables which only indicating bits?
Click for a good post on encoding categorical features

In general, it "okay" to apply to binary encode high cardinality datasets. In the sense of it will create numerical features that can be learned by a machine learning model.
However there are often better options, such a label encoding, frequency encoding, target encoding, or embeddings.
It is an empirical question which encoding scheme is best for your specific data and model. The best empirical coding scheme can be found through cross validation.


Can I make kaggle kernels read directly from my computer? For some reason jupyter notebooks never work the same as kaggle kernels for me. I want to use kaggle kernels but the downside is I don't know how to make it read from a file on the computer like a regular jupyter notebook

No, you cannot write a Kaggle kernel that will read your local data directly.

The best you can do is upload your data to the kernel. See https://www.kaggle.com/docs/kernels#adding-data-sources


Multi-class clasification Just getting my toes wet with running some models on the Iris dataset and was wondering if using One-vs-Rest is required or not? Because I can fit a linear model without it, but using OVR yields different results.

Any explainations would be great, thank you!

Linear regression is not a classification technique. Most of this is meaningless as a result. Did you mean logistic regression? You do not need OneVsRestClassifier explicitly if you set LogisticRegression to handle multi-class classification internally with 'ovr' (one vs rest again) or 'multinomial' (softmax loss)


Is there any domain where Spiking Neural Networks outperform other algorithms (non-spiking)? I'm reading about reservoir computing techniques like Echo State Networks and Liquid State Machines. Both of the methods involve feeding inputs to a population of randomly (or not) connected spiking neurons, and a relatively simple readout algorithm that produces the output (e.g. linear regression). The neuron population weights are either fixed, or trained via a Hebbian-like local activity rule like STDP.

These techniques perform well when modelling multi-dimensional inputs that have significant temporal components. However, computing the spiking neuron membrane potentials involves differential equation integration and can be computationally expensive.

Are there any examples of where the additional computational complexity of reservoir computing techniques is outweighed by gains in a prediction or classification task?

For example, are there any cases of SNN techniques outperforming comparably complex architectures based on RNNs, ANNs, SVMs, DNNs, CNNs, or other algorithms?

My answer comes from experience more than from experiments or benchmarks published.

As far as I know, Spiking Neural Networks do not outperform other algorithms in any task. There have been advances in robotics and reservoir computing but  reservoir computing algorithms are as good as other algorithms (like reinforcement learning) according to recent publications.  There are rumours that some companies are interested in these algorithms because they've hired a few reservoir computing researchers recently but these are only rumours.

Here is one of the most recent publications detailing advances and limitations of reservoir computing in robotics
https://www.cs.unm.edu/~afaust/MLPC15_proceedings/MLPC15_paper_Polydoros.pdf

I began experimenting with Liquid State Machines in college using Wolfgang Maass' proposed architecture. It looked promising, specially the idea of inhibitory neurons forming part of the circuit. But in reality using these algorithms in real life data applications (language classification, image classification among others) was not enough to get close to the benchmarks like RNNs, ANNs, SVMs. Sometimes even vanilla Multilayer Neural Networks perform better than Liquid State Machines. My understanding is that these kind of models are good for robotics and other autonomous related tasks like sensors and autonomous navigation (but that was not my area of research) but not so much for other types of data. There are a few labs, mainly in Europe working with this algorithm but so far I haven't heard of many advances in this area in the past years.

I do believe that brain inspired algorithms are the next big step in AI, and while many companies like Numenta and Deepmind are researching towards this direction, as of today there is still a lot of work to be done in order to have the next breakthrough in AI.