Keras exception: Error when checking input: expected dense_input to have shape (2,) but got array with shape (1,) I have an understanding of this error, it means that the input that I'm passing to the model is of a different dimension that what was expected. The error also states that the input that I'm passing is of the dimension (1,) while it was expecting (2,)

I have tested the input value dimension by using x.shape and it prints out (2,) still the error exists. As a counter-intuitive move I picked one of the data that was in the training data and printed the shape of the zeroth element x1[0].shape also used that as an input, the error still exists.

model.fit works well, having error with model.predict (tried passing one of the training data hardcoded, still doesn't work)

CODE:

import tensorflow as tf
import numpy as np
from tensorflow import keras
import csv

x1, ys = [], []

with open('./house.csv') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    line = 0
    for row in csv_reader:
        if line > 0:
            x1.append([row[1], row[3]])
            ys.append(row[5])
        line += 1

model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[2])])
model.compile(optimizer='sgd', loss='mean_squared_error')
x1 = np.asarray(x1, dtype=float)
ys = np.asarray(ys, dtype=float)
model.fit(x1, ys, epochs=500)

print(x1[0].shape)
while True:
    house_size = float(input('Enter the house size: '))
    house_size = house_size/3000
    bhks = float(input('Enter the BHK: '))
    bhks = bhks/3
    x = np.array([house_size, bhks])
    try:
        value = model.predict(x)
    except Exception as e:
        print(e)
        print(x)
        print(x.shape)
    else:
        value = value[0][0] * 500
        print(value)

Yoy always need to pass the data for prediction in batches, although this batch is of size one (one sample). Try changing this line:

x = np.array([house_size, bhks])

into this:

x = np.array([[house_size, bhks]])

This should work.


Feature Normalization/Scaling: Prediction Step I'm just doing a simple linear regression with gradient descent in the multivariate case. Feature normalization/scaling is a standard pre-processing step in this situation, so I take my original feature matrix $X$, organized with features in columns and samples in rows, and transform to $\tilde{X}$, where, on a column-by-column basis,
$$\tilde{X}=\frac{X-\bar{X}}{s_{X}}.$$
Here, $\bar{X}$ is the mean of a column, and $s_{X}$ is the sample standard deviation of a column. Once I've done this, I prepend a column of $1$'s to allow for a constant offset in the $\theta$ vector. So far, so good.

If I did not do feature normalization, then my prediction, once I found my $\theta$ vector, would simply be $x\cdot\theta$, where $x$ is the location at which I want to predict the outcome. But now, if I am doing feature normalization, what does the prediction look like? I suppose I could take my location $x$ and transform it according to the above equation on an element-by-element basis. But then what? The outcome of $\tilde{x}\cdot\theta$ would not be in my desired engineering units. Moreover, how do I know that the $\theta$ vector I've generated via gradient descent is correct for the un-transformed locations? I realize all of this is a moot point if I'm using the normal equation, since feature scaling is unnecessary in that case. However, as gradient descent typically works better for very large feature sets ($> 10k$ features), this would seem to be an important step. Thank you for your time!

I have learned what the correct answer is: you have to transform your prediction location in precisely the same way you do for the columns of matrix $X$: first, subtract the means element-wise from each component of the prediction location $x$. Second, divide element-wise by the standard deviation. Third, prepend a $1$ to allow for the bias. Finally, perform the dot product with your $\theta$ vector. Since the $\theta$ vector was calculated on transformed data, it's meant to operate on transformed data. It will contain units and scaling appropriate to produce an answer in engineering units. Reference: see Week 2, FAQ, Question 8 in Andrew Ng's Machine Learning course on Coursera. (Login is probably necessary.)


Binary Classification Comparing two time series of variable length Is there a machine learning model (something like LSTM or 1D-CNN) that takes two time series of variable length as input and outputs a binary classification (True/False whether time series are of same label)?

So the data would look something like the following

date        value label
2020-01-01  2     0     # first input time series
2020-01-02  1     0     # first input time series
2020-01-03  1     0     # first input time series
2020-01-01  3     1     # second input time series
2020-01-03  1     1     # second input time series

Is there something like that available out of the box, and if not how would you build a minimal working example model in Keras?

My best guess is to use a shared LSTM layer for both inputs and Concatenate both resulting vectors before feeding to the final Dense layer.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

n_lstm_blocks = 50

input_1 = keras.Input(shape=(None, 1)) # unknown timespan, fixed feature size 1
input_2 = keras.Input(shape=(None, 1))
shared_lstm = layers.LSTM(n_lstm_blocks)
encode_1 = shared_lstm(input_1)
encode_2 = shared_lstm(input_2)
concat = layers.concatenate([encode_1,encode_2])
output = layers.Dense(1, activation='sigmoid')(concat)
model = keras.Model(inputs=[input_1,input_2],outputs=output)
model.compile(optimizer='adam', loss='binary_crossentropy')

A comparable task would be Siamese Networks / One-Shot learning which is used for face recognition. But in this case the task is to compare to time series and detect if they are of the same label, but knowing each label is NOT task of the network!

I think both LSTM and 1D-CNN could work, but it depends on your data first.

In your code you used a shared LSTM, which implies that you think your two input sequences are actually just different length sequences of a same variable? If that's the case, why not just use a LSTM to label, and then compare their label.

If that's not the case, you can then use separate input.

So the restriction for using 1D-CNN for sequence is that your sequence should have same length among samples (to my current knowledge). So if you have two types of sequences with fixed length $l_1$ and $l_2$ among samples, then you can try this approach.

If your have two types of sequences, both with variable length, then you will have to use one dynamic input LSTM for each of them, and then concat this two LSTM.


Result Dithering? Why randomly shuffle results? Most of us want to build a recommendation engine as accurate as possible, however, an experienced chief data scientist believes a practical machine learning algorithm should randomly shuffle the results, therefore non-optimal results. This is known as "Results Dithering". Slide 15 at:

http://cikm2013.org/slides/ted.pptx

While I understand how to do it (adding a small fixed Gaussian noise), but why would we do it to make the performance worse.

This is not an answer but rather a comment. I would need 50 reputation to comment.

I think in its nature, a recommendation problem is a greedy algorithm to a search/optimization problem. However, the process is biased and local. The users see and react to what they can see, and the algorithm will only reinforce it. Dithering is a way to break up the bias and make the search more global.

What I have been doing is a bit different than dithering, but similar idea. Instead of ranking, I converted the matching score to log linear probability, and "sort" the item/person by probability. An example is, P(a)=0.2, P(b)=0.8, then b has 80% of chance on top, while a has 20% of chance on top.

Like to hear how others hear about it too.


Can thresholds be modeled as weights? I'm trying to model a random threshold as a weight,the threshold should help the error to decrease, the weights are not random, they are 1. It's possible to change the threshold so that the error will be 0?

import numpy as np

# input dataset
X = np.array([[0, 0],
              [0, 1],
              [1, 0],
              [1, 1]])

# output dataset
y = np.array([[0, 0, 0, 1]]).T

syn0 = np.zeros((2, 1)) + 1

threshold = np.random.random_integers(-5, 5)

# forward propagation
l0 = X
l1 = np.dot(l0, syn0)

# how much did we miss?
l1_error = y - l1
print(l1_error)

This is what I have so far..!

In this example, you have 2 classes which are separated with a line with negative slope (if you don't know why please comment). So if you set the slope as +1 then you can not separate classes. The whole point is to learn the line and line is defined with its coefficients (including threshold but threshold is JUST ONE OF THEM!).

PS: If you later want to upgrade to multilayer perceptron and use backpropagation algorithm, then you need to choose randomly different initial weights for mathematical reasons according to the backpropagating error of layers


Is there away to change the metric used by the Early Stopping callback in Keras? When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure) instead of validation loss?
All the examples I have seen so far are similar to this one:
callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')

You can use any metric function that you specified when compiling the model.

Let's say you have the following metric function:

def my_metric(y_true, y_pred):
     return some_metric_computation(y_true, y_pred)

The only requirement to this function is that it takes accepts the true y and the predicted y.

When you compile the model, you specify this metric, similarly to how you specify build in metrics like 'accuracy':

model.compile(metrics=['accuracy', my_metric], ...)

Notice that we are using the function name my_metric without ' ' (in contrast to the build in 'accuracy').

Then, if you define your EarlyStopping, just use the name of the function (this time with ' '):

EarlyStopping(monitor='my_metric', mode='min')

Make sure to specify the mode (min if lower is better, max if higher is better).

You can use it just like any build-in metric. This probably also works with other Callbacks like ModelCheckpoint (but I have not tested that). Internally, Keras just adds the new metric to the list of metrics available for this model using the function name.

If you specify data for validation in your model.fit(...), then you can also use it for EarlyStopping by using 'val_my_metric'.


Why is this Random Forest perfect? I'm learning Random Forest Classifier from a video, where the instructor got a score of 0.44, while I'm getting 0.9985 ( But actually it's perfect). Did I overfit it? If so what is the next step? Shouldn't it 'forget' and 'relearn' every time I compile the code again?
Please check the code below:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

bird = pd.read_csv('image_attribute_labels.txt', sep= '\s+', header=None,
        error_bad_lines=False , warn_bad_lines = False,
        usecols= [0,1,2], names =['imgid', 'attid', 'present'])

bird2 = bird.pivot(index = 'imgid', columns= 'attid', values ='present')

imglbl = pd.read_csv('image_class_labels.txt', sep=' ', header=None, names= ['imgid', 'label'])

imglbl =imglbl.set_index('imgid')

#merging the two arrays
df= bird2.join(imglbl)
#Shuffling the array
df= df.sample(frac=1)

df_att = df.iloc[:,:312]
df_lbl = df.iloc[:,312:]

#choosing the train and test data
df_train_att = df_att[:8000]
df_train_lbl = df_lbl [:8000]
df_test_att = df_att[8000:]
df_test_lbl = df_lbl[8000:]

df_train_lbl = df_train_lbl['label']
df_test_lbl = df_test_lbl ['label']

clf= RandomForestClassifier(max_features=50 , random_state= 0, n_estimators= 100)
clf = clf.fit( df_train_att, df_train_lbl)

print(df_train_lbl.head())
print(clf.predict(df_train_att.head()))
score = clf.score (df_train_att, df_train_lbl)
print (score)

The output for three times was as follow:

1:

imgid              #]
4499     78        #|
7442    127        #|
3200     56        #| The actual data
7271    125        #|
2601     46        #]
Name: label, dtype: int64
[ 78 127  56 125  46] # The predicted ones
0.99875 # The score

2:

imgid
10982    187
11632    198
1536      28
5449      94
8503     145
Name: label, dtype: int64
[187 198  28  94 145]
0.9985

3:

imgid
6782     116
10906    186
2465      43
6660     114
9257     158
Name: label, dtype: int64
[116 186  43 114 158]
0.9985

instead of this:

print(clf.predict(df_train_att.head()))
score = clf.score (df_train_att, df_train_lbl)

better try this:

from sklearn.metrics import accuracy_score
test = clf.predict(df_test_att)
score = accuracy_score(test, df_test_lbl, normalize=False)


What is the use of @ between two matrices I was looking into a repo, where I came across this snippet of code:

priors = x[None, :, :, None, :] @ self.route_weights[:, None, :, :, :]

It basically tried to incorporate weights of an input image matrix.
I am really not sure what is the use of @ out here and what are the advantages of using this. Is there any simple alternative too, like won't simple matrix multiplication work?

According to here,  the matmul function implements the semantics of the @ operator introduced in Python 3.5 following PEP465.

Hence, rather than np.matmul(a,b).

We can use a @ b instead.


Which model to use for multiclass audio classification? I am working on a project wherein I want to classify Tabla taalas(patterns) and I didn't find any dataset regarding it. I am recording them myself and I've ~500 data samples recorded. What model shall I use to classify the patterns if I have less than 500 samples and 6 classes?

500 samples for 6 classes is not so much. You should put aside about 100 samples for validation and 100 for testing, leaving 300 samples for training.
I'm assuming that these drum loop are on the order of 1 second long (0.5-5 seconds). Then I would recommend trying a pretrained audio model, which typically have 1 second analysis windows. For example OpenL3, which is a powerful and easy to use audio embedding. It even has pretrained versions trained on audio.
On top of the audio embeddings try a simple linear classifier like LogisticRegression or a RandomForest.


Does encoding missing data with fixed values help in classification? I have a lot of missing values for some variables in my data (70-80%). I have seen some people deal with missing values this way: encode the variable with missing values as 0 or 1. Where 0 is the value is missing and 1 as non missing.

I want to know if that technique is of any use, because I don't see any valuable information algorithm would able to pick from such variables. Also I am thinking of imputing them using mice but the problem is that in future use, we may not be able to get those variables with missing data, so the train and test set will have different number of columns

You can treat the mere presence of any value as signal - hence the 0 or 1.

What help this would be to your project depends on your dataset.


Given time series data, how to model the frequency of someone changes his job? I am given a time series data vector (ordered by months and years),which contains only 0s and 1s. 1 s represent a person changes his job at a particular a month.

Questions: What model can i use to determine model how frequently this person change his job ? In addition, this model should be able to predict the probability of this person changing his in the next 6 months.

A poisson process ? (I have studied poisson process before however I have no idea when and how to apply it). Any assumptions that data need to meet before applying the poisson process ?

Would love to gather more information on how to model something like this. Thanks

A simple and perhaps somewhat naive approach would be to assume that a person changes jobs at a constant rate and that previous job changes have no influence on future ones. Under these assumptions you could model the job changes as a Poisson process and estimate the rate parameter using MLE (http://en.wikipedia.org/wiki/Poisson_process and http://en.wikipedia.org/wiki/Poisson_distribution).

Of course one should explore how well these assumptions hold in the data. To do this, you could study whether or not job changes are independent of one another by computing the correlation between events at various lags (http://en.wikipedia.org/wiki/Correlation). You could also plot the distribution of time between job change events. If the process is Poisson-like then you should observe little to no correlation between events at any number of lags and the distribution of time between job change events should be exponentially distributed (http://en.wikipedia.org/wiki/Exponential_distribution).


Labeling classes conditionally I am working with a time series predicting whether web traffic will increase or decrease each day compared to the previous day for a given user.

Initially I used binary classes: labeled 1 for next day traffic increases and 0 for traffic decreases (which distributed into 60/40 split). Next I tried something conditionally: if the user has increased traffic for 3 previous days in a row and they increase tomorrow, that is labeled 1, else 0. Otherwise, if the user has decreased traffic the previous 3 days and decreases traffic tomorrow, that is labeled 1, else 0. So 1 doesn't always/necessarily correspond to traffic increases, it depends on the condition which can easily be observed when using this algorithm for real life predictions (by simply looking at the data).

With this 'conditional' dependent label encoding I have gotten much better results. The new binary classes are split 55/45 and accuracy and f1 have greatly improved for testing and training sets.

Is this kind of class labeling acceptable and/or good practice? I think it is positive as I am introducing more data without increasing dimensions but I am worried about mixing up the classes with this approach.

Thank you for your help!

There is no problem doing that, you can define your variable in every way. Just make sure you don't use the variables used to define your Y as predictors.

The only problem is that, after you defined your new variable, your model could be facing nonlinearities which are difficult to model (meaning: I can get a true from Decrease,Decrease,Decrease and from Increase,Increase,Increase, so some variables could be working for the first case, but not for the second).

If you have obtained better results with this new Y variable, I think you are fine.


Decompose annual time series in R I have a time series. Data points are available for each year from 1966 to 2000. Using R, I want to decompose this time series into trend, seasonal and random components. When I run the decompose command, I get the error "time series has no or less than 2 periods". Since my data is annual I have specified a frequency of 1. What am I doing wrong?

Here is the R code that I am using:

dat=c(37.2,37,37.4,37.5,37.7,37.7,37.4,37.2,37.3,37.2,36.9,36.7,36.7,36.5,
      36.3,35.9, 35.8,35.9,36,35.7,35.6, 35.2, 34.8, 35.3,35.6,35.6, 35.6,
      35.9,36,35.7, 35.7, 35.5, 35.6, 36.3, 36.5)

whts <- ts(dat, frequency = 1, start=1966, end=2000)
is.ts(whts)
plot.ts(whts)
whtimeseriescomponents <- decompose(whts)


Orange Text Mining Data Format Is there a special header structure required for text classification problems?

I'm trying to do Tweet classification using the text mining plugin. I have the data in a CSV file, with one column labeled category and containing my two categories, and the other labeled text containing the tweets.

When I load the file with the corpus widget and follow the workflow given at http://orange3-text.readthedocs.io/en/latest/widgets/bagofwords.html the test widget gives the error "Train data input requires a class variable."

I opened the sample bookexcerpts.tab data and discovered it has a different header structure than just category and text.  If I modify my tweet CSV to match it seems to work. Is this a bug, or am I missing a step?

No, no special structure is required for text classification problems in comparison with core Orange.

However — and this holds for core Orange as well as for text addon — if you would like to perform classification one feature must be set as class variable. Otherwise Orange cannot know which feature you would like to predict. When you are reading the data from CSV by default no feature is selected as class variable and hence the message.

There are two possible approaches how one can mark a feature as class variable. The more permanent one, as you already figured, is to modify the data file itself. Second, a bit more user friendly one, is to do this in Orange. To do so you just pass the data to Select Columns widget, drag the desired feature to the Target Variable field and click Send. The data on the output of the widget will have a class variable and can be passed to Test & Score. The following scheme should work without modifying the header of a file:

If you would like to know more about how the Select Columns widget works you can also check its documentation.

So in a nutshell you were missing a step and what happened is expected. If the header does not state that some feature is a class variable — as was the case for your CSV — Orange simply won't mark any feature as class variable. We don't use any heuristics to guess what might be a class variable but solely rely on what is written in the file.


Why won't my SVM learn a sequence of repeated elements I recently started playing with SVMs for a one class classification, I was able to get some reasonable classifications from real data and but was trying to optimize the nu and gamma parameters when I came across this example:

In the code below, I train an SVM with an array of ones, then I present the same array of ones for classification and it classifies all ones as outliers.

import pandas as pd
from sklearn import svm
import numpy as np

nu = 0.01
gamma = 1
ones = pd.DataFrame(np.ones(100))
clf = svm.OneClassSVM(nu=nu, kernel="rbf", gamma=gamma)
clf.fit(ones)
ones["predicted"] = clf.predict(ones)
#Returns -1 for all entries

My question is: why does this happen? I thought this data would be trivial for any parameter configuration.

What you are facing is a small but crucial definition difference:
novelty detection:
The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.
outlier detection:
The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.
OneClassSVM is an Unsupervised Outlier Detection. Therefor your data needs to have outliers in order for the algortihm to detect them. My best guess, why its prediction every input as an outlier is, that if there are no real outliers, everything must be an outlier.
Let me demonstrate this quickly. I adjustet the kernel to linear
import pandas as pd
from sklearn import svm
import numpy as np

nu = 0.5
gamma = 1.0
ones= pd.DataFrame(np.ones(100))

clf = svm.OneClassSVM(nu=nu, kernel="linear", gamma=gamma)
clf.fit(ones)
clf.predict(-1) # -1
clf.predict(1) # -1
clf.predict(1.00001) # 1
clf.predict(2) # 1
clf.predict(10) # 1


How to use a one-hot encoded nominal feature in a classifier in Scikit Learn? I'm working on a genre classification problem on a songs dataset. Since genre is a nominal feature, I used sklearn's LabelBinarizer to get the one-hot encoding for this feature for every row in the dataset. I'm then left with a dataframe(df_train_num) with two columns, both numeric in nature and a Series object for which every row value is a numpy array - the one-hot encoding of the genre. I now want to fit a classifier on this data. What I did was:

svm_classifier = LinearSVC()
svm_classifier.fit(df_train_num,df_train_genre)

This gives me:

  ValueError: Unknown label type: 'unknown'

What exactly is causing this error? Am I not allowed to use a Series object with a DataFrame object in the to fit a classifier? Although replacing df_train_genre with df_train_genre.values so as to pass the numpy array directly to the fit method also doesn't change anything. Same error.

Here is a view of the two pandas objects:

df_train_num.head(5)

Unique_Word_Count   Sentiment Polarity
157277                  126   0.027766
90109                   114  -0.199545
106224                  16    0.000000
221087                  103  -0.058025
247082                  409  -0.170143

df_train_genre.head(5)

157277    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
90109     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, ...
106224    [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
221087    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
247082    [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
Name: Genre_Encoded, dtype: object

I think you should try pd.get_dummies to code the categories; which will create new columns in dataframe and then use that df to pass it to the classifier.


IDE alternatives for R programming (RStudio, IntelliJ IDEA, Eclipse, Visual Studio) I use RStudio for R programming. I remember about solid IDE-s from other technology stacks, like Visual Studio or Eclipse.

I have two questions:

What other IDE-s than RStudio are used (please consider providing some brief description on them).
Does any of them have noticeable advantages over RStudio?

I mostly mean debug/build/deploy features, besides coding itself (so text editors are probably not a solution).

RIDE -   R-Brain IDE (RIDE) for R & Python, Other Data Science R IDEs, Other Data Science Python IDEs. Flexible layout. Multiple language support.
Jupyter notebook -   The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. The Jupyter Notebook App can be executed on a local desktop
Jupyter lab - An extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture.
Radiant – Open-source platform-independent browser-based interface for business analytics in R, based on the Shiny package and can be run locally or on a server.
R Tools for Visual Studio (RTVS) -
A free, open-source extension for Visual Studio 2017, RTVS is presently supported only in Visual Studio on Windows and not Visual Studio for Mac.
Architect -  Architect is an integrated development environment (IDE) that focuses specifically on the needs of the data scientist. All data science tasks from analyzing data to writing reports can be performed in a single environment with a common logic.
displayr - Simple and powerful. Automation by menu or code. Elegant visualizations. Instant publishing.
Collaboration. Reproducibility. Auto-updating. Secure cloud platform.
Rbox - This package is a collection of several packages to run R via Atom editor.
Use below for more IDEs:
RKWard - an easy to use and easily extensible IDE/GUI for R
Tinn-R - Tinn-R Editor - GUI for R Language and Environment
R AnalyticFlow - data analysis software that utilizes the R environment for statistical computing.
Rgedit - a text-editor plugin.
Nvim-R - Vim plugin for editing R code.
Rattle - A Graphical User Interface for Data Mining using R.
How to Turn Vim Into an IDE for R


Spark ALS: recommending for new users The question

How do I predict the rating for a new user in an ALS model trained in  Spark? (New = not seen during training time)

The problem

I'm following the official Spark ALS tutorial here:

http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html

I'm able to build a good recommender with a decent MSE but I'm struggling with how to input new data to the model. The tutorial changes the first user's ratings prior to training, but this is really a hack. They give the following hint:

  9.2. Augmenting matrix factors:

  In this tutorial, we add your ratings to the training set. A better way to get the recommendations for you
  is training a matrix factorization model first and then augmenting the
  model using your ratings. If this sounds interesting to you, you can
  take a look at the implementation of MatrixFactorizationModel and see
  how to update the model for new users and new movies.

The implementation does not help me at all though. Ideally, I'm looking for something like:

predictions = model.predictAllNew(newinput)

But not such method exists. I could go and modify the original RDD, but I think that would require me to retrain the model, so that wouldn't be an ideal solution either. Surely there must be a more elegant way?

Where I am right now:

I think I need to find the latent representation of the new vector. According to the original paper we can compute this like so:

$X_u = (Y^T C^u Y + \lambda I)^{-1} Y^T C^u p(u)$

But when I calculate using the values in the paper, it doesn't match the values from the model. I fix alpha and the regularization parameter, but I think the MLLIB implentation has a different $C^u$ implementation. It's defined here (see line 1304), but not being adept at Scala, this is very hard to reverse engineer for me ...

My current attempt:

V = model.productFeatures().map(lambda x: (x[1])).collect() #product latent matrix Y

Cui =  alpha * np.abs(newinput)
Cui =  (1. + Cui) / (Cui)
Cui[np.where(newinput == 0)] = 0
Cui = np.diag(Cui)

lambdaI = len(np.where(newinput!=0)) * regularization_parameter * np.eye(np.shape(V)[1]) #
term   = np.dot(np.dot(Vt,Cui),V)+lambdaI
term   = np.dot(np.linalg.inv(term),Vt)
term   = np.dot(term,Cui)
term   = np.dot(term,newinput)
latentinput = term

But this doesn't match.

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations.

Of course, once you have any data, and you can rebuild the model to incorporate the user, you can make recommendations. You can do that in Spark but you already know that. This will take too long if you need to add information about new users at runtime. The technique you want is called "fold-in", used to figure out what the new user vector is (approximately) given the items the user interacts with. It's just some linear algebra and does follow from the equation you give.

I dug out an old slide which might help:

The "Cu" is not really different. I added an 'extension' to handle the case of negative input, but it's the same for positive input.

Here's an implementation of fold-in, although I think it's going to be too dense to be of much value:

https://github.com/OryxProject/oryx/blob/2c01d496f93f2825a076eb7fe492aa39a5290aa6/app/oryx-app-common/src/main/java/com/cloudera/oryx/app/als/ALSUtils.java#L74

Computing the new user vector implied by a user-item interaction is fairly easy linear algebra. The tricky part I've found is deciding how much to weight it.

Hope that is a push in the right direction.


Using Machine Learning to Predict Musical Scales It's possible to use Machine Learning techniques to cluster songs into musical-scale groups? I mean: "this song was written in C"... or "this song was written in Am" etc. I made a fast search about the subject and I found no software that can do this. If you know some software, or research (academic papers), related to that subject, could you link it here for me? I'm very interested in that subject but I'm not sure from where I can begin. I have a little experience with Random Forests and Neural Networks, maybe I can accomplish the classification task with one of those algorithms, but, again, I'm not sure which kinda of features I should pass to the algorithm. Thanks in advance.

From a very high level -- You can convert the song to a spectrogram, there are a large number of implementations to do this. From there you can analyze the sound waves. In the case of the key, for instance, the note A is equal to 440 hz. Look into FFT as well.  Hope this helps get you started. I know spotify trains neural networks on spectrograms of songs to find similar songs based on "sound".


Hello, anyone able to direct me to a "cheat sheet" of Neural Network equations with legends? Can anyone here can direct me to a site that provides a cheat sheet of equations for Neural Networks with a legend for the notation?

It can be on any and all aspects of NN, be it forward or back propagation, activation function, CNN, RNN etc.

I have some reading material with the equations and I have worked through them.  I am looking for a site that breaks it down by notation by legend.

For example, without posting a specific equation, a landing page/cheat sheet that identifies each element in the equation, such as: b= bias
w= weight  a=activation the signification of what numbers are vectors in the equation etc.

I work through equations but find most literature lacking in descriptions within the equation, which makes it difficult for someone who is trying to refresh their linear algebra and calculus.

Thank you.

Look further for the "zoo neural network posters"


Is numpy.corrcoef() enough to find correlation? I am currently working through Kaggle's titanic competition and I'm trying to figure out the correlation between the Survived column and other columns.  I am using numpy.corrcoef() to matrix the correlation between the columns and here is what I have:

The correlation between pClass & Survived is: [[ 1.         -0.33848104]
 [-0.33848104  1.        ]]

The correlation between Sex & Survived is: [[ 1.         -0.54335138]
 [-0.54335138  1.        ]]

The correlation between Age & Survived is:[[ 1.         -0.07065723]
 [-0.07065723  1.        ]]

The correlation between Fare & Survived is: [[1.         0.25730652]
 [0.25730652 1.        ]]

The correlation between Parent-Children & Survived is: [[1.         0.08162941]
 [0.08162941 1.        ]]

The correlation between Sibling-Spouse & Survived is: [[ 1.        -0.0353225]
 [-0.0353225  1.       ]]

The correlation between Embarked & Survived is: [[ 1.         -0.16767531]
 [-0.16767531  1.        ]]

There should be higher correlation between Survived and [pClass, sex, Sibling-Spouse] and yet the values are really low.  I'm new to this so I understand that a simple method is not the best way to find correlations but at the moment, this doesn't add up.

This is my full code (without the printf() calls):

import pandas as pd
import numpy as np

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv")

survived = train['Survived']
pClass = train['Pclass']
sex = train['Sex'].replace(['female', 'male'], [0, 1])
age = train['Age'].fillna(round(float(np.mean(train['Age'].dropna()))))
fare = train['Fare']
parch = train['Parch']
sibSp = train['SibSp']
embarked = train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])

On a side note, I don't think correlation is the correct measure of relation for you to be using, since Survived is technically a binary categorical variable.

"Correlation" measures used should depend on the type of variables being investigated:

continuous variable v continuous variable: use "traditional" correlation - e.g. Spearman's rank correlation or Pearson's linear correlation.
continuous variable v categorical variable: use an ANOVA F-test / difference of means
categorical variable v categorical variable: use Chi-square / Cramer's V


Why GridSearchCV returns nan? I am using gridsearchcv to tune the parameters of my model and I also use pipeline and cross-validation. When I run the model to tune the parameter of XGBoost, it returns nan. However, when I use the same code for other classifiers like random forest, it works and it returns complete results.
kf = StratifiedKFold(n_splits=10, shuffle=False)

SCORING = ['accuracy', 'precision', 'recall', 'f1' ]

# define parametres for hypertuning
params = {
    'Classifier__n_estimators': [5, 10, 20, 50, 100, 200]
}

XGB = XGBClassifier()
UnSam = RepeatedEditedNearestNeighbours()

pipe = Pipeline(steps=[('UnderSampling', UnSam ), ('Classifier', XGB)])
# ___________________________________________

mod = GridSearchCV(pipe, params, cv =kf, scoring = SCORING, refit='f1', return_train_score=True)
mod.fit(X_train, y_train)

Here is my code and when I run it, the following results are obtained:
{'Classifier__n_estimators': 5}
__________________________________________________
F1 :  [nan nan nan nan nan nan]
 Recall :  [nan nan nan nan nan nan]
 Accuracy :  [nan nan nan nan nan nan]
 Precision :  [nan nan nan nan nan nan]

Another thing that is weird is that when I apply the same code for tunning the penalty in Logistics Regression, it returns nan for l1 and elasticnet.
kf = StratifiedKFold(n_splits=10, shuffle=False)

SCORING = ['accuracy', 'precision', 'recall', 'f1' ]

# define parametres for hypertuning
params = {
    'Classifier__penalty': ['l1','l2','elasticnet']
}

LR = LogisticRegression(random_state=0)
UnSam = RepeatedEditedNearestNeighbours()

pipe = Pipeline(steps=[('UnderSampling', UnSam ), ('Classifier', LR)])
# ___________________________________________

mod = GridSearchCV(pipe, params, cv =kf, scoring = SCORING, refit='f1', return_train_score=True)
mod.fit(X_train, y_train)

The results are as follows:
{'Classifier__penalty': 'l2'}
__________________________________________________
F1 :  [  nan 0.363   nan]
 Recall :  [   nan 0.4188    nan]
 Accuracy :  [   nan 0.7809    nan]
 Precision :  [   nan 0.3215    nan]

By default, GridSearchCV provides a score of nan when fitting the model fails.  You can change that behavior and raise an error by setting the parameter error_score="raise", or you can try fitting a single model to get the error.  You can then use the traceback to help figure out where the problem is.
For the LogisticRegression, I can identify the likely culprit: the default solver is lbfgs, which cannot handle L1 or ElasticNet penalty.  Use saga.
I don't immediately see an issue with the XGBoost model or parameters.  Get the error traceback using the first paragraph, and search/ask that as a separate question if needed.


Reinforcement Learning on data only (NO emulators) My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...).

It is too costly for us to emulate the agent (executing the action and assessing the reward) meaning our only option is to learn the optimal policy on our dataset using an off-policy algorithm. However I still do not know how to do that, Q-learning, while off-policy, still require an emulator.

Could you please provide me some guideline on how to do that? Which type of algorithm should I use?

Q-learning does not in fact need to be online or need an emulator, it can learn exclusively from experience replay. If you put all your history into a table or state, action, reward, next state and then sample from it, it should be possible to train your agent that way.
To do this, you will need to skip the algorithm steps that take actions and store results. The algorithm will then learn from the data you have. It will just not be possible to collect more. Depending on the problem you are trying to solve, this could be OK, or it may inhibit learning.
RL algorithms learning optimal control in complex environments benefit from sampling near to their current policy, so it is possible in your case that your agent will reach a limit on what it can learn from historic data. It may end up still quite far from optimal behaviour, although it should stand a reasonable chance of improving on the best that the historic data shows.
If you need to use function approximation (e.g. a neural network) due to size of the state, action space, then take extra care, because it will be hard to detect whether the action values have converged correctly. This is because you are learning the optimal Q values, and you will have no test data that demonstrates what those should be (to collect that data, you need to follow the optimal policy and measure the total reward).
Here is roughly what an experience-replay-only Q-learning algorithm would look like:

Input: History $H$, consisting of rows of $S,A,R,S'$
Initialise the NN for calculating $\hat{q}(s,a)$
Repeat until NN converges:
$\qquad$ Sample $S,A,R,S'$ from $H$
$\qquad$ $tdtarget \leftarrow R + \gamma \text{max}_{a'}[\hat{q}(S',a')]$
$\qquad$ Train NN single step, $\hat{q}(S,A) \rightarrow tdtarget$

You can make use of mini-batch processing to generate multiple $tdtarget$ values at once and train on them. A worthwhile improvement for stability is to use a frozen copy of the neural network when calculating the $tdtarget$ value, and update it only every N steps to be a copy of most recent network, with N maybe 1000 steps.


Where can I find a social network image dataset? I am supervising a programming project whose goal is to detect offensive images on social networks.
I would like to have a representative dataset of social network images. It would be best if the dataset were already classified. Otherwise, the classification of a (smallish) dataset could be a part of the project.

I looked around the internet and searched various dataset collections that are freely available, but I haven't been able to find anything appropriate.

What is the best way to get a large social network image dataset these days?

Google just released a beta search tool for datasets. This can help you find any kind of datasets you want: https://toolbox.google.com/datasetsearch


Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor' I want to store JSON Data into MySQL Database using Python.
I used dataframe of pandas. I found that to_sql() can do this job easily.

Python Code:

jdata=json.loads(json_data)
df=pandas.DataFrame(jdata)
df.to_sql(con=con, name='crashTable', if_exists='replace', flavor='mysql')

Output:

TypeError: to_sql() got an unexpected keyword argument 'flavor'

I got this error message. How to resolve this error?

Based on the documentation 0.22 and 0.24.1, the flavor does not exist in the argument list of the to_sql method. You're probably running the 0.24.1 version which does not need flavor argument.


classification feature selection I have a system which sends invitations to users to participate in online questionnaires and want to use machine learning in order to predict the likelihood of fulfilling the questionnaires in a predefined time ( i.e. within 1 day , 2 day , 3 days ,a week, 2 weeks, etc) based on various feature related to the users to whom the invitations are sent to , details of the questionnaires ( i.e. how long are they , their topic ,etc.) , other contextual data ( time of day , day of year , in which media the invitation are sent - i.e. sms / email etc).
I can train with positive examples ( invitations that were responded to by users ) and negative examples ( invitations that WERE not responded to ) , however , I'm not sure how to take into consideration the "predefined time" into the feature vector. for example, should I simply include a feature of "days since invitation was sent" and in the positive examples include the time , and in negative examples replicate each example X each of the predefined times to indicate that the users didn't respond at all ?

Any advise would be welcome  !

You have a problem of data leakage.
The "days since invitation was sent" feature contains all the information on the concept. Therefore, adding it as a feature will prevent most common classifiers from using rules based on other features and will lead to misleading results.

Duplicating the negative row for each predefined time value will lead to a different distribution than the one you should be tested one, which need needles problems.

Note that even that way, the feature might leak information (e.g., a small ratio of the positives will replay in two weeks while its distribution in the negatives will be higher, making it a good predicator for negative).

In case that you will have more complex representation of time (e.g., was the email sent on weekend? was the reply sent on weekend), the duplication of rows will be more complex.

I would try a different direction. It is likely that the negative rows has no influence on the time to reply in the positive ones.
Therefore I would do a first research aimed to differ between replies and no replies without using the predefined time.

After that, do a second research only on the positives where the concept will be the predefined time, aiming to find what is influencing out given that the user had replied.


Is it always possible for validation accuracy to be as high as training accuracy? I have a very small dataset (40 training examples, 10 validation examples, 120 classes) for which I'm getting very high accuracies with a very simple model in Keras (batchnorm, flatten, and dense layers only).

My training accuracy is 94-95% and validation accuracy is 76-78%. I know it's overfitting and I have tried a few things. The data is not images, so I cannot augment the data. I also cannot add data because it's a specific type. I'm using two dropout layers with 0.5 levels, and the architecture is very simple so I don't think I can reduce the architecture complexity. I can paste the model if anyone likes.

My question is: Is there ever a situation where validation accuracy cannot be as high as the training accuracy? Is there a limitation based on the size of the dataset? Or is it ALWAYS possible for validation accuracies to match training accuracies and the network just needs the right parameters?

Yes, it is possible to have a situation in which validation accuracy cannot be as high as training accuracy. Any situation in which noise (as opposed to generalizable properties of the feature set) in the training set is more predictive of the target variable within the training set would produce this.

Consider a situation in which a random property of the training sample considered is perfectly predictive, but this is found not to be true of all other examples outside the sample. The predictive power in the training set would be perfect, but outside the training set, less than perfect. This phenomenon is broadly referred to as "overfitting".

For example, let's consider a case where you have a set of fruit data and you're trying to establish whether a given fruit is an orange or a tangerine. You have 4 features- the circumference of the fruit around the stem-medial axis, the height of the fruit across the stem, a numerical value of the hue of the fruit skin, and the first letter of the amateur baseball team whose home park is closest to the field the fruit was grown in. Let's imagine that by some baffling coincidence, the baseball team letter in the training set was perfectly predictive of whether the fruit would be a tangerine or orange.

We can imagine that this would not hold true across the country or the world, which would produce a situation where the training set accuracy would be perfect, but the validation set accuracy would not be able to approach that using the same methods.


How to use boolean data in DecisionTreeClassifier in sklearn? I am trying to build a decision tree using python and sklearn DecisionTreeClassifier.

One of the data_type used for splitting the tree  is Boolean(let it be x).However the tree that is generated contains comparisons like x<=0.5 .
This does not make sense.

Can anybody suggest how to use boolean values in Decision trees.

Why would this not make sense? It tries to create a split with maximal separation between signal and background. Any float >0 and < 1 will result in the same split.


Finding Feature Importance in CNN's? Let's say I have images of cars. For each image in the dataset, I have let's say 3 pictures of the same car but in different angles.

1) The first image is the picture of the car from the front.
2) The second image is the picture of that car from either of the sides.
3) The third image is the picture of that car from the back.

If I have 1000 cars, I would have 3000 images as my dataset. Is there a way to find which angle of the car represents its label the most? In other terms, Which angle of the car image is most helpful in determining whether a particular image is a car or not?

My thoughts were to build a random guessing model to determine the lower bound of our loss and then iteratively use each of angle of images as train and predict on the same angle and come up with a loss score. Ex: First I train and predict with all images of angle 1 and come up with a loss score. So If I have 3 angles of images, I would end up with 3 loss values. So the angle with the lowest loss score is determined as the winner. But the problem is, I'm not sure if this is approach is right because if I have multiple classes in the dataset then the dominant class would have lower loss compared to any other non-dominant classes and moreover we would not be using the entire data at any point of time and so would possibly end up with wrong conclusions.

Thanks for replying in advance. This question has been lingering in my head for a long time now.

If you are training a classifier on only one angle, you will overfit to that particular position - it won't be able to recognise the same car from a different angle. It may learn something, I guess, and you could test generalisation by testing on the unseen images from the other two angles. So train on each set individually and test on all 3 sets individually with the usual caveats on training vs testing.

A better approach, however, would be to train on all your data. The network will automatically learn what is the most efficient and you can then look at the results on each of the 3 test sets to see which gives the best results. I am guessing that this is an example problem, so more generally it makes sense to do a statistical analysis on why the classifier fails. If you can identify which features make classification hard, you can work harder to create a training strategy that will improve the model. The main goal would be to make a model that generalises. I suspect that training using extra information from other angles will only improve the effectiveness for all cases.


An average of averages problem, or two valid results? From an event log I have derived the days between each user's logins. This leaves me with a dataframe like this:
I then took the average of column 'days_between_logins' which gave me a result of 15.9 days.
But then, if I calculate the average number of days between logins on an individual user basis and then average that across all users, the result is 34 days.
I think I have an average of averages problem? Then again, I think both results are valid.
I would really welcome any thoughts or supporting/opposing conclusions.

In you first approach you take into account the number of logins for each user, which you do not do in the second approach. Take the following example with two users with a different amount of logins:
$
u_1 = {7, 5, 10, 10, 8} \\
u_2 = {1}
$
Using the first approach the average would be 6.8 days whereas the second approach would give an average of 4.5 days. The second approach does not take into account that we have more observations for the first user (with a higher number of days between logins) than for the first user. Which approach is the best would depend on what question you are trying to answer, but I think generally the first approach would be more accurate in getting the average time between logins.


Estimate battery voltage based on scheduled events and previous behaviour My goal is to estimate if a battery will have enough charge for certain other systems to be powered. The power state of the other systems is recorded (i.e. if they are turned on or not), as well as the times when the battery is being charged by solar panel during sunlight. These records are several month in the past.

My question is now, in what topic/algorithms should I research when I want to use the previous data and events to estimate/extrapolate the voltage into the future (to make scheduling of powering systems easier, such as canceling them if the battery will be drained too much)? The upcoming charging times can be calculated, and it'll also be known when a system will be powered.

All the datasets (i.e. voltage, charging times, events) are available as timeseries (but with different timestamps/sample rates).

One of the simplest ways to get to some predictions would be to use a model like ARIMA, which looks at recent previous observations to predict a number of steps ahead. ARIMA stands for Autoregressive Integrated Moving Average:

Autoregressive means that something looks at its own (auto) historical values
Integrated refers to the _differencing, a step tp help remove non-stationarity
Moving Average just means the model also takes the moving average into account (helps keep predictions in reasonable ball park)

Here is a good little explanation of the main points in more detail.

In your case, given there are factors such as weather and time of data to be used, you would probably benefit from using seasonality. For this there is an extended ARIMA, called SARIMA - where S stands for seasonal.
There is an implementation in Python's statsmodels package - or if you want to use R, then maybe take a look at the forecast package by the great Rob Hyndman, or the sarima package.

If you want to use something a little more modern and cutting edge, then you are really talking about Recurrent Neural Networks (RNNs). I am not sure how familiar you are with there? One key term you should understand: LSTM - a type of unit that considers past data, current data and maintains a state of your model at a specific point in time.

Have a look at a walkthrough (example) to see if it makes sense to you.

ARIMA type models are easy to get running and the results are easy to interpret, so you really know what is going on. The performance of things like RNNs should be better, but it is rarely a short walk to success... of course it will depend on your data.


How to implement single Imputation from conditional distribution? In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as:

I did not find any code implementation of this approach. My question is, how to implement it? Should we integrate the distribution w.r.t an assumed interval of $\bf{x}_{mis}$? Otherwise, is this just an intuitive mathematical representation that should be understood but the implementation is different.

[*] Theodoridis, S., & Koutroumbas, K. “Pattern recognition. ” Fourth Edition, 9781597492720, 2008

This is just an intuitive explanation of a group of a strategy for imputing missing data.

In practice, the distribution $P(x_{mis}|x_{obs};\mathbf{\theta})$ is unknown and can be estimated at best. The best way to estimate this probability is use-case specific. Understanding how the training data was collected can help you in estimating/defining this conditional distribution.

In practice, we often do not try to get a good estimation. Keeping things simple and assuming all features are sampled from a normal distribution might get you started.

This is $x_{mis}$ follows $N(\mu, \sigma)$ where

$\mu = \sum_i^N \frac{x_{obs,i}}{N}$
$\sigma = \sum_i^N \frac{(x_{obs,i}-\mu)^2}{N-1}$

. However, such assumptions are rarely realistic and will do guarantee good models.
See this.


Machine Learning in Insurance industry Is anyone here aware of any research study or industrial application where ML algorithms have been employed to reduce Adverse Selection and Moral Hazard risk in Insurance market?

I haven't seen the specifical cases you mention , but I know of machine learning within insurance, and the types of models I've seen being used always depended on the criticality of the data and use of the results.

So a more general answer: if they need to justify all modelling decisions (as is often the case in insurance), they stick to more traditional models, such as logistic regression, tree-based methods and the like. More cutting-edge models are considered black-boxes.

If the project if more about analysing ad-hoc topics and non-core businesses, but which could perhaps support core business decisions (or otherwise), then anything is game: from stacked bidirectional LSTMs on word embeddings of their contracts to capsule networks on images scraped from social media websites.

I suppose you could model Moral Hazard similarly to fraud, in which case there are plenty of examples to be found online. I think there was even a Kaggle competition on that.


XGBoost Tree 'starting feature break' I am fairly new to learning the XGBoost algorithm and had a question about how the algorithm knows which feature to break the tree on first.
Here is my understanding (and please correct me if I'm wrong):
Each tree starts as a single leaf and all of the residuals go to that leaf. Then we calculate the similarity score and try to split amongst some feature characteristics. For example, say we only had one feature - height. We could split the residual if its respective x-value was say height > $180cm$ or height < $180cm$. And then proceed to calculate the similarity scores and the gain for the new leaves and continue down our tree.
My question is now say we had two or more features such as height, age, weight, education, etc. How does the algorithm know where to break the initial residuals? Does it compute it over all possible features and finds the best gain? I can imagine for large datasets to go through all features, then break it at some threshold and compare must take a long time.
Or does it start at some random feature and work its way down kind of like a random forest? I know there is a parameter within the algorithm that allows you to set the percentage of features per tree so adding on to my question once we specify this percentage how does it pick the features and more importantly once those features are picked how does it know which one to pick to break the initial leaf of residuals.
Edit: I tried reading the paper where the algorithm was first published but found it a bit too difficult to conceptualize and understand.

Each tree starts as a single leaf and all of the residuals go to that leaf. Then we calculate the similarity score and try to split amongst some feature characteristic. For example say we only had one feature - height. We could split the residual if its respective x-value was say height > 180 or height < 180. And then proceed to calculate the similarity scores and the gain for the new leaves and continue down our tree

Here it seems like you are describing just a decision tree. Watch this tutorial to understand it a bit more

My question is now say we had two or more features such as height , age , weight , education etc. How does the algorithm know where to break the initial residuals? Does it compute it over all possible features and finds the best gain? I can imagine for large datasets to go through all features , then break it at some threshold and compare must take a long time.

Again you are talking about the decision tree and its  complexity, but yes, for large datasets is overly computationally expensive. To avoid this several strategies are used, as sampling, feature selection or quantization

Or does it start at some random feature and work its way down kind of like a random forest? I know there is a parameter within the algorithm that allows you to set the percentage of features per tree so adding on to my question once we specify this percentage how does it pick the features and more importantly once those features are picked how does it know which one to pick to break the initial leaf of residuals.

The random forest does random feature selection per split of the tree. The trees are grown greedy, so it tries all possible splits and chooses the best. If the original paper was too difficult try following this tutorial about random forest


Clustering a labeled data set I have a large labeled dataset with 29 classes. Is is possible to use a clustering algorithm (like k-means) in this dataset, or it's not possible since clustering algorithms are unsupervised ?

You can do many things:

Forget about the labels: just use the features that are not labels and cluster along those features using the k-means algorithm (or another).
Forget about the features: this is the dummiest way of clustering. Cluster the data in 29 clusters according to the labels that they have. If you want less clusters, you can compute the centroids of the classes and use them to join clusters of different labels.
Use everything: create a categorical variable refering to the class that every example belongs to. Then, with this new variable and all the features perform a classical clustering algorithm.

The way to proceed depends on if you want to use the labels or not, and how much importance you want them to have.


Calculating derivative of error at point x with respects to weight w_j I don't know how the equation below goes from line 2 to 3 after the derivative term is moved inside the brackets.  Specifically, how is it calculating the derivative of log(y_hat)?  Also, if anyone can point to a good textbook or website to learn this stuff.  I've just started on this free course (option to pay for extras) at edx that is so far pretty good because it has easy-to-understand lectures and jupyter notebook assignments that get you coding:  https://courses.edx.org/courses/course-v1:Microsoft+DAT256x+1T2019/course/

Based on chain rule we have:

$$\frac{\partial \mbox{log}(f(x))}{\partial x}=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=f(x)}\frac{\partial f(x)}{\partial x}=\frac{1}{z}\Bigr|_{z=f(x)}\frac{\partial f(x)}{\partial x}=\frac{1}{f(x)}\frac{\partial f(x)}{\partial x}$$

More close to the notations used in the question:

$$\frac{\partial \mbox{log}(f_{\mathbf{w}}(\mathbf{x}))}{\partial w_j}=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}=\frac{1}{z}\Bigr|_{z=f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}=\frac{1}{f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}$$

Now by setting $f_{\mathbf{w}}(\mathbf{x})=\hat{y}$, or $f_\mathbf{w}(\mathbf{x})=1-\hat{y}$, line 2 to 3 follows. For example for $f_\mathbf{w}(\mathbf{x})=1-\hat{y}$:

$$\frac{\partial \mbox{log}(1-\hat{y})}{\partial w_j}
=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=1-\hat{y}}
\frac{\partial (1-\hat{y})}{\partial w_j}=\frac{1}{z}\Bigr|_{z=1-\hat{y}}\frac{\partial (1-\hat{y})}{\partial w_j}=\frac{1}{1-\hat{y}}\frac{\partial (1-\hat{y})}{\partial w_j}$$

Note that input $\mathbf{x}$ to the network is constant w.r.t. changes in  $\mathbf{w}$. Also $x|_{x=y}$ means replace $x$ with $y$.


Association rules in R / Subset rules based on feature I tried using the apriori algorithm in the arules library for association rules mining in R. There's two thing I'd like to ask:

Why are the lhs and rhs of the rules in intervals? For example U get something like this
lhs : Variable1[0,4] - rhs : Variable2[0,3]

How can I subset the rules extracted for a certain variable? Say I want rules for Variable 1. How can I  subset the result to get rules containing that variable only?

I do not recall lhs or rhs having intervals. Can you share an example?
You can use the "appearance" parameter when calling apriori. You can find an example in the documentation of APappearance-class  of arules or at
http://r-statistics.co/Association-Mining-With-R.html under section "How to Find Rules Related To Given Item/s"


Find type of relation between variables I have time-series data of a few metrics. I know which metric is the response variable and the independent ones. I need to fit a model between them. The relationship could be linear, quadratic, logarithmic, piecewise linear, multiple linear, etc. Basically, it could be anything.

Is there any technique / properties I could use to find the relationship between the metrics and fit a model?

Right now, I have written a brute-force script in R.
For example, I have response variable A which depends on X1, X2 and X2.

  A = C1*f1(X1)+C2*f2(X2)+C3*f3(X3) is my model.

My script tries all possible combinations of f1, f2 and f3.
By combinations, I mean that I initialy start with all linear, then one of them is quadratic, then cubic, then logarithmic, etc.

I am using lm().

I then choose the model which has the least AIC as my final model.

This obviously takes too long.

I definitely need an automated way of finding this model.

Could you suggest a better way to do this?

How about using neural networks instead of linear models? The neural net will have an easier time learning arbitrary non-linear relationships.


Validation loss differs on GPU vs CPU I am consistently seeing higher validation loss when I train & evaluate a model on AWS GPU vs local CPU.

I am using the exact same train/eval datasets and the exact same Tensorflow code and configuration for both runs. Has anyone else experienced this before and know how to explain?

See screenshots below comparing GPU vs CPU loss for equivalent steps.

Since one could see there is no difference in the minmal loss value, this could well be an issue of differnce in floating point precision with respect to CPU and GPU. You should try to cross check unit operations in the intermediate layers(Considering its an ANN) and look for possible discrepencies to make sure it is a floating point precision variance.


Is there a term for "this month last year" in a report? I'm building a report that has month over month data, but also "this month last year". Is there a better/standard way of describing this?

Same period last year is what you want.

BTW, your question does not appear to be about data science.

References

https://msdn.microsoft.com/en-us/library/ee634972.aspx

http://www.wiseowl.co.uk/blog/s2477/same-period-previous-year.htm


How to handle highly Imabalanced classification? I have been dealing with a classification problem. Real issue is the imbalance here
I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification  Personally I am not a big fan of oversampling/under-sampling since they mess the distribution up. I also tried stratified sampling so that same proportion is kept while training but its not working. For my present approach I take a sample of -ve points (say 100,000) and then train with 300 +ve samples and getting an ROC-AUC of ~85 but almost no case predicted reaches probability >51% (after using model.predict_proba). Any tips on handling such an extreme imbalance? additionally I would only like to stay with tree based or boosting models since they provide interpretability (hence avoiding neural network, anomaly detection , autoencoder for now).
Any hints, resources appreciated!
Below is the code I am using.
df = read_data(path)
Pkl_Filename = path + "/catboost.pkl"
# if not os.path.exists(Pkl_Filename):
X = df.drop(ep_config['target_field'], axis=1)
y = df[ep_config['target_field']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, stratify=y)
clf = CatBoostClassifier(eval_metric='Recall', use_best_model=True, scale_pos_weight=...)
sm = SMOTE(random_state=27, ratio=1.0)
X_train, y_train = sm.fit_sample(X_train, y_train)
smote = clf.fit(X_train, y_train, eval_set=(X_test, y_test))

# smote_pred = smote.predict(X_test)

# Checking accuracy
# accuracy_score(y_test, smote_pred)
# clf.fit(X_train, y_train, eval_set=(X_test, y_test))
with open(Pkl_Filename, 'wb') as file:
    pickle.dump(smote, file)
# else:
#     clf = pickle.loads(Pkl_Filename)
logging.info('the test roc is :{:.6f}'.format(roc_auc_score(y_test, smote.predict_proba(X_test)[:,1])))

have you already tried using only very little of the -ve cases? So for example to train your model on 900 points total, 600/300? Then stratified sampling should still work fine. Then I'd evaluate your model based on it's ability to predict -ve cases and just monitor the performance it get's on the (in your case) gigantic test dataset that the model hasn't seen before.
Also, you could generate "synthetic" +ve cases by bootstrapping from the real data to see how that influences the model. Similar to what is described here: https://sci2s.ugr.es/keel/pdf/specific/articulo/dupret_science.pdf


What applications does linear programming have in data science? I'm currently learning about linear programming in my degree. I'm wondering how this is relevant to anything in data science?

Whenever you have an optimization problem the first question that you have to ask yourself is.

Can I make it a Linear Programming problem?

For python I normally use Gurobi. Here is a basic example to get started: https://www.gurobi.com/resources/food-manufacture-i/

You can also do Machine Learning with it, that is the hot topic nowadays, but if you can make it a linear programming problem you will achieve optimality.


Classifying potential triggers for events (many-to-many relationship): Which algorithm to use? I want to use Data Mining/Machine Learning for a problem and I'm not sure if there is a standard algorithm for my problem.

The problem is as follows: There is a set of Events and a set of Potential Triggers. Each trigger can give rise to none, one or several Events. I want to classify Potential Triggers based on their features into ones that do not cause any event and ones that do. So far this is a standard classification task. Now comes the twist: For some of the Events, there are multiple Potential Triggers. This usually means that one is the real trigger and the others have been labeled accidentally. If the algorithm generates a rule (assuming rule learning for the moment) that covers one of the potential triggers for an event, this event shall be considered "explained" so that the other potential triggers for this event do not have to be classified as triggers for this event any more.

The twist in the problem smells a bit like set cover (https://en.wikipedia.org/wiki/Set_cover_problem), so I'm wondering if there is a good algorithm at all.

Is there a published algorithm? And does my problem have a standard name under which I can search for further information?

I did not find an existing algorithm, but I have since published an algorithm for "set cover rule mining" myself. It is called GIMO: https://doi.org/10.1016/j.eswax.2020.100040


Similarity between two words I'm looking for a Python library that helps me identify the similarity between two words or sentences.

I will be doing Audio to Text conversion which will result in an English dictionary or non dictionary word(s) ( This could be a Person or Company name) After that, I need to compare it to a known word or words.

Example:

1) Text to audio result: Thanks for calling America Expansion
will be compared to American Express.

Both sentences are somehow similar but not the same.

Looks like I may need to look into how many chars they share. Any ideas will be great. Looks a functionality like Google search "did you mean" feature.

The closest would be like Jan has mentioned inhis answer, the Levenstein's distance (also popularly called the edit distance).

  In information theory and computer science, the Levenshtein distance
  is a string metric for measuring the difference between two sequences.
  Informally, the Levenshtein distance between two words is the minimum
  number of single-character edits (i.e. insertions, deletions or
  substitutions) required to change one word into the other.

It is a very commonly used metric for identifying similar words. Nltk already has an implementation for the edit distance metric, which can be invoked in the following way:

import nltk
nltk.edit_distance("humpty", "dumpty")

The above code would return 1, as only one letter is different between the two words.


Determining if time series follows a pattern I was wondering if anyone had any idea how to solve this problem.

So basically I have a dataset where some person approximately comes at some regular interval and I don't know what that interval is. I need to determine if the person comes in at approximately regular intervals, not necessarily what is the specifically value of the interval. For example if a certain person comes to my house to deliver milk over some period on say

Week 1: Mon, Wed, Fri,

Week 2:Mon, Wed, Fri

Week 3:Mon, Thu, Fri

Week 4:Mon, Wed, Fri

Week 5:Mon, Wed, Fri

Week 6:Mon, Wed, Fri

Week 7:Mon, Wed, Fri

Week 8:Mon, Thu, Fri

So as we can see out of these 8 weeks only in 2 weeks the person didn't come on Wednesday and instead came on Thursday which can be attributed to maybe a holiday the day before. So the solution to this example is that the person does follow a regular pattern.

Similarly this is another example. Say the person came on -

Mon, Thu, Sun, Wed, Sat, Tue, Fri, Mon, Wed, Sun

where this person follows a regular pattern because except for the last Wednesday he comes every fourth day.

This is an example of where a person doesn't follow a pattern, say the person came on

Mon, Wed, Sat, Fri, Thu, Fri, Wed, Fri, Sun, Sun, Sat

I have to do this for hundreds of different people.

Another equivalent problem is if I know which days over a certain period of time (say a month) some person arrives I need to determine if they follow some pattern or not

I thought about trying to fit the data into a sinusoidal curve but I'm not sure if it will work when I have a person who comes every say Mon, Tue, Fri or the 7th of every month or the first and third monday of every month etc.

I'm open to any method as long as it has good accuracy. Also, depending on whichever algorithm you think is best, if possible could you share a link to some code which solves a similar problem so I can get a general idea of how I'm supposed to implement my algorithm. I'm pretty new to machine learning / data science. Thank You!

First of all, digitalize your data on a regular time scale with a time delta shorter than the shortest assumed period. For instance, in the case of the milkman, you could create a table with one record per day having the value 1 if the man come and 0 if not.
The table would look like a regular time-serie :

day, presence
  1, 1
  2, 0
  3, 1
  4, 0

Then perform a autocorrelation analysis of your time-serie. This is a mathematical tool often used in signal processing that consists in, as suggested by @MichelGrogan in his comment, calculating the correlation between the original serie and copies of the serie delayed by distinct time deltas (1 day, 2 days, ...). The result can be plotted as values of correlation in function of the lag value and is called an autocorrelation spectrum. Read autocorrelation for more details.

The max values of the spectrum will give you the values of the most important frequencies observed in your data and confirm that pattern exists. If you do not see clear max values, you can conclude there are no pattern in your data.

Here is an example in Python from your data :

import numpy as np
import matplotlib.pyplot as plt

day =['mon','tue','wed','thi','fri','sat','sun', 'mon','tue','wed','thi','fri','sat','sun', 'mon','tue','wed','thi','fri','sat','sun' ]
num =[    1,    2,    3,    4,    5,    6,    7,     8,    9,   10,   11,   12,   13,   14,    15,   16,   17,  18,    19,   20,   21]
pres=[    1.,    0.,   1.,   0.,   1.,   0.,  0.,    1.,   0.,   1.,   0.,    1.,  0.,    0.,   1.,    0.,  0.,    1.,   1.,    0.,   0.]

corr=np.correlate(pres,pres,'full')
lag=[i-(len(corr)/2) for i in range(len(corr))]
plt.plot(lag,corr/np.max(corr))

And the spectrum plot :

As you can see there are clear maximum at 2 days and 7 days lags.


Is there an equivalent of the Bandit package in r (Used in implementation of Multi arm bandit algorithm) in python? I want to implement MAB(Multi arm bandit algorithm ) in python. There is a package called bandit in R Check here, is there anything similar to this in python

Check this one from a book on MAB
There's also a flaskimplementation if you want a web service


Error in Model with caret Package I am running the following code with data traing with targetBuy as response
variable

# load the library
library(caret)
library(klaR)

# define training control
train_control <- trainControl(method = "repeatedcv", number = 10, repeats = 3)
model <- train(TargetBuy ~ ., data = traing, trControl = train_control, method = "nb")
print(model)

Getting the following error

  Error in train(TargetBuy ~ ., data = traing, trControl =
  train_control,  :     unused arguments (data = traing, trControl =
  train_control, method = "nb")

Please read carefully caret::train since S3 method for class formula is slightly different than default S3 method for caret::train (and takes different arguments)

This should do the work:

model <- caret::train(y = traing$TargetBuy, x = traing[, colnames(traing) != 'TargetBuy'], trControl = train_control, method = "nb")

You can also use model.matrix, but then you need to remember that it removes missing values. See this post for solution:

model <- caret::train(y = factor(traing$TargetBuy), x = model.matrix(TargetBuy ~ ., data = traing), trControl = train_control, method = "nb")


DQN - target values vs action values? I'm trying to understand the difference between target-values and action-values in Deep Q Networks.

From what I understand, action-value tries to approximate the reward of a given action (at some state). The target-value is also an approximate of the reward. How are they different?

Reference
- https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

Action value and target value in DQN refer to the same thing in terms of what they measure, but are obtained and used in different ways.

An action value does not approximate the reward of a given action, but represents the related concept of expected return - the expected discounted sum of future rewards when taking action $a$ in state $s$. You see it very often associated with the action value function $q(s,a)$.

Action values are not inherently approximated, they are more conceptual than that. However, in practice in Reinforcement Learning (RL), the action value function must be learned, and some form of approximation used based on observed data. When an action value function it is approximated by a neural network during learning you may see it written $\hat{q}(s,a,\theta)$ which makes explicit that the NN is approximating some "true" function that you don't know, and that it is parametrised by $\theta$, the neural network weights and biases.

During learning, the agent takes actions and observes resulting states and rewards. It must use these observations to improve its estimates of values. To do so in DQN, the agent constructs a temporal difference (TD) target - for single-step Q-learning this is $G_{t:t+1} = r_{t+1} + \gamma\text{max}_{a'}\hat{q}(s_{t+1},a',\theta)$. This is the "target value" and there are multiple ways to construct it, giving rise to variants of RL algorithms.

The value $G_{t:t+1}$ is also an estimate of expected return. Technically it is a sampled estimate, which may vary and may be biased (due to starting conditions of the neural network). However, it includes some real experience, so may be used to improve the neural network through training. You can use it like ground truth associated with the state and action, to train the neural network.

The TD target or "target value" gets its name because by updating a Q table or training a NN with it as a ground truth, the estimator will output values in future closer to the supplied value. The estimator "gets closer to the target".


What does it mean for an image to have a distribution? While studying Independent Component Analysis, specifically SVD to be used for the task of image separation. The lines given in the textbook is like this:-
"The first assumption will be that the two images are statistically independent."
Now, Statistical independence means images must have some kind of distribution to start with. But I am unable to grasp the meaning behind this. What does it mean for an image to have a distribution? Like how is a image having Gaussian distribution or Uniform distribution look like. It would be great if some examples are presented.

First, we must understand that an image can be represented as the numerical values of its pixels. For example, a 256x256 grayscale image can be represented by a vector of 65536 integer values between 0 and 255.
Now, let's consider that we are talking about images of human faces, like the CelebA dataset, but in grayscale.
And finally, let's imagine that we sample a vector of 65536 components from a uniform distribution between 0 and 255, that is, from $x \sim U[0, 255]^{65536}$. If we interpret such a randomly sampled vector as an image, like the interpretation we described in the first paragraph, do you think that is it probable that we see a face? Well, probably not. Probably we would see just noise, like this:

Why do you know that we won't see a face? Well, because you know that a face is not composed of uniformly random pixels, but has some structure in it. In other words, because you intuitively know that the distribution of pixel values of a face image does not match $U[0, 255]^{65536}$, but has a different distribution.
As a summary, the distribution of an image domain is the distribution the pixel values follow. Note that this is applicable not only to grayscale images, but to color images expressed in the RGB space or any other numeric space like HSB.
P.S.: this is how an image of Gaussian noise looks like (it was generated with Imagemagick, with convert -size 256x256 xc:gray +noise Gaussian out.png):


Pandas - how can I calculate a field to count the number of occurences that values in the ID field appear in sequential order I know how to read in the data frame in Pandas and do the basic manipulation, but how do I populate the order column based on the ID column? For instance, if bike occurs twice such as seen below, how can I populate the order column with 1 for the first occurrence and then 2 for the second occurrence and then do the same for car.

    ID      Color       Order?????
    bike    red         1
    bike    black       2
    car     green       1
    car     orange      2
    car     blue        3

Welcome to Data Science! Here I create your data frame and show one way to create the column you need. I use numpy in addition to pandas:

import pandas as pd
import numpy as np

# Create df
ID = ['bike', 'bike', 'car', 'car', 'car']
Color = ['red', 'black', 'green', 'orange', 'blue']
df = pd.DataFrame(data={'ID': ID, 'Color': Color})

Add a new column called 'Order', with counts based on each group - these are mini dataframes that contain only one of the ID values only. This makes use of the groupby method on a Pandas dataframe.

df['Order'] = df.groupby('ID').transform(lambda x: 1 + np.arange(len(x)))

The transform method takes a function and applies it to each group. I use an anonymous function (just a function with no name), also called lambda functions, using the keyword lambda.

The result:

    Color    ID  Order
0     red  bike      1
1   black  bike      2
2   green   car      1
3  orange   car      2
4    blue   car      3

As anymous.asker mentioned, it would be helpful in the future if you post the code that creates your dataframe!


In R, can I integrate different classifying algorithms in one bagging model? I use R to do data analysis. I have a dataset. When I use different classifying algorithms, such as random forest, SVM, etc, I have the different accuracy. So, I want to integrate all the algorithms into one framework, let's say adaboost.

We know that adaboost framework use multiple "weak" classifying algorithms to combine a strong classifier. So, can I customize the "weak" classifying algorithms as I want? Here is just my current idea: In this framework, I use SVM first. Then give the data that are classified incorrectly more weights. Then, I use random forest. ... In the end, all the classifiers in this framework will work together.

This is just what I think about this issue. If there is other method working such as voting, please let me know too.

Any help is appreciated.

What you're looking for is called an ensemble model which means it is a compilation of several models to improve the results. This is a very common technique for winners in Kaggle competitions. Since you're using R and caret is a popular way to do ML in R, here's a package just for that purpose on caret:

https://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html


Predictive models with class value belonging to a set of observations I would like to know whether it's possible to build a predictive model where I could define a set of rows with their attributes, and a class belonging to that set of rows, instead of having the typical model one observation - one class.

What I'm trying to do is to create a model which should be able to predict the future level of a rider in professional cycling - more likely to be used to forecast young riders' level . The input data are the results of each rider (race,category of race, type of parcours, position, and so on). The class is represented with the value (from 1 to 10) each "known" rider has been assigned for each skill (mountain, flat, time trial, sprint, etc.), which means that in principle, I would make one predictive model for each skill (what means for each class). For example, for mountain class I would have the following:

Rider   Category    Race Type   Parcours    Distance    Position    Class(Mountain)
-----------------------------------------------------------------------------
Froome  GrandTour   Stage       Mountain    185         1           10
Froome  WorldTour   Stage       MidMountain 210         25          10
Nibali  Contin.Tour Stage       Mountain    170         15          9
Nibali  WorldTour   Classic     MidMountain 245         3           9

I see here two main challenges, apart from the fact of dealing with both numerical and categorical attributes.

A rider gathers a lot of results, and each of them will result in the same class value. For example, a victory in a Grand Tour category stage from Chris Froome would be rated with a 10 in mountain, but he would also be rated with a 10 in mountain when he ended up in a bad position in a lower level stage - just as in the example. Ok, it's supposed that with thousands of rows the model (regression model probably, rather than a classification algorithm, but not sure of that either) should be able to detect the "common" pattern (a good performance in mountain from a good climber like Froome), so it could be overcome. I guess it'd be a matter of testing and finding the best-fitted models.
I find the predictive part the most problematic one, because what I need isn't to predict the class value for each observation (which is represented by each result), but the class value for each rider. That's why I said at the beginning of the thread that I would need to somehow treat all the results of a rider as a unique observation (something like a set or group of observations), and assign the class value to that set. A workaround could be to take all the observations of the rider, predict the class for each of them, and then calculate the average, but it doesn't look very smart.

So, any idea of how to build this kind of models?

Thanks in advance.

Looks like you have two problems: time-dependant predictive modeling and feature engineering.

1) Time-dependant data

Key words:

  create a model which should be able to predict the future level of a rider in professional cycling

The future Level of the Rider.

That means there is current Level. And Level in the past. And the history of Level change in time for each Rider.

The problem you are trying to solve is time dependant. Your source data can look like:

2) Build a variable to predict

Second. You should build the target feature. This Rider Level (Class).

As far as I understood:

the overall rider 'level' can be a function of his 'classes' in each particular race types
and... the race 'observation' class is a kind of function of {Category, Race Type, Parcours, Distance, Position}

So the only thing you realy need to predict is race result.


How to choose the number of output channels in a convolutional layer? I'm following a pytorch tutorial where for a tensor of shape [8,3,32,32], where 8 is the batch size, 3 the number of channels and 32 x 32, the pixel size, they define the first convolutional layer as nn.Conv2d(3, 16, 5 ), where 3 is the input size, 16 the output size and 5 the kernel size and it works fine.

in_size = 3
hid1_size = 16
hid2_size = 32
out_size = len(labels)
k_conv_size = 5

class ConvNet(nn.Module):

    def __init__(self):
        super(ConvNet, self).__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(in_size, hid1_size, k_conv_size ),
            nn.BatchNorm2d(hid1_size),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2))

        self.layer2 = nn.Sequential(
            nn.Conv2d(hid1_size, hid2_size, k_conv_size),
            nn.BatchNorm2d(hid2_size),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2))

        self.fc = nn.Linear(hid2_size *  k_conv_size * k_conv_size, out_size)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)

        return out

I change the output size from 16 to 32 and that of the next layer from 32 to 64 and it still works.
But when I resize the tensor to have the shape [8, 3, 64, 64], it throws a mismatch error that says size mismatch, m1: [16 x 5408], m2: [800 x 4]
I understand m2 is what it's expecting and m1 is what I'm giving.

But I don't understand where the values of m2 and m1 come from and how to change the hid1_size accordingly.

I understand the relationship between the shape of input data and the neurons in the first layer when building regular linear layers but how to define the relationship between the shape of the input and the number of channels produced by the convolutional layer in cnns?

I recommend you reading the  guide to convolution arithmetic for deep learning . There you can find very well written explanations about calculating the about size of your layers depending on kernel size, stride, dilatation, etc.

Further you can easily get your intermediate shapes in pytorch by adding a simple print(x.shape) statement in your forward pass and adapting your number of neurons in your fully connected layers.

Last but not least. When you cange your input size from 32x32 to 64x64 your output of your final convolutional layer will also have approximately doubled size (depends on kernel size and padding) in each dimension (height, width) and hence you quadruple (double x double) the number of neurons needed in your linear layer.


Similarity of Dictionaries Using Cosine Similarity I'm trying to use Cosine Similarity in Python to compare users, but can't seem to get it quite right.

from math import*

ratings={'Mr. Perfect': {1: 1, 2: 2, 3: 1, 4: 3, 5: 3, 6: 1, 7: 3, 8: 1, 9: 2},
         'The Progressive': {1: 3, 2: 3, 3: 3, 4: 3, 5: 3, 6: 3, 7: 3, 8: 3, 9: 3},
         'Mr. Villain': {1: 3, 2: 3, 3: 3, 4: 3, 5: 3, 6: 3, 7: 3, 8: 3, 9: 3}}

def square_rooted(x):

    return round(sqrt(sum([a*a for a in x])),3)

def cosine_similarity(x,y):

    input1 = {}
    input2 = {}
    vector2 = []
    vector1 =[]

    if len(x) > len(y):
        input1 = x
        input2 = y
    else:
        input1 = y
        input2 = x

    vector1 = list(input1.values())

    for k in input1.keys():    # Normalizing input vectors.
        if k in input2:
            vector2.append(float(input1[k]))
        else :
            vector2.append(float(0))

    numerator = sum(a*b for a,b in zip(vector2,vector1))
    denominator = square_rooted(vector1)*square_rooted(vector2)
    return round(numerator/float(denominator),3)

print("Similarity between Perfect and Progressive")
print (cosine_similarity(ratings['Mr. Perfect'],ratings['The Progressive']))

print("Similarity between Villain and Progressive")
print (cosine_similarity(ratings['Mr. Villain'],ratings['The Progressive']))

Basically, found the formula here https://stackoverflow.com/questions/35980910/return-similar-score-based-on-two-dictionaries-similarity-in-python but I keep getting results of 1. Any ideas would be helpful!

This is wrong: vector2.append(float(input1[k]))
Replace by vector2.append(float(input2[k]))


What is the best practice for data folder structuring? I work for a small data science consultancy firm and we are trying to standardize our project folder structure.
We started from the cookiecutter structure which is a great base.

However one of the discussion point lies in the subfolders of the data folder, which is structured as:

Raw
Interim
Processed

Let's think about the following situations:

The client gives you a manually extracted csv file -> This obviously goes into Raw
You have acces to SQL databases and make a no-modification extract
    -> Still into Raw I guess?
Because of very large databases, you create a semi-complex SQL query
    as base for a feature -> Is this Raw or Interim?

What are the best practices you apply? What would you recommend?

Ps: links to Github projects constructed following this kind of structure are very welcome

What are the best practices you apply? What would you recommend?

My practice is "kind of structured but quite different every time", and I don't recommend it ;)

I suspect that I'm not the only one but I don't have any stats.

Thanks for the link to Cookiecutter, this looks interesting! After reading a bit about it, it looks to me like a key criterion is this: "anyone should be able to reproduce the final products with only the code in src and the data in data/raw."

Based on that I would argue that:

  You have acces to SQL databases and make a no-modification extract -> Still into Raw I guess?

Yep, because this data cannot be obtained by processing some other part of the raw data.

  Because of very large databases, you create a semi-complex SQL query as base for a feature -> Is this Raw or Interim?

Assuming this query is run on the raw sql database, the resulting data goes to interim, because it can be re-generated by running the query again.
The query itself should be stored in src/features I think.


Boosting with highly correlated features I have a conceptual question. My understanding is, that Random Forest can be applied even when features are (highly) correlated. This is because with bagging, the influence of few highly correlated features is moderated, since each feature only occurs in some of the trees which are finally used to build the overall model.

My question: With boosting, usually even smaller trees (basically "stunps") are used. Is it a problem to have many (highly) correlated features in a bagging approach?

Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables.

In general, tree based models are not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares. You can easily add a variable twice without numeric problem. Note however that most interpretability tools like split importance or partial dependence plots are affected by multicollinearity. So be careful with them in such cases.


Convert a binary neural network classifier to one verses all classifier I have a neural network model (implemented from scratch) which gives me some continuous outputs and I have used a sigmoid layer, in the end, to convert it into a binary classifier. But my original problem is a multiclass problem.
I tried to use the softmax layer but somehow that cannot be applied to my model architecture as the output is not in that format (I get some continuous value as output).

Thus I realized that an alternative way would be to use one versus all classifier but I am not able to figure out how to put that on top of my binary classifier.
Please forgive me if my concepts are a bit weak, but I would really be thankful if someone could help me with this.

Like, how can I convert the binary classifier to a multiclass classifier while keeping the original architecture and what kind of loss function should be used? Would that be different from the binary classifier loss function?

Thanks in advance.

You may keep N neurons and Sigmoid and treat is like multi-label though your data will always be multi-class. So it should learn like multi-class

Create your own One-vs-One

Have 3 copy of the same model
Fit with A|B, B|C and C|A respectively
For new data, you need the average of two for each Class probability

Use scikit-learn Classifier

OneVsOneClassifier Link
OneVsRestClassifier Link


How to handle columns with categorical data and many unique values I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

For categorical columns, you have two options :

Entity Embeddings
One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects.  So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner