How to deal with different audio formats for audio classification? I am working on an audio classification problem statement to classify between two audio classes. I have collected samples from jotform, they are providing audio widget to collect .wav audio but it turned out that widget is storing data in .mp3 format :
In my problem statement, Classification classes are from different formats :
class A : all the 100 samples are in .mp3 format ( jot form collection )
class B : all the samples are in .wav format

I am adding both types of classes' sample here :
Class A sample audio  : it's in .wav format

Details :

General
Complete name                            : count_class_1.wav
Format                                   : Wave
File size                                : 1.41 MiB
Duration                                 : 15 s 445 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 768 kb/s

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 15 s 445 ms
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 1.41 MiB (100%)

Class B sample audio Jotform says it's .wav format but only extension is .wav, file is .mp3 format.
Details :
General
Complete name                            : count.wav
Format                                   : MPEG Audio
File size                                : 183 KiB
Duration                                 : 9 s 360 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 160 kb/s
Writing library                          : LAME3.99.5
FileExtension_Invalid                    : m1a mpa mpa1 mp1 m2a mpa2 mp2 mp3

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Format settings                          : Joint stereo / MS Stereo
Duration                                 : 9 s 360 ms
Bit rate mode                            : Constant
Bit rate                                 : 160 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Frame rate                               : 41.667 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 183 KiB (100%)
Writing library                          : LAME3.99.5

What i am doing before feeding it to neural network :

Downsampled to 16kHz,  the level of the signal was normalized
Segmented in audio segments, by removing the silences in the signal
High filtered (pre-emphasis filter). Audio segments were then
divided    in non-overlapping Hamming-windowed frames of 25ms.

Now after this extracting various features from each frames including MFCCs, Zero-crossing rate (ZCR), Formants (the first 4) etc and at last feeding all these features to simple dense layer neural network or CNN (spectrogram format).
But the problem is both classes' audio files are in a different format class A audio samples are in .wav and class B audio samples in .mp3 and there are high chances that network can be biased towards format or audio encoding.
Solutions I have thought :

Downgrade all files to 16kHz frequency ( But format issue is still there)
or convert all files into one universal format, for example I am converting all .mp3 files to .wav files then all files will be having same format, I could convert one into another, but I am afraid I will lose quality on the converted files.

My doubt is if I downsampled both classes audio samples (.wav and mp3 both) to 16kHz will my neural network still be format biased?
What would be a good strategy for me for Audio classification when audio files are in different formats?

If the goal of the project is to classify the groups, then just use the file format information. It is common in machine learning to use meta-data information as a feature.
If you want to contrive the problem by not using metadata, then convert them to format that audio quality matters less such as spectrogram or Fourier transform.


Feature selection filter methods I am confused about when to use which filter methods for feature selection. I tried to learn them through online resources and found methods like chi-square, variance threshold, F-test, Mutual information, etc. As I read through the methods, I got confused with which one I should use. How do I know if the method really selected the most relevant features and did not drop any features that were, in fact, contributing to the cause of the output?

In general, you need to wrap the feature selection filter method within a cross validation scheme. Here is an example of such an algorithm, assuming you are also tuning hyperparameters.

Split the data into training and testing.
Split the training set only into k folds.
For each iteration of hyperparameter tuning, use cross validation, i.e. do:

  For each of the k folds, for iterations i = 1 to k, do:

  1) Let the validation set be fold i, and the "inner training" set be all folds not equal to fold i.

  2) Using your filtering method, select some subset of your features using some cutoff p on the inner training set. For example, I might filter out variables that have absolute correlation with the target variable y less than p = 0.3.

  3) Fit a model.

  4) Predict the validation set = fold i. Score the model, i.e. evaluate your function you are trying to optimizer like log loss, mean squared/absolute error, Brier, etc. Save the score.

  end

After you are done with your cross validation for one set of hyperparameters, take the mean score over all of the k folds. If the score is better than a previous set of hyperparameters found in a prior iteration, save these hyperparameters.

Note: you can choose to have p, your filter cutoff, also be a hyperparameter here. For simplicity I kept it constant but in general you will probably do better if you tune the filtering cutoff as well.

end

Using the same (or best, if you let it be a hyperparameter and optimized over it in step 3) filter cutoff p, and the best hyperparameters that you found in the cross validation process in step 3, use the training set created in step 1, and fit your "final" model. That is, filter your training set with the cutoff value p using the exact same filtering method you did in step 3. Then, fit a model to this data set using the optimal hyper parameters, also found in step 3.
Predict the test set using your fitted model in step 4. Score the model.

Finally, repeat the entire process again but this time, don't do any filter selection method (omit the filtering process). Use the exact same seeds in your cross validation and at the initial split in step 1 to make sure you are splitting your data the same way from when you used filtering. Now, compare test set scores and see for yourself if there is any benefit to using your filtering method.

In general, I am not a fan of filter based methods because they all introduce some arbitrary cutoff = p and also are all univariate (they don't take into consideration other variables in the model, just in isolation). That being said, they are fast and they might work for your problem.

EDIT: Also, depending on how much data you have and how variable your validation scores are, multiple repeats of the entire algorithm above (steps 1 through 5) might be necessary in order to obtain valid comparisons between using filtering and not using filtering. You would repeat the algorithm above say, N times, once  with filtering and once without for a total of 2N times. You would then calculate an average of averages (find the average score over N repeats using filtering, and find the average score over the same N repeats with no filtering). Then, you could compare scores, form confidence intervals, etc. for a more trustworthy (but also more time consuming) comparison.

Hope this helps.


faster alternatives to sparse.model.matrix? I have a large dataset that is entirely categorical. I'm trying to train with it using xgboost, so I must first convert this categorical data to numerical. So far I've been using sparse.model.matrix() in the Matrix library but it is far too slow. I found a great solution here, however, the sparse matrix it returns in not the same one that sparse.model.matrix returns. I know there is a way to force sparse.model.matrix to return identical output as the solution in the link (by providing contrasts), however, that is not an effective solution as it is still too slow and results in different representation (and hence different training model).

Is there a way to accomplish the job sparse.model.matrix does even nearly as fast as the solution I posted? For my data, the solution I posted does it in about 15% of the time.

Try that but remove LHS variables from dataset before hands? We use that for the data processing part of putting our xgboost models in production.

fspmatrix <- function(df) {
  # Building sparse matrix with sparseMatrix to save time vs sparse.model.matrix

  # Details about the df to help build triplet
  nr <- nrow(df)
  nc <- ncol(df)
  nlevels <- pmax(1, sapply(df, nlevels))
  fac <- which(sapply(df, is.factor))
  # Check that all factors have at least 2 levels
  stopifnot(min(nlevels[fac]) > 1)

  # Building the value vector removing zero positions and first levels of factors after the first factor
  x <- as.numeric(unlist(unlist(df)))
  jx <- x # for use in building j
  # Positions of values coming from factors
  pos <- as.vector(sapply(fac * nr, function(x) x - nr + (1:nr)))
  # Positions of first levels of factors after the first factor
  pos1 <- pos[-1:-nr]
  pos1 <- pos1[which(x[pos1] == 1)]
  # Replacing the values of factors with 1 for one hot encoding
  x[pos] <- 1
  # Positions of non-zero values
  nzpos <- which(x[-pos1] != 0)
  # Values to build triplet around
  x <- x[-pos1][nzpos]

  # Building the i vector, the first dim part of the triplet
  i <- rep(seq_len(nr), nc)[-pos1][nzpos]

  # Building the j vector, the second dim part of the triplet
  jnlevels <- nlevels
  jnlevels[fac[-1]] <- jnlevels[fac[-1]] - 1
  jx[-pos] <- 1
  jx[pos[-1:-nr]] <- jx[pos[-1:-nr]] - 1
  cs <- cumsum(c(0, head(jnlevels, -1)))
  j <- jx + rep(cs, each = nr)
  j <- j[-pos1][nzpos]

  # Building the dimnames vector
  tag <- TRUE
  fspnames <- as.vector(unlist(sapply(names(df), function(n) {
    clvl <- as.character(levels(df[[n]]))
    isL <- is.logical(df[[n]])
    isF <- is.factor(df[[n]])
    indx <- if (tag & isF) {
      1:length(clvl)
    } else {
      -1
    }
    if (tag & isF) tag <<- FALSE
    paste0(n, clvl[indx], ifelse(isL, "TRUE", ""))
  })))

  fspm <- sparseMatrix(i = i, j = j, x = x, dims = c(nr, tail(cs, 1) + tail(jnlevels, 1)), dimnames = list(NULL, fspnames))

  return(fspm)
}


How to select the split point for Continuous Attribute Age For the above table, midpoints for possible split points are 22.5 and 35. I have calculated the entropy and gain for each value and 35 had the minimum Entropy and highest gain. Is it correct ?
Given High -> (-), and Low -> (+)
D<22.5 => [0+, 2-],
Entropy (D<22.5) = 0, since all the values are of the same class High.
D>22.5 => [2+, 2-],
Entropy (D>22.5) = 1, since the values are distributed equally among Low and High classes.
D<35 => [2+, 3-],
Entropy (D<35) = -[2/6 x $log_2$(2/6) + 3/6 x $log_2$(3/6)]= 0.5
D>35 => [0+, 1-],
Entropy (D>35) = 0, since all the values are of the same class High
Gain (D, Age>22.5) = 0.918 - 2/6 (0) - 4/6 (1) = 0.2513
Gain (D, Age>35) = 0.918 - 5/6 (0.5) - 1/6 (0) = 0.5103
Is that right?

For 35 split, there is an error in the denominator. It should be 5 as the total number of items for D<35 is 5

D<35 => [2+, 3-], Entropy (D<35) = -[2/5 x $log_2$(2/5) + 3/5 x $log_2$(3/5)]


Mean of mean and average In order to establish an overall rating for a product from a series of user ratings (from 1 to 5), I thought that the median would be a good idea so that extreme values would not have too much influence. But in doing so, it is hard to rank products since they will all have a whole ranking.
So I thought about averaging the mean and the median. Is this a known measure? Is it relevant in this case?

No, it sounds like an odd metric. I suspect the parametric distributions often calculable for the median and mean would be rather nasty for the average of the two which would make this unpopular among statisticians.
I suspect you mean "outliers" by extreme values. If you consider the definition of an outlier:

In statistics, an outlier is a data point that differs significantly from other observations.

https://en.wikipedia.org/wiki/Outlier
(I checked a few definitions and most of them are like this).
I think on a discrete scale of 1 to 5, the idea of an outlier sounds odd. As you can see "differs significantly" is open to interpretation and I don't think 1 or 5 differ greatly from 2-4 unless the rest of your user ratings are at 3 only.
In any case, it is probably best to just use the mean. If you lack the variation in the data to seperate out the objects you want using the mean +- std dev or confidence intervals then that's just the data and there's not much you can do about it.


What is a subspace and what is a shared subspace? I'm totally new in machine learning. The first confusing concept is subspace. In multi label classification we have to share the subspace. What does one mean by that shared sub spaces?

I don't know what you mean by "In multi label classification we have to share the subspace.", but the concept of subspaces is fairly easy to explain.

  In mathematics, a space is a set (sometimes called a universe) with some added structure.

Source: Wikipedia

The most common spaces are $\mathbb{R}$ (the number line), $\mathbb{R}^2$ (the plane) and $\mathbb{R}^3$ (the space we often think in, with $x,y,z$ axes). Lets assume our main space is $\mathbb{R}^3$. Our data are datapoints in this space. However, we have to write a report. This report has to be printed on paper. Hence we project the data points from a 3-dimensional space to a two-dimensional subspace. So a subspace is just a part of the original space.


A/B testing: How to calculate p-value on post test segments? My question on A/B testing is about doing post test segmentation analysis.

For example:

  I run an A/B test on my website to track bounce rate.  On the
  treatment group, i put a video to explain my company. On the control
  group i put just plain text. I pick a segment of users who are first
  time users from USA to be split 50/50 into the 2 groups.

Metric that i am tracking is average bounce rate (assume 20%).
Power effect (0.8)
effect size i expect to see(10% so bounce rate should fall to (20% - 0.10 * 20% = 18%))
Calculated sample size required is say 1000 for each group.

Say i run the test for the correct amount of time. At the end of the test, i get a p-value of 0.06. i do not reject the null hypothesis.

  However, when i do post test segmentation analysis, for example, i saw
  that users who signed up for a free trial, 44% of them played the
  video.

In this case, how do i calculate if the 44% was significant? (while taking into account the multiple comparison problem?)
Like in the Airbnb experiment, they did post segmentation analysis on the browser type and was able to calculate the p-value.

My approach

Does this mean that for every segment that i want to analyze, i need to have at least 1000 samples? Also how would i recalculate the p-value given that the p-value of this A/B test was already generated above as 0.06?

I recently wrote about this in a blog post.
Given this is a rate evaluation metric, you will want to use the z-test. The basic steps are (more details in the blog post)

calculate the pooled standard error of your pairwise comparison
calculate z-statistic by normalizing the delta or lift by the standard error
look up the cdf value of the normalized delta
p-value = 1-cdf(z)
given that this is also an A/B/n test, you should also want to apply multiple testing correction using the bonferroni procedure or the benjamini hochberg procedure, when evaluating significance


Use prediction as feature for a decision tree I'm working at classifying documents according to their content.

First I built a decision tree model that gives 90% of goods results.

Then I tried a TFIDF/SVC approach which also gives 90% of good results.

So now i'd like to combine both. My first thought was to add the prediction of TFIDF/SVC as a feature of the decision tree.

I saw this post about bagging/stacking/boosting. For me, adding the feature to the decision tree is equivalent to stacking. Is that correct ?

Yes. Stacking is essentially feeding the predictions of the base learners to a meta learner. Sort of like a model of the models. Here's a good explanation of that.

Bagging,Boosting and Stacking


Text Mining on Large Dataset I have a large data set(460 Mb) which has a column - Log with 386551 rows.
I wish to use clustering and N-Gram approach to form word cloud. My code is as follows:

library(readr)
AMC <- read_csv("All Tickets.csv")
Desc <- AMC[,4]

#Very large data hence breaking it down for creating corpus
#DataframeSource has been used insted of VectorSource is to be able to       handle the data

library(tm)
docs_new <- data.frame(Desc)

test1 <- docs_new[1:100000,]
test2 <- docs_new[100001:200000,]
test3 <- docs_new[200001:300000,]
test4 <- docs_new[300001:386551,]
test1 <- data.frame(test1)
test1 <- Corpus(DataframeSource(test1))
test2 <- data.frame(test2)
test2 <- Corpus(DataframeSource(test2))
test3 <- data.frame(test3)
test3 <- Corpus(DataframeSource(test3))
test4 <- data.frame(test4)
test4 <- Corpus(DataframeSource(test4))

# attach all the corpus
docs_new <- c(test1,test2,test3,test4)

docs_new <- tm_map(docs_new, tolower)
docs_new <- tm_map(docs_new, removePunctuation)
docs_new <- tm_map(docs_new, removeNumbers)
docs_new <- tm_map(docs_new, removeWords, stopwords(kind = "en"))
docs_new <- tm_map(docs_new, stripWhitespace)
docs_new <- tm_map(docs_new, stemDocument)
docs_new <- tm_map(docs_new, PlainTextDocument)

#tokenizer for tdm with ngrams
library(RWeka)
options(mc.cores=1)
BigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max =2))
tdm <- TermDocumentMatrix(docs_new, control = list(tokenize = BigramTokenizer))

This is giving me results as follows:

TermDocumentMatrix (terms: 1874071, documents: 386551)>>
Non-/sparse entries: 17313767/724406705354
Sparsity           : 100%
Maximal term length: 733
Weighting          : term frequency (tf)

I then converted it to dgMatrix using following :

library("Matrix")
mat <- sparseMatrix(i=tdm$i, j=tdm$j, x=tdm$v, dims=c(tdm$nrow, tdm$ncol))

While trying to use the following I am getting memory size error:

removeSparseTerms(tdm, 0.2)

Please suggest further as I am new to Text Analytics.

You are using R, and everything that you are currently working on will be held in memory hence the error. You just don't have enough of it.

You might be better off creating the frequencies of the terms from the original splits as opposed to creating one big file.

Then after you have the frequencies adding them all together.

Personally I use this code to create my wordclouds.

##Clean code for single column of a dataframe, in this case named Text
  x = alltweets
  tweets.text <- x$text
  tweets.text.cleaned <- gsub("@\\w+ *#", "", tweets.text)
  tweets.text.cleaned <- gsub("(f|ht)tp(s?)://(.*)[.][a-z]+", "", tweets.text.cleaned)
  tweets.text.cleaned <- gsub("[^0-9A-Za-z///' ]", "", tweets.text.cleaned)
  tweets.text.corpus <- Corpus(VectorSource(tweets.text.cleaned))
  tweets.text.final <- tm_map (tweets.text.corpus, removePunctuation, mc.cores=1)
  tweets.text.final2 <- tm_map (tweets.text.final, content_transformer(tolower), mc.cores=1)
  tweets.text.final2 <- tm_map(tweets.text.final2, removeNumbers, mc.cores=1)
  tweets.text.final2 <- tm_map(tweets.text.final2, removePunctuation, mc.cores=1)
  tweets.text.final2 <- tm_map(tweets.text.final2,removeWords, stopwords("English"), mc.cores=1)
  tweets.text.final2 = tm_map(tweets.text.final2, removeWords, c("amp", "&"))
#create corpus
   housing.tweets.corpus <- Corpus(VectorSource(tweets.text))

#clean up by removing stop words
   housing.tweets.corpus <- tm_map(housing.tweets.corpus, function(x)removeWords(x,stopwords()))

#install wordcloud if not already installed
   install.packages("wordcloud")
   library("wordcloud")

#generate wordcloud
   wordcloud(housing.tweets.corpus,min.freq = 2, scale=c(7,0.5),colors=brewer.pal(8, "Dark2"),  random.color= TRUE, random.order = FALSE, max.words = 500)


How to maintain CBOW dataset dimension and fit it in Neural Network? I am new to neural network. I'm trying to train word embeddings without using word2vec package.

Using titles from reddit worldnews dataset I'm have done some CBOW representation.

For window size three here is some of my outputs:

0 Context : ['scores', 'killed', 'pakistan'] ---> Target: clashes
1 Context : ['japan', 'resumes', 'refuelling'] ---> Target: mission
2 Context : ['us', 'presses', 'egypt'] ---> Target: gaza
3 Context : ['presses', 'egypt', 'gaza'] ---> Target: border

For Vocab size = 513, I've collected 369 of target words against 369 of 3-grams context words. Each context word is one-hot codded of length 513.

Therefore, my dataset length becomes:
X.shape = (369, 3, 1, 513)
Y.shape = (369, 1, 513)

Now I'm having trouble in fitting the data in neural network. My Neural Network model is constructed with keras.

# create model
model = Sequential()
model.add(Dense(100, input_dim=1, init=369 'uniform' , activation= 'sigmoid' ))
model.add(Dense(1, init= 'uniform' , activation= 'sigmoid' ))
model.compile(loss= 'binary_crossentropy' , optimizer= 'sgd' , metrics=['accuracy'])

#train
history = model.fit(X, Y, nb_epoch=100)

Raised Error:

ValueError: Error when checking input: expected dense_9_input to have 2 dimensions, but got array with shape (369, 3, 1, 513)

This shows you did not properly flatten your output before the final dense layer. Your first model.add should be an embedding layer since that represents word vectors that'll be trained. The last layer should be the dense one. Hard to tell on the spot which hidden layers to consider, but try GlobalAveragePooling and Dense with the relu activation.


System to provide guide to students about getting admissions to universities of their choice or some specific courses I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear to ultimately land up in universities of their choice in future.

I want to this project for the process of learning and i need some starting points about how should i go about doing this .

First of all, interesting question!

Most important thing to recommend you a starting point is to know what kind of data you have in hand. So my answer is regardless of that fact.

If you have a road-map of studies of students who got into University XXX then you can use model it with an HMM (Hidden Markov Model).

In this case the observations are the sequence of courses one passed and you want to calculate the probability of landing in desired university. Transition matrix is the probability of getting course A after getting course B. Emission matrix might be something like how successful they were in the course. Transition and emission matrices can be scored using facts like dropping a course or failing or the education track they exhibit in.


ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1') X = train_encoded_df.iloc[:, 1: ]
y = train_encoded_df["Loan_Status"]

print("Precision:",metrics.precision_score(y_test, y_pred))

My training data contains the categorical features encoded using get_dummies().

This is causing the error:

> ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='U1')

How to fix this?

pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive one and, if not given explicitly (like in your case here), it assumes the default value of 1 (again, check the docs).

Since it seems that the positive label in your case is 'Y', replace the last line with:

print("Precision:",metrics.precision_score(y_test, y_pred, pos_label='Y'))


CNN image to image translation: multiple image inputs to one image output I am interested in training a CNN to take in inputs where each input is a set of low-resolution images and each ground truth is a single high-resolution image. The ground truth high-resolution image was generated by averaging the information from a set of 80 low-level images. What I would like the CNN to do is to generate the same high-resolution image from a smaller set of low-level images, for example 5 low-level images.

I was looking into TensorFlow GAN and was thinking I could input each training image as a concatenated array of low-resolution images and each ground truth as a single high-resolution image. However, I am unsure whether this is the best method since the dimensions of the input and output will be different. Could someone advise what is the best approach using TensorFlow (or alternative deep learning frameworks if there is a better option)? Thanks!

GAN is a good choice also have a look at Autoencoder as well at the links below
https://www.tensorflow.org/beta/tutorials/generative/cvae

and

https://towardsdatascience.com/deep-autoencoders-using-tensorflow-c68f075fd1a3

these two are very similar and will solve your problem statement for sure.


ACF vs PACF in ARIMA Given a time series problem,

Should ACF and PACF be done before or after differencing that make the time series stationary?
If ACF and PACF has shown different results, should the number of orders of AR/MA follows ACF or PACF?

After differencing! A stationary time series is a key (as in, crucial) assumption of all ARIMA models. You are presumably trying to determine the orders of your AR and MA parts, using the Box-Jenkins methodology (looking at graphs of acf/pacf). In this case, if the true process is indeed AR/MA/ARMA then the series must be stationary by assumption (which means, differencing the time series if needed to make it stationary).
Look at both of the plots. For the AR(p) part of your model, look at the PACF plot and look for consecutive significant lags and then a sharp cutoff to zero (insignificant lags). For the MA(q) part, look for the same thing but using the ACF plot. If you see that one plot shows something as described above and the other not, then perhaps your model is only AR(p) or MA(q) rather than ARMA(p,q). If none of the plots seem to suggest anything, then likely you do not have a stationary time series and/or an ARMA/ARIMA model is inappropriate.


sklearn.GridSearchCV predict method not providing the best estimate and accuracy score I was playing around with the credit default dataset in UCI
("https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls")

These are the steps i have undertaken so far:

Created a Pipeline for Perceptron
Parameter values were provided for learning rate and epochs.
Passed the estimator and param grids to GridSearch to get the best estimator
GridSearch provided me with best score for a particular learning rate and epoch
used predict method on the gridsearch and recalculated accuracy score

Parameters provided for gridsearch

{'perceptron__max_iter': [1,5,8,10], 'perceptron__eta0': [0.5,.4, .2, .1]}

Question: The best parameters provided by GridSearch are not really the best parameters. I am not getting same scores using predict method on the gridsearch. Why is this the case? WHat am i doing wrong?

So acc to gridsearch best param are :
{'perceptron__eta0': 0.5, 'perceptron__max_iter': 8}

Accuracy score : 0.7795238095238095

However if i use these best parameters and call predict on gridsearch gives a totally different value, accuracy score dips to  0.5882222222222222

Please find code below.

import pandas as panda

from sklearn.model_selection import learning_curve, train_test_split,GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn.metrics import accuracy_score, mean_absolute_error, classification_report
from sklearn.linear_model import Perceptron, LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

from matplotlib import pyplot as plot
import seaborn as sns

from numpy import bincount, linspace, mean, std

remote_location = "https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls"
data = panda.read_excel(remote_location,sheet_name = "Data", header = 1)

data.rename(str.lower, inplace = True, axis = 'columns')

_y_target = data['default payment next month'].values

columns = data.columns.tolist()
columns.remove('default payment next month')

_x_attributes = data[columns].values

## meaning of stratify = _y_target. returns test and training data having the same proportions of class label '_y_target'
_x_train,_x_test,_y_train, _y_test = train_test_split(_x_attributes, _y_target, test_size =0.30, stratify = _y_target, random_state = 1)

## lets check the distribution. we can see 4times the lower value as was the case before as well. train/test set distributed well
print("label counts in y train %s" %bincount(_y_train))
print("label counts in y test %s" %bincount(_y_test))
parameter_grid = {'perceptron__max_iter': [1,5,8,10], 'perceptron__eta0': [0.5,.4, .2, .1]}

pipeline = make_pipeline( StandardScaler(),Perceptron(random_state = 1))
gridsearch = GridSearchCV(estimator = pipeline, param_grid = parameter_grid, cv = 10, n_jobs = 1, scoring = 'accuracy')

search = gridsearch.fit(_x_train, _y_train)
print(search.best_params_)
print(search.best_score_)

_y_prediction = gridsearch.predict(_x_test)

print("Accuracy score %s" %accuracy_score(_y_test,_y_prediction))
print("Classification report  \n %s" %(classification_report(_y_test, _y_prediction)))

My output is as below:

{'perceptron__eta0': 0.5, 'perceptron__max_iter': 8}
0.7795238095238095
Accuracy score 0.5882222222222222
Classification report
              precision    recall  f1-score   support

          0       0.78      0.66      0.71      7009
          1       0.22      0.35      0.27      1991

avg / total       0.66      0.59      0.62      9000

Summarizing your results - your trained a model using gridsearch.
accuracy score on the train set is ~0.78.
accuracy score on the test set is ~0.59.
Rephrasing you questions: why do my model performance on the test set is worse than on my train set?

This phenomena is very common - and I can think of two potential explanations:
1) Overfitting: your trained model had learned the 'noise' in the train set and not the actual pattern.
 Then when you use your model to predict on the test set, it predicts the noise he had encountered(which is not relevant for the train set - thus lower accuracy).
2) Train set and data set are not generated from the same process/describe different parts of it. In this case - the pattern learnt by the trained model is not relevant to the test set and accuracy will drop.
This may happen in situations where the train/test split is done without considering the actual underlying process. For example - an image classification problem where you model whether this picture is an animal or not and the train data has only human and dogs sample and the train set has only fish and airplanes pictures. This is less likely to be the explanation, since your using grid search, which adds randomness to this splitting.


Combining results via arithmetic mean For combining the results of different measurements, I want to calculate their arithmetic mean. Imagine the following dataset:

id;attribute;time
A;AB;1
B;AB;2
C;AB;3
B;AB;4
B;AB;5
A;AB;6

I want to combine all results with id A and get their mean. The expected dataset looks like the followind. Be aware, that the value of the in between attribute, is not important, I've chosen the first entry.

id attribute     time
A  AB            3.500000
B  AB            3.666667
C  AB            3.000000

The code below does the right thing, but I'm wondering if there is a better solution for this kind of problem.

# Read in the data
data <- data.frame(read_delim("~/...", ";", escape_double = FALSE, trim_ws = TRUE))
# Visualize head of the data
head(data)

# Define function to calculate the mean
calculate_mean = function(data){

  # Prepare empty result data set
  result = data.frame(id=character(),
                      attribute=character(),
                      time=integer())

  # Iterate over specified id <-- Change this one to the one you want to iterate over
  iterator = unique(data$id)

  # Fill result dataset with mean
  for(i in iterator){
    subset = subset(data, data$id == i)
    arithmetic_mean = mean(subset$time)
    result = rbind(result, data.frame(subset[1,]$id,
                                      subset[1,]$attribute,
                                      arithmetic_mean))
  }

  # Set colnames properly
  colnames(result) = colnames(data)

  return(result)
}

# Call function
result = calculate_mean(data)

# Display the result
result

Do you have an idea? Thanks for your support.

Try it the R tidyverse way:

library(dplyr)
dfr <- read.table(text = "id;attribute;time
A;AB;1
B;AB;2
C;AB;3
B;AB;4
B;AB;5
A;AB;6", header=TRUE, sep=";")

dfr %>% group_by(id, attribute) %>% summarize(mean=mean(time))

Result

# A tibble: 3 x 3
# Groups:   id [?]
      id attribute  mean
  <fctr>    <fctr> <dbl>
1      A        AB 3.500
2      B        AB 3.667
3      C        AB 3.000


Supervised multiclass classification : is ANN a good idea ? or use other classifiers? I have a problem deciding what to use since i'm just beginning to creating predictive models.

Let's say I have a training dataset with 5 or 6 features and a testing dataset. (With around 50k rows in training / 5k in testing). My samples are people that I would like to assign to types of products they would buy. (more than 2 classes).

I'd like to know the whole process of what to use, and based on what exactly. Also, is there a serious difference between the results rendered by an ANN and other classifiers on this type of classification?

Note: I have 10 possible classes in the output

Thanks in advance.

In general, if you only have $5$ to $6$ scalar features. I would simply start with easy methods like logistic regression and discriminant analysis. I would guess that you should be able to get good results.

You should also look at the distribution of the scalar features. Maybe you can derive new features that are helpful in separating. A simple visual way to see if it is possible to separate the classes by a linear hyperplane is to use a principal components analysis (short PCA) and extract 2 or 3 factors. Then use these factors to visualize your datapoints (maybe use a random sample from the training data set and repeat this three or more times to see if the trend is there in all random samples that you looked at). You should see if the classes are well sparable.

If you see that your model performance is not good enough I would try out decision trees (these are very interesting as they allow you to see how the decisions of your classifier are made).

Depending on the model performance you could also use neural networks. I personally would rather try it with simpler models first. Neural networks are very powerful function approximators, but you will most likely not be able to extract some useful information about the relationship between the features and the classes of products.


conv net data retrieval on unseen class I have build a  conv net for image classification which work "well"

Now I extract features from last fully connected layer and use it for image retrieval (find image most similar to my target image) using hamming distance. it's working prety well even if I'm not able to predict how it'll be rotation invariant, sensible to noise..

I have try image retrieval only on class that have been seen by my model while training (but not training data). Do you know if it could work on class that have never been seen by the model while training?

e.g

Let's says model has been train on car and truck. I want to find most similar image to that yellow small car and it's work well it's return me yellow and small car.

Now let's imagine I apply it on a new data set with Plane. I want to find most similar image to that small yellow plane. Should I expect to find yellow small plane or that'll be totaly random?

Since network has never seen plane while training, is it possible to predict, or at least have an intuition, of the result?

It depends. Let me explain you what features are detected when you train a classification model.

Let's take same example as yours - car and truck classification. The CNN will extract features from both type of images - tires, windows, shape, texture etc.

Then it'll separate the features - if it sees something like bigger tires, bigger windows, rectangular shape, it'll give more weight to truck class. Similarly for car class.

Now coming to your image retrieval part. This is what you're doing - passing all the image through CNN and extracting features for each image. Then comparing your test image features with other features to find most similar one.

Now think, what if you pass new image to the same CNN ? What features will be extracted.

CNN will extract the same features as it used to for car and truck - tires, windows, shape etc. Because it's trained for that only. It won't start extracting wings as feature if you pass airplane.

So if you pass airplane as test image - same features as car and truck would be extracted and you will get results based on the comparison of these features. Hence if you pass yellow small airplane, you might get yellow small car having same tire size as your airplane or yellow small truck having shape similar to your plane or in best case yellow small airplane.


Should I use GPU or CPU for inference? I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?

Adding more details from comments below.

I'm new to this so guidance is appreciated.

Memory: GPU is K80
Framework: Cuda and cuDNN
Data size per workloads: 20G
Computing nodes to consume: one per job, although would like to consider a scale option
Cost: I can afford a GPU option if the reasons make sense
Deployment: Running on own hosted bare metal servers, not in the cloud.

Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU.

It is true that for training a lot of the parallalization can be exploited by the GPU's, resulting in much faster training. For Inference, this parallalization can be way less, however CNN's will still get an advantage from this resulting in faster inference. Now you just have to ask yourself: is faster inference important? Do I want this extra dependencies (a good GPU, the right files installed etc)?
If speed is not an issue, go for CPU. However note that GPU's can make it an order of magnitude faster in my experience.


Word2Vec / Doc2Vec - map change of words overtime I have text data for 10 years and I want to run a Word2Vec model on the text data each year and calculate the distances between words and I would like to compare them over time.

Are there any papers in which the Word2Vec model is compared to previous years data? Or if I have a matrix of the Word2Vec vectors are there ways to compare these matrices year-on-year or from period t to t-1?

I can compute the cosine distance of the matrix but cannot think how to compare these "scores" for each word over time.

Yes, you can check Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change


Does a neural network continue to change after SGD stops improving? I would like to understand the behaviour of stochastic gradient descent (SGD) over long time periods for a fixed learning rate.

Let us assume that after a number of epochs we expect no further improvement to our cost function.  If we continue to train over more epochs then one of two things can happen:

The solution keeps changing but remains close to a local minimum; in the space of weights it remains in a local region.  Or,
The solution keeps changing but explores a large region of the space of weights.

Let's make this precise.  Take a fixed shape of neural network and use the Euclidean distance metric on the space of weights.  Now train this network using SGD on a dataset, e.g. MNIST, and for a fixed choice of learning rate.  After a large number of epochs, N start recording the weights of the network,
$$ {\bf w_N}, {\bf w_{N+1}} \ldots $$
How does $d({\bf w_N}, {\bf w_M})$ behave?  To gauge whether these distances are large or small, they can be compared to the distances you get by comparing ${\bf w_N}$ with equivalent networks given by permuting the neurons within a layer.

Ofcourse the answer may vary depending on the neural network, the data, the learning rate, etc.  I suppose I would like to know the answer for typical "real world" networks.  Unfortunately I lack the processing power (and expertise) to adequately investigate this myself, and am confident this will be known to experts.

The evolution of the improvement of the performance of the network if you keep training it depends on:

1) the surface of the loss function with respect to the network parameters.

2) the learning rate, which is the scale factor you apply to the gradient in order to compute the next set of weights.

If the learning rate is too high (with respect to the shape of the loss surface) and you keep training, you may escape a local minimum you just reached.

The problem is that you don't know the loss surface and so we cannot choose the perfect learning rate for it. There are a lot of heuristic tricks to try to improve the exploration behaviour of SGD, like momentum.

So the answer is: it depends. The network could stabilize around a local minimum and so the weights would only suffer small oscillations, or it might ruin everything by escaping the local minimum.


Watermark detection using Deep Learning I have images with and without watermark. There is only one type of watermark. I have tried VGG16 transfer learning, but results were bad.

What are the methods to do image classification if it has a watermark or not?

You could consider finding watermark an object detection problem, not an image classification problem. Faster R-CNN and "You only look once" (YOLO) are both general object detection architectures.


What arguments should I pass to input_shape parameter of LSTM function in Keras? My dataset has 2944424 rows and 6 columns. I am using an LSTM in Keras to forecast taxi demand. I am having problem with the input_shape parameter of the LSTM.

It gives the error:

ValueError: Error when checking input: expected lstm_15_input to have 3 dimension

Following example shows that by printing shape of your train data you can pass the right value to input_shape(). Also it depends you want to feed your data in 2D or another way. As it can be seen in snippet:

trainX.shape[1]=10

trainX.shape[2]=1440

and in output based on test shape Dense(units = 960)

#Shape of train and test data
trainX, testX, trainY, testY = train_test_split(trainX,trainY, test_size=0.2 , shuffle=False)
print("train size: {}".format(trainX.shape))
print("train Label size: {}".format(trainY.shape))
print("test size: {}".format(testX.shape))
print("test Label size: {}".format(testY.shape))
#train size: (23, 10, 1440)
#train Label size: (23, 960)
#test size: (6, 10, 1440)
#test Label size: (6, 960)

# create and fit the Simple LSTM model

#model_LSTM = Sequential()
model_LSTM.add(LSTM(units = 1440, input_shape=(trainX.shape[1], trainX.shape[2])))
model_LSTM.add(Dense(units = 960))

Edit: Following tips based on this example to help you when preparing your input data for LSTMs:

The LSTM input layer must be 3D.
The meaning of the 3 input dimensions are: samples, time steps, and features.
The LSTM input layer is defined by the input_shape argument on the first hidden layer.
The input_shape argument takes a tuple of two values that define the number of time steps and features.
The number of samples is assumed to be 1 or more.
The reshape() function on NumPy arrays can be used to reshape your 1D or 2D data to be 3D.
The reshape() function takes a tuple as an argument that defines the new shape.


Should I use a regression or classification framework to predict rankings? My dependent variable is a performance ranking (relative to the other individuals). For example, one individual would be ranked first, another second, another third, etc. Would I use a classification or regression framework to model this?

Sounds like a classification task. Since your $y$ is ordered, you may also consider "ordered probit" or "ordered logit". These methods make sure that the order in $y$ is accounted for.


Resources for data science applications in finance/banking What are good resources for data science applications in finance/banking ?

I want to know about the following:

Trends in finance/banking area
What creative companies do in finance/banking area with data science
Major open data sources in finance/banking in Canada

I subscribed to several data science news but none of them focus on finance/banking. I need daily/weekly news and reports for my internship. This is my job and I will be evaluated based on this so it is very important for me to get enough information in time.

Any suggestions? Thank you very much!

Take a look at O'Reilly free Ebooks There are a couple of resources for banking/finance/fintech.

There are some sites from which you can work on for learning more about predictive modelling in this sector:

1.) https://inclass.kaggle.com/c/name-that-loan-open/data - Kaggle dataset   for prediction of interest rates

2.) https://datahack.analyticsvidhya.com/contest/practice-problem-loan-prediction-iii/ - Dataset on Analytics Vidhya for loan prediction

3.) There is also a twitter handle with the name @StatCan_eng posting datasets related to Canada specifically.


Are Convolutional Neural Networks a one model, multiple output layers.? I'm writing a university report on for the Toxici comment classification Kaggle competition, comparing different attempt made with different models and I want to know if Convolutional Neural Networks are One model, multiple output layers model.

ONMO are also known as multi-task learning, this approach would have one input layer, one set of hidden layers, and one output layer for each label.

On the other hand, CNN are several layers of convolutions with nonlinear activation functions like ReLU or tanh applied to the results. It uses convolutions over the input layer to compute the output rather than a fully connected layer, or affine layer.

Convolutional neural networks implement the convolutional filter technique. This is more or less a sliding window, usually (but not limited to) convolving across the width, heights and depths of images.

If you have a single task or multi learning learning task depends on what you want this network to output. Convolution rather describes the behavior of a single layer processing some inputs into some intermediate outputs. The characteristics of your last layer determine if it is used for single or multi tasks. To my understanding this is rather independent of the convolution.


Feature importance in neural networks Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, there is no feature importance model in keras. There are three options I can use, correlation ratio between the variables, kendals rank coefficient values and lasso regulation. Which one do you think is suitable to be used for feature selection in neural networks?

For feature importance, you might want to consider using Shapley values or LIME. There are some examples here.


What are some ways to transition from additive to multiplicative(or vice-versa) changes in time-series quickly? Let's say I have a variable x and as time progresses, I want to update the value of x in additive/multiplicative manner using some attributes.
At t=t, new_x = x * some_percentage or new_x = x + some_absolute_number.
Is there some way to transition from additive change to multiplicative change or vice-versa and the number by which we can? It's fine if we focus only on one of the additive or multiplicative by reactively changing the percentage_increase or absolute_change.

A toy example task (hypothetical)
For a salary of 100k, an increase of 50% is significant but for a salary of 100(could be pocket money of some child), a 50% increase might not be enough. When we are discussing a smaller number of salaries, we need to increase the salary by absolute number. For a salary of 100, we can increase it by 70 to make it 170)

Are there any ways or literature to handle this which can adapt quickly?

The problem with your request is that both ways are always possible. For any pair of values x and new_x you can choose to represent the increase either as an addition or a multiplication:

additive increment: new_x -x
multiplicative coefficient: new_x / x

So you need a criterion to decide which one to choose, for instance a threshold: if x is lower than 100, use addition otherwise use multiplication.
Additionally if any kind of change can happen at anytime it's completely arbitrary, there's no way to represent this in a systematic way.


RandomForestClassifier : binary classification scores I am using sklearn's RandomForestClassifier to build a binary prediction model. As expected, I am getting an array of predictions, consisting of 0's and 1's. However I was wondering if it is possible for me to get a value between 0 and 1 along with the prediction array and set a threshold to tune my model.
Many thanks in advance

Referring to http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html, you are using teh predict method to derive the binary response.  If you call predict_proba, you will get an array of probabilities corresponding to the binary response.


No graph is displaying while plotting value with time interval using python I have a large dataset with values and date and time. So I want to plot graph of value with time and date. So I wrote a code for time difference. Afterthat I tried to store value into that time difference. Afterthat I tried to plot it. It run with no graph. Can anyone helps me to solve this error?
here is my code:

x= df1,iloc[:,2]
time_interval = 14400 #interval in seconds (14400s = 360 minutes)
date_array = []
date_array.append(pd.to_datetime(df1['date'][0]).date())
start_time = []
end_time   = []
temp_date  = pd.to_datetime(df1['date'][0]).date()
start_time.append(pd.to_datetime(df1['time'][0], format='%H:%M:%S').time())
for i in range(len(df1['date'])):
cur_date = pd.to_datetime(df1['date'][i]).date()
if( cur_date > temp_date):
    end_time.append(pd.to_datetime(df1['time'][i-1], format='%H:%M:%S').time())
    start_time.append(pd.to_datetime(df1['time'][i], format='%H:%M:%S').time())
    date_array.append(cur_date)
    temp_date = cur_date
    end_time.append(pd.to_datetime(df1['time'][len(df1['date'])-1], format='%H:%M:%S').time())
  datetime_array = []
  for i in range(len(date_array)):
s_time = datetime.datetime.combine(date_array[i],start_time[i])
e_time = datetime.datetime.combine(date_array[i],  end_time[i])

timediff = (e_time - s_time)
#num_periods = int(timediff.total_seconds()/t3ime_interval) +1
num_periods = abs(int(timediff.total_seconds()/time_interval)) + 1
time_list = pd.date_range(start=s_time, end = e_time, periods=num_periods ).to_pydatetime()
datetime_array.extend(time_list)
time_stamps = [datetime.datetime.strftime(t,'%H:%m:%S') for t in datetime_array]
x = np.zeros([num_periods], dtype='timedelta64[s]')
plt.xticks(np.arange(num_periods), time_stamps)

my csv file:

after running the code output came like this:

image:

In python you generally have all the libraries available to you. It is hard to find sometimes but you should rarely need to write out so much code. Try this out.

I created some dummy data using the same date formats as you have:

import pandas as pd
import matplotlib.pyplot as plt

data = {'date': ['08/06/2018', '8/6/2018', '8/6/2018', '9/6/2018'],
        'time': ['6:15:00', '12:45:00', '18:15:00', '6:15:00'],
        'x2': [1, 4, 8, 6]}

Now we will make a pandas DataFrame with this dummy data

df = pd.DataFrame(data)

Now we can get our x-axis datetimes by first concatenating the dates and times together separated by a space. Then we will get pandas to parse these datetimes.

datetimes = pd.to_datetime(df['date'] + ' ' + df['time'],
                           format='%d/%m/%Y %H:%M:%S')

You can then plot your data using

plt.plot(datetimes, df['x2'])

Put your csv file in your workspace. Then you can use this following code

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv(r'temp.csv')
datetimes = pd.to_datetime(df['date'] + ' ' + df['time'],
                           format='%d/%m/%Y %H:%M:%S')

plt.plot(datetimes, df['x'])
plt.show()

import matplotlib.dates as mdates

fig, ax = plt.subplots(1)
fig.autofmt_xdate()

plt.plot(datetimes, df['x'])
plt.xticks(rotation=90)

xfmt = mdates.DateFormatter('%d-%m-%y %H:%M')
ax.xaxis.set_major_formatter(xfmt)
plt.show()


Converting sparse IndexedSlices to a dense Tensor I got the following warning:

94: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 1200012120 elements. This may consume a large amount of memory.

For the following code:

from wordbatch.extractors import WordSeq
import wordbatch
from keras.layers import Input,Embedding
...
wb = wordbatch.WordBatch(normalize_text, extractor=(WordSeq, {"seq_maxlen": MAX_NAME_SEQ}), procs=NUM_PROCESSOR)
wb.dictionary_freeze = True
full_df["ws_name"] = wb.fit_transform(full_df["name"])
...
name = Input(shape=[full_df["name"].shape[1]], name="name")
emb_name = Embedding(MAX_TEXT, 50)(name)
...

That is I make use of WordSeq (from WordBatch) output from the Embedding layer of a GRU network. How should I modify the code to make it work without converting to dense tensor?

I've had the same issue with the Embedding layer in Keras. The solution is to explicitly use a TensorFlow optimizer, like here:

model.compile(loss='mse',
               optimizer=TFOptimizer(tf.train.GradientDescentOptimizer(0.1)))


How to load and run feature selection on a dataset with 5,000 samples and 500,000 features? I have a dataset with 5000 samples and 500,000 features (all categorical with a cardinality of 3).
Two problems I'm trying to solve:

Loading the dataset - I can't load it in memory despite using a computing cluster, so I'm assuming I should use a parallelization library like Dask, Spark, or Vaex. Is this the best idea?
Feature selection - how to run feature selection within a parallelization library? Can this be done with Dask, Spark, Vaex?

5000 samples and 500,000 is not that big - it all depends how much memory you have. Also remember you can always and always optimize your data format. e.g. if they are float64 - do they need to be ? if they are categorical, how they are encoded ? (one character or a 20 character word?) and such. so Yes, if you can load the data into memory good for you if not here are the suggestions:

if you only and only have 5K samples - you must not use all for feature selection.
you can drop features that have very low variance - in an extreme scenario if the variance of a column is 0 - for sure it is useless.
there is something called feature-screening proposed by Fan et. al from Princeton https://orfe.princeton.edu/abstracts/feature-screening-distance-correlation-learning - in short: you can lower your dimension by using a univariat model and then afterwards use multivariate-feature selection models.


Support Vector Machine Errors What is the difference between SVM classification error, SVM margin error, and SVM total error ? Is there any clear definition for them ? And what is C parameter in SVM ? Its totally confusing me !!!

A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin.

When a missclassification occurs, it is because a given point is on the
wrong side of the separating hyperplane, and that's called a
classification error.
Whenever a point is inside the margin, that counts as a margin error.
The total error of a SVM, is the sum of the classification error and
the margin error.

Now, to understand the C parameter, you've got to know that there are 2 types of SVMs:

Hard margin: these try to maximize the margin without introducing any
kind of errors.
Soft margin: which also share the same objective, but they allow for some classification and margin errors to occur. The number of errors allowed is controlled by the C parameter (which is often called the penalty parameter): if C is small, the SVM allows for some errors and therefore reaches better generalization; but as C increases, the SVM penalizes these errors more and more, eventually reaching the point that it allows no errors at all.

As you can see bellow on the left, a very large C restricts the model by not allowing any errors, showing very bad results when outliers are present. On the right, by using a small penalty parameter, we let that negative outlier to be in the positive area, therefore maximizing the margin value and reaching a more respectful separating hyperplane.

Although, observe that if you have a soft margin SVM with a very large C value, it will behave just like a hard margin SVM.


How to visualize Ensemble Models ( Random Forest) with 1000 estimators I am working on classification problem where I need to categorize  the user in buy/ non-buy category. I have around 100 + features or predictors to predict the behavior of user.

I tried to implement with Random forest and Gradient Boosting to get better prediction compare to decision tree. I am getting better performance when I evaluate against performance parameters like roc_auc,accuracy, precision and recall when using ensemble techniques.

I also extracted important features that are responsible for my predictions but I am not able to visualize the model fully. Some how random forest works as black box where i am not getting what is the contribution of each tree, which features are been considered in each trees, etc.

Is there any way through which I can find out more information from  Random Forest Model?

You can try SHAP which visually explains the output of (many) machine learning model(s) including LightGBM and XGBoost.

However, please note that it will not give you the entire Ensemble Model (Trees) as picture.
Further note that it doesn't work for RandomForest


Method to assess text credibility I am searching for an automated method (ideally a python package) that produces a score to assess the credibility of a given text (e.g. from a webpage).

I am not searching for:

text complexity assessments (i.e. how long sentences are and how many difficult words are used) as for example flesch reading ease, smog
index, flesch kincaid grade, coleman liau index, automated
readability index, dale chall readability score, difficult words
index, linsear write formula, or gunning fog.
text coherence (i.e. how well the next sentence fit with the previous one) as for example Text Coherence Analysis Based on Deep Neural Network

Why is complexity/coherence not the same credibility? Because many texts advertising for example homeopathy use long complex scientifically sounding and complex word loaded sentences while being nonsense in terms of trueness. Therefore I am wondering if there is any method to assess the credibility/reliability of a given piece of text/webpage information automatically?

I don't think there's anything close to doing this:

It would be very hard to even define the task objectively, as different humans wouldn't agree about what is credible or not.
It would require a complex system to represent reliable background knowledge... and again people wouldn't agree what should be considered "reliable" or not.
Generally the state of the art in NLP is still far from solving tasks related to Natural Language Understanding in a satisfying way. Juding the credibility of a text requires not only a real understanding of the text but also an ability to reason at a higher level. It's not clear whether this level of AI can ever be reached.

If you find a package which pretends to achieve this task, try to apply it to its own documentation because it's not credible ;)


Create new performance indicators (error metrics) I am wondering if any of you happen to know of a procedure/approach/rationale to develop new performance indicators (error metrics) that can be used to evaluate the prediction capability (say, accuracy) of a ML model? I am looking for any information that could be helpful to develop metrics for regression or classification problems. Thanks!

In research, new evaluation measures are proposed all the time but very often for specific use cases: either because it's a new task for which there is no established evaluation standard, or because the new measure provably solves some issues/limitations with the previous evaluation standard.
The purpose of an evaluation measure (or performance indicator),  is to represent the quality of the predictions by any system for a given task. This implies that:

An evaluation measure is selected specifically for a task. Some measures are very common because many tasks can be framed in a standard way, for example as a classification problem.
The central problem of evaluation is how to define "quality" in a formal way. A performance indicator is always a simplified representation of "quality", but it must correlate at least approximately with what a human expert of the task would call "quality".

Naturally the first step for developing a new evaluation measure is to select the target task (or group of tasks). Then one has to demonstrate that the new evaluation measure brings some significant advantage in terms of measuring the quality of this task. Of course this is the hard part, and it always depends specifically on the task.
For example, there have been numerous evaluation measures proposed for Machine Translation (there's even a regular Shared Task devoted to the topic). The way a MT evaluation measure is itself evaluated is against human judgements: if the automatic measure correlates strongly with human judgements about the quality of translation, then it's a good measure.


How do I encode the categorical columns if there are more than 15 unique values? I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have more than 15 unique values such as country.
Do I still use one-hot encoding? Or is there an alternative? Is using regression on this dataset a good idea?

[edit] See also Carlos' answer, I think it's better than mine.
You should use one hot encoding for the categorical features. Replacing categorical values with numerical ones would be a bad idea, because it introduces order between the values and the model would try to find patterns based on this order (e.g. 'x < 4').
If there were really too many different values then it's often a good idea to remove or replace the least common ones, but it doesn't seem to be an issue with this data.
For the record, more than 15 different values is nothing to be afraid of. For example when working with text data it's common to work with thousands of values for a word.


How to encode a job description for machine learning I'm working on a sample project and one of the features is the job description of a person (categorical, for example: blue-collar, retired, unknown, unemployed, student, etc.). Since in the future more job descriptions could be used, I don't think one hot encoding it's the best approach. How would you encode it without using one hot encoding??

I would first start by using something like scikit-learn LabelEncoder. Examples are in the documentation.

EDITED : With this option, you convert every string to a integer starting from 1.

You can also think about trying to convert the strings into hash values.


How to adaptively sample n-dimensional data and build an optimum training set My input space is at least 10 dimensional (after reducing it by various component analysis such as PCA) and the output space is 4 dimensional. I am building a neural network that works something like a function approximator which takes the above 10D data as input (10 neurons in input) and spits out 4D data as output (4 neurons at the output). In between there are hidden layers.

I need to build a good training set that covers all the possible values of input and output. Although it may seem like there is a very large combination possible for 10D input + 4D output, in reality, the input/output is restricted by real world constraints. The problem I'm facing is this: In some part of the training sets, I need to sample the data with higher resolution, in other parts, I can go away by sampling with lower resolution. I can obviously sample the entire dataset with high resolution, but then the number of samples is more than 10 trillion and I know that most part of the dataset is pretty slowly varying and sampling this part with lower resolution is enough. My guess-estimate is, if I could sample it properly, I could get away with 4-5 orders of magnitude less numbers of sample.

My questions is, what should I do/build in order to have something that will adaptively sample a high dimensional dataset (change resolution whenever necessary) and give me an optimum training set?

NB: one of my colleagues advised me to use Markov Chain Monte Carlo method. I am not sure whether it is the best way to go. Please share your opinions.

Take a look at active learning.

https://en.m.wikipedia.org/wiki/Active_learning_(machine_learning)

In this approach you can train with a subset of samples and add more depending on what samples are most informative for the model.

Alternatively you could try reducing the number of samples by clustering and choosing a small number of samples from each cluster.


Facebook's Huge Database I assume that each person on Facebook is represented as a node (of a Graph) in Facebook, and relationship/friendship between each person(node) is represented as an edge between the involved nodes.

Given that there are millions of people on Facebook, how is the Graph stored?

Strange as it sounds, graphs and graph databases are typically implemented as linked lists. As alluded to here, even the most popular/performant graph database out there (neo4j), is secretly using something akin to a doubly-linked list.

Representing a graph this way has a number of significant benefits, but also a few drawbacks. Firstly, representing a graph this way means that you can do edge-based insertions in near-constant time. Secondly, this means that traversing the graph can happen extremely quickly, if we're only looking to either step up or down a linked list.

The biggest drawback of this though comes from something sometimes called The Justin Bieber Effect, where nodes with a large number of connections tend to be extremely slow to evaluate. Imagine having to traverse a million semi-redundant links every time someone linked to Justin Bieber.

I know that the awesome folks over at Neo4j are working on the second problem, but I'm not sure how they're going about it, or how much success they've had.


How do you apply hypothesis testing to your features? How do you apply hypothesis testing to your features in a ML model? Let say for example that I am doing a regression task and I want to cut some features (once I have trained my model) to increase performance. How do I apply hypothesis testing to decide whether that feature is useful or not? I am just a bit confused about what my null hypothesis would be, level of significance and how to run the experimentation to get the p-value of the feature (I have heard that a level of significance of 0.15 is a good threshold, but I am not sure).

For example. I am doing a regression task to predict the cost of my factory, considering the production of three machines (A,B,C). I make a linear regression with the data and I find out that the p-values of machine A is greater than my level of significance, hence, it is not statistically significant and I decide to discard that feature for my model.

The data looks something like this

d = ('Cost': [44439, 43936, 44464, 41533, 46343], 'A': [515, 929, 800, 979, 1165], 'B': [541, 710, 675, 1147, 939], 'C': [928, 711, 824, 758, 635, 901])
df = pd.DataFrame(data=d)

If you want to see the full data, I have taken this example from a video on Youtube. I couldn't really understand how he drew that conclusion about the level of significance and how he used hypothesis testing in this case.

The link is below, it starts from Min 4:00 (it is only 3 mins long after that)

Linear Regression using hypothesis testing

welcome to StackOverflow.

I will try to summarize as much as possible, but we have to cover a lot of concepts to properly answer your question, if something is not clear, please let a comment and I can change/correct the answer.

First of all, I will assume from the tone of your question that you know what a hypothesis test is.

Second, in the context of Linear Regression, you usually assume that your features are normally distributed, iid (identical independent distributed), constant error variance (that is, they do not vary with X attributes) and so on. These are strong assumptions, and it is important to have it in mind since we are calculating our regression coefficients using data that have these properties.

Put in simple terms, you have your set of features X, in this case A, B, and C, and you want to predict cost, therefore you have the following function:

Thus, you use above equation to minimize the RSS (Residual Sum of Squares) and find out what are your coefficients.

Now, in the context of hypothesis test, you want to verify if your coefficients are statistically relevant, that is, saying it on simple words, you want to check if they are far enough from zero in order you can say they are relevant, not a statistical incident. How exactly you do that? Creating a Null Hypothesis that you coefficient is zero, and an Alternative Hypothesis saying otherwise.

(Example to calculating Hypothesis Test for Beta_1 coefficient)

Then you calculate t-test for your coefficient:

So you are assuming your coefficient has a t-distribution and you want to test if it is far enough from zero to determine its relevancy.

You then calculate your t-test value and estimate a p-value. Therefore, in the end, you will ask the following question: What is the probability that my coefficient comes from a distribution centered around zero? If you have a high p-value, it shows you that probably your coefficient is zero, or close to it and not significant. Otherwise, if you have a low p-value, then it is not likely that it comes from a distribution around zero, then you should take this coefficient (and feature) in your analysis.

In the example from the video, he uses a rule that all coefficients above 0.05 are not statistically relevant. Here, 0.05 is widely used as a cutoff point, but you must be aware that there are a lot of issues regarding this choice and the debate inside scientific community is not settled.

If you really want to understanding all nuances involving Linear Regression, Statistical Tests and problems when modelling data using this technique, I highly recommend that you read the Introduction to Statistical Learning Book - Chapter 3. All images and insights were taken from there, and for me it is my reference book for Machine Learning techniques.

I hope this helps to clarify your question.


Is there any named entity reconginition algorithm trained for the french language? I am trying to implement a utility for my mobile application to perform some actions based on user questions. I need an algorithm to extract named entities from a text string (French grammar). I have used nltk's interface to Stanford's NER models but it works only for English (A subset of other languages is supported but i can't find French). I have also used Polyglot but it seems that it doesn't do the work very well (Maybe the models I am using are not very well trained). I don't know if there is any free REST API that can do NER for the French language or any other algorithm or even an already trained model for nltk/Stanford NER.

Yes, there is a french model free and ready to use via the spaCy package!

Here are the small amd medium sized models, that should be ready to go.

Here is the basic summary of the dataset, shown at the spaCy website:


Layman's comparison of RMSE I don't have a maths / stats / data science background and need to evaluate which of the two evaluations below (numerical regression on Amazon Machine Learning) predict more accuracy. Both models use the same data set but it's looking at different time frames both on the independent and dependent variables.

How can I evaluate which one of the two models is more accurate? And is there a way to tell how accurate these two models are in general (e.g. 75%)?

Model 1

Model 2

Typically you want a smaller RMSE and without getting into detail it should be sufficient to just take the smaller one.  However, I am concerned because you state that the models were ran on the same dataset but at different timeframes.  Since RMSE scale depends on the dependent variable scale, it's entirely possible that these two timeframes are scaled different.  A somewhat contrive example would be energy consumption.  I would expect a model trained on daytime consumption to have a higher RMSE than for one trained between 1am and 3 am.  In that case, comparing the RMSE may be meaningless.  You can try to normalize your data and RMSE to help with this, but i'm unsure if AWS provides this ability.

As for your second questions, you really won't get a 75% accurate number for regression.  You can look at the deviations of the residual or do cross valiadtion and see how well the model performs.

Again this may not be possible in AWS.

edit:  I juse realized that the histograms were residual plots.  Do three things  Increase bin size.  Check to see if the residuals are centered around 0 and then check if there is skewness in the data.  If the data is centered around 0 and symmetric then you can say the model error is basically random and does not favor over or under predicting.  If the data is not centered around 0 and there is skewness, then the errors can be systematic and then in that case considering adding more variables.


Useful metrics to compare network-output image to true image? I'm designing a supervised network that would require to output an image. I'm wondering what are the best metrics to find similarity between the output and actual target image. So far, my best assumption is to calculate the distance of RGB values, and give less weight to the background. Are there better ways?

Widely adopted method to measure similarity between two images is structural similarity:

Wikipedia
Useful links
Implementation

However, if you aim on training a network that should produce images of desired properties, I would consider GANs framework. In this case, the similarity between the generated and target image is defined by a special discriminator network.
Depending on what is your task, there are many GAN sub-types.


Data augmentation in deep learning I am working on a deep learning project for face recognition. I am using the pre-trained model VGG16.

The dataset has around 100 classes, and each class have 80 images. I split the dataset 60% training, 20% validation, 20% testing. I used data augmentation (ImageDataGenerator()) to increase the training data.

The model gave me different results when I change ImageDataGenerator() arguments. See the following cases:

Case1:

 train_datagen = ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest')
validate_datagen = ImageDataGenerator()
Test_datagen = ImageDataGenerator()

Case1 result: High training accuracy and validation accuracy, but the training accuracy is lower. check the following image:

Case2:

train_datagen = ImageDataGenerator(
      rescale=1./255,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True)

validate_datagen = ImageDataGenerator()
    Test_datagen = ImageDataGenerator()

Case2 result: Overfitting. check the following image:

Case3:

train_datagen = ImageDataGenerator(
      rescale=1./255,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True)

validate_datagen = ImageDataGenerator(rescale=1./255)
    Test_datagen = ImageDataGenerator(rescale=1./255)

Case3 result: High training accuracy and validation accuracy, but the training accuracy is lower.. check the following image:

1-  Why does using augmentation in validation and testing data ImageDataGenerator(rescale=1./255) in case3 give different result than case2?

2- Is adding ImageDataGenerator(rescale=1./255) to the testing and validation better than not adding it?

3- Do you think there is a problem in the result of the first case?

1 and 2: If you rescale you images, you should do it on all partitions: training, validation and test. If you only rescale your images on the training set, then your network will see very different values (0~255, vs 0.0~1.0) on validation/test set and therefore give poor accuracy. That's your case 2.
I don't see any obvious problem.


How are Q, K, and V Vectors Trained in a Transformer Self-Attention? I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that one will understand their purpose after reading about how they are used for attention. I believe I understand what they do, but I'm unsure about how they are created.

I'm aware that they come from the multiplication of the input vector by three corresponding weights, but I'm not sure how these weights are derived. Are they chosen at random and trained like a standard neural network, and if so how if there's no predefined attention data in the training corpus?

I'm very new to this, so I hope everything I'm saying makes sense. If I've got something completely wrong, please tell me!

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states.

In Equation 1 of the Attention is all you need paper, these are just parameters that come from outside:

It just says, what do you do, if get some queries, keys, and values from somewhere outside. This also the case of the unnumbered equation on top of page 5. Here, you only project the keys, queries, and values for the heads of multiple attentions.

Here $W_i^Q$, $W_i^K$, and $W_i^V$ are learnable parameters and they learned by the standard back-propagation algorithm.

Note that although keys and values (and queries in the self-attention) are equal only at the input of the $\text{Multihead}$ function. The $\text{Attention}$ function already gets the projected states.


Models systematically underestimate values on the test set, why? I used different models to train on the test set and to predict on the test set. The commonality is that all models underestimate the true values on the test set. Which steps should I take to understand why?

Welcome to data science SO. It seems your model has a bias towards a lower estimate of the true values. This can happen for a variety of reasons. Say you are trying to fit a straight line on data that was generated by a second degree polynomial. The straight line fit will have a bias. Typically bias happens because the model you are trying to build is insufficient able to capture the dynamics of the data you are trying to fit it on. I understand this is very generic but without more info about what model you used or any plot of the test set it is hard to help you more.


How to persist data scaler for predictions I have a Support Vector Machine in Scikit-learn (Python) that gets trained once in a while when enough new data has accumulated (user help train the model by submitting new data).

I store the model in .pkl format for persistence. However, the SVM needs scaled data and I'm wondering what would be a good way to persist the scaler, since it was fitted on the train data that changes over time (as users submit more data).

Is there a common solution for this? Or is there a value that I can store with the model that works as some sort of seed to recreate the scaler?

EDIT (added implementation):

I have a database with a dataset. users periodically add data points to this dataset. I also have an estimator database which keeps track of all the trained models and their .pkl file locations. When enough new data points are added the system will start to train a new estimator. It extracts all the features and scales the data, then trains a model. After that it compare the existing best model with the newly trained one and if the new one is better, it sets the new one to 'active'. There is only one active model a a time and that is the one that will be used for predictions on new data. (I hope this helps)

So after looking some more I found this question: save scaler model in sklearn
which says that one can picle the scaler aswell, which works.


Is there an algorithm for sampling shortest paths? I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an algorithm to do it? Would swapping min->sample in A* be sufficient?

I don't know whether this answers your question. But have a look in these algorithms:

The Shortest Path Faster Algorithm (SPFA) is an improvement of the Bellman–Ford algorithm which computes single-source shortest paths in a weighted directed graph. The algorithm is believed to work well on random sparse graphs and is particularly suitable for graphs that contain negative-weight edges.

Dijkstra's algorithm solves the shortest-path problem for any weighted, directed graph with non-negative weights. It can handle graphs consisting of cycles, but negative weights will cause this algorithm to produce incorrect results.

The Shortest Path algorithm calculates the shortest (weighted) path between a pair of nodes. In this category, Dijkstra's algorithm is the most well known. It is a real time graph algorithm, and can be used as part of the normal user flow in a web or mobile application.

More Info at:

https://www.sciencedirect.com/topics/computer-science/shortest-path-problem
https://medium.com/basecs/finding-the-shortest-path-with-a-little-help-from-dijkstra-613149fbdc8e
https://link.springer.com/chapter/10.1007/978-3-319-05813-9_9


How do I print full date in the x axis of the line plot here? My code:
import matplotlib.pyplot as plt
plt.style.use('ggplot')
plt.rcParams["figure.figsize"] = [12,6]
def time_series(start, end):
    time_series_df = df[['Date', 'Value']][(df['Date'] >= start) & (df['Date'] <= end)]
    x = time_series_df.Date
    y = time_series_df.Value
    plt.plot(x,y)
    plt.xlabel('Time')
    plt.ylabel('PM2.5 Value')
    plt.title('PM2.5 Time Series')
    return plt.show();

time_series('2014','2019')

For better control over the x-axis formatting, you can use the matplotlib.dates methods. In your case, MonthLocator and DateFormatter could be of interest. These can be used to adjust the x-axis as follows:
import matplotlib.dates as mdates

def time_series(start, end):
    time_series_df  = df.loc[(df['Date'] >= start) & (df['Date'] <= end), ['Date', 'Value']]
    time_series_df = time_series_df.set_index('Date')

    fig, ax = plt.subplots(1)
    ax.plot(time_series_df)

    # Adjust the x-axis
    ax.xaxis.set_major_locator(mdates.MonthLocator(interval=3)) # Month intervals
    ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d')) # date formatting
    # automaticall set font and rotation for date tick labels
    plt.gcf().autofmt_xdate()

    plt.xlabel('Time')
    plt.ylabel('PM2.5 Value')
    plt.title('PM2.5 Time Series')
    plt.show()

Note the interval=3 argument for MonthLocator. This is the frequency of the x-axis ticks.
Resulting plot:

Dataframe use for the plot above:
df = pd.DataFrame({'Date': pd.date_range(start='2014-01-01', end='2018-12-31'), 'Value': np.random.uniform(size=1826)})
df['Date'] = pd.to_datetime(df['Date'])


Words as a input to network I want to build neural network, where my input will be a word(not a sentence). My set of words has many words with different lenght (number of chars). My idea to do it is transform chars to numbers using predefine dictionary, next use it as input. Number of neurons in input layer set as a number of chars in a longest word in my set. In vectors witch are shorter I'll fill them by zero.

My set of words looks that:

1. head
2. hello
3. butterfly
4. hotel
5. fly
6. spy

So for this set my input to netowrk should has 9 neurons. When I transform 'head', I will get vector with 4 numbers and 5 zeros.

It's a good idea? Or maybe you have another better idea how input of this network could look.

EDIT:

My task is described here: ML model to transform words:
And I want to build GAN as an answer suggest.
So this words will be my 'real words' and I want to generate fake words.

The simple way here is to make a one-hot over letters. So you will have many columns with ones and zeros. This may now be an input layer for your nn.


Sparse Matrix - Effect and Solution Can anybody explain me what are the effects on the model if we have sparse data in our dataset. And also how to deal these sparse matrices ?

Thank you.

The idea is really simple, just look at some online resources like https://en.m.wikipedia.org/wiki/Sparse_matrix

The implementation is also really simple. For pandas this page might help you https://pandas.pydata.org/pandas-docs/stable/sparse.html


Percentage of missing values so that we can't perform imputation (This is not a question on ways to handle missing data)

I have a dataset with around $80$ or so features and around $100000$ rows. Several of those features have missing (NULL) values for a "large proportion" (more than 20%) of the records.

Is there a general way to decide the % of the total rows for which a feature has to be missing, so that we can consider it useless and remove it from our analysis? Or does this depend on the context? Indirectly, I want to know the % of missing values for a feature so that imputation is meaningless.

In my opinion that depends on the context, like the question you want to answer, and also on the domain.

If the remaining absolute data set is still big and diverse enough to allow conclusions about that feature, the percentange of records that can not add to that imputation is not a problem, in my understanding.

It might also be that features sometimes are not just missing, but  do not apply to the particular kind of record.


Time series prediction of discontinuous data In the context of time series prediction, I have read that time series is a series of data that taken at successive equally spaced points in time (which means its in order). What if I have a discontinuous time series data, for example:

If I have data that represnt a room temperature within the working hours, specifically from 7:00 am - 3:00 pm. And this data is repeated on the day of the week. So we have something similar to this:

Saturday: [ Time series data from 7:00 am - 3:00 pm]

Sunday: [ Time series data from 7:00 am - 3:00 pm]

Monday: [ Time series data from 7:00 am - 12:00 pm]  -- Only up to 12 pm

The first question:

Is that considered as time series data
Now if I want to predict the room temperature on Monday @ 1:00 pm. How can I do this using given this type of datasets?

Yes, it is time series with missing values. In fact you have process $X_t$, which is room temperature in moment $t$, but you have observations in specific time intervals. So your goal is to find out missing values between non-observed intervals.  There are several methods and packages for solving problems like this, here is an implementation of some methods in R:


Open source data science projects to contribute Contribution into open source projects is typically a good way to get some practice for newbies, and try a new area for experienced data scientists and analysts.

Which projects do you contribute? Please provide some intro + link on Github.

The Julia project is one which I actively contribute to, including the advanced computing and XGBoost libraries. So, I can definitely vouch for it's maintenence and the quality of the community.

Some really good open source data science projects where even the beginners can contribute are:

Sklearn: Always developing at a rapid pace, the sklearn community is always open to new developers and contributors.
H2O: H2O is another fast growing data science projects, working on scalable machine learning and Deep Learning solutions.
Go: Open source data science road map and resources. Not really a technical project, but is very helpful for absolute beginners and aspiring analysts.
Pylearn2: Another fast growing Machine Learning and Deep Learning project.
Vowpal Wabbit: The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research.

Here is a Quora discussion on such projects and some more which are not mentioned in this answer.

Here is a another nice discussion about open source Data Science and ML projects in Python.


Which plot to use for data spanned on multiple years? I've a dataset of - Number of accidents that happens in each state. The dataset spans from 2001 to 2014.

Which plot should I use to represent the information - number of accidents in each state per year,  in 1 plot.

For time data, you should often use a simple Line Chart, with time on the X axis and the other variable dependent of time (here the number of accident) on the Y axis. Since you have several states to plot, you can simply draw one line for each state. to differentiate them, you can use a visualization variable : color, width, pattern ... Color is probably the most relevant here. You can as well write the name of the state near each line.


How to recreate T-SNE dimensions deterministically? So I have a set of 3000 features from which I would like to generate clusters. I passed my features through the T-SNE algorithm to reduce dimensionality to 2 features, and clusters are really visible in this reduced space. How would I go about figuring out a deterministic way to create this same T-SNE output to use for clustering?

Do not use tSNE visualizations for clustering. The results are misleading.

See this great answer: https://stats.stackexchange.com/a/264647/7828

Apart from that, you just need to fix the initial positions. For examples by fixing the random generator seed. But the fact thar it does not work every time should already warn you that it is not too reliable...


Accuracy of Stanford NER I am performing Named Entity Recognition using Stanford NER. I have successfully trained and tested my model. Now I want to know:

1) What is the general way of measuring accuracy of NER model ?? For example what techniques or approaches are used ??

2) Is there any built-in method in STANFORD NER for evaluating the accuracy ??

http://en.wikipedia.org/wiki/Named-entity_recognition#Formal_evaluation :

  To evaluate the quality of a NER system's output, several measures
  have been defined. While accuracy on the token level is one
  possibility, it suffers from two problems: the vast majority of tokens
  in real-world text are not part of entity names as usually defined, so
  the baseline accuracy (always predict "not an entity") is
  extravagantly high, typically >90%; and mispredicting the full span of
  an entity name is not properly penalized (finding only a person's
  first name when their last name follows is scored as ½ accuracy).

  In academic conferences such as CoNLL, a variant of the F1 score has
  been defined as follows:

  Precision is the number of predicted entity name spans that line up exactly with spans in the gold standard evaluation data. I.e. when
  [Person Hans] [Person Blick] is predicted but [Person Hans Blick] was
  required, precision for the predicted name is zero. Precision is then
  averaged over all predicted entity names.
  Recall is similarly the number of names in the gold standard that appear at exactly the same location in the predictions.
  F1 score is the harmonic mean of these two.

  It follows from the above definition that any prediction that misses a
  single token, includes a spurious token, or has the wrong class,
  "scores no points", i.e. does not contribute to either precision or
  recall.


Which algorithms should I use for recommendation system using a graph database? Basically I'm developing a recommendation system using a graph database (specifically neo4j), and I want to apply recommendation algorithms. Since i'm using a graph database, I can see the recommendation problem as a graph problem, and intuitively i can use graph based algorithms for the recommendation system.

From my research, recommendation systems are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And there exists basically two types, collaborative filtering and content based.

I've done a research on the algoritms, and i found some interesting ones:

Weighted Bipartite Graph algorithm
Energy Spread Activation
Union Colors

My question is simple, which other graph algorithms exists that can be used for graph based recommendation system? Or if I use a graph database for recommendation system, the algorithm doesn't necessary need to be a graph based?

Thanks. Any suggestions are welcomed.

A biadjacency matrix of a bipartite graph admits matrix factorisation.

For $m$ items and $n$ users, the biadjacency matrix is an $m \times n$ matrix which can be factorised into two lower-rank factor matrices of sizes $m \times k$ and $k \times n$ respectively. This provides a lower-dimensional representation of how users' preferences vary over products.

This model also has the property that the inner product of the $i$th row and $j$th column of the two factor matrices provide an approximation of user $j$'s rating of item $i$. In other words, the product of the two matrices is an approximation of the original ratings matrix.


Determining whether a Machine Learning model is overfitted with regard to the stability of the features I need to know how would I get to know if I have overfitted my Machine Learning model on the train data. The performance metric I have used is Logistic Loss. Does the stability of the features affect the performance of my model? If yes, how do they relate?

You need to look for the differences in the training loss and the cross-validation & test losses. If those are low, it means the model performs fairly well. Ideally, train loss should be roughly equal to the cross-validation and test losses. If not, the model may be overfitted.
This difference also hints at an insignificant overlap between the train data points and the cross-validation and test data points. Such features are said to be unstable. In such a case, the model only gets to see the data points in the train data and not those in the cross-validation and test data, and is simply overfitted thereby. It, hence, performs poorly. You can verify the same by computing the percentage of data points present in the cross-validation and test data from those in the train data for different features in your dataset.


Comparing two ordered lists Hi I would appreciate it if someone can point me in the right direction.  I'm looking for an algorithm or mathematical theory which I would use to compute the similarity between two ordered lists, where each list element can have n sub-elements.  I will explain with an example:

Suppose I go to a baseball game and I record the sequence of strikes and balls for each of the first 30 players at bat.  My list looks like this, where P is a player, S is a strike and B is a ball. Order matters.

L1: {P1=(S,S,S)}, {P2=(B,B,S)}, {P3=(B,B,S,S)}, ...

My friend goes to a baseball game and does the same thing.  Later, we meet up and compare our lists.  We find that our lists are almost identical except that I recorded a strike for player 16 where my friend recorded a ball.  What are the chances we were at the same game and one of us made a mistake at player 16?

Thanks in advance...

What about a model stating that you have two vectors of size N (where N is the total number of players, maybe unknown to us) where each element belongs to a space of {B,S} sequences, maybe empty. If you then define a distance function between two arbitrary sequences (say, normalized Levenshtein distance for two non-empty ones and some fixed cost when one is missing), you can define cosine similarity between the vectors.

(Obviously, you just consider your sequence a compact representation of sparse vector in this case.)