Python Sklearn TfidfVectorizer Feature not matching; delete? I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.

The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error:

ValueError: X has 4877 features per sample; expecting 2799

Is there a way to delete any features in x_test that were not used in x_train?

I know if I had used a countverctorizer, i could have bypassed the error by not using fit_transform on x_test. But since it is TfidfVectorizer, it won't let me bypass.

I also tried imputation but couldn't get it to work.

Thanks

You should not experience this problem if you use TfidfVectorizer properly.

Demo:

In [58]: from sklearn.feature_extraction.text import TfidfVectorizer

source text

In [59]: text = """I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.
    ...:
    ...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error"""

let's tokenize it to a list of sentenses:

In [60]: from nltk import  sent_tokenize

In [61]: vect = TfidfVectorizer()

In [62]: data = sent_tokenize(text)

yields:

In [63]: data
Out[63]:
['I trained a classifier using TfidfVectorizer in Sklearn.',
 'I then pickled the model for future use.',
 'The new x_test that I want to make predictions on, has more features than the x_train from the model.',
 'This is the resulting error']

now we can fit and transform our data set:

In [64]: X = vect.fit_transform(data)

result:

In [65]: X
Out[65]:
<4x31 sparse matrix of type '<class 'numpy.float64'>'
        with 34 stored elements in Compressed Sparse Row format>

In [66]: vect.get_feature_names()
Out[66]:
['classifier',
 'error',
 'features',
 'for',
 'from',
 'future',
 'has',
 'in',
 'is',
 'make',
 'model',
 'more',
 'new',
 'on',
 'pickled',
 'predictions',
 'resulting',
 'sklearn',
 'tfidfvectorizer',
 'than',
 'that',
 'the',
 'then',
 'this',
 'to',
 'trained',
 'use',
 'using',
 'want',
 'x_test',
 'x_train']

now let's feed it a data set with unknown words (features):

In [67]: new_dataset = ["let's see what happens to unknown words", "Yet another sentence."]

In [68]: X2 = vect.transform(new_dataset)

In [69]: X2
Out[69]:
<2x31 sparse matrix of type '<class 'numpy.float64'>'
        with 1 stored elements in Compressed Sparse Row format>

it worked properly - all unknown features (words) have been ignored:

In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)
Out[70]:
   classifier  error  features  for  from  future  has   in   is  make   ...     the  then  this   to  trained  use  using  want  \
0         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  1.0      0.0  0.0    0.0   0.0
1         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  0.0      0.0  0.0    0.0   0.0

   x_test  x_train
0     0.0      0.0
1     0.0      0.0

[2 rows x 31 columns]


Scaling neural networks While using Neural Networks (TensorFlow: Deep Neural Regressor), when increasing your training data size from a sample to the whole data (say a 10x larger dataset), what changes should you make to the model architecture (deeper/wider), learning rate and hyper parameters in general?

How much of trial and error how much of heuristic logic is involved in making these changes?

I don't think you ought to change much in the model definition.

You should, however, consider the amount of time it takes to train on the complete dataset. If it takes too long and you are still in the testing phase, you want to reduce the number of epochs to obtain results faster and make changes in the model accordingly.

I suggest plotting all the metrics and try to understand if the trend is positive or negative. If it's positive the changes you are doing are correct of course!

Then, once you are happy with the hyperparameters, put like epochs=100 and leave the model to train for a whole night, then plot again the learning curves and decide when to stop earlier, or use early_stopping.


How do we decide on the classification algorithm to use with huge training size? I am solving a questions binary classification problem and the training size for this is huge(291 billion). The data has bloated because of using tfidfvectorizerfor the questions column. Here, in the problem, I have to classify the questions.

I have used Logistic regression and have also kept MultinomialNB, Randomforest and svm for training. However, instead of doing such trial and hit method, Is there a logical explanation why one classification algorithm must perform better than others in this context.

Previously, I have tried randomforest and logistic regression for spam filtering and observed that the training error was less for logistic regression as compared to randomforest. I understand that it could be an overfitting solution.
But Is there a way I can say for certain that 'this' is the classification algorithm you must use.

Note: I'm am yet to remove stopwords and do some dimensionality reduction.

I don't think there is a clear-cut criterion to decide on what method to use. From what I see, you have a good amount of data and the problem is "complex" (language). This are reasons to go for "deep" learning such as neural nets or boosting. The reason is that both can handle "non-linearity" very well.

Another thing that comes to my mind is that when you use for instance logit, you man have the problem that you have many features (words) and that only a certain share of the features might be relevant. So in this case you would try to get rid of features which are not helpful in making good predictions. With logit, a lasso approach (l1 penalty) would be a natural thing to do, to "shrink" features.


Sampling with replacement, specify the probabilities I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class.

This is how I would do it in R.

# Vector to sample from
x <- c("User1","User2","User3","User4","User5")

# Occurenciens from which to obtain sampling probabilities
y <- c(2,4,4,3,2)

# Calculate sampling probabilities
p <- y / sum(y)

# Draw sample with replacement of size 10
s <- sample(x, 10, replace = TRUE, prom = p)

# Which yields (for example):
[1] "User5" "User1" "User1" "User5" "User2" "User4" "User4" "User2" "User1" "User3"

How can I do the same in Scala / Spark?

Scala:

def weightedSampleWithReplacement[T](data: Array[T],
                                     weights: Array[Double],
                                     n: Int,
                                     random: Random): Array[T] = {
  val cumWeights = weights.scanLeft(0.0)(_ + _)
  val cumProbs = cumWeights.map(_ / cumWeights.last)
  Array.fill(n) {
    val r = random.nextDouble()
    data(cumProbs.indexWhere(r < _) - 1)
  }
}

Spark has an RDD.sample() method that can sample without replacement, though not with weights. You could probably adapt that method along the lines above to do this however.


Weighted degree in Multidimensional networks Does there exist a definition for weighted degrees of multidimensional networks?

I understand that the basic definition would be:

  Let $v\in V$ be a node of a network $G=(N,V,L)$. The function Weighted Degree: $$V \times P(L) \longrightarrow \mathbb{N}$$ is defined as

  $${weighted\_degree}(v,D)=\sum_{(u,v,d)\in G, d\in D}\ weight(u,v)$$

  where $D\subseteq L$ and $weight(u,v)$ is the weight of the edge $(u,v)$.

I cannot find this definition anywhere on the Internet. But would that be right? Is a multidimensional version of any interest?

I think your weight function should depend on $d$ as well.

The contribution from a single dimension is just the sum of incoming weights. Even for simple graphs this does not seem to have a standard name, although it is quite commonly encountered.
In the context of random walks it is sometimes called simply degree, in the context of empirical networks, I found node strength.

I think this is a simple and natural statistic to look at if you wish to model something as a multidimensional network. However, being a linear sum of rather trivial functionals over simple graphs, I doubt it is of much fundamental interest.


Deep model ensemble giving different results I am making an ensemble of deep models for solving a classification problem. The initial weights follow the default distribution of keras layers. Each time I run the model(train the 'n' models, make predictions and then get their mode), I get a different result. What could be the possible reasons(rather errors) for this? And how shall I handle them?

Getting varying results at the level of F1 score is possible but should be high, as your are essentially taking the average over many values (assuming a reasonably sized dataset).

There are however many sources of random behaviour that could be creeping in. From most obvious to most obscure:

Are the definitions and parameters of each of your models identical each time?
Do you use the same train/validation/test splits for each session?
Are your training hyper-parameters (number of epochs, learning rate, early stopping) the same each time?
Do you set the random seed in Numpy/Keras/Tensorflow?
Did you set any Nvidia CUDA flags for determinism?

Regarding the final point, GPU determinism - this is a very new thing in the context of deep learning and many people believed it was impossible, but there is good progress being made. Check out these video/slides from Nvidia GTC conference this year.

These are really just the components of each individual building block i.e. each deep model. You could also introduce random behaviour perhaps in the way you chain these models together to create your ensemble.


Difference - Batch Training or training multiple times? I have a question on batch learning of neural network.
A neural network learns in batches and modifies weights in every iteration. Question: If I save checkpoints after a batch, and then load the weights at a later time and train with new batch, will it be different from training both batches in 1 go?

Example: If I have a batch size of 100 and training data of 1000 points. So would it be different, in outputted checkpoint file, if I train with 9 batches (900 data points) in one go -> save checkpoint -> load checkpoint next day -> train with last batch -> save checkpoint ... Vs give all 1000 datapoints (i.e., 10 batches) -> train -> save checkpoint file ?

There will not be any difference between :

Training all batches + Epoch in one go
Saving a checkpoint and resuming the training later

Saving a checkpoint is a very good practice. It is not uncommon for training to fail due to environmental issues (E.g.: Spot instance on AWS gets kicked out , PC crash).

With Libraries like Keras, it is very straight-forward to checkpoint models.


How to deal with spelling errors NLP I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in other words. I have more than a thousand possible product names.

Currently I'm just using regular expressions with a list of product names to find and extract the product names in each row of the dataset, but it's not working well in the cases of misspelled product names.

Since I already have more than 50,000 rows in which the product was extracted by hand in cases where there is some product from the list in the column. I am wondering if after breaking each description into multiple lines (tokenization) could apply some classification/search method to detect if there is some product and which is in the description.

  Example: Medical product used in cancer treatment "PRODUCT XXXXYYYY"
  where the XXXX and YYYY are the first and second name of the product.

I think most of the description would be unused, because as in the example above, simply being a medical product would not help at all since there are several possibilities, but the presence of the wrongly spelled words that describe a product specific would be helpful. I am open to suggestions on how to deal with these spelling errors.

Other options would be to...

Compare similar text sequences
Compare similar string sequences
Use fuzzy matching

Fuzzy Matching:
library(fuzzyjoin)
# https://stackoverflow.com/questions/26405895/how-can-i-match-fuzzy-match-strings-from-two-datasets

a <- data.frame(name = c('Ace Co', 'Bayes', 'asd', 'Bcy', 'Baes', 'Bays'),
                price = c(10, 13, 2, 1, 15, 1))
b <- data.frame(name = c('Ace Co.', 'Bayes Inc.', 'asdf'),
                qty = c(9, 99, 10))

# Find matches
stringdist_join(a, b,
                by = "name",
                mode = "left",
                ignore_case = FALSE,
                method = "jw",
                max_dist = 99,
                distance_col = "dist"
) %>%
  group_by(name.x) %>%
  top_n(1, -dist)

This gives you a "distance" between one and another word. So if you know the real product name, you might be able to find the erroneous names via the distance.
# A tibble: 6 x 5
# Groups:   name.x [6]
  name.x price name.y       qty   dist
  <fct>  <dbl> <fct>      <dbl>  <dbl>
1 Ace Co    10 Ace Co.        9 0.0476
2 Bayes     13 Bayes Inc.    99 0.167
3 asd        2 asdf          10 0.0833
4 Bcy        1 Bayes Inc.    99 0.378
5 Baes      15 Bayes Inc.    99 0.2
6 Bays       1 Bayes Inc.    99 0.2

Alternatively, you could look into similarity in string sequences:
If your product names are not to common from their string sequences and the erroneous names get only part of the name wrong, you could try something like:
library(dplyr)
library(tidytext)
library(fuzzyjoin)
library(tokenizers)

##############################
# Compare text sequences

text1=as.character("Hi my name is Bixi and I like cycling a lot. It is just great!")
mytext1=data_frame(text1)

text2=as.character("Hi my name is Lissi and I'm good in swimming. It is just great!")
mytext2=data_frame(text2)

ngram1 = unnest_tokens(mytext1, ngram, text1, token = "ngrams", n = 4)
ngram2 = unnest_tokens(mytext2, ngram, text2, token = "ngrams", n = 4)

# Find matching sequence(s)
semi_join(ngram1,ngram2)

##############################
# Compare sequences of single letters

ngram3=tokenize_character_shingles(mytext1$text1, n = 10, n_min = 10, strip_non_alphanum = FALSE)
ngram4=tokenize_character_shingles(mytext2$text2, n = 10, n_min = 10, strip_non_alphanum = FALSE)

ngram3=as.data.frame(ngram3)
ngram4=as.data.frame(ngram4)

# Find matching sequences of single letters
semi_join(ngram3,ngram4)

You may have a look at my Github. There are some more related options/solutions.


Cross Validation (sklearn train test split) - ValueError: not enough values to unpack I have written this code and getting error:

from sklearn.model_selection import train_test_split
X0 = pd.read_csv('C:/Users/Desktop/Dataset for Experiment - 2019/ANFIS modeling data/china_data.csv')
train_test_split. X0 = X0.drop(columns = ['EFFORT'], axis = 1)
X_train, X_test, y_train, y_test = train_test_split(X0, test_size=0.8)

The error which is showing for this code

X_train, X_test, y_train, y_test = train_test_split(X0, test_size=0.8)

  ValueError: not enough values to unpack (expected 4, got 2)

I tried many times to remove this error but it is still persistent. Can anybody explain to me why I am getting this error & how can I remove this error?

Sklearn expects to get X and y in train_test_split().

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

You only parse X.

See the docs: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html


Softmax function forms The constraint that the n outputs must sum to $1$ means that only $n−1$
  parameters are necessary; the probability of the $n^{th}$ value may be
  obtained by subtracting the first $n−1$ probabilities from $1$. We can
  thus impose a requirement that one element of $z$ be fixed. For example,
  we can require that $z_n = 0$. Indeed, this is exactly what the sigmoid
  unit does.

What does it mean by "impose a requirement that one element of $z$ be fixed." I understand you can get the $n^{th}$ one if you know n-1 but "fixed"?

  Defining $P (y = 1 | x ) = \sigma(z )$ is equivalent to defining $P
(y = 1 | x) = softmax(z)_1$ with a two-dimensional $z$ and $z_1 = 0$. Both
  the $n − 1$ argument and the $n$ argument approaches to the softmax can
  describe the same set of probability distributions, but have different
  learning dynamics.

How is it equivalent to having $z_1=0$?

The book quotes in your question are the outline of a proof that shows single output sigmoid represents the same class probabilities as a two dimensional softmax.

  What does it mean by "impose a requirement that one element of $z$ be fixed." I understand you can get the $n^{th}$ one if you know n-1 but "fixed"?

It doesn't mean the value has to be fixed. It means that we can choose any $z_n$ value, and have the other's adjusted to represent any valid target probabilities. So for argument's sake, we can pick any fixed value.

  How is it equivalent to having $z_1=0$?

It is an equivalence between $\sigma(z)$ and $softmax(z)_1$ with $z_1 = 0$:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

For softmax, this is the function for the first element from a 2-dimensional output:

$softmax(z)_1 = \frac{e^{z_1}}{\sum_{i=1}^{2}e^{z_i}} = \frac{e^{z_1}}{e^{z_1} + e^{z_2}}$

But we have set $z_1 = 0$, and $e^0 = 1$ so:

$softmax(z)_1 = \frac{1}{1 + e^{z_2}}$

This is the same function as $\sigma(-z_2)$, so the two functions have identical form (if you plot them, they overlap), although the parameters drive the specific value differently.

Note that as per step 1, we have not lost anything about the general behaviour by setting $z_1 = 0$, we just did it to simplify the equation. In fact it is possible to make a very similar argument without fixing any $z$ values, and you end up with a more generic linear combination of $z_1$ and $z_2$ in the softmax function being equivalent to just $z$ in $\sigma(z)$.


What are some challenges with big data? How did the phrase "big data" come about? What were some of the new challenges that arose? How does big data differ from a small amount of data and how do we distinguish the two?

There has/is been a lot of hype around the word "Big Data". So, as someone who has been setting up an analytics infrastructure at a fast growing company, I would like to share my experiences.

You would like to check this out, which I have written for an almost similar question on CV.

As the tools like H2O, R, Julia, Spark, etc; are being evolving for dealing and playing with Big Data, the size of the data is becoming a less of a problem these days.

As there is a lots of data available, and a wonderful set of tools at disposal, the art of analytics and data science is becoming more of a pleasure for data scientists rather than pain.

But, the real challenge is the availability of people who know what they are doing, who know how to use these tools for mining gold from the data.
Just anyone who has run a Linear Regression on a data set can't call himself a data scientist. One has to explain what the data is, what can be pulled out of it, and why is the algorithm applied on it, and why not something else.

How do they differ from a small amount of data?

In my experience, there isn't any huge differences. The only differences which I have found are algorithms which help find the needle in the hay stack, like MinHash, HyperLogLog, etc; which had to evolve for dealing with the size.


Python: Fast indexing of strings in nested list without loop I have an array dataset from which I need to build D whose each element is a list of strings (Third column of dataset). The dataset looks like:
600,900,3418309
600,900,3418309
600,900,3418314
600,900,3418314
600,900,3418319
600,900,3418319
610,800,3418324
610,700,3418324
600,900,3418329
620,900,3418329
600,900,3418329
600,900,3418334
610,900,3418334
600,900,3418339
600,900,3418339
600,900,3418339
660,700,3418339
610,800,3418339
660,700,3418339
600,900,3418339
600,900,3418339

I want to check for every new string, and if it is not a part of any of the array element then append new string to only that array element.
Since the data on new strings to be added is large, I do not want to use loop. Is there any method to do it fast. I have to use Python.
Right now I am using the code, which is very slow:
for i in range(len(dataset)):
 for j in range(int(int(dataset[i,0]-600)),int(int(dataset[i,1]-600)+1)):

    if(str(dataset[i,2]) not in D[j]):

        D[j].append(str(dataset[i,2]))

This answers assumes I have correctly understood the question... I can alter my answer if OP updates the question with more details

Using your example data, you can use Pandas easily drop all duplicates.

Setup

First dump your data above into a Dataframe with three columns (one for each of the items in each row.

Import pandas:

import pandas as pd

import your data - assuming it is a list of lists - each of your rows is a list  of three items, so we have three columns:

df = pd.DataFrame.from_records(your_list_of_lists, columns=["col1", "col2", "col3"])

Have a look at the first 5 rows:

df.head()
   col1  col2     col3
0   600   900  3418309
1   600   900  3418309
2   600   900  3418314
3   600   900  3418314
4   600   900  3418319

The values will be integers by default, not strings (if they all were). But the solutions below should work the same in either case.

Solutions

If you want to get all unique values of col3, you can do one of the following:

uniques1 = set(df.col3)              # returns a Python set
uniques2 = df.col3.unique()          # returns a NumPy ndarray
uniques3 = df.col3.drop_duplicates() # returns a pandas Series object

If you want to remove only those rows where col3 repeats itself consecutively (which in some cases could produce the same result as the methods above, depending on the data), then you can look at the methods here. [An example] adapted for your columnar situation2:

def drop_consecutive_duplicates(a, col_name):      # returns the dataframe
    ...:         ar = a[col_name].values
    ...:         return a[np.concatenate(([True],ar[:-1] != ar[1:]))]

This will return the entire dataframe, which those rows of consecutive values removed.

Performance

There are many other ways to achieve the same result. Of the above, the first method is the fastest (on your small dataset example). Here are the benchmarks:

In [23]: %timeit df.col3.drop_duplicates()
263 µs ± 883 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [24]: %timeit df.col3.unique()
37.2 µs ± 3.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [25]: %timeit set(df.col3)
10.5 µs ± 45.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

The consective version:

In [26]: %timeit drop_consecutive_duplicates(df, "col3")
266 µs ± 3.61 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

I imagine the Pandas methods might be able to scale better to a DataFrame with many rows, so this example might be rather biased on the dummy dataset with only ~20 rows.

The final method clearly has a little bit of overhead as it has to perform some extra operations


Ways to convert textual data to numerical data I've been looking for ways to wrangle my data which contains both text and numerical attributes.

There are of course several algorithms for numerical data, but I am looking for suggestions regarding how to deal with textual data, for instance: for sorting based on K-means clustering and predicting missing data using kNN. I would really appreciate any thoughts regarding that. I am using scikit-learn.

A renown method is using two layers Neural Network which transform each word to a vector. you can use word2vec library also to do this. To know more about this you can find this online article useful.


What happens to the left over unpicked data in Random Forest I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not affect the random forest training? Do we use that left over later at some point.

I suppose it is possible that not all samples are selected during training, depending on the parameters you specify (or that are available in the implementation).

Looking at Scikit-Learn's RandomForestClassifier documentation, we can see that there is a bootstrap argument that can be set to False to ensure all data points are used to fit each of the trees. Otherwise, say you pick some arguments to all be 1 (num_estimators, max_depth, min_samples), then not much data would be used at all! Looking through the source code, there doens't seem to be a check that all data was used.

Another classifier, ExtraTrees (Extremely Randomised Trees) is generally designed to used all samples to train each estimator. However the SciKit learn implementation does allow you to disable that and use random bootstrapping, as is default with the other random forest algorithms.

So to answer your question; it seems the unused samples are simply left out!


How to train a model to predict if 2 samples refer to the same thing? I have 2 ddbb with around 60.000 samples each. Both have the same features (same column names) that represent particular things with text or categories (turned into numbers). Each sample in a ddbb is assumed to refer to a different particular thing. But there are some objects that are represented in both ddbb, yet with somewhat different values in the same-name column (like different open descriptions, or classified as another category).
The aim is to train a ML model that recognizes when the two “descriptions” refer to the same thing.
We have manually recognized a few thousand "duplicated" cases that we have labelled correspondingly. This label is the target variable to learn. Yet, the supervised classification examples I've seen, work with a single data frame and/or predict something about a single row of features.
How does it work when we are not trying to predict what a sample represents, but the relation between 2 samples (specifically now if they refer to the same object or not)?
I don't even know how to feed 2 dataframes to sk-learn or even auto-sklearn (or similars) so that it can handle the task of recognizing if a sample from one represents the same thing than another sample in the other one... Maybe I should concat them so that there's just 1 df, but then it would have to compare all with all the rests... Or doesn't make it a lot of difference?
Any idea or hint about how to proceed here or how to frame the problem better?

If I understand correctly, it sounds somewhat like an NLP problem. For each row, and each column, you will compute the similarity between entries across dataframes e.g.
Your two dataframes:
Row182, ColumnA, DataFrame1 = XYZ_ABC_123
Row182, ColumnA, DataFrame2 = XYZ_ABC_152

New data frame:
Row182, ColumnA = 0.8

Repeat for all rows and columns. You would then set some threshold at an acceptable similarity rate / accuracy of your similarity metric matrix. To highlight the most severe cross-overs.
You could have a look at https://www.nltk.org/, but also depending on the data entries you could just do a pure pythonic solution e.g. 80% match between string entries.
I'm guessing at what your data looks like, so some examples there could help


Simple 3 layers Neural Network cannot be trained #3 layers neural network
import numpy as np
from __future__ import division
def nonlin(x,deriv=False): #activation function
    if(deriv==True):
        return np.exp(x)/(1+np.exp(x))**2
    return 1/(1+np.exp(-x))

X = np.array([
                [0,0,1],
                [0,1,1],
                [1,1,1],
                [1,0,0],
                [0,1,0],
                [1,1,0]])
Y = np.array([[0,1,0,1,1,0]]).T
np.random.seed(1)
l0 = X
syn0 = 2*np.random.random((3,30))-1
syn1 = 2*np.random.random((30,1))-1

for i in xrange(60000):
    l0 = X
    l1 = nonlin(np.dot(l0,syn0))
    l2 = nonlin(np.dot(l1,syn1))
    l2_error = Y-l2
    l2_delta = l2_error*nonlin(l2,deriv=True)

    l1_error = l2_delta.dot(syn1.T)
    l1_delta = l1_error*nonlin(l1,deriv=True)
    syn0 += l0.T.dot(l1_delta)
    syn1 += l1.T.dot(l2_delta)
print l2

I have been messing with the neural network implementation at https://iamtrask.github.io/2015/07/12/basic-python-network/

This is the output of the code:

[[1.85572928e-04]
 [9.99755942e-01]
 [5.21248255e-09]
 [9.99767481e-01]
 [9.99963580e-01]
 [2.07334909e-04]]

I expect something like Y

[[0]
 [1]
 [0]
 [1]
 [1]
 [0]]

What could have possibly gone wrong here?

The results you are getting are the following:

[[1.85572928e-04]  =  0.000185572928         ~ 0
 [9.99755942e-01]  =  0.999755942            ~ 1
 [5.21248255e-09]  =  0.000000000521248255   ~ 0
 [9.99767481e-01]  =  0.999767481            ~ 1
 [9.99963580e-01]  =  0.999963580            ~ 1
 [2.07334909e-04]] =  0.000207334909         ~ 0

These are indeed very close to your expected results. You are computing and predicting floating point numbers, and not binary zeros and ones. You could for example add a simply rule that will accepts values below a threshold to be zero and above the threshold to be one.


How to choose between classification Vs regression approach? I need to predict the profitability of the products of a retailer. I can either predict the absolute value of the profit the products will make (continuous outcome) or predict whether the products will make a profit or not (categorical outcome). Is there any advantage of approaching this as a classification problem rather than a regression problem or vice versa? Why?

I would say that the main thing is what information you think is the most valuable. There are some differences.

The benefit with a regressor is that it can give you a number of how much profit you can expect. That is obviously useful when considering a new product. The drawback is that it cannot tell you how likely you are to see this profit. It will not tell you if a product is risky.

Many classifiers can tell you with what probability it thinks a product will be profitable. This can alert you to potentially risky products. But the drawback is that you won't get to know how big the potential profit is.

As I said, it depends what you value most. But nothing is stopping you from training two models and using information from both of them to make your decisions.


How many Layers in the original Yoon Kim CNN implementation? I saw some implementations of Yoon Kim's Convolutional Neural Network (Paper: http://www.aclweb.org/anthology/D14-1181)....

...in some implementations they put one more Dense(..) Layer before the output layer (with softmax activation). In my opinion there is no additional Dense(..) Layer in the original paper:

So what's the original implementation?

Based on the model architecture from original paper below, it looks like there is just one dense layer at the end. Conv layers --> Max Pool over time --> Dense


How are weights calculated in a feed-forward neural network before they are summed up with bias? I have read a lot of papers and watched different videos, it seems like they explain how they are summed up with bias before entering the activation function.

What I am trying to understand is the whole flow process of data from the matrix all the way through to the output layer. What I am struggling with is how do they gain their initial weights?

I have drawn a diagram hoping to explain what I understand so far. Please let me know if I am in the right track and help me understand how weights are calculated. Many thanks in advance.

Many thanks in advance.

Initial weights are simply initialized by the programmer, usually according to some random distribution (e.g. Gaussian).
There are several well known initializers, such as xavier,
that have been shown to improve the training process.

If you read ML papers, you will see that researchers always mention how they initialized weights for their models so that their results are reproducible.


What is the meaning of term Variance in Machine Learning Model? I am familiar with terms high bias and high variance and their effect on the model.

Basically your model has high variance when it is too complex and sensitive too even outliers.

But recently I was asked the meaning of term Variance in machine learning model in one of the interview?

I would like to know what exactly Variance means in ML Model and how does it get introduce in your model? I would really appreciate if someone could explain this with an example.

It is pretty much what you said. Formally you can say:

Variance, in the context of Machine Learning, is a type of error that occurs due to a model's sensitivity to small fluctuations in the training set.

High variance would cause an algorithm to model the noise in the training set. This is most commonly referred to as overfitting.
When discussing variance in Machine Learning, we also refer to bias.

Bias, in the context of Machine Learning, is a type of error that occurs due to erroneous assumptions in the learning algorithm.

High bias would cause an algorithm to miss relevant relations between the input features and the target outputs. This is sometimes referred to as underfitting.
These terms can be decomposed from the expected error of the trained model, given different samples drawn from a training distribution. See here for a brief mathematical explanation of where the terms come from, and how to formally measure variance in the model.
Relationship between bias and variance:
In most cases, attempting to minimize one of these two errors, would lead to increasing the other. Thus the two are usually seen as a trade-off.

Cause of high bias/variance in ML:
The most common factor that determines the bias/variance of a model is its capacity (think of this as how complex the model is).

Low capacity models (e.g. linear regression), might miss relevant relations between the features and targets, causing them to have high bias. This is evident in the left figure above.

On the other hand, high capacity models (e.g. high-degree polynomial regression, neural networks with many parameters) might model some of the noise, along with any relevant relations in the training set, causing them to have high variance, as seen in the right figure above.

How to reduce the variance in a model?
The easiest and most common way of reducing the variance in a ML model is by applying techniques that limit its effective capacity, i.e. regularization.
The most common forms of regularization are parameter norm penalties, which limit the parameter updates during the training phase; early stopping, which cuts the training short; pruning for tree-based algorithms; dropout for neural networks, etc.
Can a model have both low bias and low variance?
Yes. Likewise a model can have both high bias and high variance, as is illustrated in the figure below.

How can we achieve both low bias and low variance?
In practice the most methodology is:

Select an algorithm with a high enough capacity to sufficiently model the problem. In this stage we want to minimize the bias, so we aren't concerned about the variance yet.
Regularize the model above, to minimize its variance.


Queries regarding feature importance for categorical features Queries regarding feature importance for categorical features:

Context:
I have almost 185 categorical features and these categorical features have either 2 or 3 or 8 or 1 or sometimes 4 categories, null's also. I need to select top 60 features for my model.  I also understand that features needs to be selected based on business importance OR feature importance by random forest / decision tree.

Queries:

I have plotted histograms for each feature (value count vs category) to analyse. What is the approach to select whether feature is important?
What is the standard practice followed across data science industry to get the feature importance from categorical data?
Is there basic and elegant way to select the top important features?
How do I engineer these categorical features?

From what I understand, you have 185 features and out of them you want to choose the 80 most informative/ important?

It really seems like the main focus of all algorithms for decision trees, which is how to
decide which attribute is the best for a split? I suggest you do some reading in the context of decision trees. For example: DECISION TREES - Presentation

What you are looking for is an Impurity-Measure which can help determine the Goodness-of-Split due to some discrete attribute (basically measuring information gain):

There are 3 measures which are used most commonly - Gini index, miss-classification error and entropy.


Does the network learn based on previous training or does it restart? Matlab, neuralnetworks In Matlab, if you build a simple network and train it:

OP = feedforwardnet(5, 'traingdm');
inputsVals = [0,1,2,3,4];
targetVals = [3,2,5,1,9];
OP = train(OP,inputsVals,targetVals);

then you train it again so another OP = train(OP,inputsVals,targetVals);

What is happens to the network? Does it train again based on what it learned the first time you did OP = train(OP,inputsVals,targetVals); or does it train as if it were the first time training the network.

It trains again based on what it learned the first time you did OP = train(OP,inputsVals,targetVals). More generally, train uses your network's weights, i.e. it does not initialize the weights. The weight initialization happens in feedforwardnet.

Example:

% To generate reproducible results
% http://stackoverflow.com/a/7797635/395857
rng(1234,'twister')

% Prepare input and target vectors
[x,t] = simplefit_dataset;

% Create ANN
net = feedforwardnet(10);

% Loop to see where train() initializes the weights
for i = 1:10

    % Learn
    net.trainParam.epochs = 1;
    net = train(net,x,t);

    % Score
    y = net(x);
    perf = perform(net,y,t)
end

yields

perf =

    0.4825

perf =

    0.0093

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0028


Both train and test error are decreasing in XGBoost iterations I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 early stopping rounds. Then when I try to apply the model on a separate set not used for testing and training I can see that a model trained over 100 rounds performs much better than 1000 one. My learning rate is 0.01.

I am wondering whether this is a normal situation (and I should just do early stopping on this independent validation set) or whether I am doing something wrong. Theoretically, the test error should start increasing at some point, but this is just not happening.

I am using XGBoost and I am splitting my dataset in train/test like this:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

Them I train it in the following way:

watchlist =  [(dtest, 'eval'), (dtrain, 'train')]
progress = dict()

# Train and predict with early stopping
xg_reg = xgb.train(
    params=params,
    dtrain=dtrain, num_boost_round=boost_rounds,
    evals=watchlist,    # using validation on a test set for early stopping; ideally should be a separate validation set
    early_stopping_rounds=early_stopping,
    evals_result=progress)

Note that I have used gridsearch to find the optimal hyperparameters (params).

A plot which I get (yellow is my cutoff point identified pretty much via testing on the independent set):

This is a common situation and can be caused by a number of things, such as:

Too much variance in the model (e.g. did the hyperparameter tuning identify low values for eta, alpha, lambda, gamma, subsampling/column sampling/row sampling, minimum child weights and/or high values for depth and number of leaves?)
Having a small dataset with high variance.  XGBoost is prone to overfitting and this is exacerbated when there isn't much data for it.
Not shuffling the data so that the validation set has lower variance than (or differs in distribution from) the training and test sets.  train_test_split shuffles by default, so this shouldn't be a problem if you've used it as indicated in the question for generating the validation set also.

Stopping training early is a perfectly valid and common method of increasing bias in a model and doing so based on a validation set is acceptable.  It is preferable for the training to stop based on the test set.  To achieve this you could address the potential issues above.  Without knowing more about the data and how the validation set was generated, the most reliable advice I can give is to alter the hyperparameters to increase model bias.


SVM - Shuffle image data before GridSearchCV or not? I have different image datasets, most of them are sorted by class, others are already mixed.
For each of these data sets, I would like to train one SVM (in Python with Scikit-Learn), whereby in each case the hyperparameters are previously optimized using GridSearchCV.
Some of the ordered data sets still need to be split into training and test data, which would then be in mixed form after using train_test_split and, to my understanding, would not have to be mixed again before using GridSearchCV.

Other ordered data sets are already split into training and test data, so I would directly apply GridSearchCV. Now I have found that StratifiedKFold does not mix the data before splitting into batches, if I understood correctly. In those cases, I should mix the data before using GridSearchCV, right?

If I am mistaken in my assumptions, I would be grateful for any hint.

GridSearchCV will not shuffle the data by default. However, as you noted, you can pass Kfold or StratifiedKfold objects to cv argument, that specify shuffling.

For example:

model = XGBClassifier()
param_grid = {"n_estimators": [100]}
cv = StratifiedKFold(n_splits=5, shuffle=True)
gs = GridSearchCV(model, param_grid=param_grid)


How to deal with imbalanced text data I am working on a problem where I have to classify products into multiple classes (more than one) based on product descriptions. For instance:
"Tresemme shampoo and conditioner - sulfate-free" = Personal Hygiene
"Lavender-scented handwash with moisturizer" = Personal Hygiene
"Doritos Ranch flavor 18 oz mega party pack" = Snacks
"Painting and Craft kit for adults above 18" = Art and Craft
However, my training dataset is highly imbalanced. A few classes have only 10 records while there is one that has 3,000 records. 50,000 records overall.
Can anyone suggest any good techniques to deal with the imbalance in text data?
Thanks,
GD

I too am working on same problem, found these below links very useful in getting started on oversampling and under sampling-

https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/

https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis


Datasource for regression model prediction : Machine Learning I am trying to work using Amazon machine learning, but the data set that I have is small. The model I want to build is for regression based predictions and the domain I am aiming for the data set to belong is financial, say product price prediction, price and demand prediction based on macro/micro economic factor.

I am looking for a data set that contains factors that lead to variations in value of a product or commodity. For example, I would like to predict the value of 1 unit of polyester yarn after 1 year.  The factors which influence the yarn price are say - prices of crude oil, GDP of country,figures of IIP, inflation etc. So I would like a data set that contains the quotes of these factors on which the final price depends.

I find difficulty assembling this data myself because I don't know all the factors that contribute to a certain predictive price. Does anyone know of a dataset I can start with that sounds like it might contain these factors?

There are multiple datasets available in the internet but you need to pay for most of them.
One of the best datasets for starting (and it is free) is the Quantquote Free dataset.

You can download it from here.

This is the description (borrowed from their website):

  This collection of daily resolution data goes back to 1998 for all symbols currently active in the S&P500. It is updated quarterly, the last update was 07/31/2013.


ROC curve interpretation I trained a CNN model and a combined CNN-SVM model for classification. I wanted to compare their performance using ROC curve but I was confused which model is better. How to interpret the given ROC curves ?

If you hear from the area under the curve (AUC), you can find that the first classifier is better as AUC of the first curve is more than the second classifier. To know more about AUC you can find this post useful.


Grad Checking, verify by average? I am running Gradient Checking, to spot any discrepancy between my mathematically-computed gradient and the actual sampled gradient - to reassure my backprop was implemented correctly.

When computing such a discrepancy, can I sum-up squares of differences, then take their average? I could then use this average as my estimate of how correctly the network computes the gradient:

$$\frac{1}{m}\sum_{i=0}^{i=m}(g_i-n_i)^2$$

or even:

$$\sqrt{\sum_{i=0}^{i=m}(g_i-n_i)^2}$$

where $g$ is a gradient from backpropagation, and $n$ is gradient from gradient checking.

However, Andrew Ng instead recommends:

$$\frac{\vert \vert (g-n) \vert \vert _2 }{
\vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$

where $\vert \vert . \vert \vert _2$ is the length of the vector.

Another post post also recommends an slightly different approach: https://stats.stackexchange.com/a/188724/187816

Why would their approaches be better than mine?

Let me give you an example where Andrew's recommendation works better than yours:

Let's say that the real gradient is $(0, 0, 0)$ and the gradient you have computed is $(10^{-4}, 10^{-4}, 10^{-4})$. Then your average would return $10^{-8}$, and Andrew's recommendation would return $1$. Your metric could fool you into thinking that your gradient is computed propperly and the error is just due to a numeric issue, while Andrew's cannot fool you into that, due to the fact that it considers the fact that the gradient can be very small.

To wrap up, if your gradient doesn't have norm close to zero, it wouldn't really matter. However, when the gradient is close to zero you can be fooled into thinking that your gradient is right when it is not.


To answer your last question - think of the model as your brain trying to give a maths test. Training data is what you encountered during homework/exercise and validating/testing data is what you encounter in the final examinations(most likely unseen data). To think of yourself as proficient in mathematics, you'd want your brain to be able to perform best on these unseen data.
Following the above reasoning, you'd want the snapshot of the best performing model in terms of epoch.
Why validation you ask? The answer can be found in the existence of separate validation and test set. The validation set is used as part of the training for model selection. Whereas the test set is used at the end for the final analysis of the model before being deployed for real-world data. Think of it as taking mock tests before taking the actual examination. You can combine the two and label them as validation but after that, you will have no reference for their performance as real-world data.
Here's an excellent answer that talks more about the difference between validation and test data.


How to correctly perform data sampling for train/test split in multi-label dataset? Problem statement

I have a text multi-label classification dataset, and I've found a problem with the dataset sampling.

I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.

The preprocessing step simply consists in transforming the labels into OneHot representation and keep only the N most frequent ones. I expect similar (the same) behaviour, but I'm getting really weird results. Let's take a closer look.

Train+Test all together and then split

Start with:

|ID   |TEXT  |LABELS|
|-----|------|------|
|1.txt|the   |A:B   |
|2.txt|lazy  |B     |
|3.txt|fox   |C     |
|4.txt|jumps |B:C   |
|5.txt|over  |C:D   |
|6.txt|crazy |D     |

After preprocessing and split:
Train

|ID   |TEXT|A|B|C|D|
|-----|----|-|-|-|-|
|1.txt|the |1|1|0|0|
|2.txt|lazy|0|1|0|0|
|3.txt|fox |0|0|1|0|

Test

|ID   |TEXT |A|B|C|D|
|-----|-----|-|-|-|-|
|4.txt|jumps|0|1|1|0|
|5.txt|over |0|0|1|1|
|6.txt|crazy|0|0|0|1|

The results are good. Let's take this as reference. F1-Score = 0.61.

Pre-made Train/Test split

Start with:

Train

|ID   |TEXT|LABELS|
|-----|----|------|
|1.txt|the |A:B   |
|2.txt|lazy|B     |
|3.txt|fox |C     |

Test

|ID   |TEXT |LABELS|
|-----|-----|------|
|4.txt|jumps|B:C   |
|5.txt|over |C:D   |
|6.txt|crazy|D     |

After preprocessing and split:

Train

|ID   |TEXT|A|B|C|
|-----|----|-|-|-|
|1.txt|the |1|1|0|
|2.txt|lazy|0|1|0|
|3.txt|fox |0|0|1|

Test

|ID   |TEXT |B|C|D|
|-----|-----|-|-|-|
|4.txt|jumps|1|1|0|
|5.txt|over |0|1|1|
|6.txt|crazy|0|0|1|

The results are totally degradated. F1-Score = 0.15.

What is going on? What could be causing the divergence in results?

Extra information

The labels predicted at the prediction step are not compatible with the labels on the test set. I've taken that into account and is correctly managed, that's not the problem.
The splits are exactly the same. The documents in train/test are the same in both situations.

I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.

Strategy 2 is "correct" from a "don't reuse your data" perspective. Data pre-processing is part of training. Your train/test pipeline should, in principle, account for this. For example, if you have a feature $A$ with 5 categories, but only 4 of those categories appear in the data, the correct thing to do is to only use 4 categories in your one-hot encoding, and treat the 5th category as an "unknown" value.

In practice, however, this isn't always practical. In some cases the pre-processing pipeline is compute-intensive (e.g. training word vectors) or requires processing a massive amount of data (e.g. ingesting 5 GB of images off of AWS). In those cases, you have to accept the fact that some knowledge of your test set will "leak" into your training set by way of your preprocessing steps, but it's usually not so bad.

Other times you don't have enough data after train/test splitting to conduct preprocessing correctly. Or you didn't stratify your splits, and so you ended up with incomplete class representation, or and otherwise imbalanced training set. This is where you find yourself. Just look at the headings to see why:

Train:

|ID   |TEXT |A|B|C|

Test:

|ID   |TEXT |B|C|D|

Class "A" is missing from your test set, and class "D" is missing from your training set. This will ruin just about any model.

There has been some research done on the stratification of multi-label data. Notably, there is "On the Stratification of Multi-label Data" (Sechidis, Tsoumakas, and Vlahavas; ECML PKDD 2011), which introduces "iterative stratification". There is an implementation of this technique in the scikit-multilearn Python library.


Difference between cell state and hidden state LSTM cells consist of two types of states, the cell state and hidden state.
How do cell and hidden states differ, in terms of their functionality? What information do they carry?

In short:

Cell state: Long term memory of the model, only part of LSTM models
Hidden state: Working memory, part of LSTM and RNN models

Additional Information
RNN and vanishing/exploding gradients
Traditional Recurrent Neural Networks (RNN) have the ability to model sequential events by propagating through time, i.e. forward and backward propagation. This is achieved by "connecting" these sequential events with the hidden state:
$a_n = f(W_n, a_{n-1}, x_n)$
The hidden state $a_n$ carries past information by applying a linear combination over the previous step and the current input.
Despite being a very successful architecture, RNN have the issue of vanishing/exploding gradients. This means that every previous step is essentially considered in the calculation of the backpropagation (how wrong my prediction has been), due to the chain rule engraved in $a_n$:
$a_n = f(W_n, a_{n-1}, x_n) = f(W_n, f(W_{n-1}, a_{n-2}, x_{n-1}), x_n)$, since $ a_{n-1}=f(W_n, a_{n-2}, x_n)$.
To summarise: RNNs are great, but issues occur with log terms dependencies because of the chain rule in their hidden state.
LSTM and the cell state
To alleviate the issues above, LSTM architectures introduce the cell state, additional to the existing hidden state of RNNs. Cell states give the model longer memory of past events. This long term memory capability is enabled by

the storage of useful beliefs from new inputs
the loading of beliefs into the working memory (i.e. cell state) that are immediately useful.

In case you wonder "how does it know what to store or what's immediately useful?": remember that this a trainable weight that learns with training, consider it as an additional piece of muscle that will learn this new activity storing and loading by training it on examples (i.e. labelled datapoints).
To summarise: LSTMs are usually better at dealing with long term dependencies, because of their capacity to store and load beliefs that are important at different parts of the sequence.

TLDR:
hidden state:

Working memory capability that carries information from immediately previous events and overwrites at every step uncontrollably
-present at RNNs and LSTMs

cell state:

long term memory capability that stores and loads information of not necessarily immediately previous events
present in LSTMs

GRUs are also very relevant but are excluded from the response.


Learn and sample from a generative model with 2000 boolean features My data is a set of about 1 million training examples, where each example is represented by about 2000 boolean features. Examples aren't labelled - it wouldn't really have any meaning in this domain. The features fully define the example. My goal is to learn a generative model that I can use to generate more examples. I would like to capture some of the dependencies between the features (e.g. if feature 10 is on, feature 14, 15, and 16 are usually on).

I don't have much background in ML, so I'm not sure what a simple way to do this would be (both algorithms and software packages). My initial thought was to find a structure learning package to learn a Bayesian network and then sample from it, but I figured I'd ask here to see if anyone had any recommendations.

Thanks

You may try an Apriori algorithm or more advanced FPGrowth algorithm. It will give you a set of association rules, which, I think, you required:

  e.g. if feature 10 is on, feature 14, 15, and 16 are usually on


Extrapolating GLM coefficients for year a product was sold into future years? I've fit a GLM (Poisson) to a data set where one of the variables is categorical for the year a customer bought a product from my company, ranging from 1999 to 2012. There's a linear trend of the coefficients for the values of the variable as the year of sale increases.

Is there any problem with trying to improve predictions for 2013 and maybe 2014 by extrapolating to get the coefficients for those years?

If you suspect your response is linear with year, then put year in as a numeric term in your model rather than a categorical.

Extrapolation is then perfectly valid based on the usual assumptions of the GLM family. Make sure you correctly get the errors on your extrapolated estimates.

Just extrapolating the parameters from a categorical variable is wrong for a number of reasons. The first one I can think of is that there may be more observations in some years than others, so any linear extrapolation needs to weight those year's estimates more. Just eyeballing a line - or even fitting a line to the coefficients - won't do this.


Music corpus sentence level clustering I provide an offline library of Music to my users.
My goal is to understand what my users are looking for, which means translate raw user searches to: Music Artists, Songs, Albums and then add music to the company library.

What are the suggested clustering algorithms to group common short sentences into a single entity. Example:

Taylor Swift Shake it Off
Taylor Swift Shaek it Off
Shake it off
Twylor Swift Shake it Off

I tried this example and works fine for a specific number of clusters (K) where K < N | K <= N. But since searches are unpredictable need to find a way to automate the number of clusters: my goal is to cluster 2 or more similar items and let alone single searches in independent clusters, example:

Cluster 1:

Taylor Swift Shake it Off
Taylor Swift Shaek it Off
Shake it off
Twylor Swift Shake it Off

Cluster 2

Avicci
Avicii Ibiza
Avicci Electro House

Cluster 3

Juan Gabriel

Cluster 4

Adele

Cluster 5

Britney Spears
Britney Spears VMA 2016

I would suggest hierarchical clustering. It's unsupervised, and you don't need to predefine number of clusters. How it works (for the bottom up version) is each sentence (or object) is initialized as its own cluster. At each iteration of the algorithm, the two clusters of the smallest intra-cluster distance are joined, all the way until there's a 'root' where everything is one cluster. The result of this is a big dendrogram, and you can cut the dendrogram at whatever point you want to define clusters. Or you can just inspect it. Here's a nice numerical example of the alg in action.

It should detect the same types of clusters you're already finding, and you wouldn't need to redefine a distance metric. And, I anticipate you'll get the same amount of class separation you seem to already be getting; in fact, the nice thing about the dendrogram from hierarchical clustering is that it illustrates class separation nicely. it's 'hclust' in R, not sure for Python.


Generating artificial time series data can anyone please offer suggestions on ways to programmatically generate time series data artificially. if possible, mimic the distribution of an existing dataset (say hourly humidity readings) and add some noise if required. Any suggestions will be greatly appreciated!

This article is great to generate time series data in python.
Hope this helps.

https://towardsdatascience.com/basic-time-series-manipulation-with-pandas-4432afee64ea


Advice on dealing with very large datasets - HDF5, Python I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets as it offers the versatility to allow many dimensions, has no file size limits and is transferable between OSs.

My question is how to best deal with very large files. I am working with datasets that have 3-dimensions, all of which have large number of components (example size: 62,500 x 500,000 x 3,500, 32-bit int) which is a little problematic. HDF5 is theoretically able to deal with this, however, I've discovered it is really slow to write the data into the file and subsequently slow to retrieve it. Because the number of items is very large, data must be compressed otherwise it becomes insanely large on the HDD, however compression requires chunking and chunking is quite tricky to get right... I receive my data in (1 x 500,000 x 3500) chunks so it makes sense to write it one row at a time so with those dimensions and I ensure that chunks have dimensions (1 x n_values x 3500) to speed-up writing and data retrieval and have relatively small size (< 1Mb) for quick compression/decompression.

Does anyone have any experience in how to efficiently write to HDF5 files when the size of each dimension is v. large? Are there any tricks that could help?

I should also mention that I mostly work in Python, however, would be happy to do it in another language. I've tried increasing the chunk cache via the h5py_cache library.

One option is to switch to pytables which is built on top of HDF5 and is designed to mitigate many of these issues.


How can I implement my own AR(p) simulation algorithm? Recall that an AR(p) process can be defined as
$$
X_t = \phi_1 X_{t-1} + \cdots + \phi_p X_{t-p} + Z_t
$$
where $Z_t$ is I.I.D. white noise. I want to write an algorithm simulating these processes, but I'm not sure if I'm taking the right approach. Here's my idea:

INPUT:
  n = length of time series
  [\phi_1,...,\phi_p] the parameters of the process

1. Construct a white noise vector Z of length n + p
2. Have the first rolling sum of the X_i's be set as the noise
3. Construct the rest of the time series from there

In this case
$$
\begin{align}
X_0 &= \phi_1Z_{-1} + \cdots + \phi_pZ_{-p} + Z_0 \\
X_1 &= \phi_1X_0 + \phi_2Z_{-1} + \cdots + \phi_pZ_{-(p-1)} \\
&\cdots
\end{align}
$$

Another option would be to start by constructing $X_1,\ldots, X_p$ by having the lower order terms in the recursion equation be set to 0. So, for the first few
$$
\begin{align}
X_1 &= Z_1 \\
X_2 &= \phi_1 X_1 + Z_2 \\
X_3 &= \phi_1 X_2 + \phi_2 X_1 + Z_3 \\
&\cdots
\end{align}
$$
Which approach is the "right" approach for constructing the simulation?

You question boils down to:

should you try to do the simulation in a vectorised manner?
should you iterate over a vector?

In general we should always try to vectorise (option 1) as it's much more efficient, and it is easy to lag vectors in R (embed) and Python Pandas (shift).

However, I can't see any efficient vectorisation-based solution. The autoregressive nature of the simulated values force us to iterate over a vector to simulate each of $X_1, X_2, \ldots, X_n$ individually (option 2).


ValueError: Error when checking input: expected embedding_1_input to have shape (4,) but got array with shape (1,) numpy related I am trying to predict the pretrained model of keras Machine translation

but i get the error and i could not solve or find suitable answer

and it seems everyone gets this error

code

model = load_model('model.h5')
single_x_test = ['how are you']
model.predict(np.array(single_x_test, ndmin=0))

  ValueError: Error when checking input: expected embedding_1_input to
  have shape (4,) but got array with shape (1,)

The model excepts the input to be a 1D array with 4 items. The current input single_x_test is a Python list with 1 item. The model implicitly converts the Python list to a 1D array but it still only has 1 item.

The model needs to be introspected to see what 4 items it expects as inputs. Typically items are separate channels of information.


Why there is very large difference between cross validation scores? I have a very simple regression model and I am doing the cross validation. When cv=10 the highest score i got is 60.3 and lowest is -9.7 which is useless.
Average will be 30.

No of row data set= 658

Your $R^2$ scores indicate that a linear model does not describe your data well. On top of this, there seems to be a large variability in data. You could try the following:

If the linear model is supposed to describe the data, check for outliers. They might be responsible for the large variation across the CV folds.
Try reducing the number of features if there are many. The model might be fitting noise.
Introducing regularization (lasso or ridge regression) might make the model more robust. This should decrease the variability of the CV errors, but the $R^2$ scores will get even worse.


Sum up counts in a data.frame grouped by multiple variables This is a snippet of the dataset I am currently working on:

> sample
    name sex count
1  Maria   f    97
2 Thomas   m    12
3  Maria   m     5
4  Maria   f    97
5 Thomas   m     8
6  Maria   m     4

I want to sum up the counts grouped by name and sex to finally get this data.frame:

> result
    Maria Thomas
f   194      0
m     9     20

I wrote a simple loop to iterate over the rows and sum up the counts:

result <- matrix(0, nrow=2, ncol=2)
colnames(result) <- unique(sample$name)
rownames(result) <- unique(sample$sex)

for (i in 1:nrow(sample)) {
    sex <- as.character(sample[i,"sex"])
    name <- sample[i,"name"]
    count <- sample[i,"count"]

    result[sex, name] <- result[sex, name] + count
}

Is it suitable to do it this way? Are there any other ways to do it in a more elegant / shorter fashion?

Edit:

I already tried it with aggregate, but the output is in a different format:

> aggregate(sample$count,by=list(sample$name,sample$sex),sum)
  Group.1 Group.2   x
1   Maria       m   9
2  Thomas       m  20
3   Maria       w 194

You can do this using the xtabs function! Here's how I did it using your example data:

# Create example data...
name <- c("Maria", "Thomas", "Maria", "Maria", "Thomas", "Maria")
sex <- c("f", "m", "m", "f", "m", "m")
count <- c(97, 12, 5, 97, 8, 4)
data <- data.frame("name"=name, "sex"=sex, "count"=count)

# Create table...
xtabs(formula=count~name + sex, data=data)

which gives the following output:

        sex
name       f   m
Maria    194   9
Thomas     0  20


Why is an activation function notated as "g"? In many cases an activation function is notated as g (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as sigmoid.

However, where does this convention come from? And for what reason did g start to be used?

The addition of the activation layer creates a composition of two functions.

"A general function, to be defined for a particular context, is usually denoted by a single letter, most often the lower-case letters f, g, h."

So it comes down to the reason that he uses the hypothesis representation h(x)=wX+b which is a function, and that is wrapped by an activation function denoted as g. The choice of g seems to be purely alphabetical.


What is momentum in neural network? While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per documentation, which is not clear, it says

For The momentum, type a value to apply during learning as a weight on
nodes from previous iterations.

Although that is not very clear. Can someone please explain?

Momentum in neural networks is a variant of the stochastic gradient descent. It replaces the gradient with a momentum which is an aggregate of gradients as very well explained here.
It is also the common name given to the momentum factor, as in your case.
Maths
The momentum factor is a coefficient that is applied to an extra term in the weights update:

Note: image from visual studio magazine post
Advantages
Beside others, momentum is known to speed up learning and to help not getting stuck in local minima.
Intuition behind
As it is really nicely explained in this quora post, the momentum comes from physics:

Momentum is a physical property that enables a particular object with
mass to continue in it's trajectory even when an external opposing
force is applied, this means overshoot. For example, one speeds up a
car and then suddenly hits the brakes, the car will skid and stop
after a short distance overshooting the mark on the ground.
The same
concept applies to neural networks, during training the update
direction tends to resist change when momentum is added to the update
scheme. When the neural net approaches a shallow local minimum it's
like applying brakes but not sufficient to instantly affect the update
direction and magnitude. Hence the neural nets trained this way will
overshoot past smaller local minima points and only stop in a deeper
global minimum.
Thus momentum in neural nets helps them get out of
local minima points so that a more important global minimum is found.
Too much of momentum may create issues as well as systems that are not
stable may create oscillations that grow in magnitude, in such cases
one needs to add decay terms and so on. It's just physics applied to
neural net training or numerical optimizations.

In video
This video shows a backpropagation for different momentum values.
Other interesting posts
How does the momentum term for backpropagation algorithm work?
Hope it helps.


Training on accurate data versus noisy data I have data currently available that is very accurate and I would like to train my classification methods on this set of clean data to learn the important markers for distinguishing between classes. But in the future, my trained classifiers will not be seeing and performing decisions on this cleaned data; instead, it will likely have a lot more noise following some unknown distribution(s). Thus I am wondering, is it 'better' to train on noisy data if I'm going to likely see noisy data in the future, or train on good data since the noisy data should (ideally) correspond to the cleaned data if noise was removed?

Intuitively, if my goal is to simply perform classifications, then training on noisy data seems the 'better' approach since this is better representative of my expected future inputs. But if my goal is to learn about the data and what constitutes a particular decision, then training on cleaned data appears the 'better' approach.

But am I overlooking anything? Would training on the clean and/or noisy data be preferable for different reasons (e.g. generalization, overfitting, reducing dimensionality)?

The answer to this severely depends on what you mean by 'noisy' data. Are the labels noisy i.e. wrong? Or are the features noisy? Or both? If only the features are noisy, definitely use the noisy data and probably also the clean data. If only the labels, definitely do not use the noisy data. If both, is it possible to correct the labels? At the very least, can you get a reliable test set (representative with correct labels)? You could try training on both the noisy and clean data and see which gives better performance. Important methods to consider here would be regularization and early stopping.

It also depends on what algorithm you are using. Linear regression, for example, is unlikely to over-fit, while neural networks are extremely sensitive to noise, and there are a variety of approaches in between those two extremes.


What is the difference between using numpy array images and using images files in deep learning? What is the difference between using numpy array images and using images files in deep learning?

Which way is better?

In order to pass an image as an input to a model first need to convert it to a numpy array. Each image actually is represented as an array of values when you load it into python. Even if you don't do it explicitly (i.e. through keras' ImageDataGenerator), it is done behind the scenes.

If your question is: Is it better to use generators than loading the images in a large numpy array?

The answer is: it depends. Is the dataset small enough to fit in your memory?

If not, you are forced to use a generator that loads the images in batches and passes each batch to the model.

If yes, you either can use a generator to save memory for other things (e.g. the model) or you can load the images into a numpy array so that you can save on computation time (i.e. the overhead of loading images again and again).


Extracting specific data from unstructured text - NER I have a reasonably simple problem to solve. I need to extract reservations numbers from unstructured text. Based on my research, it seems to be an NER problem. Based on a visual analysis of the dataset, I could notice that the reservation number is frequently near specific keywords, such as 'confirmation', 'reservation', 'confirmation number', 'reservations number', etc.

First, I decided to try a Regex rule to extract the data, but some minimum variations might render this solution inefficient. The reservation number can have very different variations, such as:

ZXC51657856,
EA5FFD4,
45615177413515,
QT454545EF,

At this moment, I don't have a dataset available to train a classifier to solve this issue.

I would like to receive some ideas from the community to guide me towards an elegant solution to this problem, as I'm pretty new to ML in general and time is limited.

From your question, I too feel it's a NER problem. And about the dataset, unless there is a data set which tags the reservation numbers and is similar to your application, you WILL have to create your own data set.

I worked on a similar problem before and my dataset looked something like this:

<TEAM>Northern</TEAM>   NNP
<TEAM>Ireland</TEAM>    NNP
man NN
<PLAYER>James</PLAYER>  NNP
<PLAYER>McIlroy</PLAYER>    NNP
is  VBZ
confident   JJ
he  PRP
can MD
win VB
his PRP$
first   JJ
major   JJ
title   NN
at  IN
this    DT
weekend NN
's  POS
<COMPETITION>Spar</COMPETITION> JJ
<COMPETITION>European</COMPETITION> JJ
<COMPETITION>Indoor</COMPETITION>   NNP
<COMPETITION>Championships</COMPETITION>    NNP
in  IN
<LOCATION>Madrid</LOCATION> NNP

You can see that that I have the entity tag and the part of speech tag in the word. When I parse this dataset for training, I also add the IOB tags (Inside, Outside, and Beginning)

[(('Claxton', 'NNP\n'), 'B-PLAYER'),
 (('hunting', 'VBG\n'), 'O'),
 (('first', 'RB\n'), 'O'),
 (('major', 'JJ\n'), 'O'),
 (('medal', 'NNS\n'), 'O'),
 (('.', '.\n'), 'O'),
 (('British', 'JJ\n'), 'O'),
 (('hurdler', 'NN\n'), 'O'),
 (('Sarah', 'NNP\n'), 'B-PLAYER'),
 (('Claxton', 'NNP\n'), 'I-PLAYER')......]

Then I just used the ClassifierBasedTagger(There are other taggers too). I can't find the source but I used this code:

class NamedEntityChunker(ChunkParserI):
def __init__(self, train_sents, **kwargs):
    assert isinstance(train_sents, Iterable), 'The training set should be an Iterable'

    self.feature_detector = features
    self.tagger = ClassifierBasedTagger(
    train = train_sents,
    feature_detector = features,
    **kwargs)

def parse(self, tagged_sents):
    chunks = self.tagger.tag(tagged_sents)

    iob_triplets = [(w, t, c) for ((w, t), c) in chunks]

    return conlltags2tree(iob_triplets)

Here features is a function which returns a dictionary of the features to be used such as the previous word, previous word's pos tag etc. Just features to train the model on.

{
    'word' : word,
    'lemma' : stemmer.stem(word),
    'pos' : pos,
    'allascii' : allascii,

    'next-word' : nextword,
    'next-lemma' : stemmer.stem(nextword),
    'next-pos' : nextpos,

    'prev-word' : prevword,
    'prev-lemma': stemmer.stem(prevword),
    'prev-pos' : prevpos
}

You can find useful theory here

I hope this helps.


Brain Tumor Database for CNN? I'm trying to build a Convolutional Neural Network model to classify and predict a brain tumor based on images. I am looking for a database containing images of brain tumor.

Can someone introduce some medical databases (small one to test or big ones) with links to download?

Some of the followings might be useful:

Brain Tumor Image Segmentation Challenge
Primary Tumor Data Set by UC Irvine Machine Learning Repository
ncbi.nlm.nih.gov
University of Cyprus
Brain Tumor Dataset


how to prevent machine crash while searching for hyper parameters of XGBoost with GridSearchCV I am searching for best hyper parameters of XGBRegressor using GridSearchCV. Here is the code:
from xgboost import XGBRegressor
from sklearn.model_selection import GridSearchCV, KFold

param_grid_xgb_b1 = {
    'num_leaves': np.arange(20, 500, 5),
    'max_depth': np.arange(5, 50, 2),
    'reg_alpha': np.arange(0.01, 0.5, 0.01),
    'min_data_in_leaf': np.arange(50, 1_000, 10),
    'colsample_bytree': [0.65, 0.75, 0.85, 0.95, 1],
    'subsample': [0.65, 0.75, 0.85, 0.95, 1]
    }
kfold_xgb_b1 = KFold(n_splits=5, shuffle=True, random_state=42).split(X=X_train, y=y_train)
grid_search_xgb_b1 = GridSearchCV(estimator=XGBRegressor(learning_rate=0.01, n_estimators=20_000, random_state=0,
                                                         tree_method='gpu_hist', subsample_freq=5),
                                  param_grid=param_grid_xgb_b1,
                                  scoring=rmse_score, cv=kfold_xgb_b1)
grid_search_xgb_b1.fit(X=X_train, y=y_train, verbose=3)

I am using Google Colab with GPU. Machine is dual core CPU with 12.72GB of RAM. And GPU is Tesla T4 with CUDA Version 10.1.
How do I prevent crash and search for hyper parameters?
EDIT 1:
Crash type: RAM filling up;
size of data: 300,000 rows and 40 columns.

I assume its crashing because not enough RAM. So I also assume your data is quite big. Your search grid is quite big. So this will definitely take some time.
In order to speed up the training and overloading the RAM. You can fit the model in a subsample of data. Theoretically if your data is big enough and you sample it, when you use the whole model the optimal hyperparameters should be quite similar.
Once you do, see what is the actual improvement of changing hyperparameters, hyperparameters in xgboost shouldn't be sooooo important. They will make a change but not a huge change.
So my suggestion will be to sample your data by 1% or something similar to search for hyperparameters and then fitting the model in the whole training set.
And not search in such a huge hyperparameter space.


Isolation Forest Score Function Theory I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point traversing an iTree, and $n$ is the sample size used to grow the iTree.

  The difficulty in deriving such a score from $h(x)$
  is that while the maximum possible height of iTree grows
  in the order of $n$, the average height grows in the order of
  $log(n)$. Normalization of $h(x)$ by any of the above terms
  is either not bounded or cannot be directly compared.

So herein lies my first question. What do they mean that the normalization of $h(x)$ by any of the above terms is either not bounded or cannot be directly compared? The final score function in this paper is given by

$$s(x,n)=2^{-\frac{E(h(x))}{c(n)}}$$

where

$$c(n)=2H_{n-1}-2(n-1)/n$$

is the average path length of an unsuccessful search from BST Theory. Note how they are taking the expectation of the path. So in that case we are averaging all the values anyway, so why can we not use the growth of the average height of the tree? Additionally they don't even mention the analytical form of average height here, which from what I gather is $2\sqrt{\pi n}$ reference, though I have not read this reference thoroughly and may be mistaken.

Am I missing something here?

I'm not certain on your first question; the paper does seem poorly written in this part.  They seem to be arguing that taking just $E(h(x))/n$ or $E(h(x))/\log n$ are inappropriate.  The former tends to zero regardless of $x$, but the latter should be reasonable.  Indeed, since $c(n)\sim 2\log n$, it's not far off from what they end up doing.  (Exponentiation pushes the scores into the range $[0,1]$ which is nice, and using the more exact $c(n)$ calibrates things better for "(c) if all the instances return $s\approx 0.5$, then the entire sample does not really have any distinct anomaly.")

The rest of the confusion seems to be stemming from what set we're averaging over.  In the isolation forest paper, the averaging is of the depths of (the leaf containing) $x$ over multiple trees in the isolation forest.  In the other reference, the average is of the height of all binary trees on $n$ nodes.


MinMaxScaler returned values greater than one Basically I was looking for a normalization function part of sklearn, which is useful later for logistic regression.

Since I have negative values, I chose MinMaxScaler with: feature_range=(0, 1) as a parameter.

x = MinMaxScaler(feature_range=(0, 1)).fit_transform(x)

Then using sm.Logit trainer I got and error,

import statsmodels.api as sm
logit_model=sm.Logit(train_data_numeric_final,target)
result=logit_model.fit()
print(result.summary())

ValueError: endog must be in the unit interval.

I presume my values are out of (0,1) range, which is the case:

np.unique(np.less_equal(train_data_numeric_final.values, 1))

array([False,  True])

How come? then how can I proceed.

I am not sure why your MinMaxScaler didn't work, but here is a function that should scale your data into the desired range:

def rescale(data, new_min=0, new_max=1):
    """Rescale the data to be within the range [new_min, new_max]"""
    return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min

Looking at the documentation of the MinMaxScaler, it seems my function above it the same as their method.

You could break your code down a little to explicitly comppute each step on its own line. This might help find the origins of your problem. I tried it out and got the expected results:

In [1]: import numpy as np

In [2]: from sklearn.preprocessing import MinMaxScaler

In [3]: x = np.random.randint(0, 10, (10, 10)).astype(np.float)

In [4]: x                                    # generate random data in range [0, 9]
Out[4]:
array([[ 1.,  4.,  5.,  4.,  6.,  1.,  8.,  1.,  8.,  9.],
       [ 3.,  1.,  4.,  4.,  6.,  2.,  5.,  1.,  0.,  8.],
       [ 2.,  0.,  6.,  1.,  5.,  2.,  5.,  8.,  8.,  4.],
       [ 8.,  9.,  2.,  8.,  5.,  6.,  0.,  5.,  0.,  5.],
       [ 1.,  3.,  2.,  2.,  3.,  2.,  4.,  1.,  7.,  5.],
       [ 7.,  0.,  8.,  8.,  3.,  6.,  6.,  6.,  4.,  3.],
       [ 4.,  3.,  4.,  4.,  7.,  6.,  4.,  5.,  6.,  7.],
       [ 9.,  0.,  8.,  9.,  7.,  1.,  2.,  2.,  4.,  6.],
       [ 7.,  4.,  2.,  8.,  6.,  5.,  2.,  9.,  9.,  9.],
       [ 7.,  6.,  9.,  2.,  9.,  0.,  1.,  5.,  7.,  3.]])

In [5]: scaler = MinMaxScaler()              # defaults to range [0, 1]

In [6]: scaler.fit(x)                        # compute the scaling factors
Out[6]: MinMaxScaler(copy=True, feature_range=(0, 1))

In [7]: scaled_data = scaler.transform(x)    # scale the data

In [8]: scaled_data.shape                    # still the same shape
Out[8]: (10, 10)

In [9]: scaled_data.min()                    # min and max are 0 and 1 as expected
Out[9]: 0.0

In [10]: scaled_data.max()
Out[10]: 1.0


Can Transformer Models be used for Training Chatbots? Can Transformer Models be used for Training Chatbots?
Note  - I am talking about the transformer model google released on the paper 'Attention is all you need'

A transformer is just a neural network. Sure, it is way more complex than a feed forward one, but it still is a neural network. As long as you provide the correct dataset (supervised case, so the correct pair input-target) your model should be able to learn the hidden representation that make it able to answer to new questions.
I found this interesting tutorial, try to check it out.


Sub topics with Latent Dirichlet Allocation I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.

I've come across this implementation of Hierarchical LDA, but I'm having a hard time implementing it (no community support). And I don't think gensim's hdpModel is what I want, given this discussion.

I'm currently doing this:

1) train LDA model on all records to get general topics

2) use this LDA model to assign each record a primary topic

3) for each topic, retrieve only the records that were assigned that topic

4) train a new LDA model only on the filtered records (for example, where topic ID == 3) to generate sub topics from filtered set

5) Assign each record a general topic ID and a sub topic ID

Is this a valid way to get topics and sub topics? Should I be weary of this approach?

Thanks for any insight.

I actually do not think your method is a good way to find subtopics.  Consider a document X with a distribution of topics z.  X is made up of a mixed model distribution of topic Z.  If you just give a document the most domiant distribution, and then run lda again, you might find subtopics but you'll also refind the topics that should should perhaps not be considered subtopics.

For example, let's consider a document that talks about food and exercising and why certain foods and cardo is good for heart health.  Let's assume that we find topics 1 and 2 such that the main topics are food, and exercise.  Suppose that the domiant distribution is topic 1.  Then running LDA on that particular document again, you're going to refine food, exercise and perhaps nutrition, viatims, etc.  However, there's no real reasons what should be considered the topic without inspecting each one and making an inference regarding it.  Also you'll have no real way to discern how deep you should go down this tree.

The original paper on hLDA can prove useful and exploits the chinese restraurant process and its relationship to dirichlet distribution to form the topic relationship and tree depth problem.  Also on David Blei github (bleilab) has an implentation in c++.  You can essentially setup a little shell script after processing the data in a langauge you may be more familar with and use his code.


Multiple output classes in keras I'm trying to predict movie genres using a neural network. I initially considered using a softmax layer as my output layer, but since a movie can have multiple genre labels, how should my output be?

Also, how would I have to format my data to make it work with Keras?

how would I have to format my data to make it work with Keras?

Your training labels in the output layer should be a binary vector that is 1 for class which is present and 0 for class which is not. For example, let us assume you have 3 classes of genres - comedy, romantic and horror. There are many ways to make it and Scikit-learn has a method which makes it very easy which I show below.

>>> from sklearn.preprocessing import MultiLabelBinarizer
>>> mlb = MultiLabelBinarizer()
>>> y = mlb.fit_transform([[0,2],[1]])
array([[1, 0, 1],
   [0, 1, 0]])

  I initially considered using a softmax layer as my output layer, but since a movie can have multiple genre labels, how should my output be?

This is a simple Keras example I suggest.

>>> from keras.models import Sequential
>>> from keras.layers import Dense, Activation

>>> model = Sequential([
    Dense(32, input_dim=784),
    Activation('relu'),
    Dense(10),
    Activation('sigmoid'),
    ])
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy')
>>> model.fit(X_train, y_train)

Refer this for more info. I used sigmoid because it is better for multilabel classification.


In Word2Vec, how to the vector values translate back to related words? Apologies for the newbie question: I've just downloadd the GoogleNews word2vec bin file and used convertvec to convert it to a text file to look at the vector values. I see they are values between -1 and 1. Can somebody explain how these values map back to related terms?

I would eventually like to write some code to take this bin/text file, transform it and load it into a MongoDB database for further manipulation.

In case you use Gensim in Python, you can create a dictionary with the word and the vector.

from gensim.models import Word2Vec

# we create a Word2Vec model using a window size of 4 and word vectors dimensionality of 100
model_1 = Word2Vec(window=4,size=100,sg=1,min_count=1, workers = -1)
# prepare the model vocabulary
model_1.build_vocab(this_is_your_text_dataset)
# train the model
model_1.train(this_is_your_text_dataset,total_examples=model_1.corpus_count,epochs=model_1.iter)

#we create a dictionary with the word and its vector as the value
vectors_model = dict(zip(model_1.wv.index2word, model_1.wv.syn0))

The above will create a dictionary in the format:

{
"word_1": vector_1,
"word_2": vector_2,
"word_3": vector_3,
....
}

With this, you can go back and forth from words to vectors and from vectors to words.


Reducing the size of a dataset I am trying to classify gestures. I am using Python's scikit learn library classification algorithms for that. I have collected depth images for this purpose. 200 samples are collected for each gesture. Each gesture is made up of 25 frames and each frame is of size 240x420. I tried PCA for dimensionality framewise for reducing the size of each gesture (200 samples each) to make it easy to run on the machine. Still the large size of the data make it difficult to run in my machine when the number of gestures to classify are larger than 4. I am looking for methods to make it run on my machine.

There are a number of ways to tackle this, I am going to focus on feature selection/extraction, because you mentioned PCA.
Sklearn itself offers a few feature selection/extraction algorithms already, see here, like SelectKBest. This would mean for you to maybe select specific frames, or samples, or even pixels (unlikely).
Further it has not only PCA but a few more, see here. I am going to mention PCA, NMF, ICA. While you already apparently tried this, it is important to note that these algorithms als have to be tuned correctly.
Now, on the other hand, like Graph4Me already mention you can use a CNN. For this you can use the structure of an AutoEncoder, which tries to learn a minimal representation of the input, to correctly restore the input as output. The Decoder-Encoder structure can be trained, to then only use the Encoder as a way to obtain dimensionally reduced training samples. A tutorial (for pytorch) is here, although it is for text, the same principle can be applied to images and videos.
As a final note, you can obviously try some simple preprocessing, like cropping the video, reducing the framerate, converting to grayscale if not already done, or even just black and white. I also hope you are already processing your data sequentially in every regard, if loading everything at once is an issue.


PANDAS Within Category Normalization I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this:

    pos product sales   week
0   1   car     250      1
1   2   tank    400      2
2   2   car     300      1
3   1   tank    500      2

The goal is to normalize the data between 0,1 for each point of sale and product, e.g the minimum and maximum relate to the minimum and maximum of sales within a specific product and a specific point of sale. I guess I can just create a column for each combination (example below) and then normalize each column but I'm looking for a more elegant solution.

    pos     1         2
    product car tank  car   tank
week
1           250  0    300     0
2           0    500    0   400

Thanks,
Al

As you require it to be per week vs point of sale, you have to group them by those columns. Once grouped, you could run do the standard split-apply-combine, where based on the grouping, you get splits or groups, you can apply a function that normalizes data in this group and then you combine all the groups back.

You can read about it more here : https://pandas.pydata.org/pandas-docs/stable/groupby.html


Understanding portfolio-level risk models I have a tremendous amount of experience training supervised machine learning models. However, I recently became a data scientist at a small financial services company, and I've been asked to build some sort of model that assesses portfolio-level risk. I'm not sure how to go about this task. Is this a supervised learning problem? It doesn't sound like one.

I used to work as a software/data engineer at a different financial services company, and while there I was briefly exposed to the concept of Basel credit risk management. Is Basel used for assessing portfolio-level risk? As far as I could tell (I could be wrong), Basel seemed to aggregate charge-off predictions by running every single client through each of the company's relevant credit risk models (which were trained in a supervised fashion). If my memory is right, how does this aggregation work? Is it effective?

I think this is more of a finance question. Assuming portfolio here refers to an investment of two assets, then

To determine the portfolio risk therefore, you'd need to either predict, or use historicals for $\sigma_1$, $\sigma_2$, $Cov_{1,2}$

Formula from: http://www.investopedia.com/exam-guide/cfa-level-1/portfolio-management/portfolio-calculations.asp


ValueError: could not convert string to float: '���' I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string of 26 characters.
array([['143347', '1325', '28.19148936', ..., '61', '0', '0'],
   ['50905', '0', '0', ..., '110', '0', '0'],
   ['143899', '1325', '28.80434783', ..., '61', '0', '0'],
   ...,
   ['85', '0', '0', ..., '1980', '0', '0'],
   ['233', '54', '27', ..., '-1', '0', '0'],
   ['���', '�', '�����', ..., '�', '��', '���']], dtype='<U26')

When I convert it to a float datatype, using
X_f = X.astype(float)

I get the error as shown above. how to solve this string formatting error for '���'?
I realize that some characters are not read properly in the dataframe, and the unicode replacement character is just a result of it.
My questions:-

How do I handle this misreading?
Should I ignore these characters? Or should I transform them to zero maybe?

Additional Information on how the data was read:-
importing relevant packages
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.functions import col

loading the dataset in a pyspark dataframe
def loading_data(dataset):
    dataset=sql_sc.read.format('csv').options(header='true', inferSchema='true').load(dataset)
    # #changing column header name
    dataset = dataset.select(*[col(s).alias('Label') if s == ' Label' else s for s in dataset.columns])
    #to change datatype
    dataset=dataset.drop('External IP')
    dataset = dataset.filter(dataset.Label.isNotNull())
    dataset=dataset.filter(dataset.Label!=' Label')#filter Label from label
    print(dataset.groupBy('Label').count().collect())
    return dataset

# invoking
ds_path = '../final.csv'
dataset=loading_data(ds_path)

check type of dataset.
type(dataset)

pyspark.sql.dataframe.DataFrame
convert to np array
import numpy as np
np_dfr = np.array(data_preprocessing(dataset).collect())

split features and labels
X = np_dfr[:,0:22]
Y = np_dfr[:,-1]

show X
>> X
array([['143347', '1325', '28.19148936', ..., '61', '0', '0'],
       ['50905', '0', '0', ..., '110', '0', '0'],
       ['143899', '1325', '28.80434783', ..., '61', '0', '0'],
       ...,
       ['85', '0', '0', ..., '1980', '0', '0'],
       ['233', '54', '27', ..., '-1', '0', '0'],
       ['���', '�', '�����', ..., '�', '��', '���']], dtype='<U26')

Though not the best solution, I found some success by converting it into pandas dataframe and working along.

code snippet

# convert X into dataframe
X_pd = pd.DataFrame(data=X)
# replace all instances of URC with 0
X_replace = X_pd.replace('�',0, regex=True)
# convert it back to numpy array
X_np = X_replace.values
# set the object type as float
X_fa = X_np.astype(float)

input

array([['85', '0', '0', '1980', '0', '0'],
       ['233', '54', '27', '-1', '0', '0'],
       ['���', '�', '�����', '�', '��', '���']], dtype='<U5')

output

array([[ 8.50e+01,  0.00e+00,  0.00e+00,  1.98e+03,  0.00e+00,  0.00e+00],
       [ 2.33e+02,  5.40e+01,  2.70e+01, -1.00e+00,  0.00e+00,  0.00e+00],
       [ 0.00e+00,  0.00e+00,  0.00e+00,  0.00e+00,  0.00e+00,  0.00e+00]])


What are the best activation functions for Binary text classification in neural networks? I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification.

I have heard that Relu is best for Binary classification (not sure if im correct)

I have used keras to train a model, which is 2 layer , Dense 512, dropout 0.3, activation = "Relu" for these layers,

But the predictions are not upto the mark.

I have also changed the Dense units to 1024, keeping others same, but still I got bad predictions. (Validation accuracy 50%)

So, can i use other activations, or change my model layers (add few more layers) ??

What can be the best option ?

Relu often works well, but ultimately you need to try. If you really have only one hidden layer, you might have a model which cannot properly learn.

I guess you use a sequential, dense model architecture? Try to add more hidden layers. Don‘t add too much capacity for a start. You could try something like 256, 8, 8, 1 with relu.

Dropout is good to prevent overfitting. It is not important for learning per se.

Just a remark for future posts: always include your basic model code if possible. So people can understand what you are really up to. Also a little data description is useful. Especially with text, preprocessing is really important. So the way you treated your data before it goes into the model is relevant.