On a multi lingual sentiment corpus I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I have searched high and low but could not find anything like this available. I already have the news articles in each language.

My question to the community is how would you achieve this as accurately as possible?

I was first looking at Mechanical Turk, where you can hire people to  label each article manually for you. And this may be the best way forward but expensive.

Next, I thought about all of those existing popular libraries (some of whom have already used Mechanical Turk) that do sentiment analysis (AFINN, Bing Liu, MPQA, VADER, TextBlob, etc.)

Sentiment Idea

My current idea is that I run each news article across a few of these libraries (for example AFINN, then TextBlob, then VADER) and for those articles who show positive, negative, neutral unanimously though all three libs are accepted into the corpus. Does that seem like a fairly strong and reasonable verification process?

Language Idea

The next issue pertains to language itself. The 3 lib pipeline above can be executed in English with no issue. However these libraries do not uniformity support many other languages (Spanish, German, Chinese, Arabic, French, Portuguese, etc.) I was thinking about doing what VADER suggests and taking the news stories in non-english languages and sending them though Google Translation API to get them into English and then send them through the existing 3 lib pipeline above. I do realize there will be a loss in semantics for many articles. However, my hope is that enough articles will translate well enough that some pass through the 3 lib pipeline.

I am aware that translating and sending news articles through this triple blind sentiment pipe may take a 100k corpus and yield 10k results. I am fine with that. The accuracy and then price is my concern. I can easily acquire more data.

What would you do that may be a more accurate way of achieving a sentiment corpus of news articles? Is there an existing best used practice for assembling a corpus like this?

Several questions and thoughts come to mind.

What languages are in the corpus?  This may impact what services you can leverage.
I like the "Sentiment Idea" for languages that are supported natively by the services you mentioned.
I would keep the "Language Idea" as the last resort as it is possible that the translation engine may not capture the sentiment of the original language.
Mechanical Turk would be a good option if you can limit the number of samples sent for classification.  For each language, you could try clustering the passages by, for example, word count into 30 (you pick) clusters and then perform sampling within the clusters to identify candidate passages to send to Mechanical Turk. I have used this technique to try to sample across the vector space more uniformly.

Don't dismiss oW_'s comment.  You should seriously consider breaking the articles into paragraphs. You can always aggregate the paragraph scores to the article, but it's hard to get one representative score as the text gets longer.

HTH


How to verify that a trained neural network is trustworthy? Currently I'm looking into possibilities to prove that a trained neural network (based on a regression task) performs well on the whole sample space based on statements made from test set metrics. In other words, I want to make sure that there is no sample or region of samples out there, where the network performs badly on, assuming that the sample space is finite.
Are there any well established approaches for this?

A definitive and deterministic answer regarding the existence of adversarial samples is only possible by checking the entire sample space, which you said is finite and hence feasible but maybe too computationally expensive.
I would suggest to generate a grid of samples over the entire space to get a somewhat representative subspace and evaluating the model on it. As you suggested, the normal error distribution out of that whould give you a more or less reliable probabilistic statement on future model errors, depending on the size of the subspace I guess.
However, I also am curious if there is a better way to evaluate ones model reliability.


It seems that the output of sklearn.metrics.pairwise.euclidean_distances is different to the formula on doc The doc of sklearn.metrics.pairwise.euclidean_distances() gives this formula

  dist(x, y) = sqrt(dot(x, x) - 2 * dot(x, y) + dot(y, y)).

Apply this formula to this example

X = [[0, 1],
     [2, 3]]

Y = [[1, 2],
     [3, 4]]

np.dot(X,X) - 2*np.dot(X,Y) + np.dot(Y,Y)

gives this result

array([[ 3,  5],
       [-1,  1]])

whilst calling sklearn.metrics.pairwise.euclidean_distances()

euclidean_distances(X , Y, squared = True)

gives

array([[ 2., 18.],
       [ 2.,  2.]])

It seems that the output of euclidean_distances() is not consistent to the formula from the doc.

The sklearn docs' formula says it is applying to row vectors $x$ and $y$.  When you call np.dot on the matrices $X$ and $Y$ it takes the matrix product.

EDIT (responding to question in comments):
It's not straightforward, as the row-vs-row operations needed aren't quite the usual matrix operations.  The source code for euclidean_distances does it this way (except that it does lots of input checks, operates on sparse inputs when possible, etc.):

(X*X).sum(axis=1)[:, np.newaxis] - 2*np.dot(X,Y.T) + (Y*Y).sum(axis=1)[np.newaxis, :]

That's not exactly straightforward itself, so I'll say a little more.  Say $X$ has $m$ rows and $Y$ has $n$ rows.  The middle term, by taking $Y^T$, gives us a $m\times n$ matrix whose $(i,j)$-entry is the dot product of the $i$th row of $X$ with the $j$th row of $Y$.  In the other terms, * on numpy arrays is the coordinate-wise product; summing along rows gives us the rows' squared-norms.  The newaxis is a nice trick: casting the first term to now be a $m\times 1$ matrix, adding it to the middle term's $m\times n$ matrix actually adds it to every column of that matrix (without needing to actually build the matrix of repeated columns of $X$'s squared-norms).  And of course similarly for the last term: casting to a $1\times n$ matrix makes it add to every row of the result.


Does anybody know where this rule of thumb came from? Rule of them: embedding vector dimension should be the 4th root of the number of categories I was taking an online ML course and the lecturer said that a rule of thumb for choosing the number of dimensions when embedding categorical data is the following
embedding vector dimension should be the 4th root of the number of categories
The lecturer worked for Google and when I looked on the internet for this I only found a Google blog which quickly mentioned it google blog link. I'm guessing it's something that they came up with at Google but was wondering if somebody else has maybe seen it in a research paper.

google has published a word embedding that has embedding dimension of 300. Following the rule you have given it should have trained on $300^4 = 8.1*10^9$ words. If google is using ngrams instead of just words, then it seems plausible.


2D-Input to LSTM in Keras I have following problem: I would like to feed LSTM with

  train_datagen.flow_from_directory

The input is basically a spectrogram images converted from time-series into time-frequency-domain in PNG format that has a dimension of: timestep x frequency spectrum. 1 sample = 1 PNG image in uint8. In my example: 3601 timesteps with 217 frequency spectrum (=features) / timestep.

The spectrogram itself is just 1D, but I think "flow from directory" function was hard-coded to only prepare 3D image matrix and thus the input shape was becoming , which is totally pity because there are some people who are only working with purely greyscale uint8 image, and some who are working with multispectral and hyperspectral images.

My codes are following:

import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.layers import LSTM
from keras import optimizers
from keras import backend as K
import tensorflow as tf

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 10
nb_train_samples = num_classes*70
nb_validation_samples = num_classes*20
epochs = 20
batch_size = 10
input_shape = (img_width, img_height)

model.add(LSTM(units=256, input_shape= input_shape, return_sequences=True))
model.add(LSTM(units=128, return_sequences=True))
model.add(LSTM(units=64))
model.add(Dense(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size)

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size)

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

And then as soon as I run the program, of course it gives an error message::

**ValueError: Error when checking input: expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)**

The message:

  expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)

clearly suggests it does not agree with my definition of input shape of: (3601, 217)

Any idea to easy fix the problem?
Thanks in advance.

Why do you define the last dimension of input_shape as $3$? Just put your desired input dimensions accordingly and it should be fine:

input_shape = (img_width, img_height)

Update with the full code:

The best way would be to use TimeseriesGenerator instead of ImageDataGenerator but there seems there not flow_from_directory method meeting your needs. So, I think the best solution is to squeeze the last dimension of the generator output. Also, you have a color_mode option that allows to generate a 1-channel only tensor for grayscale images.
Full code of concerned parts:

model = Sequential()
model.add(Lambda(lambda x: x[:,:,:,0], input_shape=(*input_shape, 1)))
model.add(LSTM(units=256, return_sequences=True))
model.add(LSTM(units=128, return_sequences=True))
model.add(LSTM(units=64))
model.add(Dense(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    color_mode='grayscale')


Linear regression - LMS with gradient descent vs normal equations I wonder when to use linear regression with stochastic or batch gradient descent to minimize the cost function vs when to use normal equations? The algorithms using gradient descent are iterative, so they might take more time to run, as opposed to the normal equation solution, which is a closed form equation. But it does use matrices to store the training data. Does this mean gradient solutions require more processing power, but using the normal equation method requires more memory because of the matrices? Which method is optimal in what scenario?

Andrew Ng answers this question succinctly in his Coursera lecture about the normal equation.  I will summarize.

You have m training examples and n features.

Disadvantages of gradient descent:

you need to choose the learning rate, so you may need to run the algorithm at least a few times to figure that out.
it needs many more iterations, so, that could make it slower

Compared to the normal equation:

you don't need to choose any learning rate
you don't need to iterate

Disadvantages of the normal equation:

Normal Equation is computationally expensive when you have a very large number of features ( n features ), because you will ultimately need to take the inverse of a n x n matrix in order to solve for the parameters data.

Compared to gradient descent:

it will be reasonably efficient and will do something acceptable when you have a very large number ( millions ) of features.

So if n is large then use gradient descent.

If n is relatively small ( on the order of a hundred ~ ten thousand ), then the normal equation


Deep Learning: Feed Forward for Unbalanced Classes Using Tensor Flow In theory, Deep Learning NN can predict a class with very few observations.  My problem, I have a class that happens less than 4% of the time.  Feeding the network data with distribution intact (96 one class and 4% the other class) results in the network predicting mostly the most common class.  Using under sampling result in a model that is able to predict 90% of the less common class, but still have a high False Positive.
I know that using F Score (with beta >= 2) in the training phase of the model will allow me to improve significantly the model performance. My question, how or where do I set my own performance metric in Tensor Flow?
Any suggestions are appreciated.
By the way, I did tune all the parameters (learning rate, momentum, etc..) performance increases but still not at the level I want.
My network has 20 inputs 5 hidden layers with 60 unites
My batch is 100 (tried up to 500), and I tried epoch from 100 to 1,200

The code is in Mathematica.

 netDrop1 =
     NetGraph[<| "i1" -> BatchNormalizationLayer[],
    "l1" -> CatenateLayer[], "l12" -> DropoutLayer[0.2], "l2" -> 60,
    "l21" -> DropoutLayer[0.5], "l3" -> Tanh, "l22" -> 60,
    "l221" -> DropoutLayer[0.5], "l33" -> Tanh , "l23" -> 60,
    "l234" -> BatchNormalizationLayer[0.5] , "l34" -> LogisticSigmoid,
    "l24" -> 60, "l2435" -> DropoutLayer[0.5], "l35" -> Ramp,
    "l45" -> 60, "l435" -> DropoutLayer[0.5], "l36" -> Tanh,
    "l55" -> 60, "l37" -> Tanh, "l4" -> 2,
    "l5" ->  SoftmaxLayer[]|>, {NetPort["Input1"] ->
     "i1" -> "l1" ->
      "l12" ->
       "l2" -> "l21" ->
         "l3" -> "l22" ->
           "l221" ->
            "l33" ->
             "l23" ->
              "l234" ->
               "l34" ->
                "l24" ->
                 "l2435" ->
                  "l35" ->
                   "l45" ->
                    "l435" ->
                    "l36" ->
                    "l55" ->  "l37" -> "l4" -> "l5", {{NetPort[
        "MaritalStatus"], NetPort["Gender"], NetPort["Input"]} ->
       "l1"}}, "Input" -> 11, "Input1" -> 2,
       "MaritalStatus" -> NetEncoder[{"Class", marital, "UnitVector"}],
       "Gender" -> NetEncoder[{"Class", {"M", "F"}, "UnitVector"}],
       "Output" -> NetDecoder[{"Class", {"no", "yes"}}]]

From reading publication about NN, I have decided to create a Dropout with = 0.2 at the input level, and at all subsequent levels to have Dropout at 0.5.
I have used both Tangent and ReLu (but found little differences in performance between them)
As you notice, a simple FeedForward Network

I do under sampling, the under sampling with great success:  the model is able to recall most of the yes, but still predict many negative samples as positive.

  <|{"yes", "no"} -> 972, {"no", "no"} -> 1082, {"yes", "yes"} ->
  81, {"no", "yes"} -> 3|>

By the way, I have used these same data,and used RandomForest with a utility function with great results http://femvestor.blogspot.com/search?q=geico

To Improve the performance of the network, I added a Batch Normalization layer, and other variables that I had to remove when using Random Forest.  One of these variables is a categorical variable with 51 levels.  I made sure to add an embedding layer that will represent the values by a vector of size 10.
As the original dataset had 15 numerical variables and 9 categorical variables, I have decided to also perform dimension reduction using auto-encoding.


Do I need to correct predict_proba by training fraction? Many algorithms provide a predict_proba function indicating probability of a case to belong to that class (e.g. https://scikit-learn.org/stable/modules/generated/sklearn.svm.libsvm.predict_proba.html ).

Quoting from the answer by @Media at Explain Binary Classification with output 0.5 (True)

  Suppose that you have a car classifier for distinguishing between
  white and blue cars. during training you had 100 images of blue car
  and 20 images of white car. During recall phase, if for an arbitrary
  image you have 50 percent for each class...

If blue cars accounted for 83% of training cases, and I get predict_proba for a car to be blue to be 0.5, do I take the probability to be 0.5 or do I need to correct it by a factor of 0.83?

If I do need to correct, do I multiply the factor (0.5*0.83) or divide it (0.5/0.83) to get the correct probability?

It depends on the model, whether your training data is representative of the testing data, and possibly on how "easy" the classification problem is.  But the short answer is that (probably) no adjustment is necessary.

The model is exposed to the imbalance in the training data; most models will have that baked into their probability scores, so to trust the given probabilities requires that the testing data have a similar distribution.  This goes to what exactly we mean by "the probability" that the new car is blue; I think in most cases, you would want the training data to be representative of the data in your use-case, so that we really do want the probability to be aware of the imbalance.  [Now, sometimes in the presence of extreme imbalance, certain models might learn better with some adjustment, in which case you'll want also some "probability calibration" post-processing, but that's generally more complex than the simple multiply-or-divide method you suggest.  In logistic regression for example, there's a well-known additive adjustment to the constant term, so that turns into a multiplicative adjustment to the log-odds, which does not translate into such a nice transformation on probabilities.]

Finally, some models implicitly skew their scores, so predict_proba should be taken with a fair bit of skepticism.  Again, look up "probability calibration."


Basic method of optimizing hyperparameters I recently read the LIPO blog post on the dlib blog:
http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html

It mentions that it can be used for optimizing hyperparameters of eg metaheuristic algorithmsike simulated annealing or genetic algorithms.

I looked for info on how optimizing hyperparameters work in general and the Wikipedia page is the most informative I found but it doesn't answer my basic questions:
https://en.m.wikipedia.org/wiki/Hyperparameter_optimization

My question is just: what is the basic idea for optimizing hyperparameters?

If I have some problem I'm trying to solve with simulated annealing, I know that the starting temperature and the cooldown rate are important in determining how well the algorithm does at finding a solution.

I know that I could completely run the algorithm with one set of parameters, modify one of the parameters, completely run it again, then reset the parameters and modify the other parameter and run it again. This could give me a numerical gradient that I could use to modify the parameters via gradient descent.

However... At this point I had to run the whole algorithm 3 times just to get a single modification of the hyperparameters.

I feel like I must be missing something obvious because optimizing the hyperparameters would take many many hundreds or thousands of times or more the cost of running the whole thing once, which doesn't seem useful at all. Can someone clue me in?

Hyperparameter optimization follows the same rules as model selection. Each set of hyperparameters effectively represents a different model you are considering, so the data you use to fit the model with some set of hyperparameters needs to be different from the data you use to evaluate which set of hyperparameters you want to ultimately use. A common approach for evaluating hyperparameters is nested cross-validation. This basically means that you need to treat hyperparameter selection as part of your model training process, and when you evaluate your model you evaluate the entire process front to back, i.e. treating hyperparameter tuning as a component of model training with respect to cross-validating over your training process. There's an excellent discussion of this in section 7.10.2 ("The Wrong and Right Way
to Do Cross-validation") of Elements of Statistical Learning, which you can read online and/or download for free. The general idea is that if you're not careful, you can actually overfit to your evaluation data. Play with this demo to see for yourself.

But yes, your intuition is correct. Hyperparameter tuning is often very computationally expensive. One way people sometimes try to minimize this cost is by limiting the search space of feasible parameters to a small discrete set, e.g. grid search. Another approach is to use guassian processes or KDEs to approximate the cost surface in parameter space. You can even use a multi-armed bandit approach.

Frankly, you can really use pretty much any non-linear optimization technique for hyperparameter tuning, as long as you follow the appropriate cross-validation rules. The trick is figuring out bang-for-your-buck in terms of how much time/effort/compute you are willing to expend exploring parameters vs. the potential improvement to your model. Additionally, there are concerns about the limits of repeated cross-validation/holdout evaluations, but that's a whole other rabbit hole.


How do you predict a continuous variable when all your independent variables are categorical I am new to data science and ML. Recently I have been given a sales dataset which contains weekly sales of a fashion brand. It has information about the product like category(t shirt, polo shirt, cotton shirts, briefs, jeans, etc.), gender (male, female, unisex), partner stores ( multiple retail outlets), the color of the apparel (some 150 color codes in hex format), MRP, Sold MRP and quantity sold. The data is for two years timeframe.
I added a new column called discount which is deriver from sold mrp/qty and mrp.
Can anyone suggest any kind of predictive modelling scenario for the above kind of data? I have already done time series forecasting and clustering based on high-performing partner stores. I tried Regression but how do you perform regression when most of the independent variables are categorical in nature ?
Any help would be much appreciated.

It depends on the problem statement, have you got some background about the source of this dataset.. if not then predicting sales in the most common type of analysis you can do here.

And regression doesn't really only depend on input values, it is also considering the past output values so do not worry if the input values are all categorical. You can also try gradient boosting regression trees for predicting sales in partner stores.

Aprt from predictive models, its also a good idea to do some data analysis and  find trends and patterns within your data by utilizing graphs


Is there a way to replace existing values with NaN I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column having different proportions of Nan values.

I'm using the Iris dataset for this experimentation to see how the algorithms work and which one works the best. The link for the dataset is here.

Thanks in advance for the help.

Randomly replace values in a numpy array

# The dataset
data = pd.read_csv('iris.data')
mat = data.iloc[:,:4].as_matrix()

Set the number of values to replace. For example 20%:

# Edit: changed len(mat) for mat.size
prop = int(mat.size * 0.2)

Randomly choose indices of the numpy array:

i = [random.choice(range(mat.shape[0])) for _ in range(prop)]
j = [random.choice(range(mat.shape[1])) for _ in range(prop)]

Change values with NaN

mat[i,j] = np.NaN

Dropout for any array dimension

Another way to do that with an array of more than 2 dimensions would be to use the numpy.put() function:

import numpy as np
import random
from sklearn import datasets

data = datasets.load_iris()['data']

def dropout(a, percent):
    # create a copy
    mat = a.copy()
    # number of values to replace
    prop = int(mat.size * percent)
    # indices to mask
    mask = random.sample(range(mat.size), prop)
    # replace with NaN
    np.put(mat, mask, [np.NaN]*len(mask))
    return mat

This function returns a modified array:

modified = dropout(data, 0.2)

We can verify that the correct number of values have been modified:

np.sum(np.isnan(modified))/float(data.size)

[out]:

0.2


Use serialized model without installing dependencies I prototyped an ML model consisting of preprocessing + multiple stacked regressors. I would like a colleague of mine to develop an API that will query the model. Is there any way to query the model (sklearn pipeline) without having to download all the dependencies (XGBoost, LGBM, CatBoost, ...).

I tried to serialize it with Joblib but when we deserialize it on another machine it requires to have dependencies installed.

The goal is really to transform the sklearn's pipeline to a complete inert black box that requires minimal setup. Is it possible?

You may be able to convert the entire model pipeline to a standardized format. PMML is such a format, and there are tools (e.g. jpmml) to convert all your named modeling package objects to PMML, though perhaps you've used something else that isn't already easily-converted.

Otherwise, just force installation of dependencies (and make it easy), through a virtual environment or docker image or ...


Why aren't Genetic Algorithms used for optimizing neural networks? From my understanding, Genetic Algorithms are powerful tools for multi-objective optimization.

Furthermore, training Neural Networks (especially deep ones) is hard and has many issues (non-convex cost functions - local minima, vanishing and exploding gradients etc.).

Also I'm that conceptually training a NN with GA is feasible. I was wondering, why aren't they used in practice? Is it a matter of performance?

Training Neural Networks (NNs) with Genetic Algorithms (GAs) is not only feasible, there are some niche areas where the performance is good enough to be used frequently. A good example of this is Neuroevolution of augmenting topologies or NEAT, which a successful approach to generating controllers in simple environments, such as games.

In the more general case though, the approach does not scale well to large, deep networks with many parameters to tune.

Genetic algorithms and other global searches for optimal parameters are robust in ways that gradient-based algorithms are not. For instance, you could train a NN with step function activations, or any other non-differentiable activation functions. They have weaknesses elsewhere. One thing relevant in the case of GAs used for NNs, is that weight parameters are interchangeable in some combinations but heavily co-dependent in other combinations. Merging two equally good neural networks with different parameters - which you would do in cross-over in a GA - will usually result in a third network with poor performance. NEAT's success is partially in finding a way to address that issue by "growing" the NN's connections and matching them up between similar neural networks.

Gradient-based approaches are much more efficient. In general, and not just in domain of NNs, if you can calculate gradient of a function with respect to parameters, then you can find optimal parameters faster than most other optimising techniques. An accurate gradient guarantees at least a small improvement from a single evaluation, and most other optimisers fall into a generate-and-retry paradigm which cannot make that kind of guarantee. The weakness of tending to find local optima has turned out not be a major hindrance for the loss functions in NNs, and has been tackled with some degree of success using extensions to basic gradient descent such as momentum, RPROP, Adam etc.

In practice on a large multi-layer network, gradient methods are likely orders of magnitude faster than GA searches such as NEAT for finding network parameters. You won't find any GA-trained CNNs that solve ImageNet, or even MNIST, where the GA has found the network weights unaided. However, GAs, or at least some variants of them, are not 100% ruled out. For instance this 2017 blog reviews recent papers including Large-Scale Evolution of Image Classifiers which explores using GAs to discover NN hyperparameters which is an important task in machine learning, and not very tractable using gradient-based methods.


Test dataset with categorical variable value not present in train dataset & transformer I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ).

I am fitting a transformer doing just that on a train dataset df and then transform the test dataset df2.

How do I deal with a value appearing solely in the test dataset ?

When fitted on the train dataset the transformer received no mean value of the target variable on that value.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 'a' , 'x' , 0 ] ,
                   [ 2 , 2 , 2 , 'v' , 'b' , 'y' , 1 ] ,
                   [ 4 , 5 , 1 , 'w' , 'c' , 'z' , 1 ] ,
                   [ 2 , 1 , 9 , 'w' , 'c' , 'x' , 1 ] ,
                   [ 1 , 0 , 4 , 'w' , 'b' , 'y' , 1 ] ,
                   [ 2 , 2 , 3 , 'v' , 'b' , 'z' , 0 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 'a' , 'x' , 0 ] ,
                      [ 9 , 2 , 2 , 'v' , 'a' , 'y' , 0 ] ,
                      [ 4 , 5 , 1 , 'w' , 'c' , 'k' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

df is my train dataset, df2 my test dataset.

We can see variable 'six' has the k value not existing in the train dataset.

Next :

df[ 'target' ] = df[ 'target' ].astype( 'float64' )

Next ( my homemade transformer ) :

class Cat2TargetMean( BaseEstimator , TransformerMixin ) :

    def __init__( self , col2trans , tgt_col ) :
        self._col2trans = col2trans
        self._tgt_col = tgt_col

    def fit( self, X, y = None ) :
        self._dic_col_p = {}

        for col in self._col2trans :
            p = X.groupby( col ).mean()[ self._tgt_col ]
            self._dic_col_p.update( { col : p.to_dict() } )
        return self

    def transform( self , X , y = None ) :
        for col , dic_p in self._dic_col_p.items() :
            X.replace( { col : dic_p } , inplace = True )
        return X

Then :

 tsf = Cat2TargetMean( [ 'four' , 'five' , 'six' ] , 'target' )

 tsf.fit( df )

 tsf.transform( df )

 tsf.transform( df2 )

Result :

    one two three   four    five    six target
    0   2.0 7.0 3.0 0.333333    0.0 0.5 0.0
    1   9.0 2.0 2.0 0.333333    0.0 1   0.0
    2   4.0 5.0 1.0 1.000000    1.0 k   1.0

'k' value of column 'six' has not been transformed.

I usually replace unseen and NaN values with the global target mean.

There are also already implemented transformers for target encoding that you could use that gives you some options such as smoothing: scikit contrib - target encoder.


Is this random forest logical correct and correct implemented with R and gbm? For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors.

I got a data set with 15 million entries and want to make a regression for a numerical target (time). The data structure is:

I have 7 categorical variables, 1 date and 4 numerical features. After data preparation I split the data into training and test data set.

Than I defined a gradient boosting machine model and search for the right parameters by try and error, research and more try and error. Is this approach correct as far?

#train and test are the prepared data frames
#RMSE is an implementation of the root mean squared error function
#gradient boosting machine training model
gbmTraining <- gbm(train$target~., data=train, distribution = "gaussian", verbose = TRUE,
                 n.trees = 500, #number of trees
                 cv.folds=5, #cross validation
                 n.cores=2, #number of processor cores
                 shrinkage = .03,       #learning rate
                 interaction.depth = 42, #number of notes
                 n.minobsinnode = 15)   #depth of tree
print(gbmTraining)
png(file = "Results/RelativeInfluence.png")
summary.gbm(gbmTraining, plot = TRUE)
dev.off()

print("best number of trees: ")
png(file = "Results/Convergence.png")
best_iter <- gbm.perf(gbmTraining, method="cv", plot.it=TRUE)
dev.off()
print(best_iter)

#make train prediction
fitTrain <- predict(gbmTraining, train, best_iter, type = "response", verbose = TRUE)

#training error
errorTrain <- RMSE(fitTrain, test$target)
errorTrain <- round(errorTrain, digits = 1)
errorDiffTrain <- fitTrain - test$target
print("meanTrainError(predicted, actual):")
print(mean(errorDiffTrain))

#write and save results
print("summary training")
print(head(data.frame("Actual" = train$target, "Predicted" = fitTrain)))
print(summary(data.frame("Actual" = train$target, "Predicted" = fitTrain)))
trainTable <- data.frame("Actual" = train$target, "Predicted" = fitTrain)

write.csv(trainTable, "Results/trainTable.csv", row.names = FALSE)

Here I want to know if I use the correct arguments if I calculate the errors. Do I calculate the errors between the fitted data and data from the test data set or do I use here the data from the training data set?

If I found good parameters and want to go to validation the implementation becomes shorter because I use the found parameters. I create a new model for the tests:

#gbm testing model
gbmTesting = gbm(test$target~., data=test, distribution = "gaussian", n.trees = 500,
                 n.cores=2, verbose = FALSE,
                 shrinkage = .03,
                 interaction.depth = 42,
                 n.minobsinnode = 15)

#make test prediction
fitTest <- predict(gbmTesting, data=test, n.trees = 500, type = "response", verbose = FALSE)

#test error
errorTest <- RMSE(fitTest, test$target)
errorTest <- round(errorTest, digits = 1)
errorDiffTest <- fitTest - test$target
print("meanTestError(predicted, actual):")
print(mean(errorDiffTest))

print("summary testing")
print(head(data.frame("Actual" = test$target, "Predicted" = fitTest)))
print(summary(data.frame("Actual" = test$target, "Predicted" = fitTest)))
testTable <- data.frame("Actual" = test$target, "Predicted" = fitTest)

write.csv(testTable, "Results/testTable.csv", row.names = FALSE)

Is this correct or do I fall into a mental tripping hazard?

p.s.
the stack exchange community is so huge I am not sure if this has to be in some of the other sites? If yes, which one?

Is the code correct?
Yes, it looks correct to me

How to calculate Errors:
Calculate accuracy for train data
Calculate accuracy for test data

If train error is high (RMSE in your case) - high bias, retrain model with more trees, less learning rate, more data (if possible)
If train error low, but test error high - high variance or overfit: regularization, increase k-fold
If tran and test error low - your model is good to go.
Hope this answeres your question.


How can I select data from a Tensorflow Dataset data collection? Is there any way to select features or labels from a tensorflow Dataset without using numpy conversion methods or iterate through?

The simplest example I found is:

import tensorflow_datasets as tfds

ds = tfds.load('mnist', split='train', shuffle_files=True)
print(ds)
>> <_OptionsDataset shapes: {image: (28, 28, 1), label: ()}, types: {image: tf.uint8, label: tf.int64}>

In this case, _OptionsDataset holds data in a dict, but the collection can be a tuple, list or even a np.ndarray.

If there is not a way to select data directly from a tensorflow Dataset, what's the most commonly used alternative?

You can use the ds.map() function to create dataset conatains only images or labels:

ds_images = ds.map(lambda d:d['images')

the original purpose of the map function is manipulating the data without converting to numpy, for example:

ds_images = ds.map(lambda d:d['images']/255)

hope I helped you.


ValueError: not enough values to unpack (expected 4, got 2) I have written this code

fig, (axis1, axis2,axis3, axis4)=plt.subplots(2,2,figsize=(10,4))

and I am getting this error

ValueError: not enough values to unpack (expected 4, got 2)

I tried many ways to remove this error but all was in vain.
Can you explain to me why I am getting this error?

Its because you have not looked how the values are packed in plt.subplot function.

>>> plt.subplots(2,2,figsize=(10,4))
(<matplotlib.figure.Figure at 0xa3918d0>, array([[<matplotlib.axes._subplots.AxesSubplot object at 0x000000000A389470>,
     <matplotlib.axes._subplots.AxesSubplot object at 0x000000000A41AD30>],
    [<matplotlib.axes._subplots.AxesSubplot object at 0x000000000A6F7EB8>,
     <matplotlib.axes._subplots.AxesSubplot object at 0x000000000BC232E8>]], dtype=object))

Instead of unpacking all values at once, unpack in steps. You will get a better idea then.

For your solution, to unpack -

>>> fig, [[axis1, axis2],[axis3, axis4]] = plt.subplots(2,2,figsize=(10,4))


Deep learning - aesthetics data modelling I want to train neural network on aesthetics. I am getting confused on how to go about for training data.

Assume, I have large data set of landscapes, portraits, wildlife etc which are aesthetic according to humans. But, I want to train them for the quality, the kind of colours involved, contrast levels, background blur etc.

How do I train images for this criteria? Is there a way for doing this by unsupervised learning?

Modeling aesthetics in media is an example of ordinal classification.
One of the most actively maintained datasets for this can be found is Jen Aesthetics
A relatively recent paper using deep learning towards aesthetics modeling is
this

Prior to deep learning era, research groups were trying to translate methods/guidelines used in the photography community to create/capture good quality pictures. There are several guidelines that you can explore with a bit of search online. One popular example is the 'rule of thirds'. Here the primary subject should not be centered in the image but offset and ideally centered at the intersection of 1/3 and 2/3 horizontal and vertical lines.

This is easy to translate into an algorithm: use salient object recognition or visual attention detection and measure the distance of the center of the salient/attention patch from the 4 rule-of-thirds points. Use this off-set as a feature. The closer the salient patch is to any one of the rule-of-third points, the higher the aesthetic ordinal score for that image.

This is another good paper that explores what makes images popular.

Some researchers have also used the tags or descriptions of photos as features. The objective here is to learn an association between lexical features and image aesthetics. They have sourced their data from online repositories like Digital Photography Challenge.

This subjective task is needless to say very complex. If you plan to address it, I'd recommend beginning with a clear definition of the context within which you aim to address aesthetics.

Ideally, you'd like to map any given image (media) to some value in [0, ..., 1] $\in \mathcal{R}$.
However, this is very difficult unless you have access to a lot of training data. I suggest trying instead to simplify the problem. If you can reliably map images to just two classes, good aesthetics and bad aesthetics.

You can successively generalize from binary classification to full fledged multi-class ordinal classification, for which you'll very likely have to keep increasing the depth of your CNN.

Good luck! Since, there is more to aesthetics than meets the eye! :-)


Problem reading python code Can someone explain the following python code.
value_geojson["features"][0]["properties"]["title"]

value_geojson is a geojson variable.
I like to think that I'm not a total python newbie but these are too many [] for me. I am a GeoJSON newbie though.
Would appreciate help on that one, eventhough I know it's probably to simple for SE

See sections 3.2 and 3.3 of the geojson standard.
It looks like value_geojson is a FeatureCollection object, which have a single member "features" which is a list of Feature objects.  So
value_geojson["features"][0]

is now just the first Feature object.  Those have a few members, including "properties", which is an arbitrary JSON object.  At that point we're outside of the geojson standard, so whatever "title" means is more context-specific.


improving accuracy of logistic model I am trying to reproduce results from one paper, where authors minimized the following loss function
\begin{align}
\min_{w \in R^d} \frac{1}{n} \sum_{i \in [n]} log(1 + exp(-y_ix_i^Tw))+\frac{\lambda}{2}\|w\|^2,
\end{align}
where $w$ are weights are $\lambda$ is regularization parameter for  ijcnn1 dataset.

This dataset is specific by unbalanced data (90%-0, 10%-1).
As a preprocessing step, I applied MinMaxScaler and StandardScaler.

I use model written in keras, which looks simply as:

model = keras.Sequential([
    keras.layers.Dense(1, activation="sigmoid", kernel_initializer='uniform', kernel_regularizer=regularizers.l2(1e-4), use_bias=True)
])
sgd = optimizers.SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_dataset, epochs=1000)

However, I manage to obtain only 91% accuracy at the best.
Looking at the prediction, I observe that my model learns to predicts almost everything to be zero. I also tried using class_weight, but it did not seem to help.
Does anybody have any suggestions, how to obtain better results?

If you have a disproportionate amount of zeros, it means you models doesn't have enough data in order to learn how to correctly classify observations. Because it sees zeros almost all the time, it probably has learned to output zeros.

The main thing you can do to solve this problem is to train your model using mini-batches, and build them by sampling 0- and 1-observations with equal probability - in other words, attributing greater weight to 1-observations compared to zeros. In that way, your model will be fed with balanced data and will learn to classify them correctly.


Given the files and their parsed content how can I reverse engineer the file format? I have access to a few thousand files binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are stored. What techniques can be used to figure out the format of the files, given this information?

I have had to perform that type of operation on so-called .DAT files. You need a  good hexadecimal editor, that allows you to display char, binary and hex contents from binary, and some patience. And probably a digit-probe friend you can talk with, to help decipher the riddle.

Seek global headers and local ones, file ID, dates and try to locate numbers or ascii code from your database. With that, you can infer field storage length with a relative ease, as least as long they are not exotic, like multiple of 4 bits. Sometimes, there are dummy bits to fill blanks. If the file is somehow compacted (one chunk on 3 bits, the next on 17 bits), it can become quite complicated. Imagine you are a private investigator,  challenge a geek friend. Performing such a task with another person can make it funnier.

I used HxD and HexEd.It.


Why gaussian assumption in GMM-HMM ASR? I am reading a book entitled Speech and Language Processing by Daniel Jurafsky and James.
For Acoustic vector, I would like to know why Gaussian assumption is made?
I searched over the internet and could not find the reason.

One reason Gaussian distributions are assumed is that it simplifies modeling since a Gaussian distribution only has 2 parameters to estimate (i.e., mu and sigma) and the properties of the distribution are well understood.
Gaussian distributions are often a reasonable assumption for acoustic data since acoustic data can modeled as a sum of independent random variables. According to the central limit theorem, the normalized sum of independent random variables  tends toward a Gaussian distribution.


Model biased towards low frequency data? Generally model gets biased towards data_samples/target whose frequency is high in training data set.
Is it possible during training that model gets biased towards low frequency training data set.

With structured data, you have in general 4 challenges:

(1) Missing data

(2) Outliers

(3) Cardinality

(4) Rare values (as a rule of thumb <5%)

Rare values in categorical variables tend to cause over-fitting, particularly in tree based methods. Ph.D. Data Scientist Soledad Galli has an amazing course on the subject (Udemy: "Feature Engineering". Below a screenshot from her course, but to be fair to her, I'm not going to post the solution.


How are data in tensorflow.examples.tutorials.mnist formatted? I am analyzing a sample code that implements a Restricted Boltzmann Machine (RBM) using Tensorflow. The input data seems to be the good old MNIST, except that apparently, it is now available in Tensorflow itself.

So, instead of running this sample code on MNIST, I want to run it on my own data. But the problem is that I do not know how

  tensorflow.examples.tutorials.mnist

has been formatted. Is it like a CSV file? If I want to feed my own data, which comes from Wizemann's horse images, how should I format my data that is compatible with how MNIST has been formatted?

I'm very new to this stuff. So, please enlighten me if I am mistaken or my question is absurd.

Based on the official documentation after loading the data,

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

You can use the following line,

mnist.train.images.shape

to see the shape of images, as you can see it is a rank-2 matrix which means images are unrolled which one dimension specifies the number of examples and another specifies the number of pixels __features.


Cost function for support vector regression The optimisation problem for support vector regression is (see http://alex.smola.org/papers/2003/SmoSch03b.pdf):

minimise:
\begin{align*}
C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \rVert^{2}
\end{align*}

subject to the constraints:

\begin{align*}
& y_{i} - <w,x_{i}> - b \leq \epsilon + \xi_{i} \\
& <w,x_{i}> + b - y_{i} \leq \epsilon + \xi^{*}_{i} \\
& \xi_{i}, \xi^{*}_{i} \geq 0
\end{align*}

I do not understand where the $\lVert w \rVert^{2}$ comes from. I understand how the $\lVert w \rVert^{2}$ is derived in the case of support vector classification (by maximizing the margin), but not in the case of regression.

The paper says that the "goal is to find a function $f(x)$ that has at most $\epsilon$ deviation from the actually obtained targets $y_{i}$ for all the training data, and at the same time is as flat as possible. In other words, we do not care about errors as long as they are less than $\epsilon$, but will not accept any deviation larger than this."

But I am not sure what "flat" means.

Does someone know?

Flat means parallel to the x axis; having a small slope. The smaller w is, the closer f(x) is to b; recall that $f(x) \equiv \left< w, x \right> + b$. One way to think about this is as a form of regularization; the flatter the function, the simpler or more parsimonious it is. This applies to both classification and regression.


Tokenisation with Spacy - how to get tokens to the left/right of token I am using Spacy for text tokenisation and getting stuck with it:

import spacy
nlp = spacy.load("en_core_web_sm")
mytext = "This is some sentence that spacy will not appreciate"
doc = nlp(mytext)

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_, token.shape_, token.is_alpha, token.is_stop)

returns something that seems to me to say that tokenisation was succesful:

This this DET DT nsubj Xxxx True False
is be VERB VBZ ROOT xx True True
some some DET DT det xxxx True True
sentence sentence NOUN NN attr xxxx True False
that that ADP IN mark xxxx True True
spacy spacy NOUN NN nsubj xxxx True False
will will VERB MD aux xxxx True True
not not ADV RB neg xxx True True
appreciate appreciate VERB VB ccomp xxxx True False

but on the other hand

[token.text for token in doc[2].lefts]

returns an empty list. Is there a bug in lefts/rights?

Beginner at natural language processing, hope I am not falling into a conceptual trap. Using Spacy v'2.0.4'.

the token.lefts and token.rights attributes return a generator of the immediate children of the word, in the syntactic dependency parse. It does not just return the tokens on the left and right of the given token.

see : https://spacy.io/api/token#rights

If you want the adjacent tokens for a doc, you can do :

for i in range(len(doc))[1:-1]:
    print(doc[i-1], doc[i+1])

It will print the adjacent tokens for all tokens of the doc, starting at the 2nd token and finishing at the penultimate one.


Single machine learning algorithm for multiple classes of data : One hot encoder I have data of the following kind:

   x1  x2  y
0  0   1  1
1  0   2  2
2  0   3  3
3  0   4  4
4  1   1  4
5  1   2  8
6  1   3  12
7  1   4  16

Is it possible to construct a single machine learning algorithm in python/scikit-learn by defining column x1 in such a way that a simple linear regression should give predict(x1=0, x2=5) = 5 and predict(x1=1, x2=5) = 20. My actual problem has multiple values of x1.

To illustrate the problem better: I have the following code with one hot encoder and it doesn't seem to give the accuracy of training the data separately.

import pandas as pd
from sklearn.linear_model import LinearRegression

# Dataframe with x1 = 0 and linear regression gives a slope of 1 as expected

df = pd.DataFrame(data=[{'x1': 0, 'x2': 1, 'y': 1},
                        {'x1': 0, 'x2': 2, 'y': 2},
                        {'x1': 0, 'x2': 3, 'y': 3},
                        {'x1': 0, 'x2': 4, 'y': 4}
                        ],
                  columns=['x1', 'x2', 'y'])

X = df[['x1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[0, 5]]))) # Output is 5 as expected

# Dataframe with x1 = 1 and linear regression gives a slope of 5 as expected

df = pd.DataFrame(data=[{'x1': 1, 'x2': 1, 'y': 4},
                        {'x1': 1, 'x2': 2, 'y': 8},
                        {'x1': 1, 'x2': 3, 'y': 12},
                        {'x1': 1, 'x2': 4, 'y': 16}
                        ],
                  columns=['x1', 'x2', 'y'])

X = df[['x1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[1, 5]]))) # Output is 20 as expected

# Combine the two data frames x1 = 0 and x1 = 1

df = pd.DataFrame(data=[{'x1': 0, 'x2': 1, 'y': 1},
                        {'x1': 0, 'x2': 2, 'y': 2},
                        {'x1': 0, 'x2': 3, 'y': 3},
                        {'x1': 0, 'x2': 4, 'y': 4},
                        {'x1': 1, 'x2': 1, 'y': 4},
                        {'x1': 1, 'x2': 2, 'y': 8},
                        {'x1': 1, 'x2': 3, 'y': 12},
                        {'x1': 1, 'x2': 4, 'y': 16}
                        ],
                  columns=['x1', 'x2', 'y'])

X = df[['x1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[0, 5]]))) # Output is 8.75
print(reg.predict(np.array([[1, 5]]))) # Output is 16.25

# use one hot encoder

df = pd.get_dummies(df, columns=["x1"], prefix=["x1"])
X = df[['x1_0', 'x1_1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[1, 0, 5]]))) # Output is 8.75
print(reg.predict(np.array([[0, 1, 5]]))) # Output is 16.25

How can I use pandas and sklearn for the combined data to get the same accuracy using one machine learning model?

I know this is late, but the question got bumped by Community, so...

Your discussion and data suggest that what you want to produce is the model
$$y = (3x_1+1)x_2,$$
but that is not a linear model and so linear regression will not find it.  You can try any number of other nonlinear models, the best type depending on your real use-case.  For instance,

If you really just want a linear model for each value of $x_1$, then it's probably best just to split the data along $x_1$ as you started with.  You might need to examine your motivation behind wanting just "one machine learning model".
You could introduce a new feature, equal to $x_1 x_2$.  This loses some information from option (1) when $x_1$ has more than two values, but it might be suitable.
You could use a tree-based model with linear regression done at each leaf, as in  https://stats.stackexchange.com/questions/78563/regression-tree-algorithm-with-linear-regression-models-in-each-leaf .  If the model fitting procedure decides to split only on $x_1$ and regress only on $x_2$, it mostly recovers option (1), though there's no reason to necessarily expect that.


Combining Datasets with Different Features I have multiple datasets, with slightly differing features. What tools can I use to make this a homogeneous dataset?

Dataset1:

featureA,featureB,featureC
1,7,3
4,8,4

Dataset2:

featureA,featureC,featureD,featureE
3,4,5,6
9,8,4,6

Homogeneous Dataset

featureA,featureB,featureC,featureD,featureE
1,7,3,,
4,8,4,,
3,,4,5,6
9,,8,4,6

You can use R to do that.

The smartbind function is the perfect way to combine datsets in the way you are asking for:

library(gtools)

d1<-as.data.frame(rbind(c(1,7,3),c(4,8,4))))
names(d1)<-c("featureA","featureB","featureC")

d2<-as.data.frame(rbind(c(3,4,5,6),c(9,8,4,6)))
names(d2)<-c("featureA","featureC","featureD","featureE")

d3<-smartbind(d1,d2)


Improve classification accuracy on Iris data set - tf.contrib.learn Quickstart tutorial I run through the Quickstart tutorial of the tf.contrib.learn, which uses the Iris data set. After training the DNN model I get an accuracy of ~0.967.

In an attempt to improve the accuracy, I tried to change the number and size of the hidden layers, I doubled the steps, and I changed optimizer and learning rates. However, the accuracy does not increase (or decrease).

Could that might be a dataset limitation?

Any ideas on how to improve the performance? I want to be able to dive into a NN and try to optimize it, before I go blindly into another tutorial.

An accuracy of 0.967 is a good score if you achieved it on the test set, so make sure that you are talking about the test set here. Secondly, increasing the number of iterations on the dataset won't do anything if your solution has already converged. Thirdly, adding more layers is not a very good idea, since you are already using a DNN classifier and could lead to overfitting. That being said, try to caculate a cross_validation score for your dataset, as then run it on a test set. In corss-validation you basically train your model on different chunks of your dataset, and validate on the chunk you leave out, then test it on a separate test set. This way you can actually see how your hyperparameters are behaving and you won't have to guess. Once you calculate the test score this way, it means you have good values of the parameters. If you still don't feel satisfied with the accuracy, its time to try a different algorithm, using more units in each layer and trying different activation functions.


Using time series data from a sensor for ML I have the following data for a little side project. It's from an accelerometer sitting on top of a washer/dryer and I'd like it to tell me when the machine has finished.

x is the input data (x/y/z movement as one value), y is the label on/off

Because the x values overlap for y=1 and y=0, I was thinking of using x and a rolling 3 minute window as inputs for a SVM:

xyz60=res.xyz.resample("60S").max()
X["x"]=xyz60
X["max3"]=xyz60.rolling(window=3, min_periods=1).max()

Is this a good approach for this kind of problem? Are there alternatives that might produce better results?

You have time series data which is used to measure the acceleration. You which to identify when the machine is in its nominal state (OFF) and anomalous state (ON). This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem.

Preparing you data

All of the methods will rely on the feature extraction method you select. Assuming we continue to use the 3 sample time window as you suggested. In this algorithm you will calculate a statistic for this nominal state $y = 0$. I would suggest the mean as I assume you are already doing, take the average of the three sample resultant accelerations. You will then be left with a large number of values in a training set $S$ defined as

$S = \{s_0, s_1, ..., s_n \}$

where $s$ is the mean of the tree samples in a window. $s$ is defined as

$s_i = \frac{1}{3} \sum_{k=i-2}^{i} x_k$

where $x$ is your sample observations and $i\geq2$.

Then collect more data if it is possible with the machine active such that $y = 1$.

Now you can choose if you want to train your algorithm on a one-class dataset (pure anomlay detection). A biased dataset (anomaly detection) or a well-balanced dataset. The balance of the dataset is the ratio between the two classes in your dataset. A perfect dataset for a 2-class classifier would be 1:1. 50% of the data belonging to each class. You seem to have a biased dataset, assuming you don't want to waste a lot of electricity.

Do note that there is nothing stopping you from keeping the neighboring samples split as an instance in your dataset. For example:

$x_i$ $x_{i-1}$ $x_{i-2}$ | $y_i$

This would make a 3-dimensional input space for a specific output which is defined for the currently taken sample.

A Biased Dataset

Easy Solution

The easiest way that i would suggest. Assume you are using a single statistic to define what is happening throughout the 3 sample window. From the collected data get the maximum $s$ of your nominal points ($y=0$) and the minimum $s$ of your anomalous points ($y=1$). Then take the halfway mark between these two and use that as your threshold.

If a new test sample $\hat{s}$ is larger than the threshold then assign $y=1$.

You can extend this by calculating the mean $s$ for all of your nominal samples $y=0$. Then calculate the mean for your anomalous samples $y=1$. If a new sample falls closer to the mean of the anomalous samples then classify it as $y=1$.

But I want to get fancy!

There are a number of other techniques you can use to do this exact task.

k-Nearest Neighbors
Neural Networks
Linear Regression
SVM

Simply put, almost every machine learning algorithm is well suited for this purpose. It just depends on how much data is available to you and it's distribution.

I really want to use SVM

If this is the case keep the three samples completely separated. Your training matrix will have 3 columns as discussed above. And then you will have your outputs $y$. Using SVM in python is very easy: http://scikit-learn.org/stable/modules/svm.html.

from sklearn import svm

X = [[0, 0, 0], [1, 1, 1], ..., [1, 0, 1]]
y = [0, 1, ..., 1]
clf = svm.SVC()
clf.fit(X, y)

This trains your model. Then you will want to predict the outcome for a new sample.

clf.predict([[2., 2., 1]])


Best algorithm to create segment in case of categorical variables I have a SampleData (2653 observation, 11 features) of bank transaction done in the 1-month timeframe. Download Dataset Size=250KB

I want to come up with algorithms (Single or Combine) that can segment users into different categories.

Since there are mostly categorical features involved except tx_amount, Which algorithm is best suited here, or how should I approach this problem to create user segments?

Do not approach this by using whatever algorithm you can get to run.

There is no use in just doing something that you don't understand.

Instead, you will first need to identify what is a good way to group data that solves your problem. Then you can begin looking for an algorithm that helps solving your problem (and not some useless other problem).


'PCA' object has no attribute 'explained_variance_' Elbow Method - Finding the number of components required to preserve maximum variance.

My code:

pca = decomposition.PCA()

vectorizer = TfidfVectorizer(min_df=10)
preprocessed_essay_tfidf = vectorizer.fit_transform(preprocessed_essay)
pca_data = preprocessed_essay_tfidf
pca.n_components = 3000

percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);
cum_var_explained = np.cumsum(percentage_var_explained)

I get the attribute error:

AttributeError: 'PCA' object has no attribute 'explained_variance_

Why is this?

I can find explained_variance_ present here.

PCA is an estimator and by that you need to call the fit() method in order to calculate the principal components and all the statistics related to them, such as the variances of the projections en hence the explained_variance_ratio.
pca.fit(preprocessed_essay_tfidf) or pca.fit_transform(preprocessed_essay_tfidf)


How to train a Text Based data for a Machine Learning problem? I am working on a project for displaying products to customer by context, based on a search query. For example, I don't want customers to have to enter a specific product name, instead searching based on functionality (e.g., "walls do not heat much" would return product names such as "Whirlpool NEO IC355 ROY 3S 340 L Double Door Refrigerator")
I have a training set comprised of the functionality associated products. I am planning to use Logistic Regression to train a model on these data. How do I process this data in Python or extract features to feed into logistic regression? I have heard of "Bag of words model", but not sure how to use this, or is it even applicable here?

I know there are plenty of NLTK libraries available. But, I want to implement it from the scratch or using minimum external libraries possible.

Please help or are there any resources to refer to?

Your problem looks more like a ranking problem than a classification problem to me. Have you tried a more naive method, like a 1-NN "classifier" with unigram text representation, Tf-Idf term weighting and a cosine similarity metric? It's far from the state of the art but it tends to give rather good results in retrieval and recommendation tasks.


What model to use for matching two datasets I've got 500 images of paper receipts scanned and OCR as one dataset. I also have a dataset of transactions from my credit card statement including amount and date.

What model is best to match the receipts to the transactions?

Let the credit card statement be the ground truth, and the receipts be the noisy inputs. For a given line item, find the receipt with the smallest distance. If the distance is small enough, declare a match. This is a threshold you will have to experimentally determine. You can let the distance be the sum of the distances for the amount and date. A heuristic for these individual distances is simply the edit distance. A more sophisticated approach would be to model the OCR error using labeled data to determine the most likely input; print something with known text in a similar typeface to the receipts so you can learn which characters are commonly confused with one another, and thereby estimate the most likely input sequence. Going another step further, you could jointly model the density between price, store, and item, so you can recognize for example that a vegetable bought at your local grocery store probably does not cost \$100 but \$1.00

If your OCR software gives you confidence estimates over the characters and possible guesses, you can use that too.


It is hard to see where the problem is with your code; given your comments, my guess is a problem in the timezone conversion.

I cannot see a problem exactly, buy my suggestion to help debug your situation would be to convert all time to timestamps - by default that will be seconds since the epoch (January 1st 1970). These are then simply normal float64 values (I convert to integer microseconds below). If your differences are still returning zero, subtracting simple number, then the problem cannot be in the differences.

Here is a minimal working example. A simple dateframe with timestamps with millisecond frequency:

import pandas as pd
from datetime import datetime

In [1]: df = pd.DataFrame(pd.date_range(start=datetime(2016,1,1,0,0,1),
    ...:     end=datetime(2016,1,1,0,0,2), freq='ms'), columns=["dates"]).head(10)    # just take first 10 rows for simplicity

Make a new column, converting the dates into microseconds since the epoch, as an integer:

In [2]: df["timestamps"] = df.dates.apply(lambda x: int(datetime.timestamp(x) * 1e6))

In [3]: df
Out[3]:
                    dates        timestamps
0 2016-01-01 00:00:01.000  1451602801000000
1 2016-01-01 00:00:01.001  1451602801001000
2 2016-01-01 00:00:01.002  1451602801002000
3 2016-01-01 00:00:01.003  1451602801003000
4 2016-01-01 00:00:01.004  1451602801004000
5 2016-01-01 00:00:01.005  1451602801005000
6 2016-01-01 00:00:01.006  1451602801006000
7 2016-01-01 00:00:01.007  1451602801007000
8 2016-01-01 00:00:01.008  1451602801008000
9 2016-01-01 00:00:01.009  1451602801009000

You could at this point check that there are no duplicates in any of your columns, using:

In [4]: df.T.duplicated()
Out[4]:
dates         False
timestamps    False
dtype: bool

If there are duplicates, that could be the cause of differences equal to zero.

Now compute the differences, in my case all 1-millisecond differences (1000 microseconds):

In [5]: df[["date_diffs", "timestamp_diffs"]] = df.diff(1)

In [6]: df
Out[6]:
                    dates        timestamps      date_diffs  timestamp_diffs
0 2016-01-01 00:00:01.000  1451602801000000             NaT              NaN
1 2016-01-01 00:00:01.001  1451602801001000 00:00:00.001000           1000.0
2 2016-01-01 00:00:01.002  1451602801002000 00:00:00.001000           1000.0
3 2016-01-01 00:00:01.003  1451602801003000 00:00:00.001000           1000.0
4 2016-01-01 00:00:01.004  1451602801004000 00:00:00.001000           1000.0
5 2016-01-01 00:00:01.005  1451602801005000 00:00:00.001000           1000.0
6 2016-01-01 00:00:01.006  1451602801006000 00:00:00.001000           1000.0
7 2016-01-01 00:00:01.007  1451602801007000 00:00:00.001000           1000.0
8 2016-01-01 00:00:01.008  1451602801008000 00:00:00.001000           1000.0
9 2016-01-01 00:00:01.009  1451602801009000 00:00:00.001000           1000.0

Add a sample zero difference and retrieve the index of zero differences:

In [7]: df.iloc[3, 3] = 0.0

In [8]: np.where(df == 0)
Out[8]: (array([3]), array([3]))

Hopefully that will be enough to find where there could actually be zero difference. If all of that works out without zero differences, I would either look into your timezone conversion (maybe they do some rounding there?) or report a bug to the pandas issues

EDIT

After your update trying my debugging method, and seeing this:

I believe there must be either a bug or an inherent limitation to your system.

Pandas Bug

A bug might be in your specific version of Pandas and its df.diff() method. Check your version of Pandas with pd.__version__ and look on the issues page I linked above for any clues ... maybe just try the latest stable version anyway.

32-bit system

Another possible solution would be that you are running on a 32-bit system and so could actually lose the precision required for my example above.
A 32-bit integer can only retain precision for 10 digits. My timestamps above require 12 digits. You can find out on like this, on Linux or Windows or Mac.

Additionally, you could retry my example, but only look at seconds instead of microseconds, just to be sure.


How to select features which have non zero importance using SKLearn's Decision Trees Classifier? I need to discard all the features which have zero importance and keep only those features which have non zero importance while implementing DecisionTreesClassifier. The feature importance here is determined using clf.feature_importance_ parameter of DecisionTreesClassifier. How can this be done?

You should use a sklearn.feature_selection.SelectFromModel instance: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html

You can specify the threshold that you want when you want to select features. You can add this transformer in a scikit-learn pipeline as well.


How do I use rnn to forecast to n periods with limited data? So this is my 1st time trying to run a small time-series dataset through an RNN, but after a lot of searching, I haven't been able to find,

1. How I can use this to forecast to n periods ? (like in the model.predict(start, end) function in ARIMA.)

2. Is there a better way of doing this using NN ?

details on the data is given as comments in the code below. thank you.

'''
total timeseries data points = 39
frequency = Months
train = 36
test  = 3
Need to forecast 3 periods ahead upto 42, currently using forecast horizon as 1
Limited data so used all in a single batch
'''

periods    = 35
f_horizon  = 1
batch_size = 35

x_batches  = dfp1[ : (len(dfp1) - (len(dfp1) % batch_size))].reshape(-1, periods, 1)          # train data for t   periods
y_batches  = dfp1[1 : len(dfp1) - (len(dfp1) % batch_size) + f_horizon].reshape(-1, periods, 1) # train data for t+1 periods

#defining test data set
def test_data(testdata):
    test_x = testdata[ :(len(testdata)-1)]   # t
    test_y = testdata[1:len(testdata)]       # t+1
    return test_x, test_y

test_x, test_y = test_data(dfp3[-4: ]) #since 35 points were used in training, used the last 4 points for testing

outputs    = 1
inputs     = 1
hidden     = 100

with tf.device('/cpu:0'):
    with tf.variable_scope('var', reuse = tf.AUTO_REUSE):
        x  = tf.placeholder(tf.float32, [None, periods, inputs])
        y  = tf.placeholder(tf.float32, [None, periods, outputs])

        #creating a basic rnn cell
        basicrnn           = tf.nn.rnn_cell.BasicRNNCell(num_units = hidden, activation = tf.nn.relu)
        rnn_output, states = tf.nn.dynamic_rnn(basicrnn, x, dtype = tf.float32)

        stacked_rnn_output = tf.reshape(rnn_output, [-1, hidden])
        stacked_output     = tf.layers.dense(stacked_rnn_output, outputs)
        outputs_           = tf.reshape(stacked_output, [-1, periods, 1])

        #using mape for loss fn
        loss               = tf.reduce_mean(tf.abs(tf.divide(tf.subtract(outputs_, y), y))) * 100

        learningrate       = 0.001
        training_op        = tf.train.AdamOptimizer(learning_rate = learningrate).minimize(loss)
        init               = tf.global_variables_initializer()

        epochs = 1501

        with tf.Session() as sess:
            init.run()

            for i in range(epochs):
                sess.run(training_op, feed_dict = {x: x_batches, y: y_batches})

                if i%100 == 0:
                    mape   = loss.eval(feed_dict = {x: x_batches, y: y_batches})

                    print("ep %s: %s"%(i, mape))

            '''
            unable to pass the 4 test data points since input size is 35,
            can I reshape x & y before I pass test input, would it alter the model ?
            also why is the loss fluctuating, and not steadily decreasing.
            '''

            #y_pred = sess.run(outputs_, feed_dict = {x: test_x})
            #print(y_pred)

there are 3 ways you can do this in NN -

1) take forecasted values ( horizon 1) and pass this again to built model to forecast next values ( horizon 2) and so on.

2) build a model with lagged values and have multiple output values( t, t+1, T+2... t+n). You need to decide window for lagged values.

3) use LSTM.


I have a pandas dataframe and i need to clear all the special characters other than space Input:

import pandas as pd
df=pd.read_excel("OCRFinal.xlsx")
df['OCR_Text']=df['OCR_Text'].str.replace(r'\W+'," ")
print(df['OCR_Text'])

Output:

The excel removes all the special characters along with the space. But i dont want space characters to be removed

import pandas as pd
df=pd.read_excel("OCRFinal.xlsx")

whitespace = "\r\n\t"

df['OCR_Text']=df['OCR_Text'].apply(lambda x: x.strip(whitespace))
print(df['OCR_Text'])


how to export the tables into a csv file pandas The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable.

for row in categorical:
    for col in numeric:
        ptable = pd.pivot_table(df, values = col, index = row, aggfunc = ['min','max','median','mean','std',lambda x: 100*x.count()/df.shape[0]])
        print(ptable)
        writer = pd.ExcelWriter('report.xlsx')
        ptable.to_excel(writer, 'Sheet1')
        writer.save()

It displays the output as in the image:

but this is not a data frame and when writing into an excel file it displays only the last iteration values.

how do I get all the iterated tables into the excel file or separate excel files?

Your current code overwrites the previous sheet, which is why only the last iteration is present. Setting each sheet to the same name (Sheet1) will overwrite the sheet. This name will need to be changed for each interation.

Try this:

for row_index, row in enumerate(categorical):
    for col_index, col in enumerate(numeric):
        ptable = pd.pivot_table(df, values = col, index = row, aggfunc = ['min','max','median','mean','std',lambda x: 100*x.count()/df.shape[0]])
        print(ptable)
        writer = pd.ExcelWriter('report.xlsx')
        ptable.to_excel(writer, 'Sheet1_{}_{}'.format(row_index, col_index))
        writer.save()

This should save each iteration as an individual sheet


Isolation forest - grouped by I'm trying to use isolation forest algorithm for outliers detection.
Data has 2 columns: id and REV. Below code gives me ungrouped result.
Could you pls advise how to get result grouped by first column (id)?

data= pd.read_excel (my_path)
outliers_fraction=0.1
scaler = StandardScaler()
np_scaled = scaler.fit_transform(data)
data = pd.DataFrame(np_scaled)
model =IsolationForest(contamination=outliers_fraction)
model.fit(data)
data['anomaly'] = pd.Series(model.predict(data))
print(data)

I have 2 columns: id and REV. I added a picture of what I expect to see as the final result:

Tried to use function, but have half of data NaN. Any thoughts please?

def groupreg(g):
        outliers_fraction = 0.1
        # train isolation forest
        model = IsolationForest(contamination=outliers_fraction)
        model.fit(g)
        g['anomalyX'] = pd.Series(model.predict(g))
        return g

    df1=data2.groupby('id').apply(groupreg)
    print(df1)

Here is solution if somebody will need it someday

def groupreg(df):
    a=df[['EUR']].values
    # train isolation forest
    model = IsolationForest()
    model.fit(a)
    return pd.Series(model.predict(a))

result=df.groupby('id').apply(groupreg)
#print(result)

#print(isolationforest(data2))
df['outlier1'] = result.tolist()
print(df)


'int' object is not iterable using panda python Here I want to calculate time interval in between row by row in time column import from csv file. my start time 6.00 a.m and my end time is next day 6.00 a.m. In between this time periods how to find time interval period. Can anyone help me to solve this problem?
Here I upload my code and csv file.

def time_diff(start, end):
start.append(pd.to_datetime(data['time'][0],formt = '%H:%M:%S').time())
end.append(pd.to_datetime(len(data['time']), format='%H:%M:%S').time())
if isinstance(start, datetime_time): # convert to datetime
    assert isinstance(end, datetime_time)
    start, end = [datetime.combine(datetime.min,i) for t in [start, end]]
if start <= end:
    return end - start
else: # end < start
    end += timedelta(1) # +day
    assert end > start
    return end - start
for time_range in range (len(data['time'])):
start = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
end = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
print(time_diff(s, e))
assert time_diff(s, e) == time_diff(s.time(), e.time())

my csv file :-

I got an error:-

TypeError                                 Traceback (most recent call last)
 <ipython-input-28-be40ae41d284> in <module>()
 12         return end - start
 13 for time_range in range (len(data['time'])):
 ---> 14     start = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
 15     end = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
 16     print(time_diff(s, e))

 TypeError: 'int' object is not iterable

for time_range in range (len(data['time'])):
start = [datetime.strptime(t,'%H:%M:%S') for t in time_range]

time_range is an integer due to using range(int). There is nothing to iterate using an int. It is not a container.


Is there a way to get gini index values for every node in rpart model? df <- tibble(x=factor(c("A", "B")), y=factor(c(1, 0)))
model <- rpart(formula=y~., data=df, method="class", control=rpart.control(minsplit=2))

Here model would have 1 parent and two child nodes. How to get gini index values for these nodes from rpart model object?

Gini impurity can be calculated as $1-p_{1}^2-p_{2}^2$ for each node. For example, if node 1 contains 40% '1' and 60% '0', gini = 1 - 0.4^2 - 0.6^2. The information of node size n, number of '0' dev are stored in model$frame. The Gini for each node could be calculated with node size n and number of '0' dev in model$frame:
frame <- model$frame
frame[['gini']] = 1 - (frame[['dev']] / frame[['n']])^2 - (1 - frame[['dev']] / frame[['n']])^2

frame[,c('var','n','dev','gini')]
>      var  n dev      gini
> 1     x3 10   5 0.5000000
> 2 <leaf>  4   1 0.3750000
> 3 <leaf>  6   2 0.4444444

The Gini improvment for each split is calculated by weighted difference between parent and children nodes.
frame[['improve']] = NA
for (i in 1:nrow(frame)) {
  if (frame[i,'var'] == '<leaf>') next

  ind = which(rownames(frame) %in% (as.numeric(rownames(frame)[i])*2+c(0,1)))
  frame[i,'improve'] = frame[i,'n']*frame[i,'gini'] - frame[ind[1],'n']*frame[ind[1],'gini'] - frame[ind[2],'n']*frame[ind[2],'gini']
}

frame[,c('var','n','dev','gini','improve')]
>      var  n dev      gini   improve
> 1     x3 10   5 0.5000000 0.8333333
> 2 <leaf>  4   1 0.3750000        NA
> 3 <leaf>  6   2 0.4444444        NA

#comparing with
model$splits
>    count ncat   improve index  adj
> x3    10    2 0.8333333     1 0.00
> x2    10    2 0.2380952     2 0.00
> x2     0    2 0.7000000     3 0.25


What's the difference between setting negative label as 0 or -1 in binary classification? I would like know what's the difference between setting the negative labels as 0 or -1 in the binary classification. I guess there is no important difference, just some notation changes in the model.
Thank you in advanced!

This will depend on the algorithm/code that you are using.
You are right that this is often not an important difference but that doesn't mean it can't create errors if you are using code which expects one or the other without being aware of it.
You should equally keep track of data types such as integers vs floats which will similarly produce different outcomes in lots of data science tools such as in Python or R.
Since lots of classifiers implemented in packages like sklearn will specify whether they are for continuous/discrete outcome spaces (classification vs regression), you shouldn't have to worry about this.
To illustrate this for sklearn:

source: https://scikit-learn.org/stable/_static/ml_map.png


predict future sale from past 3 month data I have the 3-month sale(branch wise) and I want to predict next month sale. There are huge differences between total sale in between months. The simple average of the month's to predict next month sale is not valuable. delta(change sale in the month) between months is important and take into account.
Any help, tips, pointers, recommendations, or directions would be extremely helpful and I would be eternally grateful!

The Facebook Prophet API might be the place to start. I imagine that the data is daily.

This package is avaiable either for R or Python, and it's very simple to use. You can get predictions easily, and if you have some knowledge about Time Series analysis you can tune some parameters to best fit your solution.

From the link above:

  Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.

Here and Example with R.


Is there any advantage in using Particle Swarm Optimization for clustering than K-Means? I have read some paper about using particle swarm optimization. It doesn't look give much different result than K-Means. I tried to use PSO for clustering but the result is pretty much the same with K-Means with some drawbacks like longer execution time and have a lot of different result caused by the random factor.

K-means makes locally the optimal decision. Most of the time (in particular when this objective works reasonably well at all) this works quite well to find a good local optimum. I doubt that using an approach such as PSO gives you any advantage here - the problem is just too simple, and k-means has the speed advantage, while PSO is unlikely to find other optima.
And in particular the better k-means algorithms such as Hamerly, Elkan, ... are so fast, they'll be able to run with hundreds of random restarts in the time needed for just one PSO.


How to train an existing word2vec gensim model on new words? According to gensim docs, you can take an existing word2vec model and further train it on new words.

  The training is streamed, meaning sentences can be a generator,
  reading input data from disk on-the-fly, without loading the entire
  corpus into RAM.

  It also means you can continue training the model later:

        model = Word2Vec.load("word2vec.model")
        model.train([["hello", "world"]], total_examples=1, epochs=1)

Source: docs

But when I actually try it, it doesn't seem to learn the new terms.

from gensim.models import Word2Vec

# initial a model
model = Word2Vec([["cat", "say", "meow"], ["dog", "say", "woof"]], min_count=1)

# count terms in model
print( len(model.wv.vocab) )

=> 5

# train existing model on new terms
model.train([['potoatoes', 'and', 'farmers']], total_examples=model.corpus_count, epochs=model.epochs)

# count terms in model
print( len(model.wv.vocab) )

=> 5

After adding new terms in the 2nd code block, the model still only has the same number of terms as before.

How can I make this work?

Add this line before training with the new terms.

model.build_vocab([['potoatoes', 'and', 'farmers']], update=True)

After training,

print(len(model.wv.vocab))
=> 8


Storing large data sets for python machine learning algorithm consumption I'm reading up on how to clean/munge/wrangle data sets in order to run machine learning algorithms on them. Lots of info on how to do the actual wrangling, but a practical detail seems to be glossed over: storage.

My question is quite simple: which is the go-to technology to store/retrieve a large data set in order to run algorithms on it in the most convenient/efficient way possible?

I'm guessing the language in which the algorithms are written is not all that relevant here.

There are a lot of technologies out there to handle. These are the most popular ones I know of:

For moderate data to operate in chunks- Pandas -  pd.read_csv('train.csv', chunksize=chunksize)

For larger data - dask, Hadoop with R

Refer this for various other suggestions.


How to reduce an unknown size data into a fixed size data? Please read details Example:
Given n number of images marked 1 to n where n is unknown, I can calculate a property of every image which is a scalar quantity. Now I have to represent this property of all images in a fixed size vector (say 5 or 10).

One naive approach can be this vector- [avg max min std_deviation]
And I also want to include the effect of relative positions of those images.

My approach would be using the Moments.
Using moments is a cool way trowing any arbitrary set of points into space, and it is used for both probability and mass calculations.
Since you have not stated what are you going to do with this vector representation I guess that a generalized "shape" like representation might work for you.

If you are using Python, then this can help.


Google Sheets: how to find max value in column B, corresponding values in column A, and max among these I have two numeric columns, A and B. I want to find the max value in column B, which will return multiple rows. Then I want to find the max value in column A from among these rows.

A   B
 5  315
 7  315
10  275

The function should return:

 7  315

It would be fine if this were a combination of two functions - one that finds the max value in B (315), and another that finds the corresponding max value in A (7). That's how I'm trying it right now using:

=index(A2:B10,match(max(B2:B10),B2:B10,0))

But this solution only finds the latest row that has column B's max value:

 5  315

Someone suggested I use the query function, but I couldn't get that to work either.

Suppose you have data in the columns A, B from row 2 to 4. Put MAX(B2:B4) somewhere (in this example, I put it in B6). Then for the corresponding max in A you can use the formula MAX(ARRAYFORMULA(IF(B2:B4 = $B$6,A2:A4))).


What does Logits in machine learning mean? "One common mistake that I would make is adding a non-linearity to my logits output."

What does the term "logit" means here or what does it represent ?

Logits interpreted to be the unnormalised (or not-yet normalised) predictions (or outputs) of a model. These can give results, but we don't normally stop with logits, because interpreting their raw values is not easy.

Have a look at their definition to help understand how logits are produced.

Let me explain with an example:

We want to train a model that learns how to classify cats and dogs, using photos that each contain either one cat or one dog. You build a model give it some of the data you have to approximate a mapping between images and predictions. You then give the model some of the unseen photos in order to test its predictive accuracy on new data. As we have a classification problem (we are trying to put each photo into one of two classes), the model will give us two scores for each input image. A score for how likely it believes the image contains a cat, and then a score for its belief that the image contains a dog.

Perhaps for the first new image, you get logit values out of 16.917 for a cat and then 0.772 for a dog. Higher means better, or ('more likely'), so you'd say that a cat is the answer. The correct answer is a cat, so the model worked!

For the second image, the model may say the logit values are 1.004 for a cat and 0.709 for a dog. So once again, our model says we the image contains a cat. The correct answer is once again a cat, so the model worked again!

Now we want to compare the two result. One way to do this is to normalise the scores. That is, we normalise the logits! Doing this we gain some insight into the confidence of our model.

Let's using the softmax, where all results sum to 1 and so allow us to think of them as probabilities:

$$\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} \hspace{20mm}   for \hspace{5mm} j = 1, …, K.$$

For the first test image, we get

$$prob(cat) = \frac{exp(16.917)}{exp(16.917) + exp(0.772)} = 0.9999$$
$$prob(dog) = \frac{exp(0.772)}{exp(16.917) + exp(0.772)} = 0.0001$$

If we do the same for the second image, we get the results:

$$prob(cat) = \frac{exp(1.004)}{exp(1.004) + exp(0.709)} = 0.5732$$
$$prob(dog) = \frac{exp(0.709)}{exp(1.004) + exp(0.709)} = 0.4268$$

The model was not really sure about the second image, as it was very close to 50-50 - a guess!

The last part of the quote from your question likely refers to a neural network as the model. The layers of a neural network commonly take input data, multiply that by some parameters (weights) that we want to learn, then apply a non-linearity function, which provides the model with the power to learn non-linear relationships. Without this non-linearity, a neural network would simply be a list of linear operations, performed on some input data, which means it would only be able to learn linear relationships. This would be a massive constraint, meaning the model could always be reduced to a basic linear model.
That being said, it is not considered helpful to apply a non-linearity to the logit outputs of a model, as you are generally going to be cutting out some information, right before a final prediction is made. Have a look for related comments in this thread.


Predicting Composition of Chemical Compounds I have a dataset which has names of compounds and their compositions. Like below

Sulphuric Acid=>[H,S,O] (Hydrogen, sulphur, oxygen)
Oxalic Acid=>[H,C,O]
Sodium Oxalate=>[Na,C,O]
Potassium Sulphate=>[K,S,O]
...

Now I would need to train a model which can tell me Sodium Sulphate as [Na,S,O]. Note that sodium sulphate represents something not in training dataset.
I have tried searching for possible ideas but nothing came up. Then I thought it could be hierarchical clustering like

 Sodium-Oxalate
    |    |
Oxalate  Sodium
|  |      |
C  O      Na

But then in hierarchical clustering the base/leafs are different. But here they are shared. Its like a graph. So What machine learning algorithm can help? Any other clustering? NLP/word clustering (if yes How)?

Another approach I could think of is like Word2Vec where I generate embeddings for each word (C,H,Sodium), all will have embeddings. And based on what is closer to the word I am asking I will give output. But this needs huge amount of data. I only have around 1k common compounds. And the approach won't generalize to any problem like this with less data.

Try a one-hot encoding of the elements in your training set (H, C, I etc.) and the same for the chemical descriptors ('acid', 'oxalic' etc.). Then try feeding the data to a simple feed-forward neural network, mapping the one-hot encodings of the descriptors as your x_train and x_val to the chemical elements, which are your y_train and y_val.

For the encodings, take a look at the Scikit learn encoders.

I think such a simple approach may work, as the chemical names are logically named, so any experienced chemist knows exactly which elements to expect in a compound, given its name. Your problem doesn't require further information in the predictions, e.g. the actual chemical makeup, like $CH_3CH_2OH$ (ethanol).

Regarding a small amount of data: look into something like K-fold cross validation. Using this, you select some portion of the data to be your validation data, and train the model. You then repeat this process, selection a different portion of the data. This will help make the most of a limited dataset, although it may introduce overfitting because your model will eventually have seen all the data!

Here is a schematic of this method:

Scikit Learn has a class that implements it for you.


Creating a valid dataset for obtaining results I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my dataset. For example, python related subreddits, stack exchange posts tagged with python, twitter posts hashtagged with python or python specific sites.
The data points taken from these places are considered positive data points and then I have retrieved data points from the same sources but relating to general topics, searched if they contain the word python in them and if they do discard them to create the negative examples in my dataset.
I have been told that I can use the training set from the dataset as is, but that I need to manually annotate the test set for the results to be valid, otherwise they would be biased. Is this correct? How would they be biased? To be clear the test set contains different entries to the training set.
There are close to 200,000 entries in the test set which makes manual annotation difficult. I have seen similar methods been used in papers I have previously read without mention of manual annotation. Is this technique valid or do I have to take some extra steps to ensure the validity of the test sets?

There are two potential biases:

With this automatic method, you might have a few erroneous labels. For example it happens regularly here on DataScienceSE that a user tags a question "python" but actually the question is not specific to python at all. Same thing for the opposite case: for example it's possible that some content contains some python code but doesn't mention "python" anywhere.
The distribution between positive and negative classes is arbitrary. Let's assume you use 50% positive / 50% negative: if later you want to apply your classifier on a new data science website where only 10% of the content is about python, it's likely to predict a lot of false positive cases so the true performance on this data will be much lower than on your test set.

It's rare to have a perfect dataset, so realistically in my opinion the first issue is probably acceptable because the noise in the labels should be very limited. The second issue could be a bit more serious but this depends on what is the end application. Keep in mind that a trained model is meant to be applied to the same kind of data as it was trained/tested on.


How to select features for a ML model I have a dataset with 5K records for binary classification problem.

My features are min_blood_pressure, max_blood_pressure, min_heart_rate, max_heart_rate etc. Similarly, I have more than 15 measurements and each of them have min and max columns amounting to 30 variables.

When I ran correlation on the data, I was able to see that these input features are highly correlated. I mean min_blood_pressure is highly correlated (>80%) to max_blood_pressure. Each measurement with its min and max feature is highly correlated. Though their individual correlation to target variable is less.

So in this case, which one should I drop or how should I handle this scenario?

I guess there is min and max variables for a reason. How would you do in a situation like this?

Should we find the average of all the measurements and create a new feature?

Can anyone help me with this?

I'd start here. Most basic idea is to run statistical tests to see how target variable depends on each feature. These include tests like chi-square or ANOVA.
Tree-based models can also output feature importance. Check this post. There's plenty of posts on kaggle with code. Might be worth checking those:

https://www.kaggle.com/willkoehrsen/introduction-to-manual-feature-engineering
https://www.kaggle.com/rejasupotaro/effective-feature-engineering
https://www.kaggle.com/willkoehrsen/automated-feature-engineering-tutorial

As your data set isn't so drastically large, you could push grid search and check how your model behaves for different factors of PCA.

It's hard to tell a priori whether you should drop some features. I guess trying each combination of 30 features is completely out of scope, though you might try dropping most redundant ones.

As your data contains categorical features, it might be good idea to give catboost a try. They claim it handles categorical features better than other gradient boosters. Just keep in mind, that default number of estimators is 10 times of that in xgboost. You might lower it for experiments.

First, I'd create base model with all the features. Now comes the question: which method to choose? Gradient boosters poses ability of learning the feature importance, those redundant ones will get little weight and you might not see much of an improvement, when dropping features. You might get more insight using more vanilla methods, but in the end you'll be certainly deploying gradient boosting to production, so I don't see much sense in it. I'd stick with xgboost or catboost and perform experiments using same parameters.

Please keep in mind: though some features might be highly redundant, they may still contribute some knowledge to your model.


merge 2 dataframe with Memory Error trying to merge 2 data frame size with (13647309, 48)
and I'm using 32 memory.

df_train = train.merge(train_lag,on=['ncodpers','int_date'], how='left')

after I run this it takes too much memory.
is there a way to minimize memory usage when merging?

The problem is that when you merge two dataframes, you need enough memory for both of them, plus the merged one. There is a workaround from a stackoverflow answer

What you can do is to read the first dataframe only (the smaller one) and then read the second in batches.

def preprocess(x):
    df = pd.merge(df_train, x, on=['ncodpers','int_date'], how='left')
    df.to_csv("final.csv", mode="a", header=False, index=False)

reader = pd.read_csv("train_lag.csv", chunksize=1000)

for r in reader:
    preprocess(r)