When optimizing the MSE, the correlation between prediction and target increases? After optimizing the MSE (mean squared error) in a regression task, how is the change in Pearson correlation coeficient between target vector and the prediction?

Is any behaviour possible?
Or is sure that it becomes larger or lowers?

It should not be lower but it does not always have to be higher.

Let's consider these two vectors:

A | B
1 | 2
2 | 3
3 | 4

The correlation between A & B is super high as I've just modeled a deterministic relationship (B = A +1). However if I perceive B as a predicted value and A as the real value then the MSE would not be 0.

Now let's say I improve my prediction and it's perfect:

A | B
1 | 1
2 | 2
3 | 3

We haven't impacted the correlation between those vectors at all because we simply removed the constant shift between the vectors. Their relation is still 1:1, however the MSE if B is the prediction and A is the real value is now much, much better than in the first example.


Trying to understand the Fowlkes-Mallows Score I recently bought Chris Albon's ML flashcards and I'm working my way through them. But this one on the Fowlkes-Mallows score has me stumped, as his definitions of false negatives and false positives seem reversed:

If y_hat are my predictions, and I've said that a pair of observations are in the same cluster when they're not, how is that a false negative and not a false positive?

Of course I read the Wikipedia article and did some Googling, but haven't turned up a clear answer yet.

Am I misunderstanding? Did he get this wrong?

Yes, its a mistake.

FN: they should be together but you said they are not, you false negatively ordered them to a different class.
FP: they should not be together, but you said they are you false positevly ordered them together


compare pandas dataframes with different lengths I have two dataframes of different lengths and I need to add a column to the first one with filtered values, e.g.

df1 = pd.DataFrame({'Object':['cup', 'brick', 'board', 'stone'], 'id':[2, 8, 9, 6]})
df1 = pd.DataFrame({'Thing':['cup', 'board'], 'color':['blue', 'grey']})

and I want to create

df = pd.DataFrame({'Thing':['cup', 'board'], 'color':['blue', 'grey'], 'id':[2, 9]})

All methods I tried to use complained about different lengths.

You can accomplish your task by using the merge operation in pandas as follows:

In [16]: df1 = pd.DataFrame({'Object':['cup', 'brick', 'board', 'stone'], 'id':[2, 8, 9, 6]})

In [17]: df1
Out[17]:
  Object  id
0    cup   2
1  brick   8
2  board   9
3  stone   6

In [18]: df2 = pd.DataFrame({'Thing':['cup', 'board'], 'color':['blue', 'grey']})

In [19]: df2
Out[19]:
   Thing color
0    cup  blue
1  board  grey

In [20]: df = df2.merge(df1, left_on='Thing',right_on='Object', how='inner')

In [21]: df
Out[21]:
   Thing color Object  id
0    cup  blue    cup   2
1  board  grey  board   9

and then drop the column (df.drop('Object', inplace=True)) that you don't need. For more details look at the official documentation here. Also, check out this to see how you can use merge and join operations in pandas to do all kinds of dataframe manipulations!


Similarity measure based on multiple classes from a hierarchical taxonomy? Could anyone recommend a good similarity measure for objects which have multiple classes, where each class is part of a hierarchy?

For example, let's say the classes look like:

1 Produce
  1.1 Eggs
    1.1.1 Duck eggs
    1.1.2 Chicken eggs
  1.2 Milk
    1.2.1 Cow milk
    1.2.2 Goat milk
2 Baked goods
  2.1 Cakes
    2.1.1 Cheesecake
    2.1.2 Chocolate

An object might be tagged with items from the above at any level, e.g.:

Omelette: eggs, milk (1.1, 1.2)
Duck egg omelette: duck eggs, milk (1.1.1, 1.2)
Goat milk chocolate cheesecake: goat milk, cheesecake, chocolate (1.2.2, 2.1.1, 2.1.2)
Beef: produce (1)

If the classes weren't part of a hierarchy, I'd probably I'd look at cosine similarity (or equivalent) between classes assigned to an object, but I'd like to use the fact that different classes with the same parents also have some similarity value (e.g. in the example above, beef has some small similarity to omelette, since they both have items from the class '1 produce').

If it helps, the hierarchy has ~200k classes, with a maximum depth of 5.

While I don't have enough expertise to advise you on selection of the best similarity measure, I've seen a number of them in various papers. The following collection of research papers hopefully will be useful to you in determining the optimal measure for your research. Please note that I intentionally included papers, using both frequentist and Bayesian approaches to hierarchical classification, including class information, for the sake of more comprehensive coverage.

Frequentist approach:

Semantic similarity based on corpus statistics and lexical taxonomy
Can’t see the forest for the leaves: Similarity and distance measures for hierarchical taxonomies with a patent classification example (also see additional results and data)
Learning hierarchical similarity metrics
A new similarity measure for taxonomy based on edge counting
Hierarchical classification of real life documents
Hierarchical document classification using automatically generated hierarchy
Split-Order distance for clustering and classification hierarchies
A hierarchical k-NN classifier for textual data

Bayesian approach:

Improving classification when a class hierarchy is available using a hierarchy-based prior
Bayesian aggregation for hierarchical genre classification
Hierarchical classification for multiple, distributed web databases


Is there any consensus on choosing an appropriate ML approach? I am studying data science at the moment and we are taught a dizzying variety of basic regression/classification techniques (linear, logistic, trees, splines, ANN, SVM, MARS, and so on....), along with a variety of extra tools (bootstrapping, boosting, bagging, ensemble, ridge/lasso, CV, etc etc). Sometimes the techniques are given context (eg. suitable for small/large datasets, suitable for a small/large number of predictors, etc) but for the most part, it seems like for any regression or classification problem there exist a dizzying array of options to choose from.

If I started a job in data science right now and was given a modelling problem, I don't think I could do any better than just try all the techniques I know with basic configurations, evaluate them using cross-validation and pick the best. But there must be more to it than this.

I imagine an experienced data scientist knows the catalogue of techniques well and follows some mental flowchart to decide which techniques to try, instead of mindlessly trying them all. I imagine this flowchart is a function of a) number of predictors; b) variable types; c) domain knowledge about possible relationships (linear/non-linear); d) size of the dataset; e) constraints around computation time and so on.

Is there any such agreed on, conventional flowchart to follow, to choose the techniques? Or does it really boil down to "try lots of things and see what works best on the desired measure eg. cross-validation"?

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning approaches has a special application domain and that is why people usually know where to use decision trees and where to choose neural networks, e.g. in situations that all of your inputs are real-valued numbers, attempting to use decision trees is not a wise choice. I try to explain the main things that an ML practitioner usually considers.

Number of Available Features

The number of features is important due to not being able to visualise them easily in cases there are so many features. This may lead to not being able to recognize whether the data is linearly separable or not. So many features do not imply that the dataset is not linearly separable. Consequently, say you want to use neural nets for modelling the problem. You shouldn't begin with a complicated network with so many layers and neurons. You have to begin with a single neuron, equivalent to logistic regression for classification tasks, to figure out whether your data is linearly separable or not. If you observe that you don't have good performance, you can add extra neurons and layers. How? Take a look at How to set the number of neurons and layers in neural networks.

Feature Space

About choosing ML approaches, it is simple to consider the limitations of each algorithm with respect to the feature space. E.g. decision trees are not very good for problems with many features which some of them are numerical features. They may get really big. SVMs are not very good for non-linear problems with so many features due to the fact that you have to specify the kernel size. For different regions in the feature space, a single kernel size may not be valuable. To generalize, problems that have very large input space are usually handled using neural networks. If the problem has so many features but they are e.g. binary features or categorical features with a small number of choices, then the problem has smaller feature space, input space, and other ML approaches can be considered.

Size of Dataset

Depending on your problem, feature types and feature ranges, the input space may be very small or very big. Consequently, the number of possible data may differ depending upon the input space. For large input space, as mentioned, neural nets are very powerful for mappings.

Distributions and Bayes Error

For different tasks, e.g. classification, you have to do statistical analysis for knowing your available dataset better. You have to investigate if there are input patterns that there are same but their labels differ. If so, why? Is that for expert error or not. The current feature space is not valid for understanding the problem. After addressing these questions you can employ Bayes error to investigate the best approach will have what accuracy on your data.


Criteria used to create and select leaf nodes in sklearn I just want to know the details of what (and how) is the criteria used by sklearn.tree.DecisionTreeClassifier to create leaf nodes. I know that the parameters criterion{“gini”, “entropy”}, default=”gini” and splitter{“best”, “random”}, default=”best” are used to split nodes. However, I could not find more information about the threshold used for spliting.
There are some methods involved in the creation of leaf nodes: post-pruning (cutting back the tree after a tree has been built) and pre-pruning (preventing overfitting by trying and stopping the tree-building process early).  It would be very useful to know more details about the criteria used for splitting to have a better understanding and be able to customize these models even more.

Pre-pruning is handled by a variety of parameters:
max_depth, min_samples_split, min_samples_leaf, min_weight_fraction_leaf, max_leaf_nodes, and min_impurity_decrease.
Post-pruning is relatively new to sklearn, and is accomplished with minimal cost-complexity pruning, parameter ccp_alpha.


Analysis of railway data - Detecting outliers I would like some pointers about the following problem:

I would like to detect anomalies in a pretty huge collection of railway data. Or create a baseline model for detecting future anomalies. The data I have at my disposal exists out of coordinates and speed at the given coordinate (also the time of measurement). Could this perhaps be approached as a regression problem where there's a clear(?) connection between location and speed of a train. For instance a train suddenly moving at snail's pace on a track that from historical data is considered high velocity could be a potential anomaly. If this could indeed be approached in such a way would something like SVM be an option or should I look into other algorithms?

There are a number of directions you can take, and there are a number of questions related to this topic (anamoly-detection-for-transaction-data).  One question to answer first is whether the data is seasonal or not.

You already suggested a valid approach.  Determine which areas are correlated to low and high speeds and than determine when a speed differs.  I think it's best to start with a heuristic approach and than try more complicated methods to try to improve.  If you want to use an SVM, a one-class SVM may work well (one-class SVM)


Proof that Gini Impurity in a Decision Tree is Monotone Decreasing? I asked this in a reply to an answer to another of my questions; but I think this merits its own question since I couldn't find an answer, and it's a pretty interesting question on its own.
Suppose we construct a decision tree for classification based on the Gini impurity function.  Can we prove that the weighted average of the Gini impurities of children nodes is always <= the Gini impurity of the parent node?
More precisely:
Let G(S)=sum_i p_i(1-p_i), where S is a finite nonempty set of points with known classification, p_i is the proportion of points in S with classification i, and the sum is taken over all classes. In the special case of binary classification, this simplifies to G(S)=2p(1-p), where p is the proportion of one of the classes.
Assume that every point x has a feature f. Denote the value of this feature by x(f). A splitting of S is defined as a partition of S into {S_left, S_right}, where S_left = {x in S: x(f) <= c} and S_right = {x in S: x(f) > c}. We require both of these sets to be nonempty.
A splitting is called good if
|S_left|/|S| G(S_left) + |S_right|/|S| G(S_right) <= G(S)

Must there always exist at least one good splitting?
Must all splittings be good?

I think I've got a proof.
Any node can be identified with the vector (p_1,...,p_k) that gives the proportion of points with each classification. The Gini impurity function can then be viewed as a function from R^k to R. The weighted average of the proportions of points in S_left and S_right belonging to a certain class is equal to the proportion of points in S belonging to that class. Thus the inequality is just stating that the Gini impurity function is concave. But that is true; its matrix of second derivatives is just a diagonal matrix whose diagonal entries are all -1.


DBPedia as Table not having all the properties I browsed a sample for available data at http://dbpedia.org/page/Sachin_Tendulkar. I wanted these properties as columns, so I downloaded the CSV files from http://wiki.dbpedia.org/DBpediaAsTables.

Now, when I browse the data for the same entity "Sachin_Tendulkar", I find that many of the properties are not available. e.g. the property "dbpprop:bestBowling" is not present.

How can I get all the properties that I can browse through the direct resource page.

The official answer:

Date: Sun, 23 Nov 2014 03:08:19 +0100
From: Petar Ristoski <petar.ristoski@informatik.uni-mannheim.de>
To: 'Barry Carter' <carter.barry@gmail.com>
Subject: RE: CSV tables don't have all properties?

Hi Carter,

The question was already answered on the DBpedia mailing list, but I will
try to clarify it again. On the DBpedia as Tables web page says that "For
each class in the DBpedia ontology (such as Person, Radio Station, Ice
Hockey Player, or Band) we provide a single CSV/JSON file which contains all
instances of this class. Each instance is described by its URI, an English
label and a short abstract, the MAPPING-BASED INFOBOX data describing the
instance (extracted from the English edition of Wikipedia), geo-coordinates,
and external links." As you can see we only provide the mapping-based
infobox properties (dbpedia-owl namespace), while the properties from the
dbpprop (raw infobox properties) namespace are completely ignored.
Therefore, dbpprop:bestBowling is missing from the file.

Also, there is a section "Generating your own Custom Tables" [1], where we
explain how to generate your own tables that will contain the properties you
need.

Regards,

Petar

[1] http://wiki.dbpedia.org/DBpediaAsTables#h347-4


What is the exact definition of VC dimension? I'm studying machine learning from Andrew Ng Stanford lectures and just came across the theory of VC dimensions. According to the lectures and what I understood, the definition of VC dimension can be given as,

  If you can find a set of $n$ points, so that it can be shattered by the classifier (i.e. classify all possible $2^n$ labeling correctly) and you cannot find any set of $n+1$ points that can be shattered (i.e. for any set of $n+1$ points there is at least one labeling order so that the classifier can not separate all points correctly), then the VC dimension is $n$.

Also Professor took an example and explained this nicely. Which is:

Let,

$H=\{{set\ of\ linear\ classifiers\ in\ 2\ Dimensions \}}$

Then any 3 points can be classified by $H$ correctly with separating hyper plane as shown in the following figure.

And that's why the VC dimension of $H$ is 3. Because for any 4 points in 2D plane, a linear classifier can not shatter all the combinations of the points. For example,

For this set of points, there is no separating hyper plane can be drawn to classify this set. So the VC dimension is 3.

I get the idea till here. But what if we've following type of pattern?

Or the pattern where a three points coincides on each other, Here also we can not draw separating hyper plane between 3 points. But still this pattern is not considered in the definition of the VC dimension. Why? The same point is also discussed the lectures I'm watching Here at 16:24 but professor does not mention the exact reason behind this.

Any intuitive example of explanation will be appreciated. Thanks

The definition of VC dimension is: if there exists a set of n points that can be shattered by the classifier and there is no set of n+1 points that can be shattered by the classifier, then the VC dimension of the classifier is n.

The definition does not say: if any set of n points can be shattered by the classifier...

If a classifier's VC dimension is 3, it does not have to shatter all possible arrangements of 3 points.

If of all arrangements of 3 points you can find at least one such arrangement that can be shattered by the classifier, and cannot find 4 points that can be shattered, then VC dimension is 3.


Business-related metric for an optimization problem We have an optimization problem on hand that is the following.
Let's say we have 10 different treatments that we might offer, all of them are equally good for us, but people have different propensity to take those treatments.
Before we've built the model the rule was to just randomly choose one of the treatments, offer it to a person, and if the person takes it - yaay, if not - so be it.
Based on this historic data we've created a binary classifier that predicts a person's propensity to agree to each treatment type and the idea is to offer a person a treatment that has a max propensity score from the model.
The model built shows great GINI and AUC numbers, but what we struggle with is a more business-related metric that we could use to evaluate the model's performance.
I'm sure some of you had been in this situation before. What metrics did you use?
Thank you!

On historical data, you can try to determine what would have happened if you add the model before. For each client you proposed some product, calculate the average gain (propensity*cost) of said product and compare with the average gain of said of best product. Say each product cost 100$.
Last year you offered product1 to client1 but he refused. Using your model you know : he only had 20% chances of buying product1 and he had 25% chances of buying product 2. On average you missed out on (25%-20%)*100 = 5 dollars. You can calculate a similar metric on all your clients, than aggregate the total. That would give you an estimation of the amount of dollars you would have made last year if you used your model. There rarely is more business-related metric than money.


Deep learning with Imbalanced classes I am trying to model a packet data with 1 dimensional CNN but I have a very imbalanced classes in my target. I have 3 classes as class 0 has 53000 cases, class 1 has 300 cases and class 2 has 150 cases. Thanks in advance!
I have tried what you have suggested but I do not think I am getting a good results(loss and accuracy), from the model.

If you are using TensorFlow to write your architecture,

You can create 3 training datasets( one for one class) using tf.data and then you can use tf.data.experimental.sample_from_datasets with the same weightage to generate the batch data. Check weights parameter in the documentation.

You can initialize well in the final layer. Please check this blog from Andrej Karpathy.

You can add the class_weight parameter while training.


Automomatic typographical error correction Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking (which can have very many candidates in relation to the correct spelling of the word in question); rather I am looking for a similar list as used by Microsoft Word (for instance correcting

  teh

with

  the

or

  becaise

with

  because

Not only can the manner in which these sort of errors are fixed be hard-coded, their frequent occurrence in text provides significant dividends in textual mining (provided that such a list of errors and corrections can be obtained, of course).

I will reference two sources:

Wikipedia has its own lists: Lists of common misspellings
Aspell is certainly a place to visit. You can find a list of misspellings and corrections here.

Still, it is a good idea to think of using a spell checker, such as Aspell, since it gives you the ability to deal with no-so-common misspellings (of course, at the cost of false positives). A good starting point is Peter Norvig's essay on creating a spelling corrector.


Data set size versus data dimension, is there a rule of thumb? I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough.

So is there a rule of thumb on how many data of dimension DIM should one collect for training a NN-Model ? For example, it depends on the number of features or the kind of NN-Model or something else ?

Any help will be appreciated.

In this video by Caltech prof. Yaser Abu-Mostafa, he explains the relationship between dimension of a dataset and it's size required for any learning model to work.

As a general rule of thumb, size of dataset should be at-least about 10x it's dimension and should be independent of the model used.

Also, this link has summaries from some of the relevant papers, viz.

  For a finite sized data with little or no a priori information, the ratio of the sample size to dimensionality must be as large as possible to suppress optimistically biased evaluations of the performance of the classifier.

This says, the ratio of size of dataset (sample) to dimension should be as large as possible to reduce classifier bias towards a particular class.

  The ratio of the sample size to dimensionality should vary inversely
  proportional to the amount of available knowledge about the class
  conditional densities.

In a classifier setting, the more knowledge we have for each class' probability density, lesser can be the sample-size to dimension ratio.

In simpler terms, we can say we should be including as much data as possible and if we are not able to do that, include as much information as possible in the small dataset itself, because for any model to work we need to feed it with high variance dataset.


Jupyter notebooks and git I am looking for any straightforward solution to work with Jupyter Notebooks and git since the cached data on the .ipynb files makes the diff difficult to read.

Does anyone know any standalone alternative and not server hosted?

One option is to strip the output from the .ipynb file. Then the git diff would only track the cell data. One package that strips the output Jupyter Notebook is nbstripout.


Storing Sensor Data for Analysis of the Office I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked on data analysis before, I would like some guidance on how to store this data (which database design to use).

What we're looking at currently are around 50 sensors that only send data when an event of interest occurs: if the temperature changes by 0.5 degrees or if the light turns on/off or if a room becomes occupied/vacant. So, the data will only be updated every few seconds. Also, in the future, I'd like to analyse some of the data. Hence, the data must be persistent in the database. What kind of technologies would you suggest to carry out this task?

I have been doing similar project in my college. I have classroom and I'm supposed to collect data like temp, humidity, light, occupancy, etc. Assuming that you have worked with sensors and motes to use, I'm going to explain rest of the structure.

You need sensor network setup and like you said you have done it. These sensor networks generally do not send data directly over internet so you need a Gateway that can collect data from sensors and send it over internet to local server.

On server side you need REST API and you could use any language to develop it and I use PHP. I find it very easy to use and develop using PHP. This REST API shall receive data from Gateway and store it into database. I use mysql database because amount of data is not so big for us. But if your data is big enough you can use big data Nosql tool like mongoDB or so. Whatever type of database you use structure remains same.

For sending data from Gateway to server you can use protocols like HTTP or MQTT whichever you feel comfortable. What I do is I have WSN controller that sends data over USB to Gateway then Gateway sends data to server over Ethernet. So I had to develop USB to Ethernet Gateway. If you can just take two UART terminals out of your controller you can build UART to Ethernet Gateway using any microcontroller or even Arduino Ethernet shield would work in that case.

In my case data is sensed periodically but as you said you are sensing data when event of interest occurs then you can use poisson distribution method over periodically collected data to predict what is average number of events per day and then you can decide if your data is big or not.


RandomForest and tree feature importance in scikit-learn What is the difference between model.feature_importances_ and tree.feature_importances_ in the following code:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

# Boston Housing dataset
from sklearn.datasets import load_boston
boston = load_boston()
# Convert 'skleran.bunch' to Pandas dataframe
data = pd.DataFrame(boston.data, columns=boston.feature_names)

# Create train and test sets for cross-validation
X,y = data.iloc[:,:-1], data.iloc[:,-1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = 123)

model = RandomForestRegressor()
model.fit(X_train, y_train)

As I understand the following is importance of features:

importance = model.feature_importances_
importance_df = pd.DataFrame(importance, index=X_train.columns,
                      columns=["Importance"])

        Importance
-----------------
CRIM    0.025993
ZN      0.002781
INDUS   0.004832
CHAS    0.000315
NOX     0.028655
RM      0.406285
AGE     0.017987
DIS     0.040696
RAD     0.003615
TAX     0.009281
PTRATIO 0.009103
B       0.012354
LSTAT   0.438106

And what is tree.feature_importance?:

[tree.feature_importances_ for tree in model.estimators_]

How do they differ, calculated and which is more important 0.2 or 0.9? Can't find in the docs.

_feature_importance of a random forest calculates the average feature importance across all trees in the forest. While tree.feature_importances_ is the feature importance for a single tree.

Since feature importance is calculated as the contribution of a feature to maximize the split criterion (or equivalently: minimize impurity of child nodes) higher is better.

You can see how it works in the source code:

The property _feature_importance of random forests is defined as following (see here for the complete source code):

@property
    def feature_importances_(self):
        """
        Return the feature importances (the higher, the more important the
           feature).
        Returns
        -------
        feature_importances_ : array, shape = [n_features]
            The values of this array sum to 1, unless all trees are single node
            trees consisting of only the root node, in which case it will be an
            array of zeros.
        """
        check_is_fitted(self)

        all_importances = Parallel(n_jobs=self.n_jobs,
                                   **_joblib_parallel_args(prefer='threads'))(
            delayed(getattr)(tree, 'feature_importances_')
            for tree in self.estimators_ if tree.tree_.node_count > 1)

        if not all_importances:
            return np.zeros(self.n_features_, dtype=np.float64)

        all_importances = np.mean(all_importances,
                                  axis=0, dtype=np.float64)
        return all_importances / np.sum(all_importances)

As you can see it returns the average feature importance across all tress in the forest and thereby uses the tree class. The tree class implements this as following:

def feature_importances_(self):
        """Return the feature importances.
        The importance of a feature is computed as the (normalized) total
        reduction of the criterion brought by that feature.
        It is also known as the Gini importance.
        Returns
        -------
        feature_importances_ : ndarray of shape (n_features,)
            Normalized total reduction of criteria by feature
            (Gini importance).
        """
        check_is_fitted(self)

        return self.tree_.compute_feature_importances()

And here compute_feature_importances is defined:

cpdef compute_feature_importances(self, normalize=True):
        """Computes the importance of each feature (aka variable)."""
        cdef Node* left
        cdef Node* right
        cdef Node* nodes = self.nodes
        cdef Node* node = nodes
        cdef Node* end_node = node + self.node_count

        cdef double normalizer = 0.

        cdef np.ndarray[np.float64_t, ndim=1] importances
        importances = np.zeros((self.n_features,))
        cdef DOUBLE_t* importance_data = <DOUBLE_t*>importances.data

        with nogil:
            while node != end_node:
                if node.left_child != _TREE_LEAF:
                    # ... and node.right_child != _TREE_LEAF:
                    left = &nodes[node.left_child]
                    right = &nodes[node.right_child]

                    importance_data[node.feature] += (
                        node.weighted_n_node_samples * node.impurity -
                        left.weighted_n_node_samples * left.impurity -
                        right.weighted_n_node_samples * right.impurity)
                node += 1

        importances /= nodes[0].weighted_n_node_samples

        if normalize:
            normalizer = np.sum(importances)

            if normalizer > 0.0:
                # Avoid dividing by zero (e.g., when root is pure)
                importances /= normalizer

        return importances


Best regression model to use for sales prediction I have the following variables along with sales data going back a few years:

date # simple date, can be split in year, month etc
shipping_time  (0-6 weeks) # 0 weeks means in stock, more weeks means the product is out of stock but a shipment is on the way to the warehouse. Longer shipping times have a siginificant impact on sales.
sales # amount of products sold

I need to predict the sales (which vary seasonally) while taking into account the shipping time. What would be a simple regression model that would produce reasonable results? I tried linear regression with only date and sales, but this does not account for seasonality, so the prediction is rather weak.

Edit: As a measure of accuracy, I will withold a random sample of data from the input and compare against the result.

Extra points if it can be easily done in python/scipy

Data can look like this

--------------------------------------------------
| date           | delivery_time| sales          |
--------------------------------------------------
| 2015-01-01     | 0            |10              |
--------------------------------------------------
| 2015-01-01     | 7            |2               |
--------------------------------------------------
| 2015-01-02     | 7            |3               |
...

This is a pretty classic ARIMA dataset. ARIMA is implemented in the StatsModels package for Python, the documentation for which is available here.

An ARIMA model with seasonal adjustment may be the simplest reasonably successful forecast for a complex time series such as sales forecasting. It may (probably will) be that you need to combine the method with an additional model layer to detect additional fluctuation beyond the auto-regressive function of your sales trend.

Unfortunately, simple linear regression models tend to fare quite poorly on time series data.


Establish relationship between two sets of data I have two data sets - Product to Features and Products to Parts

A = { (P1, F1, F2, F3), (P2, F2, F4, F6), (P3, F1, F6, F8)...}
B = { (P1, M1, M2, M3), (P2, M4, M7), (P3, M1, M5, M7, M9, M10)..}

where:

P1, P2, P3... are products
F1, F2, F3... are features
M1, M2, M3... are parts used in building products.

Is it possible to come up with relationship amongst features and parts?

e.g. (F1, M1, M2), (F2, M1, M4, M6)... and so on?

I think you may be able to shed light on relationships between features and parts using association rule learning.  You can treat the parts and features similarly to items in a market basket.

Updated:

A = {('P1', 'F1', 'F2', 'F3'),
     ('P2', 'F2', 'F4', 'F6'),
     ('P3', 'F1', 'F6', 'F8')}

B = {('P1', 'M1', 'M2', 'M3'),
    ('P2', 'M4', 'M7'),
    ('P3', 'M1', 'M5', 'M7', 'M9', 'M10')}

parts = {'P1','P2','P3'}

basket = {}
for k in parts:
    temp = []
    for a in A:
        if k in a:
            temp += list(a)
    for b in B:
        if k in b:
            temp += list(b)
    temp.remove(k)
    basket[k] = temp

Then use each value in the basket dictionary as a basket.


What are some general tips to improve my MNIST classifier? I have built a CNN from scratch in python using Numpy, to tackle the MNIST hand-written digit recognition problem. It's composed out of a convolutional layer (3 3x3 filters), a maxpooling layer (2x2 pooling) and the 10-label output layer. I'm using softmax for the output activation function and cross-entropy as loss function. I've tried running it with a couple of different hyperparameters and so far the best accuracy i've gotten is 97%, when training on the whole train dataset (60000 images) for a single epoch and using SGD. The accuracy does vary a bit though, usually around 92-95% under these conditions. I've only tried one epoch when using the whole dataset because it already takes maybe 15 min for my algorithm to train using 60000 images (with cpu on my low grade school laptop). The thing is that I don't have an perception of how good/bad this is, i.e. how much time one would expect such an network to take and how accurate it would be. Is this really slow and inaccurate? I'd love some general tips on how I can improve my network, whether it's through some optimization methods or brute force (increased amount of layers/neurons). I've also tried implementing mini-batches but for some reason (maybe faulty implementation?) this only seems to decrease the accuracy.

You may find this OpenML interesting
In there several benchmarks on different datasets are shown so that you can see how different models score across the same dataset
For your task MNIST the plot shows:

that might give a good idea of how well your model is performing as compared to other implementations/models
Metric button in the upper left corner will allow you to compare in terms of classification metrics and also in terms of execution time


How to summarize multiple time series like dataset I am running an experiment where I am sampling CPU utilization every second.

A snippet of the data from each experiment run is something like this:

timestamp   cpu1  cpu2  cpu3  cpu4
1.581015e+09    0.0   0.0   0.0   0.0
1.581015e+09   41.6  57.4  54.5  47.0
1.581015e+09   84.3  72.5  46.6  82.8
1.581015e+09    1.0  83.7   1.0   2.0
1.581015e+09   33.0  43.3  44.4  37.8

I am using pandas to explore the data.

To ensure validity of the experiment, I repeat the experiment 20 times.

Now I have 20 of each of the dataset above.

My question now, is what is the best strategy to:

Summarize the data.
Graph a summary of the data

If I only had one run, I would like to graph the data in a line chat, but now that I have 20, not sure how to proceed.

Find the average at each second across all the 20 dataset? Or is there any recommended approach to handling this?

That depends what you want to show and how much information fits in the graph. Typically you can think of:

Simply using a different colour for every dataset, but 20 is probably too many and that will make the graph hard to read.
Using boxplots (or violin plots) at every instant, where the boxplot represents the 20 values of the dataset.


Is Categorical Crossentropy Loss always bounded in the [0, 1] interval? Is Categorical Crossentropy always bounded between 0 and 1, or is it possible that during training of a Neural Network it can get higher values?

More specifically, I'm referring to the TensorFlow 2.0 function.

You can actually get values from $[0, \infty[$. Consider the cross entropy loss be following formula:

$loss (y_{pred}, y_{true}) = - \log{y_{pred, class_i}}$

, whereas the index $class_i$ states that you only use the output of the $i$th class.

For example think that your true class is $dog$, which is represented by the $3$rd output neuron. On the $3$rd output neuron your network outputs a probability of $10\%$. Your loss is then $-\log{10\%}= 2.3$. Further if your probability is only $0.1\%$ your loss will equal $6.9$ and so on.

Remember that:

$\lim\limits_{x \rightarrow 0}{\log{x}} = -\infty$


Pytorch - Gradient distribution between functions https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb

Hi I am trying to understand the NN with pytorch.
I have doubts in gradient calculations..

import torch.optim as optim

create your optimizer
optimizer = optim.SGD(net.parameters(), lr=0.01)

# in your training loop:
optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update

From the about code, I understood loss.backward() calculates the gradients.
I am not sure, how these info shared with optimizer to update the gradient.

Can anyone explain this..

Thanks in advance !

Recall that you passed net.parameters() to the optimizer, so it has access to the "Tensor" objects, as well as their associated data. One of the associated data fields associated to each learnable tensor parameter is a gradient buffer. Hence, backward() not only computes the gradients, but stores them in each parameter tensor, so that the gradient vector per parameter is stored along with that parameter. In other words, for some parameter $\theta_i$, backward() stores $ \partial \mathcal{L}(\Theta)/\partial \theta_i$ along with that parameter. The optimizer.step() call then simply updates each parameter via the gradient stored along with it.


What is the minimum amount of data required for sales prediction with ML I have historical data from the MySQL DB which contains 33 months of data.
The features in the data are state, depot, product type, purchase date, sales person name, volume, and price.

Using this information, I would like to predict/forecast product sales weekly, monthly, and yearly.

Since I only have 34 months of data, I do not think that I can apply statistical modeling techniques like Exponential Smoothing or ARIMA.

So, I just wanted to check if this data is sufficient for Machine learning modeling such as Linear Regression, SVM, or Random forest? Or any kind of feature engineering.

Note: I am unable to get more data, as DB has only 34 months of data.

Please suggest.

As @Ethan said there is no general answer to this question. There are multiple perspectives that you need to take in consideration.

Amount of the data: The general tendency is more data will lead to better results.
Variability of the data: A very important aspect is the variability of your data. If you have millions of data points that have almost the same input and output values it is empirically equivalent to a situation in which you only have one single data point.
The timescale of data acquisition: The variability will strongly depend on the relevant timescale of the problem that you are trying to solve. If you are looking at products (e.g. tickets for public transportation) that are sold at a very high frequency the data acquisition rate should be shorter (short/fast time scale). If your data acquisition rate is too small your model will not be able to detect relevant changes e.g. tickets sales during the day. If you are looking at products (e.g. car sales) then your data acquisition rate should be longer (long/slow time scale). In the second case, I would still use the fastest possible data acquisition rate, because we can always reduce the given data. Note, that the data acquisition rate is linked to the variability of the data.
Diversity of the data conditions: This is related to the variability of the data. But the focus here is on the conditions under which your data was collected. For example, imagine a situation in which we have some time series measurements of machines in a production plant during summer. If we know that the process is heavily dependent on the temperature we must also get measurements during spring, autumn, and winter in order to increase the diversity of our data set.

As the data problem is more involved it is difficult to give concrete numbers but the following rules of thumb can be applied. These must always be considered with the previous points.

In statistics, we often have the lower bound for samples to be $30$. This implies that you should never have less than $30$ observations.
For estimating parameters it is often advised to have $50$ (worst case) or better more than $200$ observations for each parameter that you want to estimate. For example, if you need to estimate $5$ parameters that would imply $250$ observations for the worst case and more than $1000$.


Implementing Frequently bought together using a DB We have a classic structure of an online shop database (products, customers, sales) and we want to implement a Frequently bought together feature. Our software is in ASP.NET and we do not know PHP to reverse engineer how this is being done in Magento.

And all we need is a simple Frequently bought together (not with discounts like Magento offers).

I understand that this is machine learning and one of the more common ways is Jaccard coefficient. Is that the recommended way?

My main question however is what the View I create in the DB should look like? What is the structure of the table that Jaccard coefficient and other relevant tests require so that used best?

It is called Association Rule mining. You can implement a very basic version of this algorithm easily which works fine for small  or mid size datasets. What this algorithm does is basically finding the most frequent items in all the transactions. In your example, you can find the frequent items purchased together by customers in your online store. I implemented one in .Net for a course it just took me 2 days to understand and code it.

The format of the data really depends on the implementation, for instance, my algorithm reads data in a text file that each line of the file is a tuple of items:

1,2,3
4,3,5
6,1,4,5,1
7,6,5

Where each line is a transaction, and the numbers can be the items purchased by a user at one checkout. The algorithm needs a number which is called the threshold. It means the algorithm finds the patterns of items that repeated X times (it can be a percentage) in all of the transactions.
The result can be a couple of rows that each is the combination of items that were bought together in a transaction and managed to satisfy the threshold.


Sales Predictions Over Time I'd like to model the evolution of the sales of a store.

Here are the data I have :

Customers are aggregated into monthly cohort depending on the date of the first purchase
eg : customers who did their 1st purchase in Jan 2015 are in the cohort 1, customers who did their 1st purchase in Fev 2015 are in the cohort 2.
Sales : aggregated for each Month * Cohort (cohort 4 is not visible because there is no line for only 1 record)

if we put all the cohorts at the same origin:

Question: How to predict the spending of the next 2 months, ie :

the spending of the cohort 1 in its aging 5 and 6
the spending of the cohort 2 in its aging 4 and 5
the spending of the cohort 3 in its aging 3 and 4
the spending of the cohort 4 in its aging 2 and 3

But also for the cohort who don't exist yet :

the spending of the cohort 5 in its aging 1 and 2
the spending of the cohort 6 in its aging 1

I have 2 methodologies in mind, but don't know if they're good:

Polynomial regression by using 2 parameters:

aging: because there is a clear evolution over the time, up then down.
cohort number: because more recent cohort seems to spend less.

Time series : I didn't perform time series recently I'm a bit rust so I would prefer to use something else but I think it could fit with this problem.

Indeed it seems obvious that yt-1 t in yt-2... are good predictors of yt but how to predict spending of future cohort with no history ?

It sounds like you are making this more complicated than it is with trying to predict spend at the cohort level. It is best to use cohorts for understanding "why" something occurred and time series analysis for understanding "what" will occur.

The Forecast package in R seems to be solution to your problem. Just forecast Monthly spend and be done with it!


How does the naive Bayes classifier handle missing data in testing? Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this missing feature?

In evaluation (test) phase, when data point $x_n$ has $d$ missing features at indices $M=\{m_1,...,m_d\}$, corresponding terms $P(x_i|C_k),i \in M$ are simply removed from the classifier. That is, classifier
$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}P(C_k)\prod_{i}P(x_i=x_{n,i}|C_k)$$
is replaced with
$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}P(C_k)\prod_{\color{blue}{i:i \notin  M}}P(x_i=x_{n,i}|C_k)$$

where $i$ iterates over features, $x_{n,i}$ denotes the $i$-th feature of data point $n$, and there is $K$ classes in total.


Highly imbalanced class Machine Learning I'm currently messing around on a wage dataset (trying to predict who did and didn't earn over 50k p.a. based on a range of factors). One of the variables - 'work-class' is very imbalanced and I was looking for some advice on how to deal with this.

100% of the people within the never-worked and without-pay categories earn over 50k.

Would imputation methods even be useful given the size of groups?

Thanks in advance,
Kelvin

What I'd do is to keep the Unknown class and treat the Never-worked and Without-pay as Unknown.

Imputation methods would just reinforce the bias towards the most dominant class, i.e. private in your case. Say you used a simple classifier to impute these $21$ values, it would assign them to another class based on the similarity of the other features (e.g. age, education, ...). Think on what this procedure has done though, it has replaced someone with without-pay to private or something else. This is conceptually wrong, though, especially for a model that tries to predict the income of someone.


What Predictions can be done for Restroom/Washroom Data? My company has installed sensors, which actually monitors the Restrooms/Washrooms in the Building. Currently the sensors collect data such as Ammonia, Nitrous and Visitors Count;

All the data collected are time-series data and I have a huge quantity of those. My question is what type of predictions can be done that can add value to this business?

Can I use the power of Data Science to avail something which can make my restroom more smarter?

Welcome to the site! In order to prevent from falling into common data science traps, I would encourage you to think about the following:

You should never take data at face value. You should be thinking about the transforms that you could be doing to your data to get more insights. For example, from the fields you have, you can also determine the time in between visitors, right? Can you determine the number of visitors per day? Also, what other datasets are out there that you can be tying together with your data?
You shouldn't be thinking about the business value of the data until you figure out what the data is telling you. This is where exploratory data analysis (EDA) comes in. You should explore your data and let the data speak to you (not the other way around!) so that you can be in a better position to think about your entire project.

Remember, the entire point of data science is to enable you to ask smarter questions. So, take the time to explore your data and figure what else you need. That will enable you to ask smarter questions and then declare some use cases for your business/client.


Can a Warm-up loss period easily be implemented in Keras? I am creating a RNN in Keras. It was suggested that I utilize a warm-up period before loss is calculated to increase accuracy down the line.

I saw some people achieved this by creating a loss function via tensorflow.

Does Keras have any simple way to directly add a warm-up period?

To my knowledge it is currently not possible to compile a Keras model first with a warmup loss function and later re-compile it with another. That would be the cleanest solution in my opinion.

You can however, with a little use of tensorflow (assuming you are using that as a backend judging from your tags) do something along these lines:

WARMUP_ROUNDS = 10

def loss(y_true, y_pred):
   seen = tf.Variable(0.)
   seen = tf.assign_add(seen, 1.)

   loss = tf.cond(tf.less(seen, WARMUP_ROUNDS),
                  lambda: warmup_loss(y_true, y_pred),
                  lambda: actual_loss(y_true, y_pred))

   return loss


how can i interpret kernel density plots from classification? all,
i have a classification problem where i am predicting likelihood of client defaulting on loan. i plotted the predicted probabilities from my model, and then plotted against the label '1' for default or 0 for non-default.

it is cut out here but y axis is the density. am i right to reason that this shows an exponential distribution, or that the fact the class 1 curve has a fat tail it shows that default is an extreme / unexpected event? woud you say class 1 is following any type of distribution?
compare this to the below:

doesn't the second graph show that the model isn't that good at distinguishing between class 0 and class 1?

In both graph it show that the model will not perform very well on the classification task as the probability distribution of the model overlaps significantly. A good model will have almost seperated curve for each class. Adding more feature will help the model differentiate between curves.


Hyperparameter Tuning in Machine Learning What is the difference between Hyper-parameter Tuning and k-NN algorithm? Is k-NN also a type of Hyper-parameter tuning?

In kNN algorithm, you only try to find a suitable value of parameter k. And some models may have many parameters that can be modified. Normal parameters are optimized by loss functions and Hyperparameter tuning allows you to set various parameters to get the best model. You set them before training. Its 2 methods: GRID METHOD AND RANDOM SAMPLING might work well.

Grid Method:
Impose a grid on possible space of a hyperparameter and then go over each cell of grid one by one and evaluate your model against values from that cell. Grid method tends to vast resources in trying out parameter values which would not make sense at all.

Random Sampling Method:
In random method, we have a high probability of finding a good set of params quickly. After doing random sampling for a while, we can zoom into the area indicative of the good set of params.

Random sampling allows efficient search in hyperparameter space. But sampling at random does not guarantee uniformity over the range of valid values. Therefore, it is important to pick an appropriate scale.

You can read this: Hyperparameter Tuning of Deep Learning Algorithm


How to determine which features matter the most? I have a large dataset that consists of search results of loans. Someone would input their details like income etc and the results would include a bunch of loans from different companies and different loan types (so there can be more than 1 loan per company).
The dataset consists of every unique search and all the corresponding results. I also have a column that shows which loan has been selected at the end by the user per each search.
I am looking to find out which features of the loan were most important to users, i.e. try to predict what loan the user will select depending on his/her inputs.
What ML model could I use for this? I am unsure how to approach the problem.

One common approach is to use Principal Component Analysis (PCA) and drop the directions with less variance. See here for instance:
Using principal component analysis (PCA) for feature selection

The latest version of sklearn allows to estimate the feature importance for any estimator using the so-called permutation importance:
Permutation feature importance
Random forest in sklearn also have other methods implemented to estimate the feature relevance:
Feature importances with a forest of trees


Is there an analog to SQL's STRING_AGG (or FOR XML PATH) function in Python? Asked this in SE but maybe this is too data-oriented so trying to post it here. I am trying to find the analog to the SQL function STRING_AGG so that I can concatenate across columns and rows of a table (or dataframe).

Here is an example of what I am trying to achieve:

input:

output:

With SQL I can easily group by the ID_No and also specify the order by via the RUN_No. The syntax for achieving what I want would be:

SELECT ID_NO,
STRING_AGG(CONCAT('(', RUN_No, ') ', Start, ' to ', Stop))
WITHIN GROUP (ORDER BY RUN_No ASC) AS "Sequence"
FROM X_TBL GROUP BY ID_NO

So what would be the way to achieve the same grouping, concatenating and ordering in Python? I do have my data stored as a dataframe. I am able to concatenate across columns using the following code, but then wasn't sure how to group by the "ID_No", or concatenate across the rows within each ID_No.

sample['Merge'] = sample['Start'].map(str) + ", " + sample['Stop']

Here a versatile solution. As you can see, you can modify the aggregation function in order to format the data as you want.

#the original DataFrame
df=pd.DataFrame({'ID_NO': [20, 20, 30, 30, 30], 'RUN_NO': [1,2,1,2,3], 'START': ['F2','F3','F9','F11','F14'],
                'STOP': ['F3','F2','F11','F14','F6',]})

#convert 'RUN_NO' to string. This will make the aggregation formula easier to read
df['RUN_NO']=df['RUN_NO'].apply(str)

#The aggregation function
def agg_f(x):
     return pd.Series(dict(Sequence = "%s" %  ' '.join('(' +x['RUN_NO']+ ') ' + x['START'] +' to ' + x['STOP'])))

df_agg=df.groupby('ID_NO').apply(agg_f)

The output will be:


Which Classification Metrics Are Appropriate For Each Class Distribution Scenario? Currently, I have a balanced dataset (that I artificially over-sampled to make it balanced). My classes are binary (0 or 1). I'm wondering if "accuracy" is the "best" metric to use in the situation when a dataset has roughly equal class balance?

More broadly though, is there a good "rule-of-thumb" or "best practice" regarding which scoring metric (accuracy, precision, recall, AUC, ...) to use based on the dataset's class label "distribution".

Common scenarios I can think of are:

Binary Classification

high imbalance of 0 class
high imbalance of 1 class
roughly equal class quantities
last, imagine a 75/25 ratio for either

Multiclass Classification

Majority of samples are dominated by one class (very common in my experience)
Classes have roughly equally the same amount
Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)

I'm aware of techniques like under-sampling and over-sampling to deal with imbalance by altering the dataset's number of samples, but let's say you didn't in this scenario. Which metrics are best for the above scenarios?

First of all don't change the distribution of your data. Your classifier won't perform good at test time if the real data is not balanced.

If you have approximately equal amount of data for each class and they have same importance, accuracy is a good metric which can help you understand how much your classifier has performed well without regarding to the number of labels, binary or multi-class classificaitons.

Binary classification
There is no difference between 0 or 1 because you yourself choose to assign each class to a label but it is common to set the rare and maybe, in some occasions, the desired output to 1.

In cases which data is distributed badly and it is so much skewed it recommended to use F1 score. For last, imagine a 75/25 ratio for either I personally prefer to use F1 because it is unbalanced.

Also in cases where one class is much important than the other class and maybe the data is skewed, it is better to use recall, although you have to change the cost function to emphasize the importance of one class over the other. As an example, suppose that you want to know whether a patient has a bad illness or not. In such cases you try to predict correctly all of those who has that illness because if you tell that someone has an illness but you're wrong, it will have less harm than saying a patient is healthy. So in these cases recall is so much better than precision.

Multi-Class classification

  Majority of samples are dominated by one class (very common in my experience)

In this case first of all, if the data is so much skewed, I suggest you using anomaly detection. But if you have more than few data, you can again use F1 score.

  Classes have roughly equally the same amount

In this case you can use accuracy if the classes have the same importance. If they don't, again use accuracy but change the cost function to emphasize on the important class.

  Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)

In this case I don't think you will have a good learning. Anyway, you have to use confusion matrix in this for tracking all classes. Although I guess your model will be over-fitted because of the amount of provided data.

Consider that in all cases you can use confusion matrix.

Finally I suggest you taking a look at here and here.


How to convert a SQLContext Dataframe to RDD of vectors in Python? I have a SQLContext data frame derived from pandas data frame consisting of several numerical columns. I want to perform multivariate statistical analysis using the pyspark.mllib.stats package. The statistics function expects a RDD of vectors. I could not convert this data frame into RDD of vectors. Is there a way to convert the data frame?

Code:

 rdd = sqlCtx.createDataFrame(df_new)
 summary = Statistics.colStats(rdd)

I am getting df_new from

 df_new = df.applymap(lambda s: dic.get(s) if s in dic else s) #df is a pandas dataframe

I am getting a PY4JJava error at the summary line. The issue is with the format of rdd.

The Dataframe Python API exposes the RDD of a Dataframe by calling the following :

df.rdd # you can save it, perform transformations of course, etc.

df.rdd returns the content as an pyspark.RDD of Row.

You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.

Note 1: dfis the variable define our Dataframe.

Note 2: this function is available since Spark 1.3


Out of Memory Error when Selecting Data from Redshift Table I am selecting data from Amazon Redshift Table with 500 millions rows. I have 64bit python installed.

code

import psycopg2
from sqlalchemy import create_engine
import pandas as pd
engine = create_engine('postgresql://'username':pwd@host/dbname')
data_frame = pd.read_sql_query('SELECT * FROM table_name ;',
engine)

Everytime I run the code I get a "Out of Memory error". I have 16gb Ram.
I am not sure how to resolve this issue.

Would really appreciate any help on this!
Thanks

First, you are trying to access a big dataset with sqlalchemy, while specialized packages like bigquery would be a more suitable choice. I suggest learning about it on https://www.kaggle.com/learn/intro-to-sql
Also, I think that you get more data than your device can handle. Maybe setting up a limit on your data will help.
data_frame = pd.read_sql_query('SELECT * FROM table_name LIMIT 1000000;', engine)


Understanding The Vertical and Horizontal stack in conditional gated Pixelcnn paper I found some confusion understanding the importance of vertical and horizontal stacks as a solution to the blind spot problem presented in the original pixel cnn architecture discussed in this paper. The vertical and horizontal stacks ideas were presented in the this paper.

Therefore, after browsing, I found this link to explain the concept.
In the vertical stack section of the web page, I still find that pixel f, still cannot see pixels c, d and e.

Any help is much appreciated!!

I also had the same problem when reading the paper, I believe it is not explicitly mentioned.
I think the whole image is padded with zeros on the top row when you are using the vertical stack. In this way, you ensure that the output of any pixel of row i will depend only on the pixels at row i-1 till row 0. But at the same time, you have the advantage of depth of field compositionality and remove the blind spot (given you are stacking enough convolutional layers).


What is the easiest way to identify a gender for a noun (in french)? I am working on an app where in order to process some data, I need to be able to identify the gender for some selected words.
My data is in French.

The feature I am looking for should be able to tell me that "les garçons" is male plural, "chienne" is singular female.

What is the easiest word to do so?
If you have a link to a library do not hesitate.

You could try to apply a French POS tagger, e.g. as suggested here: https://stackoverflow.com/questions/44468300/how-to-pos-tag-a-french-sentence

Note that a POS tagger usually works better from full sentences than isolated words.


what are the next step after ML prediction and how to proceed? I have trained an ML model with a good accuracy but what next?

I am facing difficulty in answering this question, how will you present your model.
Which framework do  you use
How do you make sure the model is learning continuously.
Where can I find definite guide for this approach?

I am good at building and predicting ML, DL & NLP. But stuck to deploy further.

If one understands the business problem you are trying to solve, then this shouldn't be that hard.

You can present the results of your model. Understand your data first, is it skewed, balanced? Understand your model, check the feature importance. Do they make sense with the domain knowledge you have? For example, if your data is highly imbalanced and you have accuracy of 0.99, then it's highly likely that your model is classifying almost everything as one class. Try and make sense of the model, may be try running some shallow models to establish a baseline.
There's no free lunch. You can't have frameworks for interpreting results, you can have a good set of metrics that you can use but metrics change based on how your data is.
Model is learning continuously?? You mean re-training the model. You need to establish a pipeline for it. You can have a python project for this or use DAG mechanisms like AirFlow.
There's no definite guide, a few text books might help but this comes with experience and understanding the concepts of machine learning thouroughly.


Situations where advanced theoretical knowledge of ML helped solve a real world problem? I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works mainly on the applied/business side of things.

By advanced theoretical knowledge, I mean things: Like understanding the details of how the Kernel trick works in SVM, the "no free lunch" theorem and its relationship to machine learning, the details of how neural networks are universal approximators, the VC dimension of a classifier, etc...

By applied Data Scientist, I mean someone who solves business problems using existing algorithms as opposed to someone who develops new ones.

So my question: Are there situations in an applied Data Scientist's life where such theoretical knowledge is useful? Or is this type of knowledge useful only for people who work on developing new algorithms?

At the end of the day, as an applied data scientist you have a bag of tools you can use to solve business problems.  It's helpful to know what your tools are capable of, where each is useful, and also what their limits are.  If you don't have at least some theory, knowing which method to use in which circumstance will seem like a meaningless list of details, and when things break, you may not know why - is there an error in my code, or am I making a fundamental error in applying an algorithm in a scenario where it has fundamental limitations?

You can be pragmatic in applying tools as you've seen them applied before.  Theory will help you have an intuition about whether it's possible or reasonable to apply given tools to new situations that you haven't seen.  It can help you answer questions like: how will this algorithm scale with the size or dimensionality of data?  SVMs in particular suffer from poor scaling with data size, precisely because they rely on kernels.  That kind of basic knowledge is probably more useful than being able to write down the kernel trick.


KNN custom transformer shows same accuracy for every k i set I built custom trasformer for KNN and i can't figure why my k-number, when i set it, always shows same accuracy...

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import StandardScaler
from scipy.spatial import distance

def minkowski_dist(a, b):
    return distance.minkowski(a, b)

class MyKNN(BaseEstimator, TransformerMixin):

    def __init__(self, k, scale_data=False):
        super().__init__()
        self.k = k
        if scale_data is True:
            self.ss = StandardScaler()
        else:
            self.ss = None
        self.X_train = None

    def fit(self, X_train, Y_train):
        self.X_train = X_train
        self.Y_train = Y_train

    def predict(self, X_test):
        prediction = []
        for row in X_test:
            label =  self.closest_n(row)
            prediction.append(label)
        return prediction

    def closest_n(self, row):
        best_dist = minkowski_dist(row, self.X_train[0])
        best_index = 0
        for i in range(1, len(self.X_train)):
            dist = minkowski_dist(row, self.X_train[i])
            if dist < best_dist:
                best_dist = dist
                best_index = i
        return self.Y_train[best_index]

Been doing KNN on iris dataset

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris_dataset=load_iris()
X=iris_dataset.data
Y=iris_dataset.target

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=42, stratify=Y)

print(X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)

And this is what i get:

for k in range(1,21,2):
    knn = MyKNN(k)
    knn.fit(X_train, Y_train)
    prediction = knn.predict(X_test)
    from sklearn.metrics import accuracy_score
    print("Test accuracy for k={}:".format(k),accuracy_score(Y_test, prediction))

You dont use self.k in your closest_n method (for starters).

U dont use it at all, you just initialise the property.

I would check sklearn implementation, just look under the hood here


Individual differences in response time (RT) experiment - searching for the right test Given a distribution of response times to different categorical variables, what's the best way to test for individual differences?

or more specifically:

There are 100 people pressing buttons in 10 colors, with more than 200k presses collected altogether.
I want to test for individual differences in response-time (RT) to the buttons, what's the best way to do this?

(remember there are differences in response time in-between the different colors).

So far I thought of:

ANOVA - calculate a general (weighted) mean RT and compare to individual people.
The distribution level: compare the mean distribution of RT to individual distributions using:

chi-squared for goodness of fit
Kullback–Leibler divergence test
Kolmogorov–Smirnov test

Find a cool bayesian way to do it (which?)
Use a mixed-effect regression model, and calculate the marginal contribution of the people (R square test).

please help me decide - what's the best practice here?
I'd love to learn about another way I didn't think of that would do the trick.

In most individual differences research, subjects are additionally coded along some dimension, such as young/old, low/high working memory, normal/cognitively impaired. Without this, you’re simply looking at individual variation, which doesn’t seem to address any question (what is your research question?). If you simply want to look at variation in overall RT on a subject level, I’d construct CDFs of mean RT collapsing over color, or perhaps for each color separately.  This will simply demonstrate that people differ, and you’ll find the distributions adhere to roughly ex-Gaussian. If you mean to test differences between colors, you can use ANOVA or compare CDFs using K-S tests, but again there isn’t a clear research question here.


How to calculate change in Loss Function w.r.t Weights for a fully connected NN I read the article Optimal Brain Damage by Yann LeCun, John S. Denker and Sara A. Solla where the authors discuss estimating the saliency of each weight of a neural network (which they define by the change in the loss function upon perturbing the weight). The larger the saliency, the more effect the weight has on the learning.
Is there any code/API in Keras or Tensorflow to calculate such "weight saliency" for a simple fully connected neural network (with one or two hidden layers)?

I doubt there is any ready-to-use facility but you can achieve this very easily with some custom code.
You can do -

Train your model(to the extent you want)
Get output using predict method on a dataset
Calculate the Loss using Loss function, y_true and last step output
Get the weight of every layer using Keras get_weights() API
Update weight as per your experiment using Keras set_weights()  API
Calculate the new Loss using the same process as mentioned in step#3

Keras weights API -
You can get weights as a simple array and can update accordingly.
May traverse in the loop on Layers Or access directly using array index.
Some code snippets(See reference links for detail)
Layer.get_weights()
Layer.set_weights(weights)

for layer in model.layers:
    print(layer .get_weights())

weights = model.layers[1].get_weights()[0]
biases = model.layers[1].get_weights()[1]

weights = [np.random.rand(*w.shape) for w in model.get_weights()]
# update
model.set_weights(weights)

References
Keras.IO
Stackoverflow
Stackoverflow
Stackoverflow


Correlation coefficient and non linear association Correlation and Linear Regression

Author:

Lisa Sullivan, PhD

Professor of Biostatistics

Boston University School of Public Health

Says there  may be non-linear association which the correlation  coefficient does not detect. I disagree with this idea. The Karl Pearson correlation coefficient actually computes the degree of relative changes in two variables which may be constant or varying.

Yes, Dr. Sullivan is right. For example, take a perfect quadratic relationship between X and Y.

Here is some Python code to show nine sample points and calculate their Pearson correlation coefficient. You can skip the code and just look at the results below if you trust me.

import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
plt.rcParams["figure.figsize"] = [5, 8]

x = np.arange(-4, 5)
y = x**2  # This is x squared.

plt.scatter(x, y)

plt.axis('equal')
plt.title('$X = Y^2$', fontsize=20)
plt.xlabel('X', fontsize=14)
plt.ylabel('Y', fontsize=14, rotation=0)
plt.xticks([-4, -2, 0, 2, 4])
plt.yticks([0, 1, 4, 9, 16])

# Calculate the Pearson correlation coefficient:
print('r =', np.corrcoef(x, y)[0, 1])

r = 0.0

Surely there is a relation between X and Y in this example, and yet the Pearson correlation coefficient is zero.


Is it possible to do Normalization before Xgboost? Currently I am working on a project which uses Xgboost Regression.
Before putting data into model, I implemented Normalization, the accuracy significantly increased compared with without Normalization.
But I saw someone said that Xgboost doesn’t need Normalization .
Did someone meet the same issue as I met ?
Thanks in advance.

Xgboost is an ensemble algorithm based on decision trees, so doesn't need normalization.
You can check this on Xgboost official github: Is Normalization necessary? and this post What are the implications of scaling the features to xgboost?

I'm new in this algorithm but I'm pretty sure of what I've written


How to use embedding to reduce features for a regression problem I’m working on a regression problem in which I’d like to predict demand of different items. I have used holidays as a feature in my model, in a hot encoded format, i.e. I have 11 binary features each representing one holiday.
I’d like to reduce the size of my features and thinking to used embedding on the holiday features to represent them in lower dimensions.
I’m new to embedding. My question whether it make sense or not and if it does any hints on how to do it?

Doing it is easy. Simply create a mapping between your 11 values and embeddings of any size. Choosing the values for the embeddings is typically done through training a neural network that embeddings are part of. You could use dimensionality reduction techniques like PCA for instance as an alternative.
Now, embeddings only make sense if they represent a feature of dimensionality $D$ with embeddings of dimensionality between 1 and $D-1$. So in your case, I don't know how much you will benefit from the dimensionality reduction itself. You may experience benefits of replacing sparse with dense features, but this needs to be empirically tested.


Combine two sets of clusters I have two sets of topics obtained from two different sets of news paper articles.

In other words, Cluster_1 = ${x_1, x_2, ..., x_n}$ includes the main topics of 'X' news paper set and Cluster_2 = ${y_1, y_2, ..., y_n}$ includes the main topics of 'Y' news paper set.

Now I want to find clusters in the two sets that are similar/related by considering the cluster attributes as given in the example below.

Example 1,
**X1 in Cluster_1** is mostly similar/related to **Y2 in Cluster_2**
**X2 in Cluster_1** is mostly similar/related to **Yn in cluster_2**
and so on.

Example 2:
News about Yet in Cluster_1 is mostly similar/related to News about Science in Cluster_2
News about Floods in Cluster_1 is mostly similar/related to News about Rains in Cluster_2

Since, I am dealing with two separate sets of clusters, what would be a suitable measurement/method I can use to connect the clusters in the two different sets?

To compare two LDA topics, you're really trying to compute the distance between two probability distributions.

One such measure that's commonly used in these circumstances is the Hellinger Distance. To find the closest match for $x_1$ in the topics for $y$, you would calulate the Hellinger Distance between $x_1$ and each $y$ topic, then take the lowest one.

Keep in mind that there's no guarantee whatsoever that the "most similar" topic in this sense would be remotely, subjectively similar.


Extracting data from a data frame with conditions Given the first data frame consisting of 4 columns

libId    studId    year    freq
 1000      3       2002     3
 1000      34      2002     2
 1000      52      2004     9
 1001      17      2003     5
 1001      34      2003     1
.
.
.
.

If I were to extract only data with maximum frequency based on year for each libId this would be the desired output.

 libId    studId    year    freq
 1000      3       2002     3
 1000      52      2004     9
 1001      17      2003     5
.
.
.
.

So far I've tried

data2<- by(data1, c(data1[c(1,2,3)]), function(df) max(df$Freq))

data2 <- as.data.frame(as.table(data2))

But my output is incorrect:

libId    studId    year    freq
 1000      3       2002     3
 1000      34      2002     2
 1000      52      2004     9
 1001      17      2003     5
 1001      34      2003     1
.
.
.
.

There is a good summary on StackOverflow from the user EDi that I'm referring to. Basically, what you want to do is group the values by year and then select the maximum frequency of each group:

# with dplyr
require(dplyr)
df %>% group_by(year) %>% summarise(freq = max(freq))

# data.table
require(data.table)
dt <- data.table(df)
dt[ , max(freq), by = year]

Check out the linked answer for further possibilities.


Is there an encoder which can automatically detect the intrinsic order of an ordinal variable and assign values accordingly? Given data with an ordinal variable, says "house quality" with values ex (excellent), gd (good), fa (fair) and bd (bad), we obviously cannot just throw data into sklearn's LabelEncoder as the resulting labels can be in wrong order, e.g. {bd: 3, gd: 2, fa: 1, ex:0}. Instead, we need to manually specify an order, right? However, if we do not have domain knowledge, how can we specify the order? Also a manual way is usually prone to error. Thus, I am curious if there is any encoder which can auto detect the correct order in an ordinal variable?

Yup, it should be target encoding. By calculating the target mean for each category, you are ordering the categories based on the target, I don't think of a best way to order them. Of course, this is only valid in a supervised learning setting. If not, I can't think of a way of automatically ordering categories.
See this question and this blogpost to dig deeper in target encoding.


What is the difference between Kafka Channel and Kafka Broker? I am a beginner in Apache Kafka. I am reading now some architecture documents and I find in a diagram 2 components:

Kafka Channel
Kafka Broker

What is the difference between these 2 components?

What is a Kafka Broker?
Kafka is a distributed streaming platform. It is run as a cluster on one or more nodes, and each node in the cluster is called a Kafka Broker.
What is a Kafka Channel?
A Kafka Channel is a type of Flume Channel. Flume is:

Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store.
The use of Apache Flume is not only restricted to log data aggregation. Since data sources are customizable, Flume can be used to transport massive quantities of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source possible.

A Channel is:

Channels are the repositories where the events are staged on a agent.

So hopefully, it is now clear that a Kafka Channel, is a Flume Channel that uses Kafka.


Understanding Imagenet training I am trying to start training Imagenet classification training using Tensorflow's inception model. I am a bit confused as I am not sure how to fully train the model.

Wherever I go everyone seems to be using the 1000 classes training list. However, while digging in tensorflows inception model files I found a list of labels containing 21000+ classes.

So I am a bit confused as to whether I should train using the 1000 classes that everyone seems to be using or should I train on the 21000 classes. I have the full imagenet data and I checked the images I downloaded and almost all 21000 classes are there (missing only 200 classes).

I really want to train against the full imagenet dataset as this should give more accuracy but I am not sure what the downfalls will be or how to properly prepare the data and how many steps are appropriate for that.

The problem with 1000 classes is the ILSVRC2012 challenge which is using a subset of the full imagenet with "only" 1000 classes. The advantage of that is that it is fixed while imagenet itself is always expanding and new classes and examples may be added.

Training on the full ImageNet set is a very long task and will require either a very long time or a long time and a lot of computing powers (talking tens of GPUs). Although it's not short, training ILSVRC2012 can be done in one day up to a few days depending on your hardware resources and complexity of the model being training.


Automatic topic labelling for topic modelling I am just curious to know if there is a way to automatically get the lables for the topics in Topic modelling. It would be really helpful if there's any python implementation of it.

Topic modelling is an unsupervised task, so by definition there is no gold-standard label. The task is a kind of clustering, i.e. it tries to group together documents with similar topics, but it doesn't label the groups.
Instead people usually use the words which are the most associated with a topic by the model as a kind of description for the topic.


NCHW vs NHWC in Machine Learning As I've been introducing myself to the various deep learning frameworks, I've noticed a difference in the default placement of channels for images.  Is there a substantial difference between NCHW vs NHCW layout?  Why would I choose one over the other?

The only difference is when you need to carry out certain operations across the channel axis such as BatchNormalisation. If you look at https://keras.io/layers/normalization/, one of the parameters for batch normalisation is as such:

  axis: Integer, the axis that should be normalized (typically the
  features axis). For instance, after a Conv2D layer with
  data_format="channels_first", set axis=1 in BatchNormalization

You need to provide the axis as the model needs to know to carry out normalisation across which axis. Other than that, it doesn't make a difference in training.


R: machine learning on GPU Are there any machine learning packages for R that can make use of the GPU to improve training speed (something like theano from the python world)?

I see that there is a package called gputools which allows execution of code on the gpu, but I'm looking for a more complete library for machine learning.

As for a complete machine learning package on GPU's, no such package exists.  However, there are actually a handful of R packages that can use GPU's.  You can see these packages on the CRAN High Performance Computing page.  You should note that most of these packages do require you to have a NVIDIA card.  Of the packages available, there are three packages you most likely would utilize unless you have a special case.

gputools - if interested in distance computations (only NVIDIA).
gmatrix - general numeric computations (only NVIDIA).
gpuR - general numeric computations (any GPU via OpenCL).*

* NOTE - At the risk of self promotion I am the author of the gpuR package.

You can likely use the latter two packages to reproduce existing machine learning algorithms.  I am actually using my gpuR package to create a GPU accelerated neuralnet package but this is in progress.

So in summary, if you are determined, the basic resources are available in R.  But if you need something in the immediate future, you will need to explore other resources/approaches as pointed out by @YCR.


Generate timeseries data Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from the same image.

My question is: how do you generate fake time series data?

ATM I know of TSimulus and TimeSynth to generate data programatically in a controlled manner (instead of generating random data).
TSimulus allows to generate data via various generators.
TimeSynth is capable of generating signal types
Harmonic functions(sin, cos or custom functions)
Gaussian processes with different kernels
    Constant
    Squared exponential
    Exponential
    Rational quadratic
    Linear
    Matern
    Periodic
Pseudoperiodic signals
Autoregressive(p) process
Continuous autoregressive process (CAR)
Nonlinear Autoregressive Moving Average model (NARMA)

and noise types
White noise
Red noise

If you are looking for a graphical way to generate data TimeSeriesMaker is the only tool able to do this.


What is the difference between and Embedding Layer and an Autoencoder? I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what are the differences between them?

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data)

Autoencoder is a type of neural network where the inputs and outputs are the same but in the hidden layer the dimensionality is reduced in order to get a more dense representation of the data.

Word2vec contains only 1 hidden layer but the inputs are the neighborhood words and the output is the word itself (or the other way around). So it cannot be an autoencoder cause the inputs and outputs are different.

Embedding layer is only a "simple" layer in a neural network. You can imagine it as a dictionary where a category (i.e word) is represented as a vector (list of numbers). The value of the vectors are defined by backpropagating the errors of the network.


How to implement random cropping during training? I'm developing a U-net like model which segments the damaged tissue of the brain between two time-points in Multiple Sclerosis patients. The model is given the baseline and follow-up images as x and the segmentation mask as y. The images are 3D (192, 218, 192) and the model input size is (128, 128, 128) and is being trained by the dice loss. The model is based on this paper: https://www.sciencedirect.com/science/article/abs/pii/S0895611120300732
What I'm currently doing is a centre crop for each image before the training, and the training dice score seems to be learning properly, but in the validation data it is not.
I have read that random cropping 10 times or so helps to reduce overfitting, but I don't exactly know how to implement it. I have written this function to do the random cropping:
def random_crop(img_bl, img_fu, mask, width=128, height=128, depth=128):
    x = random.randint(0, img_bl.shape[1] - width)
    y = random.randint(0, img_bl.shape[0] - height)
    z = random.randint(0, img_bl.shape[2] - depth)
    img_bl = img_bl[y:y + height, x:x + width, z:z + depth]
    img_fu = img_fu[y:y + height, x:x + width, z:z + depth]
    mask = mask[y:y + height, x:x + width, z:z + depth]

    return img_bl, img_fu, mask

Do I just apply this function 10 times before the training for each image? Or is there a way to include the random cropping inside the model and overlap the predictions of these 10 subvolumes?
Any tips, links to articles or insights are greatly appreciated. Thanks!

For segmentation, random cropping cannot be included inside the model definition, since random crop of same state also has to applied on mask. I suggest before each epoch, apply the 'random_crop' function on the entire dataset and shuffle(very important) the dataset.


Is this over-fitting or something else? I recently put together an entry for the House Prices Kaggle competition for beginners. I decided to try my hand at understanding and using XGBoost.

I split Kaggle's 'training' data into 'training' and 'testing'. Then I fit and tuned my model on the new training data using KFold CV and got a score with scikit's cross_val_score using a KFold with shuffle.

the average score on the training set with this cross validation was  0.0168 (mean squared log error).

Next, with the fully tuned model, I check its performance on the never before seen 'test' set (not the final test set for the Kaggle leader board). The score is identical after rounding.

So, I pat myself on the back because I've avoided over-fitting... or so I thought. When I made my submission to the competition, my score became 0.1359, which is a massive drop in performance. It amounts to being a solid 25 grand wrong on my house price predictons.

What could be causing this, if not overfitting?

Here is the link to my notebook, if it helps: https://www.kaggle.com/wesleyneill/house-prices-walk-through-with-xgboost

You've followed the right process well. (Well it's possible there's an error, like not randomly sampling the test set.)

I think the issue is simply that you have nevertheless overfit. The Kaggle held-out test set may not be, due to chance, that like the provided training data.  There's not a lot you can do except favor low-variance models more than low-bias models in your selection process.


Software for reweighted L1 minmization? I am trying to solve a sparsity-promoting optimization problem. It is well known that the L1 norm is a good surrogate to the L0 norm, and it is studied in (Candes et al, 2008: Enhancing sparsity by reweighted L1 minimization https://arxiv.org/abs/0711.1612) that a better approximation can be obtained by successively reweighting the L1 norm penalization term, where the result of a step gives the new weights for the next iteration.
So far I am implementing this directly in my code but I wonder if there is any software/optimization package that does this automatically.

I am not sure that it will answer the question exactly, but it appears that the glmnet package (R : glmnet, Python : glmnet) do something similar : enforcing sparsity trough iteration.
A small exemple of what you can get : coefficients as a function of the L1 norm. You can see the number of non-zero coefficients (degrees of freedom) evolving .


Sequence extraction in a dataset I am looking for a way to extract sequences/patterns from a dataset such as this one:

dataset = ['sample1', 'sample2', 'sample3', 'sample1', 'sample2', 'sample3', 'sample3', 'sample2'...]

And my goal is to know that the sequence ['sample1', 'sample2', 'sample3'] occurs 2 times in this dataset.
Ideally, I would also like to know all sequences that occur more than once in my dataset.

Is there a library (sklearn...) that could help me do that or do I just have to iterate over my dataset and test each and every possible combination? I assume there must be a more intelligent way to do that.

Thanks for your help!

You can use nltk.util.ngrams for ngram extraction.
See an example below:

To extract bigrams:

dataset = ['sample1', 'sample2', 'sample3', 'sample1', 'sample2', 'sample3',\
'sample3', 'sample2', 'sample2', 'sample3', 'sample1', 'sample2']

from nltk.util import ngrams
import collections

bigrams = ngrams(dataset, 2)
result = collections.Counter(bigrams)
result.most_common()

Out[1]:
[(('sample1', 'sample2'), 3),
 (('sample2', 'sample3'), 3),
 (('sample3', 'sample1'), 2),
 (('sample3', 'sample3'), 1),
 (('sample3', 'sample2'), 1),
 (('sample2', 'sample2'), 1)]

To extract trigrams:

trigrams = ngrams(dataset, 3)
result = collections.Counter(trigrams)
result.most_common()

Out[2]:
[(('sample1', 'sample2', 'sample3'), 2),
 (('sample2', 'sample3', 'sample1'), 2),
 (('sample3', 'sample1', 'sample2'), 2),
 (('sample2', 'sample3', 'sample3'), 1),
 (('sample3', 'sample3', 'sample2'), 1),
 (('sample3', 'sample2', 'sample2'), 1),
 (('sample2', 'sample2', 'sample3'), 1)]

Fourgrams:

fourgrams = ngrams(dataset, 4)
result = collections.Counter(fourgrams)
result.most_common()

Out[3]:
[(('sample2', 'sample3', 'sample1', 'sample2'), 2),
 (('sample1', 'sample2', 'sample3', 'sample1'), 1),
 (('sample3', 'sample1', 'sample2', 'sample3'), 1),
 (('sample1', 'sample2', 'sample3', 'sample3'), 1),
 (('sample2', 'sample3', 'sample3', 'sample2'), 1),
 (('sample3', 'sample3', 'sample2', 'sample2'), 1),
 (('sample3', 'sample2', 'sample2', 'sample3'), 1),
 (('sample2', 'sample2', 'sample3', 'sample1'), 1)]
 ...

You only need to specify the length of your n-grams and what number of repetitions is representative in your case.

Hope this helps!


How can I use a string as input in a neural network? I'm making a program that can determine if a user will like a car from different auctions based on the cars that he/she has bought in the past. Therefore, I want to use the make and model (which are represented as strings) as well as the year, mileage etc... but I'm having trouble to decide on how I'll pass them to the neural network

I've been thinking of using a vector of different makes as such:

 car_make = [[10000]  # Nissan
             [01000]  # Toyota
             [00100]  # Ford
             [00010]  # Dodge
             [00001]] # Mazda

I could pass those numerical values as input but making this is limited as there are tons of different makes, since I'm looking for cars in many different auctions. I have no idea how I can represent the models too as they also change in every auction.

You have any ideas on how I can pass the strings onto the neural network?

p.s I'm using python to build the neural network.

there is no other way i guess. since each one is of different type(Categorical varaible) you can only one hot encode them.


Understanding output probabilites of xgboost in multiclass problems I would like to understand the output probabilities of a xgboost classifier (or any other decision tree ensemble based classifier) in the case of a multiclass problem.
For example: We have 5 different classes and a trained model on some data belonging to those classes. I would expect, given some test data, the model would give output probabilities of up to 100% when it is really sure the data belongs to some class. Obviously the probabilites still have to add up to 1.
On the contrary I was told that in the case of 5 different classes, the model would be already sure when outputting 20% (100/5) for the data. I also see this in a problem I am facing, the probabilities are never higher than 30%.
I would like to understand this apparent fact, ideally pointing me to a source where this is explained.

What you were told is a worst case scenario. With 5 labels, 20.01% is the lowest possible value that a model would need to choose one class over the other. If the probability for each of the 5 classes are almost equal then the probabilities for each would be approximately 20%. In this case, the model would be having trouble deciding which class is correct.
About the 30% comment, while it seems low, keep in mind that the model mainly uses the probability to classify the most likely label, calculating a highly accurate probability value is not the first priority.
Here is a good reference. This paper discusses a few of the problems with probability accuracy and mentions some theories on how to improve the accuracy:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.381.4254&rep=rep1&type=pdf
It mentions (see excerpts below) how a probability estimation in classification is a complex and difficult problem, and that some models can be very accurate at classification but poor at probability estimation, and how multi-class models are much more difficult than a binary classification model.
Excerpt 1:
"Probability estimation is known to be a quite hard problem, especially in comparison to standard classification. This comes at no surprise, noting that proper probability estimation is a sufficient but not necessary condition for proper classification: If the conditional class probabilities (1) are predicted accurately, an optimal classification can simply be made by picking the class with highest probability"
Excerpt 2:
"More generally, the Bayes decision can be taken so as to minimize any loss
in expectation. On the other hand, a correct classification can also be obtained
based on less accurate probability estimates. In fact, the classification will remain correct as long as the estimated probability is highest for the true class. Or, stated differently, an estimation error will remain ineffective unless it changes the result of the arg max operation in (2). This is also the reason for why methods like naive Bayes show competitive performance in classification despite producing relatively inaccurate probability estimates [7]."
Excerpt 3:
Methods like naive Bayes and decision trees are multi-class classifiers and
can in principle be used to produce probability estimates in this setting. In
practice, however, one often prefers to estimate probabilities in the two-class
setting, especially because estimating a single probability (of the positive class) is much simpler than estimating K − 1 probabilities simultaneously
Example with probabilities close to 1:
from sklearn import datasets
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
iris = datasets.load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)
from sklearn.tree import DecisionTreeClassifier
dtree_model = DecisionTreeClassifier(max_depth = 2).fit(X_train, y_train)
dtree_predictions = dtree_model.predict(X_test)

print(dtree_model.predict_proba(X_test) )
[[0.         0.02564103 0.97435897]
[0.         0.91666667 0.08333333]
[1.         0.         0.        ]
[0.         0.02564103 0.97435897]
...