How to predict on part of image after training on other part of image? I have images of identity cards (manually taken so not of same size) and I need to extract the text in it.

I used tesseract to predict bounding boxes for each letter and am successful to some extent but some letters are not bounded.

So, I have around 5000 bounding boxes in all images combined.

I want to train it so as to predict bounding boxes for remaining letters.

After predicting the bounding boxes I will try to classify the image into characters.

This is different from conventional machine learning problem where I donot have training and testing data separately.

As you seem to plan to build your own character classifier, I assume that you are happy to do some programming but just want to avoid heavy graphics stuff. A practical approach would be to use tesseract's bounding boxes just as a guide. Look to the left and right for more characters. Knowing where the rows are and what height and width the characters are is a good basis. An extension of rows gives you the upper and lower boundaries of the new candidate boxes. To find the left and right boundaries, you can project the pixels to the baseline of the row and look for positions with a low number of black pixels. A second factor for the boundary decision can be the expected width, which should be somewhere between the width for "I" and "W". A third factor can be the letter/number that your classifier recognises and how well this fits into the context (e.g. use character language model).


How does class_weight work in Decision Tree? I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that there are a lot of algorithms Cost Sensitive Decision Tree. So what exactly does class_weight do in Decision Tree?

It is used, for example, when classes are imbalanced, so different weights are assigned to different classes, instead of equal ones.
Another case is when some class is more significant than others, so loss wrt this class counts more.
The class_weight parameter (eg for decision tress) is used by giving different weight to different class samples (doc: https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html, src: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/class_weight.py#L11), which is then used to position the sample accordingly.
Note that class_weight can be used in different ways depending on algorithm and model used. What makes sense for each algorithm. Mathematicaly, usually, is a simple multiplication of the sample value in some loss function
Related: How does the class_weight parameter in scikit-learn work?.
For CARTs with Gini Criterion:
The original gini impurity is defined as:
$${\displaystyle \operatorname {I} _{G}(p)=\sum _{i=1}^{J}\left(p_{i}\sum _{k\neq i}p_{k}\right)=\sum _{i=1}^{J}p_{i}(1-p_{i})=1-\sum _{i=1}^{J}{p_{i}}^{2}}$$
If classes are assigned weights $w_i$ then the weighted gini impurity is computed as follows:
the weight of all the observations in a potential child node, $c$, is
$$t_c = \sum_i w_i * n_i$$
where $n_i$ is the number of observations of class $i$ in $c$, and $w_i$ is the weight assigned to class $i$.
The impurity of child node $c$ is then
$$i_c = 1 - \sum_i (\frac{w_i * n_i}{t_c})^2$$
where $n_i$ is again the number of observations of class $i$ in the node, $w_i$ is the weight assigned to the class and $t_c$ is as calculated previously. The impurity of the entire potential split is then
$$\sum_c \frac{t_c}{t_p} * i_c$$
where $t_c$ and $i_c$ are as calculated previously, and $t_p$ is the total weight of all observations in the parent node that is being split.
Reference: How is the Weighted Gini Criterion defined?


In elbow curve how to find the point from where the curve starts to rise? I am computing a distance metric on my data. The result is then being sorted in ascending order.
The samples having distance more than a specific threshold are to be marked as outliers and will be discarded. Below is a plot of all distance values.

As evident from the graph, after a certain point, the graph rises quite rapidly and even the datapoints get sparse. I need to calculate that point from where this happens and mark that point as threshold value.

TL;DR

Use the two functions from below to get the index of the elbow:

elbow_index = find_elbow(data, get_data_radiant(data))

Edit: I put all of the code below into a python package called kneebow. Now, you can simply do it like this:

from kneebow.rotor import Rotor

rotor = Rotor()
rotor.fit_rotate(data)
elbow_index = rotor.get_elbow_index()

Long Answer

If this curve is representative for all of the curves (e.g. unimodal and continuous) then a quick and dirty method is to rotate it to some degree and simply take the minimum value.

The rotation can be done by multiplication with the rotation matrix

$$\left(
\begin{array}{cc}
\cos\theta&-\sin\theta\\
\sin\theta&\cos\theta
\end{array}
\right)$$

where $\theta$ is the desired angle in radians.

In python, you can do this with the following function:

def find_elbow(data, theta):

    # make rotation matrix
    co = np.cos(theta)
    si = np.sin(theta)
    rotation_matrix = np.array(((co, -si), (si, co)))

    # rotate data vector
    rotated_vector = data.dot(rotation_matrix)

    # return index of elbow
    return np.where(rotated_vector == rotated_vector.min())[0][0]

Note that theta is the angle in radians. You can caculate that by np.radians(angle).

Important: One thing to remember is that the x- and the y-axes may have different scales. So on your plot, it may look like a 45° rotation would be enough, while actually, it is not. Therefore, you can use the following function to calculate which radiant you should use. It takes the slope from the minimum to the maximum values in your data and converts it to radians:

def get_data_radiant(data):
  return np.arctan2(data[:, 1].max() - data[:, 1].min(),
                    data[:, 0].max() - data[:, 0].min())

If you want to get the angle, run np.rad2deg(get_data_radiant(data)).

Example

How to use

Let's test the approach on a sample data similar to yours:

# Let's define our sample data:
data = np.array([
    [1, 1],
    [2, 2],
    [3, 3],
    [4, 4],
    [5, 5],
    [6, 6],
    [7, 7],
    [8, 8],
    [9, 16],
    [10, 32],
    [11, 64],
    [12, 128],
    [13, 256],
    [14, 512]
])
# y is linear until (8,8) and increases exponentially afterwards

plt.scatter(data[:, 0], data[:, 1])

Plotting the data gives us the following figure:

Now, let's try to find the elbow by combining the functions from above:

elbow_index = find_elbow(data, get_data_radiant(data))

print(elbow_index)        # 10
print(data[elbow_index])  # array([11, 64])

In Detail

To sum it all up, we just calculated the slope from the min to the max value and then rotated the plot in such a way that the slope was zero. Subsequently, we took the min value of the data to get the elbow.

We can get the angle of rotation the following way:

angle = np.rad2deg(get_data_radiant(data))
print(angle)  # 88.543

The plot on the left has the slope included as an orange line. The scales of the axes make it look like it would have an angle of 45° while in reality, it has an angle of 88.5°! After vector rotation, the data looks like the plot on the right. From this data, we took the minimum value which was the 11th data point.

Drawbacks

Note that this method has a drawback: The more inequal the scales of your axes are, the more it will choose points in favor of the larger axis.

You can try to use the scikit-learn MinMaxScaler to scale your data before you use this method in order to reduce this effect. If you use the kneebow package, the data gets scaled by default.


Reformulating the maximal margin classifier optimization problem Ok, so I've been trying to read up on how SVM's work and started with maximal margin classifiers. At page $132$ in ESL (Elements of Statistical Learning) the authors "reformulates" the optimization problem but I can't seem to understand what they are doing from $(4.47)$ to $(4.48)$. Does anyone know?

Here is an excerpt:

Edit: I guess, what I don't understand is why we can arbitrarly set the magnitude of beta to $\frac1M$. What does a positively scaled multiple mean in this case? Just a multiple larger than $0$?

If $(\beta, \beta_0)$ satisfies the inequality $(4.47)$, then for any positive $k$, $k>0$, $(k\beta, k\beta_0)$ would satisfies the inequality as well.

Also, $(\hat{\beta}, \hat{\beta}_0)=\left( \frac{\beta}{M\|\beta\|}, \frac{\beta_0}{M\|\beta\|} \right)$ satiesfies the inequlity as well since

$$y_i(x_i^T\hat{\beta}+\hat{\beta}_0)=\frac{y_i}{M\|\beta\|}(x_i^T\beta+\beta_0) \ge \frac{M\|\beta|}{M\|\beta\|}=1=M\|\hat{\beta}\|$$

Let's find $\hat{\beta}$ and $\hat{\beta}_0$ directly. Note that $\hat{\beta}$  satisfies the property that $\|\hat{\beta}\|=\frac1M.$

Hence we let $\|\hat{\beta}\|$ to be $\frac1M$, hence reducing the inequality to be

$$y_i(x_i^T\hat{\beta} + \hat{\beta}_0) \ge 1$$
$$\|\hat{\beta}\|=\frac1M$$

We want to maximize $M$, hence, we minimize $\frac1M$, which is equivalent to minimizing $\frac12\|\hat{\beta}\|^2$.


How to increase model's test accuracy? I am using the InceptionV3 model for training. Here is the link for the code (https://github.com/maxmelnick/tensorflow/blob/no_random/tensorflow/examples/image_retraining/retrain.py) Initially I have a small size dataset. So, I used the augmentation technique to increase the size of the dataset.
While training phase dataset was divided into training, validation, and testing. During the training phase, it shows 96% accuracy for 11 classes. But When I predict any new input image(Unseen data) it gave 56% accuracy. What's the problem lies with the model?
I have already used Dropout, Cross-validation, OverSampling techniques but not achieved good results over the new input image.
Parameters used while training.
Training Samples - 800 images in each class

Training Samples - 70%
Validation Samples - 20%
Testing Samples - 10%

Testing Samples (Unseen data other than Training Samples) - 51 images in each class
Epochs - 10,000
Thank You.

While Yolov3 is a great model it may underperform for certain datasets. A simple example would be this. Take a model and train it on cifar 10. The same model might significantly perform lower on cifar 100 or imagenet!
Things you could do to improve your model:

Introduce Batch or group normalization?
Study the data better, if possible see if the data is used elsewhere and how they approach it.
Epochs isn't the problem, so that should stay the same.

If nothing continues to work then you might have to opt for some other model. Try yolov5 or rcnn!


Probability of event given two depandant events Is there anyway, to compute the probability of the event given two dependent events?
I know that Bayes can help if those events are independent, but what if condition events are dependent?

Let $A$, $B$, $C$ be three (potentially dependent) events. By the probability chain rule:
$$ P(A,B,C)=P(A|B,C)P(B,C)=P(A|B,C)P(B|C)P(C) $$
So we can do
\begin{align}
P(A|B,C)
&= \frac{ P(A,B,C)}{P(B|C)P(C)} \\
&= \frac{ P(B,C|A) P(A) }{ P(B|C)P(C) } \\
&= \frac{ P(B|A,C) P(A) P(C) }{ P(B|C)P(C) } \\
&= \frac{ P(B|A,C) P(A) }{ P(B|C) }
\end{align}

See this related post and this one as well.


A Text Sections Classifier I have a database of text files and I would like to classify each section(i.e references, abstract, etc...) in the text.

For example:

  Abstract Indexes: 100-300

  References Indexes: 9000-10000

Is there a package that is doing something like that?

Is there a recommended approach for doing so?

You might try to use the metadata of the individual sections for classification:

Examples:

Length -> Abstracts are usually short.
Existence of Figures -> Do not usually appear in abstract of references
Relative Abundance of quotations. The references should have a significant higher share of reference keys than the others.

You could then use the metadata as input for any supervised learning approach. For example a Neural Network or a (boosted) decision tree.


How to create US state choropleth map I have a value associated with each US state (let's pretend it's the average temperature in January for each state). I want to display this data as a heat map of the United States. To be clear, it would be a map of the US with each state having a color from a color gradient that corresponds to a quantitative value. I don't know what this plot type is called, so I'm having trouble figuring out how to do this in MATLAB or another program, but does anyone know an easy way to do this? I have a vector of data (of length 50) and just need to make a US map plot.

Plotly in Python provides a nice solution:
!pip install chart_studio
from chart_studio import plotly as py
import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_us_ag_exports.csv')

for col in df.columns:
    df[col] = df[col].astype(str)

scl = [[0.0, 'rgb(242,240,247)'],[0.2, 'rgb(218,218,235)'],[0.4, 'rgb(188,189,220)'],\
            [0.6, 'rgb(158,154,200)'],[0.8, 'rgb(117,107,177)'],[1.0, 'rgb(84,39,143)']]

df['text'] = df['state'] + '
' +\
    'Beef '+df['beef']+' Dairy '+df['dairy']+'
'+\
    'Fruits '+df['total fruits']+' Veggies ' + df['total veggies']+'
'+\
    'Wheat '+df['wheat']+' Corn '+df['corn']

data = [ dict(
        type='choropleth',
        colorscale = scl,
        autocolorscale = False,
        locations = df['code'],
        z = df['total exports'].astype(float),
        locationmode = 'USA-states',
        text = df['text'],
        marker = dict(
            line = dict (
                color = 'rgb(255,255,255)',
                width = 2
            )
        ),
        colorbar = dict(
            title = "Millions USD"
        )
    ) ]

layout = dict(
        title = '2011 US Agriculture Exports by State
(Hover for breakdown)',
        geo = dict(
            scope='usa',
            projection=dict( type='albers usa' ),
            showlakes = True,
            lakecolor = 'rgb(255, 255, 255)',
        ),
    )

fig = dict( data=data, layout=layout )

url = py.plot( fig, filename='d3-cloropleth-map' )


How do we standardize arrays with NaN? I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there other methods?

My search results came up with a solution

df['col']=(df['col']-df['col'].min())/(df['col'].max()-df['col'].min())

But this works only with panda dataframes (they have column names). Is there a way to implement column headers in the matrix?

import pandas as pd
import numpy as np
import random
import sklearn.preprocessing import StandardScaler

data = pd.DataFrame({'sepal_length': [3.4, 4.5, 3.5],
                     'sepal_width': [1.2, 1, 2],
                'petal_length': [5.5, 4.5, 4.7],
                'petal_width': [1.2, 1, 3],
                    'species': ['setosa', 'verginica', 'setosa']})

#Shuffle the data and reset the index
from sklearn.utils import shuffle
data = shuffle(data).reset_index(drop = True)

#Create Independent and dependent matrices
X = data.iloc[:, [0, 1, 2, 3]].values
y = data.iloc[:, 4].values

#train_test_split
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1, random_state = 0)

#Impute missing values at random
prop = int(X_train.size * 0.5) #Set the % of values to be replaced
prop1 = int(X_test.size * 0.5)

a = [random.choice(range(X_train.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
b = [random.choice(range(X_train.shape[1])) for _ in range(prop)]
c = [random.choice(range(X_test.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
d = [random.choice(range(X_test.shape[1])) for _ in range(prop)]
X_train[a, b] = np.NaN
X_test[c, d] = np.NaN

This is where I get the error :Input contains NaN, infinity or a value too large for dtype('float64').

from sklearn.preprocessing import StandardScaler #importing the library that does feature scaling

sc_X = StandardScaler() # created an object with the scaling class

X_train = sc_X.fit_transform(X_train)  # Here we fit and transform the X_train matrix
X_test = sc_X.transform(X_test)

This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform:
https://scikit-learn.org/stable/whats_new/v0.20.html#id37 (fourth bullet)
https://github.com/scikit-learn/scikit-learn/issues/10404


Can BRAT be used for text classification annotation? BRAT (brat rapid annotation tool) can be used for named-entity annotation:

Can BRAT be used for text classification annotation? I.e., given the text, annotate whether it belongs to some classes?

According to their documentation , BRAT does a lot of things, but text classification isn't one of them. BRAT is "too powerful" for that. I'd recommend you use a tool like Prodigy instead.

This should do what you're looking for.


Naïve Bayes and Training Data I am creating my own implementation of a Naïve Bayes classifier. While it’s behaviour and functionalities are clear to me, my concerns are on the nature of the training and testing data.

I acquired several sets of product reviews from Amazon. The first thing I do is parsing them, that is, taking the rating (1 to 5 stars) and the text, which I parse with a regex to only contain alphabetical lowercase characters and spaces. Next, I convert ratings to polar values, so 1 and 2 stars become “-“ and 4 and 5 stars become “+”. I’m intentionally skipping reviews with 3 stars; could this be an issue?

Here come my real concerns. When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews (such as 7 positive and 7 negative reviews for training and 3 positive and 3 negative reviews for testing)? Right now I’m acquiring as many positive as negative reviews from the chosen set, but I’m wondering if that should be the case. For instance, if a set contains 7 positive reviews and 4 negative ones, I discard 3 positive reviews to equate them.

Furthermore, I observed that negative reviews tend to contain longer texts on average. So, if I’m using an equal number of positive and negative reviews, but they differ on average text length, would this have an impact on the way my classifier tries to predict?

I’m intentionally skipping reviews with 3 stars; could this be an issue?

It's not a problem, it's just your definition of your question. It assumes rate 3 as neutral and blow/above that as negative/positive. One may say I assume every rate is positive so each number is a level of positiveness then rate 3 works for him. The main point is that you get the answer for the question you define so be careful about what you want from your classifier to set up your question correctly.

  When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews?

Nice question. Before specifically taking Naive Bayes into account, it is a general machine learning problem when the population of classes are imbalanced. If this is the case then better to balance them, not only in train/test split but also during train itself as the dominating class will bias your result. If the classes are not that imbalanced then you can split things randomly and it's fine. Regarding Bayes Classifier itself the balance inside training set should be more important as NB learns from the statistics of your training set. On test set, it just uses the already learned statistics so portion of classes should not impact.

  I observed that negative reviews tend to contain longer texts on average. would this have an impact on the way my classifier tries to predict?

Again nice question. Yes it does but that is not necessarily that bad. Longer text means that the probability of having larger number for term frequencies are higher. This is a feature itself! (Just think again that you already found a difference between classes. Well ... you want NB to do the same right?!!) The only thing is that if the difference in lengths of comments is very huge, then it might affect the value of features in a way that you need normalization (e.g. instead of counting terms as features, you can use TF-IDF which is a bounded score)

Hope it helps. Good Luck!


How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice? Let's say I used minmaxscaler while creating my model.
Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

I just noticed that none of the answers contained the most important instruction. So here is how it's done since i already resolved this a long time ago.
from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
                ('normalization', MinMaxScaler()),
                ('classifier', RandomForestClassifier())
            ])
pipeline.fit(X)

The pipeline.fit() was missing from other answers which is very important. Once scikit function has learned the parameters of your training data X , you can dump that pipeline into a file and do the rest like this.
joblib.dump(pipeline, 'transform_predict.joblib')
pipeline = load('transform_predict.joblib')
transformed_data = pipeline.transform(new_data)

This will apply the same transformations to a dataset (even if containing 1 sample) that was done on your training dataset.
Update
for newer version, I believe one has to read. I hit the memory limit and the following change helped.
with open(conf['le_embedder_file'], 'rb') as f:
    pipe_le = joblib.load(f)

and write
with open(conf['tecv_embedder_file'], 'wb') as f:
    joblib.dump(pipe_tecv, f, compress='zlib')

I had to read


unable to predict by LinearRegression Should I add csv as text in SO question? There's lot more data.
%matplotlib inline
plt.xlabel('Year')
plt.ylabel('Income($US)')
plt.scatter(df.year,df.income,color='red',marker='+')

reg = linear_model.LinearRegression()
reg.fit(df[['year']],df.income)

Output : LinearRegression()

reg.predict('10000')

--------------------------------------------------------------------------- ValueError                                Traceback (most recent call
last)  in
----> 1 reg.predict('10000')
~/.local/lib/python3.9/site-packages/sklearn/linear_model/_base.py in
predict(self, X)
236             Returns predicted values.
237         """
--> 238         return self._decision_function(X)
239
240     _preprocess_data = staticmethod(_preprocess_data)
~/.local/lib/python3.9/site-packages/sklearn/linear_model/base.py in
decision_function(self, >X)
218         check_is_fitted(self)
219
--> 220         X = check_array(X, accept_sparse=['csr', 'csc', 'coo'])
221         return safe_sparse_dot(X, self.coef.T,
222                                dense_output=True) + self.intercept
~/.local/lib/python3.9/site-packages/sklearn/utils/validation.py in
inner_f(*args, **kwargs)
61             extra_args = len(args) - len(all_args)
62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
64
65             # extra_args > 0
~/.local/lib/python3.9/site-packages/sklearn/utils/validation.py in
check_array(array, >accept_sparse, accept_large_sparse, dtype, order,
copy, force_all_finite, ensure_2d, allow_nd,

ensure_min_samples, ensure_min_features, estimator)
628             # If input is scalar raise error
629             if array.ndim == 0:
--> 630                 raise ValueError(
631                     "Expected 2D array, got scalar array instead:\narray={}.\n"
632                     "Reshape your data either using array.reshape(-1, 1) if "

ValueError: Expected 2D array, got scalar array instead: array=10000.
Reshape your data either using array.reshape(-1, 1) if your data has a
single feature or >array.reshape(1, -1) if it contains a single
sample.

I am not sure why I am getting above error. I have income lists of some years. So, I graph it. When I was trying to predict a data from linearRegression I got the error. I am new to ML(Machine Learning)
How to solve it? What am I missing?

The input to the predict method should be a 2d array of shape (n_samples, n_features), which in your case with one feature and one sample would be (1, 1). Therefore add a second dimension for the number of samples: reg.predict([['10000']]).


Capture pattern in python I would like to capture the following pattern using python
   anyprefix-emp-<employee id>_id-<designation id>_sc-<scale id>

Example data

strings = ["humanresourc-emp-001_id-01_sc-01","itoperation-emp-002_id-02_sc-12","Generalsection-emp-003_id-03_sc-10"]

Expected Output:

[('emp-001', 'id-01', 'sc-01'), ('emp-002', 'id-02', 'sc-12'), ('emp-003', 'id-03', 'sc-10')]

How can i do it using python.

You can also solve this problem by the following ways;

import re
regex = re.compile("(emp-.+)_(id-.+)_(sc-.+)")
strings = ["humanresourc-emp-001_id-01_sc-01","itoperation-emp-002_id-02_sc-12","Generalsection-emp-003_id-03_sc-10"]
print([regex.findall(s)[0] for s in strings])


Effects of L2 loss and smooth L1 loss Can any one tell me what the effects of $L_2$ loss and smooth $L_1$ loss (i.e. Huber loss with $\alpha = 1$) are, and when to use each of them ?

First, Huber loss only works in one-dimension as it requires $$\left\|\boldsymbol{a}\right\|_2=\left\|\boldsymbol{a}\right\|_1=\delta$$at the intersection of two functions, which only holds in one-dimension. Norms $L_2$ and $L_1$ are defined for vectors. Therefore, in my opinion, Huber loss better be compared with squared loss rather than $L_2$ loss, since "$L_2$" presumes a multi-dimensional input compared to "squared".

Huber loss is the same as squared loss for differences less than $\delta$, and the same as absolute loss for differences larger than $\delta$, i.e.
$$\begin{align*}
L_{\delta}(y_n, f_{\theta}(\boldsymbol{x}_n))
=\left\{
\begin{matrix}
\frac{1}{2}\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right)^2 & \left|y_n - f(\boldsymbol{x}_n)\right| \leq \delta,\\
\delta\left|y_n - f_{\theta}(\boldsymbol{x}_n)\right| - \frac{1}{2}\delta^2, & \text{otherwise.}
\end{matrix}
\right.
\end{align*}$$

where $y_n$ is the target of data point $n$, and $f_{\theta}(\boldsymbol{x}_n)$ is model's prediction. Note that $L_{\delta}$ has nothing to do with $L_p$ norm, despite the similar notations.

Because of this definition, for large differences due to outliers, gradient of loss function remains constant $\pm \delta$, the same as absolute loss, i.e.
$$\frac{\partial \delta\left|y_n - f_{\theta}(\boldsymbol{x}_n)\right|}{\partial \theta_i} = \pm \delta \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}$$
compared to squared loss, where gradient increases with the difference, i.e.
$$\frac{\partial \frac{1}{2}\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right)^2}{\partial \theta_i} = -\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right)\frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}$$

which leads to large contributions from outliers when we update a parameter solely based on squared loss as follows:
$$\begin{align*}
{\theta}'_i &=\theta_i + \lambda \sum_n \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right) \\
&= \theta_i + \lambda\sum_{n \notin \text{outliers}} \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}(\text{small}) +\lambda\sum_{n \in \text{outliers}} \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}(\text{large})
\end{align*}$$

It is worth noting that, here, outliers are irregularities in the joint input-output space $(\boldsymbol{x}_n, y_n)$, not necessarily just in the input space $\boldsymbol{x}_n$ as we usually visualize in unsupervised tasks. For example, in a linear trend, none of $(x, y)=\{(1, 2)$, $(5, 10)$, $(10, 20)\}$ are outliers, but $(1, 10)$ is, which leads to large difference $(10 - 2)$ when model expects (predicts) $f_{\theta}(1)=2$.

  When to use each of them?

Reminding that we are only talking about one-dimensional targets, Huber loss is a complete replacement for squared loss to deal with outliers. However, the challenge is the choice of $\delta$, which makes it a less favorable "first choice" when we are not yet familiar with the problem at hand. Therefore, we may start with squared loss (or other losses), and after a while try to experiment with Huber loss for different values of $\delta$.


What is the best way to replace NaN values for ranked columns I have a column named BsmntQual that gives a ranking on the height of the basement per each house.  These are all of the unique values in this column:

print(train['BsmtQual'].unique().tolist())

>>> ['Gd', 'TA', 'Ex', nan, 'Fa']

This is the legend of this particular column:

BsmtQual: Evaluates the height of the basement

       Ex   Excellent (100+ inches)
       Gd   Good (90-99 inches)
       TA   Typical (80-89 inches)
       Fa   Fair (70-79 inches)
       Po   Poor (<70 inches
       NA   No Basement < Not to be confused with the nan value above

This is what I did for my other ranked columns but this one did not have NaN values:

train['ExterQual'] = train['ExterQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [4, 3, 2, 1])  # Exterior Quality

For numerical values, a common way is to fill all NaN values with the mean of the column.  But what is a good way of replacing the NaN values for columns such as these?

Here is the full dataset

Your legend clearly states that missing values mean that there is no basement. You could fill the missing values with ’NoBase’ to make that point clearer (train[‘BsmtQual’].fillna(‘NoBase’, inplace=True)).
When you rank them then, you just add another ranking for ‘NoBase’, maybe 0, based on your example rating that gives a higher value to more quality.


Error plotting with datetime and value using matplotlib in python Here I have a dataset with three values. I plot these three values in one graph using python. While I plot the graph first plot line is not coming properly. Because start point and end point combined with a line. So does anyone know how to remove this line? I'm stuck with this error. Please can you all help me to solve this error?
Here is my code

data=pd.read_csv('data1.csv')
data = pd.DataFrame(['date', 'time','x1','x2','x3'])
x = pd.to_datetime(data['date'] + ' ' + data['time'], format='%d/%m/%Y %H:%M:%S')
y =data.iloc[:,2]
y1=data.iloc[:,3]
y2=data.iloc[:,4]
idx = ~np.isnan(y)
idx1=~np.isnan(y1)
idx2=~np.isnan(y2)
import matplotlib.dates as mdates
fig = plt.figure(figsize=(6,6))
ax=fig.add_subplot(111, label="1")
ax2=fig.add_subplot(111,label="2", frame_on=False)
ax3=fig.add_subplot(111,label="3", frame_on=False)
ax.plot(x[idx],y[idx], color="C0")
ax.set_xlabel("date1 label 1", color="C0")
ax.set_ylabel("y label 1", color="C0")
hours = mdates.HourLocator(interval =10)
h_fmt = mdates.DateFormatter('%H')
ax.xaxis.set_major_locator(hours)
ax.xaxis.set_major_formatter(h_fmt)
#ax.set_xlim([xmin,xmax])
#ax.set_ylim([ymin,ymax])
ax.tick_params(axis='x', colors="C0")
ax.tick_params(axis='y', colors="C0")
ax2.plot(x[idx1], y2[idx1],'g*')
ax2.xaxis.tick_top()
ax2.yaxis.tick_right()
ax2.set_xlabel('x label 2', color="C1")
ax2.set_ylabel('y label 2', color="C1")
ax2.xaxis.set_label_position('top')
ax2.yaxis.set_label_position('right')
hours = mdates.HourLocator(interval =10)
h_fmt = mdates.DateFormatter('%H')
ax2.xaxis.set_major_locator(hours)
ax2.xaxis.set_major_formatter(h_fmt)
#ax2.set_xlim([xmin,xmax])
#ax2.set_y1lim([y1min,y1max])
ax2.tick_params(axis='x', colors="C1")
ax2.tick_params(axis='y1', colors="C1")
ax3.plot(x[idx2], y1[idx2],'k.')
hours = mdates.HourLocator(interval =10)
h_fmt = mdates.DateFormatter('%H')
ax3.xaxis.set_major_locator(hours)
ax3.xaxis.set_major_formatter(h_fmt)
#ax3.set_xlim([xmin,xmax])
#ax3.set_y3lim([y3min,y3max])
ax3.set_xticks([])
ax3.set_yticks([])
fig.autofmt_xdate(rotation=90)
plt.show()

Here I attached the my csv file link.
https://docs.google.com/spreadsheets/d/1y6LKGIftb7F0ovF2dYsMwmvkR_diJBhBNC6sIiaagbI/edit?usp=sharing

my graph:

Matplotlib plots data in lists from left to right.

Hence, in the list x[idx] which is used in ax.plot(x[idx],y[idx], color="C0"), I assume that either the first element (according to x-axis) is in the back of the list, or the last element is at the beginning of the list.


Where do I start? I have a massive dataset (from web scraping) and want to predict y from 20 variables I'm trying to do my best to recreate an algorithm that a company is using by sucking in all of its outputs and adding in some relevant variables (I won't be able to perfectly recreate it as I'm missing inventory data). I'm guessing this is a job for machine learning but I was wondering where I start? Are there specific models I should start with? Software I should use? Courses I should dive into?

Since you are really new to all this, here are some suggestions.

Download and install R and RStudio Desktop.

Look at the book "Introduction to Statistical Learning". The book comes with code examples (in R) which are ideal to start learning. For you it will be useful to have a look at Chapter 2 to understand what statistical learning is all about and also to get a good idea of how to use R.

However, what is really relevant for you is Chapter 3 (linear regression) and Chapter 7 (moving beyond linearity).

Now you can get started. Here are some examples how regression can be done in R:

# Install a package if needed
# install.packages("ISLR")

# Lead packages
library(ISLR)
library(gam)
library(Metrics)

# Look at data
head(mtcars)
# Look at one variable in our dataframe
summary(mtcars$mpg)

# Make a test/train split
set.seed(123)
smp_size <- floor(0.75 * nrow(mtcars))
train_ind <- sample(seq_len(nrow(mtcars)), size = smp_size)
train <- mtcars[train_ind, ]
test <- mtcars[-train_ind, ]

# Linear regression
# Estimate mpg (y) dependent on ...
reg1 = lm(mpg~disp+wt+qsec+hp, data=train)
summary(reg1)

# Regression with polinomials
reg2 = lm(mpg~poly(disp,2)+poly(wt,2)+poly(qsec,2)+poly(hp,2), data=train)
summary(reg2)

# Generalised additive models (regression splines)
reg3 = gam(mpg~s(disp,3)+s(wt,3)+s(qsec,3)+s(hp,3),data=train)
par(mfrow=c(2,2))
plot(reg3)

# Predict mpg
pred1 = predict(reg1, newdata=test)
pred2 = predict(reg2, newdata=test)
pred3 = predict(reg3, newdata=test)

# See how well our model fits
mae(test$mpg, pred1)
mae(test$mpg, pred2)
mae(test$mpg, pred3)


How the Support Vector Machine will perform if the bias b = 0 in the equation of hyperplane? We have a soft margin linear SVM and the equation is as follows :

How the SVM will perform if b = 0, means the hyperplane is passing through the origin ?

I think that it will simply behave as a normal SVM, with the difference that being the hyperplane anchored to the origin, you are blocking one degree of freedom and as such, the optimization algorithm used to find the parameters w might not find the best solution.
At this point, I am wondering why you should set b=0.


Using the Stanford Named Entity Tagger in R I am experimenting with the Stanford Named Entity Tagger  here http://nlp.stanford.edu:8080/ner/process and I feel it would be useful in my research. Does anyone know of a example that I could follow so that I could do the analysis in R? Ideally I'd want to provide a string and get back a count (as a list) of the number of organisations, persons, etc recognised in the string. Thanks.

https://github.com/statsmaths/coreNLP  can be used as a wrapper for this library in R. Documentation has good examples for most use cases lie NER and POS.


Is a good practice to sum each rate (i.e. crime rate per 100.000 people)? Consider a dataset from 1990 to 2017 that contains the crime rate per 100,000 people in some cities from Latin America.

I want to measure which city is more complex according to this data and other indicators. I'm using The analytic hierarchy process proposed by Saaty.

Consider this example:

city_code  r1990  r1991 r1992 r1993 r1994 r1994
  12345     120     80    91   110   105   99
  23456      10     15    16    12     7   11
  34567      90     91    85    75    77   65

According to my question, I want to sum each rate in each year, and then get for each year the mean of each rate based on the total.

Based on the example above:

city_code  total
  12345     605
  23456      71
  34567     483

city_code  mean1990  mean1991 mean1992 mean1993 mean1994 mean1994
  12345      0.20      0.13     0.15     0.18     0.17     0.16
  23456      0.14      0.21     0.23     0.17     0.10     0.15
  34567      0.19      0.19     0.18     0.16     0.16     0.13

So, is this a good practice? I couldn't find an example following this method? Any orientation about this subject will be appreciated.

This can be fairly good if you deal with few years.

In general it's not a good practice, since asymptotically will lead you to have almost the same fraction of crimes per all the considered years. But unless you have data from the last 10000 years, you can simply ignore the asymptotic behavior a continue the way you are doing.

Otherwise, you can normalize the data to a reference value, say the maximum number of crimes on the series. This is more robust.


L2 loss vs. mean squared loss I see some literature consider L2 loss (least squared error) and mean squared error loss are two different kinds of loss functions.

However, it seems to me these two loss functions essentially compute the same thing (with a 1/n factor difference).

So I am wondering if I have missed anything? Is there any scenario that one should use one of the two loss functions?

Function $L_2(x):=\left \|x \right \|_2$ is a norm, it is not a loss by itself. It is called a "loss" when it is used in a loss function to measure a distance between two vectors, $\left \| y_1 - y_2 \right \|^2_2$, or to measure the size of a vector, $\left \| \theta \right \|^2_2$. This goes with a loss minimization that tries to bring these quantities to the "least" possible value.

These are some illustrations:

$L_p$ norm: $L_p(x) := \left \|x \right \|_p = (\sum_{i=1}^{D} |x_i|^p)^{1/p}$,
where $D$ is the dimension of vector $x$,
Squared error: $\mbox{SE}(A, \theta) =\sum_{n=1}^{N} \left \| y_n - f_{\theta}(x_n) \right \|^2_2$,
where $A=\{(x_n, y_n)_{n=1}^{N}\}$ is a set of data points, and $f_{\theta}(x_n)$ is model's estimation of $y_n$,
Mean squared error: $\mbox{MSE}(A, \theta) =\mbox{SE}(A, \theta)/N$,
Least squares optimization: $\theta^*=\mbox{argmin}_{\theta} \mbox{MSE}(A, \theta)$$=\mbox{argmin}_{\theta} \mbox{SE}(A, \theta)$,
Ridge loss: $\mbox{R}(A, \theta, \lambda) = \mbox{MSE}(A, \theta) + \lambda\left \| \theta \right \|^2_2$
Ridge optimization (regression): $\theta^*=\mbox{argmin}_{\theta} \mbox{R}(A, \theta, \lambda)$.

In all of the above examples, $L_2$ norm can be replaced with $L_1$ norm or $L_\infty$ norm, etc.. However the names "squared error", "least squares",
 and "Ridge" are reserved for $L_2$ norm. For example for $L_1$, "squared error" becomes "absolute error":

Absolute error: $\mbox{AE}(A, \theta) =\sum_{n=1}^{N} \left \| y_n - f_{\theta}(x_n) \right \|_1$,


Why bias is not considering in Regularization? Most of the Regularization (L1, L2 ) techniques focused mostly on the weight term only .But Regularization is   not considering Bias.From my understanding  large bias doesn’t make a neuron sensitive to its inputs in the same way as having large weights and vice versa .which means Bias is also important as weight .

Can any one help me why Bias is not considering?

The point of regularization is to avoid overfitting, and overfitting happens when you have too many predictor variables (i.e. neurons) contributing to the outcome. So, by regularization you are essentially excluding some neurons.

However, bias is an intrinsic property of every neuron. It is not a connection transporting the result of a neuron from the previous layer. Maybe you can imagine all biases in the network being connections to a single, virtual neuron with a constant output of one, but still, as it is a constant, it is not a result of a computation and does not depend on the input.


How does Seaborn calculate error bars when using estimators other than the arithmetic mean? If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?

This information is in the documentation of Seaborn.
They show a bootstrap confidence interval, computed by resampling units (rows in the 2d array input form). By default, in seaborn version 0.8.1 it uses 95% of confidence interval, which is equivalent to a standard error. This value is a parameter that can be changed.


Multi-Label Loss function and model training I'm working on Multi-Label problem i.e output can predict 1 or more label as an output and hence training data also have multiple labels.

Somehow I'm not able to map such ML model training. Please provide me the answer of below mentioned questions:

How loss is getting calculated for such type of problem?
How the ML model predicts?

I found one old thread Multi Label old query, but again Im not able to understand the solution.

In a normal multi-class setup we use softmax over the last layer because we know that the sum of probability of the classes is 1 as only one of these classes is the actual answer. After that, we pass the output to a suitable loss function like BCE (binary cross-entropy).
For multi-label, we know that each class can be the output so the sum of probability of these classes will not necessarily be 1! However, the individual probability of each class still needs to be between 0 and 1 to be a valid probability distribution. Therefore, simply apply a sigmoid layer on top of each class output and pass this to BCE loss which treats each class as a separate mini BCE loss, and finally adds these losses across all the classes to get the final BCE.
For e.g., in Pytorch, you just apply sigmoid and pass it to the BCE such that the dimensions of the output passed to the BCE layer are (batch size, num of total class).
Look at the BCE loss section in the pytorch docs, where the dimensions of the input to BCE loss is (N, *) which implies the dimension which I just pointed out: TORCH.NN | Pytorch Docs.


How can I read in a .csv file with special characters in it in pandas? I am trying to read in a .csv file containing some data. I only need to read in specific chunks of rows from the file, such as line 15- line 20, line 45-line 50, and so on. However, the file contains text and copy write information like, such as ©1990-2016 AAR,All rights reserved in several places. Such lines seem to be producing the error ValueError: No columns to parse from file, because when I just copy lines without such information using pd.read_csv(), it works fine. My goal is to automate the process of downloading these files from the web and reading them into pandas to grab chunks of rows and then do some processing with it, so I can't just manually specify the windows of text lacking such characters.

Here is what I tried:pd.read_csv("filename.csv",encoding=utf-8, skiprows = 14) and pd.read_csv("filename.csv",encoding=utf-16, skiprows = 15), after looking at similar answers in stack exchange, but this didn't work. Can anyone give me some guidance on this?

There is df.drop command that can be used as follows to remove certain rows (in this case, 15 & 16):

df.drop(df.index[[15,16]])

If the rows you don't need are regular (e.g. you never need row 15) then this is a quick and dirty solution.

If you only want to drop arbitrary rows containing some value, this should do the trick:

df = df.drop([df.column_name == ©1990-2016 AAR])


How to get Big Data Sets? It sounds like dump question but as a beginner, I'm really getting confuse.
For my academic thesis, I choose a conference paper on Big Data in Healthcare field. Now, problem is to get the Data sets.
I can't find any resources to download the data sets to work on it. When I google it, people suggested some resources which are really good but those file is not Big to call Big Data. I'm really confuse in this point and can't get right answer on this from anyone.
I was thinking, may be we work on small batch of data sets and further apply the models on Big Data in production sphere. However, it's just an assumption. I want to know, for thesis or analytical research in academic level, can we get real life Big Data sets to download for free?
Please correct me if I'm missing something. Thanks in advance.

Take a look at https://goo.gl/yCZvSb, a view of the Gapminder health data sets. Of the 519, you may find one suitable.

For purposes of your thesis, I'd encourage you to think of big data in terms of high-dimensional data with n >> the typical clinical trial. I'd be very surprised if you found health-related observational data other than vital statistics, such as number of births/deaths that involved even 10^9 scale data.


using simple autoencoder for feature selection I am using a simple autoencoder to extract the informative features and I have multiple Q:

I know that the features extracted will be a linear combination of the original features so I consider that the feature that has a larger mean weight (has the highest percentage in the formation of new features) will be important so I will take that features but I don't know if this is true or not

the second things is that I want to apply the grid search to find the optimal hyperparameters for the model but I can't do that please if anyone can help me in this and save my life

Autoencoders normally aren't linear models. If you make them linear (i.e. you create a shallow Autoencoder with linear activations) then you get exactly a PCA result. The power of Neural Networks is their non-linearity, if you want to stick with linearity go for PCA imho.

Keep a Train-Validation-Test set split, and try different configurations of hyperparams checking their performance on Validation data. Alternatively there are many libraries, such as hyperopt, that let you implement more sophisticated Bayesian hyperparameter searches, but unless you want to be published at a conference or win some competition it's a bit overkill. If you're still interested, the internet is plenty of tutorials like this one.


A clear visualization of a two-way ANOVA To provide a full yet simple picture of a 3-level, one-way ANOVA, I use the following visualization where variation within each group (the filled circles) and variation between the groups (black arrows) are simple to be understood.

But I'm wondering if it could be possible to extend the current visualization to a 2 x 3 two-way ANOVA (adding another way with two groups to the current visualization)?

(Note: the dashed vertical lines denote each group's mean)

You could go for a 3D plot, but they are only useful in specific situations (e.g. simple bar plot, a surface etc.), and definitely not here. Your best option, while keeping the overall design of your plot, seems to be to split your data by the factor with 2 levels, and plot each separately in 2 panels side by side. That way, you can compare similarities, baseline shifts, and interactions.


Best Python NLP library for supervised topic classification I have a labeled dataset that I have ingested into a dataframe. It consists of news articles,
>>> df.columns

Index(['title', 'headline', 'byline', 'dateline', 'text', 'copyright',
       'country', 'industry', 'topic', 'file'],
      dtype='object')

where the text column contains the body (text) of the article and the topic column contains a list of associated topics.
I want to train a model from this dataset to predict the article topics. I was considering using transformers (https://huggingface.co/transformers/index.html) to do this, along with tensorflow, but I from what I know of transformers, it's not really good for this.
What would be the best NLP library to perform this task with high accuracy?

The recommendation is going to be still Huggingface transformers. You extract the features from BERT, then, some dense layers and then, feed them into sigmoid layer with unit equal to number of classes. Pose it as a multi-label classification.


Your train/validation loss curves are a classic example of overfitting.
It looks like you have 1425 data samples to train a model with > 28 million parameters.
I would suggest trying any/all of the follow:

use a smaller model -> less parameters means less complexity and less ability to overfit
data augmentation -> more data samples means more variation in the dataset for the model to capture
early-stopping -> use something like the Keras callback, which will stop the model training once the validation loss doesn't decrease for a number of epochs

If you happen to be using image data, you might take a look at the Keras ImageDataGenerator, which can do things like flip/rotate your images.


What algorithms can be used to predict the outcome of a cricket match? I am doing a project to predict the outcome of a cricket match, I have the data that states which matches were won by whom for ODIs. [Espn data]

Which algorithm could be used to predict the outcome of the coming matches?
Would Quadratic Regression be a good idea? Or would predicting based on probability algorithms such as the markov's algorithm is what is generally used?
Any other algorithm I should use?

So basically, I want to know which algorithm I should use, I will implement in C++ ultimately but I will do it in R or python first.

P.S.:-

I am a newbie in this field, hence pardon if the question sounds too stupid.
I have learnt regression so far in data analytics.

Sharing what sort of outcome you are most interested will be helpful in directing you towards a proper answer. However, consider this:

If you are interested, in exploring the likelihood of will win or lose think classification algorithms:

Linear Machine Learning Algorithms
 - Logistic regression
 - Linear Discriminant Analysis

Non-linear Machine Learning Algorithms

K-nearest neighbors
Naive Bayes
Support Vector Machines

If you are interested in exploring the scores of each team in a given game think regression algorithms:

Linear regression
Logistic regression
Stepwise regression
Multivariate Adaptive Regression Splines (MARS)
Locally estimated Scatterplot Smoothing (LOESS)

In my judgement, giving us more direction on your desired outcome we can give you more direction with a solid answer.


How long does it typically take to train a MNIST data on a Mac Pro? My code is below:

# define a simple CNN model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Conv2D(30, (5, 5), input_shape=(1, 28, 28), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Conv2D(15, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(50, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

# build the model
model = baseline_model()
# Fit the model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=500, batch_size=200)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)

My Mac's capacity:

Grphics Radeon Pro 555 2048 MB
Intel HD Graphics 630 1536 MB

Memeory 16 GB 2133 MHz LPDDR3

Processor 2.8 GHz Intel Core i7

It is taking several hours to train this the MNIST image classification dataset on my computer. Is this normal?

Did you try to do a model summary and check the size of your model. Yes, i may take hours as you are adding lot of layers. Try starting with 10 or 20 epochs and check the accuracy of your model gradually. May reduce your training time.

When you do the above exercise its better to have a validation data set, rather than fitting the model on the training data set time and again.


Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:

Storm alone and implement streaming naive bayes on my own in storm topology.
Storm + TridentML
Storm + SAMOA
Spark Streaming + MLlib

What is the best framework set to choose and start working on. Any suggestion will be of great help.

If I were you, I would pick anyone of the frameworks I am comfortable with and implement the use-case.
Spark-Streaming + MLlib should work and would be my choice since its user base is on the rise and it is one of the most popular project under the Apache Umbrella with good enterprise business plan. Both Cloudera and Hortonworks provide enterprise level support.
Now, in theory Spark-Streaming lacks behind Storm in stream processing, but the framework is cool in a way that it provides you the option to do streaming, common map and reduce, graph processing and SQL under the same framework. So once you have the pipeline to convert your data to RDD you are good for most of the common jobs related to Data Analysis. It's written from scratch in Scala which is a very powerful language and provides huge scalability in a distributed setup when handling concurrency.
Hope this helps, feel free to reach out to me with any questions you have.


Where to start, which books What is a good book or resource to start learning Artificial Intelligence ?

The standard textbooks that covers AI is "Artificial Intelligence: A Modern Approach" by Russel & Norvig. The book's website can be found here.

I also recommend "Artificial Intelligence: Foundations of Computational Agents" by Poole and Mackworth. The book can be read online.


Are there any image classification algorithms which are not neural networks? Image classification is the task of assigning one of $n$ previously known labels to a given image. For example, you know that you will be given a couple of photos and each single image has exactly one of $\{cat, dog, car, stone\}$ in it. The algorithm should say what the photo shows.

The benchmark dataset for image classification is ImageNet; especiall thy large scale visual recognition challenge (LSVRC). It has exactly 1000 classes and a huge amount of training data (I think there is a down-sampled version with about 250px x 250px images, but many images seem to be from Flicker).

This challenge is typically solved with CNNs (or other neural networks).

Is there any paper which tries an approach which does not use neural networks in LSVRC?

To clarify the question: Of course, there are other classification algorithms like $k$ nearest neighbors or SVMs. However, I doubt they work at all for that many classes / that much data. At least for $k$-NNs I'm sure that prediction would be extremely slow; for SVMs I guess both fitting and prediction would be much to slow (?).

Part of the problem with answering this question is there are actually two questions.  The first:

Are there any image classification algorithms which are not neural networks?

Yes, lots. But now the actually question:

Is there any paper which tries an approach which does not use neural networks in LSVRC?

In your question, you rule out methods such as kNN and SVM because of speed. Bag of Words is one method used to solve this problem.  MATLAB has a good demonstration (http://www.mathworks.com/help/vision/examples/image-category-classification-using-bag-of-features.html).  But BoW incorporates k-means clustering, so that may not fit your needs.

There are some other interesting image classification methods such as texture analysis. TA is being researched as a way to classify malevolence of disease in medical images (such as tumors).  Here is a commonly referenced paper:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701316/

Here's an overview of image classification:
http://www.tandfonline.com/doi/full/10.1080/01431160600746456


Text summarization with limited number of words I am reviewing summarization techniques and haven't (yet) found an approach to limit the length of a summary. So for example a summarization function that gives me a summary that is < 500 words.

Can you point me in the right direction? Are there approaches/implementations out there that try to solve this challenge?

Appreciate your replies!

You sure can,
for example in latent semantic analysis you can fixate number of topics (which is actually size of the decomposition matrix) beforehand.


Training the Variational Autoencoder After applying the reparameterization trick In a Variational Autoencoders, z cannot be simply sampled from the output of the autoencoder directly since the network would not be differentiable. Instead, we have to sample from a normal distribution who's parameters are the output of the autoencoder, mu and stddev.

I So cannot understand how the network isn't differentiable (lets say we take z = mu + srd). So the reparameterization is not clear to me.

Any help would be much appreciated!!

Have you read Doersch's tutorial on VAE? In paragraph 1.1 the author shows main intuitions behind the latent model and in the following sections he goes deeper into the explanation of how a VAE works. I think it's a good starting point to approach to VAEs.


How can I tune LSTM hyperparameters? If anyone is there to answer these, that'll be great. I'm in the midst of a Final Year Project on LSTM.

Currently, I’m stuck and confused over LSTM codes. There are 4 hyperparameters that I can play around with:

Look back
Batch size
LSTM units
No. of Epochs

Can you explain what will happen to my results if I tune each of these hyperparameters? And also is it common if we get different results each time we run the codes?

Look back, I don't know look back as an hyper parameter, but in LSTM when you trying to predict the next step you need to arrange your data by "looking back" certain time steps to prepare the data set for training, for example, suppose you want to estimate the next value of an episode that happens every time t. You need to re-arrange you data in a shape like:
{t1, t2, t3} -> t4
{t2, t3, t4] -> t5
{t3, t4, t5} -> t6
The net will learn this and will be able to predict tx based on previous time steps.

Batch size (is not referred to LSTM only), roughly is how much samples will be trained per single step, as bigger the batch size is the faster the training is but more memory is needed.
In a GPU is better to have bigger batch sizes because copying the values from GPU to memory is slow.

LSTM units, refers to how much "smart" neurons you will have. This is highly dependent on your dataset, usually you determine this depending on your vector dimensions.

No. of Epochs, how much times the algorithm will run to approximate the observations. Usually to much epochs will overfit your model and to little will end up in an under fitted one.


Can I use Linear Regression to model a nonlinear function? I have recently started studying the basics about regression, and as a beginner I started by Linear Regression.

I read this article that says that for this particular type of regression the relationship between independent and dependent variables has to be linear, which to me implies that I can only predict "lines" with Linear regression:
https://www.analyticsvidhya.com/blog/2015/08/comprehensive-guide-regression/

But then I started wondering about how to model functions like "y = log(x)" or "y= sqrt(x)" or "y=exp(x)" or "y=tan(x)" or other nonlinear functions by definition which are not "lines" but "curves".

Then I carried on doing research until I found this article that says that it is not the relationship between the independent and dependent variables that should be linear, but the final functional form passed to the model:
https://medium.freecodecamp.org/learn-how-to-improve-your-linear-models-8294bfa8a731

I want to know if that is really the case, and is it always possible to do this "change" in the functional form? Also if it is possible to use linear regression for nonlinear functions, is it still correct to measure the performance of the model using R_square metric?

Thank you.

You are asking two different questions:

What is linear regression?

Linear regression means that, given a response variable $y$ and a set of predictors $x_i$, you are assuming (whether or not this is true is another matter) to model your response variable as
$$
y^{(j)}=\sum_{i=1}^N x_i^{(j)}\beta^i + \epsilon^{(j)}
$$
for each observation $y^{(j)}$, where $\epsilon^{(j)}$ is an error term with vanishing expectation value. The purpose of the algorithm is to find the set of $\beta^i$ to minimise the errors between the above formula and the actual values of the response.

May I use linear regressio to model non-linear functions?

You may use the linear regression to model anything you want, this does not necessarily mean that the results will be a good fit. The mere decision to use a model makes no assumptions on whether the underlying equation is in fact reflected by the model you choose. In case of linear regression you are essentially approximating an $N$-dimensional manifold (where all true points belong) with their projections onto a plane. Whether or not this is a good idea it depends on the data.

  I want to know if that is really the case, and is it always possible to do this "change" in the functional form?

By using this or that other model you are not changing the functional form of the underlying variables. You are just dictating that the original relation (that you do not know) can be approximated by the model you choose.

  Is it still correct to measure the performance of the model using R_square metric?

The $R^2$ is defined as the ratio between the residual sum of squares of your model over the residual sum of squares of the average. Basically it tells how much of the variance of the data is explained by your model compared to just taking a straight line (in correspondence of the average) passing through all your data points.


trying to decrease overfitting with regularisation in CNN I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:

training steps: 20,000
learning rate: 0.075
test accuracy :72.9%
train accuracy for final iteration : 98.0%
val accuracy for final iteration : 75.0%

clearly overfitting, so I tried L2 regularization. Here are my parameters and results, for the highest accuracy until now :

training steps: 40,000
learning rate: 0.1
test accuracy :71.1%
alpha for L2 : 0.00075
train accuracy for final iteration : 91.0%
val accuracy for final iteration : 71.0%

Looked promising so I decided to go further with the following hoping to get better accuracy :

training steps: 80,000
learning rate: 0.075
alpha for L2 : 0.005

go the following graph :

Since, the graph was getting saturated so I quit at 50,000 steps.

My question :

Should I continue with it, hoping it will get better or should I try with other values(Please suggest some good range of parameters that I should try)? or may be other techniques like L1 regularization or dropout ?

Note : Since, it is real-world data so getting new data would be difficult, and I cant flip the image to generate new data since orientation is also a factor to determine the label for output classes

Steps for overcoming overfitting are the followings:

 1. Add more data
 2. Use data augmentation
 3. Use architectures that generalize well
 4. Add regularization
 5. Reduce architecture complexity.

I would try to add Dropout, lately it is being the most used regularization technique in complex networks architectures. Start adding it in the later layers, since it is best no to lose initial info. Something like 0.6 or like that, and if it is not enough add dropout in the previous layer, and so on.


Multi-task learning, finding a loss function that "ignores" certain samples I'm training a convolutional neural network with 1 input, and 3 outputs: a classification and 2 regression outputs.

I initially used 3 separate models, but I found that combining the 1st and 2nd loss functions into 1 model improves accuracy/rmse for both. I'm applying multi task learning.

Now I'm experimenting with incorporating the 3rd loss function into the same model with the first 2.

My challenge is that the 3rd loss function is only valid when the 1st classifier identifies a positive sample (it's a regression output that measures the identified sample). These labels are known at training time.

I attempted:

Standardizing the labels and impute a 0 value for the 3rd regression for all negative samples. That approach biases towards 0 in some cases, so I threw it out.
Alternate training between all samples on the 1st and 2nd loss, and only positive samples using the sum of all 3 loss functions. Mixed results here, but the 3rd loss function seems to perform worse than it does in its own separate model.

Question:

I'm considering how I might re-structure the loss function (currently L2 loss) so that it treats a 0-value as irrelevant (e.g. no gradient is propagated when the label is 0).

Any thoughts?

I seem to have a workable solution using this minor tweak to square loss:

$L=(y-\hat{y})^2 \cdot \left(\frac{y}{y + \epsilon}\right)$

Where $y$ is the label and $\hat{y}$ is the prediction, and $\epsilon=1e-6$ is to avoid a divide by zero error (it's ~ the smallest number within 32 bit precision)

In tensorflow:

epsilon = 1e-6
loss = tf.reduce_sum(
          tf.mul(tf.square(prediction - label),
                 tf.div(label, label + epsilon))
)

The second term is 0 when the label is 0 (e.g. loss is 0 and no gradient is added for that sample), and it'll round to 1, or near enough, when the label is non zero. As long as the range of labels does not cross 0 this should work, and that's a simple matter of scaling the labels if it's not the case by default.


Using BM25 to rank words How effective is it to use BM25 to rank words, to be more specific i have a dictionary of words and i want to rank only words in a document that are also in my dictionary. I want to rank all words in my dictionary for each document and then add the BM25 value of each word for the specific document.

Lets say i have a document and dictionary like this:

myDictionary=['bad', 'dangerous','hide', 'following]

corpus=[
['human', 'intelligence', 'computer','bad', 'dangerous'],
 ['survey', 'user', 'human', 'system', 'time', 'hide', 'following],
]

Now iam going to run the below BM25 formula in a loop for all words in my dictionary and then sum
the results of each word to get a bm25 value for each document.

BM25 is usually used in information retrieval. In this task, you have a query and a lot of documents(maybe millions), and then you want to find a subset of these documents that are most relevant to your query. A ranking of a set of documents will be provided from the most relevant to the least.
If by efficient you mean fast in a computational way. I would say BM25 is pretty fast with respect to other algorithms that are using deep neural networks.
But if your asking if BM25 results are promising or not. This is debatable as BM25 is being used for a long time. People usually use it for the first step of ranking, then they do re-ranking with other powerful tools. It doesn't mean that BM25 is giving the best answers. But when you are dealing with thousands or millions of documents, This is a good choice in order to select only a subset of documents that have high scores with BM25 then later use a more accurate algorithm to rerank the results of BM25.


Does autoencoder with few images not work? I am trying to load 2 images to an autoencoder but for some reason, it does not rebuild the input image.
An autoencoder is supposed to compress and decompress an image. However, when passing two images with which it trains, it only shows a black color instead of the corresponding image.
import keras
from keras import layers

# This is the size of our encoded representations
encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats

#28*28=784  tamaño de images de nmist

ancho=28
alto=28
n_pixels=alto*ancho

# This is our input image
input_img = keras.Input(shape=(n_pixels,))

# "encoded" is the encoded representation of the input
encoded = layers.Dense(encoding_dim, activation='relu')(input_img)
# "decoded" is the lossy reconstruction of the input
decoded = layers.Dense(n_pixels, activation='sigmoid')(encoded)

# This model maps an input to its reconstruction
autoencoder = keras.Model(input_img, decoded)

# This model maps an input to its encoded representation
encoder = keras.Model(input_img, encoded)

# This is our encoded (32-dimensional) input
encoded_input = keras.Input(shape=(encoding_dim,))#la salid del encoder osea la entrada del decoder
# Retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# Create the decoder model
decoder = keras.Model(encoded_input, decoder_layer(encoded_input))

This code is to load two images with the help of pil module:
##load custom data
from PIL import Image
import numpy as np

#data1=np.linspace(0,0,2)
#data1_test=np.linspace(0,0,2)
data1=[0,0,0]
data1_test=[0,0,0]

data2=np.linspace(0,0,n_pixels*2).reshape(2,n_pixels)

data1[0]=Image.open("/content/zero.png")
data1[1]=Image.open("/content/eight.png")

#custom_imagen_arr=np.array(list(custom_imagen)[0]).reshape(1,784)
custom_data_arr=np.linspace(0,0,n_pixels*2).reshape(2,n_pixels)
for  i2 in range(2):
       imm=list(data1[i2].getdata())
       for  i in range(len(list(data1[i2].getdata()))):
            custom_data_arr[1,i]=imm[i][0] / 255.
       for  i in range(len(list(data1[i2].getdata()))):
            data2[i2][i]=custom_data_arr[1,i]
data2 = data2.astype('float32') / 255.

data2 = data2.reshape((len(data2), np.prod(data2.shape[1:])))

Then I train:
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
import numpy as np
autoencoder.fit(data2, data2,
                epochs=152,
                batch_size=1,
                verbose=1,
                validation_data=(data2, data2))

After finishing training, I choose one of the loaded images and make a prediction, but the output is black.
#give an arbitrary image to the autoencoder and generate an image

from PIL import Image
custom_imagen=Image.open("/content/eight.png")
#custom_imagen_arr=np.array(list(custom_imagen)[0]).reshape(1,784)
custom_imagen_arr=np.linspace(1,0,n_pixels*2).reshape(2,n_pixels)
imm=list(custom_imagen.getdata())
for  i in range(len(list(custom_imagen.getdata()))):
        custom_imagen_arr[1,i]=imm[i][0] / 255.
encoded_imgs = encoder.predict(custom_imagen_arr)
decoded_imgs = decoder.predict(encoded_imgs)

# Use Matplotlib (don't ask)
import matplotlib.pyplot as plt

n = 2  # How many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # Display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(custom_imagen_arr[i].reshape(alto, ancho))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # Display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(alto, ancho))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()

How do I so that the output of the autoencoder is not black and the same number is the one I train?
The code is modified from here.

I'm doing the same thing actually. I had to find tools because I was getting black images too.
This one worked for me:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvas
from PIL import Image
import tensorflow as tf

keras_imager = tf.keras.preprocessing.image

# creating a figure to plot data
fig = Figure(figsize=(3, 2), dpi=96) # adjust your params
canvas = FigureCanvas(fig)
ax = fig.add_subplot(111)

# plot the data
ax.plot(your_data)

# draw the canvas, cache the renderer
canvas.draw()
s, (width, height) = canvas.print_to_buffer()

# converting to a NumPy array.
X = np.frombuffer(s, np.uint8).reshape((height, width, 4))

# converting to 3D rgb array data
img = keras_imager.array_to_img(X)
rgb = img.convert('RGB')
arr = keras_imager.img_to_array(rgb)
# standardize each image
arr = tf.image.per_image_standardization(arr)

# If you want to resize
arr = tf.image.resize(arr, (32, 32), # <- adjust your param
                     preserve_aspect_ratio=True)
# you can view the keras transformed plot
keras_imager.array_to_img(rgb)

Honestly, I don't quite understand all the parameters, but it's the simplest way I could tweak it to work for my situation. Good luck.


Target encoding with KFold cross-validation - how to transform test set? Let's say I have a categorical feature (cat):
import random
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold

random.seed(1234)
y = random.choices([1, 0], weights=[0.2, 0.8], k=100)
cat = random.choices(["A", "B", "C"], k=100)
df = pd.DataFrame.from_dict({"y": y, "cat": cat})

and I want to use target encoding with regularisation using CV like below:
X_train, X_test, y_train, y_test = train_test_split(df[["cat"]], df["y"], train_size=0.8, random_state=42)
df_train = pd.concat([X_train, y_train], axis=1).sort_index()
df_train["kfold"] = -1
idx = df_train.index
df_train = df_train.sample(frac=1)

skf = StratifiedKFold(n_splits=5)
for fold_id, (train_id, val_id) in enumerate(skf.split(X=df_train.drop("y", axis=1), y=df_train["y"])):
    df_train.iloc[val_id, df_train.columns.get_loc("kfold")] = fold_id

df_train = df_train.loc[idx]

encoded_dfs = []

for fold in df_train["kfold"].unique():
    df_train_cv = df_train[df_train["kfold"] != fold].copy()
    df_val_cv = df_train[df_train["kfold"] == fold].copy()

    means = df_train_cv.groupby('cat')['y'].mean()
    df_val_cv['cat'] = df_val_cv['cat'].map(means)
    encoded_dfs.append(df_val_cv)

encoded_dfs = pd.concat(encoded_dfs, axis=0).sort_index()
encoded_dfs.drop('kfold', axis=1, inplace=True)

However, I have some doubts about the way how I should then encode test set. As there is no single mapping deduced from train set I think we should use the whole train set to fit the encodings and then use it on test set:
means = df_train.groupby('cat')['y'].mean()
X_test['cat'] = X_test['cat'].map(means)

It seems to be the natural way to do it as, in fact, this is exactly mimicked by CV step. But the results of the model I got were off and it made me think if I am missing something. Please note that, for sake of simplicity, I omitted additional smoothing I did as well. Therefore, my question is: is it the correct way to encode test set?

I have some doubts about the way how I should then encode test set. As there is no single mapping deduced from train set I think we should use the whole train set to fit the encodings and then use it on test set

Yep, that seems fine, they way that you do it there its a bit more complicated than using a pipeline. The idea of splitting into train and test is mimicking how the model will behave in production/unseen data. Doing target encoding with the test, is doing data leakage and getting a miss representation of how the model will behave in production. So you get the target values in train and then move to test.
If you do this, and then you have a category in test that is unseen, it will through an error. If you have a look at the target encoding library of category encoders, you can deal with this.:

handle_missing: str
options are ‘error’, ‘return_nan’ and ‘value’, defaults to ‘value’, which returns the target mean.

You can handle it in different ways, the best is depending in your problem. The default is returning the target mean.
They best practice to do is to create a pipeline where the target encoding is a step(transformer). This will allow you to do CV, evaluate your model on test and many other functionalities. (Here a tutorial on how to)
A code snippet:
import random
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from category_encoders.target_encoder import TargetEncoder
from category_encoders.m_estimate import MEstimateEncoder
from sklearn.linear_model import ElasticNet,LogisticRegression

random.seed(1234)
y = random.choices([1, 0], weights=[0.2, 0.8], k=100)
cat = random.choices(["A", "B", "C"], k=100)
df = pd.DataFrame.from_dict({"y": y, "cat": cat})

X_train, X_test, y_train, y_test = train_test_split(df[["cat"]], df["y"], train_size=0.8, random_state=42)
skf = StratifiedKFold(n_splits=5)

clf = LogisticRegression()
te = TargetEncoder()

pipe = Pipeline(
        [
         ("te", te),
          ("clf", clf),
        ]
    )

#Grid to serch for the hyper parameters
pipe_grid = {
    "te__smoothing": [0.0001],
    }

# Instantiate the grid
pipe_cv = GridSearchCV(
        pipe,
        param_grid=pipe_grid,
        n_jobs=-1,
        cv=skf,
    )

pipe_cv.fit(X_train, y_train)

# Add some unseen category to the test.
X_test['cat'] = 'UUUUU'

pipe_cv.predict(X_test)

Note that the code is not optimal but it should show you how to deal with this problem of doing target encoding with the train and test using a pipeline, and working with unseen data :)
Note that the category has been assigned randomly. So the model detects that the best is predicting the most frequent class. If you change for ElasticNet (a regressor) you will get the mean.
If you take out the unseen category assignation to test you will still get the same results


How can a CNN learn colours? I've created spectrogramms of different classes of low-hertz signals. They all have a plain blue foreground with hardly other coloured pixels, even for me its not easy to distinguish the classes by human sight. Now I'd like to train a CNN to do binary classification on these spectrogramm images. No matter how I build the network with no matter what configurations and parameters - it doesn't learn, loss doesn't decrease.
I figured out the reason: the convolutional filters of CNNs are good at distinguishing forms and shapes, but not really plain colours.
How can CNNs learn colours? Am I missing something? Perhaps there are more suitable models than CNNs?
Here is an example of the two classes that hardly differ, only in the colour hue:

2 more images:

EDIT1: The two classes have images that not all look the same with the same prevailing colour; some have yellow/greenish stripes in them, some have lighter or darker colour etc. So the classes do not only consist of the feature colour. My goal is to classify these images using CNNs, but somehow CNNs fail at learning from images, that almost only consist of colour, whereas CNNs learn well if there are sharp edges, boundaries, object-like elements in the image.

CNNs can learn colors, there are CNNs developed just to check colors on cars. But there are some points worth analyzing when dealing with classification when color is more relevant feature than shape.
1 - Color Space
Digital Color Images are usually represented as a pixel 2D Grid, where each pixel is a vector with 3 or 4 elements (the later case to define opacity). Usually, we deal with images using the RGB Color Space, as this is a representation used by our retinas to color cells.
But there are multiple Color Spaces that can be used to represent images, mostly 3 dimensional as RGB but each one has a different approaches and the transformation between them is not always linear. Check using HSV and HSL to check which is best for your problem. (The previously mentioned paper, has other examples).
2  - Architecture:
There are a bunch CNNs architectures, developed with different intuitions and objectives. For example U-Net is developed with image segmentation in mind, reducing information loss by keeping shallow paths along with the deep paths.
Most architectures are created with shape in mind, trying to catch local nuances in the image, and usually use small kernels such as 3x3, 5x5 etc.
Dealing with colors as the main feature, these local differences might not be the most relevant feature you want your kernels to learn to extract, more global patchs (i.e larger kernels) might be better.
Specifically for your problem:
These plots can be seem as surfaces and have some known relations between vertical and horizontal axes, maybe non-symmetric kernels might be more functional, and also avoid the speed loss in using too large kernels.
I would go with layers using alternating sizing. For example 30x3 then 3x30 kernels, with no padding.


NLP : Rules for chunking Verb Phrases I have read a lot of documentation surrounding NP chunking but what about Verb Phrases? Has there been a fixed set of rules for VP chunking?

There are many ways to perform chunking/phrase structure parsing. The most common methods are:

Rule-based
Statistical models
Deep Learning

Fixed rules are simple and fast. But they make many mistakes given the complexity of human language. The other methods are typically more useful because they generate probabilistic rules.

Context-free grammar (CFG) is a set of rules for the production of formal language and can be used for parsing.


NLP - Simple approach to identify commonalities in text comments between people For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following:

What are the common things (things defined as phrases/sentences) they are saying (Some way to quantify the commonality too if possible). The point is to identify what seems to be areas of agreement about their review
What are things that are not common (basically...what are those on-off sentences/phrases that have been told that are very uncommon)
Where is there disagreement (i.e. are there sentences/phrases where there is disagreement potentially between the responses)

The goal is to find a simple solution to this and not necessarily model driven (there is paucity of data). Also...it needs to be directional at this time...as the goal is to prove that this can work and can product reasonable results.

Any help advise?

Thanks much!

PS: We would need some 'intelligence' in identifying the commonality or vice-versa (i.e. different words meaning same within a phrase should be considered common).

Note: The questioner requests for the response to be simple. Although this approach below is not simple, it is an attempt to provide a perspective that could help.

Understanding the problem stated:
At a conceptual level, there are arguably 3 concepts in this question as given below.

Semantic similarity i.e. how similar the responses (in meaning) are to one another. In a way, this similarity could be loosely inferred as similar proposition.
Syntactic similarity i.e which aspects (tokens or entities or chunks) overlap between the responses.
Text classification i.e. "is one feedback in agreement with another or neutral or against?".

Evaluating approaches:
There could be many potential solutions. Following steps are one such attempt.

Step 1 Deploy a cosine similarity algorithm to measure the similarity between responses. In order to bring it a step closer to semantic similarity, use WORDNET to build the features for computing cosine similarity. This will ensure that tokens such as "path" are treated closer to token "road".
Step 2 Group responses (for the same question) beyond a threshold cosine value (example: 0.75) as similar responses. This can be loosely considered as a set of different responses for a question.
Step 3 Train a model to identify agreement or disagreement between responses. This training can be based on a classification algorithm or at the least based on a bag of words approach(hard coded tokens such as "i dont agree", "it is incorrect" etc). This step is perhaps the least scientific one but the only pragmatic one that the author can think of.


Error on CGAN / InfoGan So i'm trying to execute the code provided here: https://github.com/eriklindernoren/Keras-GAN/blob/master/infogan/infogan.py

Both for InfoGAN and CGAN i get the same error:

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-489e5ad1b538> in <module>()
      1 infogan = INFOGAN()
----> 2 infogan.train(epochs=50000, batch_size=128, sample_interval=50)

7 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/cbook/__init__.py in to_filehandle(fname, flag, return_opened, encoding)
    390             fh = bz2.BZ2File(fname, flag)
    391         else:
--> 392             fh = open(fname, flag, encoding=encoding)
    393         opened = True
    394     elif hasattr(fname, 'seek'):

FileNotFoundError: [Errno 2] No such file or directory: 'images/0.png'

Can someone please help?

In Line 207, generated pictures are written to a directory images, which must exist. In your case it does not, that's why it throws that error.


Standardizing features by one specific feature I am working on a project with a dataset that looks something like the following:

      velocity   accel_amp         f_vert   tau_vert      f_pitch_filt   tau
0    3.778595      -5.796777     2.400000  32.753227      1.600000   27.844535
1    1.970611      -6.087134     2.272727  32.638705      1.704545   30.639998
2    3.581163      -6.241817     2.400000  32.850969      1.600000   30.449256
3    4.735210      -6.109532     1.400000  28.809865      1.000000  127.749313
4    5.340568      -6.614317     1.400000  20.249699      1.000000  124.549628

I was suggested to standardize the last 5 features by velocity in order to improve my PCA.
Does this simply mean to take each element in these last 5 columns, subtract the mean of the velocity column, and then divide by the standard deviation of the velocity column?

That is how I interpreted this suggestion. Is there a functionality in Python for doing this? Any suggestions or clarification would be appreciated.

Thanks.

If this is a pandas DataFrame:

vel_mean = df.velocity.mean()
 vel_std = df.velocity.std()
 df = df.apply(lambda x: (x - vel_mean) / vel_std)

In case it is a numpy array:

data = (data - np.mean(data, axis=0)[0]) / np.std(data, axis=0)[0]


You can use the built-in Pandas functions to do it:

df["Time stamp"] = pd.to_datetime(df["Time stamp"]) # Convert column type to be datetime
indexed_df = df.set_index(["Time stamp"])           # Create a datetime index
indexed_df.rolling(100)                             # Create rolling windows
indexed_df.rolling(100).mean()                      # Then apply functions to rolling windows

This code is not 100% correct because the datetime conversion is not correct but it should help you get started.


Python Keras NN, handling n_samples of float outputs I am using Keras with Theanos backend in Python.

I have 2117 samples and each sample has an individual target (on purpose) ie. 2117 outputs.

As opposed to categories, the targets are ratings eg. (16.4714494876, 17.4129353234, 17.4476570289) the entirety of the number is important.

I am having problems/don't know where to start.

1) When i run the NN it only outputs the targets as whole integers as opposed to the format of the actual values. eg. 16 instead of 16.xxxxxx

2) Presumably i will only be able to gauge the accuracy of predictions based on how close the output is to the target since there are so many targets, does this type of classification problem have a name that i can research?

3) In 3 research papers i have read that apply NN to my specific classification problem they list the output layer as only having 1 neuron but provide no further explanation, how could this be?

Here is my model.

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

X = np.array(df[FEATURES].values)
Y = np.array(df["MTPS"].values)

# define baseline model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(10, input_dim=(len(FEATURES)), init='normal', activation='relu'))
    model.add(Dense(2117, init='normal', activation='softmax'))
    # Compile model
    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
#build model
estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=2)

#cross validation
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=seed)
estimator.fit(X_train, Y_train)

#print class predictions
print estimator.predict(X_test)
print Y_test

Thanks for any help.

I'm confused. As you have said, the targets are ratings. It's definitely a regression problem to me, instead of a classification problem.
There's several problems in your code:

For regression problem, we usually use linear as activation function of the last layer (sometimes relu, even sigmoid).
And also we use mse as metric(sometimes mae, msle, etc). categorical_crossentropy is used for classification problem, and sparse_categorical_crossentropy is used for sparse input classification problem. Ref
KerasClassifier is used for classification problem, use KerasRegressor instead.
metrics=['accuracy'] is used for classification problem, and it's meaningless in regression problem. Ref
Assuming df is a Pandas DataFrame, then df.values is naturally a ndarrary, there's no need to cast np.array.

Now answering your question:

As I've never used wrappers for scikit-learn API, I'm not very sure why you have integer output, my best guess is because of KerasClassifier and .predict(in scikit-learn APIs, .predict returns integer, basically the predicted class and .predict_proba returns float, indicating the probability of each class). Try to use .predict_proba, it would help. BTW, you should really use KerasRegressor.
As you have mentioned, there's so many classes to predict. In fact, it's a regression problem.
Could you please add the links of these 3 papers? Neural networks having only 1 neuron in output layer seem to be a regression NN to me. Regression NN usually uses Dense(1, activation='linear') as the output layer.

Here's my version of your code, it might work:
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

X = df[FEATURES].values  # no need to cast np.array
Y = df["MTPS"].values

# define baseline model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(10, input_dim=(len(FEATURES)), init='normal', activation='relu'))
    model.add(Dense(1, init='normal', activation='linear'))  # one neuron, linear activation
    # Compile model
    model.compile(loss='mse', optimizer='adam')  # mse loss
    return model
#build model
estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=2)  # KerasRegressor for regression problem

#cross validation
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=seed)
estimator.fit(X_train, Y_train)

#print class predictions
print estimator.predict(X_test)
print Y_test

References


How is PACF analysis output related to LSTM? I was going through a recent paper “A Novel Hybrid Data-Driven Model for Daily Land Surface Temperature ForecastingUsing Long Short-Term Memory Neural Network Based on Ensemble Empirical ModeDecomposition” found here.

The EEMD-LSTM scheme mentioned is indeed a very well designed model for predicting land surface temperature with more accuracy. I was reading and trying to implement the given steps (as mentioned in the publication) working with similar climatological time series data. I got the different IMFs and the residual.

For the next step, which is to feed all the IMFs and Residual to LSTM, as in Section 3.5 (Forecasting IMFs), the paper mentions that to fix input of the LSTM, a PACF (Partial Auto Correlation Function) analysis has to be done and the number of lags beyond 95% Confidence Interval(CI) is the number of inputs to LSTM.

I understand you are getting the PACF order for the decomposed results (for example: 4,6,5,5,6,6,1,1,1,1,1 for the max pooling station as shown in Fig 7 of the publication where 4 is the number of lags for original data series, 6 is the number of lags for IMF1 series,...., 1 is the number of lags for IMF9 series, and 1 is the number of lags for Residual series).

My question is: how exactly is this PACF order related to input of LSTM (does it determine the number of past data points to look back)?

It would be great if someone could provide some explanation on this.

Thanks.

When used for time series, LSTM (or any Neural Network approach) fall under the category of auto-regressive models. The most general formulation of an auto-regressive model is:

$Y_t = f(Y_{t-1}, Y_{t-2},...,Y_{t-n})$ with n the number of past lags to include in your model.

When using a neural network for univariate time series modeling, n will determine the number of input neurons to use (It could be n, or could be n+k, with k additional causal features).

The PACF will help you determine n.

The key here is that neural networks are a generalization of AR(p) models.


How to classify features into two classes without labels? I have a big dataset with nearly 200 features. However, I do not have class labels for these data. I want to divide these data into two classes based on these features. I know, when we do not have class labels we have to use some clustering method. However, since I do not have any labels, I am just wondering how to measure the accuracy of the models.

Please let me know the most suitable approach that I should follow?

I am happy to provide more details about my featureset if needed :)

As you know, clustering is Unsupervised learning algorithm. Since you don't know the number of clusters, it becomes hard to find the best possible separation (number of clusters). There is a very good paper published on validating clustering techniques. There are 3 criteria defined in this paper for validating your clusters. You can take a look at that.

Also, the link below gives you some code in R for cluster validation. If you want you can try this too-

http://www.sthda.com/english/articles/29-cluster-validation-essentials/97-cluster-validation-statistics-must-know-methods/


These columns are actually arbitrary, they do not represent anything for humans. However, it does not mean they are useless, quite the opposite - computers can extract features from this highly dimensional space more easily, such as in neural networks.

To extract information which is useful for humans, I recommend to have a look at examples in original paper by Tomas Mikolov:

We can have a look at common capital city.  Athens is to Greece (in sense that they have similar relationship) as is Oslo to Norway.
We can use afterwards vectors obtained using GloVe and construct

$$v(Athens) - v(Greece) + v(Norway)$$

and this vector in 'globally' (meaning that trained on universal dataset, such as wikipedia) trained model would be closest to $v(Oslo)$.


one single value target variable I have a set of 20 observations, the target variable which is a binary variable (0 or 1), in this set has one unique value which is 1.

I want to predict this variable for a large sample, so I want to use a regression algorithm. The problem is that the target variable take one value, there is no observation that can describe the 0 class.

Is there any solution to handle this situation.

Thank you in advance.

Welcome to the site! As Media has mentioned it is a classification problem not prediction problem. If you meant Logistic Regression then it makes sense.

You need to remember one thing if the outcome is only one value then algorithm would also return the same value irrespective of anything. if you don't have negative cases then you cannot use Logistic regression and it doesn't know when to say 0 as the model wasn't trained on it.

If possible using Business Matter Experts advice populate some dummy data for 0's and balance the data. Train your model and see what the model is going to say.

Else you cannot use any kind of classification algorithm.


A timestep's prediction depends on future data Consider an LSTM model with 100 timesteps, each of which with input and target data. Let f(99) be the function mapping the input data of the 99th timestep and hidden state of the 98th timestep to the output data of the 99th timestep.

The mapping f(99) of another LSTM model will be different if the other LSTM model is fitted only to the data in the first 99 timesteps. It follows that future data -- that is, data from the 100th timestep -- was used to generate the 99th timestep's prediction in the first LSTM model.

What are the implications of this for forecasting methodology?

I think you have assumed that future timesteps also are used for learning by a simple LSTM. But, that actually is Bidirectional LSTM which learns from the past as well as future timesteps. Simple Recurrent Neural Networks don't have the access to future input information. You might want to refer to a book on recurrent neural networks. You can also look this question on cross validated for training an LSTM.