Gradient descent and partial derivatives I'm on the first chapter of Nielsen's book on neural networks. He gives an example of a gradient descent and that change in function $C(v_1,v_2)$ is described by this formula:

$$\begin{eqnarray}\Delta C \approx \frac{\partial C}{\partial v_1} \Delta v_1 +\frac{\partial C}{\partial v_2} \Delta v_2\tag{7}\end{eqnarray}$$

Though I'm familiar with partial derivatives, i'm confused about how you would compute partial derivatives of this function.

Could someone give an example please?

What's the point?

First, it is good to understand what we are doing that leads us to need these tools. When we are trying to apply machine learning we want to infer some meaning from data. This means that given an instance we want to put it through our model and then we can have some output that tell us something about this data.

Let's look at the example where we try to label whether an image contains a cat or a dog. If we have a perfect model, we can give the model a picture and it will tell us if it is a cat or a dog. However, no model is perfect and it will make mistakes.

When we train our model to be able to infer meaning from input data we want to minimize the amount of mistakes it makes. So we use a training set, this data contains a lot of pictures of dogs and cats and we have the ground truth label associated with that image. Each time we run a training iteration of the model we calculate the cost (the amount of mistakes) of the model. We will want to minimize this cost.

Many cost functions exist each serving their own purpose. A common cost function that is used is the quadratic cost which is defined as

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

This is the square of the difference between the predicted label and the ground truth label for the $N$ images that we trained over. We will want to minimize this in some way. The cost is also commnly called the loss function.

So we have a minimization problem on our hands!

Indeed most of machine learning is simply a family of frameworks which are capable of determining a distribution by minimizing some cost function. The question we can ask is "how can we minimize a function"?

Let's minimize the following function

$y = x^2-4x+6$.

If we plot this we can see that there is a minimum at $x = 2$. To do this analytically we can take the derivative of this function as

$\frac{dy}{dx} = 2x - 4 = 0$

$x = 2$.

However, often times finding a global minimum analytically is not feasible. So instead we use some optimization techniques. Here as well many different ways exist such as : Newton-Raphson, grid search, etc. Among these is gradient descent.

How does gradient descent work?

Let's use a famously used analogy to understand this. Imagine a 2D minimization problem. This is equivalent of being on a mountainous hike in the wilderness. You want to get back down to the village which you know is at the lowest point. Even if you do not know the cardinal directions of the village. All you need to do is continuously take the steepest way down, and you will eventually get to the village. So we will descend down the surface based on the steepness of the slope.

Let's take our function

$y = x^2-4x+6$

we will determine the $x$ for which $y$ is minimized. Gradient descent algorithm first says we will pick a random value for $x$. Let us initialize at $x=8$. Then the algorithm will do the following iteratively until we reach convergence.

$x^{new} = x^{old} - \nu \frac{dy}{dx}$

where $\nu$ is the learning rate, we can set this to whatever value we will like. However there is a smart way to choose this. Too big and we will never reach our minimum value, and too large we will waste soooo much time before we get there. It is analogous to the size of the steps you want to take down the steep slope. Small steps and you will die on the mountain, you'll never get down. Too large of a step and you risk over shooting the village and ending up the other side of the mountain.

$\frac{dy}{dx} = 2x - 4$

$\nu = 0.1$

Iteration 1:

$x^{new} = 8 - 0.1(2 * 8 - 4) = 6.8 $
$x^{new} = 6.8 - 0.1(2 * 6.8 - 4) = 5.84 $
$x^{new} = 5.84 - 0.1(2 * 5.84 - 4) = 5.07 $
$x^{new} = 5.07 - 0.1(2 * 5.07 - 4) = 4.45 $
$x^{new} = 4.45 - 0.1(2 * 4.45 - 4) = 3.96 $
$x^{new} = 3.96 - 0.1(2 * 3.96 - 4) = 3.57 $
$x^{new} = 3.57 - 0.1(2 * 3.57 - 4) = 3.25 $
$x^{new} = 3.25 - 0.1(2 * 3.25 - 4) = 3.00 $
$x^{new} = 3.00 - 0.1(2 * 3.00 - 4) = 2.80 $
$x^{new} = 2.80 - 0.1(2 * 2.80 - 4) = 2.64 $
$x^{new} = 2.64 - 0.1(2 * 2.64 - 4) = 2.51 $
$x^{new} = 2.51 - 0.1(2 * 2.51 - 4) = 2.41 $
$x^{new} = 2.41 - 0.1(2 * 2.41 - 4) = 2.32 $
$x^{new} = 2.32 - 0.1(2 * 2.32 - 4) = 2.26 $
$x^{new} = 2.26 - 0.1(2 * 2.26 - 4) = 2.21 $
$x^{new} = 2.21 - 0.1(2 * 2.21 - 4) = 2.16 $
$x^{new} = 2.16 - 0.1(2 * 2.16 - 4) = 2.13 $
$x^{new} = 2.13 - 0.1(2 * 2.13 - 4) = 2.10 $
$x^{new} = 2.10 - 0.1(2 * 2.10 - 4) = 2.08 $
$x^{new} = 2.08 - 0.1(2 * 2.08 - 4) = 2.06 $
$x^{new} = 2.06 - 0.1(2 * 2.06 - 4) = 2.05 $
$x^{new} = 2.05 - 0.1(2 * 2.05 - 4) = 2.04 $
$x^{new} = 2.04 - 0.1(2 * 2.04 - 4) = 2.03 $
$x^{new} = 2.03 - 0.1(2 * 2.03 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $

And we see that the algorithm converges at $x = 2$!

Where do partial derivatives come into this?

We just minimized $y$ for $x$. But if we had a multi-dimensional $x$ then we will need to identify where is the steepest slope in each dimension. To do this we will take the derivative across each dimension. In 2 dimensions gradient descent is performed using the gradient of the function $\nabla y$.

For the function

$y = 2x_1^2 + 5x_1 - 4x_2^2$

we will take the gradient in both dimensions defined as

$x^{new} = x^{old} - \nu \nabla y$.

The operator $\nabla$ is called the gradient it is defined as

$\nabla y = \frac{\partial y}{\partial x_1} \hat{\textbf{x}_1} + \frac{\partial y}{\partial x_2} \hat{\textbf{x}_2}$.

$\hat{\text{x}_1}$ and $\hat{\text{x}_2}$ are the two dimensions that we are optimizing over. So in our analogy we will go down this 2D surface a bit in the $x_1$ dimension and the $x_2$ dimension. Kind of like going down the mountain in the North-West direction. A bit in front and a bit to our left. That's a 2D direction!

The total size of the step that we take across all dimensions is then described by

$\Delta y = \frac{\partial y}{\partial x_1} \Delta x_1 + \frac{\partial y}{\partial x_2} \Delta x_2$.

Relating all this to your question

Change our function $y(x_1, x_2)$ by $C(v_1, v_2)$. The size of the step we are taking is defined by

$\Delta C = \frac{\partial C}{\partial v_1} \Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2$.

At each gradient descent iteration we will update $v$ as

$v^{new} = v^{old} - \nu \nabla C$

where

$\nabla C = \frac{\partial C}{\partial v_1} \hat{\textbf{v}_1} + \frac{\partial C}{\partial v_2} \hat{\textbf{v}_2} = (\frac{\partial C}{\partial v_1}, \frac{\partial C}{\partial v_2})^T$

Supplemental

2D gradient descent of a function

$y = x_1^2 - x_1 + x_2^2 - 3x_2$

The partial derivatives are

$\frac{\partial y}{\partial x_1} = 2x_1 - 1$

$\frac{\partial y}{\partial x_2} = 2x_2 - 3$

Thus,

$\nabla y = (2x_1 - 1, 2x_2 - 3)$.

Our update rule with a learning rate $\nu = 0.1$ will then be

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.0 \\
0.0
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.0 - 1 \\
2 * 0.0 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.1 \\
0.3
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.1 \\
0.3
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.1 - 1 \\
2 * 0.3 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.18 \\
0.54
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.18 \\
0.54
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.18 - 1 \\
2 * 0.54 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.24 \\
0.73
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.24 \\
0.73
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.24 - 1 \\
2 * 0.73 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.29 \\
0.88
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.29 \\
0.88
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.29 - 1 \\
2 * 0.88 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.33 \\
1.00
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.33 \\
1.00
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.33 - 1 \\
2 * 1.00 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.36 \\
1.10
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.36 \\
1.10
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.36 - 1 \\
2 * 1.10 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.39 \\
1.18
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.39 \\
1.18
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.39 - 1 \\
2 * 1.18 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.41 \\
1.24
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.41 \\
1.24
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.41 - 1 \\
2 * 1.24 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.43 \\
1.29
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.43 \\
1.29
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.43 - 1 \\
2 * 1.29 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.44 \\
1.33
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.44 \\
1.33
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.44 - 1 \\
2 * 1.33 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.45 \\
1.37
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.45 \\
1.37
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.45 - 1 \\
2 * 1.37 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.46 \\
1.39
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.46 \\
1.39
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.46 - 1 \\
2 * 1.39 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.47 \\
1.41
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.47 \\
1.41
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.47 - 1 \\
2 * 1.41 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.47 \\
1.43
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.47 \\
1.43
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.47 - 1 \\
2 * 1.43 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.48 \\
1.44
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.48 \\
1.44
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.48 - 1 \\
2 * 1.44 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.48 \\
1.45
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.48 \\
1.45
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.48 - 1 \\
2 * 1.45 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.48 \\
1.46
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.48 \\
1.46
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.48 - 1 \\
2 * 1.46 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.47 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.47 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.48 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.48 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.48 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

And we have convergence in 2D at those points above!


Can an R^2, or coefficient of determination be used on non-linear data? I have used the $R^2$ metric to determine how well my neural network performs a non-linear regression. And it seems to work. The plots look almost identical, and I get an $R^2$ value of 0.93... it seems to work perfectly,  however I have been told that $R^2$ should only work for linear data. Is this true or false?

The $R^2$ calculation for nonlinear regression models can very a lot depending on the software / application / etc. Also, they involve a lot of approximations in order to make any calculation and are not as easily interpretable as the linear case. There are many other methods that will give you a good and very interpretable result directly from your data. For example, some kind of cross-validation like K-fold cross-validation or bootstrap methods give you a good sense of how trustworthy your estimates are directly from your data without relying on a bunch of normal approximations. Lastly, for most applications I would also recommend using some goodness of fit measure that takes into account parsimony like the AIC or BIC.

If it's a linear regression (linear in the parameters), then yes the $R^2$ is fine and is interpretable in the commonly thought of way.

$R^2$ is good for cases where you have no model for the noise in your measurement. A good way to think of it is just as a measure of the fraction of the variance in the measurement that's covered by your fit model. But it's not necessarily what you want for cases where you're concerned you might be overfitting your data (i.e. using a fit function with more parameters than you can justify) or where the error bars on different data points are very different from one another (as happens, for example, when Poisson noise dominates). Chi2, and especially reduced chi2 (i.e. chi2 per degree of freedom) are probably more appropriate for these cases.


Python: multiplication of sparse matrices slower in csr_matrix than numpy I am trying to multiply a sparse matrix with itself using numpy and scipy.sparse.csr_matrix. The size of matrix is 128x256. Its 93% values are 0. Ironically the multiplication using numpy is faster than scipy.sparse. I do not know why? The code I am using is:

import numpy,time
W=numpy.random.choice([0, 1], size=(128,256), p=[0.93,0.07])
start=time.time()
W1=numpy.matmul(W,numpy.transpose(W))
end=time.time()
print(end-start)

from scipy.sparse import csr_matrix
start=time.time()
W1=csr_matrix(W).dot(csr_matrix(W).transpose())
end=time.time()
print(end-start)

Numpy gives time 0.0006 and scipy gives 0.004. Why. Comparing times for dense matrix, numpy gives smaller time on dense matrix as well and scipy takes more time. Why is the time for scipy.sparse not less than numpy for sparse matrix

Here, you do not time only the time taken to make the matrix multiplication but also the time taken to convert your matrix from dense to sparse. If you convert your matrix before the timing starts, you will see that multiplication with scipy is indeed more than twice faster

import time, numpy, scipy
from scipy.sparse import csr_matrix
import numpy as np

W = np.random.binomial(n=1, p=0.01, size=(100, 100))
start=time.time()
numpy.matmul(W,numpy.transpose(W))
end=time.time()
dt_dense = end - start
print ('time taken for the dense matrix {}'.format(end - start))

sparse_W = csr_matrix(W)
start=time.time()
sparse_W.dot(sparse_W.transpose())
end=time.time()
dt_sparse = end - start
print ('time taken for the sparse matrix {}'.format(end - start))
dt_dense/dt_sparse


Adding extra variables to XGboost model is worsening the train and test accuracy I am fitting a multi class model using Xgboost. I am getting an accuracy of 96% on Train and 95% on test. I am using the 80-20 train/test split.
However, when I am adding two new features , the accuracy drops down to 92% for train and 89% for test.

Doesnt XGBoost:

Pick the most important variable that can be used to split a node and leave out the rest?
Handle multi colinearity?

I have not used cross validation. Could it be that I am still overfitting the data ?

This is the code I used

from sklearn.model_selection import train_test_split
df_new_train, df_new_test, y_train, y_test = train_test_split(df, labels2, test_size = 0.2)

dtrain = xgb.DMatrix(df_new_train, label=y_train)
dtest = xgb.DMatrix(df_new_test, label=y_test)

param = {
        'max_depth': 10,
        'early_stopping_rounds': 10,
        'eta': 0.01,
        'subsample': 0.6,
         'colsample_bytree': 0.5,
        #'alpha': 0.5,x`
        #'lambda': 0.5,
        'gamma': 10,
        'min_child_weight': 1,
        'watchlist': [(dtrain, 'train'), (dtest, 'valid')],
        'objective': 'multi:softprob',  # error evaluation for multiclass training
        'num_class': 4}  # the number of classes that exist in this datset
num_round = 1500

bst = xgb.train(param, dtrain,  num_round)

preds = bst.predict(dtest)
preds_train = bst.predict(dtrain)

best_preds_train = np.asarray([np.argmax(line) for line in preds_train])

best_preds = np.asarray([np.argmax(line) for line in preds])

print(classification_report(y_test,best_preds,target_names=label_encoder.classes_ ))
```

This is something you can try, remove the parameter early stopping rounds and the number as I see is relatively low. It could be that xgboost stopped training before it is supposed because it may not have seen any improvement. Also you might consider setting random seed to get consistent result between attempts.


Understanding cosine distance with word vectors I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, there's an obvious association with the "ing" in king and the "ing" in other words that probably don't have much to do with king. Am I missing something about how a word2vec formula uses the characters inside the words it is mapping? Or is my net just really untrained?

Also, what does the cosine distance between those example words say to you about the results, if anything? Thank you for your advice!

   List<String> abc = vec.similarWordsInVocabTo("king", 0.8); //80% similar
   System.out.println(abc);

   String[] words = {"woman", "king", "boy", "child", "human"};
   for (String word : words) {
       System.out.println(vec.similarity("man", word));
   }

Output - Similar words to king:

[taking, drinking, kingly, picking, waking, singing, wringing, knight, feigning, beginning, ink, thinking, kin, knocking, making, bringing, knowing, lingring, winking, neighing, king-, kings, asking, stinking, king, liking]

Output - Vector similarity between "man" and woman, king, boy, child, human:

woman:   0.8305895924568176
king:    0.00203840178437531
boy:     0.2974374294281006
child:   0.4752597510814667
human:  -0.10414568334817886

Word2Vec algorithm does not go inside words. Word “king” is never used as a gerund, so there is no reason why it should be similar to gerunds.

My guesses are:

Your corpus might by wrongly tokenized. Maybe there are some OCR-related errors with word splitting something like “li-↲ king”.
You might be using a different algorithm for getting the embeddings (e.g., FastText) that goes inside the words and infers the word embedding as some embeddings of character n-grams the word consists of.

On the other hand, words similar to man look fine. If you think about how Word2Vec (and also FastText) is trained, you should not ask a question: ”Do the words have as similar meaning as possible?” but rather “Does the word appear similarly frequently in a similar context Shakespear's works?”

(Of course, when the embeddings are trained on data which is large enough, there is almost no difference between these two questions.)


Best Model for Multi-label image classification Usually using CNN architecture with a Sigmoid function as an activation function in the last layer and using binary cross entropy can be used to output a probability for each class. However, the problem is when we have more than 2 labels per image, the model will result in lower accuracy. Anyone has a suggestion for a robust state-of-art models that do Multi-label image classification  with a high accuracy.

The models will stay the same for the case like ResNets, EfficientNets, NFNets and MobileNet amongst many others. To start with go EffNetB4, good training speed and accuracy. These are great feature extractors what you do with them bring another complexity so in multi-label, look at using label_smoothing for regularization.


Semantic segmentation of an image with multiple labels per pixel I am building a model for a multiclass sematic segmentation of a skin disease. At a moment I am using U-Net for binary classifications.
In this multiclass problem I have the following cases. There are four types of skin damage. There are four degrees of damage for each skin damage type: healthy, mild, moderate, severe. Healthy skin is A0_B0_C0_D0, "mild b and severe c" corresponds to A0_B1_C3_D0. I would like to train a single multiclass model to predict a dictionary {A: scoreA, B: scoreB, C: scoreC, D: scoreD}. Given that
4(damage types)**4(damage degrees) = 256 combinations
Do I need to train a 256 class model? My concern is that I only have 200 images in my training set and some of the combinations will not appear at all in the training set. Is there a way to train a 12 class model returning four values corresponding to  the most likely damage type for each damage type?
Update. Consider an rgb image. You want to classify brightness of each color channel in categories
Intensity   class
 0-63 -> "0"
64-127 -> "1"
128-191 -> "2"
192-255 -> "3"

Then each pixel belongs to one of 64 classes (r0g0b0, r0g0b1, ..., r3g2b2, r3g3b3). The training set has pixels of colors r0, r1, r2, r3, g0, g1, g2, g3, b0, b1, b2, b3, but it has no pixels of color r0g1b2 or of color r2g3b0. Three separate models (one per channel) will easily learn to predict the channel category, but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen those classes. How to overcome this problem?

Three separate models (one per channel) will easily learn to predict the >channel category,  but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen these classes. How to overcome this problem?

The only way to solve this problem is to use channels (one for each skin damage type) for each pixel, and treat it as a regression rather than a classification.
In other words, use a multi output regression.
For your output, use a convolutional layer that gives the same number of rows and columns as the input, but 4 channels (one for each skin damage type).
Your ground truth (y_true) should be an array the same width and height as your input, but with 4 channels (one for each skin damage type) with each holding the severity rating of that pixel for the corresponding skin damage type.
Your loss function could be something used for regression such as MAE (mean absolute error).
This is because a classification will output 0 for classes it has never seen samples of since that is what minimizes the loss. A regression, on the other hand, will treat the target variable as continuous, and even if it hasn't seen examples of all severity levels of Type A skin damage (for example), it can still output them.
You could then choose thresholds to classify the outputs into the corresponding labels.
Ex. for each output <0.5 is 0, >0.5 is 1, >1.5 is 2, >2.5 is 3, >3.5 is 4
so [0.1, 1.6, 0.7, 5] means {A: 0, B: 2, C: 1, D: 4}
This is also more useful and easier to interpret in a clinical context as the severity is ordinal rather than just categorical, so a doctor would be better off knowing that a particular mild case (for example) had a prediction of 1.4 rather than 0.6, because they both correspond to a mild prediction but a 1.4 is closer to moderate and may be treated differently than a 0.6
This is an interesting problem from a learning and research perspective, but I develop deep learning based prognosis and diagnosis models using medical images at a big pharma/life sciences company and can tell you that a dataset of 200 images for a task this complex will be insufficient for decent performance or reliable results. multi-fold training/validation/testing AND some slight image augmentation will be necessary, but probably still insufficient.


Transform multi-label problem to multi-class problem What are the downsides of modelling a multi-label problem as a multi-class problem with a single classifier?

Let my clarify what I mean.

There at least two ways that one multi-label problem can be transformed to a multi-class problem with a single classifier (suppose there are N labels at our problem):

1)

Create a class for each element of the powerset of the labels.

Therefore, each element for each combination of the labels.

In this case, the output vector will have $2^N$ length.

2)

Have an output vector of $N$ length (every element of the vector will be one label) but the problem will be treated as a multiclass problem with one classifier.

In this case, let's say the classes which have output probability more than 0.2 will be considered as the classes/labels of this instance/observation.

Obviously, the output probabilities of all the classes should sum up to 1.

What are the implications of transforming a multi-label problem to multi-class problem at each of these cases?

A multi-label problem is when an instance can have several labels, for instance a system which classifies news articles by topic could do this:

instance 1: politics, society
instance 2: sports
instance 3: culture, sports
instance 4: society
...

To turn this into a multiclass problem and still do the exact same task, one needs to create one class for each possible subset existing in the data, for instance:

{ politics-society, sports, culture-sports, society, ...}

If the original multi-labels problem contains $N$ labels, the number of classes in the multiclass problem is $2^N$ in the worst case (number of partitions of the set).

The main problem is that the classifier needs a representative sample of every class in order to perform well. The classes sports and culture-sports (for example) are now independent from the perspective of the classifier, so the class sport cannot benefit anymore from the instances belonging to culture-sports, as it would be the case in the multi-label problem.

So in general one would need many more instances to train a multiclass classifier than a multi-label classifier for doing the same thing.


Why multiply by 2 when calculating partial derivatives during backpropagation? I'm wondering why we multiple by 2 when calculating partial derivatives.

I'm referencing the 2's that I've circled below, from here.

We also see this in the python implementation,
m_deriv += -2*X[i] * (Y[i] - (m*X[i] + b)), there is a 2.

How can I understand the intuition here?

My math is not strong so apologies in advance if this is a silly question.

You're using the Mean Square Error $\Sigma\frac{1}{N}(y-(Wx+b))^2$as the loss function, if you take the derivative, you will have the $2$. In some materials, we will use $\frac{1}{2}\Sigma\frac{1}{N}(y-(Wx+b))^2$ as the loss function to cancel out the $2$. In fact, this doesn't matter at all and it has no impacts on params optimization.


Building a machine learning model to predict crop yields based on environmental data I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data.

Please note that the dataset has DAILY values for temperature and precipitation, but only 1 value per year for the yield, since harvesting of crop happens at end of growing season of crop.

I want to build a regression or some other machine learning based model to predict 2015 yields, based on a regression/some other model derived by studying the relation between yields and temperature and precipitation in previous years.

I am familiar with performing machine learning using scikit-learn. However, not sure how to represent this problem. The tricky part here is that temperature and precipitation are daily but yield is just 1 value per year.

How do I approach this?

For starters, you can predict the yield for the upcoming year based on the daily data for the previous year. You can estimate the model parameters by considering each year's worth of data as one "point", then validate the model using cross-validation. You can extend this model by considering more than the past year, but look back too far and you'll have trouble validating your model and overfit.


Regex remove 1-2 character sequences: hyphens literal, not word boundaries I need a regex in R to exclude 1 or 2-character words, but which does not treat hyphens as word boundaries.  Here is an example:

z <- c("regulatory protein SR-B1 na na na na", "Dr Foo is na Editor-in-chief", "na P3 protein is popular na na")

The intended 'gsub' would produce the result

  [1] "regulatory protein SR-B1" " Foo Editor-in-chief"     "P3 protein  popular"

This is not produced from the gsub statements below. Instead:

A - 2-character segments unintentionally dropped from hyphenated words (and hyphens kept).

B - 2-character segments unintentionally dropped within hyphenated words (hyphens also dropped):

C - Produces: Error: '-' is an unrecognized escape in character string starting ""
*\b[[-"

D - Nothing dropped or changed:

gsub(" *\\b[[:alpha:]]{1,2}\\b *", " ", z) # A

gsub(" *\\b[-[:alpha:]]{1,2}\\b *", " ", z)  # B

gsub(" *\\b[[\-][:alpha:]]{1,2}\\b *", " ", z)  # C

gsub(" *\\b[[\\-][:alpha:]]{1,2}\\b *", " ", z) # D

Input would be much appreciated.


Number of parameters in an LSTM model How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the number of parameters is useful for training models using LSTMs.

The LSTM has a set of 2 matrices: U and W for each of the (3) gates. The (.) in the diagram indicates multiplication of these matrices with the input $x$ and output $h$.

U has dimensions $n \times m$
W has dimensions $n \times n$
there is a different set of these matrices for each of the three gates(like $U_{forget}$ for the forget gate etc.)
there is another set of these matrices for updating the cell state S
on top of the mentioned matrices, you need to count the biases (not in the picture)

Hence total # parameters = $4(nm+n^{2} + n)$


Is it normal for some classifiers to change their prediction performance from fixed amount of data each time I run them to train? I have a fixed amount of dataset, say text data with 100,000 records, to rank classifiers, such as gradient boosting, random forest, and other ones, according to their classification performance. The dataset is divided randomly for training and validation but the issue is that I get different prediction performance even if I use the same dataset overtime.

My understanding is that once the classifiers are ranked after the first training and validation run on the dataset, the rank should always show the same way as the previous rank. I don't add or delete the fixed amount of dataset that I use. Even though the same dataset is divided randomly, what I think is there shouldn't be a significant impact on the performance because the ratio of data used between training and validation is 80(training):20(validation). If they change their behaviors each time I run on the same unchanged fixed amount of data, how can I tell which one shows better performance on the current dataset? I understand the training data changes but its percentage is 80% of the dataset.

How are you dividing? You say randomly, but is this being redone each time your analysis is run? If it is, then this is perfectly normal, expected behavior. You're getting a different prediction performance because you have a different random training set each time.

You could also be using a stochastic method, that doesn't give the same result each time, say regression fit using gradient descent versus least squares. Since you didn't give any more details, I can't be more specific than that.


How do I assess whether two time series change together? In this example, at timepoint 5, both signals move up together. I would like to quantify these similar movements, and ideally disregard the parts where the signals are almost constant.
What correlation or similarity measures would be best here?

One option is a Granger causality test which a statistical hypothesis test for determining whether one time series is useful in forecasting another. One time series is said to Granger-cause another time series if the lagged values of one times series provide statistically significant information about future values of another time series.


Sequence pattern mining on continuous dataset I have a dataset on broadband usage and bills for a set of customers for an year. For every metric, like Upload usage, I have 12 values corresponding to each month.
Below are the columns and the format.

  CustomerId,Upload_Jan,Upload_Feb...Upload_Dec,
  Download_Jan,Download_Feb....Download_Dec,
  bill_Jan....bill_Dec

The values in all columns are continuous.
I want to see if there are some sequences appearing in the data frequently.

Can we do sequential mining on continuous data set?
Is it mandatory to bin each column to separate bins? How to decide the number of bins?
What is the best way of pre processing the data set for tools like SPMF which requires transactionized data?

I looked at TraMineR package in R and wrote a few scripts, but I could not make out much from the sequences generated.

The established procedure is called symbolic aggregate approximation or SAX in which you first first do piecewise aggregate approximation (PAA), namely send non-overlapping mean (boxcar) windows of width $w$ over the data, and then transform the obtained values into symbols by equal-frequency-splitting the resulting data distribution into $k$ letters of the alphabet.

How to determine appropriate $w$ and $k$? They are a tradeoff between dimensionality reduction and information loss. One way to adapt $w$ is to look at window' data points variance; if the variance is too large, reduce the window size, and vice versa. Another information-theoretic is MDL as explained in this question.

Since you only have 12 values in a series, the windowing step may not be necessary.

Alternatively, you could try reshaping the data into three series:

$$Dd_1^{(c_1)}\dots{}d_{12}^{(c_1)}Dd_1^{(c_2)}\dots{}d_{12}^{(c_2)}D\dots$$

$$Uu_1^{(c_1)}\dots{}u_{12}^{(c_1)}Uu_1^{(c_2)}\dots{}u_{12}^{(c_2)}U\dots$$

$$Bb_1^{(c_1)}\dots{}b_{12}^{(c_1)}Bb_1^{(c_2)}\dots{}b_{12}^{(c_2)}B\dots$$

where D, U, and B are string literals (used as start markers) and $d_m^{(c_i)}$, $u_m^{(c_i)}$, and $b_m^{(c_i)}$ are the values of download, upload and billed, for customer $c_i$ in month $m$. Then you could use one of the frequent subsequence mining algorithms on those three sequences (separately) to find any prevaling patterns.


How to decided which test of normality to use Given a data set with features, that you want to check for normality, one feature at a time w/o a multivariate normal test, how do you decided which test of normality to use? For example, using the python module scipy I could either use:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html

or I could use:

http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.mstats.normaltest.html

For example on the same feature I get these outputs:

scipy.stats.shapiro(data[:,0])
(0.9985173940658569, 0.77845299243927)

scipy.stats.mstats.normaltest(data[:,0])
NormaltestResult(statistic=1.492603328675163, pvalue=0.47411675723570479)

scipy.stats.anderson(data[:,0], dist='norm')
AndersonResult(statistic=0.17542490527580412, critical_values=array([ 0.573,  0.653,  0.783,  0.913,  1.086]), significance_level=array([ 15. ,  10. ,   5. ,   2.5,   1. ]))

Mathematical detail of the two different tests can be found in two papers:

[1]. http://sci2s.ugr.es/keel/pdf/algorithm/articulo/shapiro1965.pdf

[2]. http://www.jstor.org/stable/2334522?seq=1#page_scan_tab_contents

To summarize, both methods are based on hypothesis testing. But they are using different test statistics.

Shapiro-Wilk test has many good analytical properties and was designed to apply to data with sample size less than $50$. However, when the sample size becomes greater, Shapiro-Walk test might be unreliable. As I quote from the second paper:

  Shapiro and Wilk did not extend their test beyond samples of size 50. A number of reasons indicate that it is best not to make such an extension.

Essentially, that is why people invented the second normality test. As I quote from the second paper:

  We present a new test of normality applicable for samples of size 50 or larger which possesses the desirable omnibus property


What math topics are most needed to become data scientist? I am trying to learn data science from very different sources like Coursera, edx and many other sources. But I can not able to find any proper syllabus that what actually topics are needed. So, if anyone can please list out topics that I can follow and able to complete data science math skills.

Even if you post the name of books which can able to complete these criteria, then also it will be a great help. Thanks in advance.

Linear Algebra, Calculus, Probability and Statistics are fundamental topics.

Lots of great books out there, Khan Academy is a good resource as well:

https://www.khanacademy.org/math/linear-algebra

https://www.khanacademy.org/math/calculus-1

https://www.khanacademy.org/math/statistics-probability


Is there a way (like planning algorithms) to draw a successive polyline to fill a specific shape (like triangle)? there is a specific shape (like triangle) area, i am trying to develop an program to draw a successive polyline inside the triangle to fill the triangle, one line 3 layers

this post demonstrates some algorithms to solve this category of problems.

none of them is suitable for this task, any ideas?

Would not it be easier to simply apply geometric computation to your triangle to get smaller triangles whose vertices can be used for the polyline.?

With Wolfram Language

You have a triangle.

shp = Triangle[];
Graphics[{LightGray, shp}]

A ScalingTransform about the RegionCentroid can be performed with TransformedRegion to get smaller inner triangles. Below, shp is scaled by factors 0.8, 0.6, and 0.4 for the inner triangles.

scaledShps =
  TransformedRegion[shp,
     ScalingTransform[{#, #},
      RegionCentroid@shp]] & /@ Range[.8, .4, -.2];
SeedRandom[456]
Graphics[{LightGray, shp, Riffle[Hue /@ RandomReal[1, Length@scaledShps], scaledShps]}]

Each of scaledShps has the vertices as the first argument. Map (/@) First to collect these and Flatten them into a single Line.

Graphics[{
  LightGray, shp,
  Orange, Thick, Line@Flatten[First /@ scaledShps, 1]
  }]

Get more layers by increasing the number of scaling factors. For example, from .95 to .05 in steps of -.05

scaledShps =
  TransformedRegion[shp,
     ScalingTransform[{#, #},
      RegionCentroid@shp]] & /@ Range[.95, .05, -.05];
Graphics[{
  LightGray, shp,
  Orange, Thick, Line@Flatten[First /@ scaledShps, 1]
  }]

Wolfram recently released a free Wolfram Engine that can be called from many languages (Python, C, ...) so you can use the above code directly in your project.

Hope this helps.


Neural Network beginner level tutorial I am trying to build a simple multi layer perceptron Neural Network in Java, but apparently my calculations are off. I am looking for a beginner-level tutorial which can help me to understand how to properly calculate forward and backward pass, preferably with examples.

One of articles which helped me a lot is : A Step by Step Backpropagation Example by Matt Mazur. It covers forward and backward pass of MLP. I hope that helps.

Another great source is http://www.deeplearningbook.org/


Slightly different results between scipy.stats.spearmanr and manual calculation I have the following dataset.
When I calculate the Spearman correlation coefficient with scipy.stats.spearmanr, it returns 0.718182.

import pandas as pd
import numpy as np
from scipy.stats import spearmanr

df = pd.DataFrame(
    [
        [7,3],
        [6,5],
        [5,4],
        [3,2],
        [6,4],
        [8,9],
        [9,7]
    ],
    columns=['Set of A','Set of B'])

correlation, pval = spearmanr(df)
print(f'correlation={correlation:.6f}, p-value={pval:.6f}')

It returns this:

correlation=0.718182, p-value=0.069096

However, when I tried to calculate it manually:

df_rank = pd.DataFrame(
    [
        [5,2],
        [3.5,4],
        [2,4],
        [1,1],
        [3.5,4],
        [6,7],
        [7,6]
    ],
    columns=['Rank of A','Rank of B'])
cov_rank=np.cov(df_rank.iloc[:,0],df_rank.iloc[:,1])[0][1]

cov_rank/(df_rank.std()[0]*df_rank.std()[1])

It returns a different value.

0.7105597124064275

After the two decimal points are different and I do not know why.

The question is if scipy.stats.spearmanr expect the data to be ranked or not.

I think you have a small error in your manual calculation. You assign rank 4 to 4, 4, and 5. The first two should have rank 3.5 and the last should be rank 5. Your calculation then gives the same answer, 0.7181818181818181


How should I retrain my neural network with new images? I trained a convolutional neural network with a large (>90k) dataset of images. Since there is some confusion between two classes, I collected all the images that were misclassified and I augmented them with several methods. Furthermore, I collected new images that were not present in the original dataset. Now I would like to re-train my network in order to reduce the confusion. What is the best approach? Should I perform fine-tuning using only the new images (case A) or should I add the new images to the whole training set and re-train the network from zero (case B)?

My concerns are the following: in case A, most of the images belong to the same class, therefore the dataset would be very unbalanced. In case B, I am afraid that the impact of the new images on the whole dataset would be very small.

I guess you can try both the solutions.

First try fine-tuning. Then, if the results aren't improving, retrain with all the data.


sklearn select N best using classifier pretty simple question here but just can't seem to find the answer in the normally great documentation for sklearn.

I am working with binary classifiers, but we can just assume i am using LogisticRegression, and I was wondering if there is a general way to have the classifier select, say only 10 best (most sure) data points?

For example, say I train a set with 500K data points, and my test set has 10K lines, and out of the 10K, I just want to choose 10 that have the highest chance of being true positives.  Does this make sense?

I have read about, and have been playing with the class_weights attribute, which works well for giving more/less weight to each of the binary outcome classes, but its not quite working for what I want in that it always give different number of position predictions, and I can't really tell how sure the classifier is about each one of those.

class_weight is used in the process of model-training to train(=fit) a better model (let us call it clf).

Your question is is about choosing the most sure predictions.
You just need to predict the probability (for binary classification this will be the probability of positive class)

y_test_predicted_probability = clf.predict_proba(X_test)

and then choose 10 points with the highest y_test_predicted_probability

#some code to do this
top_picks_indexes = y_test_predicted_probability[:,1].argsort()[-10:] # chose top 10 probabilities for class = 1

# create a vector, Y_top_picks, with all zeros except ones for the selected top probabilities
Y_top_picks = np.zeros(len(X_test))
Y_top_picks[top_picks_indexes] = 1


Using vgg16 or inception with wights equals to None When using pre-trained models like vgg16 or inception,
it seems that one of the benfits of using pre-trained model, is to save up time of training.
Is there a reason to use the pre-trained models without loading the weights ? (use random weights) ?

The advantage of using a pre-trained model without loading the weights (which would mean you are only use the model, not a pre-trained version) is that you can easily use an existing model architecture and applying it to your problem. This can save you quite some time since you don't have to build the model architecture yourself in tensorflow/keras/pytorch and can go straight to applying the model on your data.


Your function

  0,otherwise 1

is not a linear combination of the inputs.

From the definition of single layer perceptronL

  A single layer perceptron (SLP) is a feed-forward network based on a threshold transfer function

Your function has two thresholds:  one for < 1 another for >1


How does class_weight work in Decision Tree The scikit-learn implementation of DecisionTreeClassifier has a parameter as class_weight.
As per documentation:

  Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one.

and

  The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

My understanding is that it should be used in case of imbalanced classes.

Question: How does the DT (classification) algorithm use this parameter while determining the ideal split for a given node? Does it consider some kind of "weighted" majority class instead of simple majority class in a region in the prediction space?

When deciding on a split at a node, the algorithm basically calculates some metric (entropy or gini impurity) for the given node and for the two resulting left and right nodes after the split. Comparing them tells you how much the split would improve the result.

The statistics for the child nodes are weighted by the number of samples in the left and right node, respectively.

When you use sample_weight this adjusts the count and replaces it with the sum of the sample weights. class_weight gives equal sample weights for each sample based on its class according to its class proportion.

For example, the improvement in impurity is calculated as

$$ \frac{N_{parent}}{N_{total}} * (impurity_{parent} - \frac{N_{right}}{N_{parent}} * impurity_{right-child}
                                - \frac{N_{left}}{N_{parent}} * impurity_{left-child}) $$

Without class_weight or sample_weights, the $N$s are just counts. With class_weight you replace them with the corresponding weights.

The idea is the same for entropy, even though calculated differently.

source code


What is the classification accuracy of a random classifier? I have a build a classification model using machine learning technique (SVM). I want to compare the classification accuracy of my model with a random classifier. My data set contains only two classes(1 or 0). The ratio of 1 and 0 instances are 35% and 65%. That means, 35% instances belong to 1 and 65% belong to 0 class. In that case, what will be the classification accuracy of random classifier (Random Guess)?

The equation of the classification accuracy for a random classifier (Random guess) is as follows:

Accuracy = 1/k (here k is the number of classes).
In your case, the value of k is 2.

So, the classification accuracy of the random classifier in your case is 1/2 = 50%


Clustering using both text and numerical features I have a dataset that contains 2 types of features, one is generated from doc2vec and one is numerical feature. I would like to perform clustering analysis on them. However, due to the size of doc2vec features, if I simply combine them into one array, clustering algorithm would distribute the "weight" on the doc2vec features more, how do I overcome this problem?
For example, for a given label, say I have features from doc2vec that look like this [1,2,3,4,5], and numerical feature [2]. I don't want to simply combine them into [1,2,3,4,5,2] and perform clustering analysis. Ideally, I would like my clustering algorithm to give the numerical feature equal importance as the doc2vec feature.

One way to achieve this is to use a clustering method based on a custom similarity/distance measure. For example you could defined the similarity measure between two instances as:
$$sim(\langle v_1, n_1\rangle,\langle v_2, n_2\rangle)=\frac{1}{2} cosine(v_1,v_2)\ +\ \frac{1}{2} \left(1-\frac{|n_1-n_2|}{max(n_1,n_2)}\right)$$
This measure gives the same weight to the similarity between the vectors ($v_1$ and $v_2$) and the similarity between the numerical values ($n_1$ and $n_2$). Note that since this similarity measure is normalized, you can convert it to a normalized distance measure: $d=1-s$.
Of course you should define the exact measure based on what the values represent, this is just an example.
You could use this measure with a hierarchical clustering method or a graph clustering method (with edges based on similarity value).


Image normalisation methods I have found some research papers specifying explicitly the normalisation  technique they used to get the results.

What difference do

IMG /255.0

And

IMG - mean / STD

Has on the performance of the CNNs?

So, before just diving into the performance of the CNNs based on the two methods, just lets start with what they both will do to the input.

The first method (IMG/255.0, and we'll call it scaling) will have the data scaled between 0 and 1. The second method (IMG - mean/STD, we'll call it centering) will have inputs ranging from arbitrary numbers (can be both positive and negative) with 0 being the mean.

Scaling will improve the convergence speed of the model, whereas centering will not only speed up the convergence of the model, but will also deal with the gradient exploding and vanishing problem.

As mentioned in CS231, centering is enough, since it does the scaling thing as well.

Lastly, as long as you don't have a very deep model, the two techniques won't make a lot of difference because batch-normalisation takes care of pretty much everything regarding image normalization, where all the layers receive the normalized data, and not only just the first one.


How to retrain Glove Vectors on top of my own data? I am using GloVe and gensim for my project. I have a corpus of data (let's say mydata.txt) which has new words which are not in the existing Glove. So, how do I retrain glove so that the existing pre-trained glove must now include the new words on my corpus mydata.txt? I have been struggling and failed to find the solution for 2 weeks. The only similar post I found is this Improving existing GloVe Model

What you should do is:

Create a new instance of a GloVe model with the old_words and new_words as vocabulary.
Replace the initial vectors/biases of the old_words with the ones you have already.
Train this model on mydata.txt.

The new old_words representations won't be the same but will be highly influenced by the old ones.


How do you compare term counts between two different periods, with different underlying corpus sizes, without bias? I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's imagine it's the term "Christmas"). Unfortunately, there are also far less news articles in the third quarter than in the second (due to the lack of news in the summer). So how do we accurately compare the counts in each quarter? We assume that there will be a greater number of occurrences in the fourth quarter, but how much does the magnitude of this difference depend on the change in size of the underlying text?

Heap's law shows the relationship between text size and number of unique terms. It's non-linearity implies that the rate of new, unique words introduced by the text decreases as you increase the size of the text, and the proportion of the text taken up by each existing word therefore increases. This applies given documents taken from the same 'distribution' of text, in other words the underlying zipfian distribution of word ranks is identical (see wiki).

In my example above this is obviously not the case, since the underlying topics, and resultant term distribution, will change between summer and winter, especially with regards to the term "Christmas". But take the same term count but over the whole of 2013 and 2014; you would reasonably expect the general underlying term distribution to be the same in each period, so Heap's law applies, but what if the volume of text has changed? Simply normalising by the size of the text, or the number of documents, does not, as far as I can tell, account for the relative change in expected value of the term count.

I have a hunch that this might be a simple application of Heap's or Zipf's laws, but I can't see how to apply them to this particular question. Appreciate any insight.

The approach suggested by j.a.gartner is good if you want to analyze the quarter-to-quarter change in frequency for a given term, considering only that term in isolation.  What it doesn't do is evaluate the relative frequency compared to all other terms in the corpus.  For this you could compare frequency rank for all terms quarter-to-quarter. This is the analysis used in Zipf's Law.  Another advantage of this analysis is that you can test whether the generating process is or is not stationary (e.g. governed by same distribution quarter to quarter).

The advantage of comparing frequency rank is that it doesn't depend on the relative size of the corpus for each quarter (as long as they are all "large").  In simple language, the frequency rank of the word "Christmas" in the 4th quarter will probably be the same, regardless of whether the (US English news) corpus is 10,000 articles or 1,000,000 articles.


Identifying templates with parameters in text fragments I have a data set with text fragments having a fixed structure that can contain parameters.
Examples are:

 Temperature today is 20 centigrades
 Temperature today is 28 centigrades

or

 Her eyes are blue and hair black.
 Her eyes are green and hair brown.

The first example show a template with one numerical parameter. The second one is a template with two factor parameters.

The number of templates and the number of parameter is not know.

The problem is to identify the templates and assign each text fragment to the corresponding template.

The obvious first idea is to use clustering. The distance measure is defined as a number of non matching words. I.e. the records in example one have distance 1, in example two distance is 2. The distance between the record in example one and two is 7.
This approach works fine, providing the number of clusters is know, which is not the case, so it is not useful.

I can imagine a programmatic approach scanning the distance matrix searching for records with lot of neighbors in distance 1 (or 2,3,..), but I'm curious if I can apply some unsupervised machine learning algorithm to solve the problem. R is preferred, but not required.

You might consider using word2vec to identify phrases in the corpus.  The presence of a phrase (instead of single tokens) is likely to indicate a 'template.'

From here, the tokens most similar to your template phrase are likely to be the values for your parameters.


Grouping probabilities of full model vs fitting model on subset of variables Suppose I want to predict some event probability with a set of features. Some of the features may be gender or some other categorical variable. Assume the probabilities are well calibrated.

Now, say, I need to report the average predicted probability for males and females (and maybe some other subset of features).

Is there a difference between

averaging the probabilities of the full model or
simply fitting a model on only the subset of variables?

And if so why? The full model would outperform the simpler model by a large margin. An example would also be appreciated.

A very thought provoking question.

Surprisingly, second approach (subset) is better in theory. The first one (full) is an unbiased estimator of second one (subset). That is, expectation of average of probabilities in full model is equal to the probability in subset model.

I said "in theory", because we assume that full and subset models estimate the true P(event | full) and P(event | subset) exactly respectively (zero error, perfect generalization, etc.). In practice, choice of learning algorithm and training set affects how accurate they can fulfill this assumption. A model could be Naive Bayes, Logistic Regression, Neural Network with a SoftMax layer, etc.

Proof

Let

$F_f$ denote the full features,
$F_s$ denote the subset features (that we group on),
$F_d=F_f-F_s$ denote the features that we average out,
$\boldsymbol{A}=\{x|F_s(x)=a\}$ denote the set of all possible instances with subset features equal to $a$, and
$A \subset \boldsymbol{A}$ denote the observed subset of $\boldsymbol{A}$ in training set.

Here is the average of probabilities in full model (first approach):
$$\begin{align*}
P_f(e|F_s=a)&=\frac{1}{N_A}\sum_{x\in A}P(e|F_d(x),a)
\end{align*}$$

Expectation of above average is:
$$\begin{align*}
E[P_f(e|F_s=a)]&=\frac{1}{N_A}\sum_{x\in A}E[P(e|F_d(x),a)]\\
&=\frac{1}{N_A} N_A E[P(e|F_f(X))|X \in \boldsymbol{A}]\\
&=\sum_{b}P(e|F_d=b,F_s=a)P(F_d=b|F_s=a)\\
&=\sum_{b}P(e,F_d=b|F_s=a)\\
&=P(e|F_s=a)
\end{align*}$$

where $P(e|F_s=a)$ is the output of second approach.

Example

To verify, here is an example:

$e=\text{has_job}$

$F_f=\{\text{sex}, \text{has_degree}\}$

$F_s=\{\text{sex}\}$

$F_d=\{\text{has_degree}\}$

Full model sees:

x     sex  has_degree  has_job
1     f    1           0
2     f    1           1
3     f    0           0

4     m    0           0
5     m    0           0
6     m    1           1

And builds

sex  has_degree  P(has_job=1|sex, has_degree)
f    0           0
f    1           0.5
m    0           0
m    1           1.0

And answers $P(\text{has_job}=1|\text{sex}=f)$ with

$$\frac{1}{3}\sum_{x \in \{1, 2, 3\}}P(\text{has_job}=1|x)=(0.5+0.5+0)/3=0.33,$$

and answers $P(\text{has_job}=1|\text{sex}=m)$ with

$$\frac{1}{3}\sum_{x \in \{4, 5, 6\}}P(\text{has_job}=1|x)=(0+0+1.0)/3=0.33$$

On the other hand, subset model sees:

x     sex  has_job
1     f    0
2     f    1
3     f    0

4     m    0
5     m    0
6     m    1

And builds

sex  P(has_job=1|sex)
f    0.33
m    0.33

And answers $P(\text{has_job}=1|\text{sex}=f)$ with 0.33, and $P(\text{has_job}=1|\text{sex}=m)$ with 0.33.


Predictive modeling on time series: How far should I look back? I am building a model for classification on a dataset which is collected by recording a system's behaviour through 2 years period of time. The model is going to be used in the same system in real time.

Right now I am using the whole dataset (2 years) to build my classifier but I doubt it might not be the right approach. Since I am trying to model the behaviour of a system in real time, it is possible that older data points in the dataset have become irrelevant or uninformative compared to system's current environment (distribution of the system's input changing drastically through time for example).

My question is how can I determine which parts of the dataset to use for training, taking the last 1.5 years instead of all 2 years for example. Is there a statistical way to help me decide that a specific time period is not helping or possibly hurting the model to correctly classify more recent data points?

Hold out the most recent block of data (maybe something like a month) to use as your validation data.

Try multiple models and different ways of training those models. For example, training only on the most recent couple of months vs the whole time period.

Take a small number of your top performers and test their performance on the validation set and see what does better.

There’s nothing special about this answer. Anytime you want to know is X better than Y, testing performance on a held out validation set is almost always the way to go.


Why is my LSTM is working best with batch size of 2 and no hidden layers? I am building an LSTM for price prediction using Keras. I am using Bayesian optimization to find the right hyperparameters. With every test I make, Bayesian optimization is always finding that the best batch_size is 2 from a possible range of [2, 4, 8, 32, 64], and always better results with no hidden layers. I have 5 features and ~1280 samples for the test I am trying.

Why is this the case? Is the lack of hidden layers due to the fact that I do not have many inputs and samples?

Also, literature always seems to suggest that a batch size of 2 is very low and a middle ground for the optimal batch_size is 32. How can I interpret this result?

There are some papers which tell us that lower batch size may generalize better than large batch size. and large batch size may cause regularization in the model too. maybe that is the reason Bayesian optimization is suggesting a lower batch size for your dataset.
Please check below papers,

https://openreview.net/pdf?id=B1eyO1BFPr

https://openreview.net/pdf?id=B1Yy1BxCZ


Standardise AND Normalise I am new to machine learning.
Does it make sense that my model works when I do both Standardise and Normalise? What does this say about my data?

Or do I do have to select one or the other? My goal is to build a binary classifier which analyses CNNs and GIST features?

No, actually it doesn’t make sense to apply two scaling methods to your data simultaneously. Generally, I would suggest using standardization because it is more robust to outlier samples compared to normalization. Having removed outlier samples from your dataset, you can safely use normalization instead.


How can I build a self-attention model with tf.keras.layers.Attention? I have completed an easy many-to-one LSTM model as following.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dropout
model=Sequential()
model.add(LSTM(2**LSTM_units,input_length=data.shape[1],input_dim=data.shape[2],return_sequences=True))
model.add(Dropout(dropout))
model.add(Dense(1))
model.fit(data,res)
prediction=model.predict(test)

Where data.shape is (dates, time_interval, factors) and res.shape is (dates,1).
I would like to replace LSTM with Attention.
How can I get the similar result by using tf.keras.layers.Attention?

How about this? https://pypi.org/project/keras-self-attention/
I use it but I have not tried it without the LSTM layer yet. Also, you have to use keras directly, not tensorflow.keras. Here is a working model I am running right now.
    model = keras.models.Sequential()
    model.add(keras.layers.LSTM(cfg.LSTM, input_shape=(cfg.TIMESTEPS,
              cfg.FEATURES),
              return_sequences=True))
    model.add(SeqSelfAttention(attention_activation='sigmoid'))
    model.add(keras.layers.Dense(cfg.DENSE))
    model.add(keras.layers.Dense(OUTPUT, activation='sigmoid'))

Also, look here, there is quite a bit https://awesomeopensource.com/projects/attention-mechanism


How to deal with classification problem where labels are non uniformly distributed? I have a data set which has around 1000 samples and are divided in 4 groups - A, B ,C , D. The problem I am facing is that there are very high number of data sample which have B and C s output. They outweigh other two by factor of 3:1. Due to this most of the classification problems are producing very inefficient results with results from other two classes being mapped to A and B quite often . Is there any way to deal with it?

Welcome to DataScience.SE! You need to ensure that your training distribution is similar to your test distribution in order to get the best results. This can be done through stratified sampling. It might be that there is no problem, and that the classifier is simply allocating its resources to best classify the majority case, though I'm a bit confused by your description because you say that B and C dominate and the "other two classes" (A and D) are "mapped to A and B quite often". One option is to use more discriminating features, if you can think of any. Another is to partition the large classes into (B becomes the union of $B_i$ and likewise for C) if applicable. This can help if the shape of the class is complex by breaking it up into learnable pieces. It helps to plot your inputs, colored by class, to understand which classes overlap or have complex shapes. Finally, you can modify your loss function if getting certain classes wrong is more important than others.


Is the prediction algorithm absolutely the same for all linear classifiers? Is the prediction algorithm absolutely the same for all linear classifiers and linear regression algorithms?

As known, any linear classifier can be described as: y = w1*x1 + w2*x2 + ... + c

There are two broad classes of methods for determining the parameters of a linear classifier (generative and discriminative): https://en.wikipedia.org/wiki/Linear_classifier

Linear Discriminant Analysis (or Fisher's linear discriminant), Naive Bayes classifier
Logistic regression, Perceptron, Support vector machine

Question: Is it true that linear classifiers differ only in the Learning algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2 + ... + c?

If I used one method for Training (for example SVM with linear kernel function), then can I use other method for prediction (for example Perceptron) with the same output result?
And can I do the same, if I used SVR (Support Vector Regression with linear kernel function) for Training, then can I use Perceptor as linear regression method for Prediction value?

Is it true that linear classifiers differ only in the Learning
  algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2
  + ... + c?

Yes, all parametric linear classifiers try to predict the weights for the same equation $y = w1*x1 + w2*x2 + ... + c$ and differ in how they try to optimize the cost function.

  If I used one method for Training (for example SVM with linear kernel
  function), then can I use other method for prediction (for example
  Perceptron) with the same output result?

You will not be able to achieve this using standard libraries. But, if you code your models from scratch and extract weights from one model after training and use those weights to make a prediction with a different model, you'll get a sufficiently close answer. This is due to the fact that once you have learned the weights $(w1, w2,...., wn)$, you know the nature of data and can predict the target variable using the equation $y = w1*x1 + w2*x2 + ... + c$. Note that, this will be true only in the case of parametric linear models as only then the above equation would be applicable.

  And can I do the same, if I used SVR (Support Vector Regression with
  linear kernel function) for Training, then can I use Perceptor as
  linear regression method for Prediction value?
  Well, yes again with the same restrictions as above. This is because you will get a decision boundary as the output which depends on the data. Note that, there will be minor changes in the output as the error of different models is different.

Also, any other differences in the output will be due to the difference in the learning algorithm and how it operates on the data. You can verify this by training two models on the same data and extracting the weights after training. If the weights are close enough (which means you have learned the correct underlying pattern), you can use any of the two models on test data to predict and you will get similar values.


NLP SBert (Bert) for answer comparison STS I've been researching a good way to automate short answer evaluation. Essentially a teacher gives a test with some questions like:
Question: why did columbus sail westward to find asia?
Answer: so he could find a new trade route to Asia through the ocean. Three goals of the Spanish in the Americas were the desire to attain great amounts of riches, to establish claims on as much land as possible, and to colonize as much as possible.
With that we have the correct answer and would like to compare that with the students answer and produce a score based on similarity. I know this isn't a reliable replacement for human grading, but for the sake of the example.
I've come across this paper and codebase:
https://arxiv.org/pdf/1908.10084.pdf
https://github.com/UKPLab/sentence-transformers
It seems like the ideal method for solving this problem, but most examples are based on scoring/ranking of semantic search. I question whether I'm on the right path, given that I'm just comparing two answers and not a cluster. Anyone with more experience, possibly can provide some guidance?

I have used Siamese Bert and I can say it does a pretty good job. However, the issue is that the data that it has been fine-tuned atop of Bert may not necessarily, entirely represent the same semantic distance as with the answers between the true and the student's one. For instance, if there is a question about engineering, where a small change of word may mean a totally different thing; SBert would still find them quite similar cause they are related to the topic.  Unless it's fine-tuned.
Moreover, you will not be able to interpret the similarity. Should a student ask you why my peer's answer is better you won't be able to explain.
My opinion: I believe you could use this tool as a way to reduce totally incoherent answers, but at some point, human evaluation will be needed. And maybe use interpretable metrics such as ROUGE or BLEU. I aware as well, that this topic is quite trendy in NLP, I won't be surprised if there is or will be good off the shelf tool for that, but I am not aware of one currently.


Agglomerative Clustering without knowing number of clusters I want to perform agglomerative clustering, but I have no idea of number of clusters before hand. But I want that every cluster has at least 40 data points in it. How can I apply this to sklearn.agglomerative clustering? Should I use dendrogram and cut it somehow? I have no idea how to relate dendrogram to this and cutting it out. Any help will be appreciated!

A minimum cluster size will not generally be satisfiable in hierarchical clustering. Instead, you have to expect many clusters with just a single point.

ELKI has some fairly interesting techniques to cut a dendrogram. Check the clustering.hierarchical.extraction (or so) package. If I remember correctly, some allow you to set a minimum size (but there will be a "noise" cluster with all the leftovers).


Linear regression assumptions I have read that we make the following assumption for linear regression:
1. Linearity (correct functional form)
2. Constant error variance (homoskedasticity)
3. Independent error terms (no autocorrelation)
4. Normality of error terms
5. No multicollinearity
6. Exogeneity (no omitted variable bias)

So are these assumptions specific to Linear Regression or applicable for all types of regression techniques like Support Vector Regression, Lasso and Ridge regression, Stepwise regression etc.

These are not (direct) assumptions for linear regression.
But rather for OLS-Ordinary Least Squares which is widely used to estimate the parameter of a linear regression model. OLS estimators minimize the sum of the squared errors (a difference between observed values and predicted values).

In order to guarantee finding best possible parameters, we make these assumptions (as with every optimisation)

But just because they are not present that does not mean that linear regression wont work.

Regarding your other models: If they use the same optimisation, than they have same assumptions.


When are p-values deceptive? What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance?  Are there specific problem types that fall into this category?

You are asking about Data Dredging, which is what happens when testing a very large number of hypotheses against a data set, or testing hypotheses against a data set that were suggested by the same data.

In particular, check out Multiple hypothesis hazard, and Testing hypotheses suggested by the data.

The solution is to use some kind of correction for False discovery rate or Familywise error rate, such as Scheffé's method or the (very old-school) Bonferroni correction.

In a somewhat less rigorous way, it may help to filter your discoveries by the confidence interval for the odds ratio (OR) for each statistical result.  If the 99% confidence interval for the odds ratio is 10-12, then the OR is <= 1 with some extremely small probability, especially if the sample size is also large.  If you find something like this, it is probably a strong effect even if it came out of a test of millions of hypotheses.


Is NN with no hidden layer is behave like a regression? Is a NN with no hidden layer is behave like a regression? What we could say that NN without hidden layer can say us?

If we have for instance 20 input and 4 output and I have no true label, is it similar to regression?

If it is a regression then it will be a supervised learning however there is no true label!

What is wrong inside this clarification?

more clarification:
Also, I have a dqn ( a deep learning structure inside q-learning approach)! Unfortunately I do not have the optimal answer of it! I am looking for a way to compare my results, Maybe I can compare by putting a un/supervised learning approach instead of DNN! For instance, you can put a DNN without hidden layer and as it behaves like the linear regression then you can put every supervised learning instead of it! As there is no True label I have problem with training section and estimate the DNN section with supervised learning! Also,as  it is MDP, I can not estimate the DNN section with unsupervised learning!

You would then call it perception model. As in 1945 or so some researcher did this with no hidden layer. They came up with AND,OR,XOR problem where you can solve AND,OR with a single perception model but that model was unable to solve the XOR problem.

There are two kinds of regression: Linear and non linear. The activation functions in neural net makes them non linear regressor.

A Neural Net would be a linear regressor if you don't put any activation function.

Then it would be like : $Wx+b$ (which is $y=mx+b$ equivalent).
But when you add activation function let it be sigmoid then it achieves a non linear space.Like $$\frac{1}{1+e^{-(wx+b)}}$$

"If we have for instance 20 input and 4 output and I have no true label, is it similar to regression?" --- In this you are just deleting back propagation thing. what as usual regression algorithms like linear,polynomial does is they just predict the function but never learns something like neural net learns proper function via optimizing weights. If you don't have true level that means you are not calculating the loss value between [true-predicted] so it would be a simple regression.


What can I use to post process an NLP tree generated from the python library `spaCy`? Using spaCy as the NLP engine for a chatbot, I call nlp("Where are the apples?").print_tree() and receive:

[{'word': 'are',
  'lemma': 'be',
  'NE': '',
  'POS_fine': 'VBP',
  'POS_coarse': 'VERB',
  'arc': 'ROOT',
  'modifiers': [{'word': 'Where',
    'lemma': 'where',
    'NE': '',
    'POS_fine': 'WRB',
    'POS_coarse': 'ADV',
    'arc': 'advmod',
    'modifiers': []},
   {'word': 'apples',
    'lemma': 'apple',
    'NE': '',
    'POS_fine': 'NNS',
    'POS_coarse': 'NOUN',
    'arc': 'nsubj',
    'modifiers': [{'word': 'the',
      'lemma': 'the',
      'NE': '',
      'POS_fine': 'DT',
      'POS_coarse': 'DET',
      'arc': 'det',
      'modifiers': []}]},
   {'word': '?',
    'lemma': '?',
    'NE': '',
    'POS_fine': '.',
    'POS_coarse': 'PUNCT',
    'arc': 'punct',
    'modifiers': []}]}]

I can easily enough parse out (arc, lemma) pairs for where (advmod, where) and (apple, nsubj), and call a function where(apple).

However, this is a really naive way of handling the parsed tree.  Any suggestions for how to handle processing this tree?  I don't think something like a multilevel SVM would work.  Maybe a NN of some kind?

What do you want to do with the chat bot? How you parse it will depend on the final use case and, believe it or not, many people get the job done by simply collecting the POS they want and using some filtering.

If you want to try to maintain more of the data and perhaps abstract it makes sense to try clustering of some kind, perhaps using hierarchical methods, such as the (relateively new) hdbscan. The features on which you cluster will again depend on what you want to achieve.

If you haven't already, check out the spaCy examples for some inspiration!

Once you have a corpus with word all tagged, you can try training models that might be able to answer questions, based on user input. This will involve steps such as creating encoding of the words (or entire user questions), using embeddings such as Word2Vec, GLoVe, or simple sparse one-hot encodings. You basically need to transform words into numerical input somehow.

I hope this gives you some keywords to help you on your search  :-)


Is Data Science the Same as Data Mining? I am sure data science as will be discussed in this forum has several synonyms or at least related fields where large data is analyzed.

My particular question is in regards to Data Mining.  I took a graduate class in Data Mining a few years back.  What are the differences between Data Science and Data Mining and in particular what more would I need to look at to become proficient in Data Mining?

@statsRus starts to lay the groundwork for your answer in another question What characterises the difference between data science and statistics?:

Data collection: web scraping and online surveys
Data manipulation: recoding messy data and extracting meaning from linguistic and social network data
Data scale: working with extremely large data sets
Data mining: finding patterns in large, complex data sets, with an emphasis on algorithmic techniques
Data communication: helping turn "machine-readable" data into "human-readable" information via visualization

Definition
data-mining can be seen as one item (or set of skills and applications) in the toolkit of the data scientist.  I like how he separates the definition of mining from collection in a sort of trade-specific jargon.
However, I think that data-mining would be synonymous with data-collection in a US-English colloquial definition.
As to where to go to become proficient?  I think that question is too broad as it is currently stated and would receive answers that are primarily opinion based.  Perhaps if you could refine your question, it might be easier to see what you are asking.


Libraries for (label propagation algorithms/frequent subgraph mining) for graphs in R General description of the problem

I have a graph where some vertices are labeled with a type with 3 or 4 possible values. For the other vertices, the type is unknown.
My goal is to use the graph to predict the type for vertices that are unlabeled.

Possible framework

I suspect this fits into the general framework of label propagation problems, based on my reading of the literature (e.g., see this paper and this paper)

Another method that is mentioned often is Frequent Subgraph Mining, which includes algorithms like SUBDUE,SLEUTH, and gSpan.

Found in R

The only label propagation implementation I managed to find in R is label.propagation.community() from the igraph library.
However, as the name suggests, it is mostly used to find communities, not for classifying unlabeled vertices.

There also seems to be several references to a subgraphMining library (here for example), but it looks like it is missing from CRAN.

Question

Do you know of a library or framework for the task described?

This is an old post, but there is a subgraph package and accompanying book/documentation for doing this in R:
https://www.csc.ncsu.edu/faculty/samatova/practical-graph-mining-with-R/PracticalGraphMiningWithR.html

Although I personally don't get the connection between subgraph mining and label propagation in this case. SVD++ might be closer to what you're looking for (supported by GraphX of Spark, which I think also supports label propagation).
http://spark.apache.org/graphx/


How to prepare data for Named Entity Recognition with BIO annotation? Assume the task here is extracting important facts for resume like a candidate skills and his education etc.., Here is resume is parsed text from pdf or docx resume.

First, I'll obtain skills and education data from various online websites, job portals etc.., the obtaining data will be two text files and in each file every row represent a skills or a university name like below,

skills.txt
___________
c
python
java
node js

education.txt
___________
massachusetts institute of Technology
harvard university

I want to know if these be enough to be able to train a named entity recognition model to recognize skills and education for raw resume text. The data I have is not sentences but just entities. I've read somewhere that we require some context along with the entity for NER model to learn better. Like this example below,

skills.txt
___________
c is used at facebook
python is my favorite programming language

If I use my collected data education.txt to train the modeel and BIO annotate them then it will be like below, it won't have O-Other token.

massachusetts B-EDU
institute I-EDU
of I-EDU
Technology I-EDU

harvard B-EDU
university I-EDU

indian B-EDU
institute I-EDU
of I-EDU
technology I-EDU

But I don't know how to access such data for my resume-extraction problem. How do I proceed further?  How to build an effective NER model for my resume facts identification domain-specific task? Any inputs/suggestions would really help.

If your data always looks like this, there is little reason to use sequence labeling: every token belongs to an entity, so it's just a matter of correctly separating the entities and classifying them. But since the entities are already separated by line breaks, there's no need to train a model to separate them. So in the end you just have to classify the entities by category, and this doesn't require sequence labeling. But even for that, from your example it looks like the skills vs. education entities are already separated, so in the end I'm not sure what you want the model to learn?


What is the difference between Missing at Random and Missing not at Random data? I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able to understand the difference between MAR and MNAR.

First I would like to give the description of the dataset and the patterns of the missing data. So it would be easier for you to explain the difference between the two and also it would enable to identify the what are those patterns in my dataset.

Description of the dataset: It is a transaction history dataset of a cycle company(Imaginary company) which contains descriptions about customers like their name, DOB, gender, geographic location, Income, social status and the transaction details like the cycle brand name bought, size of the cycle(small,M,L), type of cycle like (Mountain, Road and Standard), Product manufacturing cost, List Price of the product and finally the transaction date.

Pattern in the missing data:
There are basically four different patterns of missing data that I have identified in the dataset:

Pattern 1: This is related to product details in the transaction history: Whenever there is some missing data about say the brand name then the details regarding Product line, Product class, Product size, standard cost and Product_first_sold_date are also missing but the list price alone is available but then these list prices seem unique as they cannot be found repeating in the same column. This could blamed on the company for not recording the details properly but there is actually no such company that exists and it is upto me to deal with.

Pattern 2: Whenever there is missing data about a customer say first name then all the other columns about the customer are also missing like Gender, Past_3_years_bike_related_purchases,Job_Industry_Category,
Wealth_Segment,Deceased_Indicator and Owns_Car. Only the customer ID in this case is available. Probably this could be taken as the customer didn't want to disclose the details but then again it is upto me to deal with.

Pattern 3: Whenever there is some missing data about the geographic location of the customer like say the Address then the Post code, state,country and property evaluation column data are also missing but the general customer details like name, gender, DOB are all available. Probably in this case the customer didn't want to reveal their geographic details and I have got to deal with it.

Pattern 4: Most interesting one: The gender column has three categories M, F and U. U can be taken as not disclosed. Whenever the gender is U then their age and tenure are also missing. This could be taken as those not ready to disclose their Gender are not ready to disclose their age and tenure.

Sometimes some patterns occur together.

What category of missing date like(MAR, MCAR and MNAR) do these fall into? And how do I deal with it. Any suggestion would be extremely helpful. Thanks.

Definitions of missingness process are tricky.
Missing completely at random occurs when the missingness is really at random (MCAR; e.g. when conducting a survey there are error in the data entry process).

Missing at random (MAR) occurs when the missingness is not really at random, but when it could be considered at random conditioning on what is observed in the rest of the data (e.g. males are less likely to express their opinion in a survey but this is completely not related to their attitude as customers).
Missing completely at random occurs when the missingness is really at random (MCAR; e.g.
when conducting a survey there are error in the data entry process).
Missing not at random (MNAR) occurs when the missingness depends on the value of the variable (those who buy more tend to not answer survey questions).

I think that it is difficult to figure out in what category are your missing data patterns. In your case seems to me that MAR is a reasonable assumption (which is impossible to test statistically) and that multiple imputation with chained equations could be an option (mice or miceadds in R).

If you adopt a mice approach, a crucial part in the application is to carry out sensitivity analyses. what happens if you:
- impute only some of your missing data pattern?
- vary the number of imputed sets
- perform a complete case analysis.


Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow) In many ptb/mini shakespeare  LSTM generator tutorials on web, people make input data, (where every batch is subsequent. for example, sequences in batch_2 are subsequent to sequences in batch_1, and batch_1 is fed right before batch_2) and feed those input data to dynamic_rnn. tutorial: https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html

During training, these tutorials manually feed back the hidden state of LSTM, like codes below.

batch_size = 32
hidden_state_in = cell.zero_state(batch_size, tf.float32)
output, hidden_state_out = tf.nn.dynamic_rnn(cell, inputs,initial_state=hidden_state_in)
...
#For loop used in training: ...
    output, hidden_state = sess.run([output, hidden_state_out],
                            feed_dict={hidden_state_in:hidden_state})

What makes me confused, is that when generating or testing ptb/shakespreare, these tutorials feed data, with batchsize==1 and varying seqlength (at training, batchsize was 32 or above).

def generate_characters(g, checkpoint, num_chars, prompt='A', pick_top_chars=None):
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        g['saver'].restore(sess, checkpoint)
        state = None
        current_char = vocab_to_idx[prompt]
        chars = [current_char]

        for i in range(num_chars):
             if state is not None:
                feed_dict={g['x']: [[current_char]], g['init_state']: state}
                #shape [BATCHSIZE, SEQLEN] with BATCHSIZE=1 and SEQLEN=1
            else:
                feed_dict={g['x']: [[current_char]]}
            preds, state = sess.run([g['preds'],g['final_state']], feed_dict)

When using dynamic_rnn with LSTM, the hidden state has shape of  [num_layers, 2, batch_size, state_size]. I just wonder how can we use data with different batch_size at training and testing. If hidden state is fed manually, what kind of trained feature is contained in LSTM graph? Is it cell_state, or the coefficients which are used in building states?

Also, when looking at generating text part of r2rt's tutorial above, first parts of generation seems weird. I guess this is because hidden_state (which is fed to dynamic_rnn's initial_state) at the beginning of generation is yet poorly configured, cause it's right after the first prompt. Am I right about this reason?

There's a lot going on inside an LSTM, so it's easy to get confused. I think you are confusing the state and the weights:

The weights are trained at training time and are not updated during prediction. This is like the weights of any other neural network.
The state updates as the model moved "forward" through the text. It is essentially what enables the LSTM to keep track of where it currently is in a sequence.

The weights have a fixed size, of course, but when generating text the LSTM acts only on the last character and the current state, which is why the sequence length is set to 1 (and if you're only generating one text at a time, the batch length is also 1. you could increase the batch size to generate multiple texts at the same time).


Why do we use gradients instead of residuals in Gradient Boosting? I have found mentions of two advantages in using gradients instead of actual residuals:

1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base learners to make them compatible with the loss function.

2) It can be computationally infeasible (in the case of MAE, we would have to calculate the median in every split).

However, I don't understand these points, namely:

1) If we just calculate the residual and have the base learner fit on those values, how exactly would that be any more difficult than calculating the gradients and then fitting on those values?

2) Following on the above question, why exactly do we need to calculate the median in the MAE example instead of just getting the residuals?

Perhaps I don't understand the exact mechanism behind optimizing an individual loss function. It would be really great if someone could demonstrate exactly how MAE is done and in which step would using residuals have been infeasible.

Hmmm, I am little perplexed by your question.

In gradient boosting, we do use the residuals. The residuals are the gradients.

You can check my simple implementation of gradient boosting. This is where the magic happens:

def fit(self, X, y):
    self.fs = [self.first_estimator]
    # step 0
    f = self.first_estimator.fit(X, y)
    # step 1
    for m in range(self.M):
        f = self.predict(X)
        # step 2
        U = self.loss(y, f)
        # step 3
        g = clone(self.base_estimator).fit(X, U)
        # step 4
        self.fs.append(g)

You start by a dummy model $f$. Then you create a new model $g$ based on the errors of the existing ensemble $L(f(x),y)$. The code should be pretty straight-forward. Check the algorithm in the wikipedia page for a more formal presentation.

Why am I using residuals and gradients as interchangeable words?

You have discrete mathematics and continuous mathematics.

In neural networks, we use the definition of gradient from continuous mathematics. You have a loss function that is $C_1$ continuous (meaning that the loss must be differentiable at least once).

$$f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}$$

In gradient boosting, we use the definition of gradient from discrete mathematics. It's usually called fininite differences, instead of derivative or gradient.

$$f'(x) = f(x+1)-f(x)$$

Personally, I think gradient is a misnomer. But it's marketing. Gradient boosting sounds more mathematical and sophisticated than "differences boosting" or "residuals boosting".

By the way, the term boosting already existed when gradient boosting was invented. It was used for AdaBoost, which changes the weights of the observations based on the loss, rather than training in the residuals. AdaBoost only works for weak models (each model must commit errors), while gradient boosting can do resample and work for strong models.

Important point on gradient boosting

One important difference between gradient boosting (discrete optimization) and neural networks (continuous optimization) is that gradient boosting allows you to work with functions whose derivative is constant.

In gradient boosting, you can use "weird" functions like MAE or the Pinball function. In my code, you can see that you can choose between MSE, MAE and quantile (which is the pinball loss). These latter two do not work very well for neural networks.

The pinball loss is used for quantile predictions, like wind forecasts in energy systems. This is the reason why gradient boosting is much used in these systems.

Small point: many people think that the reason why MAE and Pinball functions work bad for neural networks is because they are not continuous. While it is true they are not continuous when $y=\hat y$, it's very easy to work-around that. This is not the reason why results are so poor with those losses. The reason why they don't work well is because the optimization steps use the gradient (in gradient descent), and the gradient is constant for the entire function $L'(y,\hat y)=1$ or $L'(y,\hat y)=-1$, so the magnitude of the gradient is always 1. The optimization will depend solely on the learning rate heuristic that you are using, which usually will not work out very well.


How to run a pyspark application in windows 8 command prompt I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] as an environment variable and called spark-submit command in the cmd prompt. I believe that it is finding the spark context, but it produces a really big error. Can someone please help me with this issue?

Environment variable path: C:/Users/Name/Spark-1.4;C:/Users/Name/Spark-1.4/bin

After that, in cmd prompt: spark-submit script.py

I'm fairly new to Spark, and have figured out how to integrate with with IPython on Windows 10 and 7. First, check your environment variables for Python and Spark. Here are mine:
SPARK_HOME: C:\spark-1.6.0-bin-hadoop2.6\
I use Enthought Canopy, so Python is already integrated in my system path. Next, launch Python or IPython and use the following code. If you get an error, check what you get for 'spark_home'. Otherwise, it should run just fine.

import os

import sys

spark_home = os.environ.get('SPARK_HOME', None)

if not spark_home:

raise ValueError('SPARK_HOME environment variable is not set')

sys.path.insert(0, os.path.join(spark_home, 'python'))

sys.path.insert(0, os.path.join(spark_home, 'C:/spark-1.6.0-bin-hadoop2.6/python/lib/py4j-0.9-src.zip')) ## may need to adjust on your system depending on which Spark version you're using and where you installed it.

execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))