How to deal with different length entities in a Keras DataGenerator? I'm solivng a prediction problem where I need to predict the demand of multiple articles based on their performance during the last 7 days. To get the most out of the data I am trying to implement a rolling window approach where I use 7 days as the training input (x) and the 8th day's performance as the training label (y).
As I am using Keras as a deep learning framework, how do I combine this rolling window approach with a custom data generator that feeds my network during training? The different articles have different data lengths (i.e. article A has 200 days worth of data, article B only has 10 days worth of data).
I implemented a version where I only use one sample per article but that drastically decreases the amount of data I have for training.

I faced a similar problem and the quick solution I came up with was iterate through the main data set, create an instance of TimeseriesGenerator for each of the subsets and feed it to the model. So in your case that would be: go through the collection of articles, instantiate TimeseriesGenerator for each article and feed the windowed samples to your model, ie.:
from keras.preprocessing.sequence import TimeseriesGenerator

# Set the training parameters
window_length = ...
ix0, ix1 = ...
batch_size = ...

for article in articles:
        # Insert your code to generate the article's features and targets
        x, y = ...

        model.fit_generator(TimeseriesGenerator(
                        x,
                        y,
                        window_length,
                        batch_size=batch_size,
                        start_index=ix0,
                        end_index=ix1
                ),
                steps_per_epoch=int(np.ceil((x.shape[0] - ix0 - ix1 - window_length)/float(batch_size))),
                epochs=50,
                verbose=1
        )

Now depending on your dataset size and composition you can either:

run a number of epochs per article
or train one article per epoch, but for that you would need another loop.


How can I compute the ideal variance threshold value for my data? I have a dataset that contains n features scaled between [0,1]. I would use an unsupervised feature selection algorithm (variance thresholding). How can I compute the threshold value?

Variance thresholding is used to select those features with a variance above the suggested threshold. Ideally you would want to take in all features with a non-zero variance but I'm not sure of the data youre handling, it would be better to calculate the variance of the individual feature, arrange them in the increasing order of variance and then select that value where the variance sharply increases.
Or you could do PCA and find the order of the importance of features and then set the threshold.


Package for SMOTEBoost in R I am solving a rare event cum classification problem. I have come across  a package called SMOTEBoost which oversamples the rare event and boost the results. But I'm not sure is that supported in R.

Could you please help me how can I use SMOTEBoost in R? Any examples?

There are two packages in R that should be able to use SMOTE to up sample the minority class:

unbalanced package, and
DMwR package

It doesn't seem like they provide the boosting that is described in the paper that you might be referring to. But it should be straightforward to sew in SMOTE with the idea of boosting using some base tree packages in R like rpart (for example).


Clustering of Weekday Weekend Time Series Data I have a dataset of the number of steps people take throughout a day over a period of months. I aggregated them so that each person will have an average weekday and weekend time series of steps. An example is below:

ID Weekday 00:00 00:30 01:00 ........ 23:00 23:30
A    1       20    30    23             154  256
A    0       15    67    121            35   101
B    1       78    21    46             78   26
B    0       55    301   22             121  451

I have tried clustering using K-means, agglomerative hierarchical and dynamic time warping by combining both weekdays and weekends together (i.e each person only has 1 case of data), and it appears to show 2 distinct groups of people, one group with generally higher levels than the other group.

How can I do clustering while also taking in account whether it is on a weekday or weekend? So for example, there might be a few clusters: one with high weekday activity and low weekend activity, one with high - high, another with low - low, etc?

I am not sure what the terminology is in this case: multidimensional time series? multivariate clustering?

As you don't do any alignments you can simply concatenate the two vectors.

KMeans will then minimize variance in both weekdays and weekends.


Intuitive explanation of Lovasz Softmax loss for Image Segmentation problems Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.

The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.

The loss can be optimized on its own, but the optimal optimization hyperparameters (learning rates, momentum) might be different from the best ones for cross-entropy. As discussed in the paper, optimizing the dataset-mIoU (Pascal VOC measure) is dependent on the batch size and number of classes. Therefore you might have the best results by optimizing with cross-entropy first and finetuning with our loss, or by combining the two losses.

Here is the Implementation of Lovasz Softmax Loss in Pytorch & Tensorflow.

A step by step explanation of the important steps of the code:

Step 1: compute the errors of the predictions:

signs = 2. * labels.float() - 1.
errors = (1. - logits * Variable(signs))
errors_sorted, perm = torch.sort(errors, dim=0, descending=True)

The "errors" vector is positive when the predictions are false and negative when the predictions are correct. Nothing special here. Note that a margin of (1) is used here, which means those correct predictions that have a margin lower than 1 is considered as an error.

Step 2: Lovasz extension w.r.t sorted errors

gts = gt_sorted.sum()
ntersection = gts - gt_sorted.float().cumsum(0)
union = gts + (1 - gt_sorted).float().cumsum(0)
jaccard = 1. - intersection / union

At this point, the vector "jaccard" contains the evolution of the Jaccard index with respect to the sorted errors. It varies between 0 and the actual Jaccard index of the prediction.

Step 3: gradient of the Lovasz extension w.r.t sorted errors

The next step of the code consists of computing the gradient of this vector. It is computed by a backward difference:

jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]

This vector says what is the effect of an error on the evolution of the Jaccard index. The idea is to minimize the errors that penalize the Jaccard index the most.

Step 4: Compute the loss

In the last step of the code, the actual loss is computed as the dot product between the error vector and the gradient of the Lovasz extension. Relu is used because only a positive part of the error vector contains prediction errors.

loss = torch.dot(F.relu(errors_sorted), Variable(grad))

More Info


Overfitting after first epoch I am using convolutional neural networks (via Keras) as my model for facial expression recognition (55 subjects). My data set is quite hard and around 450k with 7 classes. I have balanced my training set per subject and per class label.

I implemented a very simple CNN architecture (with real-time data augmentation):

model = Sequential()
model.add(Convolution2D(32, 3, 3, border_mode=borderMode, init=initialization,  input_shape=(48, 48, 3)))
model.add(BatchNormalization())
model.add(PReLU())
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(256))
model.add(BatchNormalization())
model.add(PReLU())
model.add(Dropout(0.5))

model.add(Dense(nb_output))
model.add(Activation('softmax'))

After first epoch, my training loss decreases constantly while validation loss increases. Could overfitting happen that soon? Or is there a problem with my data being confusing? Should I also balance my testing set?

It is very unlikely for such huge dataset(450k ) to overfit after just one epoch.

Try to run the code for 20-30 epochs and see if you see any decrease in the validation set error (increase in validation accuracy.)

Try to change your learning rate if (unless you are using adaptive learning parameter)


Transformer decoder output - how is it linear? I'm not quite sure how's the decoder output is flattened into a single vector.
As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and we input the decoder with a length M sentence, the output of the decoder will give us M x units output. M is not fixed (M should be the length of the decoder's raw input) and will change during the different steps of inference.
How do we go from here to a single vector?

Screen shot from "Attention is all you need"

I'm not quite sure how's the decoder output is flattened into a single vector

That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


How to model compositional data? What is the best way to model compositional data problems?

Compositional data is when each example or sample is a vector that sums to 1 (or 100%). In my case, I am interested in the composition of minerals in a rock and I have sensors that tell me the sum of the minerals but not the components that make up the sum.

For example, lets say I have two minerals, $m_1$ and $m_2$, that are made up of 3 elements (like copper and other elements from the periodic table) which form a vector of length 3:

m1 = [0.1, 0.3, 0.6]
m2 = [0.6, 0.2, 0.2]

If a rock has 25% of $m_1$ and 75% of $m_2$, the sensor reading produces the sum of the two minerals (shown in bottom-left subplot below):

$$
\begin{align}
&0.25*m_1 + 0.75*m_2 \\
=&0.25*[0.1, 0.3, 0.6] + 0.75*[0.6, 0.2, 0.2] \\
=&[0.475, 0.225, 0.3]
\end{align}
$$

I would like to know how to model and solve the problem of unmixing a composition into its underlying components, where the sum of the elements is normalized to 100% (e.g. $0.25m_1 + 0.75m_2$ has the same composition as $0.50m_1 + 1.50m_2$).

Furthermore, my example is simplistic; in reality a composition can have more than just 2 minerals (up to 3000) and each mineral is made up of 118 elements, not just 3 (all the elements of periodic table - though many elements will be zero). The elemental composition of a mineral is assumed to be known (definition of $m_1$ and $m_2$ in the example). Also, the sensor reading is noisy - each element of the observed composition is assumed to have Gaussian noise.

If I understand correctly, in your problem the mineral-element vector $m1$ and $m2$ is known.

If that is the case, I would suggest LASSO regression. Suppose you have $n$ minerals $m_1$, $m_2$, ..., $m_n\in\mathbb{R}^p$ arranged into a matrix $M=[m_1;m_2,...,m_n]\in\mathbb{R}^{n\times p}$, and an unknown mixing vector $x\in\mathbb{R}^n$ (in your case, $x=[0.25; 0.75]$). Your sensor measures $y=M^\top x\in\mathbb{R}^n$ and you want to recover $x$ from $y$.

The LASSO regression aims to solve $\hat{x}=\arg\min_x ||y-M^\top x||_2 + \lambda||x||_1$. The $\ell_1$ penalty promotes sparsity in $x$ and could be beneficial when you have a lot of candidate minerals.

I don't think you need any special handling due to the compositional nature of the data, other than normalizing the final $\hat{x}$ if needed.


Brown clustering, graph partitioning, agglomerative clustering - libraries/software I need to do some experimenting with Brown clustering, graph partitioning, agglomerative clustering.

1) Are there Python/Matlab libraries for that? I know sklearn.cluster but it doesn't have algorithms I need.

2) Is it possible to install graphical interface for Cluto on Mac OS?

3) Overall, are there useful tutorials on using Cluto?

4) Other software for clustering that I could learn within a couple of hours?

Thanks

For graphs I definitely suggest Networkx and igraph libraries. They support many graph partitioning algorithms.


Forget Layer in a Recurrent Neural Network (RNN) - I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track. The next picture and equation is from Colah's blog post "Understanding LSTM Networks":

where:

$x_t$ is input of size $m*1$ vector
$h_{t-1}$ is hidden state of size $n*1$ vector
$[x_t, h_{t-1}]$ is a concatenation (for example, if $x_t=[1, 2, 3], h_{t-1}=[4,
   5, 6]$, then $[x_t, h_{t-1}]=[1, 2, 3, 4, 5, 6]$)
$w_f$ is weights of size $k*(m+n)$ matrix, where $k$ is the number of
cell states (if $m=3$, and $n=3$ in the above example, and if we have
3 cell states, then $w_f=3*3 $ matrix)
$b_f$ is bias of size $k*1$ vector, where $k$ is the number of cell
states (since $k=3$ as the above example, then $b_f$ is a $3*1$
vector).

If we set $w_f$ to be:
        \begin{bmatrix}
    1 & 2 & 3 & 4 & 5 & 6 \\
    5 & 6 & 7 & 8 & 9 & 10 \\
    3 & 4 & 5 & 6 & 7 & 8 \\
    \end{bmatrix}

And $b_f$ to be: $[1, 2, 3]$

Then $W_f . [h_{t-1}, x_t] =$

$$
    \begin{bmatrix}
    1 & 2 & 3 & 4 & 5 & 6 \\
    5 & 6 & 7 & 8 & 9 & 10 \\
    3 & 4 & 5 & 6 & 7 & 8 \\
    \end{bmatrix}
.        \begin{bmatrix}
    1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \\
    \end{bmatrix}
=\begin{bmatrix} 91 & 175 & 133\end{bmatrix}$$

Then we can add the bias, $W_f . [h_{t-1}, x_t] + b_f=$

$$\begin{bmatrix} 91 & 175 & 133\end{bmatrix} + \begin{bmatrix} 1 & 2 & 3\end{bmatrix}=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$$

Then we feed them into a sigmoid function: $\frac{1}{1+e^{-x}}$, where $x=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$, hence we perform this function element wise, and get \begin{bmatrix} 1 & 1 & 1\end{bmatrix}.

Which means for each cell state, $C_{t-1}$, (there are $k=3$ cell states), we allow it to pass to the next layer.

Is the above assumption correct?

This also means that the number of cell state and hidden state is the same?

Great question!

tl;dr: The cell state and the hidden state are two different things, but the hidden state is dependent on the cell state and they do indeed have the same size.

Longer explanation

The difference between the two can be seen from the diagram below (part of the same blog):

The cell state is the bold line travelling west to east across the top. The entire green block is called the 'cell'.

The hidden state from the previous time step is treated as part of the input at the current time step.

However, it's a little harder to see the dependence between the two without doing a full walkthrough. I'll do that here, to provide another perspective, but heavily influenced by the blog. My notation will be the same, and I'll use images from the blog in my explanation.

I like to think of the order of operations a little differently from the way they were presented in the blog. Personally, like starting from the input gate. I'll present that point of view below, but please keep in mind that the blog may very well be the best way to set up an LSTM computationally and this explanation is purely conceptual.

Here's what's happening:

The input gate

The input at time $t$ is $x_t$ and $h_{t-1}$. These get concatenated and fed into a nonlinear function (in this case a sigmoid). This sigmoid function is called the 'input gate', because it acts as a stopgap for the input. It decides stochastically which values we're going to update at this timestep, based on the current input.

That is, (following your example), if we have an input vector $x_t = [1, 2, 3]$ and a previous hidden state $h_t = [4, 5, 6]$, then the input gate does the following:

a) Concatenate $x_t$ and $h_{t-1}$ to give us $[1, 2, 3, 4, 5, 6]$

b) Compute $W_i$ times the concatenated vector and add the bias (in math: $W_i \cdot [x_t, h_{t-1}] + b_i$, where $W_i$ is the weight matrix from the input vector to the nonlinearity; $b_i$ is the input bias).

Let's assume we're going from a six-dimensional input (the length of the concatenated input vector) to a three-dimensional decision on what states to update. That means we need a 3x6 weight matrix and a 3x1 bias vector. Let's give those some values:

$W_i = \begin{bmatrix} 1 & 1 & 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 & 3 & 3\end{bmatrix}$

$b_i = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}$

The computation would be:

$\begin{bmatrix} 1 & 1 & 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 & 3 & 3\end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \\5 \\6 \end{bmatrix} + \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 22 \\ 42 \\ 62 \end{bmatrix}$

c) Feed that previous computation into a nonlinearity: $i_t = \sigma (W_i \cdot [x_t, h_{t-1}] + b_i)$

$\sigma(x) = \frac{1}{1 + exp(-x)}$ (we apply this elementwise to the values in the vector $x$)

$\sigma(\begin{bmatrix} 22 \\ 42 \\ 62 \end{bmatrix}) = [\frac{1}{1 + exp(-22)}, \frac{1}{1 + exp(-42)}, \frac{1}{1 + exp(-62)}] = [1, 1, 1]$

In English, that means we're going to update all of our states.

The input gate has a second part:

d) $\tilde{C_t} = tanh(W_C[x_t, h_{t-1}] + b_C)$

The point of this part is to compute how we would update the state, if we were to do so. It's the contribution from the new input at this time step to the cell state. The computation follows the same procedure illustrated above, but with a tanh unit instead of a sigmoid unit.

The output $\tilde{C_t}$ is multiplied by that binary vector $i_t$, but we'll cover that when we get to the cell update.

Together, $i_t$ tells us which states we want to update, and $\tilde{C_t}$ tells us how we want to update them. It tells us what new information we want to add to our representation so far.

Then comes the forget gate, which was the crux of your question.

The forget gate

The purpose of the forget gate is to remove previously-learned information that is no longer relevant. The example given in the blog is language-based, but we can also think of a sliding window. If you're modelling a time series that is naturally represented by integers, like counts of infectious individuals in an area during a disease outbreak, then perhaps once the disease has died out in an area, you no longer want to bother considering that area when thinking about how the disease will travel next.

Just like the input layer, the forget layer takes the hidden state from the previous time step and the new input from the current time step and concatenates them. The point is to decide stochastically what to forget and what to remember. In the previous computation, I showed a sigmoid layer output of all 1's, but in reality it was closer to 0.999 and I rounded up.

The computation looks a lot like what we did in the input layer:

$f_t = \sigma(W_f [x_t, h_{t-1}] + b_f)$

This will give us a vector of size 3 with values between 0 and 1. Let's pretend it gave us:

$[0.5, 0.8, 0.9]$

Then we decide stochastically based on these values which of those three parts of information to forget. One way of doing this is to generate a number from a uniform(0, 1) distribution and if that number is less than the probability of the unit 'turning on' (0.5, 0.8, and 0.9 for units 1, 2, and 3 respectively), then we turn that unit on. In this case, that would mean we forget that information.

Quick note: the input layer and the forget layer are independent. If I were a betting person, I'd bet that's a good place for parallelization.

Updating the cell state

Now we have all we need to update the cell state. We take a combination of the information from the input and the forget gates:

$C_t = f_t \circ C_{t-1} + i_t \circ \tilde{C_t}$

Now, this is going to be a little odd. Instead of multiplying like we've done before, here $\circ$ indicates the Hadamard product, which is an entry-wise product.

Aside: Hadamard product

For example, if we had two vectors $x_1 = [1, 2, 3]$ and $x_2 = [3, 2, 1]$ and we wanted to take the Hadamard product, we'd do this:

$x_1 \circ x_2 = [(1 \cdot 3), (2 \cdot 2), (3 \cdot 1)] = [3, 4, 3]$

End Aside.

In this way, we combine what we want to add to the cell state (input) with what we want to take away from the cell state (forget). The result is the new cell state.

The output gate

This will give us the new hidden state. Essentially the point of the output gate is to decide what information we want the next part of the model to take into account when updating the subsequent cell state. The example in the blog is again, language: if the noun is plural, the verb conjugation in the next step will change. In a disease model, if the susceptibility of individuals in a particular area is different than in another area, then the probability of acquiring an infection may change.

The output layer takes the same input again, but then considers the updated cell state:

$o_t = \sigma(W_o [x_t, h_{t-1}] + b_o)$

Again, this gives us a vector of probabilities. Then we compute:

$h_t = o_t \circ tanh(C_t)$

So the current cell state and the output gate must agree on what to output.

That is, if the result of $tanh(C_t)$ is $[0, 1, 1]$ after the stochastic decision has been made as to whether each unit is on or off, and the result of $o_t$ is $[0, 0, 1]$, then when we take the Hadamard product, we're going to get $[0, 0, 1]$, and only the units that were turned on by both the output gate and in the cell state will be part of the final output.

[EDIT: There's a comment on the blog that says the $h_t$ is transformed again to an actual output by $y_t = \sigma(W \cdot h_t)$, meaning that the actual output to the screen (assuming you have some) is the result of another nonlinear transformation.]

The diagram shows that $h_t$ goes to two places: the next cell, and to the 'output' - to the screen. I think that second part is optional.

There are a lot of variants on LSTMs, but that covers the essentials!


What is the scalability of linear regression and decision trees? Recently I'm studying machine learning algorithms among them linear regression and decision tree so I have a question regarding the scalability of both algorithms. Can anyone provide what is the scalability of both algorithms and examples?

If you use the standard OLS formula :

$$ \beta = (X^TX)^{-1}X^Ty$$

the overall linear regression complexity is $O(np^2)$, where n is the number of exemple and p the number of features.

For simple trees the theoretical complexity if of the same order $O(np^2)$.

In practice other techniques are used, like gradient descent for the regression. Also you may have simplifications depending on your problem ($n>>p$ for exemple, may lead to the use of sparse matrices).

Practical implementation like sklearn have following complexities (see https://www.thekerneltrip.com/machine/learning/computational-complexity-learning-algorithms/):

LinearRegression : $O(n^{0.72}p^{1.3})$

ExtraTreesRegressor: $O(n^{1.03}p^{0.88})$

Overall they are relatively simple algos and thus are pretty much as scalable as you can get. (or conversely we wouldn't use algorithms that are not scalable)


Number of parameters in Simple RNNs Please, I am stuck, I can not understand the number of parameters of a simple RNN, here the example and the model summary.
the example is simple:
x = np.linspace(0,50,501)
y= np.sin(x)
df= pd.DataFrame(data=y, index=x, columns=['Sinus'])

Then I would to build a simple RNNs to predict this sine wave,
test_percent = 0.1
test_point= np.round(len(df)*test_percent)
test_ind = int(len(df)-test_point)
train = df.iloc[:test_ind]
test = df.iloc[test_ind:]

from sklearn.preprocessing import MinMaxScaler
scaled_train = scaler.transform(train)
scaled_test = scaler.transform(test)

from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator
generator = TimeseriesGenerator(scaled_train, scaled_train, length=50, batch_size=1)

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, SimpleRNN

n_features=1
model= Sequential()
model.add(SimpleRNN(units=50, input_shape=(50,1)))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam')
model.summary()

I do not understand the number 2600 in the model summary!
because according to this intuition of RNNS

in my case the dimension of U is (50,1), so 50 weights, then for V, also 50 weights, and for W, also 50 weights, for me, only 150 parameters, why this number in the summary: 2600 parameters?

Keras SimpleRNN is a fully connected RNN so indeed each unit connected with all other units. So the equation becomes:

(input_feature +1) x units + units x units
2x50 +2500
+1 comes from the bias


Why do decision trees have low accuracy? It seems to be generally acknowledged that decision trees have low prediction accuracy. Is there a concise explanation for why they have low accuracy?

I've read this so much, I've accepted it to be true, but I realize I don't have any intuition as to why it's true.

As an example, here's an excerpt from Elements of Statistical Learning (page 352):

  Trees have one aspect that prevents them from being the ideal tool for predictive learning, namely inaccuracy. They seldom provide predictive ac- curacy comparable to the best that can be achieved with the data at hand.

Or on Wikipedia, under the heading Disadvantages of Decision Trees: "They are often relatively inaccurate. Many other predictors perform better with similar data. "

It's not true in general. Decision trees tends to overfit in comparison to other algorithms, which provide too low accuracy. But if you use a decision tree in the right way i.e you prepare data in the proper format, use feature selection and perform k-fold cross-validation everything should be ok.

I am sure that you misread it. There is no reason why DT could be a much worse algorithm compared to others.


Modeling price vs demand I have a dataset consisting of products, clients, price policy, discounts, quantities, and net sales. The task as put in words by the business is quantity vs price. I have noted a few observations from looking at the dataset :

Discounts: Discounts nullify the effect of any change in the Price policy. So in the end the net sales don't follow this variation. And i observe this for so many client-product pairs.

Seasonality : Variation of quantity for client-product pairs simply follows a seasonality pattern and its not driven by any of the timeseries variables in the dataset. ( I should statistically verify this for now i just did a visual check).

Because at this point i dont see any logic behind how discounts are decided for the clients. Hence there is literally no affect on net sales vs price changes.
How should I model this ? Is this even a machine learning problem because there is simply no causal relation between the variables. If not Price vs Demand then what other things can I propose to the business ?
Edit : 1. Product-client scatter plots UNITARY_NET_SALES VS QUANTITY

The first column of the plot shows products are demanded at the same quantities across varying net sales. So no price vs demand effect here.

Timeseries plot for a particular product

Price_list and Discounts have the same behaviour. So whenever the business increases the prices they increase the discounts too, hence the overall affect on net sales is none. And Quantity simply follows a seasonal pattern.

Describe on the columns

75% of Quantity is less than 8 units !
Thank you !

I've tangled with modeling pricing systems over the last two years and one of my key learnings applies here:
Available sales data is often a bad basis for straight-forward prediction tasks and the reason for this is fairly simple:
If you classify all prices (or transactions of a given product at a price x) into "Accepted" and "Not accepted" by the customer you will realize that the data provided by your customer only contains instances of "Accepted" prices.
Therefore a straight-forward modeling of y ~ x with y = demand and x = price is impossible because your y does not vary!
There are several ways around this however. In my comment I mentioned discount being a valuable information!
You have already noticed that discounts are not depended on logical variables, they are seemingly random, this isn't true!
Discounts in most organizations are very, very flexible and often applied manually based on negotiations. This means they are a great indicator of our target y "Acceptance"/"Non Acceptance".
Consider this:
Discount = Demand x Undiscounted_Price
This means that large discounts indicate that the demand is low / negative for the undiscounted price and low discounts indicate demand is high.
To truly discover this relation you might need to model codependent factors and then remove them by training new models on the residuals.
Edit:
An important thing to add especially for the B2B domain. Demand for a certain product is almost set in stone for a customer. Unlike consumers companies do not buy surplus or refrain from purchase due to the prices.
What they do is switch suppliers! This means that there is a really important unknown variable "Customer Demand for product X". You do not want model this variable but you need it to model what you actually want to do:
Share of Wallet or the percentage of the fixed customer demand that was satisfied by your company with the goal being to identify the price that will optimize that percentage. This is helpful to keep in mind because it constrains the performance of any model as you never know whether the historic demand you have in your data is already 0% or 100% of the total demand and thus could not decrease/increase regardless of price.


Any pytorch tools to monitor neural network's training? Are there any tools to monitor network's training in PyTorch?

I am looking for an equivalent to tensorboard in tensorflow.

I am using tensorboardX. It supports most of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. Haven't tried the rest like audio and graph. But the repo also contains examples for those usecases. Installation can be done easily with pip. It's all explained in the readme.

There are also other software which implement a wrapper for PyTorch (and other languages/frameworks) of TensorBoard. As far as I know, they support fewer functionalities. But have a look at Crayon and Tensorboard-Logger.


Importing Common Constants to R and Python I use Python for data munging, R for data analysis, and I combine both by running the munging/analysis sequentially in a Makefile. My Python and R scripts depend on various constants. I typically store my Python constants in a settings.py file, and I import this file across several Python scripts which depend on those constants. My R constants are typically not used in more than one script, so I store those constants in the script which uses them.

Some constants are shared by both the Python scripts and R scripts. Right now, I am defining them separately in R/Python files. However, it would be better if there was some way of defining constants in common across Python and R files, since this redundancy can lead to mistakes.

What would be the best way for me to define constant in common across files? Obviously, there is the problem that these are two different languages; so, for example, a list in Python is not quite the same as a list in R. However, there might be some solution which is more elegant than simply repeating constants across files.

I believe this would be the best place for this question, but if it belongs in the Python or R StackExchanges, please let me know.

You should consider storing this sort of data into a format that is more universally accessible.  I would suggest either JSON or YAML. JSON can be readily parsed by both R and Python

Your data would look something like:

{
    "FIRST_YEAR": 1952,
    "IV_TYPES": [
        ["p", 6],
        ["n", 6],
        ["w", 4]
    ]
}


Does feature selection removes highly corelated variables? I know that feature selection helps in removal of irrelevant features. Do they also remove redundant features(features which convey the same info as as some other feature)? If not, which methods are used for removing these highy corelated variables?

Lasso is a very common strategy as it is able to handle correlated variables.

https://stats.stackexchange.com/questions/7935/what-are-disadvantages-of-using-the-lasso-for-variable-selection-for-regression


Threshold to consider new feature as a new finding to a model? I am working on binary classification problem with 5K records and 60 features.

Through feature selection, I narrowed it down to 14 features.

In existing literature, I see that there are well-known 5 features.

I started my project with an aim to find new feature that can help improve the predictive power of the model

However, I see that with well known features (reported in literature), it produces an AUC of 84-85 and having all my 14 features decreases it to 82-83.

So I tried manual add and drop and found out that if I add only one feature (let's say magic feature), it increases the AUC to 85-86.

I see that there is a difference of 1 point in AUC.

1) Is it even useful to be happy that this adds some info to the model?

2) Or me looking at AUC is not the right way to measure model performance?

3) Does it mean the other new features (9 out of 14) that I selected based on different feature selection/ genetic algorithm aren't that useful? Because my genetic algorithm returned 14 features, so I was assuming that was the best subset but still through my previous experiments I know that model had better performance when it had 5 features. Any suggestions here? What can I do?

4) I am currently using train and test split as my training and testing data. I applied 10 fold cv to my data. Should I be doing anything different here?

5) If I add around 16-17 features, I see the AUC is increased to 87 but this can't be over fitting right? Because if it's overfitting, shouldn't I be seeing the AUC as97-100or just100? I know we haveoccam razor's principleto keep the model parsimonius but in this case, just having 16-17 features in model is not too complex or heavy. Am I right? Because it's increasing theAUC`. Any suggestions on this?

A lot of questions here. Here are some thoughts.

should you be happy about a 1 point increase in AUC? Yes. An effect can be genuine, but small. A 1 point advantage is still an improvement. But do I trust that outcome? Not sure.
you need some more data. Your sample size is not large. Furthermore, cross-validation is a wonderful thing, but you have been running a lot of tests on the same small data set, so cross-validation notwithstanding -- we really don't know how your classifier will perform on brand new, unseen data.
point 3. This sounds like an issue with over-fitting.
point 4. I'm not sure what you are doing here. Cross-validation is an alternative to "train" and "test" sets, since each fold acts as the "test" set to the model fit on the "Non-fold" part of the data. Did you mean to say that you were doing CV on the train part, and then used the "test" part for a final check on performance? That would be a good thing to do, but if you have been using the same test set to check every model you have tried, then that test set is beginning to look like a training set. You will need another "test" set.
point 5. You don't need a ridiculously high AUC to be guilty of over-fitting. Over-fitting happens when you fit features to the noise in your data set. An over-fit model will have a higher mean squared error on a fresh test set than the optimal model, although it will do better on the training set. Having 16 to 18 features is too many if, in fact, the outcome is well explained with only 5.
Given that you have reduced your candidate features to 14, you now have a much smaller problem, and it should be possible to examine the features from a subject matter perspective. Which features would a SME recommend you to retain? And you can also examine correlations between the 14 features and check for redundancy that way. With a small data problem (which is what this is), you can work to understand your data directly. That approach might yield some interesting insights.
You might want to use a different method of model selection than maximal AUC. This paper discusses the merits of AUC and offers an alternative. Could be interesting.


Classification with heavily overlapping data points of different labels I am trying to build a classifier that will classify data points of different classes, but which significantly overlap, but I'm not sure how to best approach the problem. I have read that tree models can perform well in this situation, and that Support Vector Machines might as well, but I'm unclear what other approaches might be good for tackling this problem. Can anyone give me some direction?

There are many approaches you could take here. Assuming data overlap in such a way that the classes are still distinguishable from a human expert—human performance is generally regarded as the performance ceiling in supervised classification tasks—you can approach this from the classifier side of the feature side.

First, I would consider the feature representation you're using for your task. A classifier is only as good as the information you give it, and there are many ways you can adjust how you're representing your input data. For example, if you're working in text classification, using a unigram representation will typically lead to different performance as compared with a bigram representation. Similarly, there may be non-linear transformations you can apply to continuous-valued data that a classifier may not pick up on its own. You should consider the domain in which your problem exists and create a set of feature generation and feature selection cross-validation experiments using a hold-out training data set.

Next I would consider what alterations you can make on the classifier end. If your data are highly skewed (i.e., one class is much more prevalent that another), it may not make sense to use an algorithm like Naïve Bayes, at least not with out some sampling approach. If your problem isn't high-dimensional, I recommend generating a graphic that will let you explore the spatial breakdown by class. There are certain overlap patterns that linear classifiers like Support Vector Machines will not be able to classify without using a kernel function, such as a Radial Basis Function.


R NbClust for Python? Is somebody know a Python equivalent to R NbClust?

I'm searching for a way to determine the optimal clusters number for many 'similar datasets' and NbClust sounds good but it's for R and i'm a Python user...

I have not found any unified package in Python like NbClust.  Check out the Python package rpy2 for using R modules in Python.


Why SVM gridsearch takes longer time? I have a dataset of 5K records and 60 features focussed on binary classification. Please find my code below for SVM paramter tuning. It's running for a longer time than Xgb.LR and Rf. The other algorithms mentioned returned results within minutes (10-15 mins) whereas SVM is running for more than 45 mins.

Questions

1) Is SVM usually slower and takes longer time?

2) Is there any issue with my code below?

3) How can I make the gridsearch faster?

from sklearn.svm import SVC
param_grid = {'C': [0.001,0.01,0.1,1,10,100,1000],
              'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
              'kernel': ['linear', 'rbf','poly'],
              'class_weight':['balanced']}
svm=SVC()
svm_cv=GridSearchCV(svm,param_grid,cv=5)
svm_cv.fit(X_train_std,y_train)

Simple, optimization problem of SVM is of quadratic order.
Just check first line of documentation

  "The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples."


Multichannel numpy array to PIL image I have a 4 channel Numpy image that needs to be converted to PIL image in order implement torchvision transformations on image. But when I try to do this using PIL.Image.from_array(<my_numpy_image>) I get the following error.

TypeError: Cannot handle this data type

I cannot loose and get rid of any channel information right now so won't be able to discard any channels.

Try specifying mode so that PIL is aware of data format.

img = Image.fromarray(source_array, mode="CMYK")

If that does not work, what is the shape of source array ?


How to transform an imbalanced attribute to make it more suitable for linear regression? I'm new to data science but trying to get better

Here I have an attribute and plotting its histogram

From what I know so far such a distribution is imbalanced and my goal is to equalize things a little bit right?
Again from what I know, I have to transform this attribute to be more suitable for linear regression?

Is it obvious (to someone more experienced than me) which kind of transformation is applicable in this case?

Note that this is an attribute and not my target, this is not what I am trying to predict. This is one of the attributes to be used for predicting

You have multiple options and you may choose the best by seeing the performance.

First one is to use a guess. For positive-only values the log-transform is a hot candidate to make sense (maybe correct for small values here to avoid exceedingly large negative transformed values).

Log-transform is natural if percentage increases have a particular real-world meaning. This is often the case for financial data. Why is log a hot candidate? You probably know that when there are addititve real world effects, the normal distribution often appears. Now, when you have multiplicative effects, you get the https://en.wikipedia.org/wiki/Log-normal_distribution .

Other common transformations are power transforms, where you take some power of the values. I don't think there are many more which are very common. Theoretically, your perfect transform would make the noise on the linear regression Gaussian, but no-one can tell what that would be and most likely reality isn't perfectly linear anyway.

A transform is more interesting when the transformed values follow a Gaussian distribution. But that is just a guess and in the end only final performance evaluation can tell more.

For a second option, be aware that you can force the transformed values to be any distribution you want. For example, if you take ranks of the values you get a uniform transformed distribution. You could even force it to be a Gaussian by a suitable mapping. However, in your case, this will lose the interesting bump on the right.

I think these are the most common options. In data science, nothing is ever obvious and most of the time you can only decide by performance evaluation (cross-validation with the whole model).

Conclusion:

Set up a performance test (cross-validation; not on final test set though if you like a fair, final evaluation) and try all of the following

Try untransformed. It might already be what has most information.
Try log-transform (while adding a small offset if you have some very small values)
Try power-transform if you like


How can I use two different datasets as a training model for svm I know that you're supposed to scale your test data using the parameters (mean and stdev) from your training data. This is relatively simple; but what if the number of samples is limited in one training data set (e.g. Set A = 5 samples) so I want to combine two data sets (i.e. Set A + Set B = 10 samples) to have enough samples for training, what can I do so that I can scale/normalize the two sets into one and then use those parameters on my test set? If I scale them individually I will have 2 means and 2 stdev.

The context is I'm trying to combine two microarray expression from two different microarray platform so their expression ranges are different.

Thank you for your help in advance

From a proper methodological standpoint, you should do the scaling you're proposing after the two sets are merged.

Either way, the model is not going to be able to differentiate between, say, an anomalous reading from one generating process that falls into the range generated by the other generating process unless you also make a variable indicating the source system that the observation came from (assuming all else is equal).

You need to make sure that both sets of observations actually represent the same population of possible observations in order to make this modeling decision.


When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc? I have a data set with categorical and continuous/ordinal explanatory variables and continuous target variable. I tried to filter features using one-way ANOVA for categorical variables and using Spearman's correlation coefficient for continuous/ordinal variables.I am using p-value to filter. I then also used mutual information regression to select features.The results from both the techniques do not match. Can someone please explain what is the discrepancy and what should be used when ?

Pearson correlation and ANOVA will measure the linear variance between x,y.
A mutual information regression method is an entropy-based method which can find even non-linear correlation.
Let's see 2 pair of data (x, y_1) and (x, y_2).
##0. Data setup
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y_1 = 3.39*x_1
y_2 = np.array([1, 1, -1, -1, 0, 0, -2, -2, 2, 2])
df_1 = pd.DataFrame([x,y_1]).T
df_2 = pd.DataFrame([x,y_2]).T

##1. Pearson corr
df_1.corr()
df_2.corr()

##2. f_test
from sklearn.feature_selection import f_regression, mutual_info_regression
f_test, _ = f_regression(x.reshape(10, 1),y_1)
f_test
f_test, _ = f_regression(x.reshape(10, 1),y_2)
f_test

##3. Mutual reg
mutual_info_regression(x.reshape(10, 1),y_1,n_neighbors=1,random_state=123)
mutual_info_regression(x.reshape(10, 1),y_2,n_neighbors=1,random_state=123)

##4. Tree-based feature importance is also based on entropy reduction
##and is more reliable as  KNN needs a good k value

 We can observe that Mutual importance is able to catch the relation of x with not just y_1 when it is simple Linear bust also with y_2 when it is not linear(but has a strong relation)
You should use either MI if you are only looking for a filter-based method. Otherwise, Tree feature importance would be best for most of the cases.
Check this link -
F_test Vs MI


Calculate correlation between two sensors regarding the time I am new in all this, so I am sorry if I am asking something stupid.
I have two-time series - the measurement of the value by two sensors every 5 min and I want to see are the measurements correlated in time (are the measurements similar). So far now I only found questions related to autocorrelation and classic correlation. My dataframe looks like this:

                  id      time                      sensor1      sensor2
                  1       24.1.2020. 00:00:00        0.052         0.631
                  1       24.1.2020. 00:05:00        0.812         0.102            ....
                  1       24.1.2020. 00:10:00        0.326         0.500             ....
                  1       24.1.2020. 00:15:00        1.021         0.999             ....
                  1       24.1.2020. 00:20:00        1.033         1.000             ....

and so on for 10 days.

So, time here is a really important aspect because measurements depend on part of the day, of climate condition during the day and so on. I saw on some question that people are suggesting pandas function:

corr = df['value1'].corr(df['value2'])

But as I see, this line doesn't include the time.
Also if you have some course for data science to recommend, it will be appreciated. I am already taking something.

Your suggested method should work, assuming the two column are in the same dataframe with the same timestamps as the  index. This means that they are already aligned in time and therefore the time is implicitly encoded.

If there are any missing datapoints, you could impute them (fill them) with something like the previous value, which is usually best for time-series data. Replacing with e.g. the mean of the entire time-series might be misleading because there are temporal fluctuations/cycles involved.

df.fillna(method='ffill')

You could also simply plot them to see which kind of correlation value you might expect, doing something like this:

import matplotlib.pyplot as plt    # <-- needed if running in script (not a Jupyter notebook)
df[["sensor1", "sensor2"]].plot()
plt.plot()


If you want to get the time duration between the start and end in seconds, you can try this :

data %>% group_by(SessionID) %>%
summarise(start_time = min(Timestamp), end_time =max(Timestamp)) %>% # using your code until here
mutate(`session end_time in seconds` = as.numeric(difftime(end_time, start_time, units = "secs"))) # gives the time interval in seconds


Does a variable with different range for rows affect the classification model I am trying a binary classification on a Time Series data using XGBOOST and have sales variables with different currencies for different customers. Will different currencies (basically variable with different range) have a significant impact on the model? The two solutions I could think of is:

Conversion for all rows for sales variable in a single currency.
Adding a region of sales/currency as a categorical variable.

Will the second solution work or I should stick with the first one.

The first solution is valid. You need to keep the same unit for all rows per a given feature. But for currencies, you should be careful about the rate of conversion which might be dependent on the date of the data collection.

For the second solution, I am not quite sure about your end application but you may use it if you are trying to incorporate the effect of currency type in your model prediction. But if you are proposing it as alternative to the first solution, your model will less likely to infer the conversion rate among the various currency types.


K mean clustering method of data mining I am reading about k-means clustering method.

How does the method calculate the mean between two coordination points?
How does the method work with more than two dimensions? How can calculate the distance and the mean if there are more than two dimension?

Are there any packages about k-means clustering in R?

To answer simply, Euclidean distance is generally used:

$$ d = |\mathbf{x}-\mathbf{y}|=\sqrt{\sum_{i=1}^n|x_i-y_i|^2}$$

Read more about k-means over here:

Introduction to K-means
k-means clustering algorithm

Using kmeans() in R:

K Means Clustering in R


Unsupervised image classification? Does this exist? What algorithm or combinations of algorithms would be able to classify images without supervision?

For example if you have many pictures of cats and dogs, then without being trained to distinguish them first, it would go through the images and realize there are two distinctive groups of images (either cat or dog)

Or if you have many random pictures but a good portion of them have trees, then going through the images, the algorithm realizes there is one consistent pattern within some of the images (trees)

As a starting point, you can think about unsuvervized image classification as a type of image clustering. You can - for instance - use VGG16 weights, extract image pseudo-features, and run some clustering on this feature set.
Here is some "starter code" (Python):
# Imports
from keras.preprocessing import image
from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import preprocess_input
import numpy as np
from sklearn.cluster import KMeans
import os, shutil, glob, os.path
from PIL import Image as pil_image
image.LOAD_TRUNCATED_IMAGES = True
model = VGG16(weights='imagenet', include_top=False)

# Variables
imdir = 'C:/indir/' # DIR containing images
targetdir = "C:/outdir/" # DIR to copy clustered images to
number_clusters = 10

# Loop over files and get features
filelist = glob.glob(os.path.join(imdir, '*.jpg'))
filelist.sort()
featurelist = []
for i, imagepath in enumerate(filelist):
    try:
        print("    Status: %s / %s" %(i, len(filelist)), end="\r")
        img = image.load_img(imagepath, target_size=(224, 224))
        img_data = image.img_to_array(img)
        img_data = np.expand_dims(img_data, axis=0)
        img_data = preprocess_input(img_data)
        features = np.array(model.predict(img_data))
        featurelist.append(features.flatten())
    except:
        continue

# Clustering
kmeans = KMeans(n_clusters=number_clusters, random_state=0).fit(np.array(featurelist))

# Copy images renamed by cluster
# Check if target dir exists
try:
    os.makedirs(targetdir)
except OSError:
    pass
# Copy with cluster name
print("\n")
for i, m in enumerate(kmeans.labels_):
    try:
        print("    Copy: %s / %s" %(i, len(kmeans.labels_)), end="\r")
        shutil.copy(filelist[i], targetdir + str(m) + "_" + str(i) + ".jpg")
    except:
        continue


How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations.
Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get the inertia_ after KMeans fitting my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.

It sounds like you are grappling with large data set sizes, for which I first suggest switching to mini-batch k-means. Mini-batch scales better so will be less frustrating.

Regarding apriori estimates of the inertia_, I suggest using a sample data set to approximate the inertia_ with appropriate margins of error. But, mini-batch may just preclude your need for apriori inertia_.

Hope this helps!


Word labeling with Tensorflow A part of our system does a limited rule-based word labeling (kinda WSD). Basically it takes a dependency tree of a sentence, finds words or POS tags of interest and assigns to each a label/sense based on the word itself, it's POS tag, position in sentence and so forth.

I was wondering if ML methods in general and Tensorflow in particular can be used for the task, and if so which ones to try. I'm familiar with theoretical basics of ML, but never actually used it in work, so I'd grateful for any pointers.

Basically, our input is a ConLL representation of a sentence that we can turn into an matrix:

[
 ['cat', 'NN', 'nsubj']
 ['jumped', 'VBD', 'root']
 ['over', 'IN', 'case']
 ['fox', 'NN', 'nmod:over']
]

we can encode words and labels to numbers and pad sentence to max length of say 60 words, which gives the final input 60x3 matrix of integers. Our output then is a vector of 60 numbers with codes for labels for corresponding input words.

So far my research points to using bidirectional RNN for the task, but all tutorials I can find use them either for classification (as with MNIST) or finding the next best word as in the Tensorflow's RNN tutorial, whereas I want to label each word/input.

So, my questions are:

Is a bidirection RNN a good choice for the task? And if so, what should the composition be for it? How many steps and hidden layers would you recommend?
If RNN isn't a good choice, what else would you recommend?
Would using word embeddings add to the accuracy if our labeling doesn't care for word meanings/similarities?

One solution to this problem can be found in Grammar as a Foreign Language. The paper outlines a system that "translates" English sentences into syntactic constituency parse trees using a sequence-to-sequence LSTM (Long Short-Term Memory model). A TensorFlow implementation can be found here.


Representing geospatial information I am trying to train a model to predict the location of a storm at a given time. The dataset includes the longitude and latitude of the storm at the given "timestamps" but I am not sure if that is the best way to represent the location as it doesn't likely have a linear relationship.

Is there a way to combine the longitude and latitude into a feature that can be used for training? I was thinking about creating "grids" to represent spaces but I'm not sure how I would go about creating these grids or converting them back to a long/lat range.

Any help is appreciated. Thanks.

There are a lot of things one could do with that information.

Lat. and long. are the most precise data one can get on location, but that data itself may not say much about anything, really.

When I read this:

  predict the location of a storm at a given time

the first things that come into my mind are: what would define each coordinate point? You could get terrain height, closeness to some geographic accident... Things that would actually impact on the weather, more precisely, storms.

This is all intuitive. I'm not sure this can help in any way, but hopefully, it will at least help to think of something actually useful.


When is a time series considered to be "complex"? (not complex-valued) what is a measure or definition for complex time series. When is a time series considered to be normal and when it is considered to be a complex?

A lot of researcher saying they predicting/analayzing complex time series. However, I am asking me what is the property that makes a time series a complex one.

Most of the times one refers to a complex time-series to indicate that this is not, for example, a predictable periodic time-series or a sequence of repeating patterns with fixed seasonality.

Complex time-series would be those coming e.g. from real-world measurements, and can be non-stationary, very noisy, have different and dynamic behaviors and seasonality, contain outliers and missing values, and so on.


Features used by MAUI key phrase extraction tool I have been trying out keyphrase extraction for a while and I want to know what are all the features that MAUI MAUI github uses for training the keyphrase extraction?
Having read this its hard for me to understand every feature that she uses can someone help me give a brief about the features that she is using? and some references for the same if any.

The references are in the link that you give: here you can find references to two papers which describe the method:

Domain-Specic Keyphrase Extraction
KEA: Practical Automatic Keyphrase Extraction

Fyi it seems there's no neural nets involved (the question is tagged with neural-network)


How to do data augmentation for Machine Learning on statistical data? I am training machine learning classification model. I have data in csv format with lets say 5 features(or columns) and 100 such observations(or rows).

I want to add more similar data to improve my model accuracy. I do not have more data from original source.

I did data augmentation on images in deep learning where I performed brightness(-/+), rotate, flip. But I do not know how to do such exercise in ML.

Please help.

You can do mainly two things: bootstrap or oversampling.
With statistical data you can do bootstrapping (random sampling with replacement)
Bagging methods help boosting you model accuracy. The pseudocode will be a bit like this.
for estimator in range(number of estimators):
    Sampling some data
    Fitting a model
    Predicting
mean(predictions)

This way you are able to train different models with the same sample of data and helps in boosting accuracy. Well known as Random Forest use this technique.
There is other way to create synthetic data and they are the oversampling/undersampling techniques that you can have a look in the imblearn documentation. Techniques like SMOTE (Synthetic Minority Over-sampling Technique) are sometimes used, but they generate synthetic data.
With statistical data you can not rotate, add brightness...


XGBoost change loss function I'm using XGBoost (through the sklearn API) and I'm trying to do a binary classification.

False Positives are much worse for me than False Negatives, how can I take this into account?

The API confuses me a bit and I found two arguments that might be related to this:

objective & eval_metric

Which one is it? And how to use it?

In a scenario where consequences of prediction errors are not equivalent, you are usually still interested in training a model to predict accurately from the data set, and would not change the objective function in supervised learning.

Typically when consequences of FP and FN differ, you would:

Use the confidence of the prediction given by the model. In XGBoost, that is objective: "binary:logistic"
Use area under ROC as a base metric for deciding best tuned model (this is eval_metric: "auc" for XGBoost). This metric is a measure of how well sorted your classes are - the higher the value, the easier and more effective it is to tune the confidence level later.
Decide a weighting for your costs FP and FN - this is entirely up to you, you say that FP is worse than FN, but in order to make an optimal decision you have to turn that into a numerical statement. If you can assign a relative financial cost to a business for each kind of mistake, then that would be a good start.
Using the best tuned model, use the weightings to calculate costs at different cutoffs of class confidence before assigning positive class that you wish to act upon in some way. Predict against a test set to get class probabilities. Then use different cutoff points for positive class, count your FP and FN given that cutoff, and multiply the totals by the costs you chose later. The lowest-scoring cutoff point should be the one to use in production. If FP has a higher cost than FN, then you will likely find that a confidence level > 0.5 is required.

You can also look into machine learning approaches that consider the consequences of actions - e.g. reinforcement learning - in order to achieve a similar result within a single learning framework. However, that is not something you can do solely within XGBoost, and as long as your problem remains balancing between FP and FN in a single step prediction, then the above approach should be OK.


Amazon Cloud Image istance most suited for R data mining I'm new to the field of machine learning, I always used my laptop for regular statistical analysis with no performance problems. Though lately I started programming with caret and I find myself stuck with hours optimizing models and resampling datasets. I saw EC2 istances but I can't understand the difference between the different classes, I know that generally the one with highest numbers of cpus and RAM are most performant but what is the best type of istance for R programming (for example p one vs c one)? Then I'll chose the right amount of memory for my applications but I'm wondering if one subset is more suited than the others

It very much depends on the calculations you are doing as well as the tools you are using to implement them and the size of data you are working with.

Some small rules of thumb:

R processes (generally) tend towards being RAM bound, so you want may want to get a RAM optimised
R Studio IDE has a profiling tool you can use to check how your code executes and where the time is spent
It may be easier and cheaper to optimise code before scaling with tools like data.table and RCPP

ML models are likely not going to be effected by this, more your data prep, check the profiler

If you are REALLY keen on getting the model running fast, look into the GPU/TPU instances, however, in the case of R using caret, I don't know if it will utilise these assets. Do your research first as these are the most expensive flavour of EC2. TPU units are (from my limited research) specifically optimised to tensorflow ML.

A plan B/alternative would be to work in aws Sagemaker notebooks to start. Then you can abstract away all the faff of managing the EC2 and just focus on building the ML.

For extra credit on your EC2, use one of the R community AMI's put together by this lovely person: http://www.louisaslett.com/RStudio_AMI/


Mismatching target dimension in Keras LSTM I have built this network for text classification:

model = Sequential()
model.add(Embedding(vocabulary_dim, 150, input_length=max_length))
model.add(LSTM(150, return_sequences=False))
model.add(Dense(1, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

When I fit the model:

model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=5, batch_size=128)

I get this error:

ValueError: Error when checking model target: expected dense_9 to have shape (None, 1) but got array with shape (12481, 3)

I suppose because y_train and y_val are one-hot encoded, with shape:

array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       .............])

but I'm not sure about this.

With a single output unit model.add(Dense(**1**, activation='softmax')), Keras is expecting a binary outcome, and so the 'softmax' activation function and the 'categorical_crossentropy' loss are not appropriate.

If your output is binary, use: model.add(Dense(1, activation='sigmoid')) and model.compile(loss='binary_crossentropy'.

If your outcome is not binary, then the number of output units needs to be adjusted to match the number of output categories.


Multivariate linear regression in Python I'm looking for a Python package that implements multivariate linear regression.

(Terminological note: multivariate regression deals with the case where there are more than one dependent variables while multiple regression deals with the case where there is one dependent variable but more than one independent variables.)

You can still use sklearn.linear_model.LinearRegression.  Simply make the output y a matrix with as many columns as you have dependent variables.  If you want something non-linear, you can try different basis functions, use polynomial features, or use a different method for regression (like a NN).


Preparing data for a live prediction engine I need to build a live prediction engine (either using Ensemble methods/ RNN/ Keras Classifier) that could learn from historical data what kinds of users are likely to transact and, in real-time, generate transaction probabilities for current users on a DAILY basis. I do have their demographics, transactional and app activities data. There is an amount limit that can be transacted by the users in a month. How do I prepare the historical data to predict the next day if he is going to transact or not? Thanks.

Demographics:

CustomerID | CustomerAge | Gender | Nationality | Joining Date | Loyalty Program Opt-in | Address

1222 | 25 | M | British |  21-Apr-2016 | Yes | Old kent

Transactions:

Beneficiary | CustomerID| TransactionDateTime | TransactionAmount

4534534 | 545544 | 19/02/2016 16:02:56 | 1888.85

4353222 | 545544 | 23/02/2016 08:49:49 | 3242.43

2453453 | 535455 | 26/02/2016 10:10:56 | 4776.57

5353532 | 435353 | 11/02/2016 12:43:38 | 4862.16

4532522 | 345353 | 23/02/2016 17:42:22 | 4923.54

You'll have to do some feature engineering on your data. Beneficiary and Customer ID has no meaning for machine learning, so you first have to join all other data, taking care of date types (split hour, day of week, month) and should be ordereed by the ones you will not use for training (beneficiary, customerID). than you can create new features like for each transaction the time spent since the last one for each customer, count of transcactions by day/day of wee/month, min, max, median amout spent and son on. It'll be a binary classification (SVM, random forest, logistic regression, etc.).


Example data source for educaional use I'm doing project on subject of affinity analysis for my statistical class in college.

In order to complete it, I have to acquire sales database with at least 200-300 records, each containing list of products bought by single client.

Are there any example sales databases available for free to use for such educational purposes?

The description of the needed data is quite general, but I have found this dataset that describes sales during Black Friday; it is possible to identify products bought by the same customer using customer_id column.

Also, this dataset describes transactions from a bakery. The products that were bought in the same transaction (and so by the same customer) are identifiable as they share the same #Transaction value.

Generally below are 2 sources where someone can find free datasets that can be used for training/ educational purposes:

Kaggle: https://www.kaggle.com/datasets (both datasets that I suggested were found here)
EU Open Data Portal: https://data.europa.eu/euodp/en/data


Why does a simpler model performs better than a complicated one? This has happened to me, a complicated model couldn't solve the problem when a simpler one solved it in a few epochs. How is that? I believed that a more complicated model means more number of parameters and more number of parameters means a higher capability to solve a problem.
I have heard that people say, A simpler Model can perform well because it has a low variance. But, what does that even mean?

High variance means that your model's performance varies from data to data, which is bad for the model.
For example:
You used a polynomial classifier with high degree, so it will overfit your training data resulting in good accuracy, but when tried on a different dataset it will yield less accuracy , resulting in variance.

The image on the right is way too complex which results in overfitting.

The middle image shows a simpler model than the image on right, but fits the data better ie. generalize the data better


How to train neural word embeddings? So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then use them in supervised DNN. But recently I read a blog post which suggested that training the word embeddings while training the neural network gives better results. This is the other link.
So my question is which one should I follow?
Some YouTube videos that I referred:

Deep Learning for NLP without Magic Part 1, 2 and 3

There is one answer to know: try both methods and take the one that gives the best result. I would say that in general pre-trained embeddings usually gives better results. You can also start with pre-treained embeddings as initial conditions and let the embeddings train maybe with a smaller learning rate.

In any case, the current state of the art for text classification is ULMFIT (https://arxiv.org/abs/1801.06146), which actually doesn't do any of this. It pre-trains embedding and RNN with a language model in the wikipedia and in the target text and then fine tunes the whole model with the target text.


How Calculate Effect (percentage) label of the input variables on the output variable by DecisionTreeClassifier a description problem below.
I have 10 words like X1 , X2 , X3 , ... , X10

and three Label like short , long , hold.

My problem is that how calculate Effect (percentage) label of the input variables
by DecisionTreeClassifier Algorithm.
DT=DecisionTreeClassifier()
DT.fit(X_train, y_train)

and how calculate Effect (percentage) label of the input variables

I don't think there is any way of doing that with decision trees, because that's not how decision trees work: the predicted label is not the result of some linear combination of the features. Instead you can look at the actual decision tree that the model represents and see which features have been used to classify a particular instance.


How to prepare masks for multiclass semantic segmentation? It's very straightforward for binary semantic segmentation: black color (0s) is responsible for background, whereas white color (1s) is responsible for objects of interest.

But what about multiclass semantic segmentation? As far as I understand, these masks must be RGB images since we use more than two colors. Is it correct? Or should I have a separate binary mask for every class?

If I can use RGB images with multiple colors as masks, should I use some specific colors for masking? If not, should I specify colors I chose somewhere in a network as class parameters? Or will any CNN automatically detect any number of different colors in my masks?

These questions may seem naive and primitive, but I was unable to find any clear explanantions of thus aspect of multiclass semantic segmentation.

You should create a separate binary mask (1 for the pixlels belonging to that class and 0 for the rest of pixels) for each class. Therefore, your mask array should have a shape of (BATCH_SIZE, WIDTH, HEIGHT, NUM_CHANNELS), where NUM_CHANNELS is the number of class.


Why do most GAN (Generative Adversarial Network) implementations have symmetric discriminator and generator architectures? For example, if the discriminator is a vanilla network of n layers, each with n(i) units, then, typically, the generator will also be a vanilla network of n layers, each with n(n-i) units (except the output of the discriminator, where n(n) = 1 whereas for the generator n(0) = NOISE_SIZE).

If the discriminator is a CNN, the generator is typically a symmetric "deconvolution network" where the i-th layer is a transposed convolution layer, symmetric to the n-i-th layer of the discriminator.

Virtually all implementations I've seen follow this pattern, although I can't see anything in the theory why it would have to be the case. And yet, I had a simple vanilla implementation of a digit drawing GAN trained on MNIST, which worked reasonably well. I tried to improve the discriminator by making it a CNN (with the same architecture which I had working well on recognizing MNIST digits) without changing the generator. The GAN no longer worked and converged to a state where the generator always produced the same gibberish drawing. Intuitively, a better discriminator should help the GAN but it was not the case. It seems that improving the discriminator without equally improving the generator does not work (and vice-versa probably). Is this the reason why people always choose symmetric architectures? To preserve a "balance of skills" between the adversaries? Or is there a deeper reason?

The main reason is that you want the Generator and the Discriminator to be equally powerful. The base intuition of GANs is that the two Networks can improve themselves through competition. If one is way better than the other, they will get stuck into some unwanted equilibrium in which one beats the other all, or almost all the times.

For this reason, they are usually chosen symmetric. It's the simplest and most effective way to make sure their competition will remain balanced.


xgboost: give more importance to recent samples Is there a way to add more importance to points which are more recent when analyzing data with xgboost?

Just add weights based on your time labels to your xgb.DMatrix.
The following example is written in R but the same principle applies to xgboost on Python or Julia.

data <- data.frame(feature = rep(5, 5),
                   year = seq(2011, 2015),
                   target = c(1, 0, 1, 0, 0))
weightsData <- 1 + (data$year - max(data$year)) * 5 * 0.01

#Now create the xgboost matrix with your data and weights
xgbMatrix <- xgb.DMatrix(as.matrix(data$feature),
                         label = data$target,
                         weight = weightsData)


Regression for discrete values? Im a noob in ml / statistical algorithm, but I do have worked with simple classifiers and regression

I like some opinions if I am going the right way, given my limited knowledge

My problem is following: I wanna best predict price of a input item, given the known price of the most similar items in database.

This at first sounded like regression but my features are mixed discrete and contiguous, (like item model type (think car model, not ml model) is discrete, year of manufactury is contiguous?)

Now I am thinking finding x closest matches via euclidean distance, and then weighted average + somehow make sure the certain features have to match exactly (like item model type)

Thanks

You can still perform regression even if your input (or part of it) is discrete.
If you think of it, even your "continues" values are actually discrete (starting from their initial measurement accuracy/resolution).

You need to decide if you want to train a model to perform estimation, or just cast your input features to a vector space and use them with some euclidean distance (to make sure some features match exactly, just add some "if" check so you can calculate distances only to relevant samples).

To train a model, it should except as input all the features (preferably normalized), and output a single number which you can later round to the closest discrete value (if you like).


Predictive Analytics on distributed systems vs standalone system I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance knowledge on this subject can enlighten me. Here is where I am getting confused:

How can a statistical model (such as linear regression model) built on Hadoop where data is distributed across many nodes, and where work is done only on subset of original data-set residing in each node have same or better accuracy than linear regression model built on single server where all data resides locally. Would the error-rate (accuracy) of a model build on hadoop be same, better or worse than model build on local system. I was thinking that since all data reside in local system, algorithm can be tuned to optimize error in a linear regression model. However, in case of Hadoop where data is distributed, what if only local optimization lead to drastically bad algorithm overall? Are the algorithms written separately for distributed system to take into account data distribution?

There is nothing special about building something across many machines, or Hadoop, so maybe we can take that out of the question.

One general reason is that, even if the model building process were worse, building on more data could result in a better result.

You're asking why building many models on subsets of data could be better, but this is precisely why ensemble models like random forests can outperform single instances.

But lastly, it's not true that distributed algorithms always work by making an ensemble of small models, so the premise is not strictly true.


What exactly does generator produce in DCGANs? Does it generate the set of the same image classes in the same order on each iteration? If yes, what's the usufullness of that ?

Does it generate the set of the same image classes in the same order on each iteration?

No, a basic generator in a GAN is typically fed a small random vector as input; e.g. a column of 100 gaussian samples, with mean 0, standard deviation 1. It then must use this random "embedding" in the feed forward network to produce output that is in the target distribution of real data that the discriminator is trying to assess it against.

So a GAN typically learns a mapping from an arbitrary relatively low dimensional space onto a higher-dimensional target space. DCGANs do this with image data.

The class of an image can be used as conditioning data, if it is fed as additional input to the the generator and discriminator. The sequence of image classes can be anything you like, and the GAN is not required to work in a particular sequence (unless perhaps you are working with a RNN-based GAN and are trying to learn to produce sequences).

  What exactly does generator produce in DCGANs?

It produces an output image depending on the input. The output is drawn from a distribution based on the training data, that the discriminator would struggle to distinguish from real data. If fed a random input vector, generated in same way as training stages, then the idealised output should appear to have been drawn equally randomly from the imagined population of all possible training data.


Can KMeans clustering be used on word2vec output? I have a dataset that has been trained on word2vec. Is it a good idea to cluster the output vectors?.

It's totally fine to cluster word2vec output to know semantically similar words. KMeans is an option, you might also want to checkout some approximate neighbor scheme such as Locality Sensitive Hashing.


Xgboost performs significantly worse than Random Forest I have a dataset of 3500 observations x 70 features which is my training set and I also have a dataset of 600 observations x 70 features which is the test set.

The target is to classify observations correctly either as 0 or 1. 2000 observations of the training set are 0 and the rest 1600 of them are 1.

I aim at the highest possible recall for precision>=90%.

I did grid search for ensemble algorithms only in relation to number of trees (from 50 to 650 trees). Analytically the best recall results for precision >= 90% for each of the algorithms are the following:

Random Forest (375 trees)

from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(random_state=0, n_estimators=375, class_weight='balanced')
classifier.fit(X_train, y_train)

Precision: 90%
Recall: 24%

Xgboost (550 trees)

from xgboost import XGBClassifier
classifier = XGBClassifier(n_estimators=n_trees, seed=0, scale_pos_weight=1.5)
classifier.fit(X_train, y_train, eval_metric='map')

Precision: 90%
Recall: 15%

Why Xgboost is performing so much worse than the Random Forest?

Different algorithms need to be tuned in different ways. For example, one important parameter in boosting is ETA or learning rate. This determines the change in weights after each boosting step. This parameter is important to reduce overfitting in boosting.

To understand the different parameters you can refer to this.

Another important thing to look at is the objective function. This will determine what your algorithm is trying to optimize.

Lastly, please make sure you are selecting the model that actually maximises the metric that you care about. You can do this by selecting an appropriate eval_metric. In this case, you probably want to use something like F1-Score or precision at a fixed recall.


K-Fold cross validation and data leakage I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the remaining k-1 folds as training set. Now I want to do feature scaling and data imputation on that training set and then apply the same transformation on that validation set. I want to do this for each step. I am trying to avoid data leakage as much as possible and at the same time rescale my validation sets to get better results.
How can I do this with a few lines of code?
Secondly, is it necessary to do this? Because I don't see many people do this for k-fold validation. I have seen many times, they do feature scaling and imputation on the entire dataset first and then do the k-fold cross validation. But doesn't this cause data leakage?

A reproducible example with no data leakage:
In there I'm scaling the data only with the train data on the k-fold stage
import numpy as np

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

X, y = load_iris(return_X_y= True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 42)

model = Pipeline([("imputing", SimpleImputer(strategy= "mean")),("scaling", StandardScaler()), ("modeling", LogisticRegression(random_state= 42, class_weight= "balanced"))]).fit(X_train, y_train)

cv_scores = cross_val_score(estimator= model, X = X_train, y = y_train, scoring= "accuracy")

print(f"Mean accuracy cv: {np.mean(cv_scores)}")

model.score(X_test, y_test)

Note that in this case it is easy to apply all the pipeline because all the feature are the same data type, but imagine you have categorical and continuous features, so you need to apply different preprocessing and imputing to each.
In that case a combination of ColumnTransformer and Pipeline would do the job.
For reference check:
https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html


Why my neural network does not predict decimal values in range [-1,1]? When it is able to predict the integer values I am trying to perform a very simple experiment, predict the input number. The concept is same as an auto-encoder. But with just one layer, which can handle the task of encoding and decoding-

Also, wanted to observe how the network learns, with different training examples.

Initially, I took 10000 training samples, of integers, in range[1, 10000]. So, example if I pass 767676 as one of the test sample it should predict the output as 767676.0
This test passed very well.
The activation function used here was 'softplus', which keeps the values in range [0, inf).
the network -

train_d = numpy.array([i for i in range(10000)], dtype=np.int)

model = Sequential()
model.add(Dense(1, input_shape=(1, ), activation='softplus'))

model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)

Now when I gave it 10000 training sample of decimal values in the range[-1, 1].  E.g. 0.3456 the expected output will be 0.3456 but rather, its giving me 0.5721.
The network -

train_d = numpy.array[round(random.uniform(-1, 1), 4) for i in range(10000)], dtype=np.float)

model = Sequential()
model.add(Dense(1, input_shape=(1,), activation='softsign'))
model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)

First of all, during the 10000 integer test, did you use all the integers from 0 to 9999 in training? If yes, then you have fully covered the whole input range. This means that while testing, you actually feed the network with data that are identical to the training data, therefore the accuracy is very high. What is the result of the network if you train it with 10000 randomly sampled integers in range of 10,000 and then test it with ALL integers in range of 10000? This test will reveal if your encoder generalizes well enough.

Also keep in mind that the smaller the number, the more difficult it is for the network to train due to vanishing gradient. Therefore, decimal numbers can saturate the learning process and lead to lower accuracy. Try changing the batch size (use smaller batches) and see if you decrease your output error.


Machine Learning to predict risk of items I'm trying to find out what I need to research and start learning to try and apply machine learning to this problem:

  In multiple offices I have 20 chairs, all of these chairs will need to have a risk assessment carried out, as the chairs become older I want to understand when they become more at risk because of other factors such as; age, usage, last time assessed and areas level of risk. I then want to be able to come up with a simple risk score of each individual chair.

I have done some research and I think it points towards using a machine learning technique such as supervised learning with a neural network but I'm uncertain. Given my scenario what should I be looking to start learning and doing?

Furthermore I believe that from my limited knowledge that providing all of the values I'm using are the same across other items I could apply and scale out this model for other items?

Further info;

As I'm familliar with Javascript I'm choosing to use synaptics.js to create a neural network and I'm going to be generating sample data inside a node server. I'm going to be using the following for my datasets that will be used as my model for learning from:

  Age (days to years), Location(low, med, high), Office(low, med, high), Last Risk Assessment Date (days-> years capped by twenty), Daily use (Low, Medium, High) and quality(low, med, high).

All of the above will be given an expected outcome manually so that I can attempt to figure out a way for the neural network to process the above and come to the same conclusion I did.

With the above labels I will be adding a weighting that will then be used to classify the risk level it should be associated to.

The factors I'm looking for the neural network to understand is that given my factors a chair with a lot of high factors will likely be placed in a higher outcome value of say 1-4 for risk factors. I'm then hoping I can use this learning to be applied to multiple other things other than chairs and increase the labels the network can use to learn from as I go along.

This sounds like a homework problem and those are frowned upon on this web site; so I'm going to be cautious about helping you and just give you general guidance:

1 - be very, very careful about picking models before you do anything else. That is a bit of Researcher Bias and you should try to avoid that; let the data tell you where to go, not the other way around.

2 - you have to think about what "factors" you can use to describe each chair. How old is it? What color is it? Does it have wheels or not? Some of these features will have an effect on your models, others will not. But it should be your goal to start with a long list of factors and then try to model them out and see which ones have an effect on your results.

Start putting a list of factors together and then you will be in a better position to continue your studies.


How to compute document similarities in case of source codes? I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token based document representation (TF-IDF or embeddings) and computing similarity on these but somehow I find this approach a bit clumsy. I want to detect "handprints" (characteristic comment and abbreviation style, folder structure, used 3rd party tools, order of elements in the code etc.) that seem out of the scope of this approach.

Moreover, I cannot find place for any proper machine learning here. Clearly, finding weights for the any future quantitative features would be nice but this similarity task is not classification/regression, so how to define the target? Clustering seems to be a better tool but we cannot define as many categories as potential authors.

Could you kindly suggest any more reliable method? Does any literature exist for this topic?

Why not use all non-code indicators as 'handprints?' For example, many IDE's will add specific comments to documents when they are started. Also, and this is one that would be easy to detect my work, I have a tendency in python to copy entire import blocks from one script to another, whether or not I actually need all the imports or not. If someone uses a copied import or include statement, then their imports will all be in the same order. If you track the order of imports, you may be able to find patterns.

There are also issues of whether you use spaces or tabs for indentation, preferred number of spaces or tabs, etc. For languages with braces, you can see if braces are used following an if statement, or at the start of the next line. I believe this will give you enough 'handprints' to identify people distinctly.

Once you do this, you can use multilevel clustering to attempt to assign documents first to a group of possible originators (i.e. all originators who use a certain IDE or text editor). Then you can go through and look for certain patterns within each group, clustering again within each cluster.


Similarity score: Can Sklearn SVR predict values greater than 1 and less than 0? I am using svm.SVR() from scikit-learn to apply Logistic Regression on my training data to solve a similarity problem. Using GridSearchCV, I am finding the best hyper parameters using the scoring as "R2". The best hyperparameters are C=1, cache_size=200, coef0=0.0, degree=3, epsilon=0.001, gamma=0.005, kernel='rbf', shrinking=True, tol=0.001.

I fit the training data and training label as model.fit(X_train, Y_train)

Now I use test data in the same model as: prediction = model.predict(X_test)

The reason I am using SVM-Regression is to find similarity between two inputs. However, for some of the test data, the prediction contains value in negative (less than 0) and for all the same vs same comparison it returns the value as 1.09469178. I am expecting the value to be between 0 and 1. Is this normal or am I doing something wrong?

I am using svm.SVR() from scikit-learn to apply Logistic Regression on my training data to solve similarity problem.

Wait a second, if you're using support-vector regression, then you're not using logistic regression.  These two are very different algorithms.  They aren't even applicable to the same type of problem.  Support-vector regression is used when you are predicting a continuous target, whereas logistic regression (despite the name) is a classification algorithm.

  However, for some of the test data, the prediction contains value in negative (less than 0) and for all the same vs same comparison it returns the value as 1.09469178.

There's nothing unusual about this if you're using support-vector regression.  A support-vector machine can (in theory) output any real number.

Logistic regression, on the other hand, is a sigmoid function.  It will take as input any real number and output a result between 0 and 1.  Maybe you meant to use scikit's LogisticRegression model rather than SVR?

  The reason I am using SVM-Regression is to find similarity between two inputs.

Could you expound on your use-case a bit more?  Support-vector regression isn't really meant to be used to compute similarity.

I'm speculating that you have a training set of (X, y) pairs (where y is a label between 0 and 1).  You are training a model to output $\hat{y}$, a prediction of y.  To find the similarity of two inputs, $X_i$ and $X_j$, you pass them both through the model and measure the difference in the $\hat{y}_i$ and $\hat{y}_j$.  Is that right?

If so, I think this is a really roundabout way of computing similarity, and is unlikely to yield better results than a more straightforward method.  I'm not sure that you need a machine learning algorithm at all.

Have you looked into other similarity metrics that might be suitable for your problem?  For similarity based on Euclidean distance, you can compute $\frac{1}{1 + d(X_i, X_j)}$ (where $d$ is the euclidean distance function).  Cosine similarity might be a good choice if you care more about the similarity in direction of two input vectors.


Preparing dataframe to carry k-means clustering Im trying to apply 3 different algorithms of clustering on my dataset.
to check which one fits the best. I'm confused how should I convert my dataframe

-k-means
-DBSCAN
-hierarchical clustering

Dataset (bank.csv)
https://archive.ics.uci.edu/ml/datasets/Bank+Marketing
(bank.csv with 10% of the examples and 17 inputs, randomly selected from
3 (older version of this dataset with less inputs))

https://archive.ics.uci.edu/ml/machine-learning-databases/00222/
(direct download)

'data.frame':   4521 obs. of  17 variables:
 $ age      : int  30 33 35 30 59 35 36 39 41 43 ...
 $ job      : chr  "unemployed" "services" "management" "management" ...
 $ marital  : chr  "married" "married" "single" "married" ...
 $ education: chr  "primary" "secondary" "tertiary" "tertiary" ...
 $ default  : chr  "no" "no" "no" "no" ...
 $ balance  : int  1787 4789 1350 1476 0 747 307 147 221 -88 ...
 $ housing  : chr  "no" "yes" "yes" "yes" ...
 $ loan     : chr  "no" "yes" "no" "yes" ...
 $ contact  : chr  "cellular" "cellular" "cellular" "unknown" ...
 $ day      : int  19 11 16 3 5 23 14 6 14 17 ...
 $ month    : chr  "oct" "may" "apr" "jun" ...
 $ duration : int  79 220 185 199 226 141 341 151 57 313 ...
 $ campaign : int  1 1 1 4 1 2 1 2 2 1 ...
 $ pdays    : int  -1 339 330 -1 -1 176 330 -1 -1 147 ...
 $ previous : int  0 4 1 0 0 3 2 0 0 2 ...
 $ poutcome : chr  "unknown" "failure" "failure" "unknown" ...
 $ y        : chr  "no" "no" "no" "no" ...

I know that my data for clustering should be converted to integers. So i converted it like this (badly):

#Import data to R

bank <- read.table(file="bank.csv", sep=";", header=TRUE,stringsAsFactors = FALSE)

# character data into numeric format
bank$job <- as.numeric(as.factor(bank$job))
bank$marital <- as.numeric(as.factor(bank$marital))
bank$education <- as.numeric(as.factor(bank$education))
bank$default<- ifelse(bank$default == "yes", 1, 0)
bank$housing <- ifelse(bank$housing== "yes", 1, 0)
bank$loan<- ifelse(bank$loan== "yes", 1, 0)
bank$month <- as.numeric(as.factor(bank$month))
bank$contact <- as.numeric(as.factor(bank$contact))
bank$poutcome <- as.numeric(as.factor(bank$poutcome))
bank$y <- ifelse(bank$y== "yes", 1, 0)

# create normalization function
normalize <- function(x) {
  return ((x - min(x)) / (max(x) - min(x)))
}
#normalize the data to get rid of outliers if present in the data set
bank <- as.data.frame(lapply(bank, normalize))

No, it does not make sense to encore the data this way.

You use Euclidean distance.

You need to encode the variables in a way that Euclidean distance computes a similarity that is useful for your problem. It strictly is not enough to encode stuff somehow as numbers! Garbage in, garbage out is what you get this way.