Can cosine similarity be applied to multidimensional matrices? I'm trying to find the similarity between two 4D matrices. Because cosine similarity takes the dot product of the input matrices, the result is inevitably a matrix. Is there a way to get a scalar value instead? Could inner product used instead of dot product? That is, is

cossim(A,B) = inner(A,B) / (norm(A) * norm(B))

valid? Or is there a better way to find the similarity between multidimensional matrices?

Don't hack. Do the math instead.

You could reshape your matrix into a vector, then use cosine.

But whether that is sensible to do: ask yourself.

You could also ignore the matrix and always return 0. That is a proper similarity, too. Just usually not useful.

Don't just use some function because you heard the name. You need to understand it's properties, and you need to check whether these properties are helpful for your problem, or not. Choose one because math says this is solving your problem; don't replace the math with random guessing.

Cosine similarity link 1
Cosine similarity link 2
Cosine similarity link 3


KNN with mixed data (feature set) I have a dataset where the feature set consists of hour of the day (between 0 to 23), day of the week (Monday to Sunday), number of shops (a positive integer) and road category (0 to 8 on an ordinal scale). I wish to predict the pedestrian volume which is a continuous variable. Hence I was looking a regression approaches and was thinking along the lines of using KNN. Will it be appropriate for this task? I am especially worried about the hour and day data as they are cyclic in nature.

df_test[['Day', 'Time','shop_count','road_category','pedestrian_volume']].iloc[0:5]

           Day  Time        shop_count       road_category  pedestrian_volume
  0    Tuesday    20                22                   0             210.0
  1    Tuesday    21                13                   4             196.0
  2    Tuesday    22                39                   4             214.0
  3    Tuesday    23                 2                   8              46.0
  4  Wednesday     0                15                   8              18.0

KNN is a classification algorithm where, typically, continuous variables are used to apply a classification. Your problem is a problem of predicting a continuous value using variables that are not continuous.

Your problem presents several challenges which, to me, would suggest KNN isn’t a good choice.

The biggest problem is probably the road_category where you have an ordinal value. Is category 4 four times greater than the difference between category 0 and category 1? Probably not.
No matter how deep your data, due to the fact that your predictors are integers or categorical, your neighbors won’t really be neighbors, more like roommates in a warehouse; you will probably have hundreds of observations on the same vector. So, now you have to choose k values from m observations. So, you will need to come up with an algorithm to do that. Then you’ll have to come up with a methodology to get a prediction. An average might work, but depending on your application, Min or max may be more beneficial.

Bottom line, KNN expects the variables to be close to the same order of magnitude otherwise, larger variables tend to determine the neighbors even if they are not necessarily more important. Some limited categorical variables might be accommodated using dummy variables, but I’d be really careful.

Based on your description of the problem, I think you should continue with regression, but use dummy variable encoding for your categorical values.


Finding sequences in two arrays Imagine you have two arrays.
arrayA = [1,2,4,1,3,9,8]
arrayB = [4,3,2,4,1,3,9,8,9]

and there's a sequence of numbers inside of them, if you look at arrayA then you can see that it has "2,4,1" and arrayB has "2,4,1" as well (it also has "3, 9, 8"). I need to write a small script in Python that can find the starting position of the sequence and the length of the matching sequence. Does anyone have any idea how I can do that? The result should look like this:
[{start_position: 1, size: 3}, {start_position: 4, size: 3}]

This should work
def findUntilNotSame(array1,array2):
    resultPos = 0
    for i in range (len(array1)):
        if(i < len(array2)):
            if(array1[i] != array2[i]):
                break
            resultPos += 1
    return resultPos

# find the length of same value
def findStartSame(array1,array2):
    for i in range(len(array1)):
        if array1[i] in array2:
            return [i,array2.index(array1[i])]

# find the solution one time
def findStartAndLength(array1,array2):

    start = findStartSame(array1,array2)
    start1 = start[0]
    start2 = start[1]
    length = findUntilNotSame(array1[start1:],array2[start2:])
    return [[start1,length],[start2,length]]

def updateAnswer(finalOrigin):
    finalResult=finalOrigin
    for i in range (len(finalResult)):
        if i != 0:
            finalResult[i][0][0] += finalResult[i-1][0][0] + finalResult[i-1][0][1]
            finalResult[i][1][0] += finalResult[i-1][1][0] + finalResult[i-1][1][1]
    return finalResult

def algorithm(array1,array2):
    i, j = 0, 0
    finalResult = []
    while(i<len(array1) or j<len(array2)):
        array1now = array1[i:]
        array2now = array2[j:]
        if(any(item in array1now for item in array2now)):
            result = findStartAndLength(array1now,array2now)
            finalResult.append(result)
            nextStart1 = result[0][0] + result[0][1]
            i += nextStart1
            nextStart2 = result[1][0] + result[1][1]
            j += nextStart2
        else:
            i += 1
            j += 1
    return updateAnswer(finalResult)


CNNs - Hyperparameter tuning with different training sizes of the same data set I would like to compare how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set.
For this I would like to use a data set like MNIST or Fashion MNIST.

I would like to start by first training a CNN with only a subset ob about 1000 images, then a CNN with a subset ob about 5000 images, then with a subset of about 10000 images etc.

For each ot these data sets an optimized CNN should be generated.

My current plan would be the following:

Create a simple base model with only 1 Conv Layer (with 32 filter), MaxPooling2D, Dropout (0.5) and a Dense Layer with about 100 Neurons.
Using this base model, I would search for the smallest dataset (with 1000 images) using GridSearchCV for batch size and optimizer und reuse both for all other datasets.
Then I would do the following for each data set:

Search for the optimal architecture of the network (Number of Conv
layer, Number of filter, Number of dense layer, Number of Neurons in Dense layer) using GridSearchCV (RandomizedSearchCV will propably be less suitable at this point)
Search for learning rate, weight decay, dropout rate etc. (using GridSearch or RandomizedSearch).

Maybe I could just use the default setting for one or the other not so important parameter and  exclude them from the search.

I am not sure if my approach is really recommendable.

I also tend to use RandomizedSearchCV to determine all hyperparameters simultaneously for each data set.

Many thanks in advance for your comments and suggestions.

First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it.

Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can easily take hundreds of thousands or millions of trainable parameters. Running cross validation on those could be computationally impossible.

In other words, keep the two problems separated.

I suggest you to take a look at common CNN architectures. There's plenty of them on Kaggle competitions (for MNIST, Fashion MNIST or any other dataset). You can choose one that works fine, and then use it to show how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set.


Hand-crafted decision tree inspired from learned decision tree Goal of this question:
As I am the only 'machine learning guy' in our group, I wanted to get an outsiders view, that is a sanity check if what I am doing adheres at least to 'decent practices' in machine learning (I know its not best practices :) ).

Problem setup:
I am working on a classification task on biomedical signals (detection of hypertension from physiological signals other than blood pressure). Since I do not have too much high quality labeled data for training a powerful classifier (say a larger conv-net), currently my procedure is as follows:

Feature engineering (manually engineered features, mainly driven by (physiology) domain insights.
Train supervised learning classifiers, in particular tree based algorithms such as random forests and simple decision trees.

Now, since currently I do not have enough high quality labels and I need to ship some classifier soon (in addition as it is a medical application I really care about stability and to some degree interpretability), I thought I could go for a manually built expert system, that is some rule-based system (if-else) using the most relevant features according to the learned random forest or decision tree classifier. For the cutoffs on the features I can use the values learned by the single decision tree.
In addition, I would start from a learned decision tree and adapt it (as some learned splits are totally non-sense, i.e., we are in the overfitting regime).

Question:
Is this procedure fine as long as I test my (hand-crafted and machine learning inspired) classifier on unseen data? I'd be happy to hear your experiences in similar situations!

Thanks

I did this once, in a position similar to yours.

My constraints were,

People skeptical of machine learning
Required interpretability
Required very fast inference
Explainable in text documentation
Implementable in any language

Traditionally, this would have been done by an expert system, so training a decision tree was essentially just one step further.

On my end, all of the work was being done in Python. I tried a handful of ML models, with and without hand-crafted features. A single decision tree performed nearly as well as anything I tried, and outperformed a fully handcrafted expert system. I then created an "inference model" in the form of if-statements for each split in the tree to confirm performance was equal to the decision tree itself. I even manually ignored some low entropy splits or where a single feature was underutilized, finding that for the same number of branches, a larger trained tree manually retaining fewer nodes outperformed a small tree where I replicated it exactly.

Some of the splits were unintuitive, but that's partially the point of throwing a model at it. Be sure to really dig into the failure modes of every alternative to help everyone involved have confidence in your work.

Given the constraints, it was a great solution to the problem.


Data Quality Assessment in Image Data I'm working on a CNN model that classifies images. After scraping image files from the Internet, I found that many of them didn't look this way as described by the searching keyword (for example keyword='dog' but image not containing a dog).
So, I cleaned my dataset manually which was quite labor intensive and time consuming.

Was my approach right or is there any tools or methods by which image data can be cleaned? Actually this sounds quite controversial because this cleaning tool should do the job for what the model are being built - classify the images.

I am building on the first part of @Dylan's answer:

For general items like "dogs" pre-trained models are easily available. A good starting point is ImageNet. There are plenty of pre-trained models available for this dataset, e.g. see here for PyTorch. Since ImageNet includes multiple categories for a given item you can check this list to see which indexes correspond to which items and define a mapping (e.g. dogs are classes 151 to 268).

Once you have done that, manually check the cases for which the labels and the transfer model disagree.


How to count observations per ID in R? I have a large amount of Data where I have to count meassurments per one ID. What I already did was creating a Data Frame over all Files and I omited the NAs. This part works properly. I was wondering if the nrow-function is the right function to solve this but I figured out that this will not lead me to the target as it returns a single number as output.

What I am looking for is if you have entries like that:

1155 2010-05-02  2.7200    1
1156 2010-05-05  2.6000    3
1157 2010-05-08  2.6700    1
1158 2010-05-11  3.5700    2

That I get a list:

ID          Number of observations
1           2
2           1
3           1


Pecularities of classification of Hidden Markov Models? Assume that we have set of Hidden Markov Models (Bayesian networks) Set{(n, m, P, A, B)} (n - number of hidden states, m - number of observable states, P - initial probabilities, A - transition matrix, B - emission matrix). Are the peculiar algorithms or approaches how to classify those HMMs into different subsets of HMMs? If n and m are equal for all the HMMs in the set then classification can be build from the ranges of elements in A and B. If n and m are different for different HMMs then one classify hidden and observable states first...

Are there some research or trends for this? Google is not helping here because it returns results for classification methods using HMMs, but we want to classify (or find clusters) of HMMs themselves.

What you are looking for is called a metric (or distance, or similarity measure) for HMMs (some people would say SVM kernel for HMMs).

Somehow, if you have a distance, you can cluster the HMMs.

The most classical distance, defined by Rabiner and Juang is a Monte-Carlo approximation of a Kullback-Leibler divergence for HMM. It requires a long observation sequence $O$ generated from the first HMM $\lambda_1$ and is then computed as:
$$ D_{KL}(\lambda_1||\lambda_2) = 1/T * (log(P(O|\lambda_1)-log(P(O|\lambda_2))$$

The advantage of this is that it does not imply any constraint on the number of hidden state. However, $O$ needs to be a long sequence if you want accurate results.
There is some research on the topic but not too much, and even less for HMMs with different topology (which is another reason why Google does not give much results...)

Edit: This paper proposes a distance measure that does not require the number of hidden states to be the same but is only valid for uni-dimensional observations.
A new distance measure for hidden Markov models by Zeng, Duan, and Wu in Experts Systems with Applications 37, p. 1550-1555, 2010. It is rather easy to implement.


How to prevent vanishing gradient or exploding gradient? Whats causing the vanishing gradient or exploding gradient to occur, and what are the measures to be taken to prevent it?

Vanishing gradient and exploding gradient are two common effects associated to training deep neural networks and their impact is usually stronger the deeper the network.

As you know, two fundamental operations when training neural networks are Forward-propagation and Back-propagation. When we carry out Back-propagation, that is, moving backward in the Network to calculate gradients of the loss function with respect to the weights, gradient values tend to decrease or increase dramatically, the further we get back in the network. This happens in cases when we have activation functions like Sigmoid, or TanH, whose non-linear regions below 0 (i.e. x << -5) and over 0 (i.e. x >> 5) return gradient values in the saturation regions. This means  (x << -5) that neurons in earlier layers will learn in a very slow pace as compared to those layers located later in the network (vanishing gradients problem). Exploding gradients are the other side of the coin, i.e. when activation functions are saturated (with x >> 5), gradient values tend to increase dramatically, making updates to weights unstable, thus not being able to converge.

Some possible techniques to try to prevent these problems are, in order of relevance:

Use ReLu - like activation functions: ReLu activation functions keep linearity for regions where sigmoid and TanH are saturated, thus responding better to gradient vanishing / exploding. You can also use different types like Leaky-ReLu, Randomized ReLu, etc.
Use Batch Normalization (BN): this is another solution you could use in order to make your network more robust against gradient vanishing / exploding, especially if you are using sigmoid or TanH as activation functions. Actually, BN gives you more flexibility during the selection of the activation function for your network. The obtained architecture gets more robust at training, given that it is less prone to diverging due to initialization values or from higher learning rates.
Reduce learning rate: if you increase your learning rate without considering using a ReLu-like activation function and/or not using BN, your network can diverge during training much more easily. By reducing your learning rate you can reduce the chance of suffering vanishing / exploding gradients problem, but your network will take longer to learn. That is why the first two options are located first in the list.
Change your architecture: If you are using Convolutional Neural Networks, for example, and you are suffering from vanishing / exploding gradients, it might make sense to move to a new architecture like ResNETs. In comparison to other networks, these structures connect different layers between each other, i.e. the so-called skip connections, acting as gradient highways, allowing the gradient to flow between the different layers unhindered.
Use proper weight initialization: you could use, for example, Xavier initialization Xavier et al. to reduce the chance of suffering vanishing / exploding gradients. By itself this option does not guarantee you will resolve these issues, but it makes your network more robust when combined with other methods.
Gradient clipping: this can be used when having exploding gradient problem. Firsthand, we select a threshold value, and in case the value returned by the function of a gradient is greater than this threshold, we set it to a different value. You can check more info here.


How to make data predictions As a total beginner I am trying to apply some "predictions" on top of a bunch of csv files which contains house transactions for the last 20 years divided per area. What I would like to predict is the trend of the transactions for lets say the next year for a specific area.

What general steps would you follow, to analyse those data and then predict?

I read different articles but what I am looking for it is a sort of "best general practice" for this sort of problem.

Regression will work well if your data set is large, but only for predicting current house prices (say, for example, estimating the value of your house). That's what people generally mean when they talk about predicting house prices from current house sales data.

The question of how house prices will behave in the next year is much, but much more complicated, and would not depend simply on the data you currently have. You would need to involve other information and a much more complex model, which would need to involve things like current level of household debt, inflation rate, economic outlook, etc. Daunting.

Generally, speculative prices follow some stochastic process. They depend on the current value, but diverge more and more the farther you go in the future.


How can I sort a data frame by groups? I am using R. I have a data frame with "year , price, mileage" columns. I want to group the df by year first and then sort each group by mileage. How can I do this?

The 'dplyr' package in R is ideal for these types of data manipulation tasks. The arrange function for example can group a dataframe by a certain column, and then sort by another column. For example:

arrange(df, desc(mileage), group_by = year)

See arrange for documentation on the arrange function, and dplyr for the dplyr package description.


Mean encoding With KFold regularization I just learned that regularizing the mean encoding reduce the leakage hence generalize better than mean encoding without it but I made 2 submissions with XGB in predict future sales competition on Kaggle  with the naive mean encoding method and got RMSE = 1.152 and with 5 folds validation and got RMSE = 1.154 which was a surprise for my. Can any one explain why this may happen ? also after making the kfolds regularization every item_id has multiple mean encoding one for each fold so should I leave the train set in this format or take their mean ? also when applying this to the test should I take the mean and apply it or what ?

I'm encoding the item_id with the target which is the number of sold items :

Naive method :

all_data['item_target_enc'] = all_data.groupby('item_id') ['item_cnt_month'].transform('mean')

Regularization :

kf = KFold(n_splits = 5 , shuffle = False)

for train_ind, val_ind in kf.split(all_data) :

train_data , val_data = all_data.iloc[train_ind], all_data.iloc[val_ind]

all_data.loc[all_data.index[val_ind] , 'item_target_enc'] = val_data['item_id'].map(train_data.groupby('item_id')['item_cnt_month'].mean())

all_data['item_target_enc'].fillna(all_data.item_cnt_month.mean(), inplace=True) # fill na that happened from values in validation not in the train with the global mean

Here is the encoding result with regularization for each item_id i got an array which was confusing for my to take the mean or not :

Try this.
item_target_enc= df.groupby(['item_id'])['item_cnt_month'].mean().to_dict()

df['item_id'] =  df['item_id'].map(item_target_enc)


Including the validation file in the training process after tunning Should I include the validation file in the training process after finishing the tuning process (e.g. searching for params using the validation file)?

It depends on the distribution of the train, valid and holdout/test set.

There are a couple of possibilities (basically permutations). In general any different distribution=covariate shift is bad and you should repair it. If this is the case, including valid is the least of your problems (but you should include it in this case to make corrections) and you should worry about covarite shift.

If distributions are the same between the sets, it wont make any negative difference and it could only help if you add the valid-hyperparam tuning dataset to the train.


Identifying baseline consumption I have data of intraday electricity consumptions (by half hours - 48 a day) over a year of 4000 households. Task is to establish baseline consumption of each of these households - possibly also differentiated on seasonality.

One way how to do this would be just taking the mean of the consumption signals. What would be more sophisticated method for this?

I would be very grateful for pointing out to methods I could look into.

I don't think there is need for very complex methods here, I would take a look at several granularity levels and take means for these levels. One important thing is if all the households are in the same area/timezone. After establishing this I would first take the general mean and see per household how they compare to this, then do this for every halfhour segment of the day to get rid of time-of-day seasonality, do this for day of the week, month of the year etc. It really depends on what you want to know what to do with this data.


Reading a wide dataset in R I originally had a wide CSV dataset of about 18000 columns(and about 80 rows) that I am trying to read in R. It was stored in an Excel sheet,which unfortunately has a limit of only 16384 columns. Hence, taking the dimension I obtain:

> dim(train_set)
[1]    83 16384

i.e 1000+ columns are getting eaten up ,and this would badly affect the accuracy of the predictions. How can I read all the columns in R?

Your suggestions are much appreciated. Thanks a lot!

Refer to my comment, I believe what you need is

df <- read.csv("./yourpath/yourfile", sep = ";", header = TRUE) # play around with the arguments per your file.


Exporting from matplotlib I'm finding a quicker method to export plots

import numpy as np
from matplotlib.backends.backend_pdf import PdfPages
import matplotlib.pyplot as plt

X = ... #define X
with PdfPages('MyFile.pdf') as export_pdf:
   plt.figure()
   plt.plot(np.arange(0,60+1), X)
   plt.xlabel('t')
   plt.ylabel('x(t)')
   plt.title('title')
   export_pdf.savefig()
   plt.show()
   plt.close()

I'm trying to find simpler code than the one I wrote attached. Any advice?

import matplotlib.pyplot as plt
import numpy as np
file = plt.figure()
plt.plot(np.arange(0,60+1), X)
plt.xlabel('t')
plt.ylabel('x(t)')
plt.title('title')
plt.show()
file.savefig("MyFile.pdf")


Random Forest vs. RainForest I have studied Random Forest and RainForest papers, but they are a bit confusing! In summary, I understand following steps for these algorithms. Could you help me to find out if I am right or not?

I appreciate your help.

In Random forest first:

define number of trees
partition data by bootstrapping
on each partition construct trees (in each node a sub sample of features is selected)
label leaf nodes
for classifying a  new instance vote over all trees.

In RainForest:

Partition dataset
Build AVC-set of a partition
Build tree over the partition by computing a purity criterion (like gini-index) over AVC-sets

Random forest is a learning algorithm. It is an ensemble learning algorithm that uses decision trees as base learners. You wrote the steps for it correctly.

Rain forest is not a learning algorithm. It is an algorithm of constructing a decision tree (how to do splitting) when the dataset is so large that it does not fit the memory. In rain forest, the whole dataset is not required for making a splitting decision. Only some aggregated information (AVC-set for an attribute or AVC-group if you have more memory) is required.

If your dataset is large, and memory is small, you can use rain forest to build several different decision trees. Then use random forest algorithm with those trees as base learners.


How do you calculate the number coefficients in polynomial regression? So I can't seem to find much on this by searching so I came here. Let's say I  had 3 variables $x_1,x_2,x_3$ and the let's say the degree of the polynomial was $d=2$, I can define the length of a vector of variables as $l$ such that $l=3$ in this case. What I'm looking for is the length of a vector of coefficients that would be needed to create a polynomial regression on these variables.
Would I be right in saying that the length of the vectors of coefficients needed for polynomial regression would be $d\times l$?

Yes - it is $d\times l$. There would be 6 $β$ coefficients for 2*3:
$β_1x_1 + β_2x_1^2 + β_3x_2 + β_4x_2^2 + β_5x_3 + β_6x_3^2$
That does not include an intercept or any interaction terms.


Considerations for multiple hidden layers in neural networks? What is a general approach or planning when it comes to selecting the number of hidden layers in a neural network? Are there datasets/industries/classification v. linear that would help in choosing the number of hidden layers?

I mostly deal with small, financial datasets (less than 10,000 records) and want to put more planning into the number of layers that I use. What should I be taking into account?

You generally won't know ahead of time how many layers to use for a particular project, so you'll have to try a variety (1-5 layers, etc). However, one key constraint that will provide an upper bound is that you don't want so many layers that your model starts to overfit. 10,000 records is a very small dataset, so I probably wouldn't try more than 3 hidden layers.

Some problems (e.g., object recognition in images) can have more layers depending on the design of the network. For example, using convolutions on images reduces overfitting, so you could have more layers. However, convolutions probably won't work on financial data. Convolutions work best on spatial data. Lastly, the number of layers is independent of whether the output is classification vs regression.


Matching Data Text of Two Place with Exception I have data of two places name and it's address in a row and i have to match it. Data is text type, I have read, it have to convert to numeric type that generated by the text. I extracted the numeric from the name place and address using fuzzywuzzy score matching and counting the length of it and make it the feature for classifying to.
This is the structure of data (this is just example but represents the data)
id     place_name    address                                         places_name_candidate_match   address_candidate_match                            match
1      Liberty Bell  143 S. 3rd Street, Philadelphia, Pennsylvania   Liberty Bell                  S. 3rd Street, Pennsylvania                        True
2      Liberty Bell  143 S. 3rd Street, Philadelphia, Pennsylvania   Liberty Bell                  S. 3rd Street, Pennsylvania                        True
3      Hershey Park  100 Hersheypark Drive, Hershey, Pennsylvania    Hershey Park                  S. 3rd Street, Pennsylvania                        False
4      Hershey Park  100 Hersheypark Drive, Hershey, Pennsylvania    Hershey Park                  100 Hersheypark Drive, Hershey, Pennsylvania       True

I would like to make an exception for a place which is a park that if it isn't park has different addresses but same place name, the result is still true, but if it is specifically park, the place name is the same but the address is different it should return false. But sometimes there are place names that don't contain the word "park", how do I create a feature to teach machines to make those exceptions? or is there another approach?

Rule of thumb: if a human with a lot of time wouldn't be able to do it, there's little chance a machine could. In this case if you ask somebody to classify whether a place is a park or not, they just can't do it correctly if the place name doesn't contain any indication that it is a park.
In a case like this you would need to use external resources in order to add indications about the place. For example you could search the place name in Wikipedia, and if Wikipedia says that it's a park then you add a boolean feature "true" for the place.
More generally I suspect that your approach is not very solid: you looked at a few results and noticed that your matching system makes mistakes about parks. If you look at more results there's a good chance that you will find more mistakes about various kinds of cases. Fuzzy matching works only when the strings are quite similar, so it's unavoidable that the system will make mistakes one way or the other. Even if you try to fix every kind of mistake one by one with some ad hoc exception, the system will never be completely correct. Importantly, this would make the system completely specific to the dataset, so it wouldn't generalize well to new instances.
This is an issue related to task specification and design: the automatic system can only achieve a certain level of quality, so maybe the task should be framed differently. For example, the system could be in charge of finding candidate pairs for matching, and a human expert would annotate the actual matches among the candidates. Or you could just decide to live with the errors, if they are not critical for the application.


Question about the simple example for batch normalization given in "deep learning" book In the section about batch normalization of Deep Learning book by Ian Goodfellow (chapter link) there is the follwing text:

  As  example, suppose we have a deep neural network that has only one
  unit per layerand does not use an activation function at each hidden
  layer: $y=x w_1 w_2 w_3 \ldots w_l$. Here, $w_i$ provides the weight used by
  layer $i$. The output of layer $i$ is  $h_i=h_{i−1} wi$. The output $y$ is a
  linear function of the input $x$, but a nonlinear function of the weights
  $w_i$.

Why y is nonlinear with respect to w_i?

I think what the statement meant was when given weights $w_1,...,w_n$ are fixed, output is linear proportional to $x$, but as it mentions

  nonlinear function of the weights w_i

Given a set of weights (more than one being varied), they do not linearly add to produce an output like

$y=w_1x_1+...+w_nx_n$

but rather non-linearly like

$y=w_1w_2..w_n*x$ (each $w_i$ is a dimension in the hyperspace)

And I think it would become more clearer from the statement "Output is linear to any weight $w_i$ but non-linear to weights $w_i$".


Understanding Terminology in Goodfellow's paper on GANS I am trying to understand Ian Goodfellow et al's paper Generative Adversarial Nets here.

In section 3 the author's write:

  The adversarial modeling framework is most straightforward to apply
  when the models are both multilayer perceptrons. To learn the
  generator’s distribution $p_g$ over data $x$, we define a prior on input
  noise variables $p_z(z)$, then represent a mapping to data space as $G(z;
$ $\theta_g)$, where $G$ is a differentiable function represented by a multilayer
  perceptron with parameters $\theta_g$.

I have a few questions.

1) From what I understand, the generator is a multi-layer perceptron input with a random vector sampled from a predefined latent space, (where wikipedia gives the example of a multivariate normal distribution). What does it mean to learn the generator's distribution $p_g$ over data $x$?

2) What does "we define a prior on input noise variables $p_z(z)$" mean? I understand that in Bayesian statistical inference, that a prior probability distribution, called a prior for short, gives a probability distribution on outcomes before some evidence is collected.

Any insights appreciated.

Question 1

  1) From what I understand, the generator is a multi-layer perceptron input with a random vector sampled from a predefined latent space, (where wikipedia gives the example of a multivariate normal distribution). What does it mean to learn the generator's distribution $p_g$ over data x?

Generally, supervised learning means to learn a distribution $p_{model}(y\mid x)$ where $x$ are your examples and $y$ your labels. However, in unsupervised learning it is not about labels but instead you are interested in unconditional probabilities and train a model to learn $p_{model}(x)$ (e.g. see p.485/486 in "The Elements of Statistical Learning" by Hastie et al). When applying GANs this $model$ is the generator $g$, i.e. you are learning $p_{g}(x)$. Accordingly, in the original paper the authors state:

  The generator $G$ implicitly defines a probability distribution $p_g$ as the distribution of the samples $G(z)$ obtained when $z ∼ p_z$.

Question 2

  2) What does "we define a prior on input noise variables $p_z(z)$" mean? I understand that in Bayesian statistical inference, that a prior probability distribution, called a prior for short, gives a probability distribution on outcomes before some evidence is collected.

Prior means we are talking unconditional probabilities, i.e. $p(z)$ is not conditioned on anything related to the task and examples on hand. While for example, $p_d(y\mid x)$ (the distribution learned by the Discriminator) is conditioned on $x$. Often $z$ is sampled from the standard normal distribution $\mathcal{N}(0,1)$.

In "NIPS 2016 Tutorial: Generative Adversarial Networks" the authors provide a good summary of the overall concept of $G$:

  The generator is simply a differentiable function $G$. When
  $z$ is sampled from some simple prior distribution, $G(z)$ yields a sample of $x$
  drawn from $p_{model}$. Typically, a deep neural network is used to represent $G$.


How to get a rebalance strategy with a cost matrix? In the case of a classification problem where a cost matrix is used to maximize the model performance, it is common to do a rebalance technique.

Let's say for example that I have the following costs for the two classes.

C(a,a) = 0, C(b,b) = 0, C(a,b) = 2, C(b,a) = 1.

Then, with a Rebalancing technique, I would need examples of class b twice as the examples of class a.

But, what should by rebalancing strategy will be when there is a cost for (a,a) or (b,b)?

For instance,

C(a,a) = 0, C(b,b) = 2, C(a,b) = -2, C(b,a) = -10

How should I handle those cases?

Is not very common to find cost functions where there is a cost associated with a correct answer C(b,b) (in your example).

But supposing there is, I think the solution to the classification could be trivial: I could say "All my predictions are 'b'" and in that way, I could have -10 as a cost many times and thus giving me a negative cost (depending on balance, of course).

I did not know the technique for applying cost you mention (rebalance accordingly), but for me, it would be more natural if the objective function changes to have this into account.

The following article talks about a possibility for addressing this (Instead of rebalancing, we should measure the cost sensitive matrix). And with XGBoost!

As far as I know, the cost function of a XGBoost classification can be personalized.

Cost Sensitive Classification

XGBoost change loss function - A related question in DS


How to build recommendation model based on resume and job description? How to build a model which will result in better recommendation of resumes based on the job description given?
I am familiar with bow or tfidf (n-grams) approach and then take a cosine similarity but I'm looking for a deep learning approach. I don't have any labelled data to evaluate.
Anything suggestions will be really appreciated.

If you want a DL approach, I recommend substituting the tf-idf by some kind of word embeddings.
For instance, you can take a pre-trained word embedding model, like glove, and average its outputs both in resume and job description, and then compute cosine similarity. However, I recommend to use a contextual word embedding (BERT-like), as the terms in resumes might be very dependent on the context.
The following article also introduces sentence-bert, which I think is very suited for your problem.


How to handle a regression problem with skewed target and only few high values? I'm currently tackling a regression problem with skewed target variable (presented below).

Naturally, my first idea was to transform the target with natural logarithm as it'll probably help both linear regression or decision-tree-based algorithms. The second idea is to prepare a validation scheme similar to stratified k-fold cross-validation with target binned into n groups. However, my concern is that I have only few highest values:

Therefore, my test set and all validation sets error are highly dependent if one of these 4 extreme values are drawed placed within them or not. That makes it hard to obtain reliable true error estimate.
Is there anything more I can do to handle that issue?

You can generate symmetrical distribution(s) by suitable transformation:
Your distribution is mainly right-skewed, therefore a log10 transformation is required.
You can also use an auto-binning method and combine it with dummy variables for the spikes.
If you then perform a feature selection, the learner will automatically choose the most significant features. Learners like XGBoost automatically take care of multicollinearity.


Understanding a Neural Network with Keras (preferably), TensorFlow or PyTorch I would like to try a technique I saw in an article a while ago where, in order to understand what each neuron is doing, you apply specific inputs to the net and see which one of them is most activated.

Is there a simple way to do it with Keras (preferably)? I can take the weights of each neuron after the NN has been trained and use forward (predict) functions to activate it, but how would I get the activation result of each individual neuron? Is there a function for that?

TensorFlow and PyTorch offer more low level functions so I could do the same thing if I programmed the NN almost from scratch, but even so, does anyone know a simpler way?

If you have tips for this task also, they would be more than welcome. I don't know if functions for this kind of task already exist, but, if they don't, I would say it would be worthwhile creating them, people on the market have complained often to me that the main reason why they are not yet using NNs is that they don't know how to understand the NN's thinking.

From Keras Issue #41:

  One simple way to do it, is to use the weights of your model to build a new model that's truncated at the layer you want to read. Then you can run the ._predict(X_batch) method to get the activations for a batch of inputs.


Converting Json file to Dataframe Python I have a json file which has multiple events, each event starts with EventVersion Key. The data looks similar to the following synthesized data.

{"Records":[{"eventVersion":"1.04","userIdentity":{"type":"R","principalId":"P:i","arn":"arn:aws:sts::5","accountId":"50","accessKeyId":"AW","sessionContext":{"attributes":{"mfaAuthenticated":"f","creationDate":"2013-09"},"sessionIssuer":{"type":"R","principalId":"WA","arn":"arn:aws:iam::6","accountId":"70","userName":"user1"}}},"eventTime":"2027-6","eventSource":"a.com","eventName":"DS","awsRegion":"UZ","sourceIPAddress":"2.1.3","userAgent":"li","requestParameters":null,"responseElements":null,"requestID":"OO","eventID":"09","eventType":"ABC","apiVersion":"2010-4","recipientAccountId":"78"},{"eventVersion":"1.04","userIdentity":{"type":"R","principalId":"P:i","arn":"arn:aws:sts::5","accountId":"50","accessKeyId":"AW","sessionContext":{"attributes":{"mfaAuthenticated":"f","creationDate":"2013-09"},"sessionIssuer":{"type":"R","principalId":"WA","arn":"arn:aws:iam::6","accountId":"70","userName":"user1"}}},"eventTime":"2027-6","eventSource":"a.com","eventName":"DS","awsRegion":"UZ","sourceIPAddress":"2.1.3","userAgent":"li","requestParameters":null,"responseElements":null,"requestID":"OO","eventID":"09","eventType":"ABC","apiVersion":"2010-4","recipientAccountId":"78"}]}

I'm using the following code in Python to convert this to Pandas Dataframe such that Keys are columns and values of each event is a row.

with open('/Users/snehahonnappa/Documents/NLP_AWSlogs/Model/Data/505728423372_CloudTrail_ap-northeast-1_20160913T1700Z_yKA3wB5Nx6juR6Kg.json') as json_data:
    sample_object = json.load(json_data)

df = pd.io.json.json_normalize(sample_object)
df.columns = df.columns.map(lambda x: x.split(".")[-1])

print df.shape

When I print shape of the dataframe its 1X1. I'm expecting (Number of unique keys X Number of records)

Snippet of how I'm expecting the dataframe to be

eventVersion               userIdentity                          eventTime
             type   principalId P   arn     accountID userName
1.04         R      P           i   arn:aws 50        user1      2027-6
1.06         Q      O           i   arn:aws 67        u2         2027-7

Appreciate any help.

Update :

I'm writing the json file into a csv and then trying to convert this to dataframe on which my models can be applied on. Following is my code.

import json
import csv
import sys

data_parsed = json.loads(open('/tmp/A.json').read())
log_data = data_parsed['Records']

# open a CSV file for writing
data = open('/tmp/log.csv', 'w')

# create the csv writer object
csvwriter = csv.writer(data)
count = 0

for i in log_data:
      if count == 0:
             header = i.keys()
             csvwriter.writerow(header)
             count += 1
      csvwriter.writerow(i.values())

data.close()

This is writing the keys as headers and values of each record as a separate row which is as expected. However the nested json objects are being written as one value.

Following is a snippet of my csv file which was obtained by executing the above code.

eventVersion eventID eventTime requestParameters                eventType
1.04         0       2016-20                                    AwsApiCall
1.04         8       2016-20    {u'tagKeys': [u'User Name']}    AwsApiCall
1.05         4       2016-30    {u'filterSet': {u'items': [{u'name': u'resource-type', u'valueSet': {u'items': [{u'value': u'*'}]}}, {u'name': u'tag:User Name', u'valueSet': {u'items': [{u'value': u'*'}]}}]}}
                                                                AwsApiCall

Any suggestions to tackle this?

I once ran into a situation like this where i wanted a complex dataframe due to the original source having a complex data structure.

I solved my issue by simplifying my structure into multiple separate dataframes instead of one big complex multi structure dataframe.

With simple separate dataframes i was better positioned to apply complex algorithmic operations.

Looking at your specific data, you could get rid of userIdentity which results in a simple 2d dataframe. This should position you to do any complex operation.

I understand this doesn't answer your specific dataframe structure requirement.

But i hope this answers the spirit of your objective.


How can I preprocess text to feed into a SVM? I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn.
The code for the above statement
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer()
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_test['text'])

When I am trying to use the data for an SVM, using a linear kernel and then fitting it into the model using
classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, df_test['label'])

I am getting the error
ValueError: Found input variables with inconsistent numbers of samples: [40000, 5000]
df_train is of the shape (40000,2)
df_test is of the shape (5000,2)
I am able to overcome this problem by using only 5000 values of df_train using
df_train.loc[:4999,'text']

but this defeats the purpose of having a bigger training dataset.
My question is how can I use the training dataset that will retain it's number of values?

X needs to be the features for your Model and Y needs to be a target variable.
As you mentioned, you are using a IMDB dateset so, all the features which you want your model to use will be stored in X variable whereas the 'LABEL' columns will be stored in the Y variable.
Instead try this code:
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_train['label'])

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, Y)

Another thing you did wrong here was that,
you passed the target variable of you testing data.
You train your model from your training data and if you are confident enough on your model then you test it using the testing data.
Concepts which can help you: Cross-Validation, train_test_split.


Python memory error: compute inverse of large sized matrix I have a big matrix of size 200000 x 200000. I need to compute its inverse. But it gives out of memory error on using numpy.linalg.inv. Is there any way to compute the inverse of a large sized matrix.

Well it does not matter the language you will use, I am surprised you can even store that matrice in memory... I had a similar problem while researching on kernel methods.

First, to put in perspective how huge that matrix is, I will assume each element is 32 bit word: $ 200,000 \times 200,000 \times 32 = 1.28 \times 10^{12} $ bits or 149 GB

The bitlenght is variable but even for a boolean matrix that is about 4,65 GB and you said your data is a bit more complex than that.

Solution 1

If you trying to solve a linear system there are many iterative solutions that might help you computing an 200,000 x 1 array aproximation of your system answer without having to store that absurdly large matrix in memory. But you should be aware that this might take a bit long because you might have to load information from the disk.

Solution 2

Another possible solution, if you really need that inverse is if that matrix can be expressed as sumation of outer products of vectors. If so, you can do this:

$$
M = \sum v_i.u_i^T
$$

Get the first set element of the sum and invert it so you have $M^{-1}_{bad aproximation}$ matrix by using pseudoinverse, instead of doing the outerproduct you can invert $u_1$ and $v_1$ and use Moore-Penrose Inverse property:

$$
M^{-1}_{bad aproximation} = (v_1.u_1)^\dagger  = v_1^\dagger . u_1^\dagger
$$

Combine this with Sherman-Morrison formula to add the other $v_i$ and $u_i$ vectors.

Sherman-Morrison formula gives:
$$
(A + uv^T)^{-1} = A^{-1} - \dfrac{A^{-1}uv^TA^{-1}}{1 + v^TA^{-1}u}
$$

Solution 3

You might try using Schur Complement which allows you to do the following:

Given a matrix $M$, it can be expressed as
$$
M =
    \begin{bmatrix}
    A & B  \\
    C & D  \\
    \end{bmatrix}
$$
where $A$ and $B$ must be square you can solve the inverse of $M$ blockwise as:

$$
M^{-1} =
    \begin{bmatrix}
    (M/D)^{-1} & (M/D)^{-1}BD  \\
    -D^{-1}C(M/D)^{-1} & (M/A)^{-1}  \\
    \end{bmatrix}
$$
where $(M/D)$ denotes the schur complement of block D of the Matrix M and is defines as

$$
(M/D) = A - BD^{-1}C
$$

also
$$
(M/A) = D - BA^{-1}C
$$

If $A$ or $D$ are singular, the matrice $M$ is singular and a pseudo inverse might be used.

This can give a reduction from inverting a $200,000 \times 200,000$ matrice to inverting 4 $50,000 \times 50,000$ matrices and some matrice multiplication, also this can be implement in parallel applications and there is way to do this recursively given certain conditions (enabling a reduction greater than 4 in size)

The $50,000 \times 50,000$ matrices store is only 9.3 GB each and will be easier to invert

Hope it helps. Also you might want to search answers in Math Stack or in Stack Overflow and even ask there.

  Edit: Added another solution with Schur Complement


Word2Vec vs. Doc2Vec Word Vectors I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access.
My question is:
Should we expect these word vectors and by association any of the methods such as most_similar to be 'better' than word2vec or are they essentially going to be the same? If in the future I only wanted word similarity should I just default to word2vec?

If you only care about word similarity, then apply Occam's Razor and use word2vec. There is no need to increase model complexity if not going to be used.
Also, the quality of embeddings is primarily increased through the size and diversity of the corpus. The algorithm has a much smaller effect on the quality of the embedding.


How do I plot linear regression results if input and target have different sizes? For a linear regression model that I conducted, I'd like to review the regression plot of results. But since I have an input of size 6 parameters and target (output therefore) of 4, I get error when I use this code : source code

pyplot.scatter(x_train, y_train)
pyplot.plot(np.sort(x_values, axis=0,prediction)
pyplot.show()

You can plot your residuals. I am sorry I can't write it in Python but in R you would do it like this:

plot(my.model$residuals)

This will give you a scatterplot that you WANT to look like a null plot. Any curve or pattern in the points means that your model is not a good fit.

If you are interested in assessing each variable's prediction power I recommend (because you only have six) doing 1 regression model on each variable by itself and then plotting the residuals of those models, again looking for a null plot. You can also look at the correlation of each predictor variable vs your response variable. Those with higher correlation will be better predictors.


pass variable length argument to mstats.kruskalwallis I am trying to run kruskawallis test on multiple columns of my data for that i wrote an function

var=['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']

def kruskawallis_test(column):
    k_test=train.loc[:,[column,'SalePrice']]
    x=pd.pivot_table(k_test,index=k_test.index, values='SalePrice',columns=column)

    for i in range(x.shape[1]):
        var[i]=x.iloc[:,i]
        var[i]=var[i][~var[i].isnull()].tolist()

    H, pval = mstats.kruskalwallis(var[0],var[1],var[2],var[3])

    return pval

the problem i am facing is every column have a different number of groups so var[0],var[1],var[2],var[3] will not be correct for every column. mstats.kruskalwallis() take input vector which contain values of each group to be compared from a particular column.(as per my knowledge).

is there a better way to do this?

or what can i do pass different number of variable for every column for example:

if a column x have a, b, c, d, e levels how can i pass 5 vectors?

i solved using

def kruskawallis_test(column):
    k_test=train.loc[:,[column,'SalePrice']]
    x=pd.pivot_table(k_test,index=k_test.index, values='SalePrice',columns=column)

    for i in range(x.shape[1]):
        var[i]=x.iloc[:,i]
        var[i]=var[i][~var[i].isnull()].tolist()

#     s=""
#     for i in range(x.shape[1]):
#         s=s+'var{}'.format(i)+','

    m=()
    for i in range(x.shape[1]):
        m=m+(var[i],)

    args=(m)

    H, pval = mstats.kruskalwallis(*args)

    return pval

i did
m=()
    for i in range(x.shape[1]):
        m=m+(var[i],)

args=(m)

by adding as many vars[i] in wanted to pass as argument of an empty tupple which i later passed to

mstats.kruskalwallis(*args)

*args helped mr pass variable length parameter.

please reply if you can suggest a better way.


How do I predict a class for each time step using the information from previous timsteps I have a classification problem but  different than usual. I have to provide 3 outputs (each of them either 0 or 1) for every input of 3 timesteps and 10 features.
What model architecture or approach is used in this type of problem?

Do I predict for each time step by separating the observations?
Do I predict for each time step and for next time step also use the previous time step?
Do I set output units to be 3 and train model to predict three values for 3 timesteps?

Regarding neural networks, and supposing your output is based on three prior timesteps, two possibilities could be:

An approach with a CNN model with a (3,10,1) input and your desired output.
LSTM where you use three cells the 10 features as an input for each timestep and output the classes you need at the end (don't know if this one will work properly though,maybe it is too small, depending on your problem).
As in you question 1. you propose you could use a DNN with the input of one of your previous timesteps, but I don't think this will be the best approach neither.

If you provide more details (like an example of your dataset) we can help you with more specific answers.


Forward or Backward Stepwise Regression I understand the process and logic of why to perform stepwise regression. To me they should always arrive at the same function, just one adds coefficients and tests for significance while the other removes coefficients as it tests for significance. Are there any significant trade-offs between using one versus the other?

Imagine you have 20 coefficients to test for and also have target accuracy (or whatever metric you're interested in) that you aim to beat. It acts as a threshold.

One tradeoff could be that performing "backwards regression" means you would in theory start with you maximum accuracy and be decreasing each time you remove a parameter. You might cut your threshold after removing only 3 parameters! So don't necessarily need to continue.

Conversely, starting with a single parameter and working your way up, you would have to try 17 models 5o reach your threshold.

This scenario could of course be reversed, so it is important to understand your data and make an informed decision about what suits your situation best.


How to visualize results/errors of multilabel classifiers? For multiclass classification you would normally choose a confusion matrix to plot the error of predicted classes against the target classes.

What is the best way to visualize errors of multilabel classifiers? As multiple classes are predicted at once a mapping of prediction against target is not always possible so confusion matrices are generelly not suitable.

My first idea is to plot a bar chart where each class has a bar for missed and one for false predictions. But is there any standard for visualizing the errors which includes more information just like a confusion matrix for multiclass problems?

There is something called macro-averaging and micro-averaging.

Macro-averaging is just the TPs, TNs, FPs, and FNs of each label class summed up and averaged. You should use macro-averaging for generalizing and evaluating on the overall. You should not use it to make specific decisions.

See details of micro-averaging and more evaluation techniques in the link below.
https://towardsdatascience.com/journey-to-the-center-of-multi-label-classification-384c40229bff#6aaa


Scikit model is not able to predict sequence correctly I am trying to create a regression model using scikit-learn for predicting car price. The input data are, car model(trim), kilometers used, past resale price of similar car and age of used car. I am trying to predict the future resale price of the car.

I have done the preprocessing of data. I have tried using ARDRegression, RandomForestRegressor and finally MLPRegressor. But the prediction model doesn't seem to predict well, the prediction results seem to fall outside the range of training data.

Example: if the Actual value(actual selling price) is 748077.0 but predicted value is 1352960

Probably I have done some mistake but I am not able to figure out what it is. I have simplified the code to focus on prediction part and have shared the link.

Could someone please guide me.

This is the google colab notebook link

Update: Inverting the index of the dataframe iresults in total change of prediction.

I don't use Python so I can't tell you exactly what is going on but I had a quick look at your data:

A few remarks:

it looks like the vast majority of the points are created artificially by interpolation. Why not, but that's unlikely to reflect the reality of the price changes: I would expect much more variation/noise in a real dataset about car prices.
there's no need to add so many points to the dataset anyway. With all this data there's not even a need to train a model, since virtually every possible instance is already in the data.
It seems to me that there's something weird with the prediction year: normally the higher the prediction year the more the price decreases right? here year 0 has no decrease at all, year 1 has the highest decrease,..., and year 4 has little to no decrease. That could confuse the model.
because it's mostly an artificial dataset the relation is very simple: a basic regression future_price = past_price * a + b would already give quite good results, and the relation can be learned perfectly when adding the prediction_year feature. At least MLP and random forest should give near perfect results.
From a quick look at the code, I suspect that the problem has to do with the scaling. I'm not sure what is supposed to happen there since I'm not familiar with these functions: it might be that the predicted values need to be "un-scaled" at the end maybe? Anyway I don't think scaling it's needed at all here.

For the record the graph was done with R like this:

library(ggplot2)
d<-read.table('interpolated_dataframe.csv',sep=',',header=TRUE)
ggplot(d,aes(past_price,future_price,colour=factor(prediction_year)))+geom_point()


Recommender system for next career step I want to build a recommender system that suggests the next step in your career.

About the dataset. About 50'000 Users with following informations:

Skills (tags, string value)
every job they did (embedded job titles)
every school they went

Now the recommender system should tell you what the next step in your career can be, and if you need a another skill or degree for that step.

How should you build such a system? (Supervised, Unsupervised (Recommender System,...), ...)

I have tried something similar once. I used an approach which you would not expect, but which gave some surprisingly good results.

I used the NMT (Neural Machine Translation) model in Tensorflow. There are some examples online for Vietnamese->English translation. I changed it to translate from "old job title" to "new job title", and trained it on my dataset. This is easy to do and you don't need to modify any code, except maybe tweak the model a little (e.g. length of document is now shorter than in the machine translation example).

The drawback of this approach is that only the most recent job title is taken into account when using it for inference. However you could try concatenating all previous job titles and other data (schools etc) as the input text, to produce a single job title as output, and this way you could make sure you are making use of their entire career path.

I would suggest tweaking aspects of the architecture (number of layers, number of previous jobs used, off-the-shelf word2vec vs domain specific trained word2vec etc) until you get the best performance against whatever evaluation metric you are using.

The other approach you could take would be to take the doc2vec of each title, and train some kind of RNN/LSTM to predict the next vec given all previous vecs. Then you need a postprocessing stage on the output to convert the output vec to a text. Unfortunately I don't know of a shortcut here to avoid building your model architecture from scratch, so this approach is more work.

Of course there are other ways of solving the problem but since no-one else has answered yet I thought I'd give my suggestion.


Using python to identify common paths that users follow among different activities I have a click-stream data of users regarding their behaviour in a platform. There are activities provided in a temporal order and users can go back and forth between these activities. Below is an example:

Here, there are 3 activities in a temporal order. Blue arrows represent one user and the green arrows represent another user. As you can see, these two users followed a different path.

In my scenario I have over 500 users , which are divided into 2 categories (let's Group A and Group B), and more than 50 activities. I want to identify the common patterns for Group A and Group B users and make a comparison among them (I am hoping to see patterns distinct to each user group).

I wonder if there are some useful (free) tools out there that I can take advantage of. I prefer Python but I could not find a python library for this. I am also looking for a tool (or python library) to visualise the paths that users follow. Any suggestions?

You can rank the paths by frequency using the database (SQL), constraining the start-, end- points as necessary once you fix the length of the window. If you let the path length be variable then you will not be able to do it all in SQL. In that case you can learn the transition probabilities between states then solve a weighted shortest path problem, where the distances are log likelihoods. Or you can use a heuristic like A* search. I do not know of any library that will do all this since it is very ad hoc, but you can do the visualization in networkx.


Identifying documents similar to specific clusters Through performing clustering on a set of 1,000,000 text documents, I have identified 100 clusters. I am particularly interested in, say, 10 of the clusters. Imagine, I now have an additional set of 100,000 documents (not part of the initial 1 mil documents). I wonder if there is a way to efficiently check whether each one of the 100K new documents belongs to one of the 10 clusters.

First Idea: Go as usual. Calculate the distance according to of each new document to the centroids of 100 clusters and see to which cluster the belong (minimum distance wins).
Second Idea: You already know which clusters are your targets. Assign a label (e.g. 1) to these 10 and and another label (0) to the rest 90. Then train a classifier and try to predict the label for new documents. (The process of labeling in supervised learning is usually based on expert annotation i.e. an acceptable accuracy in labeling. But with this method you lose that accuracy so expect that not all the labels are perfectly fine as a clustering algorithm is blindly do that)


Can i find similar players using a clustering method like the k-mean algorithm? I am working on a data mining project on NBA data. I want to make a recommendation system similar to the google one, where you search for players and you get recommendation for similar players. I intend to make such system based on their stats. My data looks like this

14  Tony Allen  709     357     15386   2178    4580    0.476   102     363     0.281   2076    4217    0.492   0.487   1220    1701    0.717   804     1567    2371    942     995     299     964     1675    5678

15  Al-Farouq Aminu     431     228     9988    1119    2592    0.432   188     607     0.31    931     1985    0.469   0.468   530     731     0.725   618     1724    2342    488     413     245     540     813     2956

One approach I have in mind is using k-means clustering, but I am not sure how I will use the clusters to find similar players. Each cluster will have hundreds of players, I will need further filtering.

Does anybody have any suggestions/tips for handling this problem?

For finding similar objects, k-means does not make much sense.

k-means is based on similar objects. If squared deviations don't work for your data then k-means will not work either.

Thus, instead of using k-means, just use a nearest neighbor search without k-means.


LSTMs: what is $W_x$ & $U_z$ in $φ(W_x + U_z + b)$? Reading On Multiplicative Integration with Recurrent Neural Networks

  Despite of their varying characteristics, most of them(RNNs) share a common
  computational building block, described by the following equation:
  $φ(W_x + U_z + b)$, where $x ∈ R_n$ and $z ∈ R_m$ are state vectors coming
  from different information sources, $W ∈ R_{d×n}$ and $U ∈ R_{d×m}$ are
  state-to-state transition matrices, and $b$ is a bias vector.

Don't get what meaning of $W_x$ and $U_z$. I know that $W$ is typicaly for weights... what does this equation mean?

If you look deeper in LSTMs or GRUs, we observe that the gates(input, output, cell or forget based on the RNN) are calculated using an equation like you specified.

Example, according to deep learning tutorial of lstm, it=sigma(Wi xt + Ui ht-1 + bi) In this, h is the hidden state vector and x is the input state vector as specified and W and U are the corresponding weights for the input gate it. SImilarly, there are gates for output and forget. So in the paper, they recall a gist of RNNs and sum it up as a general equation. It is a common computational block in RNNs despite their minor differences.

Refer Colah's blog or wildml, I think they are one of the best to understand RNNs.


how to create outputs for key points of bounding boxes on image in Neural network in Python I've made my first convolutional neural network which can classify one object on image (220, 220). The objects classes are only two - dogs and cats. I have 3385 '.jpg' pictures of dogs and cats on my PC. I have information about class of each object (1 or 0). Also I have information about two corners coordinates (from 0.0 to 1.0) of bounding boxes (0.0 is left/top corner of image and 1.0 is right/bottom corner of image) around of the objects. Convolutional net must have 5 outputs (1 output for classes and 4 outputs for two corners coordinates: x1,y1,x2,y2).
I have 3047 train images 'x_train' shape (3047, 220, 220, 3) and labels information in 'y_train' shape (3047, 5). Also 338 test images in 'x_test' shape (338, 220, 220, 3) and test labels in 'y_test'. For example, y_test[0] has such elements array([1.        , 0.555     , 0.18      , 0.70833333, 0.395     ]).
But my neural network have such prediction results prediction[0] is array([276.47525 ,  51.961666,  20.63266 , 125.419945, 106.816345],dtype=float32). And np.sum(prediction) is 338.0 which is number of test images. During the training parameters increased from  'loss: 4.8534 - accuracy: 0.5938' to 'loss: 533785.6875 - accuracy: 1.0000'. Could anyone help me with an advice what do I do wrong?
#Summary:
#Model: "sequential_1"
#conv2d_2 (Conv2D)            (None, 220, 220, 32)      896
#max_pooling2d_2 (MaxPooling2 (None, 110, 110, 32)      0
#conv2d_3 (Conv2D)            (None, 110, 110, 64)      18496
#max_pooling2d_3 (MaxPooling2 (None, 55, 55, 64)        0
#flatten_1 (Flatten)          (None, 193600)            0
#dense_2 (Dense)              (None, 128)               24780928
#dense_3 (Dense)              (None, 5)                 645
#Total params: 24,800,965
#Trainable params: 24,800,965
#Non-trainable params: 0

import os
import cv2
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
import numpy as np
import matplotlib.pyplot as plt
from tensorflow import keras
from tensorflow.keras.layers import Dense, Flatten, Dropout, Conv2D, MaxPooling2D
import time

model = keras.Sequential([
    Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=(220, 220, 3)),
    MaxPooling2D((2, 2), strides=2),
    Conv2D(64, (3, 3), padding='same', activation='relu'),
    MaxPooling2D((2, 2), strides=2),
    Flatten(),
    Dense(128, activation='relu'),
    Dense(5, activation='relu')

])

print(model.summary())
model.compile(optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy'])
his = model.fit(x_train, y_train, batch_size=32, epochs=2, validation_split=0.5)
model.evaluate(x_test, y_test)

Your network can be improved in many ways :
Adding a few convolutional layers could help reduce the dimension as you have a lot a weights on your first FC layer, and this will improve the speed of your training.
But your main problem is your loss function, cross entropy is used for classification, which means your network is considering all your 5 ouputs as classes (a softmax function is applied to all 5 of them before computing cross entropy).
For your task, i would feed the outputs of the convolutional network into 2 networks:

One is the classifier that tells you if it is a cat or a dog (1 output).
The second outputs the bounding box (4 outputs).

These networks can be trained independently using different losses and will give you coherent results.


What is the branch of data science that deals with generating data? Looking for tutorials So far all the tutorials (Python/Tensorflow) I have been following dealt with processing the data, they are classification problems. Things that give Yes or No answers. Now I want to find tutorials on actually creating things using ML from large data sets.

For example lets say the task is to generate a face using millions of face photos. This is not a problem that produces statistics from data, but rather a problem of using statistics to generate data. My understanding is, the algorithm would generate random images and run them through the classifier until they closely resemble the face. Is this AI? I wouldn't consider it to be intelligent anything. I understand there is also a ton of creativity that goes into generating the images. But let's say I just wanted to use a +-RNG on every pixel. Is there an adaptive process where it would start matching at higher convolutional levels, where the image is just blobs of information and then use that information, adding or subtracting color, edges, etc, to move to lower levels that more closely resemble a face.

What is this type of problem called and what should I be looking for when searching a tutorial. If there is one that shows you how to do something similar, can you point me to this?

Thank you

This is known as generative models, they basically infer the probability of the data based on the training examples. Algorithms usually are discriminative, i.e they perform prediction by estimating:

$$
p(Y|X)
$$

so given your data they are able to predict a label.

Generative models are interested in $p(X)$ i.e. how is the data generated. Once the model learns this, it can produce "fake" images that look like your inputs.

You can look at:

Generative Adversarial Nets (which have had a lot of attention lately)
Variational Autoencoders (were in vogue before GANS were introduced)


4 Class Classification - Machine Learning Model I have a data set which contains nearly 150 features and 60k data. And my target feature is continuous variable represents hours. I divided this period into 4 categories of user engagement (4 ranges of hours). Implement GA with SVM, GA with logistic regression, Random forest, GA with KNN with suitable normalization of data wherever required. Used GA for best feature subset selection.

All the algorithms gave similar results of around 46% accuracy ( for nearly balanced test set).

Note: Training is also done on a balanced data set. I am wondering where am I gone wrong?

I believe I went wrong somewhere in input to target mapping. Could anyone please confirm that categorization of the continuous target variable (hrs) into 4 sets are reasonable?

Yes, binning a continuous variable so that in can take discrete values is reasonable, as long as you are OK with transforming this regression problem into a classification one.

Just note that when dealing with a balanced 4-class classification problem, if an algorithm hadn't learnt anything and was predicting randomly it would achieve a 25% accuracy. Your results indicate that the algorithms are at least learning something. It might just be a very difficult task to solve, in that the features might have little to no correlation with the output variable.

I'd suggest trying out more algorithms with and without GA.


Model Predicts Badly on Real-World Data I know this is probably a really easy fix, but I've been stuck on this for a while. I'm a noobie to both machine learning and programming in general.  My data is from this dataset. https://www.kaggle.com/ozlerhakan/spam-or-not-spam-dataset
I made a model that can predict whether or not an email is spam email, and it does so reasonably well, getting an 84% accuracy with the validation data. But when I try to make it predict on a new string, it seems to be predicting on a character-by-character basis. Below is the code that I used to preprocess the training(and testing) data. I would've copy-pasted the required code, but when I look at the draft, it looks terrible. So here's the link to my notebook. https://colab.research.google.com/drive/1EnoNFR5CFi85hOrW8ihx66QWb6sToKAL?usp=sharing
I tried to debug the function by putting various print statements all over. I've come to the conclusion that the problem spot is when I try to one-hot encode the words. I need those words to be one-hot encoded because that is how I trained the model.  But I don't know how to fix that. I would really appreciate some help with this. Thank you in advance.

You are using multiple frameworks (pandas, keras / tensorflow, nltk, reg, scikit-learn) and doing a lot of manually processing. It would be better if you used a single framework. A single framework would automatic many of the steps and appropriately handle training and prediction. The result would be creating the same one-hot encoding scheme at the token level.


How to properly set up neural network training for stable accuracy and loss I have a DenseNet121 implemented in Pytorch for image classification. For now, the training set-up is pretty straightforward:

the data is loaded. An important characteristic here is that the validation data is fixed from the outset and can never change. The rest of the data I split into training and testing. None of the data sets overlap!
for every epoch iterate through the training data loader, calculating loss, optimizing etc.
every 100 batches evaluate the loss using the validation data loader
at the end of an epoch compare the current state’s loss on validation data with the state that previously had the best validation loss (for the first epoch just compare this with a random high number like 1e5) and save the current state if it better or keep the older state.
after all epochs are finished save the state with the lowest validation error as the best model
the best model is then applied to test data. Accuracies are calculated and ROC curves drawn

I was wondering how to extend my set-up to make it a proper statistical experiment set-up, with the goal of getting stable accuracy and loss results ie I want to be able to say that my model gives consistently the same results. I was thinking along the lines of running the testing step say 10 times and averaging the error? Or do you see some deficiency in the training that I could improve to improve stability?
Thanks in advance

The learning rate is one of those first and most important parameters of a model, and one that you need to start thinking about pretty much immediately upon starting to build a model. It controls how big the jumps your model makes, and from there, how quickly it learns.
There are learning rate technique called Cyclical Learning Rates.
Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. Check this out : Cyclical Learning Rates for Training Neural Networks. And implementation in pytorch and details blog : Adaptive - and Cyclical Learning Rates using PyTorch.
By this small trick, you can build a stable version of your model. Best of luck.


How to make overfitting (powerful) model? According to my professor one of the first steps in modelling a NN is to use a powerful enough model.

  The first step is to create a model that is powerful enough to
  achieve very high accuracies (very low loss) on the training data, at
  least when no regularisation is used.

What are some of the things (obviously apart of regulazing and adjusting learning rate), that I can do to make my model "powerful" enough, in other words to let it overfit on the training data?

Am I looking in the right direction with the following things?

Add extra layers
Make layers thicker (more neurons)

Here are some general hyperparameter adjustments to increase model capacity:

Architecture-related:

add additional layers
increase the number of hidden units per layer
remove dropout or decrease dropout rates

Optimizer related:

find the optimal learning rate
train for more epochs

Section 11.4.1 in the Deep Learning Book provides a good overview. The paper "Practical recommendations for gradient-based training of deep architectures" is another good source to understand the effect of different hyperparameters (though it is a bit older).

Moreover, there are specific hyperparameters to increase model capacity for different network types, e.g.

kernel size for CNNs,
sequence length for LSTMs and
embedding dimension for embedding layers.


Visualize changes other time I am analyzing a population movement pattern and I would like to design a visualization like this one:

Is there any tool or language (preferably R) I can use for that purpose?

Here is a sample of my dataset.

Of course you can reproduce that graph with some ggplot!

Here is the code to create it:

library(ggplot2)
library(scales)

df <- read.csv2('Graphs_SK.csv')

p <- ggplot(data = df, aes(x = Governorate)) +
  geom_segment(aes(y = Mid.May.2015, yend = Dec.16, xend = Governorate),
               color = 'grey', size = 1) +
  geom_point(aes(y = Mid.May.2015, color = 'start'), size = 2) +
  geom_point(aes(y = Dec.16, color = 'end'), size = 2) +
  scale_colour_manual(name = '(Population in millions)',
                      labels = c('May 2015', 'Dec 2016'),
                      values = c('start' = 'steelblue4', 'end' = 'steelblue')) +
  scale_y_continuous(labels = comma, name = '', position = 'top') +
  coord_flip() +
  ggtitle('1 Year of something of Governorate') +
  guides(colour = guide_legend(title.position = 'right')) +
  theme(panel.background = element_rect(fill = 'white'),
        legend.position = c(-.15, 1.03),
        legend.direction = 'horizontal',
        legend.justification = c(0, 0),
        legend.key = element_rect(fill = NA),
        legend.title = element_text(size = 10),
        axis.text.y = element_text(hjust = 0),
        axis.ticks.y = element_line(size = NA),
        plot.title = element_text(hjust = .5))


Optimization problem: Given Beta Bounds Maximize sharpe I would like to maximize a portfolio's Sharpe Ratio while keeping Beta in bounds.
Could anyone supply a calculation please?

***************Starting Metrics**********************
symbols          : ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H']
Beta             : [ 0.09  1.2   2.1  -1.3   1.1   0.3   0.9   1.2 ]
Sharpe           : [-1.7  5.4 -0.3  0.8  0.4  2.7  0.9  1.9]
optimized weights: [x0, x1, x2, x3, x4, x5, x6, x7]

***************Unadjusted Portfolio**********************
AvgBeta unadjusted  : 0.69875
AvgSharpe unadjusted: 1.2625000000000002

***************Ajusted Portfolio**********************
Beta bounds: -.3 < x <.3
Maximize sharpe: X

Well, you didn't give a whole lot of information, so I'm going to make some assumptions here, but I think this is pretty darn close to what you want.  Keep in mind, Beta = (Covar/Var), so you need to calculate that with what's given below, and then you will have everything you need.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.optimize as sco
import datetime as dt
import math
from datetime import datetime, timedelta
from pandas_datareader import data as wb
from sklearn.cluster import KMeans
np.random.seed(777)

start = '2019-02-28'
end = '2020-02-28'
# N = 90
# start = datetime.now() - timedelta(days=N)
# end = dt.datetime.today()

tickers = ['MMM',
'ABT',
'ABBV',
'ABMD',
'ACN',
'ATVI',
'ADBE',
'AMD',
'AAP',
'AES',
'AMG',
'XEL',
'XRX',
'XLNX',
'XYL',
'YUM',
'ZBH',
'ZION',
'ZTS']

thelen = len(tickers)

price_data = []
for ticker in tickers:
    prices = wb.DataReader(ticker, start = start, end = end, data_source='yahoo')[['Adj Close']]
    price_data.append(prices.assign(ticker=ticker)[['ticker', 'Adj Close']])

df = pd.concat(price_data)
df.dtypes
df.head()
df.shape

pd.set_option('display.max_columns', 500)

df = df.reset_index()
df = df.set_index('Date')
table = df.pivot(columns='ticker')
# By specifying col[1] in below list comprehension
# You can select the stock names under multi-level column
table.columns = [col[1] for col in table.columns]
table.head()

###################################################

def portfolio_annualised_performance(weights, mean_returns, cov_matrix):
    returns = np.sum(mean_returns*weights ) *252
    std = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights))) * np.sqrt(252)
    return std, returns

def random_portfolios(num_portfolios, mean_returns, cov_matrix, risk_free_rate):
    results = np.zeros((3,num_portfolios))
    weights_record = []
    for i in range(num_portfolios):
        weights = np.random.random(thelen)
        weights /= np.sum(weights)
        weights_record.append(weights)
        portfolio_std_dev, portfolio_return = portfolio_annualised_performance(weights, mean_returns, cov_matrix)
        results[0,i] = portfolio_std_dev
        results[1,i] = portfolio_return
        results[2,i] = (portfolio_return - risk_free_rate) / portfolio_std_dev
    return results, weights_record

returns = table.pct_change()
mean_returns = returns.mean()
cov_matrix = returns.cov()
num_portfolios = 10000
risk_free_rate = 0.0178

###################################################

def display_simulated_ef_with_random(mean_returns, cov_matrix, num_portfolios, risk_free_rate):
    results, weights = random_portfolios(num_portfolios,mean_returns, cov_matrix, risk_free_rate)

    max_sharpe_idx = np.argmax(results[2])
    sdp, rp = results[0,max_sharpe_idx], results[1,max_sharpe_idx]
    max_sharpe_allocation = pd.DataFrame(weights[max_sharpe_idx],index=table.columns,columns=['allocation'])
    max_sharpe_allocation.allocation = [round(i*100,2)for i in max_sharpe_allocation.allocation]
    max_sharpe_allocation = max_sharpe_allocation.T

    min_vol_idx = np.argmin(results[0])
    sdp_min, rp_min = results[0,min_vol_idx], results[1,min_vol_idx]
    min_vol_allocation = pd.DataFrame(weights[min_vol_idx],index=table.columns,columns=['allocation'])
    min_vol_allocation.allocation = [round(i*100,2)for i in min_vol_allocation.allocation]
    min_vol_allocation = min_vol_allocation.T

    print("-")
    print("Maximum Sharpe Ratio Portfolio Allocation\n")
    print("Annualised Return:", round(rp,2))
    print("Annualised Volatility:", round(sdp,2))
    print("\n")
    print(max_sharpe_allocation)
    print("-")
    print("Minimum Volatility Portfolio Allocation\n")
    print("Annualised Return:", round(rp_min,2))
    print("Annualised Volatility:", round(sdp_min,2))
    print("\n")
    print(min_vol_allocation)

    plt.figure(figsize=(10, 7))
    plt.scatter(results[0,:],results[1,:],c=results[2,:],cmap='YlGnBu', marker='o', s=10, alpha=0.3)
    plt.colorbar()
    plt.scatter(sdp,rp,marker='*',color='r',s=500, label='Maximum Sharpe ratio')
    plt.scatter(sdp_min,rp_min,marker='*',color='g',s=500, label='Minimum volatility')
    plt.title('Simulated Portfolio Optimization based on Efficient Frontier')
    plt.xlabel('annualised volatility')
    plt.ylabel('annualised returns')
    plt.legend(labelspacing=0.8)

display_simulated_ef_with_random(mean_returns, cov_matrix, num_portfolios, risk_free_rate)

Result:

Maximum Sharpe Ratio Portfolio Allocation

Annualised Return: 0.2
Annualised Volatility: 0.16

             AAP  ABBV  ABMD   ABT   ACN  ADBE   AES    AMD   AMG   ATVI  \
allocation  3.29  5.51  0.59  4.29  2.89  9.09  8.42  11.22  3.01  10.03

             MMM    XEL  XLNX  XRX   XYL   YUM   ZBH  ZION   ZTS
allocation  0.12  11.39  2.95  6.7  2.89  5.91  3.02  0.79  7.88
-
Minimum Volatility Portfolio Allocation

Annualised Return: 0.05
Annualised Volatility: 0.13

             AAP  ABBV  ABMD   ABT   ACN  ADBE   AES   AMD   AMG  ATVI   MMM  \
allocation  6.35  8.14  2.55  4.32  8.71  2.18  5.62  1.99  2.08  4.86  6.89

             XEL  XLNX  XRX  XYL   YUM   ZBH  ZION   ZTS
allocation  10.1  1.93  1.9  5.5  7.92  5.57  6.01  7.39


Is there an algorithm that imputes missing values based on n nearest columns? (KNN hybrid) I have a dataset of 70 columns that have missing values. Each column has a few columns (3-5) that it is significantly more correlated than the others but each column's correlated columns are very different from other columns. I would like to perform a tweaked version of KNN imputation and before I start writing this from scratch, I'd like to know if there's something similar out there, So, I don't go about reinventing the wheel.
On initialization, a correlation matrix is obtained between all the variables. For every missing cell that needs to be imputed, the algorithm retrieves from the correlation matrix the top n highest correlated columns and uses only those N columns to select the k nearest neighbors and use the mean value of those neighbors.
Thanks and apologies if I am not very clear with my question. I'll try to clarify if you have any doubts.

You can extend scikit-learn's sklearn.impute.KNNImputer. The metric parameter can accept the custom function you described.


Kappa near to 60% in unbalanced (1:10) data set As mentioned before, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples.
I have trained a Generalized Boosted Regression model using gbm() from the gbm package in R and get the following output:

  interaction.depth  n.trees  Accuracy  Kappa  Accuracy SD  Kappa SD
  1                  50       0.906     0.523  0.00978      0.0512
  1                  100      0.91      0.561  0.0108       0.0517
  1                  150      0.91      0.572  0.0104       0.0492
  2                  50       0.908     0.569  0.0106       0.0484
  2                  100      0.91      0.582  0.00965      0.0443
  2                  150      0.91      0.584  0.00976      0.0437
  3                  50       0.909     0.578  0.00996      0.0469
  3                  100      0.91      0.583  0.00975      0.0447
  3                  150      0.911     0.586  0.00962      0.0443

Looking at the 90% accuracy I assume that model has labeled all the samples as majority class. That's clear.
And what is not transparent: how Kappa is calculated.

What does this Kappa values (near to 60%) really mean? Is it enough to say that the model is not classifying them just by chance?
What do Accuracy SD and Kappa SD mean?

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing with text/linguistics.

What it does is compare the level of agreement between the output of the (human or algorithmic) annotater and the ground truth labels, to the level of agreement that would occur through random chance. There's a very good overview of how to calculate Kappa and use it to evaluate a classifier in this stats.stackexchange.com answer here, and a more in-depth explanation of Kappa and how to interpret it in this paper, entitled "Understanding Interobserver Agreement: The Kappa Statistic" by Viera & Garrett (2005).

The benefit of using Kappa, particularly in an unbalanced data set like yours, is that with a 90-10% imbalance between the classes, you can achieve 90% accuracy by simply labeling all of the data points with the label of the more commonly occurring class. The Kappa statistic is describing how well the classifier performs above that baseline level of performance.

Kappa ranges from -1 to 1, with 0 indicating no agreement between the raters, 1 indicating a perfect agreement, and negative numbers indicating systematic disagreement. While interpretation is somewhat arbitrary (and very task-dependent), Landis & Koch (1977) defined the following interpretation system which can work as a general rule of thumb:

Kappa Agreement
< 0 Less than chance agreement
0.01–0.20 Slight agreement
0.21– 0.40 Fair agreement
0.41–0.60 Moderate agreement
0.61–0.80 Substantial agreement
0.81–0.99 Almost perfect agreement

Which would indicate that your algorithm is performing moderately well. Accuracy SD and Kappa SD are the respective Standard Deviations of the Accuracy and Kappa scores. I hope this is helpful!


Building Customers/Patient Profiles I am looking for ideas on how to proceed with a situation. I have historical data of appointments for many users. I can (easily) predict their future behaviour (whether the next appointment will be positive or not)/

I now need to build profiles, i.e., classify a user as "good" or "bad" user. So, I won't be predicting future appointments, but just say "this user is a good/bad user", and then compare with, e.g., a monthly behaviour.

Any ideas where I could look for information on how to proceed?

I apologize for my vague first question. I will try to be more clear.

I have ~200k users, with their doctor appointment history. As demographic, I only have their age and gender. The rest of the variables relate to their appointment: time of the appointment, which doctor (service), day of the appointment, and so on. I also have whether they went or not to the appointment (show / no show).

The assignment consist on classifying the users as "good" or "bad", i.e., whether they will go or not to the appointment. I do not have to predict if they will go to the next appointment, just to have a list of the users classification. By doing this, if user A, whom I classified as "good", calls for an appointment, I know he is "good" and he will make it to the appointment. I do not to take any action on him.

Again, I do not have to predict future appointments, or the behavior of a new user, just to classify the existing ones.

I hope now it is a bit more clear. If not, let me know. Thank you!

I know perhaps the information I am giving is a bit vague, but I didn't know if to go into full detail or not. If you want me to share more info, just let me know.

Thanks!

Maybe I'm being a bit too simplistic, but I would build a set of training data that looks like this (Good=1 means patient showed up for appt and is good by your definition, 0 = bad)

Recid, PatId, SurgeryId, DrId, DateAppt, TimeAppt, Gender, Age, Good
1, 1, 100, 10, 01jan16, 10:30, M, 31, 1
2, 1, 100, 12, 05jan16, 15:20, M, 31, 1
3, 1, 100, 10, 06mar16, 11:45, M, 31, 0
4, 2, 101, 15, 02Feb16, 12:35, F, 75, 1
....

I would then use one of the machine learning tools in R - there are a variety of them, to train a model of your data.

Then with another set of data, I would test the model you have just built to see how correct it is. If you don't have a second set of data, then randomly partition your original training set and only train with half of the data.

Some further suggestions to make your model more powerful, is create additional variables with the following information

a Flag to indicate if a person has missed a previous appointment with any doctor.
a Flag to indicate if a person has missed a previous appointment with the particular doctor they are going to be visiting - I know from person experience there are some doctors in my surgery that I have a preference to see.
The number of days since the last appointment
Day of week of appointment

Sounds like a nice dataset to be working with!


Hardware requirements for Linux server to run R & RStudio I want to build a home server/workstation to run my R projects. Based on what I have gathered, it should probably be Linux based. I want to buy the hardware now, but I am confused with the many available options for processors/ram/motherboards. I want to be able to use parallel processing, at least 64GB? of memory and enough storage space (~10TB?). Software wise, Ubuntu?, R, RStudio, PostgreSQL, some NOSQL database, probably Hadoop. I do a lot of text/geospatial/network analytics that are resource intensive. Budget ~$3000US.

My Questions:
What could an ideal configuration look like? (Hardware + Software)
What type of processor?

Notes:
No, I don't want to use a cloud solution.
I know it is a vague question, but any thoughts will help, please?
If it is off-topic or too vague, I will gladly delete.

Cheers B

There is no ideal configuration, for R or in general - product selection is always a difficult task and many factors are at play. I think that the solution is rather simple - get the best computer that your budget allows.

Having said that, since you want to focus on R development and one of R's pressing issues is its critical dependence on the amount of available physical memory (RAM), I would suggest favoring more RAM to other parameters. The second most important parameter, in my opinion, would be number of cores (or processors - see details below), due to your potential multiprocessing focus. Finally, the two next most important criteria I'd pay attention to would be compatibility with Linux and system/manufacturer's quality.

As far as the storage goes, I suggest considering solid state drives (SSD), if you'd rather prefer to have a bit more more speed than more space (however, if your work will involve intensive disk operations, you might want to investigate the issue of SSD reliability or consult with people, knowledgeable in this matter). However, I think that for R-focused work, disk operations are much less critical than memory ones, as I've mentioned above.

When choosing a specific Linux distribution, I suggest using a well-supported one, such as Debian or, even better, Ubuntu (if you care more about support, choose their LTS version). I'd rather not buy parts and assemble custom box, but some people would definitely prefer that route - for that you really need to know hardware well, but potential compatibility could still be an issue. The next paragraph provides some examples for both commercial-off-the-shelf (COTS) and custom solutions.

Should you be interested in the custom system route, this discussion might be worth reading, as it contains some interesting pricing numbers (just to get an idea of potential savings) and also sheds some light on multiprocessor vs. multi-core alternatives (obviously, the context is different, but nevertheless could be useful). As I said, I would go the COTS route, mainly due to reliability and compatibility issues. In terms of single-processor multi-core systems, your budget is more than enough. However, when we go to multiprocessor workstations (I'm not even talking about servers), even two-processor configurations can go over your budget easily. Some, not far away, such as HP Z820 Workstation. It starts from 2439 USD, but in minimal configuration. When you upgrade it to match your desired specs (if it's even possible), I'm sure that we'll be talking about 5K USD price range (extrapolating from the series' higher-level models). What I like about HP Z820, though, is the fact that this system is Ubuntu certified. Considering system compatibility and assuming your desire to run Ubuntu, the best way to approach your problem is to go through Ubuntu-certified hardware lists and shortlist systems that you like. Just for the sake of completeness, take a look at this interesting multiprocessor system, which in compatible configuration might cost less than from HP or other major vendors. However, it's multimedia-oriented as well as it's reliability and compatibility are unknown, not to mention that it's way over your specified budget.

In terms of R and R-focused software, I highly recommend you to use RStudio Server instead of RStudio, as that will provide you with an opportunity to be able to work from any Internet-enabled location (provided you computer will be running, obviously). Another advice that I have is to keep an eye on alternative R distributions. I'm not talking about commercial expensive ones, but about emerging open source projects, such as pqR: http://www.pqr-project.org. Will update as needed. I hope this is helpful.