"R" write/export dataframe preserving the class information Short Question

After formatting a dataframe in R with class information like "Date", "Factor", and etc. how to save the dataframe while preserving the class information. Consequently how to load the information back to a new R Project with preserving the above class information.

Long Question
I have created following dataframe with class information,
names <- c("Adrew", "Vikie", "Penny", "Samantha")
date <- as.Date(c("1984-12-28", "1986-10-18", "1986-11-30", "1987-11-30"))
gender <- c("M", "M", "S", "S")

df <- data.frame(names, date, gender)

df$gender <- factor(df$gender,
                   levels = c("M", "S"),
                   labels = c(1,2),
                   order = FALSE)

and the classes are factor, date, factor
method 01
then write the dataframe in csv format and read it
write.csv(df, "df.csv")
dfcsv <- read.csv("df.csv")

and the daraframe classes are changed
method 02
then tried to write the file using table export method
write.table(df, "df.R")
dftable <- read.table("df.R")

again the data frame classes had been changed
Question

is there a proper method to write or save a data frame with preserving the class information like date, factor, etc information intact, and conversely how to load it back.

The best way to achieve this would be to save the data as an R data file using either save() or saveRDS():
# option 1
save(df, file="data.Rdata")
load("data.Rdata")

# option 2
saveRDS(df, file="data2.Rds")
df <- readRDS("data2.Rds")


How do I merge two data frames in Python Pandas? I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like tearing a piece of paper that contains a list in half so that half the columns go on one paper and half the columns go on the other. I would like to merge them back together. How do I do it?

Pandas has a built-in merge function. Please refer to the documentation

End-result should be something like this:
df_merged = pd.merge(df1, df2, left_on=['name_indexcolumn_df1_here'],
              right_on=['name_indexcolumn_df2_here'],
              how='inner')


Run Apriori algorithm in python 2.7 I have a DataFrame in python by using pandas which has 3 columns and 80.000.000 rows.

The Columns are: {event_id,device_id,category}.[]

each device has many events and each event can have more than one category.

I want to run Apriori algorithm to find out which categories seem together.

My idea is to create a list of lists[[]]: to save the categories which are in the same event for each device. like: [('a'),('a','b')('d'),('s','a','b')] then giving the list of lists as transactions to the algorithm. I need help to create the list of lists.

If you have better idea please tell me because I am new in Python and this was the only way I found out.

df.groupby('device_id')['category'].apply(list).tolist()

There's your transactions LOL.

If you aren't limited to Python 2.7, I'd suggest Orange3-Associate which contains a frequent_itemsets() function based on FP-growth algorithm, which is orders of magnitude faster than Apriori.


Different results for LogisticRegression on python 2.7 and 3 I have different results for the same kernel on python 2.7 (local machine) and python3 (the system running on kaggle) for LogisticRegression. How it is possible?

Here my results from my local machine:

Here from the kaggle notebook:

The amount of data is different, because I split a bit more for the train data, but the predictions are totally different.  Can this be because of the versions of python?

Assuming that you are using sklearn, if you go to the source in the github
you will see that they do not differ.

But you have variability elsewhere. For example train_test_split of sklearn also takes random seed, so it could be that your data is fundamentally different between these approaches (you could do cross valdiation for example to include all the data in train and test)


Combining textual and numeric features into pre-trained Transformer BERT I have a dataset with 3 columns:

Text
Meta-data (intending to extract features from it, then use those i.e., numerical features)
Target label

Question 1: How can I use a pre-trained BERT instance on more than the text?
One theoretical solution suggests having BERT fed the text and another neural network with the numerical features fed into this one, then aggregating their output, into another neural network.

Is that the most efficient approach?

Question 2: How can you connect neural networks?

You get the output from each, but then what?

You get classification output from BERT, you get classification
output from MLP based on numerical features.

You concatenate these and feed them to another MLP, and you get the
final prediction? Wouldn't that last prediction be less robust ?

In other words, does the last MLP encapsulate the other 2 networks?

If so, what happens if BERT predicts on 90%, but the first MLP just
50%, will we get a lesser outcome?

Question 3: Any tips on how to implement this in pytorch?

When they are talking about aggregating their outputs, they mean the final embeddings (just before the classification layer) not the output of the network itself.
You take the embeddings from both of the networks and concatenate them vertically. This concatenated embedding you use to predict the final output you wanted.
Torch has a function torch.cat for this purpose.


Time-stamp for linear model How can we extract information from time-stamp variable for modelling? I have a variable with format mm-dd-yyyy hh:mm:ss I want to predict an outcome variable using time-stamp as input variable. I do not think i can directly use this column for modelling and will need to do some transformation before i can use this in model. Not sure what type of transformation i need to do.

For example, if you have date field, you can create dummy variables of day o week and use that as input variable in model. Not sure how to proceed in the case of time-stamp.

Any help?

Welcome to Datascience.SE!

Like you said, you can extract the day of the week. Also extract the hour of the day, then encode these two variables using sines and cosines with their respective periodicities (7 and 24). Also create a column for the UNIX/epoch time. If there are "special days", such as holidays or sales, create boolean columns for them too.


How to create a training set for sequence labelling I have a bunch of unlabeled text data that I would like to hand label. Are there any tools out there that you can use to produce your own labels? Ideally it would show a sentence at a time and give me the option of choosing a label for each word.

There are several tools for this you could check the Stanford Simple Annotation Tool, also Brat has online demos. For a comprehensive list you could check this question on Quora


Capturing movement importance - logistic regression output I'm studying some event for a set of objects that can be plotted on a square $[0, 100] ^ 2$. I have used logistic regression to calculate probabilities that event occur for different objects and the output can be plotted as below:

We can observe a high density around point $(100, 50)$ and that basically the farther the less probable the event is. A distance to this point is one of the predictors, but there are also few others. What I'm interested in is to capture a potential movement, i.e. how valuable is to move from point A to point B. But I would like to reward also those movements which are made far from the point $(100, 50)$, not only those near to the point. And for the former the absolute increase in probability will be small even if a movement is pretty significant (long distance), while for the latter we can observe a large increase in probabilities even with a tiny move. So I think what I would need is to somehow nullify/smooth effect of distance from $(100, 50)$, i.e. make it less significant for calculating our movement gain. And I don't really know how to accomplish that. I can't calculate the percentage gain, because this results in a really large gains. I think I need some monotonic transformation that would make a large probabilities lower and low probabilities higher. Any ideas what I could apply here?

You can calculate a distance in Euclidean space.

Given your specific goal, you can define a custom distance metric that would the equivalent of a hinge loss. For the most of the space, distance is not defined because the probabilities can be thresholded to zero. For a small section of space near (100, 50), there is a linear or nonlinear distance defined.

An alternative is to switch classifier to a support vector machine (SVM) with a hinge loss which would more directly model your problem.


General question on EDA, correlations, classification, ML I am looking for a general best practices regarding classification and correlations. I created a new predictor feature call it B, based on a certain threshold in a feature A. Now I started to do EDA and I am not sure which feature to include in my EDA, A or B. When I do correlations plots, nothing correlates with feature B, but some features do correlate with feature A. Which one should I take into account then, A or B correlations? Also, how can I make use of those correlations and scatterplots and pairplots anyway and are they important? If I am using random forest or NN, do I even need to bother with all of the pairplots and correlations to extract features from? I have around 150 features and not sure how to approach the problem of which features to use. I haven't found a source saying how to make a proper use of all of this in a real world scenarios. Any help is appreciated.

If the relation between predictors is nearly 0, it's always better to drop that feature, the caveat here depends on the domain knowledge you have.

Did you check the correlation between B and the target variable and also A and target variable? if it's negative drop it, If it's significantly high .i.e greater 0.7, use that as your feature.

Yes, pair plots and scatter plots are really important, but it would be tedious to plot features with 150 variables.


XGBClassifier: make the output of predict_proba ascending regarding a specific feature I'm trying to build a classifier using Xgboost on some high dimensional data, the problem I'm having is that I have the prior knowledge that the output probabilities should be ascending regarding a feature(say x), but I don't know how can I make the model understand this!

For example for a data point with features Feat I want to have:

predict_proba(Feat[x=1]) <= predict_proba(Feat[X=2])

where the rest of the features are the same.

You can enforce such monotonicity with the monotone_constraints (hyper)parameter:

https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html
https://stackoverflow.com/questions/43076451/how-to-enforce-monotonic-constraints-in-xgboost-with-scikitlearn


Stemmer/lemmatizer for Polish language I'm looking for a stemmer/lemmatizer for Polish language, preferably in Python. What would you recommend?
I have a list of ingredients in a recipe. Plural forms are inflected differently, depending on the counter, e.g.: for tomatoes

5 pomidorów
2 pomidory
1 pomidor

I want my parser to recognize all those ingredients as for one product, hence the need for stemming.

There is bunch of lemmatization solutions for polish language.
One of the best implementation is in polish morphosyntactic analyser, which you can download here.

It has bindings to python, but you have to install them manually. It is "morphosyntactic analyser" which means, that you get all possible lemmas for a given word. If you want also disambiguation, you can use this tool which is provided as docker container.

My team is currently working on implementing polish language support for Spacy, and we will cover lemmatization, so tool with good integration and support will be available soon.


How to implement a Restricted Boltzmann Machine manually? I am learning about Restricted Boltzmann Machines and I'm so excited by the ability it gives us for unsupervised learning. The problem is that I do not know how to implement it using one of the programming languages I know without using libraries. I want to implement it manually, which means that I want to use native functionalities of a language as much as possible.

The programming languages I know are Java, C, PHP (my preferred language), JavaScript, R and Python. I am not familiar with TensorFlow or Scikit-Learn or similar stuff.

Thanks in advance

ML From Scratch has an implementation in Python


Metric to use to choose between different models - Hyperparameters tuning I'm building a Feedforward Neural Network with Pytorch and doing hyperparameters tuning using Ray Tune. I have train, validation and test set, train and validation used during the training procedure.
I have different versions of the model (different learning rates and numbers of neurons in the hidden layers) and I have to choose the best model. But I'm unsure on which metric should I use to choose the best model. Basically I don't know the XXXX in this line of code:
analysis.get_best_config(metric='XXXX', mode='min')

Should it be test loss or validation loss?

Essentially, the function of your testset is to evaluate the performance of your model on new data. It mimics the situation of your model being put into production. The validation set is used for optimizing your algorithm.
Personally I would recommend tuning your algorithm using your validation set and using the hyperparameters of the training epoch with the lowest validation loss. The accuracy of the model using these hyperparameters on your testset can be used to estimate the performance of your model in the 'real world'.


How to perform Cleaning of a very large set of addresses I have a dataset where I have a large number of addresses. The problem lies in the fact that many addresses although they are same but haven't been noted down in the same manner. So I want to find these addresses which are similar and clean my data from the duplicate data.

Is there any standard approach or algorithms that can help me? How should I go about this problem?

I guess you are in a situation that different fields of an address are mixed together. Try break the address text into Shingles and then try Locality-sensitive Hashing

Prof. Jeff Ullman's Text Book may help you put together all the techniques you need, starts with Section 3.2


Custom Lambda layer Keras outputs predictions. I get 'An operation has `None` for gradient' error I have a  Lambda layer that takes input from previous layer, makes some preprocessing. Output of the Lambda layer is a prediction, and keras.losses.mean_squared_error is used.

#model
inputs = Input(shape=(len_train_data_columns,))
dense1 = Dense(777, activation='relu')(inputs)
dropout1 = Dropout(0.4)(dense1)
softmax = Dense(3, activation='softmax')(dropout1)
predictions = Lambda(custom_layer,
            output_shape= custom_layer_output_shape,
            arguments= {'experiment_config': experiment_config},
               )(softmax)

model = Model(inputs = inputs, outputs = predictions)
model.compile(optimizer=adam(lr=0.0001), loss=keras.losses.mean_squared_error, metrics=keras.losses.mean_squared_error)

Output of Lambda is (num_rows, 1).

I get ValueError: An operation has None for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

Why is that? I use Keras loss...

I think it comes from your 'custom_layer'. Not all functions defined in keras backend are differentiable (as the error says, for example K.argmax) if you use these functions on your layer and they don't have a defined gradient, it will rise this error. You have to make sure that you use only functions that has a gradient associated. Although there is no an official list of these functions, basic operations (+/-*) are, for example equal_toor greaterare not differentiable.


How would you describe the trade-off between model interpretability and model prediction power in layman's terms? I know it depends on the data and question asked but imagine a scenario that for a given dataset you could either go for a fairly complex nonlinear model (hard to interpret though) giving you a better prediction power perhaps because the model may see the nonlinearities present in the data, or have a simple model (perhaps a linear model or something) with less prediction power but easier to interpret. Here is a very good post discussing ideas on how to interpret machine learning models.

Industries, while being very cautious, are slowly becoming more interested in adopting more complex models! Still they want to know the trade-off clearly? A data scientist perhaps is the one sitting between data team and decision-makers, and often need to be able to explain these stuffs in layman's terms.

I am trying to brainstorm here to see what analogy you would come up with to describe such trade-off to a non-technical person?

Interesting question. I think that you can illustrate this by thinking about different use cases. The one example I've heard that I like is around lending decisions for loan applications. That's an algorithm but, because of regulations, it can't be strictly "black box". The decision has to be, effectively, interpretable because the bank has to give you a reason for decline on the loan. So, there's certainly better algos out there for loans that can give a binary result, but do you want a bank to just tell you yes or no?


LSTM BPTT with wide input? I was following Tensorflow's own time series/LSTM tutorial, and there's something I don't quite understand about the whole process around Backpropagation Through Time (BPTT).
The resources I've found explain how prediction for a time in a future with an offset work (the "offset" in the tutorial, and the t+3rd time instance in the Wikipedia article)
What I don't understand is how does this generalize to where the input width is more than 1?
So in the Wikipedia article we see the case for inputWidth=1, outputWidth=1 and offset=3.
I think I understand if outputWidth>1 then you'd just start using not only the last, but also previous cell predictions as output. I also understand if offset>1 the you just unroll the network into offset number of steps (as shown on Wikipedia).
But what if inputWidth>1? Let's say inputWidth=k (k>1)!
In this case, do you unroll the same RNN multiple times in parallel, and somehow average the weights? Do you unroll the same RNN k times sequentially? What does that look like?

You unroll the same RNN multiple times, but not in parallel. The RNN needs both an input and the previous step, therefore, you unroll the same RNN for the first time step, then for the second, and so on.
For each step it is always the same RNN, so the weights are the same each time:


sine, cosine transformed cyclical features - am I losing information? If I use sine, cosine transformation for cyclical features (e.g. weekday or hour of the day), do I lose information if the first ordinal value was 0 respectively?
Assume hours of the day are encoded as follows: 0, 1, ..., 23
If I apply the sine, cosine transformation, I get 23 data points instead of 24 (cf. plot)

Am I losing information on the first ordinal value (0)?
Many thanks in advance!

Are 0 and 23 supposed to be the same?
If not then you should use $( \sin(2i\pi/24),\cos(2i\pi/24) )$ instead of $( \sin(2i\pi/23),\cos(2i\pi/23) )$ with $i \in \{0,...,23\}$


Splitting Temporal Data I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in the data so I need to split into train/test by some cutoff date (rather than randomly sampling 70/30 split from the data).
The issue I'm running into is that no matter what date I split by, the distribution of my target variable is different in the testing & training set. Earlier targets in the set are a lot larger, and as a result the model fit on the training set tends to way over-predict on the test set.
Any advice on how to tackle something like this?

It sounds likes you have non-stationary data which can be a challenge to model.
One option is to completely discard all older data and only train/test on newer data. This way the model will only capture the newer relationships. This approach assumes there is enough newer data to train a model.
Another option is heavily regularize the model. The goal of regularization  is to increase generalization.


How to generate 12 independent random weights which all add up to one I'm using Palisade's @Risk software with a triangular distribution to generate 12 random weights which must add up to one, but I get a lot of negative numbers.  Is there a straightforward way to set this up?

I am afraid I do not know the software you mention, but I can show you the principles and suggest why maybe you are getting negative numbers.

I will do this in Python, making use of the numerical library, numpy.

I import numpy and generate 12 random integers between 0 and 9 (10 is excluded as upper limit):

In [1]: import numpy as np; samples = np.random.randint(0, 10, 12)

In [2]: samples
Out[2]: array([8, 5, 8, 4, 4, 7, 2, 2, 0, 5, 9, 1])

To scale the values to a range that makes their sum equal to 1, we can do the following. First sum up all values:

In [3]: total = np.sum(samples)

Now simply divide each value by the sum (the division happens here individually for each element of samples:

In [4]: normalised = samples/total

In [5]: normalised
Out[5]:
array([0.14545455, 0.09090909, 0.14545455, 0.07272727, 0.07272727,
       0.12727273, 0.03636364, 0.03636364, 0.        , 0.09090909,
       0.16363636, 0.01818182])

We can see that the result does indeed sum to 1:

In [6]: np.sum(normalised)
Out[6]: 1.0

What you may have is a set of samples that contains some negative numbers, like the following in samples_neg, with ten integers ranging from -5 to +9:

In [7]: samples_neg = np.random.randint(-5, 10, 10)

In [8]: samples_neg
Out[8]: array([ 4,  7,  0,  4, -3,  0,  9, -3,  9,  3])

We can follow the same recipe as before, summing the values and dividing each value by the sum:

In [9]: total_neg = np.sum(samples_neg)

In [10]: normalised_neg = samples_neg / total_neg

We see that the result this time includes negative values, as you mentioned:

In [11]: normalised_neg
Out[11]:
array([ 0.13333333,  0.23333333,  0.        ,  0.13333333, -0.1       ,
        0.        ,  0.3       , -0.1       ,  0.3       ,  0.1       ])

However, this does still satisfy the constraint you originally had, which was that they sum to 1:

In [12]: np.sum(normalised_neg)
Out[12]: 0.9999999999999999        # this is 1, within rounding errors of floating point values

A suggestion would be to first normalise the values in a range of [0, 1] and afterwards, re-weight the values such that their sum is 1.


Neural network back propagation gradient descent calculus So I've drawn a neural network diagram below:

where $x_1, x_2,\ldots,x_m$ are the input layer, $h_1, h_2$ are the hidden layer and $\hat y_1, \hat y_2,\ldots \hat y_k$ are the output layer. In the $W^h_{im}$ notation, it represents the weight, where $i$ is representing to which node it's pointing to in the hidden layer, which is either $h_1$ or $h_2$ and $m$ is representing from which input node.

For example, if $W^h_{11}$, this means it is the weight represented by the line connecting $x_1$ node to $h_1$ node. $W^o_{ki}$, where $k$ is the output node and $i$ is the hidden layer node. Therefore, $W^o_{11}$ is the arrow connecting $h_1$ node with $\hat y_1$ node.

I'm currently working on the back propagation process of updating the gradient descent equation for $W^h_{11}$. Before this, I've only learnt to deal with a diagram with only one output node, but now it's multiple output nodes instead.

So I've attempted the beginning part of the calculus, where I'm not entirely sure if $\hat y$ equation i wrote below is correct and also for the loss function?

For $W^h_{11}$:

$Input: (x,y)$

$\hat{y} = forward(x)$

$\hat{y} = \sum ^K_{k=1} h_1W^o_{k1} + \sum ^K_{k=1} h_2W^o_{k2}$ *am i doing this correctly?

$h_1 = \sigma(\bar{h}_1)$ where $\bar{h}_1 = \sum_{j=1}^{M} x_jW^h_{1j}$

$Loss function: J_t(w) = \frac{1}{2}\sum(\hat{y}-y)^2$ where $\hat y$ is a vector of $\hat y_1,\hat y_2,..\hat y_k$ *would the loss function be included with a summation?

Would like to know if I've done the beginning part right.

We have $$\hat{y}_{\color{blue}k}=h_1W_{k1}^o + h_2W_{k2}^o$$

If we let $\hat{y} = (\hat{y}_1, \ldots, \hat{y}_K)^T$, $W_1^o=(W_{11}, \ldots, W_{K1})^T$, and $W_2^o=(W_{12}, \ldots, W_{K2})^T$

Then we have $$\hat{y}=h_1W_1^o+h_2W_2^o$$

I believe you are performing a regression, $$J(w) = \frac12 \|\hat{y}-y\|^2=\frac12\sum_{k=1}^K(\hat{y}_k-y_k)^2$$

It is possible to weight individual term as well depending on applications.


I want to know which machine learning algorithms can be used for trajectory classifications? I am working on project for clustering of air objects based on their trajectories.
Like I want to train a model on dataset of different flying object's trajectories so later I can predict what type of object is based on trajectory data.
Now trajectory data include 4 things (Altitude, Longitude, Latitude, Time).
So based on set of such dataset we may be able to classify objects like plane, rocket, missile etc.
What I cant figure out is which algorithms can be used?
I first thought about SVM. Later I thought "Long Short Term Memory" can be used. But I am not very sure.
And I am new to machine learning. SO any help is appreciated.

Depending on the amount of data, any classification algorithm can be suitable. LSTM, however, are likely to be an overkill, considering that you probably won't be having much variation in the time series for each object.

Instead of pondering about the algorithm, you'd better think of useful features you can extract from your data. My guess would be that speed, accelerations, and altitude would be most informative.


Confidence of this particual prediction I am looking for a confidence of model to predict well in a given situation.

So I have a model $f$ (generic, let's exemplify with a regression model of explicit form for brevity).
It well fits on the train-set (by looking at say $r^2$), it well performs on the test set (by looking on RMSE).
The fit is fine enough to predict with this model.
Yet I want to know how confident am I to predict in this particular case, so while typically I look at the confidence $C$ of the model:

$C(f)$

while now I want to know what is the confidence of predicting with this model (subject to $x$):

$C(y=f(x))$

and I believe this shall strongly be related to presence of similar observations in the train/test sets and performance on similar observations.

Any hints on this welcomed.

If I understand your question correctly, then this is not possible for a "generic" model. For a generic model, you can only use generic methods. One of the most generic would be bootstrap sampling. In this scenario, you take your model and train it on different bootstrap samples and derive confidence intervals from the distribution of these predictions.

However, it sounds like you are looking for a method that does not require training. In this case, you would need to use models from which confidence intervals can be derived directly based on the model assumptions, such as linear regression, Gaussian processes or Bayesian models.


How to assess that a cross entropy based model has converged I have a question regarding cross entropy convergence using Stochastic Gradient Descent. I am a little bit confused about how the convergence should be assessed. Should we treat the model as converged if the loss hit minimum on any single example or may be a certain number of examples? I am asking because what if the model converged on a single example just randomly.

Pure, hardcore Stochastic Gradient Descent (when you feed just one observation at a time) is not advisable at all. The descent of the Gradient is so noisy that after a certain minimal loss reduction it will stop learning anything. It will wander around the loss function in unpredictable ways. In this case, you're right: there's no way to assess final convergence. Even if the Gradient hits the global minimum, it will jump away from it at the following iteration.

The most powerful technique is a robust version of SGD: Mini-Batch Gradient Descent. In this case, you feed a batch of data at each training iteration (usually of size between 30 to 250, but it's up to you). Batch size is another hyperparameter, therefore you must find a trade off between robustness to noise on one side, and on the other side to avoid get stuck in local minima and lower computational times.


Recommender Model for Human Action in Income Protection Problem Domain
I'm working on a project that involves building a model to provide recommendations on the next best step for Human supervisors to take on income protection claims.
Income protection is an area of insurance that involves providing the insured party with a proportion of their income if ever they fall ill or cannot work. It's a safety net designed to help you if you ever get sick or fall on hard times. Many of our customers get Cancer, Depression, Broken bones, etc. and avail of the insurance service, which is usually provided by their company.
On our side, we have a number of human supervisors that manage the ongoing claims. This means scheduling doctor's appointments, getting up to date medical records and assessments/diagnoses, arranging rehab, talking with the customer, etc. (the cost of all of this is also covered under their insurance). These management items are finite, categorical and described as actions.
The supervisor's job is to help the customer Return To Work (RTW) as soon as possible, both to help the customer heal/get back on their feet, and to reduce costs incurred in handling the claim for the business.
I have datasets that describe what actions each supervisor has taken for every claim for the past 20 years (time series/ sequential data), along with text data that describes the progress of the claim, and finally categorical data describing the claim and customer (type of disease, age, occupation, etc.)
What I'm trying to build is a recommender system which harnesses the data to recommend which action maximises the probability of a Return To Work for a given customer in an on going claim.
What I've Tried
Using the data described above, I have built a neural network to predict the outcome of a closed claim. The NN is a combination of LSTM and Dense layers - LSTM for the sequential action data, Dense for the categorical data and Dense for the TfIdf transformation of the text data. There are 39 possible outcomes, one of which is Return to Work, and the network performs relatively well, achieving ~70% accuracy.
I had planned to use the finished model like so: Iterate through all possible actions, and for each action append it to the claim in question to produce a new "hypothetical" action sequence. For each hypothetical sequence do a feed forward through the trained model and see which action maximises the "score" at the index of the output vector which corresponds to return to work. The action that maximises this score is then the recommendation. A simple toy example for an ongoing claim with 10 actions is shown below:
scores = []
# ongoing_claim.shape = (10, 30)
for action in potential_actions:
    hypothesis = ongoing_claim.append(action)
    # ongoing_claim.shape = (11, 30)
    hypothesis = pad_sequence(hypothesis)
    output = model.predict(hypothesis)
    scores[action] = output[index_for_return_to_work]

The recommendation would be the key corresponding to the highest value in scores.
Unfortunately this doesn't work. No matter which action, when I feed forward the new action sequences in the model the scores stay constant for Return to Work - no variation whatsoever. I think this is because simply adding a tiny new data-point considering the amount of data used to train the model does not add enough variance to perturb the output. I have tried using both sigmoid and softmax activation functions on the final layer but this didn't work either.
Question
What kind of model should I use which harnesses all of the data to then examine an ongoing claim and recommend the action which maximises the probability of a particular outcome (in this case Return to Work)?

It looks like the inverse reinforcement learning problem defined by Stuart Russell as

Given

measurements of an agent’s behaviour over time, in a variety
of circumstances.
measurements of the sensory inputs to that agent;
a model of the physical environment (including the agent’s
body).

Determine the reward function that the agent is optimizing.

It is particularly difficult in your case because you only have observations of a tiny subset of the actual state of a customer and the environment.
I don't think it is solvable without being able to actively explore the actions or without really strong assumptions or a model of the dynamics.
If you can assume that the historical supervisors did their best to help their clients to return to work, your task simplifies to: given the historical actions up to a given point, predict the next historical action.
Even with this simpler problem, you still have to be careful with conditioning, the possible non-stationarity in time, choosing a good model (perhaps similar to what you tried), etc.


Recommending movies with additional features using collaborative filtering I am trying to build a recommendation system using collaborative filtering. I have the usual [user, movie, rating] information. I would like to incorporate an additional feature like 'language' or 'duration of movie'. I am not sure what techniques I could use for such a problem.

Please suggest references or packages in python/R.

Here some resources that might be helpful:

Recommenderlab - a framework and open source software for developing and testing recommendation algorithms. Corresponding R package recommenderlab.

The following blog post illustrates the use of recommenderlab package (which IMHO can be generalized for any open source recommendation engine) for building movie recommendation application, based on collaborative filtering.

Research on recommender systems - a nice webpage with resources on the topic, maintained by Recommenderlab's lead developer Michael Hahsler.

Mortar Recommendation Engine - an open source customizable recommendation engine for Hadoop and Pig, written in Python and Java. Company, sponsoring the development of this project, Mortar Data, offers general commercial cloud platform for development and hosting data science software projects, including ones based on the Mortar Recommendation Engine (development and hosting of public projects are free): http://www.mortardata.com. Mortar Data provides help in form of public Q&A forum (https://answers.mortardata.com) as well as a comprehensive tutorial on building recommendation engine using open technologies (http://help.mortardata.com/data_apps/recommendation_engine).

"Introduction to Recommender Systems" - a relevant Coursera course (MOOC), which content and description provide additional resources on the topic.

PredictionIO - an open source machine learning server software, which allows building data science applications, including recommendation systems  (source code is available on GitHub). PredictionIO includes a built-in recommendation engine (http://docs.prediction.io/current/engines/itemrec/index.html) and supports a wide range of programming languages and frameworks via RESTful APIs as well as SDKs/plug-ins. PredictionIO maintains an Amazon Machine Image on AWS Marketplace for deploying applications on the AWS infrastructure.

Additional open source software projects, relevant to the topic (discovered via MLOSS website on machine learning open source software: http://www.mloss.org):

Jubatus
MyMediaLite
TBEEF
PREA
CofiRank

The following relevant R blog posts are also interesting:

"Simple tools for building a recommendation engine"
"Recommendation System in R"


Currency Normalization for Salary Prediction I have a dataset (350k data points) with data of employees across different regions over the last 10 years. The dataset consists of their skills, the region they are in, the industry, their current role, their salary in the respective currency.
After doing some analysis, I have found 60% of the salaries are in SGD, 30% in INR, and the rest are divided across 15 other currencies.
Is it recommended that I have a model for each currency or is there a way I can convert all the currencies to a universal value so I can use all my data points to train?
Currently, I have used the 40% of the points available in SGD to train a random forest model and I have found that the results on the test set are reasonably accurate. For this model, I have considered skills, role, and industry as features and nothing else. Is there any better model I can explore?
Thank you

Within each currency (say SGD or INR), you can divide the salary by quartiles, for e.g the top 25 percentiles of salaries within each currency can be assigned a value of 4, the next 25th percentile a 3 and so and so forth. This way you are able to normalize salaries across currencies.


Why use regularization instead of decreasing the model Regularization is used to decrease the capacity of a machine learning model to avoid overfitting. Why don't we just use a model with less capacity (e.g. decrease the number of layers). This would also benefit the computational time and memory.

My guess would be that different regularization methods make different assumptions of the dataset. If so, what assumptions are made for the common regularizations (L1, L2, dropout, any other)

Thanks in advance!

Regularization does decrease the capacity of the model in some sense, but as you already guessed, different capacity reductions result in models of different quality and are not interchangeable.

L1 can be interpreted as making the assumption that the influence of different factors (represented by neurons) on each other shouldn’t be assumed without significant support by data (i.e. the gain achieved by larger influence has to outweight the L1 loss associated with increased absolute value of the parameter that „connects“ them).

L2 does the same, but makes this dependent on the connection strength, i.e. very light connections basically need no support (and are therefore not driven further to exact zero) and very large connections are almost impossible.

Dropout can be interpreted as training a large amount of smaller networks and using the approximated average network for inference: „So training a neural network with dropout can be seen as training a collection of 2^n thinned networks with extensive weight sharing, where each thinned network gets trained very rarely, if at all.“ Dropout: A Simple Way to Prevent Neural Networks from Overfitting

All these methods make certain network parameter combinations highly improbable or even impossible to achieve for a given dataset, which otherwise could have been the result of the training. In this sense, the capacity of the model is reduced. But as one can imagine, some capacity reductions are more useful than others.


kNN vs Logistic Regression Good day, I had this question set as optional homework and wanted to ask for some input.

Suppose an individual was to take a data set, divide it in half into training and test data sets and then try out two different classification procedures.
First they use logistic regression and get an error rate of 20% on the training data and 30% on the test data. Next they use 1-nearest neighbours and get an average error rate (average over both test and training data sets) of 18%. According to these numbers, which method would you prefer to use for classification purposes (of new observations)? Why?

I am inclined to say kNN as it is a rather flexible approach and provides a lower error on average. But this doesnt sound formal enough and may likely be a flawed and naive conclusion. Any input?

Hint: what's the error rate of 1-nearest-neighbor on any training set?

From there, determine the test error on this dataset.


Efficient self study plan I am hoping for a bit of guidance from experienced practitioners / academics.

I want to work through the Bishop ML book, but have minimal background.

What is the fastest way to get the pre-requisites (specific books would be appreciated)?

From searching around I found this potential self-study path:

Statistical Inference - Casella / Berger
Probability Theory and Examples - Durrett
Linear Algebra - Hoffman / Kunze

I checked these books out from the library, but they will take me over a year to work through thoroughly, so it does not seem to be practical.

I have searched around on the internet, but most of the advice doesn't list any specific books, just what subjects I should learn.

About me

Graduated in an unrelated discipline many years ago
Willing to dedicate many hours to this (I am doing this to build a background for a degree in machine learning)
I can code pretty well due to my job

I'm not super familiar with Bishop, but it looks like Durrett, which deals with measure-theoretic probability, would be overkill.  Plus you would probably have to add real analysis to your prerequisites.  I would just work through the first 4-5 Casella chapters on probability and random variables.

You probably also don't need to go through the whole linear algebra book to get started.  Just make sure you understand the basic matrix operations well enough to deal with multivariate calculus (ex. taking multivariate derivatives and computing multivariate integrals).

The second half of Casella would likely be useful, but I suspect not strictly necessary.

I think if you can handle those things you'll have most of Bishop covered; you can always pick up more as needed.


Explaining instances of a machine learning regressor I trained an XGBoost regressor, and I'm looking for a way to interpret new instances that the model will receive in production.
There's a nice solution for classification models (Lime), but it doesn't offer explanations to regression problems.

Does anyone know of an existing "explainer" for regressions?

Lime introduced support for regression problems in their latest version. A notebook with examples is available here.


Understanding and find the best eps value for DBSCAN I'm trying to run the DBSCAN algorithm on this .csv. In the first part of my program I load it and plot the data inside it to check its distribution. This is the first part of the code:
import csv
import sys
import os
from os.path import join
from sklearn.cluster import DBSCAN
import matplotlib.pyplot as plt
import numpy as np

def load_data(file_path, file_name):
   with open(join(file_path, file_name)) as csv_file:
       data_file = csv.reader(csv_file,delimiter=',')
       temp1 = next(data_file)
       n_samples = int(temp1[0])
       print("n_samples=")
       print(n_samples)
       n_features = int(temp1[1])
       temp2 = next(data_file)
       feature_names = np.array(temp2[:n_features])

       data_list = [iter for iter in data_file]

       data = np.asarray(data_list, dtype=np.float64)

   return(data,feature_names,n_samples,n_features)

# --- Main program ---

file_path="Datasets/"
file_name3="CURE-complete.csv"
data3,feature_names3,n_samples3,n_features3 = load_data(file_path, file_name3)
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111)
fig.subplots_adjust(top=1)
ax.set_title('Dataset n. 3 of data points')
ax.set_xlabel(feature_names3[0])
ax.set_ylabel(feature_names3[1])
plt.plot(data3[:,0], data3[:,1], '.', markersize=1.2, markeredgecolor = 'blue')
plt.show()

This is how the data are in represented:

Since I need to know how many clusters should I split the dataset into, I used the Means algorithm which returned 4 (this is the elbow of the diagram, the best value):
sse = {}
for k in range(1, 11):
    kmeans = KMeans(n_clusters=k, random_state=0).fit(data3)
    sse[k] = kmeans.inertia_

kn = KneeLocator(
    list(sse.keys()),
    list(sse.values()),
    curve='convex',
    direction='decreasing',
    interp_method='polynomial',
)

plt.figure()
plt.plot(list(sse.keys()), list(sse.values()))
plt.vlines(kn.knee, plt.ylim()[0], plt.ylim()[1], linestyles='dashed')
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.show()

This is the diagram:

Then, I run the DBSCAN algorithm with the following parameter: min_sample = 10 (fixed value, I need to use it) and eps = 2. This is the code:
np.random.seed(5)
dbscan2 = DBSCAN(eps=2, min_samples=10).fit(data3)
fig = plt.figure(figsize=(20,10))

ax = fig.add_subplot(111)
fig.subplots_adjust(top=1)
ax.set_title('Clustered points in dataset n. 3')

ax.set_xlabel('x')
ax.set_ylabel('y')

# set the list of colors to be selected when plotting the different clusters
color=['b','g','r','c','m','y','k','w']

# number of clusters given by the diagram
k=4

#plot the dataset
for clu in range(k):
    # collect the sequence of cooordinates of the points in each given cluster (determined by clu)
    data_list_x = [data3[i,0] for i in range(n_samples3) if dbscan2.labels_[i]==clu]
    data_list_y = [data3[i,1] for i in range(n_samples3) if dbscan2.labels_[i]==clu]
    plt.scatter(data_list_x, data_list_y, s=10, edgecolors='none', c=color[clu], alpha=0.5)

plt.show()

This is the result:

Is my process incorrect? I do not understand why the plots looks so different. I would expect a digram like to the first one but with different color for each cluster.

KMeans and DBSCAN are two different types of Clustering techniques.
The elbow method you used to get the best cluster count should be used in K-Means only.
You used that value i.e. K=4 to assign colors to the scatterplot, while the parameter is not used in DBSCAN fit method.
Actually that is not a valid parm for DBSCAN
You will have to control "esp" to control the number of Clusters
Fit with esp=6 resulted in 112 Clusters. You only need these few lines of code
dbscan2 = DBSCAN(eps=6, min_samples=10).fit(data3)
fig = plt.figure(figsize=(12,7))

# max(dbscan2.labels_) # This is the number of Cluster
plt.scatter(data3[:,0], data3[:,1], s=10, edgecolors='none', c=dbscan2.labels_, alpha=0.5, cmap='hsv')


Getting the earliest date (duplicates due to several call ids and agents) I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates:

select customer, agent, min(call)
from call_data;

I still get the following:

This solved the problem:

select distinct on (customer)
customer, agent, min(call)
from call_data;

I am using PostgreSQL.


Printing Feature Contributions in a Random Forest algorithm from the Treeinterpreter library leading to errors I am working on a dataset where I predict the risks of developing pancreatic cancer with respect to a number of variables. I have created a random forest, and want to find the feature contributions. I have already used the "Treeinterpreter" library, resulting in a contributions array that is three-dimensional. I want to display the contributions in the array beside the name of the factor/variable. I have used the code below to do so, however, the code responsible for displaying the contributions does not work. I have tried multiple methods, including converting the dataframe to a numpy array, and other methods such as .all() and .any(). However, none are producing the desired result.

What can be the right way to display the feature contributions with respect to each of the feature it represents?

     # -*- coding: utf-8 -*-
"""
Created on Mon Apr 15 13:39:19 2019

@author: GoodManMcGee
"""

import pandas as pd
from sklearn.metrics import accuracy_score
from sklearn import tree
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn.metrics import confusion_matrix
from sklearn.ensemble import RandomForestClassifier
from IPython.display import Image
from sklearn.tree import export_graphviz
from treeinterpreter import treeinterpreter as ti
import matplotlib.pyplot as plt
import numpy as np
import itertools

data = pd.read_csv("pancreatic_cancer_smokers.csv")
target = data['case (1: case, 0: control)']
data.drop('case (1: case, 0: control)', axis=1, inplace=True)
x_train, x_test, y_train, y_test = train_test_split(data, target, test_size = 0.2)
clf = RandomForestClassifier(n_estimators=100)
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
clf_accuracy = accuracy_score(y_test, y_pred)
clf_pred, clf_bias, contributions = ti.predict(clf, x_test)

#The code below was taken from DataDive's treeinterpreter tutorial.
#The aforementioned messages applies to all code between the underscores
#///////////////////////////////////////////

for i in range(len(x_test)):
    print ("Instance", i)
    print ("Bias (trainset mean)", clf_bias[i])
    print ("Feature contributions:")
    for c, feature in sorted(zip(contributions[i], data.feature_names),
                             key=lambda x: -abs(x[0])):
#An error occurs in the "data.feature_names" method in the code above:AttributeError: 'DataFrame' object has no attribute 'feature_names'. I have tried referenceing columns from datasets also, but that also leads to errors: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
        print (feature, round(c, 2))
    print ("-"*20)
#///////////////////////////////////////////

Try this in the last part of your code:

for i in range(len(x_test)):
    print ("Instance", i)
    print ("Bias (trainset mean)", clf_bias[i])
    print ("Feature contributions:")
    for c, feature in sorted(zip(contributions[i,:,0], data.columns),key=lambda x: -abs(x[0])):
        print (feature, round(c, 2))
    print ("-"*20)

The problem is that you are sorting contributions without taking into account that contributions is a 3D array and the column names is accesible with data.columns, not data.feature_names.


How to start the process of coming up with the predicted math score? I am working on a problem(non competition) from hacker rank https://www.hackerrank.com/challenges/predict-missing-grade

Basically you're given test data of a bunch of students of their scores in other subjects but math and you are to predict their score in math based off all their other test scores. Say you were passed data of

{"SerialNumber":1,"English":1,"Physics":2,"Chemistry":3,"ComputerScience":2}

How would you go about generating that student's score in mathematics or coming up with a prediction engine to generate the math score? I know that's the whole point of this question but can someone give me a hint or a resource to go to so I can have a chance of figuring this out and actually get started? I really want to learn.

What you are looking for is a machine learning algorithm. Although the easiest way would be to take the average scores and use that, there are much more accurate ways to make predictive models.

This was the first data science tutorial I did. It's perfect for getting started. Here is it in R and in Python.

If you're looking for a short answer, something you can just look up how to implement, I'd check out Random Forests.


Training a model for fall detection My parents are elderly and a fall is a BIG DEAL.  I'm pretty good at coding and such.  So I thought I'd use some fall datasets and plug them into a machine learning system.  As you can see from this statement, I am new to machine learning.  How should I start getting my first model?  Literature has pointed me to trying a decision tree model as better than KNN...do I pick up TensorFlow and start plying a model with datasets, or how would you start?  Thank you and I apologize for my noob question.

I don't know of such a dataset, but I would suggest you to borrow ideas from mobile phone accelerometer projects. e.g.  the links below can be a good inspiration.

To proceed, I would record their daily movement and generate the normal movement thresholds (probably you could just use an off the shelf android phone utilizing its motion sensor).

Based on the collected dataset, you will define what are the normal movements (e.g. you can start with fixed rules). Later you dynamically adjust based on the feedback from your parents.

Concerning machine learning algorithms, After you collect high dimensional data for long enough time you can proceed with the algorithms you mentioned above.

(e.g. setting it as a classification algorithm)

(speed, time, pt1, pt2, ...) -> fall
(speed, time, pt1, pt2, ...) -> no_fall

good luck!

[1] step detection https://github.com/danielmurray/adaptiv

[2] sensors overview https://developer.android.com/guide/topics/sensors/sensors_overview


Would K-means be Appropriate to Use with Four or More Variables? Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, although I haven't done it before, that you can plot a k-means cluster with three variables utilizing the x, y, and z axes. But now I'm currently in a position where I have four variables, normalized by their z-scores, in which I'm not sure how to use the k-means clustering appropriately. Should I be using a k-means cluster in this circumstance?

Thanks

You can use k-Means clustering in all the dimensions you need. This technique is based on a k number of centroids that self-adjust to the data and "cluster" them. The k centroids can be defined in any number of dimensions.

If you want to find the optimal number of centroids, the elbow method is still the best. You iterate the algorithm changing the value of k each time, and record how much error the clustering is producing; once you're done you plot the error levels against each k value and check the optimal k visually. There is plenty of tutorial on it, you can start here for example.


Binary Classification of a ship Dataset What are the best options for a big dataset classification?
I am thinking of 2 solutions-

1.Autoencoders

2.PCA(Principal Component Analysis)

I think the first approach is better.Does it works for my problem?Or maybe someother solution is better?

Thank you

None of the methods you described may classify a dataset alone whereas both can be used to transform your data into another domain in an unsupervised fashion.

PCA projects your data onto n-orthogonal components. A trained encoder (first component of the autoencoder) can project your data onto a latent space.

Both of those representations can be used in conjuction with a classifier, such as a decision tree, to form a classification pipeline.

There is no globally best solution, try them both for your specific problem. PCA is, in my opinion, way faster to try out using sklearn instead of building your own autoencoder using PyTorch or Keras. Give PCA a first go and if the results are poor, try using an autoencoder.


Can neural networks have multi-dimensional output nodes? I'm trying to understand what's possible with TensorFlow's output layer. Specifically, are outputs always a flat array?
Since a neuron (or 'unit', in TF) has just one number, and there is only one set of outputs, it seems that output must have a single dimension. With one-hot probabilities, this is easy to understand. But what about an image?
If my output is going to be a picture, can I have TF output a multi-dimensional array of pixels, e.g. [[r0, g0, b0], [r1, g1, b1], ...]? If so, how would that network be constructed? How would I define the output layer's dimensionality/shape?
The only param I know of that defines output shape is this, from tf.layers.dense, which seems inherently one-dimensional:

units (number) Positive integer, dimensionality of the output space.

Any help you can provide is greatly appreciated!

Reference: https://js.tensorflow.org/api/latest/#layers.dense

The output layer does not have to be 1D (excl. batch size) but even if it is, it does not necessary mean you cannot transform it to a n dimensional space. Consider an autoencoder used to reconstruct an image:

In the simplest case we could flatten a image (e.g. 24 x 24 pixels) and learn a network to predict the 24 x 24 pixels (output a 1D image). These pixels can then be transformed back to an 2D image (https://www.tensorflow.org/tutorials/generative/autoencoder). So in other words, even if your network outputs a 1D shape, nothing prevent you from reconstructing it to a higher space.

We can achieve similar results as stated in the point above, by using an encoding network (convolutional + pooling layers) followed by a decoding layer (transposed convolutional + up sampling layers). In this case you can effectively generate a 2D image directly (https://www.tensorflow.org/tutorials/generative/cvae).

You can also look at image segmentation networks for inspiration of how higher dimensions outputs can be generated.


Classification problem where one attribute is a vector Hello I am a layman trying to analyze game data from League of Legends, specifically looking at predicting the win rate for a given champion given an item build.
Outline
A player can own up to 6 items at the end of a game. They could have purchased these items in different orders or adjusted their inventory position during the course of the game.
In this fashion the dataset may contain the following rows with:
   champion id   |                 items ids               | win(1)/loss(0)
----------------------------------------------------------------------------
       45        |   [3089, 3135, 3151, 3157, 3165, 3285]  |       1
       45        |   [3151, 3285, 3135, 3089, 3157, 3165]  |       1
       45        |   [3165, 3285, 3089, 3135, 3157, 3151]  |       0

While the items are in a different order the build is the same, my initial thought would be to simply multiply the item ids as this would give me an integer value representing that combination of 6 items.
While there are hundreds of items, in reality a champion draws off a small subset (~20) of those to form the core (3 items) of their build. A game may also finish before players have had time to purchase 6 items:
                items ids
------------------------------------------
   [3089, XXXX, 3151, 3285, 3165, 0000]
   [XXXX, 3285, XXXX, 3165, 3151, 0000]
   [3165, 3285, 3089, XXXX, 0000, 0000]

XXXX item from outside core subset
0000 empty inventory slot

As item 3089 compliments champion 45 core builds that have item 3089 have a higher win rate than core builds which are missing item 3089.
The size of the data set available for each champion varies between 10000 and 100000. The mean is probably around 35000.
Questions

Is this a suitable problem for supervised classification?
How should I approach finding groups of core items and their win rates?

1) If you want to build a model with:

Input: Items bought
Output: Win/Loss

then you will probably want to learn a non-linear combination of the inputs to represent a build.  For example item_X may have very different purpose when paired with item_Y than with item_Z.

For the input format, you may consider creating a binary vector from the item list.  For example if there were only ten items, a game in which the champion purchased items 1,4,5,9 (in any order) would look like row 1; a game where he also purchased item 2 and 7 would look like row 2:

item_ID   | 0  1  2  3  4  5  6  7  8  9
________________________________________
champion_1| 0  1  0  0  1  1  0  0  0  1
champion_1| 0  1  1  0  1  1  0  1  0  1

There are a variety of models that might suit this task.  You might use decision trees for interpretability.  A simple neural net or SVM would likely also do a good job.  These should all be found in most basic ML packages.

2) The win rates of various items are directly computable.  Simply count the number of times a champion used the items in question and won and divide by the total number of times a champion used that item combination.  You can do this for any given group size (1 to 6)


Train Naive Based Classifier For (a) I have calculated $P(G)=\frac{5}{8}$, $P(O|G)=\frac{2}{5}$, $P(B|G)=\frac{1}{5}$, $P(C|G)=\frac{4}{5}$, and $P(A|G)=\frac{4}{5}$. Now how do I calculate the maximum likelihood estimate of these values?

And how do I go about part (b)? I get that $O,B,C,A$ are independent so I can multiply them to get joint probability. But for values like $O_i$ for sample $i=9$, that is just $0$, since sample 9 doesn't have outdoor seating. And how am I supposed to calculate $P(G_i)$ if I don't know what $G_9$ is?

Done: MLE is a somewhat abstractly defined concept, but in
essence it is your best guess at a parameter. In this case we assume
that the observed frequency is your best guess.
You want to calculate the probabilty of your observation (HasOutdoorSeating=0 in this case), given
IsGoodRestaurant=1. That probability is not 0 (check the first sample for instance)


Collaborating on Jupyter Notebooks I have prepared Jupyter Notebook with some findings and I shared it with other team members through GitHub to get their feedback in a written form. It used to work like this when working together on a piece of code but does not work for Jupyter Notebook. In GitHub that would mean commenting on HTML or JSON level (internal markup for .ipynb files), not on the document level. An alternative would be for team members to clone the repo and puts inline comments in the document. That's an additional effort for other team member I would like to avoid.

What is the way you collaborate, peer review and provide feedback when working on Jupyter Notebooks?

CoCalc provides Jupyter notebooks with realtime collaboration, unlike Colab, Kaggle, etc.  You just make a project drag and drop ipynb and data files, add collaborators, and everybody can edit everything simultaneously.  You can also share content publicly at the share server.   I think CoCalc is currently the overall most mature of the realtime Jupyter collaboration platforms (and it is the only open source one 4), but Deepnote is another option that is more focused on data science (but is closed source).


Data Conversion to Time Series in R I am having Sales data of 2018 and 19. I need to convert to time series. The data is not having daily sales

View(df)

Sales          Date

75606         11/01/18

95620         16/01/18

55666         21/01/18

56270         29/01/18

45600         11/02/18

65620         18/02/18

50660         26/02/18

76200         10/03/18

75606         20/03/18

95620         27/03/18

55666         28/03/18

56270         29/03/18

45600         17/04/18

65620         24/04/18

50660         02/05/18

76200         16/05/18

And so on till 2019 Dec

Here the problem is if I convert to time series by using following code

tsn <- ts(df[,1], start =c(2018, 1), frequency=12)

It's considering only 1st 12 rows
The output will be having 1st 12 values

So how to convert the whole data into time series. Should I aggregate monthly wise and convert that into time series or is there any way to do.

This is an example of multi-parallel-time-series.

Example shared on github MultiParallelTimeSeries, believe that will be helpful.


Finding the best "depth" of ICD9 codes with pseudo-hierarchical clustering Here is a common problem in health care modeling.  Did I just invent a new algorithm or has someone already thought of this?

The goal is to find the most homogeneous partition of patients by medical costs using ICD9 codes.  There are 13,000 individual codes in the data set, so using the full code results in many only having a few observations.

ICD9 codes are in a nested hierarchical structure.  For instance, all infectious diseases are 001-139, one particular disease is Cholera (ICD9 001), and this can have several other suffixes which further specify the illness.  This is a "drill down" so to speak.  001.0 and 001.1 are different types of the same disease.

The goal is to find the best level of detail for the data set.  For example, say that these codes are binary for simplicity, and are only 3 digits long.  Then the only possible codes are 000, 010, 011, 001, 111, 110, 101, 001.

One way of searching through all possible substrings of these would be to use a decision tree.  The x matrix would be all substring combinations, and the response would be the average costs for that particular code.

X would have a column for substring 0, 01, 010, 011, etc.  Each row of X would have 2^3*3 = 24 columns.  Each row would represent a single code.

The top 3 rows of X would look like this.  The response Y would be the average dollar amount of healthcare costs for that ICD code.

Once the data is in this format, a decision tree (or other model) could be used in order to determine which substring prefixes should stay in the model.  Because different ICD codes have different number of patients, this would be used as a weight vector.  The model could be tuned to allow deeper trees (more digits).

Is there a name for this already?
Thanks

It's called a prefix tree.

And because the ICD codes are human made, and humans tend to think in categories, there is indeed such a structure in this codes.

It's just that humans first thought of the structure, then made these codes to reflect that structure, not the other way round. What you are now doing is reverse-engineering this...


Prediction with non-atomic features I would like to use non-atomic data, as a feature for a prediction.
Suppose I have a Table with these features:

- Column 1: Categorical - House
- Column 2: Numerical - 23.22
- Column 3: A Vector - [ 12, 22, 32 ]
- Column 4: A Tree - [ [ 2323, 2323 ],[2323, 2323] , [ Boolean, Categorical ] ]
- Column 5: A List [ 122, Boolean ]

I would like to predict/classify, for instance, Column 2.

I am making something to automatically respond to questions, any type of question, like "Where was Foo Born?" ...

I first make a query to a search engine, then I get some text data as a result, then I do all the parsing stuff (tagging, stemming, parsing, splitting ... )

My first approach was to make a table, each row with a line of text and a lot of features, like "First Word", "Tag of First Word", "Chunks", etc...

But with this approach I am missing the relationships between the sentences.

I would like to know if there is an algorithm that looks inside the tree structures (or vectors) and makes the relations and extract whatever is relevant for predicting/classifying. I'd prefer to know about a library that does that than an algorithm that I have to implement.

When it comes to dealing with many disparate kinds of data, especially when the relationships between them are unclear, I would strongly recommend a technique based on decision trees, the most popular ones today to the best of my knowledge are random forest, and extremely randomized trees.

Both have implementations in sklearn, and they are pretty straightforward to use. At a very high level, the reason that a decision tree-based approach is advantageous for multiple disparate kinds of data is because decision trees are largely independent from the specific data they are dealing with, just so long as they are capable of understanding your representation.

You'll still have to fit your data into a feature vector of some kind, but based on your example that seems like a pretty straightforward task, and if you're willing to go a little deeper on your implementation you could certainly come up with a custom tree-splitting rule without actually having to change anything in the underlying algorithm. The original paper is a pretty decent place to start if you want to give that a shot.

If you want pseudo-structural data from your text data though, I might suggest looking into doc2vec, recently developed by Google. I don't think there are any good open-source implementations now, but it's a pretty straightforward improvement on the word2vec algorithm, which has implementations in at least C and python.

Hope that helps! Let me know if you've got any more questions.


Is it good practice to convert columns with a number to a range between 0 and 1? Relatively new to data science. I heard something about converting columns which contain integers into a range between 0 and 1. I think the reasoning was that so all the columns will be more similar in their range. I think along with that there might have been a step of removing outliers (very high integers) to that they wouldn't cause all other results be skewed as a low fraction.

Is this accurate?

If yes, is there an easy command to make it happen with a Pandas dataset?

I think there si some confusion with the quantile transformer : https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer it actually scale the values to a 0-1 range. the goal is to get a uniform distribution. It serves a different purpose than minmax scaler imho. Notably it could be helpfull if you get outliers.

To get a bigger overview on what different scalers do :  https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py


What has to be considered when creating a radar chart? I'm currently analyzing a WhatsApp chat history. One thing I'm interested in is the time when the two people communicated. I thought this is a perfect use case for a radar chart (aka spider chart, star chart). So here is one example:

I find this super hard to read. In contrast, have a look at the following bar chart with exactly the same information:

Here it is way easier for me to see patterns:

0 - 5: No activity (sleeping)
6 - 9: A morning peak (messages after waking up)
10 - 16: Little activity (work times)
17 - 19: Increased activity (work is done)
20 - 21: High activity
22 - 23: Sinking activity (going to bed)

Question

What are typical use-cases for radar charts? Are there specific requirements on the data? When are they better than bar charts? How should I set the ticks for radar charts?

Before people starting to close-vote: I could imagine very well that there are similar studies like for color maps - how many errors are done in analysis? How fast can people pick up main insights?

Example code

In case you want to improve the existing visualizations:

# core modules
from math import pi

# 3rd party modules
import matplotlib.pyplot as plt
import pandas as pd

def main():
    df = pd.DataFrame({'date': [209, 13, 1, 2, 1, 25, 809, 3571, 1952, 1448, 942, 1007, 1531, 1132, 981, 864, 975, 2502, 2786, 2717, 3985, 4991, 2872, 761]},
                      index=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23])
    create_bar_chart(df)
    create_radar_chart(df)

def create_bar_chart(df, output_path='bar_chart.png'):
    df.plot(kind='bar')
    plt.savefig(output_path)

def create_radar_chart(df, output_path='radar_chart.png'):
    """
    Create a radar chart.

    Parameters
    ----------
    df : pandas.DataFrame
        Has a column 'date'
    """
    values = df['date'].tolist()

    df = df.T.reset_index(drop=True)
    df.insert(0, 'group', 'A')

    # number of variable
    categories = list(df)[1:]
    N = len(categories)

    # What will be the angle of each axis in the plot?
    # (we divide the plot / number of variable)
    angles = [n / float(N) * 2 * pi for n in range(N)]
    angles += angles[:1]

    # Initialise the spider plot
    ax = plt.subplot(111, polar=True)

    # Draw one axe per variable + add labels labels yet
    plt.xticks(angles[:-1], categories, color='grey', size=8)

    # We are going to plot the first line of the data frame.
    # But we need to repeat the first value to close the circular graph:
    values = df.loc[0].drop('group').values.flatten().tolist()
    values += values[:1]
    values

    # Plot data
    ax.plot(angles, values, linewidth=1, linestyle='solid')

    # Fill area
    ax.fill(angles, values, 'b', alpha=0.1)
    plt.savefig(output_path)

if __name__ == '__main__':
    main()

Whenever it comes about a specific graph critique, I check the data-to-viz.com (I have not affiliation with it)

About the Radar or Spider Charts, you can find some drawbacks here. Let me summarize them for the readers.

Circular layout = harder to read

This is a common problem in every circular layout. The human eye can easily understand and compare data in a single vertical or horizontal axis, but not with the same convenience in circular plots.

Supporting the ranking

Bar charts and other similar plots, such as lollipops allow you to instantly graph the ranking of the data. Which is higher and which lower. This is not easy at all in Radar Charts where the are no starts and ends.

Category order has a huge impact

You need to be really careful with the order you will put your categories. Readers meant to focus a lot on the shape of the radar which will change significantly when you change the order of your categories.

About scales

It's common in Radar Charts to compare different variables with different scales at the same time. In those cases, you need to clearly specify the scale for each of them, otherwise, readers expect to have the same scale and unit everywhere.

Overplotting

One series of data is nice. A second one adds some eye-catching details. If you need more than that, it will become unreadably quite easy.

Over-evaluation of differences

Humans are able to understand the difference in horizontal or vertical axis linearly. However, when it comes to the area of a shape, it becomes quadratically. As a result, it's easy to overestimate a difference between two points in the Radar Chart.

Workarounds

Unfortunately, there aren't many things to do. If you believe your Radar Chart is not readable enough, then you can only change to a bar chart or a lollipop plot.

In your example, you might try to put the scale outside to keep the inside area more clear. Like this one

Image source: https://blog.scottlogic.com/2011/09/23/a-critique-of-radar-charts.html#occlusion


Is label-encoded data quantitative or qualitative? If you label-encode something which is qualitative, like brand of toothpaste or colour of hair, would you describe the resulting data as quantitative since it is now expressed in numbers? Or would you continue to describe the data as qualitative, since it is still describing the same underlying information?

Mapping to numbers in a random fashion e.g. Label encoder doesn't make it quantitative. It's still qualitative.
It just that most of the library implementation of ML Models only expect numeric data.
So, Label encoding will work fine with models that don't do any quantitative operations on any two encoded values i.e. Tree-based models that simply use the label to divide the space
If you use it with models that treat the mapping as quantitative e.g. Neural Networks then the result might be error-prone e.g. 10 will become 2 times 5 although no such relationship was intended.


SVD for recommendation engine I'm trying to build a toy recommendation engine to wrap my mind around Singular Value Decomposition (SVD). I've read enough content to understand the motivations and intuition behind the actual decomposition of the matrix A (a user x movie matrix).

I need to know more about what goes on after that.

from numpy.linalg import svd
import numpy as np

A = np.matrix([
  [0, 0, 0, 4, 5],
  [0, 4, 3, 0, 0],
  ...
])
U, S, V = svd(A)

k = 5 #dimension reduction
A_k = U[:, :k] * np.diag(S[:k]) * V[:k, :]

Three Questions:

Do the values of matrix A_k represent the the predicted/approximate ratings?
What role/ what steps does cosine similarity play in the recommendation?
And finally I'm using Mean Absolute Error (MAE) to calculate my error. But what I'm values am I comparing? Something like MAE(A, A_k) or something else?

You can use SVD to build a recommendation engine, but I don't think it's the best way to get intuition around what's going on under the hood. Regardless, here's a presentation with more details, I'd recommend reviewing slide 9.

And to answer your questions:

A_k represents an embedding dimension (i.e. the low-rank approximation) that is used to predict the user-rating matrix.
The cosine similarity is just the dot product for user $i$ and item $j$, which maps to the predicted rating for user $i$ and item $j$. The dot product is what defines the users and items as being similar.
Yes, you should use the MAE on A and A_k. You may prefer to use MSE instead. This measures the quality of your predictions for user $i$ and item $j$. Note, this is obviously the MSE of a matrix, which is the Frobenius Norm.

I think an easier way to understand SVD is to see it applied to image compression for different components. See this presentation here.


Investigate why data is missing? After finding out reasons, what should I do next? x1 x2 x3 x4 - - - - - - x10 .... x21 x22 x23
 1                                           |
 .         Complete                          |
 .           data                            |
 88                                          |
 89                                          |
 90 ------------------------------------------
 .          |              |
 .complete  |   missing    |  complete
 .          |              |
 .          |              |
 . -------------------------------------------
 100
 101           Complete data
 102------------------------------------------

The dataset looks like this. 10 % of the data is missing. It doesn't appear missing at random.

From row 90 to 99. Variables x4 to x10 are missing. All other rows do not have any missing values. It is not missing at random. Is there any statistical way to investigate why they are missing.

My initial plan is to create a new column, 0 is not missing, 1 is missing.
Run a logistic regression on non-missing columns. Is this the correct way to do it or not?

 My questions:
     How should I investigate why data is missing by just playing around with the data set?

     If I found out the reasons or data is not missing not random, What should I do next?

Add the binary column but start with summary statistics by group (missing/not missing) for all of the x’s. That may reveal an implied ‘why’, eg the values are missing whenever some other x is always above/below some value.


"ValueError: Index contains duplicate entries, cannot reshape" error when I try to use pd.MultiIndex.arrays I have data which includes id , gender , collected time test name and Test values , Units of measurement

Test Names will include all tests that a patient taken and Value col will have its corresponding test result.

I want to analysis on only certain tests and retrieve corresponding test values from "value" col . The analysis will be on those tests and their values , so I thought it would be good idea to pivot on those test names and test values. However when I add TS col I get an error and adding any other test name in the multiindex code does not throw an error.

Steps:

 df_s.head(30).dropna()

Here in the below screenshot we can see there multiple test taken for each requisition id:

In the below code Iam only getting tests which I want to do analysis
1#

df_s2 = df_s[df_s['Test'].isin(['TOTAL TRIIODOTHYRONINE (T3)','TOTAL THYROXINE (T4)','FREE THYROID 3','FREE THYROID 4','Human Chorionic Gonadotropin (hCG)','BILRUBIN'])]

2# Resetting the index:

df_s3=df_s2.set_index(['ID', 'Name', 'Age', 'Sex', 'CT', 'RT', 'Test', 'Test_Result', 'Units']).reset_index()

3# applything multiindex

idx = pd.MultiIndex.from_arrays([df_s3['ID'], df_s3['Name'], df_s3['Age'], df_s3['Sex'], df_s3['CT'],df_s3['RT'], df_s3['Units'], df_s3['Test'],  ])
#, df_s3['Unit of Measure']
df_s5 = df_s3.set_index(idx).Test_Result.unstack(fill_value='')
df_s5.columns.name = None
df_s6= df_s5.reset_index()
df_s6.head(100)

Result:

I get this result if do not add TSH (from Test Col)
 Code with TSH test:
Retry 1# with TSH

df_s2 = df_s[df_s['Test'].isin(['TOTAL TRIIODOTHYRONINE (T3)','TOTAL THYROXINE (T4)','THYROID STIMULATING HORMONE (TSH)','FREE THYROID 3','FREE THYROID 4','Human Chorionic Gonadotropin (hCG)','BILRUBIN'])]

df_s3=df_s2.set_index(['ID', 'Name', 'Age', 'Sex', 'CT', 'RT', 'Test', 'Test_Result', 'Units']).reset_index()

idx = pd.MultiIndex.from_arrays([df_s3['ID'], df_s3['Name'], df_s3['Age'], df_s3['Sex'], df_s3['CT'],df_s3['RT'], df_s3['Units'], df_s3['Test'],  ])
#, df_s3['Unit of Measure']
df_s5 = df_s3.set_index(idx).Test_Result.unstack(fill_value='')
df_s5.columns.name = None
df_s6= df_s5.reset_index()
df_s6.head(100)

Question1  (Retry 1# with TSH ): Please help me with the correct approach, what I understand the error is because once it convert it is not finding any unique index but not sure how to resolve it.

Question2: When I proceeded to go ahead without tsh, after conversion of test col- rows to cols, I get blank values in respective test col ( example T4 col) because 1) the person has taken the test but there is no value in the dataset(python is treating it as Null value and can be imputed/rejected - no issue 2) the patient has not taken this test but has taken atleast one other tests may be T3, hcg etc but not this test- this is considered as string '' . I want to get rid of these rows for amy analysis .. is there an approach while transforming the data to take care of so that I only want the result of the code to have T4 and its value( numeric or null). I do not want a scenario where the person has not taken test at all. OR is there a way to impute these values so I will know the person has taken T4, T3 but not Hcg , bilrubin etc?

Please advise. Long questions but I hope it this explanatory

Thank you very much to who ever had taken time into looking into this.  I have resolved this issue by adding a new unique col before apply multiIndex

df_s2 = df_s[df_s['Test'].isin(['TOTAL TRIIODOTHYRONINE (T3)','TOTAL THYROXINE (T4)','THYROID STIMULATING HORMONE (TSH)','FREE THYROID 3','FREE THYROID 4','Human Chorionic Gonadotropin (hCG)','BILRUBIN'])]
[df_s2['ID1'] = range(1, len(df_s2.index)+1)

df_s2.set_index(['ID1'])

idx = pd.MultiIndex.from_arrays([df_s2['ID1'],[df_s2['ID'], df_s2['Name'], df_s2['Age'], df_s2['Sex'], df_s2['CT'],df_s2['RT'], df_s2['Units'], df_s2['Test'],  ])

df_s5 = df_s2.set_index(idx).Test_Result.unstack(fill_value='')
df_s5.columns.name = None
df_s6= df_s5.reset_index()
df_s6.head(100)
```


How to code an SVM's equation including kernels? So I need to code an SVM from bottom up in Python, and I cannot use stuff like libSVM or scikit-learn, for reasons of my own.

Referring to Andrew Ng's excellent notes on Support Vector Machines, I need to specifically implement the second equation on page 20,

$$\max_\alpha W(\alpha) = \sum_{i=1}^m\alpha_i - \frac 1 2 \sum_{i,j=1}^m(y^{(i)} \times y^{(j)} \times \alpha_i \times \alpha_j \times \langle x_i, x_j\rangle)$$

Subject to the given constraints, $\sum_{i=1}^m\alpha_i y_i = 0$, and $0 \le \alpha_i \le C$.

Does someone have the code for this in any language, or can explain the intuition behind coding this? Most SVM implementations I could find either stick to scikit-learn or neglect kernel functions altogether. While I am working in Python, I would try to understand code in any language.

The SVM problem is a quadratic programming problem. It depends on whether you are willing to call a quadratic programming packages to solve your problem, if you are willing to do so, you just have to reformat the question in the input format of a programming language. Here is a resource for how to call quadratic programing solver in Python and here is how to do it in matlab.

Note that

$$\sum_{i=1}^m \alpha_i-\frac12\sum_{i,j=1}^my^{(i)}y^{(j)}\alpha_i\alpha_jK( x_i, x_j)=e^T\alpha-\frac12\alpha^TA\alpha$$

where $A_{ij}=y^{(i)}y^{(j)}K(x_i,x_j)$ , $K$ is the kernel function, and $e$ is the all one vector.

Now, suppose you are not willing to call a quadratic programming solver. If you read on the document that you attached, you will reach a section called sequential minimal optimization. Here is the wikipedia page of the algorithm.

The idea of the algorithm is that we can't update a single $\alpha_i$ at a time due to the linearity constraint, hence we do the update in pairs of $\alpha_i$ and $\alpha_j$. You might want to refer to Platt's paper for details including heuristic on how to choose the pair for update.


What is the difference between symmetric bipartite graphs and a complete bipartite graph? I am studying Restricted Boltzmann Machines (RBMs), and it is described as a symmetrical bipartite graph. Link

How is this different from a Complete bipartite graph? They seem to be the same to me, which is why I'm curious to why there is such a clear difference in terminology.

Your first link's usage of 'symmetrical bipartite graph' is indeed puzzling. From my knowledge of RBMs it would have been better for them to say 'complete bipartite graph' as you suggest.

The only formal reference I can find online for 'symmetry' in bipartite graphs is this link. But that does little to explain the usage in your case.


How to implement "one-to-many" and "many-to-many" sequence prediction in Keras? I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently see two different kind of codes:

Type 1 is where no TimeDistributed applied like this:

model=Sequential()

model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1], border_mode="valid", input_shape=[1, 56,14]))
model.add(Activation("relu"))
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=pool_size))

model.add(Reshape((56*14,)))
model.add(Dropout(0.25))
model.add(LSTM(5))
model.add(Dense(50))
model.add(Dense(nb_classes))
model.add(Activation("softmax"))

Type 2 is where TimeDistributed is applied like this:

model = Sequential()

model.add(InputLayer(input_shape=(5, 224, 224, 3)))
model.add(TimeDistributed(Convolution2D(64, (3, 3))))
model.add(TimeDistributed(MaxPooling2D((2,2), strides=(2,2))))
model.add(LSTM(10))
model.add(Dense(3))

My questions are:

Is my assumption correct that Type 1 is the one-to-many kind and Type
2 is the many-to-many kind? Or TimeDistributed has no relevance in
this aspect?
In either case of one-to-many or many-to-many is the last dense layer
supposed to be 1 node "long" (emitting only one value in turn) and
the previous recurrent layer is responsible to determine how many
1-long value to emit? Or the last dense layer is supposed to consist
of N nodes where N=max sequence length? If so, what is the point of
using RNN here when we could produce a similar input with multiple
outputs with N parallel "vanilla" estimators?
How to define the number of timesteps in RNNs? Is it somehow
correlated with the output sequence length or is it just a
hyperparameter to tune?
Inn case of my Type 1 example above what is the point of applying
LSTM when the model emits only one class prediction (of possible
nb_classes)? What if one omits the LSTM layer?

The point of using any recurrent layer is to have the output be a result of not only a single item independent of other items, but rather a sequence of items, such that the output of the layer's operation on one item in the sequence is the result of both that item and any item before it in the sequence. The number of timesteps defines how long such a sequence is. That is, how many items that should be handled in a sequence, and affect each other's resulting output.

A LSTM layer operates in such a way that it accepts input on the form number_of_timesteps, dimensions_of_each_item. If the parameter return_sequences is set to False, which it is by default, the layer "compounds" the inputs of all the timesteps into a single output. If you consider a sequence of, say 10 items, a LSTM layer with return_sequences set to False will from such a sequence produce a single output item, and the attributes of this single item will be a result of all the items (timesteps) in the sequence. This is what you want in the case of a many-to-one design.

A LSTM layer with return_sequences set to True will for each item (timestep) in an input sequence produce an output. This is done in such a way that at any timestep, the output will depend not only on the item that is currently being operated on, but also the previous items in the sequence. This is what you want in the case of a many-to-many design.

As a LSTM layer takes a sequence of items as input, any layer before a LSTM layer in your model will need to produce a sequence as an output. In the case of your Type 1 model, the first few layers do not operate on sequences, but rather a single item at a time. This does hence not produce a sequence of items on which to operate for the LSTM.

Using TimeDistributed makes it possible to have a layer operate on every item in a sequence without the items affecting each other. TimeDistributed layers thus operate on sequences of items, but there is no recursion.

In the case of your type 2 model, the first layers will produce a sequence 5 timesteps long, and the operations done on each of the items in the sequence will be independent of each other, since the layers wrapped in TimeDistributed are non-recurrent. As the LSTM layer uses the default settings, return_sequences = False, the LSTM layer will produce a single output for each such sequence of 5 items.

The final number of output nodes in your model depends completely on the use case. A single node is suitable for something like binary classification or for producing some sort of score.


ADHoc Information Retrieval I want to extract the total bill from image receipts. I could extract the entire data present in the image but now I am struck with the problem of extracting only the information that I need.

This is the image that I have.

I am pasting the extracted information from the image

m cm lnnk 3mm: :33; no 1 z m

x Visut all! ms“; (or nulnunn mfn an an: nan.

Sub Iota] 19.56
TOTAL 19.56
VISA 1956
Fun 19.56
D!!! You Know 0

For ureat-tastlru dessens under 200
cahries, try our Triple Berry Frozen
Yogurt Sunda: a dish of Frozen Yogurt.
or a Vanma rozen Vugurt Done.

From this data I just want to extract the total bill. To get this I found out that I could use Ad Hoc Normalization (Adhoc retrieval). Can someone provide any insights on Adhoc retrieval. If there are any other option to extract the data from the image please let me do so. I am using tesseract to extract this information. Sometimes it is no giving the proper output. I could use some help in improvising the output given by the tesseract.

receipt-parser is receipt parser package in Python built on tesseract.


Voting patterns similarities I'm interested in any research materials on voting patterns.

I have a data set of how PMs (members of parliament) voted in my country during last couple of years. Each PM has 3 buttons: Yes, No, Abstain. There are also two special situations: PM can be absent during the voting and he/she can be present but not vote (it's wrong but they do it sometimes). So any particular voting result is represented as a huge vector whose values are yes/no/abstain/present_not_voted/absent.

I'd like to find some similarities in PMs' voting patterns and cluster them in groups based on these similarities. There are some interesting questions like coalition stability and party loyalty which (in theory) could be answered with the data described above. Unfortunately i deeply lack any references.

While googling i've come across some site which formed an undirected graph based on American senators voting. It looked cool but i didn't find any explanation on how such graph can be formed. Intuitively "similar" is a symmetric binary relation so undirected graph representation sounds quite natural. But how can one define "similarity" for my data? Probably there are some standard ways of doing this. Any help will be highly appreciated.

The politicians and Political parties follow a set of principles and characteristics. They always follow these things during voting on legislations and bills. you can use T-distributed Stochastic Neighbor Embedding (t-SNE), machine learning algorithm for visualization developed by Laurens van der Maaten and Geoffrey Hinton. This method helps us to visualize the clusters and similarities of the observations.

Like-minded MP's and political parties with similar principles cluster together(neighbors).

The t-SNE 1algorithm comprises two main stages. First, t-SNE constructs a probability distribution over pairs of high-dimensional objects in such a way that similar objects have a high probability of being picked, whilst dissimilar points have an extremely small probability of being picked. Second, t-SNE defines a similar probability distribution over the points in the low-dimensional map, and it minimizes the Kullback–Leibler divergence between the two distributions with respect to the locations of the points in the map. Note that whilst the original algorithm uses the Euclidean distance between objects as the base of its similarity metric, this should be changed as appropriate.


How can I predict the acceptance of an article by publisher? I am working on this project where I have to predict the acceptance probability of a literary article, let's say a research paper by a publisher, which in this case would be a Journal.

I would like to use a machine learning algorithm for this project which would learn from the pool of articles/research papers published by the Publisher/Journal till date. Later I want to use the current article/research paper at hand to predict the probability of acceptance of this article/research paper on the insights that the algorithm has learnt. Here are my queries:-

What algorithm are there to tackle such task?
How should I make these algorithms learn the word vector, grammars, topics etc?
How do I convert the article into a feature set from which the algorithms can learn?

Applied Predictive Modeling book has a case study of acceptance rate of grant proposals, which was a Kaggle competition. You can get many good ideas from there.

from the dataset, what kinds of features you would look at
from the book, step-by-step explanations of how to build models
from the Kaggle competition, approaches by the contestants

I think it is very hard to predict the quality of the paper from the text of the paper, but you might have much more luck by including features such as academic rank of the author(s), their past publishing history and success, acceptance rate of the journals the paper was submitted to, and things such as even the length of the paper. Such proxy features for paper quality actually might be very successful...


How to interpret the mean for output clusters for expected-maximization? I am trying to cluster data using scikit's expectation-maximization. So I created two different data sets from a normal distribution which is I have shown in the graph below.

The mean for each of the distribution is:

Mean of distr-1: 0.0037523503071361197
Mean of distr-2: -0.4384554574756237

But after I run the EM using scikit, I get the mean as follows:

Mean after EM: [[-0.12327634  0.39188704]
[-1.31191255 -4.4292102 ]]

How am I supposed to interpret this mean? I am trying to create two clusters from the data. Here is my code:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture

distr_1 = np.sin(2 * np.random.randn(100) + np.random.randn())
distr_2 = (3 * np.random.randn(100)) + np.random.randn()

x = list(range(0,100))

X_train = np.concatenate((distr_1, distr_2))

plt.scatter(x,distr_1)
plt.scatter(x,distr_2)
plt.gca().legend(('sin', 'linear'))
plt.savefig('cluster_data.png')
plt.clf()

print("Mean of distr-1:",np.mean(distr_1))
print("Mean of distr-2:",np.mean(distr_2))

gmm = GaussianMixture(n_components=2)
gmm.fit(X_train.reshape(100,2))

print("Mean after EM:",gmm.means_)

Am I doing this incorrectly? What does the output mean?

It seems like you are trying to create a mixture of 2 univariate distributions, but you happen to get a bivariate distribution (which is why you have a 2x2 array).

This is because you reshaped the X_train array. You need to change your penultimate line to:

gmm.fit(X_train)


How to convert Hindi/Telugu/Marathi text to vector for text classification problem? sentence = 'अच्छा होगा अगर इसमें और गहने ना हों'

Which method will work for this task?
Is any pretrained model available to convert this text to vectors?
Please help by giving the code.

Hindi text is in Unicode format and can be read as follows in Python:

separators = [u"।", u",", u"."]
text = open("indiantext.txt").read()
#This converts the encoded text to an internal unicode object, where
# all characters are properly recognized as an entity:
text = text.decode("utf-8")

#this breaks the text on the white spaces, yielding a list of words:
words = text.split()

counter = 1

output = ""
for word in words:
    #if the last char is a separator, and is joined to the word:
    if word[-1] in separators and len(word) > 1:
        #word up to the second to last char:
        output += word[:-1] + u"(%d) " % counter
        counter += 1
        #last char
        output += word[-1] +  u"(%d) " % counter
    else:
        output += word + u"(%d) " % counter
    counter += 1

print

It is not clear from the question as to what exactly you mean by pretrained model for converting into vector but assuming that you are asking about the language model for hindi, ULMFiT is something you must refer to