Does it make sense to parallelize machine learning algorithms as part of PhD research? I'm developing machine learning algorithms to aid in the diagnosis and prognosis of various cancers for my PhD. My lab is an Nvidia teaching center (CUDA).

My supervisor thinks that I need to also optimize ML by parallelizing it in CUDA. However, as I see it, a model is trained once and there is no need to train again. Testing a model is also not time consuming. My interests are in ML, not Parallel Processing.

1) Should I spend a large chunk of my time parallelizing with CUDA?
2) Is CUDA still a viable framework for research?
3) In the world outside of research, will this make it easier to get a ML job?

From practise point of view, just sharing some thoughts. I don't have any research (PhD) type of experience, and your supervisor may communicate similar contents below.

And, paralleling calculation here, personally, I defined as single PC that utils graphical card GPU cores for calculation acceleration as oppose to clustering calculation.

Some Theoretical Thoughts

Sometimes, it worth building multiple models, or even, same model multiple times on the same data in ML world.

The example would be like:

using cross validation (same method multiple times) to have robust model output or parameters. Or
model ensemble with lots of weak learners (multiple models) to have better accuracy.

Those ML process are time consuming, and paralleling can help reducing the time.

Also, from the info provided, I assume your ML project is image recognition with GPU acceleration. But I've no idea what's the main purpose of the project.  It could be developing/improving new ML methods, or comparing known ML methods to form a academy review.  Whatever the case, I assume, the result should still achieve some certain accuracy level.

Hence, it would make sense to consider some efficient method (e.g. paralleling calculation) to accelerate the modelling process.

Some Practical Considerations

In practise, efficiency is very import.  When you have a theoretically accurate model, but takes a long time to build models, it won't be acceptable.

You can step back to say let's come back to some simple quick model with less accuracy. But what's the point of ML when traditional methods have similar or better accuracy?

Personal Answer to Questions

1) Should I spend a large chunk of my time parallelizing with CUDA?
Paralleling is useful to make the ML training quicker.  And CUDA is a cool technique to learn and apply to ML.  Just balance the time between main purpose of the project and time.

2) Is CUDA still a viable framework for research?
This meant to be the first chapter of your report / essay / dissertation.  At least, more and more ML uses deep learning, which can be much quicker with GPU involved.

3) In the world outside of research, will this make it easier to get a ML job?
I cannot definitely say knowing CUDA/paralleling would make you a top player (as other aspects would also be considered). But when other candidates have similar background, one with paralleling calculation experience would stand out.


Does it make sense to build a ROC for a decision tree where there are multiple threshold you can adjust? I understand building a ROC curve when the output is a probability, say, from a logistic regression model. You can build a ROC curve by varying the cutoff threshold.
But what about decision trees of the form:
if attribute_1 > x:
  decision = positive
else:
  if attribute_2 < y:
     decision = position
  else:
     decision = negative

You can adjust the cutoff for both attributes and all will affect your confusion matrix. Does it make sense to build a ROC curves when there are multiple thresholds?
Thanks

The ROC Curve has no relation with the way your model works, but instead, with its outputs. If the target is binary and your model outputs anything in between 0 to 1 (e.g. [0, 0.2, 0.4, ..., 1] or continuous probabilities), there's a sense in building the ROC Curve. If instead, the only outputs of your model are 0 or 1, the ROC Curve would be kind of useless, and computing simpler metrics like Precision and Recall would make more sense.


How to deal with a binary classification problem, where the instances in the negative class are very similar? Let's say, one wants to detect, whether a picture of a fixed size contains a cat or not. But as a dataset, you have 10000 pictures of cats, and 30000 pictures which don't contain a cat, but are very similar to each other. For example, let's assume, the 30000 pictures in the "not cat" class contain only pictures of one or two kinds of spiders.
When training a CNN, you will find that you achieve a high score on the test set (here high score = almost fully diagonal confusion matrix) but when you want to use the CNN in the real world, you find that almost everything gets classified as a cat.
Why does the network generalize badly in this case? Even if the dataset doesn't represent the kind of data, the CNN would see in the real world, shouldn't it be easy for the CNN to say "I have seen 10000 examples of cats, therefore anything which doesn't look like a cat is not a cat" ?
How would one deal with this problem (besides gathering more data)?

The CNN in this case does not learn what is a cat but rather what differentiate an image with a cat from one without a cat.
If all of your "no-cat" images contain spider(s), the CNN could also converge only by detecting the spider(s) : images with spider(s) belong to "no-cat" and others belong to "cat". That explains why you have such a good confusion matrix on testing data and such poor performance in real-world.
You have some options to avoid such a situation:

add other images to your "no-cat" label as you said (using public datasets for instance)
try one-class classification approaches. If you are familiar with neural networks, you could dig into reconstruction error of autoencoders (see this post: How to use a dataset with only one category of data or this sklearn guide)

Hope it helps


Data science degree or Computer Science degree? I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for awhile, so I know it's something I enjoy. The only questions I have are:

How relevant is the curriculum? (For example, the UCSD data science major)
How reliable is the degree in the upcoming future? (Would I be able to easily get a job like if I had a CS BS)
Am I better off doing CS and minoring in stats or something else?

I am also planning on getting either a masters or PhD in DS/ML. I don't know if it would better to get a lower level understanding of computers like I would probably get in a CS curriculum and then just focus on data science in grad school.

Thanks!

This is a bit opinion based, but here are my thoughts:

  How relevant is the curriculum? (For example, the UCSD data science major)

Since most DS majors have been developed the past 2-3 years they tend to be well structured and curated. Their subjects are usually a mix of some theoretical background (e.g. stats, linear algebra), some basic programming courses (usually python or R) and some ML topics at a near state-of-the-art level.

However, some face organization issues due to them operating only for a couple of years.

  How reliable is the degree in the upcoming future? (Would I be able to easily get a job like if I had a CS BS)

No one can tell, but the trend seems to be that DS will be adopted universally by tech companies in the future. From what I can tell you'll have no problem at all finding a well paying job right out of college.

  Am I better off doing CS and minoring in stats or something else?

Personally, I think no. DS at the moment is very undermanned in the labor market and demand is rising like crazy. To fill this demand there will be a lot of former programmers with a heavy CS background and years of experience making the transition. This, however, will change when people with DS degrees flood the market and I don't think it will be as easy in the future for CS to shift to DS, as it is now...


What are some solutions for dealing with time series data that are recorded at uneven intervals? Let's say I have a time series data which is a bunch of observations that occur at different time stamps and intervals. For example, my observations come from a camera located at a traffic intersection. It only records when something occurs, like a car passes, a pedestrian crosses, etc... But otherwise doesn't record information.
I want to produce a LTSM NN (or some other memory based NN for time series data), but since my features don't occur at even time intervals, I am not sure how having a memory would help. For example let's consider two sequences:
Scenario 1:

At 1PM, I recorded a car passing.
At 105, some people cross.
At 150 some more people cross.
At 2PM, another car passes.

Scenario 2:

At 1 PM a car passes
At 2 PM a car passes

In the first scenario, the last car passed 3 observations ago. In the second scenario, the last car passed 1 observation ago. Yet in both scenarios, the last car passed 1 hour ago. I am afraid that any model would treat the last car passing in scenario 1 as 4 time periods ago and the last car passing in scenario 2 as 1 time period ago, even though in reality, the time difference is the same. My hypothesis is that the time difference is a very important feature, probably more so than the intermediate activity between the two car passing. In other words, knowing that the last car passed 1 hour ago is equal or likely more important than knowing that there were some people crossing in the last hour. With that said, knowing that people crossed is important too so I can't just remove that feature.
Another example of my issue can be scene below:
Scenario 1

1PM Car passes
2PM Car passes

Scenario 2

1PM Car passes
10PM Car passes

Once again, in my data set, this would be treated as adjacent observations, but in reality, the time gap is vastly different and thus, the two scenarios should be viewed as very dissimilar.
What are some ways to solve these issues?

I've thought of just expanding the data set by creating a row for every possible time stamp, but I don't think this is the right choice as it would make my dataset humongous and most rows would have 0s across the board. I have observations that occur in microseconds so it would just become a very sparse dataset.
It would be nice to include time difference as a feature, but I am not sure if there's a way to include a dynamic feature in your data set. For example, in the first scenario, at 105, the 1PM observation needs a feature that says this occurred 5 minute ago. But at 150, that feature needs to be changed to this occurred 50 minutes ago, and then at 2PM, that feature needs to now say that it occurred 1 hour ago.
Would the solution to just give the NN the raw data and not worry about it? When building a NN on word prediction, I guess if you give the model enough data, it'll learn the language even if the relevant word happened 10 paragraphs ago...  However, I am not sure if there are enough examples of the exact same sequences (even with the amount of data) for it to obtain the predictability I want.

Any ideas on ways to solve this problem while keeping in mind that the goal is to build a NN? Another way to think about it is, the time when a data point occurred relative to when the prediction will be made, in my situation, is a crucial piece of information for prediction.

So, the question essentially asks how to model sequential data with inconsistent time intervals.
I would say option 2 would be more logical in my mind.
I would propose to have the timestamp when the image was taken from the camera as an input feature for each example. Here, it would still be in a format which you could model this using an RNN/LSTM (any sequential model architecture).
By adding the timestamp feature to your input, you at least have given the model the time representation of the images. You could represent the timestamp fo the image as number of seconds (minutes/hours, whichever is more appropriate to the dataset you have) from the earliest image you have in the dataset.
Of course, it's best to normalise this so you don't end up with very large values, which could reduce the overall generalisation performance of the model.


how to match a sentence to a cluster of keywords? I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below.

year of education               Education
years education                 Education
years of educational            Education
two years of education          Education
years of education beyond       Education
education four year             Education
..........
of proven sales experience      Ability
knowledge of and                Ability
experience or education high    Ability
assigned knowledge skills       Ability
accountable for driving         Ability
..........
administrative and leadership skills    Experience
advanced negotiations skills            Experience
must have keyboarding skills            Experience
must have skills                        Experience
activities preferred skills             Experience
of clinical skill                       Experience

I have to give a string and find it whether it belongs to either Experience or Education or Ability based on the trained model. The examples of strings .

string1 = "There is a requirement of four year professional degree"
string2 = "Able to drive the teams to higher levels"
string3 = "Must have programming experience in C, C++"

when I test these string it should be able to classify the string in to either of the clusters.

What are the possible ways to train my model ? Refering to word2vec and doc2vec, will these models work? I couldn't find any relevant examples to train model on words and test on strings. Any ideas on how to work?

I am kind of confused by the question. But I am guessing you are asking what model to use and how to represent words.

This seems like a pretty introductory example so you may want to try simple Bag-of-words representation. Try pushing that into a Naive Bayes classifier as a first pass. If you are unhappy with the results, you can implement word2vec and more complex methods.


When to use Random Forest I understand Random Forest models can be used both for classification and regression situations.

Is there a more specific criteria to determine where a random forest model would perform better than common regressions (Linear, Lasso, etc) to estimate values or Logistic Regression for classification?

Adding some extra general points to the previous answer:

As a decision tree algorithm, Random Forests are less influenced by outliers than
other algorithms. A good discussion about it is here.
They also do not make any assumptions about the underlying distribution of your data,
and can implicitly handle collinearity in features, because if you have two
highly similar features, the information gain from splitting on one
of the features will also use up the predictive power of the other
feature. Read about it here.
Random Forests can be used for feature selection because if you fit
the algorithm with features that are not useful, the algorithm simply
won't use them to split on the data. It's possible to extract the 'best'
features (which could be the total number of
times a feature was used to split on the data, or the mean decrease
in impurity etc). However, as with my point above, you cannot read too much into the
relative importance of the features, especially if you have ones that are highly correlated.
I think the biggest deciding factor of whether to use a RF or another algorithm is probably if you want to understand more about the relationship that
features have with the target and the degree of influence they have.
If this is important for your use case (for example, you want to know if a feature has a positive or negative relationship
with the target and the degree to which it affects the outcome) then others like Logistic Regression and Lasso are better choices.
Also, if you want your model to extrapolate to predictions for data that is
outside of the bounds of your original training data, a Random Forest
will not be a good choice.

Hope this helps!


Custom trained model in Azure ML I have a predictive model which I trained on a training set. I have written it in R. Now I want to deploy it as a web service so anyone can just input the data into it and get the output from the predictive model.

I wanted to use Azure ML for deploying. I wanted to know whether I can drag and drop my already created/custom trained model to Azure ML studio instead of re-training it there ? I know we can train and save models in AML Studio but I am not sure about adding already created models and using them in AML solution. Help regarding this will be appreciated.

No you cannot. I had a discussion with someone in their development/support team on the MSDN forums and currently they don't support 'drag and drop' type of functionality. However you CAN serialize the model output and then de-serialize them in Azure.

Note that the answer in the image is a bit outdated and there is the 'Create R Script' module to replace the serialization-deserialization steps within Azure. However I believe you can still serialize outside Asure (in your Desktop version of R) and deserialize them in Azure.

Link to the conversation in Image:
https://social.msdn.microsoft.com/Forums/azure/en-US/5944c342-79ac-4ada-8006-8edf40f36ee1/r-script-as-a-trained-model?forum=MachineLearning


Upload 'tsv' file to Google Colab / Jupyter Notebook TSV(Tab separated Value) extension file can't be uploaded to google colab using pandas

Used this to upload my file

import io
df2 = pd.read_csv(io.BytesIO(uploaded['Filename.csv']))

import io
stk = pd.read_csv(io.BytesIO(uploaded['train.tsv']))

What i want is a tsv file should be uploaded and read into the dataframe stk

You first need to upload your file. The io.BytesIO only reads from the uploaded. So first run:

from google.colab import files
uploaded = files.upload()

and select the file you would like to upload.

Also, when you load it into your pandas, you need the sep='\t':

tsk = pd.read_csv(io.BytesIO(uploaded['train.tsv']), sep='\t')


How to compute modulo of a hash? Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then proceed to "bucketizing" them and being able to do random sampling with the help of the modulo function.

That said, if I have a hash such as 367b50760441849e, I want to be able to use hash % 20 == 0 to randomly pick 5% of the population (hence, 20 "buckets"). This is the approach that is used in Kusto hash() with a modulo argument.

With this in mind, what is the approach that should be used to calculate an integer value from the hash, so that I can calculate the modulo?

Any good hash will be uniformly distributed, which means that you can assume a uniform distribution when you apply modulo n, as long as $n < 2^{M/2}$, where M is the number of bits in your hash, see here. So for SHA1-32 you would at most modulo by $2^{16}$.

There is no approach to calculating an integer value; what you have there is an hexadecimal representation of a hash, you just need to convert it to a numeric type if you obtained it as a string. XXH32() and XXH64() both already produce an unsigned int output.


Question about sklearn's StratifiedShuffleSplit I'm reading through the book Hands-On Machine Learning with Scikit-Learn and Tensorflow by Aurélien Géron. In a regression project on California Housing Prices, he goes over the concept of stratified sampling.

I think I understand the concept as his explanation "the population is divided into homogeneous subgroups called strata, and the right number of instances is sampled from each stratum to guarantee that the test set is representative of the overall population."

So in my own words, simply splitting the dataset with sklearn's train_test_split leaves the train and test set vulnerable to misrepresenting the ratios of categorical variables (ie population has 40% one category, 60% another, but the train/test set are totally different ratios of these categories), so stratifying ensures the sample is 'random', but still maintaining proper ratios within test and train splits. Please correct me if I'm wrong.

Here's the code to his stratified sampling based on income categories (housing is the main dataframe):

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing['income_cat']):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]

I'm pretty confused with this code:

1) What does the variable 'split' represent? Does it comprise both the train and test split...?

2) In the 2nd line of code, what does split.split mean? I guess I'm confused with most of how StratifiedShuffleSplit divides the train and test set and why he needed to create this 'for' loop in order to create strat_train_set and strat_test_set.

Thanks,

Greg

split

Is the object that allows us to do stratified split, and split.split is the split’s object method/function called split that can be used to perform stratified split.

Depending on how many splits you want using the n_split parameter you can split, hence the for loop.


Can pytorch / keras support dataloader object of Image and Text? Is there a way of creating a dataloader object, or the equivalent in Keras, where every observation is an image AND some text? I could create two models to do classification but I want to see if I can build it all in one to pull information from both to get the prediction.

Are there any examples of this someone can point me to? Or is it even possible?

The closest I've found is in fast.ai there's this article that combines tabular and text.

I can only guess what you are trying to achieve, but I think this might be helpful even though it is a different approach. The idea is to create two separate models for text and image and merge them into third.


NNDSVD to initialize Convex-NMF I'm working with the Convex Nonnegative Matrix Factorization Algorithm described in Ding, Li, Jordan 2008 ("Convex and Semi-Nonnegative Matrix
Factorizations").

Good initialization strategies make all the difference and using the described k-means clustering to get started works very well.

But there is a paper describing NNDSVD (Boutsidis, Gallopoulos, 2007) to initialize "traditional" NMF Algos. I wanted to test this, to see if it improves my results.

The nonnegativity constraints for Convex-NMF are relaxed. X can have mixed sign data, where X ~ XWG', with factors W and G having only positive data.
I've implemented NNDSVD just like in the paper (in C++ w/ OpenCV), but since X has mixed sign data, the resulting W contains negative values as well.

Has anyone tried to adapt this initialization strategy? Are there any other recommended initialization strategies for Convex-NMF, aside from the ones mentioned in the Ding et al Paper (and random init)?

I've played before with a library for ("classic") NMF from University of Vienna: libNMF

While it wasn't useful for me at first, since they don't implement Convex-NMF, I had a look at their NNDSVD implementation.
They simply replace all negative values and values smaller than the machine epsilon with zero.

I tried that, but it fails, since the update rules for Convex-NMF are such that I get a division by zero. Since they add 0.2 to all values in their proposed initialization scheme (for "smoothing"), I just did the same here.

It works, the precision is worse than expected (a bit worse than k-means, which is my best approach for now), but I'll tweak it a bit more.


scalable tools to build kNN graph over sparse data I'm looking for scalable tools to build kNN graph over sparse data points.

The dimension and number of data points can be both up to millions.

What I have tried already:

sklearn.neighbors.kneighbors_graph: which does brute-force search for sparse data, giving quadratic time.
flann: only supports dense arrays
pysparnn: the running time is not very satisfatory (maybe because it's written in Python)
knn search in mlpack: which only supports dense data
scipy.spatial.KDTree: which converts the sparse data to dense one
SparseLSH: which is implemented in Python, so I'm not quite sure about the scalability
elasticsearch: it seems to only support indexing documents, instead of sparse features.

the reason I thought of elasticsearch is: knn over sparse data can be framed as retrieving the top-k "documents" in IR.

Thanks for any comments/answers :)

I think what you might be looking for is,

L2Knng: Fast Exact K-Nearest Neighbor Graph Construction with L2-Norm Pruning

They have multiple runtime options specifically for different kinds of datasets (including sparse data). The link for the same is : http://glaros.dtc.umn.edu/gkhome/node/1162


What's the advantage of multi-gpu training in real? The decreasing speed of training loss is almost the same between one gpu and multi-gpu.

After averaging the gradients, the only benefit from multi-gpu is that the model seems to see more data in the same time.

But why average the gradients?
Is it that the model is indeed feed with more data in the same time?

I see two main advantages of using multi-GPU instead of one as they distribute certain resources:

using large DNN models - some recent models occupy vast space in memory so they simply cannot fit regular GPU and using multiple GPU allow to distribute some parts of the model to different GPU instances.
speed-up DNN training is also a very positive effect of using multiple GPU but only if you have a high-speed connection among GPUs as NVIDIA came with their NVLink


Building a multiclass classifier that can handle classes it has never seen? I am given a dataset that has free-form text and a category associated with it. There are 100 different categories and 3000 records for each category. The goal is build a multiclass classification model. I have created a simple neural network with 10,000 input features/words and the results were fairly good (~88%).
The issue I am facing is with an unlabeled dataset that I have, that is missing the category label. This dataset is very large, and has much more than 100 categories. I am only interested in being able to classify unlabeled data for the 100 categories that I have, but I am not sure how to approach this.
One thought I had was to build a Word Embedding model for the labeled data. This model could be used to calculate a document vector for the unlabeled data and find a similar document from the labeled dataset. This would allow me to assign labels to some of the data in the unlabeled dataset. Is there a better way to approaching this problem?

You basically want your model to say - I dont know.
For the test set created out of the original dataset with labels, plot a histogram of the probabilities of the predicted class. Are they generally higher than some threshold say 0.2 ? If so experiment using that as a threshold to flag items from your unlabeled dataset for a manual inspection. The hope is that the probabilities are low because the model is unsure.
The manual review of these flagged items will allow you to verify if this approach even works. If it works, then use that as a labeling strategy for the "I dont know" category and retrain along with this new label.


Using max-pooling is not a good idea on its own. The reason is that by employing that, you ignore $75$% of the information each time. If your input is a signal which is quite a small pattern, as the number you have referred, it is better not to use max-pool that much. Although the use of max-pooling adds relative spatial invariance to the objects in the input and is useful for classification tasks, the main reason it is employed nowadays is to reduce the number of parameters to train. For instance, for images which belong to $R^{224\times224}$, it is wise to use them for some layers due to lessening the number of training parameters significantly. For images with smaller sizes, the input signal on its own has a smaller number of entries, features, and by employing max-pooling, you are chiefly discarding information which may be necessary and prominent for generalisation. You are actually shrugging them off!


Early stopping and final Loss or weights of models In a deep model, I used the Early stopping technique as below in Keras:

from keras.callbacks import EarlyStopping

early_stopping = [EarlyStopping(monitor='val_loss',
                          min_delta=0,
                          patience=2,
                          verbose=2, mode='auto')]

model.fit(train_x, train_y, batch_size=batch_size,
                epochs=epochs, verbose=1,
                callbacks=early_stopping,
                validation_data=(val_x, val_y))

model.fit(train_x, train_y, batch_size=batch_size,
                epochs=epochs, verbose=2,
                callbacks=early_stopping,
                validation_data=(val_x, val_y))

Now, when I run this code, in the output it prints the loss value for training and validation of each epoch.

I set the patience=2 in the early stopping. So, it continues the training process two times after when the validation loss increased instead of decreased.

Some things like this:

Epoch 1/10
- 198s - loss: 99.7160 - val_loss: 123.0397
Epoch 2/10
- 204s - loss: 78.7000 - val_loss: 109.0344
Epoch 3/10
- 208s - loss: 65.4412 - val_loss: 78.0097
Epoch 4/10
- 268s - loss: 61.9812 - val_loss: 79.0312
Epoch 5/10
- 298s - loss: 59.1124 - val_loss: 79.3397
Epoch 6/10
- 308s - loss: 57.2200 - val_loss: 218.0397
Epoch 00007: early stopping

In the end, what will be the final weights of the model and the Loss values? The final time of training or two times before it?

If it considers the final epoch, so should it be better if I set the patience as little as possible to overcome the overfitting?

Thank you

EarlyStopping

The final weights will be saved, not the weights where your patience parameter is triggered.

Looking at the documentation for EarlyStopping, it seems not to be involved with saving weights at any point - it isn't mentioned.

EDIT

Upon further investigation (reading the source code), it seems you can indeed save the best, using the EarlyStopper callback

The class can be initialised with the aprameters restore_best_weights, as seen here. Then at the end of training, when your waiting period has overshot the patience parameter, the model's weights are returned to be the best weights (weights of the model at the time of loewst validation loss:

if self.restore_best_weights:
    if self.verbose > 0:
        print("Restoring model weights from the end of the best epoch")
    self.model.set_weights(self.best_weights)

It does this by tracking a chosen metric and comparing it to the recorded best value. By default this will be the validation loss. Check out the course code and it's description of the class there.

If these parameters don't work with your code, you will need to upgrade to the latest version (master branch from GitHub). This new parameter was only added 10 days ago!

ModelCheckpoint

If you want to use a callback to prevent overfitting, have a look at the ModelCheckpoint callback class. This has options to save the model weights at given times during training, and will allow you to keep the weights of the model at the end of the epoch specifically where the validation loss was at its minimum. This is selected using the the save_best_only parameter - use it like this:

from keras.callbacks import EarlyStopper, ModelCheckpoint

checkpointer = ModelCheckpoint(filepath, monitor='val_loss', verbose=0,
                               save_best_only=False, save_weights_only=False,
                               mode='auto', period=1)

# your early stopper as before
early_stopper = ...

All callbacks must be placed in a list:

my_callbacks = [checkpointer, early_stopper]

# pass to model along with other parameters as you did already
model.fit(..., callbacks=my_callbacks, ...)

If you don't use save_best_only, the default behaviour is to save the model at the end of every epoch.

You can set the filepath by using certain dynamic variables available during training, so that the filenames have some useful information:

E.g. filepath is weights.{epoch:02d}-{val_loss:.2f}.hdf5

  the model checkpoints will be saved with the epoch number and the validation loss in the filename.

Have a look at the linked documentation for how to use the other parameters of ModelCheckpoint.


What is a product most frequently bought together with I want to find out what people also buy when they buy a "Bike Helmet". There are only 77 different products in my dataframe.

I might solve this using a for loop, and creating a counter. So whenever ProductName == 'Bike Helmet, look at the OrderID and increase the counter by 1 for all those product names. Then find the next instance of a bike helmet purchase. This approach, however, is slow.

  OrderID           ProductName  PricePerOrder  ProfitPerOrder  Quantity  \
239    10337           Bike Helmet     805.200000      178.080000        24
238    10337           Mehmet-Skor       5.200000        0.626667        28
237    10337          Mehmet-Tröja       7.990000        1.540000        20
236    10337          Oyaki Kimono      10.025000        2.558333        25
240    10337           DSW Sandals       3.600000        0.628000        40
40     10262          Runner Shoes      14.362500        1.946250        15
42     10262       O-Man Underwear       0.535680       -0.030720        12
41     10262         Slip-on Shoes       4.606667        0.050667         2
336    10374           Onesy Dress      38.677500       10.800000        15
337    10374  Rossi Bermuda Shorts      13.665000        2.370000        30

Two approaches, both based on discrete mathematics. The first one is a set-theoretic approach to frequent item-set mining and association rule mining. For instance A-priori algorithm works here.

the next approach is creating a graph based on orders and do graph mining on them. A-priori might be a better approach as in this case you probably need to deal with hypergraphs which makes life a bit complicated.

I suggest you first extract sets of different orders from your dataframe. As your problem is not general and you already know that there is only association to one product is important, you can prune your data much. Then simply rank other products bought together with Bike Helmet by counting number of their occurences and dividing by number of all (it gives you the probability estimate of each product is bought together with bike helmet).


What is the model architectural difference between transductive GCN and inductive GraphSAGE? Difference of the model design.

It seems the difference is that GraphSAGE sample the data.

But what is the difference in model architecture.

The main novelty of GraphSAGE is a neighborhood sampling step (but this is independent of whether these models are used inductively or transductively). You can think of GraphSAGE as GCN with subsampled neighbors.

In practice, both can be used inductively and transductively.

The title of the GraphSAGE paper ("Inductive representation learning") is unfortunately a bit misleading in that regard. The main benefit of the sampling step of GraphSAGE is scalability (but at the cost of higher variance gradients).


DistinctCount for relatedtable with additional filter In powerBi, a column is created with below formula, and it is working good.

activedays = COUNTAX(RELATEDTABLE(ActivesInactiveData),ActivesInactiveData[Surface Date])

However, I want to add a filter some thing like

ActivesInactiveData[Topic]<> null

Could you please advice on how to do so, most of the examples on web are using on relatedtable etc - where relation is already build, could you please advice/direct to me what I could find an example of adding filter to relatedtable or filter data from a unrelated table based on field like ID column etc.

Here there is a relation user ID in current table has more than 1 rows in ActivesInactiveData table.

I would wrap your expression in a CALCULATE function, so something like:

activedays = CALCULATE (
    COUNTAX ( RELATEDTABLE ( ActivesInactiveData ) , ActivesInactiveData[Surface Date] )
    , ISBLANK ( ActivesInactiveData[Topic] ) = FALSE )

I assumed [Topic] is text, so NULL will converted by DAX to BLANK().


Scalable way to calculate betweenness centrality for a graph in spark I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job. Has anyone successfully calculated betweenness centrality of a large network with around 10 million vertices and 100 million edges?

Sorry, I do not think you can compute the exact betweenness centrality of nodes in a graph this size, as its complexity is $O(n\cdot m)$ where $n$ is the number of nodes, $m$ the number of links.
The good news is that you may approximate it, and in a way that may take benefit from parallel computations. Indeed, computing betweenness centrality relies on counting the number of shortest paths from any node to any other. You may (randomly) select some nodes and compute the numbers of shortest path from each of them to all others, and use the obtained number to approximate the betweenness. The more nodes you select, the best the approximate will be, but it is empirically rather good even with a small sample set.


Mining of massive datasets Can someone answer this question:

It is from an exercise in the book:
  Mining of massive datasets:

Chapter 3: Finding Similar Itemsets

What is the largest number of k-shingles a document of n bytes can have?

  Exercise 3.2.3 : What is the largest number of k-shingles a document of n bytes can have? You may assume that the size of the alphabet is large enough that the number of possible strings of length k is at least as n.

the answer is (N-k)

I wanted an explanation

I checked the exercise, there is an additional assumption that simplifies the question

  Exercise 3.2.3 : What is the largest number of k-shingles a document of n
  bytes can have? You may assume that the size of the alphabet is large enough
  that the number of possible strings of length k is at least as n.

Shingling is a common technique of representing documents as sets. k-shingle is said to be all the possible consecutive substring of length k found within a document. For example if the document is "The sky is blue and the sun is bright"
the k-shingles when k=3 is given below:

[1)"the sky is"    2)"sky is blue"     3)"is blue and"     4)"blue and the"
 5)"and the sun"   6)"the sun is"      7)"sun is bright"]

Now we can say that for a document of 9 words, the number of 3-shingles is 7. if you try multiple sentences you conclude that:

The number of k-shingles = the number of words - k+1

The question now is what is the largest number of possible words in an n-bytes document?

This depends on the smallest possible size of a word. In UTF-8 encoding each letter occupies 1 byte(8 bits). for example a word of 4 letter "Mars" occupies 4 bytes.

The smallest possible size of a word is 1 byte, that's when it consists of only 1 letter. Hence the largest number of words in n bytes document is  n . Now let's substitute it in the largest number of k-shingles

The number of k-shingles = n-k+1

At the end this answer is valid under the following assumptions:

UTF-8 encoding
Each word consists of 1 letter only
The size of the alphabet is large enough (>n) so that all sets are different


How can I merge 2+ DataFrame objects without duplicating column names? This is for work. TLDR: Bottom-line question at the bottom.

I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've made good progress on parsing the output data into lists of strings, booleans, etc., but I'm having trouble pulling the data together into an easily searchable and retrievable whole. The data looks something like this:

- test_case_A
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 100  pass/fail   supporting data
- test_case_B
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 70   pass/fail   supporting data
- test_case_C
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 10   pass/fail   supporting data
- test_case_D
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 30   pass/fail   supporting data

I parse them into individual DataFrame objects like so:

df_case_A_pass_fail = pd.DataFrame({
    "case" : "test case A",
    "header" : *big header string*,
    "test ID" : [*list of IDs*],
    "test passed" : [*list of bool*],
    "test data" : [*list of strings*],
    })
*repeat for test case B, C, and D*

Now I try to merge them together.

big_df = fancy_merge_step_probably_involving_reduce_and_a_lambda(...)

Problem 1: The "case" and "header" strings appear to be duplicated all the way down their DataFrame. Like so:

             case        test ID  test passed               header      supporting data
0   "test case A"    "test 1 ID"         True  "big header string"  "all kinds of stuff"
1   "test case A"    "test 2 ID"         False "big header string"  "all kinds of stuff"
...
99  "test case A"  "test 100 ID"         True  "big header string"  "all kinds of stuff"

I checked the DataFrame size according to this get_real_size(...) algorithm and the size explodes exponentially as I merge in more results (~80KB for 1 test case, ~800KB for 2 test cases), so the big header string is definitely getting duplicated. I want to establish one->one relationship between test case and header and one->many between test case and test data, but all I'm seeing is duplication until there is a unique permutation of every line. Am I making the DataFrame wrong for what I need?

Problem 2: (Possibly related to problem 1.) I attempt to merge the DataFrames with an outer join and get the NaN results for places where the column sizes don't match up (expected), but also get duplicated columns "case", "header", "test passed", "test data" (any column that wasn't merged on), appendend with automatic suffixes ("_x", "_y"). I know that pandas does this automatically when there is a column name class, but it is now a problem. Result: searching on column "case" fails because the merged DataFrame has no column "case". All columns formerly named "case" are now "case_x" or "case_y".

I want to query like this:

match1 = (big_df["case"] == "test case D")
match2 = (big_df["test ID"] == "test 3")
single_test_df = big_df.loc[match1 & match2]

match1 = (big_df["case"] == "test case A")
match2 = (big_df["header"] == "header")
header_str = big_df.loc[match1 & match2].values[0]

Question: How do I set up these DataFrame objects and merge them so that I can query the test data as mentioned earlier?

Welcome to the site!

Problem 1 seems to be an issue that could be solved by treating your dataframes like tables in a relational database. By this I mean: you have your main data table and secondary tables called 'headers' and 'supporting_data', and you store each header in the 'headers' table once with an ID (a key, maybe just an integer) assigned to it, and in the main table you just store the integer corresponding to the header data. If you ever need to refer to the header data, you look up the integer and go over to the 'headers' table to retrieve it. You can do something similar with the supporting data, but not knowing what that data is it's up to you to figure out if you should parse out parts of that data or store it as whole text blocks or whatever. But if you do this, you'll just have two integers on each row of the main table instead of all that superfluous repeated data.

Problem 2 I'm less sure about, but I'd look into the pandas concat function. I think you should probably just have a case column containing 'A', 'B', etc, and be appending the rows from those files into the main table; concat should be able to do that if all the data types work out, etc, etc.

Good Luck!


Association between continuous input and categorical output I have two independent continuous variables like Age, Price and an outcome variable like purchased or not purchased
I know if we have continuous variables on input and output, we can use pearson correlation or chi-square test if both input and output are categorical.
Now, which test should I use to test the association of continuous input variable with categorical outcome ?
So, can someone please help me?

Try logistic regression. The sigmoid function and its variants allow for conversion of continuous inputs into a binary outcome variable.


How do the linear layers in the attention mechanism work? I think I now the answer to my question but I dont really get confirmation.
When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that there are three linear layers applied on the key, query and value matrix. And then one layer at the end, which is applied on the output of the matrix multiplication of the score matrix an the value.
The three linear layers at the beginnging: When the key/query/value with shape (seq-len x emb-dim) enter the linear layer the output is still (seq-len x emb-dim). Does that mean, the same linear layer is applied on every "index" of the input matrix. Like this (pseudo-code):
fc = linear(emb-dim, emb-dim) # in-features and out-features have the shape of emb-dim
output_matrix = []

for x in key/query/value:
    # x is one row the input matrix with shape (emb-dim x 1)
    x = fc(x)
    # x after the linear layer has still the shape of (emb-dim x 1)
    output_matrix.append(x)

# output_matrix has now the shape (seq-len x emb-dim); the same as the input-matrix

So is this indeed what happens? I couldn't explain why the output is the same as the input otherwise.
The linear layer before the output: So the output of the matrix multiplication of the score matrix an the value is also (seq-len x emb-dim) and therefore the output of the linear layer is too. So the output of the whole attention block has the same shape as the input.
So Im just asking for comfirmation if the explaination I wrote is correct. And if not: What am I understanding wrong?
Extra question: When I want to further use the output of the attention block further for classification, I would have to take the mean along the seq axis in order to get a vector of fixed shape (emb-dim x 1) so I can feed it into a classification layer. But I guess that valueable information is getting lost in that process.
My question: Could I replace the last linear layer with an RNN to get the desired output shape and without losing information?

Your understanding is not correct.
The relevant information is described in the original paper in section 3.2.2:
The three sets of projection matrices you are referring to are $W^Q_i \in \mathbb{R}^{d_{model} \times d_k}$ for the Queries, $W^K_i \in \mathbb{R}^{d_{model}\times d_k}$ for the Keys and $W^V_i \in \mathbb{R}^{d_{model}\times d_v}$ for the Values. Notice that the $i$ subindex in the matrix names refers to the attention head, indicating that there is a different matrix for each attention head.
The final projection matrix is $W^O \in \mathbb{R}^{hd_v \times d_{model}}$.
Given the number of attention heads, $h = 8$, the dimensions of the matrices are defined by $d_k=d_v=d_{model}/h=64$.
The three sets of matrices project the embedding dimensionality $d_{model}$ into a space with 8 times smaller dimensionality ($d_k=d_v=d_{model}/8$). However, note that for each of $W^K$, $W^V$ and $W^Q$ there are 8 matrices (one per attention head) and, analogously, 8 scaled dot products are computed. The results of the dot products are 8 vectors of dimensionality $d_{model}/8$; those 8 vectors are concatenated (see figure below), giving a tensor with the original dimensionality $d_{model}$.
Then, the final matrix multiplication by $W^O$ doesn't change the dimensionality, obtaining again the original one.

About your second question, either of the approaches you describe (averaging and using an RNN) is technically feasible, but what people normally do when using transformers for classification is to use BERT's approach, that is, adding a special token [CLS] at the beginning of the sequence and using the output at that position for the classification, just projecting with a matrix multiplication into a space with dimensionality equal to the number of classes.


Naive bayes, all of the elements in predict_proba output matrix are less than 0.5 I've created a MultinomialNB classifier model by which I'm trying to label some test texts:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import preprocessing
from sklearn.naive_bayes import MultinomialNB

tfv = TfidfVectorizer(strip_accents='unicode', analyzer='word',token_pattern=r'\w{1,}',
                      use_idf=1,smooth_idf=1,sublinear_tf=1)

# df['text'] is a long string text of words
tfv.fit(df['text'])

lbl_enc = preprocessing.LabelEncoder()

# df['which_subject'] is one of the following 7 subjects: ['Educational', 'Political', 'Sports', 'Tech', 'Social', 'Religions', 'Economics']
y = lbl_enc.fit_transform(df['which_subject'])

xtrain_tfv = tfv.transform(df['text'])

# xtest_tfv has 7 samples
xtest_tfv = tfv.transform(test_df['text'])

clf = MultinomialNB()
clf.fit(xtrain_tfv, y)

y_test_preds = clf.predict_proba(xtest_tfv)

Now y_test_preds is as follows:

0.255328    0.118111    0.129958    0.123368    0.119301    0.131098    0.122836
0.122814    0.265444    0.117637    0.13531     0.116697    0.122812    0.119286
0.131485    0.114459    0.258224    0.122414    0.118132    0.134005    0.12128
0.125075    0.131948    0.122668    0.258655    0.116518    0.119995    0.12514
0.124356    0.116987    0.121706    0.119796    0.266172    0.127231    0.123751
0.132295    0.1192      0.13366     0.119445    0.123186    0.257318    0.114895
0.126779    0.118406    0.123723    0.127393    0.122539    0.117509    0.263652

As you see, all of the elements are less than 0.5. Does this table show anything? Can I conclude that the classifier is not able to label test text?

No, your classifier can label text. It doesn't do it well but it is still almost 2 times better than random (for 7 classes, random will get you ~0.15 accuracy).

Looking at the test set is not enough. You need to create the same confusion matrix for you training set.

If the results you will get for the test set are similar in magnitude than maybe your model is too simple for the task or maybe you haven't trained it long enough.

If the results of the test set are good, than you might have a generalization problem (overfitting), which means that you need to increase the regularization during training. It also might mean that your training set comes from a different distribution than your test set.


Effect of weights on the Louvain communities detected The Louvain method for community detection aims to optimize modularity and hence detect communities in the given graph. In case of a weighted graph would it be valid to assume that an edge with a higher weight would be prioritized to be part of a community as against an edge with a lower weight while optimizing modularity ?

Yes, weights are interpreted as a strength of connection in the context of modularity.  This is perhaps easiest to see using the interpretation of unweighted modularity as

The fraction of edges within communities minus the expected such fraction in a random graph with the same degree sequence.

Switching the adjacency matrix to the weighted adjacency matrix to get the weighted modularity just changes this to

The fraction of total weights within communities minus the expected such fraction in a random weighted graph with the same total weights at each vertex.

That said, you do have to be careful.  There are community detection methods that do treat weights as distances:
SO: meaning of weights in community detection algorithms
It may be worth running a simple example when first using a specific algorithm implementation; e.g., with $G=C_4$, weight one perfect matching very much higher than the other, the communities should end up being two $K_2$s, and you want to know whether they contain the high- or low- weight edges.


What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem? I have a multi-class classification problem with class imbalance. I searched for the best metric to evaluate my model. Scikit-learn has multiple ways of calculating the F1 score. I would like to understand the differences.
What do you recommending when there is a class imbalance?

F1Score is a metric to evaluate predictors performance using the formula

  F1 = 2 * (precision * recall) / (precision + recall)

where

  recall = TP/(TP+FN)
  and
  precision = TP/(TP+FP)

and remember:

When you have a multiclass setting, the average parameter in the f1_score function needs to be one of these:

'weighted'
'micro'
'macro'

The first one, 'weighted' calculates de F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:

$$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$$

therefore favouring the majority class.

'micro' uses the global number of TP, FN, FP and calculates the F1 directly:

$$F1_{class1+class2+class3}$$

no favouring any class in particular.

Finally, 'macro' calculates the F1 separated by class but not using weights for the aggregation:

$$F1_{class1}+F1_{class2}+\cdot\cdot\cdot+F1_{classN}$$

which resuls in a bigger penalisation when your model does not perform well with the minority classes.

The one to use depends on what you want to achieve. If you are worried with class imbalance I would suggest using 'macro'. However, it might be also worthwile implementing some of the techniques available to taclke imbalance problems such as downsampling the majority class, upsampling the minority, SMOTE, etc.

Hope this helps!


How to understand clearly the feature importance computing in random forest model I want to know the details of feature importance computing of random forest with gini index. I refer to this Chinese blog:https://www.jianshu.com/p/7a876bb876b9. But I don't understand clearly these mathematical forms in blog.

for example

As the node m after twos branches(Il, Ir) splitting, the gini index is computing as following

And  feature Xj in the decision ith tree ,and be in nodel of M set, and the feature importance of Xj in the ith tree is computing as following:

And I confusing about M set, what's the meaning of M set? I think that the  feature Xj in ith tree just one node not a M set? If you guys have detailed computing method, please tell me!

Let's start with this formula for the Variable Importance Measure (VIM) based in Gini Impurity (GI)*:

$$VIM_{jm}^{(Gini)} = GI_m - GI_l - GL_r$$

Here, $m$ is a given node in a single decision tree which performs a split on feature $j$. For this node you can compute the Gini Impurity $GI_m$ (see * note at the very bottom for the formula). Now this node has two childs: a left node $l$ and a right node $r$. For these the Gini Impurities are $GI_l$ and $GL_r$ respectively. If you substract the Gini impurities of the child nodes from the Gini Impurity of node $m$, then you know by how much the split performed by this node has decreased the overall Gini Impurity. Because before the split it was $GI_m$ and after the split the remaining impurity is the sum of $GL_l$ of $GL_R$. Therefore, the difference ($GI_m - (GI_l + GL_r)
= GI_m - GI_l - GL_r$) is the corresponding impurity reduction.

(as a side node: this is the unweighted gini importance calculation since $GI_l$ and $GI_r$ are considered with equal weights. There are also different weighted versions. For details and a comparison please refer to "The effect of splitting on random forests", Hemant Ishwaran, Mach Learn (2015) 99:75–118)

For feature importance however, you're interested in the overall importance of each feature and not a single node. And a given feature can be present in different branches of a tree.

Here is an example:

(Source: "Artificial intelligence - a modern approach"; Norvig, Russell; 2010; 3rd Ed)

As you can see "Alternate?" is being used at multiple nodes. That is why they use set $M$ in the blog you posted to derive the overall importance of a feature $j$ in a tree $i$:

$$VIM_{ij}^{(Gini)} = \sum_{m \in M}VIM_{jm}^{(Gini)}$$

As a side node: in some papers and books the authors do not calculate the impurity indexed by features $j$ but indexed by a split value of a given node, e.g. $s$, (which of course operates on features). Don't get confused by that.

*
The Gini Impurity of a given node $m$ with $k \in K$ being the classes is defined as follows:
$$GI_m = \sum_{k \in K} \hat{p}_k(m)(1- \hat{p}_k(m)).$$

Where $\hat{p}_k(m)$ is the class frequency of class $k$ in node $m$.


Anomaly detection thresholds issue I'm working on an anomaly detection development in Python.
More in details, I need to analysed timeseries in order to check if anomalies are present.
An anomalous value is typically a peak, so a value very high or very low compared to other values.

The main idea is to predict timeseries values and, using thresholds, detect anomalies.

Thresholds are calculated using the error, that is the real values minus the predicted ones.
Then, mean and standard deviation of the error are performed.

The upper threshold is equals to mean + (5 * standard deviation).
The lower threshold is equals to mean - (5 * standard deviation).

If the error exceeds thresholds is marked as anomalous.

What doesn't work with this approach is that if I have more than one anomalous value in a day, they are not detected. This because error, mean and standard deviation are too much influenced by anomalous values.

How can i fix this problem? Is there another method that i can use to identify thresholds without this issue?

Thank you

Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard deviation formula.

You will probably have to change your critical value to something other than 5 to get the same kind of coverage. According to Wikipedia, you'll want the new critical value to be $5\sqrt{\frac{\pi}{2}}$ if your data are iid Gaussian.

An alternative that might be more difficult to implement, but is probably more statistically appropriate, is to use trimmed estimators for the mean and standard deviation. With trimmed estimators, you throw away the most extreme values in your data (the proportion of which is specified beforehand), and estimate your statistics on the remaining data.

The estimator for the mean would be the truncated mean, and the Wikipedia page for trimmed estimators mentions how to get a decent estimator for the standard deviation from the interquartile range.

I hope this helps!


How do you perform multilabel classification that is also a multiclass problem? I have a data set in which each row of data belongs to certain classes/labels.

text
class1
class2
class3

text1
pos
neg
na

text2
na
neg
na

text3
na
neu
na

text4
pos
neg
neg

text5
neg
neg
na

There are basically 4 classes with 3 labels each (pos, neg, neu, na). I suppose this is both a multiclass and multilabel problem. How do I approach this? I am using the BinaryRelevance function from multisklearn but the result always returns 2 classes only (0 and 1). What is the correct way to do this?

From the description this is not a multilabel problem because:

Each of the three "classes" (columns) must have a label. In a multilabel problem every class is optional.
Every "class" (column) appears to have a specific purpose subdivided into 4 labels. In a regular multilabel problem the labels are exchangeable, e.g. a document can have topics "sports" and "society" but not "politics", all of these labels have no order and no specific role distinct from the others.

It seems that you simply have three regular independent multiclass problems:

problem 1 = predict "class 1"
problem 2 = predict "class 2"
problem 3 = predict "class 3"

Note: the word "class" for the columns is confusing because these don't correspond to the regular concept of class.


Populate free space between two dates I'm new to data science. I'm trying to increase the time-series length for a special calculation. In the original time-series I have 20 weekly reports and I want to increase the amount of occurrences to 200. Is it ok just to use the range from the first date value to second date value between two neighbor dates?

For example if I have 1 for the first date and I have 5 for the second one, is it fine to populate the empty space between them with 2, 3, 4. Or do I need to use more advanced techniques here?

It depends on the further analysis that you want to perform. Increasing amount of occurrences can significantly effect the properties of your time series data, because you have only 10% of the data available.

In general, this is an interpolation question. There are fundamentally different statistical models that you can use to find the missing data. The Pandas dataframe.interpolate() is a good way to start.


How do I predict a set of frequently bought items? I have a dataset of retail transactions wherein different users buy certain items together. For example, a user A buys a toothpaste, a toothbrush and a floss at the same time, and a user B buys a toothpaste, a shampoo and a soap together.

I'm required to find out for each item in the dataset a list of items that are frequently bought together with it. For instance, one may expect that whenever a toothbrush is purchased, a toothpaste can be one of the items that are frequently bought together with it.

I tried to make use of the Apriori Algorithm but that would only give me sets of items that are bought together or, I'm unaware if it can somehow be modified to predict frequent purchases for a list of items in the dataset.

Which other algorithm can I put to use here?

The simplest way is to build a matrix representing how often two items are bough together: $M[i,j]$ is the count of how many times items $i$ and $j$ are bought together (note that you need to fill only a diagonal matrix, since it's symmetric).

From there you can calculate whatever you need: most frequent $B$ item bought with a given item $A$, top most frequent pairs of items in general, etc.

You could also consider the matrix as a graph representation, and use algorithms specific to graphs.


filter a sequence of coordinates Let's say that I receive lists of coordinates (latitude,longitude), each representing a 2D route. Later I want to use this data for route prediction, but first I have a preliminary problem to solve: I was expecting these lists to be sorted and, although the majority of the coordinates of each route are indeed sorted, some are not. For example:

a) expected:

b) actual:

where t is a timestamp.

I am looking for solutions to address this problem, for example remove all coordinates like b.t3 or sort the lists appropriately. Does anyone have any suggestion?

This raises many questions that you may or may not be able to answer, and this may not be helpful but hopefully it is.

Regarding the data, what are it's origins?  Is it all electronically collected or is it an amalgamation of, say delivery driver logs that could be easily falsified or recorded incorrectly? Asked another way, how do you know that the order, based on timestamps, is not correct?

On the other hand, if you know the relationship between the points on the route, why bother with the timestamp values?  From your comments, it seems as though the participant traveled in a straight line.  If that is the case, you know the order based on the coordinates. You could order them by distance from the mean of the coordinates.

Since there is some disagreement between the "known" route and the "recorded" route I think to resolve the data you need to understand what causes this disagreement?  Is the timestamp produced by one device or multiple?  If one device, maybe someone is not driving the advertised route.  If multiple devices, maybe one is set to the wrong timezone or just has the wrong time.

Again, not really an answer, but hopefully helpful.


Ingredients, Recipes and recipe ratings. I would like to predict the rating based on combination of ingredients I would really appreciate some help on the first steps to my problem, suggestions of modeling techniques i could use or relevant research (i could not find any).

I have a list of ingredients (150 in total) and a list of recipes (100 or so) with data on which combination of ingredients are present in each recipe (on average 4-5 ingredients per recipe).

My output variables are 4 metrics of ratings of those recipes (taste, smell, look, and texture) for each of those 100 recipes from a scale of 1-10 for each.

My aim is to predict the 4 metrics based on a new combination of ingredients (picked from the list) that i specify.

What modeling technique should I look at to help solve this prediction problem?

Here are a few ideas to start with:

A simple model would be to use multiple linear regression (MLR), or a Random Forest

If you want to evaluate only based on if an ingredient is used, your input dataset could look like this:

Butter   Flour    Eggs  ...  Test   Smell   Look   Texture
1        0        1          1      5       2      4

If you want to predict on how much of each ingredient (some is better, too much is not good) your dataset could look like this:

Butter   Flour    Eggs  ...  Test   Smell   Look   Texture
1.5       3.0        2          1      5       2      4

You can get some other ideas from this website:

https://machinelearningmastery.com/multi-output-regression-models-with-python/


Perform clustering on individual data in categories I am looking for ways to perform clustering on an aggregated category while still using individual data.

For example, assume we have a multitude users that we have acquired through different channels. I would like to cluster groups of channels together, so that it is easier to deal with similar channels in similar ways. However, if I aggregate the user data into averages for each channel, then because these averages are heavily driven by outliers (high-spenders), it doesn't give that much insight to the behavior of the majority of users from that channel.

Is there a way to use the individual user data but still gain insight into and cluster based on the channel they come from?

As always clustering is about what is the meaning of distance, since that encodes at least some part of your question. So you question is which channels are similar, where similarity is defined on users. Usually you do not assume some nesting on your instances, but what you have is basically a nesting of users in channels. So you have to incorporate this kind of nesting into the distance / similarity function.

I would start with some observation on similarity function. We denote similarity between instance $i$ and $j$ with $d(i,j)$. Usually this function obeys the following condition $d(i,j)\ge0$ for any $i$ and $j$. Note that we usually have equality only on identical data points. The main consequence of this observation is that we can have basically an identity at user level. What you want is an identity at channel level.

One way would be to define the similarity function like:

$$d_c(i,j) = I_{c_i\ne c_j}d(i,j)$$

where $I_{c_i\ne c_j}$ is $1$ when channel of instance $i$ is different that channel of instance $j$. The main effect is that now all the points from the same channel are considered identical since the distance between them is zero. When the identity function is $1$ the distance is given by your real business distance which should be constructed by you and answer your question.

If you use such kind of function in a hierarchical clustering basically it will start to find first some clusters which are identically with your grouping on channels and later on it will join clusters which are similar. This kind of approach will work even with a kmeans algorithm and perhaps with most of the clustering approaches.

A slight different approach would be to define your $I$ function to return $1-\lambda$ when cluster is different and $\lambda$ when channels are equal, and have $\lambda$ a positive value close to $0$. This will not guarantee that all the clients will go into the same cluster, but it gives you the benefit that you have a slider which you can use to fine tune the compromise between all user of the same channel goes into the same cluster and a distance measure which is more robust to outliers.

A totally different approach from an implementation point of view would be to define a more complex function directly on channel samples. This would be similar with how hierarchical clustering works since in order to joins to clusters it should have a distance function which would measure inter clusters similarity. See more on linkage-criteria. For example average linkage clustering. Note that I said that the approach is different only algorithmic. I would bet that the results would be similar with the first approach.

A totally different approach would be to use a more robust criteria. It is known that sample average is not robust since one point can blow away the estimation. Median instead is much stable. You can use a median or a trimmed mean to have a more robust aggregation values. This would have the advantage that the clustering would be much faster since you would work with channels instead of clients and the running time for computing the clustering would be reduced.

And finally, another approach which comes to my mind would be to go further with comparing channels, but this time using a distance which would be based on statistical tests. I will give you a scenario to clarify. Suppose that your users have an attribute named Weight, which as expected would be a continuous variable. How could you define a distance between the Weight of users of one channel and Weight of users of other channel? If you can assume a distribution, like a Gaussian on that weight you can build a two sample t-test on the two samples which are the two clusters. If you can't assume a distribution you can employ a homogeneity / independent test like two sample KS test. KS test is valuable since does not assume any distribution and is sensitive to both changes is shape and location. If you have nominal attributes you can employ a chi-square independence test. Be careful with what you can use from that tests. In order to have equal contribution for each attribute used in distance function you have to you p-values. Also note that if the test is significant it will have a small p-value, since the null hypotheses for both tests is the independence assumption, which can be translated as same marginal for both samples. So, smaller p-values means bigger distance. You can use $1-\text{p_value}$ or even you can try $\frac{1}{\text{p_value}}$.


Feature Selection and PCA I have a classification problem. I want to reduce number of features to 4 (I have 30). I'm wondering why I get better result in classification when I use correlation based feature selection(cfs) first and then employ pca in comparison with just employing pca (the latter one is worse than the first one). It also should be mentioned that data loss in the second approach (just pca) 0.2-variance cover:0.8- and in the first one is 0.4 -variance coverd: 0.6!

Thank you in advance

PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be correlated with each other but have minimal relevance to your target variable. That's probably what is happening in your case.

Consider a toy example. Lets say I want to predict stock prices. Say I'm given four predictors:

Year-over-year earnings growth (relevant)
Percent chance of rain (irrelevant)
Humidity (irrelevant)
Temperature (irrelevant)

If I apply PCA to this data set, the first principle component would relate to weather since 75% of the predictors are weather related. Is this principle component relevant? It's not.

The two options you've highlighted boil down to using CFS or not using it. The option that uses CFS does better because it explicitly selects variables that have relevance to the target variable.


How to separate words that are together in a large data set in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'

i tried this but this is slow

def split(sentence, word_to_index):
# words to index dictionary of valid words
    sentence_words = sentence.split()
    lst12=[]
    for w in sentence_words:
            lst=[]
            ws=w[:]
            # Set the (i,j)th entry of X_indices to the index of the correct word.
            while True:
                for wd in lst:
                    ws = ws.replace(wd,"")
                if ws=="":
                    lst=" ".join(lst)
                    lst12.append(lst)
                    break
                wrdind = ws[:]
                for kl in range(len(ws)):
                    try:
                        word_to_index[wrdind]
                        lst.append(wrdind)
                        break

                    except KeyError:
                            wrdind = wrdind[:len(wrdind)-1]
                            continue
   return " ".join(lst12)
```

This commonly called a "word break" problem. There are a variety of approaches, the most common use dynamic programming or tries. You can recursively try candidates and keep the candidates if they can split the entire string.

Here is a version (inspired by this answer):

from functools import lru_cache

@lru_cache(None)
def wordbreak(string):
    if len(string) < 1:
        return " "
    else:
        for i in range(len(string)):
            substring = string[:i+1]
            if substring in valid_words:
                remaining_split = wordbreak(string[i+1:])
                if len(remaining_split) > 0:
                    return substring + " " + remaining_split
                else:
                    return ""
            if (i == len(string) - 1) and (substring not in valid_words):
                return ""

valid_words = {'man', 'boycott', 'bears'}
assert wordbreak('boycottbears') == 'boycott bears'
assert not wordbreak('abc')

However, creating a complete collection of valid words is impossible for Twitter data since there are always new words.


Word2Vec for Named Entity Recognition I'm looking to use google's word2vec implementation to build a named entity recognition system.  I've heard that recursive neural nets with back propagation through structure are well suited for named entity recognition tasks, but I've been unable to find a decent implementation or a decent tutorial for that type of model. Because I'm working with an atypical corpus, standard NER tools in NLTK and similar have performed very poorly, and it looks like I'll have to train my own system.

In short, what resources are available for this kind of problem?  Is there a standard recursive neural net implementation available?

Two recent papers use a Deep learning architecture called CharWNN to address this problem. CharWNN was first used to get state of the art results (without handcrafted features) on Part of Speech (POS) tagging on an English corpus.

The second paper by the same author uses the same (or similar) architecture for predicting whether a word belongs to 10 Named Entity classes, with apparent state of the art results.


Are there any python libraries for sequences clustering? I have a problem which I explained in other question. I've understood that my dataset is a sequence of states or something like that. Is there libraries to analyze sequence with python? And is it right way to use Hidden Markov Models to cluster sequences?

Is there libraries to analyze sequence with python

You can take a look at here. You can also use TensorFlow if your task is sequence classification, but based on comments you have referred that your task is unsupervised. Actually, LSTMs can be used for unsupervised tasks too depending on what you want. Take a look at here.

  And is it right way to use Hidden Markov Models to cluster sequences?

Markov hidden models are those that your current state does not depend on all previous states. If you your task has longterm dependencies, you can use LSTM networks. If your data does not have longterm dependencies you can use simple RNNs.


What does 'Linear regularities among words' mean? Context: In the paper "Efficient Estimation of Word Representations in Vector Space" by T. Mikolov et al., the authors make use of the phrase: 'Linear regularities among words'.

What does that mean in the context of the paper, or in a general context related to NLP?

Quoting the paragraph from the paper:

  Somewhat surprisingly, it was found that similarity of word
  representations goes beyond simple syntactic regularities. Using a
  word offset technique where simple algebraic operations are performed
  on the word vectors, it was shown for example that vector(”King”) -
  vector(”Man”) + vector(”Woman”) results in a vector that is closest to
  the vector representation of the word Queen [20].

  In this paper, we try to maximize accuracy of these vector operations
  by developing new model architectures that preserve the linear
  regularities among words. We design a new comprehensive test set for
  measuring both syntactic and semantic regularities1 , and show that
  many such regularities can be learned with high accuracy. Moreover, we
  discuss how training time and accuracy depends on the dimensionality
  of the word vectors and on the amount of the training data.

By linear regularities among words, he meant that "Vectorized form of words should follow linear additive properties!"

V("King") - V("Man") + V("Woman") ~ V("Queen)


HDBSCAN cluster: still unclear to me how to chose 'min_cluster_size` Hdbscan is an excellent technique to find the "optimal" number of clusters within your data when you have little a priori idea how many clusters should exist. This makes the method great for exploratory analysis:

http://hdbscan.readthedocs.io/en/latest/comparing_clustering_algorithms.html

Here's my problem: All results using hdbscan with the python implement in the link above rely on the crucial min_cluster_size

http://hdbscan.readthedocs.io/en/latest/parameter_selection.html

If users have a priori little idea how many clusters best fit the data, what is the correct approach above? Isn't there a metric one uses to decide what the optimal number of clusters is?

Optimal in which sense?

The crucial thing with clustering is that there is no optimal solution. Different solutions tell you a different part of the story. And to be able to get different views, you will need parameters. It is a exploratory technique.

Various attempts at defining "optimal" solutions have failed for practical use, just think of k-means.


Load large .jsons file into Pandas dataframe I'm trying to load a large jsons-file (2.5 GB) into a Pandas dataframe. Due to the large size of the file  pandas.read_json() will result in a memory error.

Therefore I'm trying to read it in like this:

S_DIR = r'path-to-directory'

with open(os.path.join(S_DIR, 'file.jsons')) as json_file:
    data = json_file.readlines()
    data = list(map(json.loads, data))

df = pd.DataFrame(data)

However, this just keeps running, slowing/crashing my pc.

What would be the most efficient way to do this?

The final aim is to have a subset (sample) of this large file.jsons dataset.

Thanks

There are a few ways to do this a little more efficiently:

JSON module, then into Pandas

You could try reading the JSON file directly as a JSON object (i.e. into a Python dictionary) using the json module:

import json
import pandas as pd

data = json.load(open("your_file.json", "r"))
df = pd.DataFrame.from_dict(data, orient="index")

Using orient="index" might be necessary, depending on the shape/mappings of your JSON file.

check out this in depth tutorial on JSON files with Python.

Directly using Pandas

You said this option gives you a memory error, but there is an option that should help with it. Passing lines=True and then specify how many lines to read in one chunk by using the chunksize argument. The following will return an object that you can iterate over, and each iteration will read only 5 lines of the file:

df = pd.read_json("test.json", orient="records", lines=True, chunksize=5)

Note here that the JSON file must be in the records format, meaning each line is list like. This allows Pandas to know that is can reliably read chunksize=5 lines at a time. Here is the relevant documentation on line-delimited JSON files. In short, the file should have be written using something like: df.to_json(..., orient="records", line=True).

Not only does Pandas abstract some manual parts away for you, it offers a lot more options, such as converting dates correctly, specifying data type of each column and so on. Check out the relevant documentation.

Check out a little code example in the Pandas user guide documentation.

Another memory-saving trick - using Generators

There is a nice way to only have one file's contents in memory at any given time, using Python generators, which have lazy evaluation. Here is a starting place to learn about them.

In your example, it could look like this:

import os

# Get a list of files
files = sorted(os.listdir("your_folder"))
# Load each file individually in a generator expression
df = pd.concat(pd.read_json(file, orient="index") for f in files, ...)

The concatenation happens only once all files are read. Add any more parameters that are required where I left the .... The documentation for pd.concat are here.


How to integrate nvidia gpu with jupyter notebook for scikit-learn I know there are answers saying that tells how to integrate/use tensorflow or for deep learning libraries. I want to use it for classification or regression scikit-learn libraries. Is it possible. If yes then how ?

There is no way to use GPU with scikit-learn as it does not officially supports GPU, as mentioned in its FAQ.


Predicting number of cars I am predicting the number of cars from a traffic dataset.
Here is my data dictionary:
The ‘Traffic-Major-Roads(kilometres)’ file contains the following variables (variable names
are in bold):

Year - Traffic volumes are shown for each year from 2000 onwards.
CP (count point) – a unique reference for the road link that links the AADFs to the road
network.
ONS GOR Name – the former Government Office Region that the CP sits within.
ONS LA Name – the local authority that the CP sits within.
Road – this is the road name (for instance M25 or A3).
RCat – the classification of the road type (see data definitions for the full list).
S Ref E – Easting coordinates of the CP location.
S Ref N – Easting coordinates of the CP location.
A-Junction – The road name of the start junction of the link
B-Junction – The road name of the end junction of the link
LenNet – Total length of the network road link for that CP (in kilometres).
PC – Traffic volume (in thousands of vehicle kilometres) for pedal cycles.
2WMV – Traffic volume (in thousands of vehicle kilometres) for two-wheeled motor
vehicles.
Car - Traffic volume (in thousands of vehicle kilometres) for Cars and Taxis.
Bus – Traffic volume (in thousands of vehicle kilometres) for Buses and Coaches
LGV – Traffic volume (in thousands of vehicle kilometres) for LGVs.
HGVR2 – Traffic volume (in thousands of vehicle kilometres) for two-rigid axle HGVs.
HGVR3 – Traffic volume (in thousands of vehicle kilometres) for three-rigid axle HGVs.
HGVR4 – Traffic volume (in thousands of vehicle kilometres) for four or more rigid axle
HGVs.
HGVA3 – Traffic volume (in thousands of vehicle kilometres) for three or fourarticulated axle HGVs.
HGVA5 – Traffic volume (in thousands of vehicle kilometres) for five-articulated axle
HGVs.
HGVA6 – Traffic volume (in thousands of vehicle kilometres) for six-articulated axle
HGVs.
HGV – Traffic volume (in thousands of vehicle kilometres) for all HGVs.
AMV – Traffic volume (in thousands of vehicle kilometres) for all motor vehicles.

I need to predict the variable AMV.
So, I have one-hot encoded Road, and kept date, time in my features.
But, number of Roads being very large. I have too many features.
Can you please suggest how should I proceed?

I am having too many features.

No, you don't :).

First of all, it is highly likely that not all of them are important for the prediction that you want to make.

I would highly recommend using a CART Random Forest for regression of the variable of interest. It literally requires minimal coding if you choose to do it in python using the RF algorithm from the sklearn package.

It's big advantage is that it is straightforward to use and understand and, moreover, it provides you with the learnt feature_importances_ of all inputs
after training, so that you can exclude the least important ones and speed up the inference/training in the future.

-EDIT-

To understand the difference between Classification and Regression Decision trees, check this helpful link.

The decision tree implementations for regression are commonly the C4.5, the C5.0 or the CART algorithm. The one that is used by sklearn is CART, please take a look at section 1.10.6 in this link.

A good example of how to use the sklearn Decision Tree for regression is this.


How to use a a trained model I just trained my first model in Python 3.7/scikitlearn (Linear Regression) (well I copied most of the code but its something ^^).

Now I want to actually Use the model. Specifically its about sons heights incorrelating to their fathers. So I now want to enter a new Father-height and get a predictions for its sons height.

How could something like this look like?

I read about "Pickle" to save a model and use it later, seems awsome but how would I use such a saved model?

If anybody can give me a simple answer or even just a link to atutorial would be great. Below is a piece of "my" code just for some context.

#Spliting the data into test and train data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)

#Doing a linear regression
lm=LinearRegression()
lm.fit(X_train,y_train)

#Predicting the height of Sons
y_test=lm.predict(X_test)
print(y_test)

You have your model saved as the variable lm. You can use the lm.predict(X_test) for any other test scenario. Note that your X_test should be similar to your X_train meaning if you have transformations made on your X_train, you need to do similar transformation on X_test too. You can use pickle in the following way

import pickle
filename = 'model.pckl'
pickle.dump(model, open(filename, 'wb'))

#To load the model from disk, use this
model = pickle.load(open(filename, 'rb'))
prediction = model.predict(X_test)


Feeding 3 consecutive video frames to a CNN to track a tennis ball I want to use CNN transfer learning to track a tennis ball from TV broadcasts of tennis matches.
I used VGG annotating tool annotation tool link (use version 1 of the tool for compatibility with matterport code) and have about 200 frames annotated with ball location and x,y coordinates given by the tool for the bounding circle.

Like this:

But, the ball is occluded by the bottom player's body or the net tape at times and at others practically invisible because it's moving too fast (other times, it is elliptical in the direction its moving).

A potential solution I saw used is below in an algorithm called Tracknet.
Tracknet:  Tracknet ball tracking I contacted the creators of it and was told that they would open source but it's been over a year so I want to try to mimic it.

Edit:  My Q and A via email with the Tracknet Team:

  No.

  We concatenate three images that are 9 slices in total and then input
  them together into a network but keep the rest of the network the
  same. In such a method, the calculation overhead is only on the first
  layer.

  If you would like to have a design like the figure in your previous
  email, we would suggest you applying an RNN or LSTM to track the ball.

  Best, Tsi-Ui

"9 slices in total"...does this mean RGB for each of 3 consecutive frames?

Also, I know from Karpathy Pong, how to feed difference frames but how would I feed 3 consecutive frames as done in Tracknet?

I know that 2 frames is not enough having manually gone through them and 3 seems to be the minimum required to have at least one visible tennis ball.

Furthermore, I was advised by Adrian Rosebrock of pyimage that I would need trajectory estimation and a high FPS camera so this is another avenue for investigation although Tracknet seemed to do it with neither of these features.

Edit:  I am reading the Deep Learning Book chapters on CNNs to learn more about how they process input info at a low level so I can figure out what concatenate 9 slices means.  Book chapter:  CNN chapter
Just as a thought, I was thinking of calculating a collection of difference frames at t - (t-1), t - (t-2),...,t- (t-n) which might help in approximating ball location.

Edit:  I just saw this video by Andrew Ng on outputting real numbers from a nn for object detection and localization:  coursera video
So, if that is possible, then I can output y =[x_0, y_0, x_1, y_1, x_2, y_2] which are the x, y coordinates of the center of the ball at t, t+1, t+2 and take the mean squared error between y-hat and y for a loss.
Note:  When feeding the network, I might have overlapping frames, ie. feed [t, t+1, t+2] and then feed [t+1, t+2, t+3] which has the intersection of t+1, t+2.  Also for frames for which the ball is too blurry, I am going to train a separate nn that gets inputs of ball location at t_0 and t+n concatenated and outputs ball location at t_j where j is sometime between 0 and n.  The use these outputs to annotate the "invisible ball" frames, then use this "complete" set of ground truth frames concatenated as the training set for the main nn tha tracks the ball in video.

The researchers who made Tracknet finally open-sourced their code and will do so with dataset as soon as they settle their copyright issues.

Here are the links:

Arxiv paper:  Tracknet Arxiv paper

Tracknet repo

If the link is broken, let me know and I'll upload the zip and tar.gz along with Readme file to Dropbox.

Here is a link to the resulting video with tracked ball.
Fed Nadal demo

It is not perfect so probably going to combine this method with blob detection, difference frames used in other papers.  Also need to take out the lines, players and other background from being detected.

Here is the code snippet where they concatenate three frames in predict_video.py.

#combine three imgs to  (width , height, rgb*3)
X =  np.concatenate((img, img1, img2),axis=2)


Feature importance and deriving rules using tree based classification models I have a dataset where I have categorical and continuous values with targets 0/1 (binary classification task). Since I need to find patterns and relationships in the occurrence of the event or target, I think I should use Decision trees. However the issue is that I have 2 categorical variables which have 700 & 150 categories respectively with the remaining variables being numerical/continuous.

My questions are as follows:

Can I estimate feature importance using random forests in sklearn for this set of variables? If yes, do I need to change the type of variables?
Can I use the variables as they are, i.e categorical with so many categories each & continuous variables, when training a decision tree using sklearn in order to be able to visualize rules from the tree?

Please feel free to suggest any other approach too. The prime objective of the task is to identify the drivers/features which impact the target & generate a pattern - if at all can be identified from the data.

Also, given these many categories for the 2 variables, how do I conduct a statistical analysis for this variables alone with the target to understand if any correlation exists or identify cause effect relationships.

You can use categorical features with decision trees in scikit-learn, but you'll need to encode them as numbers. If your categorical features are ordinal (such as ranking ‘bad’, ‘fair’, ‘good’), they are easy to encode in numbers that respect the underlying ordering (e.g. 0, 1, 2). For nominal features, given the high cardinality you mention, you can try some version of the target encoding or the frequency encoding. With the target encoding, make sure to use some form of regularization in order to avoid overfitting.

After the features are encoded, you can build, for example, a random forest and obtain the impact of each feature with attribute feature_importances_.


Classifying survey response text SVM I have 800 responses to an open-ended survey question. Each response is categorized into 3 categories based on a list of 70 categories. These categories are things like "stronger leadership", "better customer service", "programs", and etc...

My question is, can I use this as a training data set in order to develop a model that I can use in the future as we get more survey responses? We would like to be able to tag, label, or classify each survey response into (up to) 3 of the 70 categories.

Is this even possible? Or do I have to use a NB with simple words? Can you please guide me to tutorials, examples, etc.?

Using R in this exercise.

Can you clarify what you're trying to predict with these responses?

My initial reaction is that with open-ended surveys you'll have a tough time implementing classification algorithms. Open-ended-ness means you don't have a finite feature space and thus you can't do the usual transformation of responses into a feature matrix.

However, there may be other ways to make this work. If you have 5 questions, for instance, you may be able to use sentiment analysis or other methods to come up with metadata about the open ended response that can help you design a classification scheme.


Seems like the Keras .fit and .evaluate methods give different training accuracy (yet the same validation accuracy). Same thing on loss.? The code below is a translation of Nielsen's first mnist code to Keras.  Surprisingly, the last accuracy value of the .fit method and the accuracy value for the .evaluate method are different for the training data.  As I expected, for the validation data the accuracies are the same. The same behavior is seen with losses.  I would appreciate finding out what is going on.
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers

num_classes = 10
input_shape = (28, 28, 1)

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)
# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)  #adds another dimension at the end for color
x_test = np.expand_dims(x_test, -1)

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

initializer = keras.initializers.RandomNormal(mean=0., stddev=1.)

model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Flatten(),
        layers.Dense(30, activation="sigmoid", use_bias=True,
                     kernel_initializer=initializer,
                     bias_initializer=initializer),
        layers.Dense(num_classes, activation="sigmoid",
                     kernel_initializer=initializer,
                     bias_initializer=initializer),
    ]
)
batch_size = 10
epochs = 2
opt = keras.optimizers.SGD(learning_rate=3)
model.compile(loss="MeanSquaredError", optimizer=opt, metrics=["accuracy"])
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=1/6)

# Evaluate the trained model
print("\n")
score = model.evaluate(x_train[:50000], y_train[:50000], verbose=0)
print("evaluate train accuracy:", score[1])
score = model.evaluate(x_train[-10000:], y_train[-10000:], verbose=0)
print("evaluate validation accuracy:", score[1])
print("history last train accuracy:",ann.history['accuracy'][-1])
print("history last validation accuracy:",ann.history['val_accuracy'][-1])

training's loss/accuracy are not calculated for the whole dataset every time.
It is calculated for the current batch and averaged successively.
test's loss/accuracy is calculated at the end of an epoch for the whole test data.
Check these references
Keras FAQ
SO Answer
You may put the validation data same as the train data to check it.
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_train,y_train))

# Evaluate the trained model
print("\n")
score = model.evaluate(x_train, y_train, verbose=0)

Or, you may write a custom callback to print appropriate logs as suggested in the SO answer.