Character recognition neural net topology/design I'm building a neural net for classifying characters in pictures. The input can be any character a-z, lowercase and uppercase. I only care about classifying the characters, and not the case, so the neural net has an output vector of length 26; one for each character. It makes sense, intuitively, to have a hidden layer of size 26*2 just upstream of the output layer. It also makes intuitive sense for this layer not to be fully connected to the output layer, but instead having two and two hidden nodes connect to each output node.

I have some questions:

a) Does this make sense? I'm getting about 75 % success rate on a pretty hard data set with just one hidden layer, but I'm not certain on how to improve from there.

b) If so, What activation function should I use from the hidden layer with 26*2 nodes to the output layer? Maybe I should use an OR function for this, since both the lowercase and the uppercase version of a character should output for a single character.

c) Would it be wiser to have 26*2 output nodes instead, and just combine lowercase and uppercase outputs after the neural net?

Your design makes some sense, but there is no need to limit connections even if you expect to represent probabilities of upper/lower case separately, because they will interact usefully. E.g if the character could most likely be one of o, O, Q, G then this might be useful information to choose the correct one.

If you went ahead, you would need to train this network without the final layer (so that it learns the representations you expect, not some other group of 52 features), then add the final layer later, with no need for special connection rules, just use existing ones. Initially you would training the new layer separately from the full output of the 52-class net i.e. probability values, not selected class. Then you would combine with the existing net and fine-tune the result by running a few more epochs with a low learning rate on the final model.

That all seems quite complex, and IMO unlikely to gain you much accuracy (although I am guessing, it could be great - so if you have time to explore ideas, you could still try). Personally I would not take your hidden layer idea further. The full 52-class version with simple logic to combine results is I think simpler. This is also not necessary, the neural net can learn to have two different-looking images be in the same class quite easily, provided you supply examples of them in training. However, it may give you useful insights into categorisation failures in training or testing.

It is not clear from the question, but if you are not already using convolutional neural network for lower layers, then you should do so. This will make the largest impact on your results by far.


Should we normalize test data by choosing maximum and minimum value of training data? I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that while performing min-max normalization we pick minimum and maximum value in each sample's features. For the test data should I choose minimum and maximum of each sample Or should I normalize it with  the minimum and maximum value of training data?

You should use your training data fit function (by fit function i mean the function of the scaling that is learning from your data)simply because your test dataset size and feature values could be different than your training which would result in a scale different than the scale used on your training data. imagine you want to infer your model with 1 row which will be a problem if you try to fit 1 row on the minmaxscaler.

Example :
Let's say your max value is 100 and min value is 0 on your training data for a particular feature, you scale that to [0,1] => 100 will be 1, 0 will stay as so.
70 will be 0.7 , 60 = 0.6 and so on. You move on to the test set and apply scaling on it but it happens that the max value for that same feature in your test set is 70 ( 100 unfortunately for you happens to exist only in the training set ). Your 70 in the test set if you scale using the test set will be a 1. You feed the 1 ( 70 ) to your model who's trained to consider a 1 as a 100 thus the error.


Difference between Word Embedding and Text Embedding I am working on a dataset of amazon alexa reviews and wish to cluster them in positive and negative clusters. I am using Word2Vec for vectorization so wanted to know the difference between Text Embedding and Word Embedding. Also, which one of them will be useful for my clustering of reviews (Please consider that I want to predict the cluster of any reviews that I enter.) Thanks in advance!

A Text embedding is a vector representation of a text. A trivial way to construct a text embedding is to average the word embeddings of each word in the text.
However using this method, you will lose contextual information.


Use of TfidfVectorizer on dataframe I have the dataframe which has two colums(Reviews and Label):

  Reviews                                     Label
0 [cameron, diaz, woman, marri, judg, play]   1
1 [turgid, dialogu, feebl, character, harvey] 0
2 [misfortun, watch, rubbish, sky, cinema]    1

I want to apply the TfidfVectorizer on the DF.
I have written the following code.

from sklearn.feature_extraction.text import TfidfVectorizer
df_x=train_df["Reviews"]
df_y=train_df["Label"]
cv = TfidfVectorizer()
df_xcv = cv.fit_transform(df_x)
a=df_xcv.toarray()
cv.get_feature_names()

which is giving an error:

  AttributeError: 'list' object has no attribute 'lower'

Why is this throwing an error?

Your reviews column is a column of lists, and not text. Tfidf Vectorizer works on text. I see that your reviews column is just a list of relevant polarity defining adjectives. A simple workaround is:

df['Reviews']=[" ".join(review) for review in df['Reviews'].values]

And then run the vectorizer again. That will fix the problem.


Understanding the multidimensional-nature of the data being fed to a RNN and its output Assuming we have a time-series dataset whose window_size = 30 and the batch_size = 4, which makes the overall input = 4*30 (2D). But as RNN expects 3D input, tf.expand_dims is used to make it a 3D input (as per the lecture, new inut becomes 4*30*1, where the last dimension is 1 as the example deals with a univariate time-series).
What I don't get is that what does adding a dimension mean? Eg. what will be the element [0,0,0] of the input?

Also in keras, the typical format for fitting is

model.fit(input, output, epochs=400)

But in an RNN sample code for time-series data, I found

model.fit(dataset, epochs=400)

where  dataset is a tf object containing the time-series data.
Why is the input and output not given explicit  for the model to train in case of the first code? The timestamp is already included in the input in a way(in the 4*30*1 input, the 2nd dimension is supposed to be time-stamps), but how does the keras know against what output labels the input has to be trained?

I think you’re confused about what a “batch” is. A batch has a very specific definition in machine learning.

In my experience:

Dimension 1 = number of bins or number of data points per each time step
Dimension 2 = window size, the number of time steps
Dimension 3 = batch size, the total number of examples which you’ll feed the network per training batch

Looks like this for [4,3,2]:

[[[0,1,4,3],[0,8,6,9],[9,6,7,4]],[[6,8,7,0],[1,7,7,9],[1,3,5,8]]]

4 values per time step, 3 total time steps per example, 2 examples in the batch.

Where [0,0,0] returns 0, [3,1,1] returns 9.

Also, one friendly piece of advice - don’t think in terms of time stamps. You’re not dealing with time in an RNN. You’re dealing with steps in a sequence. The sequence can be related to time.

Edit: With respect to the tensorflow stuff, a tf.dataset object can contain both the input data and the labels for each example. The point is to make writing code easier, managing data easier etc.

The process for training/testing is the same.


Dealing with correlated features when calculating permutation importance I have implemented the permutation importance calculation as found here in my attempt to identify features that contribute little to my model's (Gradient Boosted Tree model) predictive power.

The issue I have encountered is that some of my features are highly correlated, potentially masking the true importance of features when being evaluated by permutation importance. Typically the solution to this would be to perform something like Recursive Feature Elimination instead. Unfortunately, I cannot do this as the cost of retraining the model is prohibitive. The model takes ~3 hours to train with a feature set of 39 features.

My question is whether it is possible to use permutation importance while dealing with correlated features? My initial thought was to invert the process and shuffle all features other than the one I want to investigate although I do not know if this would have the same level of explanation.

Using mutual information namely the correlation of each feature and the output is not something that can be helpful very much. The reason is that the correlation coefficient is just capable of finding whether they are linearly dependent or not. Other than Gaussian distribution, as far as I know, it is not even capable of finding out whether the inputs are independent or not which means if you see the coefficient is equal to zero, if you don't know the distribution you cannot conclude that they are independent but you can say they are not linearly dependent. In real-world applications, it scarcely happens that features just have linear dependencies with the output. Consequently, approaches like this are not very helpful because you may have two features that can have a better relationship with the output but none of them has a good separation with the output.

For cases that you have a lot of features, you can decide to choose other feature selection and extraction methods. I guess for your case wrapper method may suit better. In the wrapper method, you don't have any criterion and you just search through all possibilities of features using heuristic-based methods in order to find the best sub-features which have the best cross-validation accuracy.


Classification algorithm that only matches trained examples I have 10 categorical features and a multi-class target.
Training data contains rows where the same 10 categorical features may map to a different target class.
What classification algorithm should I choose that fulfils the following criteria:

prediction output should only show results if the prediction input matches a row example that existed in the training data. (I don't want to make assumptions on inputs it has never been trained on, as this model will deal with people's lives in the real world)
prediction output should be targets that match exactly the prediction input, ordered by their frequency in the training data, with the highest as 'the best match' at the top

I realize this looks like it could be 'easily' solved with databases but I would like to use Classification and train a model instead of having a cumbersome db infrastructure with columns, keys, indexes and other boring things.

Strictly speaking Machine Learning is not the answer to this problem imho, because a ML method always works by generalizing what is in the data. In other words ML methods are meant to make some guesses while statistically minimizing the risk of error: if you don't want any generalization or risk of error, you don't want ML.
There might be some symbolic methods which could do what you describe, but essentially it's just a very simple deterministic method:

Obtain the frequency for each distinct instance (including label) in the training data in a map
For each distinct instance (only features) keep the most frequent one and discard all the others.
When applying, just look up the instance in the map and assign the corresponding label.

a cumbersome db infrastructure with columns, keys, indexes and other boring things.

There's a confusion here: the question of using a database or not is irrelevant here since it's a matter of how you store the data, it's just not related to using ML or not.
Usual advice: use the right tool for the right job... even if it's boring ;)


Can we use median to replace all the missing values from a column If we can use the median to replace all of the missing values from a column, then what is the advantage of using median over mean for replacing the missing values in a column with numeric values?

imputing with median/mean affects distibution of the data, essentially squeezes it, and it does not account for variance in the data, not to mention we'll be making up data. Of course, there're some cases that it may not matter as much and some cases where it will.

If it's worthwhile your time and in case it adds value to your work, you can look into other methods such as proper stochastic regression imputation. Using this method you can randomly add value using linear regression imputation method and create several models and aggregate the results. In this case you'll be accounting for variance in your data and not essentially makin up data, but it's more computationally consuming of course in comparison to just fillna.

There's new method to impute values based on missing patterns, where you'll break your data based on the pattern and create models for each and aggregate.
Of course if it's worth the time and effort, these methods should be more beneficial.
Also you can look into python's fancyimpute

Here are some good resources that cover various methods of imputing missing values:
1) http://www.stat.columbia.edu/~gelman/arm/missing.pdf
2) http://www.bu.edu/sph/files/2014/05/Marina-tech-report.pdf
3) https://stefvanbuuren.name/fimd/sec-nonnormal.html
4) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2727536/


Data Visualization with multiple dimension, and linear separability I have a dataset of two classes with several features, how can I visualise such data using Matlab to decide if it is linear separable or not?

You basically need a t-SNE plot, the t-SNE will convert the high dimensional feature vector (several features in your case) to a 2d point and then you can use matplotlib to plot, while plotting you need to send in the class of the corresponding feature to get different colour for different classes of data points. once you do this, you will be able to judge if they are linearly separable.

the code goes as follows

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=0)
transformed_data = tsne.fit_transform(features)
k = np.array(transformed_data)
plt.scatter(k[:, 0], k[:, 1], c=class_labels, zorder=10, s=0.4)

and the above code should produce an image as shown below

Since you're looking for MATLAB code you can get the t-SNE with the function tsne(x) click on it for documentation. Also matplotlib scatter plot is pretty easy to do on Matlab, the documentation is extensive!

vote me if i was able to help ;)


(Scikit-learn) differences between LinearSVC, 'linear' kernel SVC and poly kernel SVC with degree 1 I would like to know the differences between:

linearSVC()
SVC(kernel='lineaer)
SVC(kernel='poly', degree=1)

Will these all determine the same boundary decisions?

You can see in the user guide that the first two will not produce the same results.  In the multiclass setting, LinearSVC uses one-vs-rest while SVC uses one-vs-one.  Aside from that, the solver used is also different, and because of that some of the options differ.  See LinearSVC docs, SVC docs.

I suspect the third will be the same as the second.  I don't see anything in the source to indicate it quickly, but it shouldn't take long to test.


Fine Tuning the Neural Nets I have recently read about Fine Tuning, and what I want to know is, when we are fine-tuning our model is it necessary to Freeze the model and train only the top part of the model and then unfreeze some layers and again train the model or one can directly begin by unfreezing some layers? Till now, I have read that one does not unfreeze the layers directly because then we risk losing the important features captured by the earlier layers

It is a good practice to freeze the early layers when you finetune the model. There are two reasons why you want to do this :

Your new layers is initialized randomly and will always start with very big loss. Big loss means big gradient and if the weights are not frozen this will be propagated . Your model will become less stable and might fail to converge. To put it simply you don't want to put the blame of false prediction on the earlier layers.
You are saving a lot of computation resources. Training a neural network, is not a small issue, and training only a small section of the model will give you decent enough accuracy without expending too much computational resource.

Unfreezing the earlier layer is up to you on the later stage of training (Some people do some people don't) but if you feel you might need to do this to push for better performance, then just try it.


Smallest Possible Dataset for Text Classification using BERT What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis?

~100 examples

~1000 examples

...

~10000000 examples

What are your experiences?

Sorry, but there’s no rule and amount we are able to quantify.

I’ve used it (multilingual) for 700 texts with a 20 multilabel classification and I had worse results than with a custom deep net (but with pretrained word embeddings).

But you know, in fact these questions are hard to answer. Why? Because with very low quantities of data, you can’t accurately assess the performance of a model - it is statistically insignificant.


Creating a new Dataframe with specific row numbers from another I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers of another.
Here is what I have tried so far:
To add a single row to a new dataframe:
test.append(original.loc[300])
test

To add a list of rows to a new dataframe:
entries_to_move = [10, 20, 30]
for i in entries_to_move:
   test.append(original.loc[i])
test

Neither method works, so help would be appreciated. The output for either code is just a __.
Thank you!

So, your question is to instantiate a new data frame df2 from another data frame df1, by simply selecting rows.
You can do this by indexing. What is great by pandas DataFrames is that you can index a DataFrame using a list of indices.
df2 = df1.iloc[[list of indices],:]


Difference between bagging and boosting Can anyone explain me the basic difference between bagging and boosting and which technique can be used in which scenario?

Bagging: Also known as Bootstrap Aggregation is an ensemble method. First, we create random samples of the training data set (sub sets of training data set). Then, we build a classifier for each sample. Finally, results of these multiple classifiers are combined using average or majority voting. Bagging helps to reduce the variance error.

Boosting provides sequential learning of the predictors. The first predictor is learned on the whole data set, while the following are learnt on the training set based on the performance of the previous one. It starts by classifying original data set and giving equal weights to each observation. If classes are predicted incorrectly using the first learner, then it gives higher weight to the missed classified observation. Being an iterative process, it continues to add classifier learner until a limit is reached in the number of models or accuracy. Boosting has shown better predictive accuracy than bagging, but it also tends to over-fit the training data as well.

Algorithms work on these techniques

Bagging: Random Forest

Boosting : Ada Boost, Gradient Boosting, XGBoosting, etc


Cross-validated average: metrics mean or ensembling probabilities? Let's say I have 5 models cross-validated via leave-one-out strategy. I have the predictions and scores of each model.
Now, it's time to calculate the average for the set of 5 models - am I supposed to:

add up the 5 losses and divide them by 5?
Or average their probabilities for each prediction and use the average probability to calculate new metrics like an ensemble/ forest?

A standard way to provide the performance of each model would be:

providing, for each split, the value of the chosen metric (accuracy, roc_auc, etc) on the train and test sets (on your case, your one-out sample), something like this (in this case with 2 models):

as a final model performance (for each one of the 5 models), a mean metric value together with its standard deviation for the test sets is a way to inform about the model quality and its robustness, something like (preferably for the test set):

You have more detail on how to automatically get this done via scikit-learn, and in this answer and this one.
By the way, consider using another strategy as stratified k-fold, in case you have a lot of samples, as leave-one-out would be very costly.


Why would you use word embeddings to find similar words? One of the applications of word embeddings (such as GloVe) is finding words of similar meaning. I just had a look at some embeddings produced by glove on large datasets and I found that the nearest neighbors of a given word are often fairly irrelevant. Eg. ‘dad’ is the closest neighbor of ‘mom’, ‘dealership’ is the seventh closest neighbor of ‘car’.

In light of this, if you wanted to find words of similar semantics why would you prefer using embeddings instead of just downloading a database of synonyms from an online dictionary that is compiled by humans?

It depends on how similarity is defined. If similarity is defined as human-defined semantics, then a synset (i.e., synonym set) is most appropriate. If similarity is defined as frequent co-occurrence, then word embeddings are most appropriate. Even within semantic similarity, there are many approaches beyond synsets.

One advantage of word embeddings over synsets is the ability to automatically find similarity with multi-word term vocabulary. For example, the common word analogy - Man is to king as woman is to queen.


How to interprete percentile information from the describe function in Pandas? I am a bit stumped on how to interpret the percentile information you see when you call the describe function on dataframes in Pandas.
I believe I have a basic understanding of what percentile means. For example if in a test someones score 40% which ranks at the 75% percentile, this means that the score is higher than 75% of the total scores.
But I don't know how to translate this knowledge to interpret what I see from the describe function.
To illustrate, given the following:
test = pd.DataFrame([1,2,3,4,5,1,1,1,1,9])
test.describe()

This prints out something similar to this:
| count | 10.000000 |
|-------|-----------|
| mean  | 2.800000  |
| std   | 2.616189  |
| min   | 1.000000  |
| 25%   | 1.000000  |
| 50%   | 1.500000  |
| 75%   | 3.750000  |
| max   | 9.000000  |

Now I do not know how to interpret the values assigned to 25%, 50% and 75%. For example 5 out of the 10 values is set to 1, but the 50% has a value of 1.50000, clearly it is not saying 1.5 has a value of 50% because there is not even 1.5 in the data set.
Also why is 25% set to 1.000000 and 75% set to 3.750000?
I know I am interpreting this wrong hence this question! Would appreciate if someone can help understand this

Percentiles indicate the percentage of scores that fall below a particular value. They tell you where a score stands relative to other scores.
For example:
a person height 215 cm is at the 91st percentile, which indicates that his hight is higher than 91 percent of other scores.
Percentiles are a great tool to use when you need to know the position of a value/score respect to a population/data distribution you're considering. Where does a value fall within a distribution of values? While the concept behind percentiles is straight forward, there are different mathematical methods for calculating them.
In your example 50% correspond to the median of the ordered values distribution.
In this case the median is calculated between two values: 1 and 2 so the median is calculated (in this case 'cause the number of values is even so the median as to be calculated between the fifth and sixth ordered values ) as the mean between them 1.5.


How to deal with unbalanced class in biological datasets? When dealing with unbalanced class, which is better,

oversampling/undersampling of the classes or
randomly selecting equal number of positive samples and negative samples from the training dataset and combining as training samples, transforming the imbalanced classification problem and replacing by multiple balanced data classification problems?

Does one of them have an advantage over the other? If so, which one? I am asking for a generalized point of view. If you had an unbalanced dataset, which option would you choose, 1 or 2?

There can be two things for an imbalanced dataset:

. For eg, in a cancer detection dataset, the number of positive(cancer present) samples, will be quite less than the negative( cancer absent) samples. In this case, you should not apply sampling technqiues. You should rather focus on minimizing False Negatives , as cancer should not be missed while diagnosing a patient at the expense of a few false positives.

.Second case, for eg, The popular  Iris Datset where the classes should be balanced. But suppose you get an imbalanced version of the dataset, then you should apply sampling technqiues to the dataset. Because, the class proportion does not match the actual proportion , which was not the case in the Cancer Example.


How would you - on-the-fly - prevent a neural network from overfitting using a Keras callback? I have a neural network that starts to overfit in that the validation loss begins to increase while the training loss stays ~ flat with epochs.

Is there a generic algorithm - obvious or otherwise, well-known or not - to stop the training early if overfitting is somehow detected?

I note that catboost implements such an algorithm but I have found it nowhere else.

https://catboost.ai/docs/concepts/overfitting-detector.html

Is this all simply a matter of rolling my own callback function and stopping when the training and validation losses start to diverge..?

Preference for TF, Keras, python3, ...

Thanks as ever

Sounds like you're just looking for EarlyStopping, which will stop training when validation loss does not improve for N epochs. It's the same as Iter in catboost.


Custom thresholds on categorical classification When assessing a binary classification task, it is possible to search for particular threshold in order to have better score on some metrics (f1,recall,etc) through numerous methods. Unfortunately, it looks like the method cannot be applied on categorical classification (more than two classes) task. I've thought about training a Simple Classifier (SVC,Log,...,Tree) on top of a already trained model in order to find best thresholds to apply on outputs to maximize similarity of results.
My proposed workflow is to train a model on training set and evaluate it, then use outputs on training set of this model as entry for another model which may learn how to handle threshold to maximize similarity to ground truth on training set (all of those steps being validated using the same strategy).
I've never heard about it and I want to know whether it makes sense ?

What you are talking about sounds a lot like meta-learning (use outputs of several models and use another model to train on that predictions). You can read through more detailed example here (link to machinelearningmastery)
Should you do it for your binary classification task - no. (this in not a strict "no", but generally you should not do it given your task can really benefit from such approach, but this has little to do with classification threshold)
About threshold search, you can check this thread with quite good answers:
https://stats.stackexchange.com/a/386433/232020
From your question, it sounds like you are dealing with the binary dataset, not sure what you mean by  Unfortunately, the method seems not to be scaled on categorical classification.


What's the proper Word2vec model to get pre-trained word embedding for a classification task? I have to use a neural network to classify whether some reviews of hotels are deceptive or truthful. I also have to use pre-trained word embeddings to fed the neural networks. So I can use Word2vec to get the word vectors from a way larger dataset of hotels reviews. However, Word2vec gives the possibility to use continuous bag-of-words and continuous skip-gram models for this task. Which one would be generally better for this specific task?

I think this article gives general idea of pros / cons between CBOW and Skip Gram,

Link

  According to Mikolov:

  Skip-gram: works well with small amount of the training data,
  represents well even rare words or phrases.

  CBOW: several times faster
  to train than the skip-gram, slightly better accuracy for the frequent
  words.

So without looking at data, assuming that you have large amount of data, CBOW might be my bet. However, if you have labels for "deceptive" and "truthful", it will be best to train both models, and use whichever outperforms via cross-validation.


how do the number of classes in an object detection model affect accuracy? If I have, say, a Yolo or RetinaNet Object Detection Model... if I train it with 10 vs 50 classes, (assuming 3000 training data images per class), will the model with 10 classes perform similarly to the model with 50 classes? Is there a 'soft limit' to the number of classes a model can successfully hold in 'weight memory' ?.
I notice for most COCO examples, the class # is set at 80. Is there deteriorating performance when they push that number up into the 200s?
Is there any benchmarks done on this type of question? I would assume its a well discussed problem, i.e. splitting object detection of many classes across multiple trained models or packing them all into a single model?

The general consensus in machine learning problems is that it becomes tougher to get higher accuracy results when there is more data with more class splits. The simplest of examples would be cifar 10 and cifar 100. While they are practically the same models tend to vary very differently with respect to the efficiency.
The moment more classes are added, there is more variability under the final output. I am not aware of any soft limit though few papers might be there which focus on this. If you take a model A and train it on data with 10 classes and 50 classes, it is bound to perform better on the 10 classes provided there is an ideal set-up in both terms.
I have not come across any benchmark which says which is the ideal number of classes. Importing many models takes a lot of computational power therefore the question of many to one models varies mostly on the cost rather than any soft limit of the performance.


Does the layer contain two matrices, one for the actual weights and one for the biases?

There could be one bias value for each of the columns in your weight matrix, depending on how you built your model.


Using a GAN discriminator as a standalone classifier The goal of the discriminator in a GAN is to distinguish between real inputs and inputs synthesized by the generator.

Suppose I train a GAN until the generator is good enough to fool the discriminator much of the time. Could I then use the discriminator as a classifier that tests whether an input belongs to a single class?

For instance, if I train StyleGAN to be able to synthesize photorealistic cats, could I use the trained discriminator to detect whether an image is a cat or not?

My thinking is that perhaps the discriminator would be more accurate than other classifier models because it has effectively trained on many, many more inputs thanks to the generator.

On the other hand, perhaps the discriminator is somehow worse because it has been trained overwhelmingly on cat-like images (assuming the generator has gotten pretty good), and hasn't seen a wide variety of negative examples. It is concerned less with "is this a cat?" than "what are the tell-tale signs of this being synthetic?"

Yes, we can use the Discriminator of the GAN to classify images. But we should make sure that the images produced by the Generator are real looking.

If you have trained your GAN on a large number of images and it is performing pretty well on the dataset then I insist you to treat the Discriminator model as a pretrained model ( like we do in transfer learning ) and again train this model on images which were not used to train the GAN earlier. Thus the model is fine tuned on the dataset on the GAN wasn't trained before.
Another similar way could be to only use the weights of the CNN layers and load them in our new model.

  Suppose I train a GAN until the generator is good enough to fool the
  discriminator much of the time. Could I then use the discriminator as
  a classifier that tests whether an input belongs to a single class?

You should definitely try reusing a Discriminator from a GAN and share your results too :-).


Siamese Network - Sigmoid function to compute similarity score I am referring to siamese neural networks introduced in this paper by G. Koch et al.
The siamese net computes 2 embeddings, then calculates the absolute value of the L1 distance, which would be a value in [0, +inf). Then the sigmoid activation function is applied to this non-negative input, so the output afterwards would be in [0.5, 1), right?
So, if two images are from the same class, your desired L1 distance should be close to 0, thus the sigmoid output should be close to 0.5, but the label given to it is 1 (same class); if two images are from different classes, your expected L1 distance should be very large, thus the sigmoid output should be close to 1, but the label given to it is 0 (diff. class).
How does the use of a sigmoid function in order to compute the similarity score (0 dissimilar, 1 similar) in a siamese neural network make sense here?

Your observation is correct @Stefan J, but did you observe that there is a Dense layer too? The absolute L1 distances are multiplied with the weights of the Dense layer. These outputs are then fed to the sigmoid function. If we are looking at a Keras implementation from One Shot Learning with Siamese Networks using Keras,

you see the last Dense layer.
Also, from the research paper,

The L1 distances are multiplied with $\alpha$ which is a training parameter. This sum is then passed on to the sigmoid $\sigma$ function. We may interpret these values of $\alpha$ as the weights of the last Dense layer. These weights get smaller after training.

Another obvious reason of a sigmoid function is to get similarity scores in ( 0, 1 ). The binary cross-entropy loss function is used with it.


Can we predict when an event will occur in the future from time series data? I would like to predict a few possible times when a particular event may occur. For instance, I have the daily activity data of a person that consists of what the person doing and when over a period of time. Currently, my data are like the following format:

Date    Time    Activity

01-01-2017  08:23:30    Breakfast

01-01-2017  16:20:30    Reading

01-01-2017  19:00:00    Dinner

02-01-2017  08:00:10    Breakfast

02-01-2017  17:40:30    Reading

02-01-2017  19:30:00    Dinner

03-01-2017  08:15:30    Breakfast

03-01-2017  16:20:30    Reading

03-01-2017  20:30:00    Dinner

Let's just say, in most of the cases, the person takes his dinner around 7-7:30 pm and sometimes at 8-8:30 pm, read books around 4 pm mostly and sometimes at 5,6 pm too. From this type of data, I would like to predict when he will take breakfast, dinner or read tomorrow. For instance, dinner at 8pm (confidence/probability 65%), or 8:30pm (20%), 9 pm (10%), 9:30pm (5%).

The challenge I am facing is, I can't identify which technique to choose to achieve the goal. Can you please provide me a few hints?

I would suggest creating a custom ontology by capturing the ranges of each activity. For example, define in your training data additional parameter for breakfast and put the range of between the minimum and the maximum hour 08:00:10 08:23:30.

Feature engineering is a must if you plan to you RandomForest, LinearRegression, etc..

Additionally, the timestamp itself has very reach data information. You can take the day of the week, the day of the month, the day of the year, week of the year. Based on this you can see whether is the weekend, a weekday(mon-fri), and you can see how this will change over time(maybe during the summer the person is eating later/earlier). Also, you can see the hourly changes. I would also strongly suggest using plots, and this way you will see how the "Breakfast" label is chaining over time(during different periods of the year, different parts of the week/month). This will give you a better understanding of what is happening in your data.

In the end, I would suggest trying RandomForest(s)(maybe a bagging approach). They are really good for time-series data, especially when you can define nice features where the differences are clearly visible. Tunning the RandomForest is a must. In similar problems to this one, the min_sample_split and the max_depth parameter are usually really helpful.

Additionally, Ridge or Lasso regression is also a good way to try your newly constructed features.

At the end, as it was pointed out, LSTM is proven to capture time-dependency in the data. However, I would reconsider using it, without any practical experience.


Which dissimilarity/similarity measure use after a dimension reduction ( PCA / AutoEncoder / ... )? Each problem required its own similarity/dissimilarity measure. Imagine we are dealing with dataset composed with vector of real. I suppose that we mostly use the euclidean distance especially in low dimension. Unfortunately we often have to deal with huge dimensional dataset and we apply divers dimension reduction techniques to make the problem easier to solve.

I will be happy to know your opinion about different similarity/dissimilarity measure you use after made a dimensionality reduction and how they impact classification/regression/clustering metrics.

Is it better to use euclidean distance after a PCA or another distance measure ? Why ?

And if i prefer to use AutoEncoder to reduce my dataset dimensions, does it exists a special one that guaranty to offer better results with a specific similiarity/dissimilarity measure ?

Does it exist a dimensionality reduction technique that is optimal with a particular distance metrics an which is bounded in $O(n.log(n))$ in time ?

This question will invite a lot of opinions and it will be very specific to the problem that you are looking at. But there's no harm in laying out the popular options -

1. Start with the data

a. Images
 - If you are working with images and plan to use a classifier like ConvNet, PCA probably isnt a very good idea. You should just resize the image using popular libs like opencv

b. Unstructured Text
 - I have personally looked at a lot of free text problems, and my goto method is using Cosine Distance after doing a TruncatedSVD (dimensionality reduction) on the bag-of-words data (or any way of structuring for that matter)

c. Structured
 - Usually PCA is the go-to solution, but only if you are ok with losing the interpretability in your downstream predictive models. Scree Plots are a good way to evaluate how good a PCA exercise was / how many Principal Components you need.

2. Methods

a. Auto-encoders
 - These are neural-network based dimensionality reduction technique and I haven't looked at a single production instance where in a predictive modelling workflow Auto-encoders outperformed PCA. I guess we just have to wait for the right use-case

b. Distance metrics - Euclidean is no-doubt the most popular one, but Cosine catches up well to free text problems. You also have Hamming distance, Mahalanobis distance and countless others but its really upto you to try that. I haven't come across a case where these were performing better.

While we are talking about distance metrics, also look at DBScan for clustering.

Also, Silhouette Distance is another effective way of measuring how good your clusters did.


Where is the error in the following code? I am trying to analyze the movie database using python, downloaded from imdb. While trying to generate some plots, I am running into errors which confuses me.
I am trying to generate a matrix of small figures which can show me any hidden pattern etc. Here is the code:

fig, axes = plt.subplots(nrows=4, ncols=6, figsize=(12, 8),
                         tight_layout=True)

bins = np.arange(1950,2012,3)
for ax, genre in zip(axes.ravel(), movieGenre):
    ax.hist(movieDF[movieDF['%s'%genre]==1].year, bins=bins, histtype='stepfilled', normed=True, color='r', alpha=.3, ec='None')
    ax.hist(movieDF.year, bins=bins, histtype='stepfilled', ec='None', normed=True, zorder=0, color='grey')
    ax.annotate(genre, xy=(1955, 3e-2), fontsize=14)
    ax.xaxis.set_ticks(np.arange(1950, 2013, 30))
    ax.set_yticks([])
    ax.set_xlabel('Year')

The first hist isn't working, but the second one is working when I am commenting out the first one. Here is the traceback:

 KeyError                                  Traceback (most recent call last)
<ipython-input-158-c2e7c2737372> in <module>()
      4 bins = np.arange(1950,2012,3)
      5 for ax, genre in zip(axes.ravel(), movieGenre):
----> 6     ax.hist(movieDF[movieDF['%s'%genre]==1].year, bins=bins, histtype='stepfilled', normed=True, color='r', alpha=.3, ec='None')
      7     ax.hist(movieDF.year, bins=bins, histtype='stepfilled', ec='None', normed=True, zorder=0, color='grey')
      8     ax.annotate(genre, xy=(1955, 3e-2), fontsize=14)

/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/matplotlib/axes.pyc in hist(self, x, bins, range, normed, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, **kwargs)
   8247         # Massage 'x' for processing.
   8248         # NOTE: Be sure any changes here is also done below to 'weights'
-> 8249         if isinstance(x, np.ndarray) or not iterable(x[0]):
   8250             # TODO: support masked arrays;
   8251             x = np.asarray(x)

/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/series.pyc in __getitem__(self, key)
    477     def __getitem__(self, key):
    478         try:
--> 479             result = self.index.get_value(self, key)
    480
    481             if not np.isscalar(result):

/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/index.pyc in get_value(self, series, key)
   1169
   1170         try:
-> 1171             return self._engine.get_value(s, k)
   1172         except KeyError as e1:
   1173             if len(self) > 0 and self.inferred_type == 'integer':

KeyError: 0

Here is the first few columns of the data:

     imdbID     title                      rating    vote      runtime  year    genre
0   tt0111161   The Shawshank Redemption    9.3      1,439,277  142    1994 [Crime, Drama]
1   tt0468569   The Dark Knight             9.0      1,410,124  152    2008 [Action, Crime, Drama]
2   tt1375666   Inception                   8.8      1,209,159  148    2010 [Action, Mystery, Sci-Fi, Thriller]
3   tt0137523   Fight Club                  8.9      1,123,462  139    1999 [Drama]
4   tt0110912   Pulp Fiction                8.9      1,117,193  154    1994 [Crime, Drama]

movieGenre is basically collecting all the different genres from 'genre' column with duplicates removed: movieGenre = set(movieDF.genre.sum()) . I then added a single column to movieDF data frame for each genre such that if a particular movie belong to that genre, then that cell is True otherwise it is False.  So for example, for the movie Inception, the Action column is marked True but Crime column is marked False and so forth.

Thanks.

First of all, since you only seem to want to know about the Python error, your question is probably better suited for Stack Overflow.

Secondly, this is not fun to answer with the information you've provided.  This is certainly not a minimal working example (MWE) - I can't copy your code as is, and run it myself.  I'm left to guess many things:

Are you importing matplotlib?
Did you rename matplotlib.pyplot as plt?
Are you importing numpy and did you rename it np?
Does your traceback literally end in KeyError is 0 or is it KeyError: 0 or KeyError: False?  Can you show the full traceback?  The traceback for a MWE is typically not long.

In the positional argument (movieDF[movieDF['%s'%genre]==1].year) on your problematic line, you ask for a key (movieDF['%s'%genre]==1) that looks like it should evaluate to a Boolean.  Is this never 0?  I'm forced to grasp at straws because you haven't provided a sample dataset; I shouldn't have to download from IMDb myself.

Edit:

Just to troubleshoot, can you place x = movieDF[movieDF['%s'%genre]==1].year on a line by itself and tell me what x[0] is?  What type would that x be?  Is it a dictionary, where the .hist() method is expecting a list or other array?  This documentation makes it sound like the method will not accept a dictionary...what type is movieDF.year?


Why there is no exact picture of softmax activation function? I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a sigmoid function?
I just found the following funny pictures from dataaspirant which is not available anymore:

Can anyone illustrate a Softmax function graph? and explain why one is faster than another one due to the above picture?

Softmax is a multivariable function, generally. You wouldn't take a softmax of a single variable just like you wouldn't take a maximum of a single variable. It's difficult to plot functions of more than 2 variables because our eyes see in 3 dimensions.

A sigmoid is a 1 dimensional function. In your picture, they are applying a sigmoid to every variable separately.


Understanding REINFORCE loss The loss used in REINFORCE algorithm is confusing me.

From Pytorch documentation :

loss = -m.log_prob(action) * reward

We want to minimize this loss.

If a take the following example :

Action #1 give a low reward (-1 for the example)
Action #2 give a high reward (+1 for the example)

Let's compare the loss of each action considering both have same probability for simplicity :

p(a1) = p(a2)
=> m.log_prob(a1) = m.log_prob(a2)
And loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
And loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
Then loss(a1) < loss(a2) since m.log_prob(X) < 0

Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss.

So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !

I think the intuition here is that you want to push the negative loglikelihood (NLL) * the reward to be as negative as possible. Since our reward is often not differentiable because it is obtained through sampling, we can only change the NLL. For actions with high reward, we have greater "pressure" / gradient to push the NLL down, than if the reward was low, as the reward acts as a multiplier on the gradient on the NLL. So you'd imagine at convergence/equilibrium, the high reward actions would have greater likelihood than the low reward ones. Let me know if that makes sense!


How to feed a table per timestamp to LSTM neural network? I have a time-series dataframe like this

         feat1  feat2  target
date id
0    1      12     16     192
     2      15      6      90
     3       2      9      18
1    1       0      3       0
     2       0      9       0
     3      56      9     504
2    1       5      9      45
     2       6      9      54
     3       5      8      40

and my problem is for regression.

What I know about the LSTM sequences, is that normally, the row_id is the date, so you build sequences of n rows.

In my data, as you can see, it's different. In each row that refers to the date, I have 3 more rows representing a product.

What I have thinked for make the sequence to my problem,  my sequence will be of 2 days:

sequence = [day0,day1],[day1,day2]

For every date, I have

date0 = [id1,id2,id3].

For every  id, I have:

id = [feat1,feat2].

the sequence[0], and it would be something like

[
    [
        [12,16],
        [15,6],
        [2,9]
    ],[
        [0,3],
        [0.9],
        [56,9]
    ]
]

Is this valid?

Will the LSTM layer understand this? Or do I have to do some kind of extra transformation?

Your data format is

                         feature1   feature2   target
             product1    1          12         2
 timestamp   product2    2          6          3
             product3    4          3          4

There are two designs for two assumptions:

Products are not related to each other. Therefore, each product could be modeled separately. That is, each timestamp is X(t) = [feature1, feature2] or, including the target, X(t)|y(t) = [feature1, feature2, target]. And we build a model for each product separately. In summary, LSTM receives two  1 x 3 sequences for t-1 and t, and outputs a 1 x 1 target for t + 1. In notation:
$$(\overbrace{X_{t-1}|y_{t-1}}^{1 \times 3}, \overbrace{X_{t}|y_{t}}^{1 \times 3}) \rightarrow \overbrace{y_{t+1}}^{1 \times 1}$$
Products are related to each other, meaning product1 can help product2 to predict its target. For this, we just need to flatten the 3 x 2 matrix to a 1 x 6 vector, where the order of values does not matter. That is,

X(t) = [product1_feature1, product1_feature2, ..., product3_feature2]

or

X(t) = [product1_feature1, product2_feature1, ..., product3_feature2]

We can also add the targets, for example

X(t)|y(t) = [product1_feature1, product2_feature1, ..., product3_feature2, target1, ..., target3]

This way, dimension of each timestamp would be 9 (6 + 3), and a sequence of two timestamps would be

[
  [product1_feature1, product2_feature1, ..., product3_feature2, target1, ..., target3], # t-1
  [product1_feature1, product2_feature1, ..., product3_feature2, target1, ..., target3]  # t
]

corresponding to a three dimensional target [target1, ..., target3] at t + 1.

In summary, LSTM receives two 1 x 9 sequences for t-1 and t, and outputs a 1 x 3 target for t + 1. In notation:
$$(\overbrace{X_{t-1}|y_{t-1}}^{1 \times 9}, \overbrace{X_{t}|y_{t}}^{1 \times 9}) \rightarrow \overbrace{y_{t+1}}^{1 \times 3}$$


Faster 3D Matrix Operation - Python I am working with 3D matrix in Python, for example, given matrix like this with size of 2x3x4:

[[[1 2 1 4]
  [3 2 1 1]
  [4 3 1 4]]

 [[2 1 3 3]
  [1 4 2 1]
  [3 2 3 3]]]

I have task to find the value of entropy in each row in each dimension matrix. For example, in row 1 of dimension 1 of the matrix above [1,2,1,4], the normalized value (as such the total sum is 1) is [0.125, 0.25, 0.125, 0.5] and the value of entropy is calculated by the formula -sum(i*log(i)) where i is the normalized value. The resulting matrix is a 2x3 matrix where in each dimension there are 3 values of entropy (because there are 3 rows).

Here is the working example of my code using random matrix each time:

from scipy.stats import entropy
import numpy as np

matrix = np.random.randint(low=1,high=5,size=(2,3,4)) #how if size is (200,50,1000)
entropy_matrix=np.zeros((matrix.shape[0],matrix.shape[1]))
for i in range(matrix.shape[0]):
    normalized = np.array([float(k)/np.sum(j) for j in matrix[i] for k in j]).reshape(matrix.shape[1],matrix.shape[2])
    entropy_matrix[i] = np.array([entropy(m) for m in normalized])

My question is how do I scale-up this program to work with very large 3D matrix (for example with size of 200x50x1000) ?

I am using Python in Windows 10 (with Anaconda distribution).
Using 3D matrix size of 200x50x1000, I got running time of 290 s on my computer.

It is faster if you use the built-in functions of numpy (instead of reimplementing them yourself):

import numpy as np
from scipy.stats import entropy

np.apply_along_axis(func1d=entropy, axis=2, arr=matrix)


Reason for square images in deep learning Most of the advanced deep learning models like VGG, ResNet, etc. require square images as input, usually with a pixel size of $224x224$.

Is there a reason why the input has to be of equal shape, or can I build a convnet model with say $100x200$ as well (if I want to do facIAL recognition for example and I have portrait images)?

Is there increased benefit with a larger pixel size, say $512x512$?

There is no requirement for specific pixel dimensions for convolutional neural networks to function normally. It is likely the values have been chosen for pragmatic reasons - such as a compromise between using image details vs number of parameters and training set size required.

In addition, if source data has a range of different aspect ratios, some portrait, some landscape, with the target object usually in the centre, then taking a square crop from the middle could be a reasonable compromise.

When you increase the input image size, you will also increase the amount of noise and variance that the network will need to deal with in order to process that input. That could mean more layers - both convolutional and pooling. It could also mean that you need more training examples, and of course each training example will be larger. Together, these increase the computation resources you need to complete training. However, if you can overcome this requirement, it is possible that you will end up with a more accurate model, for any task where the extra pixels could make a difference.

One possible rule of thumb for whether you would want higher resolution is if, for goal of your network, a human expert could make use of the extra resolution and perform better at the task. This might be the case in regression systems, where the network is deriving some numerical quantities from the image - e.g. for face recognition extracting biometrics such as distance between facial features. It might also be desirable for image-processing tasks such as automated masking - state of the art results for these tasks may still be lower resolution than the commercial images where we would like to apply them in practice.


From http://www.deeplearningbook.org/contents/convnets.html,

  The only reason to ﬂip the kernel is to obtain the commutative property. While the commutative property is useful for writing proofs, it is not usually an important property of a neural network implementation. Instead, many neural network libraries implement a related function called the cross-correlation, which is the same as convolution but without ﬂipping the kernel.

The discrete cross correlation function with 1-based indexing for $$I = U*V\ and\ K=M*N$$ is given by  $$C(x,y)=\sum_{m=1}^M\sum_{n=1}^NI(x+m-1,y+n-1)K(m,n)$$

where $$x\in(U-M+1)\ and\ y\in(V-N+1)$$

When applying 2-D convolutions, many neural network implementations will reduce the size of the output. To retain the original size in output and to retain information at the borders, many practical use cases add a padding of zeros to the input image.

Coming back to your specific question, in the case where indices go out of bounds, either on the positive side or on the negative side, (which happens when you try to retain the size of the input in your output), values corresponding to them are usually taken as zeros, which is akin to zero-padding in implementation.

Discrete convolutions are actually given by $$(f * g)[n] = \sum_{m=-\infty}^\infty f[n-m] g[m]$$ In practice the limits are made finite because of the same assumption of zero amplitude signal for out of bound indices.


How can I use variable length inputs to train a regression model? I'm working predicting a value $y \in \mathbb{R}$ from the value of $x_{n+1}$, where $n$ is the number of samples ($x_{i \in [1,n]}$) used for training.

Each training sample $x_{i}$ is a time series of variable lengths. How can I engineer features to predict $y$ while making the most of the samples available?

Nota bene:

I'm working with Python & Scikit-learn
the length of the time series may be considered as an explanatory variable

Thanks for your help :)

A quick approach can be using a library like tsfresh, which extracts features from your time series, e.g. max value, number of peaks, median value, etc.

Normally, a thoughtful solution to this kind of problem involves domain knowledge, that is, insights from an expert that can tell you which aspects of your time series are important, e.g. frequent variations, very high peaks, plateaus, local patterns, etc. This knowledge is then used to derive ad hoc feature extractors.


XGBOOST CV() producing error I am getting the following error while using xgboost.cv() (scikit-learn interface). I am working on a regression problem. Below is the code and trace. No idea why it is giving this error.

from hyperopt import hp
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

space = {
             'n_estimators' : hp.quniform('n_estimators', 100, 1201, 100),
             'eta' : hp.loguniform('learning_rate', np.log(0.01), np.log(0.2)),
             'max_depth' : hp.choice('max_depth', np.arange(1, 14, dtype=int)),
             'min_child_weight' : hp.quniform('min_child_weight', 1, 6, 1),
             'subsample' : hp.quniform('subsample', 0.5, 1, 0.05),
             'gamma' : hp.quniform('gamma', 0.5, 1, 0.05),
             'colsample_bytree' : hp.quniform('colsample_bytree', 0.5, 1, 0.05),
             'eval_metric': ['rmse'],
             'objective': ['reg:linear'],
             'n_jobs' : [-1]
             }
dtrain = xgb.DMatrix(X_train, label=y_train)
cv_results = xgb.cv(space,dtrain,num_boost_round = 1000, nfold= 5, obj = ['reg:squarederror'],
                        stratified = False, early_stopping_rounds = 100, metrics="rmse", seed = 44)

cv_results

The following is the part of the trace:

---------------------------------------------------------------------------
XGBoostError                              Traceback (most recent call last)
<ipython-input-19-b45243537e3b> in <module>
      1 dtrain = xgb.DMatrix(X_train, label=y_train)
      2 cv_results = xgb.cv(space,dtrain,num_boost_round = 1000, nfold= 5, obj = ['reg:squarederror'],
----> 3                         stratified = False, early_stopping_rounds = 100, metrics="rmse", seed = 44)
      4
      5 cv_results

...
...
...
XGBoostError:src/objective/objective.cc:23: Unknown objective function ['reg:linear']\n\nStack trace returned 7 entries:\n[bt]

Any suggestion as to how to avoid this would be helpful.
Thanks

You've passed an objective function reg:linear in space; there is no such objective function.  The objective reg:squarederror that you pass to cv should be fine.


Why do we need to concatenate in a U-Net? You might be familiar with the U-Net, a machine learning network deceived for image segmentation.
It's basically an encoder/decoder network with some direct links between encoder and decoder segments:

I would like to better understand the reason behind those gray lines!
What is the idea behind copying one part (cropped) of the outputs of the convolutional segments  (before max-pooling) and concatenating it to the inputs to the individual up-convolutional layer?

These types of connections between layers are called skip connections, searching for this will give you a way to find more in-depth information. Broadly speaking, there are two advantages to using skip connections in this case:

They allow gradients to more freely flow through the model, mitigating the issue of vanishing gradients
They allow features from the encoder side of the network to the decoder side of the network, adding extra information that might be lost because of the downsampling on the encoder side of the network.


Unsupervised Algorithm for hybrid data I have a hybrid data that contains 15 categorical data and 4 continuous data.
I need to implement a prediction on the data. So as I don't have any labeled data, I need to implement the unsupervised algorithm.

Now Using PCA my dimensions reduced to 5 but still it has 3 categorical and 2 continuous data. Now to create a prediction algorithm, I can use KMeans for continuous data or KModes for categorical data, But I couldn't find any algorithm that works on both types of data.

Can anyone know any algorithm that works on both the data types for unsupervised prediction?

I think you are looking for K-prototype algorithm


Can sampling like SMOTE/UP/DOWN applied on Validation set? I am trying to predict classification problem. For that I have used Ranger, Xgboost and naive bayes. My Response class is imbalance . 92:8 ratio. My positive Response is only 8% of whole data.

Because of class imbalance I am getting more FP / FN.

I tried different sampling on training set and predicted with original set gives me more FP.

Can I do something like ?:
- split data into train and test
- apply sampling method individually on train and test
- and use sampled test to predict.

Thanks.

Oversampling on your test set will only artifically improve your performance. What you may want to do instead is changing your objective function to give more importance to you imbalanced class. There are already a lot of question about class imbalance on this website, such as : Classification problem: custom minimization measure


Would you recommend feature normalization when using boosting trees? For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using regularization. However, in my experience, boosting tree regression works less well when I use normalized features, for some strange reason. How is your experience using feature normalization with boosted trees does it in general improve our models?

Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching.

Wikipedia for decision tree:

Requires little data preparation. Other techniques often require data normalization....

However, it's always a good idea to normalize your features because:

It's easier to visualize and interpret your model
It's easier to compare another model (e.g. SVM) with the same data set


General methods outlier detection What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of bedrooms, living rooms and bathrooms. I want to identify outliers in this data, where some of the variables are discrete and some of them are continuous. Any ideas on how to do this?

You can use proximity-based outlier detection methods. These methods fall under 2 different categories, 1) Cluster-based, and 2) Distance-based.

Cluster-based methods will define a point as an outlier if its distance to cluster centroids is large. If you have both categorical and continuous variables, then it is important to ensure that the clustering method you use is appropriate for mixed data-types. Clustering methods appropriate for both types include k-Prototypes or Squeezer.

To use a distance-based method, you could use the k-nearest neighbor algorithm, and define an outlier as a data point whose distance from its k neighbors is much larger than normal data points.


What is affine transformation in regards to Neural Networks? I have been reading a paper recently on Highway Neural Networks and found the following:

$y=H(x,W_H)$
$H$ is usually an affine transform followed by a non-linear activation function, but in general it may take other forms.

After googling about affine transform I can't say I understand fully what it means. Can somebody please elaborate?

It's a linear transformation, for example lines that were parallel before the transformation are still parallel. Scaling, rotation, reflection etcetera. With regards to neural networks it's usually just the input matrix multiplied by the weight matrix.


Why a GAN trained on same data and same parameters may produce different results? I am trying to train a Generative Adversarial Network and ran the training a few times with same dataset and same parameters but it seems tp produce different results. Why this may happen?

The loss function error surfaces of both discriminator and generator are complex. There are many different optima on both surfaces that can yield usable solutions. There is a element of randomness where on the loss function error surfaces a training run with stop.


determining batch size and epoch size of neural network what should be my batch size and epoch size for 2 layer neural network if I am working with 1700 data?
I also want to know about neurons.how to determine the number??

Depends on many things. Batch size is how many instances will be accepted as input by your model, in order to make a complete forward pass and propagate the appropriate gradients backwards. Having a batch size too small could lead to gradients that derived from a batch which is not representative of the data distribution, thus these gradients could be harmful. Having a large batch size in a huge dataset could lead to memory issues. Having said that, use 128.


Data Cleansing - Handling CSV files I would like to hear some views on a problem I have with my dataset (which I presume to be a common one).

Let's say I have the following dataset

SKUID   PRODUCT QTY        MFGDT      ......     EXPDT     SUPPLIERID SUPPLIERPH CUSTOMFIELD
FD001     MILK    3   12/01/18 14:12:02 ...    18/01/18    SV01    04053XXXX
FD002    CREAM    3   12/01/18 14:12:02 ...    18/01/18    SV01    04053XXXX
FD003   CHEESE    5   12/01/18 14:12:02 ...             18/01/18    SV01    04053XXXX
FD004   BUTTER    2   12/01/18 14:12:02 ...    18/01/18    SV01    04053XXXX
FD005 ICECREAM    1   12/01/18 14:12:02 ...      SV01   04053XXXX

The dataframe is of shape (123078, 199) and there are few records where the field values are jumbled.

When reading this csv using pandas, I used the error_bad_lines=False attribute to skip the lines where there is a mis-match in the field.

However, I was wondering if there is a way to FIX the data by some means (say, pattern matching with the previous items in the column; based on the dtype etc.)

How do we generally handle a scenario where every record is crucial (or say inter-dependent) and there is mismatch in the field?

If you read the entire file without skipping jumbled lines, does it stil work? What values appear in those jumbled cells?

You could go down the road of imputation, i.e. filling the missing gaps, based on something you can deduce from the data that is there.

Example are:

fill-forward: fill will preceding non-missing value
fill-backwards: fill with following non-missing value
average-filling: if numerical, fill gaps with the mean/median etc. of that column. You could also use something like a moving average.
model-based: fill missing values using e.g. a regression model on available values (this needs a target variable, or at least timestamps)

One last snity-check: I assume it isn't possible for you to try creating the CSV file again and addressing the problem at its source?


how to find classification accuracy in autoencoders? how can we find the accuracy autoencoders for classification of images? because we will get the reconstruction of the image and when we will plug in the test data it will spit out the image but how would we able to calculate the accuracy? or we should take difference between the prediction and input of the image .

Are you using the autoencoder for classification or reconstruction?
if you are pre-training the autoencoder for classification then you use the usual logloss to determine the accuracy of your classifier. If on the otherhand, if you are using the autoencoder for reconstruction, then there is no classification, and you can use something like KL divergence (described in detail here) to measure and track your reconstruction and compression performance.


How to solve ValueError: Negative dimension size caused by subtracting 3 from 1 for '{{node model/Conv1/Conv2D}} = Conv2D... in mobilenet_v2 I'm trying to apply a retrained model of mobilenet_v2 presented in https://github.com/balajisrinivas/Face-Mask-Detection
The main part of my adapted code is the following:
faces = []
locs = []
preds = []

for coord in coords:

    x = person[0]
    y = person[1]
    w = coord [2]
    h = person[3]

    # extract the face ROI, convert it from BGR to RGB channel
    # ordering, resize it to 224x224, and preprocess it
    face = frame[y:y+h, x:x+h]
    face = cv2.cvtColor(face, cv2.COLOR_BGR2RGB)

    if (face.shape) > (224, 224):
        face = cv.resize(face, (224, 224))

    face = img_to_array(face)
    face = preprocess_input(face) #from tensorflow.keras.applications.mobilenet_v2

    # add the face and bounding boxes to their respective lists
    faces.append(face)
    locs.append((x, y, w, h))

    # only make a predictions if at least one face was detected
    if len(faces) > 0:
        # for faster inference we'll make batch predictions on *all*
        # faces at the same time rather than one-by-one predictions
        faces = np.array(faces, dtype="float32")
        preds = maskNet.predict(faces, batch_size=32) #maskNet = load_model("mask_detector.model")

The error is raised when I try to apply maskNet.predict, which is the is the mobilenet_v2 model saved in https://github.com/balajisrinivas/Face-Mask-Detection/blob/master/mask_detector.model.
The error message is the following: ValueError: Negative dimension size caused by subtracting 3 from 1 for '{{node model/Conv1/Conv2D}} = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], explicit_paddings=[], padding="VALID", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true](model/Conv1_pad/Pad, model/Conv1/Conv2D/ReadVariableOp)' with input shapes: [?,1,1,3], [3,3,3,32].

I found out my mistake!
I shouldn't use a condition before resizing the frame:
face = cv.resize(face, (224, 224)) #without if (face.shape) > (224, 224):

I've used the condition because I had to do that for SSD Resnet in the past, but I just realized it's not necessary for mobilenet_v2


Is PCA (by eigendecomposition) or SVD better in decorrelating the predictors of a machine learning model? Is there any reason to think that SVD is better than PCA (by eigendecomposition) in decorrelating the predictors of a machine learning model?

To the best of my knowledge, the answer to your question is no. Regarding finding the correlations of different variables, they work the same. They both capture linear associations and do not capture nonlinear ones. The difference between them is mostly about numerical computation which makes SVD more handy than traditional PCA. I recommend having a look at this answer and this explanation.

  As a final remark, let’s discuss the numerical advantages of using SVD. A basic approach to actually calculating PCA on a computer would be to perform the eigenvalue decomposition of $X^TX$ directly. It turns out that doing so would introduce some potentially serious numerical issues that could be avoided by using SVD.


Where can I find an algorithm for human activity classification using thigh and shank sensors? I am working on a project where I initially need to classify the type of activity a subject is performing based on raw accelerometer and gyroscope data from the thigh and shank of both legs. The type of activities would be day-to-day tasks such as standing, sitting, running, sit-to-stand, stair climbing, if playing sports, cutting, etc.

I have tried reading papers and looking on git to find an algorithm that will do this for me. I know that there are a lot of algorithms available, but none seem to utilize sensors on the leg. Most are sensors on the chest or through a person's phone.

Where would I be able to find an algorithm that will take the data I have an classify the activities being performed? Can I do this with algorithms that aren't initially made for the sensor placement I am using?

Any help would be greatly appreciated.

What you want to do can be called activity recognition or time-series classification. Almost all classification algorithms can be made to do this. A popular choice is to use a type of RNN (recurrent neural network) called a LSTM (long short-term memory). It is made to work with sequences and handles time series quite well.

Here are some relevant readings that might get you started:

Activity recognition tutorial using RNN models

Similar tutorial using various models


Career switch to Big Data Analytics I am a 35 year old IT professional who is purely technical. I am good at programming, learning new technologies, understanding them and implementing. I did not like mathematics at school, so I didn't score well in mathematics. I am very much interested in pursuing a career in Big Data analytics. I am more interested in Analytics rather than Big Data technologies (Hadoop etc.), though I do not dislike it.  However, when I look around in the internet, I see that, people who are good in analytics (Data Scientists) are mainly Mathematics graduates  who have done their PHds and sound like intelligent creatures, who are far far ahead of  me. I get scared sometimes to think whether my decision is correct, because learning advance statistics on your own is very tough and requires a of hard work and time investment.

I would like to know whether my decision is correct, or should I leave this piece of work to only intellectuals who have spend their life in studying in prestigious colleges and earned their degrees and PHDs.

Due to high demand, it is possible to start a career in data science without a formal degree. My experience is that having a degree is often a 'requirement' in job descriptions, but if the employer is desperate enough, then that won't matter. In general, it's harder to get into large corporations with formalized job application processes than smaller companies without them. "Knowing people" can get you a long way, in either case.

Regardless of your education, no matter how high demand is, you must have the skills to do the job.

You are correct in noting that advanced statistics and other mathematics are very hard to learn independently. It is a matter of how badly you want to make the career change. While some people do have 'natural talent' in mathematics, everybody does have to do the work to learn. Some may learn more quickly, but everybody has to take the time to learn.

What it comes down to is your ability to show potential employers that you have a genuine interest in the field, and that you will be able to learn quickly on the job. The more knowledge you have, the more projects you can share in a portfolio, and the more work experience under your belt, the higher level jobs that will be available to you. You may have to start  in an entry level position first.

I could suggest ways to study mathematics independently, but that isn't part of your question. For now, just know that it's hard, but possible if you are determined to make a career change. Strike while the iron is hot (while demand is high).


Where can I get labels for small ImageNet? I have recently downloaded the small image net dataset:

http://image-net.org/small/download.php

The archives contain a lot of images, but no other files. How do I know which label the images have?

I am affraid this dataset does not include labels,

You can use datasets that are described here:

https://patrykchrabaszcz.github.io/Imagenet32/

Those files will have the same format as CIFAR dataset

And they will contain labels according to the map file that is available in original Imagenet devkit.


How to arrange the dataset/images for CNN+LSTM I am working on an image classification problem using Transfer Learning with Resnet50 as base model (in Keras) (For example Class A and Class B).

There is a time factor involved in this classification. For example, I need sufficient evidence to make transition from one class to another.
So one of the thought came to my mind to make smooth transition is adding LSTM layer to the CNN layer (CNN+LSTM).

Need your help in understanding below queries.

Is there any way that I can add LSTM layer to the transfer learning process (assuming the CNN layer weights are not trainable)
How do I need to prepare the dataset (image frames). For example I have 10 videos each for class A and Class B. Do I need to keep the images in sequential order as it is in video. (As of now for the normal image classification, I have shuffled the image frames)
Any thought on building my own CNN + LSTM model.

A reference link would be great for 2 and 3.

Whether the image frames need to be updated as it is in the video ?

Any updates on this please.

Thank you,
Deep Guy

I'll expand my answer if there is any interest. I propose building a simple test model using Vgg and two custom fully connected layers that end with a single sigmoid per the document below. If you prefer early fusion I recommend using stacked black and white images.
https://www.cs.cmu.edu/~rahuls/pub/cvpr2014-deepvideo-rahuls.pdf

if you then want to go the route of LSTM, then this graphic from Karpathy will help. Use the many to one architecture. The inputs to the RNNS or LSTMS vary by their implementation. You would most likely want to pass the final convolutional layer though a fully connect layer and then into the RNN/LSTM. https://karpathy.github.io/2015/05/21/rnn-effectiveness/


pandas datetime to Unix I need to convert a datetime.date(2007, 4, 30) to a Unix timestamp in pandas.

thank you.

my datable:

0     2007-04-30
1     2007-05-31
2     2007-06-30
3     2007-07-31
4     2007-08-31
Name: datetime, dtype: object

finally worked something out though I wonder if it is the best solution?

dt = pd.DatetimeIndex ( dt ).astype ( np.int64 )/1000000

cheers.


K-Means visualisation problem 8 numerical feutres Hello im looking for exemple with python for K-Means clustering when i have data set with more than 6 feutres.
thanks

It's not clear enough what you try to do. If I understand correctly, you want to train a K-Means clustering and visualize the results. However, you have 8 dimensions in your dataset and obviously, you cannot plot such a space.

What you can do is to reduce the dimensionality in 2 dimensions and then create that plot.

For example,

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans

# read my data with pandas into a dataframe
data = pd.read_csv("data.csv")

# run a KMeans model with 3 clusters. Change that number to what you want
clustering_kmeans = KMeans(n_clusters=3, precompute_distances="auto", n_jobs=-1)
clusters = clustering_kmeans.fit_predict(data)

# run PCA to reduce the dimensionality to 2 dimensions
reduced_data = PCA(n_components=2).fit_transform(data)

# create a new dataframe that contains the 2 dimensions and the cluster label
results = pd.DataFrame(reduced_data,columns=['pca1','pca2'])
results['label'] = clusters

# plot the results with a scatterplot
sns.scatterplot(x="pca1", y="pca2", hue=label, data=reduced_data)
plt.show()


Computing number of batches in one epoch I have been reading through Stanford's code examples for their Deep Learning course, and I see that they have computed num_steps = (params.train_size + params.batch_size - 1) // params.batch_size [github link].

Why isn't it num_steps = params.train_size // params.batch_size instead?

The double-slash in python stands for “floor” division (rounds down to nearest whole number), so if the result is not an integer, it will always miss the last batch, which is smaller than the batch size.

For example:

Given a dataset of 10,000 samples and batch size of 15:

# Original calculation:
# (10000 + 15 - 1) // 15 = floor(667.6) = 667

# Your calculation:
# 10000 // 15 = floor(666.667) = 666

Your formula calculates number of full batches, theirs total number of batches.


Keras decision threshold for Multiple Label prediction I'm training a Neural Network to predict multiple labels for a given input. My input is a 200 sized vector of integers and the output should be a boolean vector of size 28. My y has a 1 on the corresponding classes the example corresponds to, i.e. the y should look like:
[0, 0, 1, 0, ... 1, 0, 0].

Now, I've used a sigmoid function on my final layer, and I'm training with binary crossentropy since I want my model to treat each class as independent from each other.

When I fit my model, I get a pretty decent categorical accuracy, i.e. ~75%, but I'm wondering which is the threshold that Keras uses to say a class is or isn't present on the example, I mean, which is the threshold where it decides:

prediction[prediction >= threshold] = 1
prediction[prediction < threshold] = 0

You can look at your multi class classification as 28 separate binary classifiers (1 for each output feature).

The output of a binary classification is the probability of a sample belonging to a class, so the threshold per each feature is 0.5.


Classification model using RNN(action detection) 1) Could it be useful to use RNN for classification problem?(e.g.
to distinguish which action is taken: car is going, walking,
digging, nothing).
If 1 question is positive, how should RNN structure look like?

I have dataset of 4 actions, many examples for each action, each example includes 124 samples.
So my X_train, X_test are arrays of float(400000, 124, 1); y_train, y_test are arrays of int(0 or 1 or 2 or 3 depends on action).

My data preprocessing:

X_train, X_test, y_train, y_test = np.array(X_train), np.array(X_test), np.array(y_train), np.array(y_test)
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))

My structure:

regressor = Sequential()
regressor.add(LSTM(units=55, return_sequences=True, input_shape=(X_train.shape[1], 1)))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50))
regressor.add(Dropout(0.2))
regressor.add(Dense(4,activation='softmax'))

regressor.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
regressor.fit(X_train, y_train, epochs=5, batch_size=32)
y_pred = regressor.predict(X_test)

@CapJS,

Instead of going ahead with an RNN, which helps you model the dependencies and relationship between your content, I would suggest you to take a look at 1D convolution networks to achieve the classification of activity.

There is a nice post talking about something similar : https://machinelearningmastery.com/cnn-models-for-human-activity-recognition-time-series-classification/

With 1D convolution, you are by design gaining better processing performance and given that you have just 124 samples per activity, you do not need a deep layered network as the data size is still quite small.

Given the data size, I would also suggest you to go ahead with something as simple as logistic regression, random forest approach. Hope this helps.


huge doubt on anomaly detection from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to find anomalies in our data

@Ram , you have correctly said. You might not need any algorithm to detect these known abnormality.

1) But even if u apply K-means, It will give you same result and will do it in automatically way.

2) Seems like you have just 1 variable -total ectivities, things get dirtier when you have multiple variables then you might have to look into LOF/ABOD other algorithm after identifying what exactly is abnormality for you.  Have a look here-

https://machinelearningstories.blogspot.com/2018/07/anomaly-detection-anomaly-detection-by.html

3) In ML , many a times you even do not know what exactly is an abnormlaity. Here you assume high activities is abnormality. depending on problem statement we need to have some unsupervised algo to identify possible abnormalities.


Which cross-validation type best suits to binary classification problem Data set looks like:

25000 observations
up to 15 predictors of different types: numeric, multi-class categorical, binary
target variable is binary

Which cross validation method is typical for this type of problems?

By default I'm using K-Fold. How many folds is enough in this case? (One of the models I use is random forest, which is time consuming...)

You will have best results if you care to build the folds so that each variable (and most importantly the target variable) is approximately identically distributed in each fold. This is called, when applied to the target variable, stratified k-fold. One approach is to cluster the inputs and make sure each fold contains the same number of instances from each cluster proportional to their size.


Does overfitting depend only on validation loss or both training and validation loss? There are several scenarios that can occur while training and validating:

Both training loss and validation loss are decreasing, with the training loss lower than the validation loss.
Both training loss and validation loss are decreasing, with the training loss higher than the validation loss.
Training loss decreasing, but validation loss is increasing.

I am aware that overfitting occurs in scenario 3, but does overfitting occur in scenario 1? If so, does this mean that overfitting only occurs when either scenario 1 or scenario 3 occur? Otherwise, if overfitting only occurs in scenario 3, does this mean that overfitting only occurs when validation loss is increasing?

In my opinion, only case 3 should be considered overfitting. As @stans has mentioned, there is not a very rigorous definition of overfitting so other people might think differently.
I wouldn't say the point where the validation loss stops decreasing is where bias and variance are minimized since there is a trade-off between bias and variance:

A constant model will have very low variance, but very high bias.
An overfitting model will have very low bias, but very high variance.

The point where the validation loss starts increasing can be considered optimal in terms of the sum of squared bias and variance, that is, an optimum of the generalization error.


Can I treat text review analysis as a regression problem? I am playing with a dataset that contains tripadvisor restaurant reviews and their labels (either 1, 2, 3, 4 or 5 stars).

Initially I was thinking of using it as a classification problem, applying softmax, cross-entropy and so on, however upon a second thought it came to me the idea to actually treat it as a regression problem, as generating an output which is continuous (such as 3.897) is perfectly fine (assuming of course that the original value would have been 4).

Does this make sense? or should I just hot-encode the answers in a vector of 5 elements (on per star?).

Also, assuming that regression is a correct way to treat this, linear activation in the last layer would be the way to go, and the metrics to look at would be the mean squared error and the mean absolute error.

UPDATE:

After browsing a bit more, I found something named Ordinal Regression, actually if one thinks a bit about it, everything makes sense. If i just treat the problem as a classification one, and I have a prediction saying that a given review has 1 star when in reality it was 4 star, how would we penalize that?

If we are using a classification problem and we are using cross entropy, that is simply a missclassification error, and labeling something as 1 start when it should have been a 4 start is consider as incorrect as labeling something with 3 stars when it should have had 4 stars. As you can see this is incorrect, both predictions (1 and 3 stars) are incorrect, but one is waaay more incorrect than others.
However if we are using a regression problem, our cost function will clearly penalize more a prediction of 1 start rather than one of 3 stars (again, here we assume that the correct label would have been 4 stars).

So given all that, I would say that regression makes sense in this case, as the categories are related to each other.

Yes, regression makes sense.

Many MLer working on similar tasks (e.g. the Yelp challenge) use classification instead of regression because they collapsed the label space to 2 (positive or negative) or 3 (adding neutral). For predicting numerical scores 1~5 regression makes more sense.


Is it right to impute Train and Test set? I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing this, I split my data (using Pandas) into training and test and then encoded them. After checking, I realized that my training and testing datasets had Nan's. Since you can not run an ML algorithm with Nan's, I decided to impute both of them with MissForest.

My question is: Is what I have done acceptable?

Yes, it is possible to impute both the train and the test set. You have to be careful not to introduce information leakage by splitting - if you impute for the train set, then use the same imputation process for the test set as well. I believe that was mentioned in a comment as well. Here is some further information:

See https://stats.stackexchange.com/questions/95083/imputation-before-or-after-splitting-into-train-and-test


How to impute missing value in Test Set using a custom Imputer created on training dataset I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique.
Under this technique, I replaced missing values with the mean value of the two categories of the target feature. The code snippet is as below:
dataframe['Feature'] = dataframe['Feature'].fillna(dataframe.groupby('Target Feature')['Feature'].transform('mean'))

Using this strategy I have designed classification models based on Logistic Regression and Support Vector Classifier. Now I have to run my models on a test set but am stuck at the pre-processing stage. The test set also has missing values in the same feature (as in the training set), now how can I update these missing values with the mean values that the models learnt from the training set.

The way you are imputing your feature can't be replicated in the test set, because it needs knowledge of the target classes!
You need to select a different imputation strategy, that doesn't rely on your target feature.

Assuming that you are using another feature, the same way you were using your target, you need to store the value(s) you are imputing each column with in the training set and then impute the test set with the same values as the training set. This would look like this:
# we have two dataframes, train_df and test_df

impute_values = train_df.groupby('Another Feature')['Feature'].mean()

train_df['Feature'] = pd.Series(train_df['Feature'].values, index=train_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

# train your model ...

test_df['Feature'] = pd.Series(test_df['Feature'].values, index=test_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

Example:
train_df = pd.DataFrame({'f1': ['a'] * 5 + ['b'] * 5, 'f2': range(10)})
test_df = pd.DataFrame({'f1': ['a'] * 3 + ['b'] * 7, 'f2': range(10, 20)})
train_df.loc[[1, 6], 'f2'] = np.nan
test_df.loc[[1, 6], 'f2'] = np.nan

impute_values = train_df.groupby('f1')['f2'].mean()

train_df['f2'] = pd.Series(train_df['f2'].values, index=train_df['f1']).fillna(impute_values).reset_index(drop=True)

test_df['f2'] = pd.Series(test_df['f2'].values, index=test_df['f1']).fillna(impute_values).reset_index(drop=True)


How to estimate student's level according to score and time spend I have some data that contains students answer, score, and time spent in many exams.

How to estimate each students level according to this data?

Sample data:

student ID, topic ID, score, time spend

  sid1    ,   tid1  ,  10  , 5s

  sid1    ,   tid2  ,  4   , 15s

  sid1    ,   tid3  ,  3   , 25s

  sid1    ,   tid4  ,  10  , 5s

  sid2    ,   tid1  ,  11  , 15s

  sid2    ,   tid2  ,  6   , 11s

  sid2    ,   tid3  ,  7   , 24s

  sid2    ,   tid4  ,  0   , 53s

I think there will be different approaches, and different results, to this problem.
The first which comes to my mind is simple:

evaluate the difficulty of each topic, by the average of the time spent by the students on it;

use the evaluations at point 1) to weight the time spent by the students on the topics;

calculate the average weighted time spent on all the topics;

a) you can rank the students based on the average weighted times, or
b) you can plot an histogram of the average weighted times, and choose some proper tresholds to classify the students


Alternative methods for novelty detection and correlations Hey mates I have the following project:

Imagine having two datasets A and B. Each dataset consits of 101 time series with the same lenght and identical time stamps. The two datasets where taken from the same experiment, therefore the data structure is identical. From the 101 time series there is one particulary signal that is of interest in both datasets. Lets call that signal X(t)_101. Now we have the following case that the signal X(t)_101 from dataset A (good dataset) is significantly different than from dataset B (bad dataset). The goal now is to analyse the remaining 100 signals and find the signals that have the biggest effect on making our signal of interest X(t)_101 go bad. So in the end I would get some sort of ranking, on which signal is responsible for the change of our signal X(t)_101.

So far I am looking into novelty detection and correlation, are there any other keywords I should consider for this topic?

Thanks

I'm doing something similar and found that the best approach has to be comprehensive. My process obtains a ranking of signals based on degree and measure of anomaly, both as a function of 'bads' (if available). If information on 'bads' is unavailable then I perform the same study but as a function of the degree of anomaly. In both cases, I perform hierarchical clusters. These are helpful in grouping signals that have similar behavior.


Human readable format for clusters of word vectors Let's say I have pretrained word2vec model and apply it to dataset consisting of article titles from "The Guardian". It seems pretty obvious that titles coming from "Science" section would form one cluster in latent space and titles from "Fashion" section would form another cluster in latent space. But the thing is my dataset doesn't have category label for each title. How can I come up with such human readable interpretation of cluster centers(probably coming from Kmeans)?

The usual way is to present the top N (e.g. top 10) words for the cluster:

With distance-based clustering like K-means, the top words can be picked as the closest ones to the centroid.
With probabilistic methods such as LDA, the top words are the ones with the highest probability for the topic.


Look up a number inside a list within a pandas cell, and return corresponding string value from a second DF (I've edited the first column name in the labels_df for clarity)

I have two DataFrames, train_df and labels_df.  train_df has integers that map to attribute names in the labels_df.  I would like to look up each number within a given train_df cell and return in the adjacent cell, the corresponding attribute name from the labels_df.

I've tried variations of the function below but fear I am wayyy off:

def my_mapping(df1, df2):
    tags = df1['attribute_ids']
    for i in tags.iteritems():
        df1['new_col'] = df2.iloc[i]
    return df1

The data are originally from two csv files:

train.csv

labels.csv

I tried this from @Danny :

sample_train_df['attribute_ids'].apply(lambda x: [sample_labels_df[sample_labels_df['attribute_name'] == i]
                                              ['attribute_id_num'] for i in x])

*please note - I am running the above code on samples of each DF due to run times on the original DFs.
which returned:

I created my own data.

train.csv

id,attrib
1,1 2 3
2,3 4 5
3,2 3 5
4,1 1 1

labels.csv

attrib_id,attrib_name
1,a
2,b
3,c
4,d
5,e

Read the csv files and create df1 and df2

After that use

def get_name(x):
    result = []
    for t in x.split(' '):
        result.append(df2[df2['attrib_id']==int(t)]['attrib_name'].values[0])
    return result

df1['attrib'] = df1['attrib'].apply(lambda x: get_name(x))

This will result in df1 looking like

I guess you also doing the same thing when you referred @Danny. The only thing important here is to convert the string into integer


Implementation of reliable rule learning I want to perform "reliable rule learning", i.e. mining a set of rules with a very low number of false negatives. I recently read the paper "Reliable agnostic learning" by Kalai et al. (https://doi.org/10.1016/j.jcss.2011.12.026) and they basically describe what I want: Rules are determined to reliably classify data points, and the reliability is partly reached by allowing "I don't know" as an additional answer. Sadly, their paper is purely theoretical and I could not find a corresponding implementation. Is there an implementation of reliable rule learning or a similar algorithm? At best, open source and Java-based?

I think this paper just compares algorithms:
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.294.9414&rep=rep1&type=pdf
If you want something specific, here's the white paper for SLIPPER:
https://www.aaai.org/Papers/AAAI/1999/AAAI99-049.pdf
Hope that helps.