Analysis for basic weight training analysis? TL;DR: I'm doing a fairly basic project which involves exercise. It seems that descriptive statistics and basic data vis (ex: line graph) would be most appropriate for this project, but I wonder if you have any recommendations for analyses.

For this project, I am performing the same set of 15 single-joint exercises each week (we'll call these "Exercises"). Every 4 weeks, I'm performing 3 different multi-joint exercises (we'll call these "Lifts"). My goals are to:

Track my progress (strength gains) from the Exercises.
See if my rate of progress on the Exercises roughly matches my rate
of progress on the Lifts, which would possibly indicate some sort of
transfer effect.
Explore relationships between strength gains measured during each
workout and other factors (including sleep quality, stress)

My hypothesis is that my strength gains from the Exercises I perform will transfer well to the strength gains from the Lifts that I perform.

The design is a bit janky (and it's more of a correlational study design than an experimental design), but this project is mostly just to demonstrate data collection and basic data analysis concerns. However, I'd like to make the analysis as interesting as possible. So far, it seems that basic descriptives are the best way to analyze this data, with something like a line plot for the gains on the Exercises and the Lifts.

Do you have any suggestions for analyzes that could be performed?

The variables I'm tracking are:

Weight, repetitions, and sets for each Exercise (15 in total, each exercise performed weekly)
Weight, repetitions, and sets for each Lift (3 in total, each Lift
performed once every month)
Sleep quality for the night before each workout
Energy before the workout and after the workout
Stress levels before the workout and after the workout
Level of physical fatigue before the workout
Weight before the workout
Heart rate (before the workout and once during the workout)
Exertion of the workout (rated after the workout)

Thanks!

I'd start with PCA to see the main correlations and maybe clustering. With the PCA you should be able to find nice sub-spaces, which you can plot and see, which parameters correlate the most/least. Clustering might help you identify "growth spurts". You might also try some linear regression to predict weight repetitions given the other parameters. Here I would be careful though, as the best predictor of performance will probably be the implicit 10th variable: time. You might want to tweak the regression problem to predict changes in performance given sleep, stress etc. In general, depending on what you are looking for, it might make sense to consider "change in repetitions" rather than the repetitions themselves.

I hope this helps.


One hot encoding of target space I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, specs, ...) out of 4 possible brands, how do you encode the brand variable? My answer was one hot encoding, it was accepted but then they asked me to do it explicitly and I sketched something like:

brand A -> [1,0,0,0]
brand B -> [0,1,0,0]
brand C -> [0,0,1,0]
brand D -> [0,0,0,1]

And then, I was corrected under the reason that these columns were not independent. And that the solution should have been three binary columns instead.

Later it hit me that I do not know why independence is required, and also that three binary variables are not independent. Two would be.

Can someone provide some explanation to help with my confusion?

For certain models, it can be important to make sure the inputs are linearly independent. I'm pretty sure this is never the case for the outputs. If we have more than two classes, it's extremely difficult if not impossible to output class-probabilities if we don't have a specific dimension for each class in the output vector. In other words, your suggestion has much more explanatory power than what was suggested by your interviewer, which is solving a problem that exists on the inputs for some models but not for the outputs.

In multi-class problems, it is indeed the norm that each class has their own dimension in the output vector as you suggested. Either the interviewer was trying to look smart and confused input and output encoding (which still would only apply to certain families of models), or you aren't remembering the events of the interview properly. If you're confident that the interview went as you described, you should consider complaining to your point of contact that you were criticized for giving a correct answer.

Feel free to cite the following as popular/canonical examples of multi-class classifiers:

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba

  Returns: array of shape = [n_samples, n_classes]

https://keras.io/losses/#categorical_crossentropy

  when using the categorical_crossentropy loss, your targets should be in categorical format (e.g. if you have 10 classes, the target for each sample should be a 10-dimensional vector that is all-zeros expect for a 1 at the index corresponding to the class of the sample)


sub-setting based on categorical variable in R the below dataset is just top 8 rows of 100 rows dataset.

>a
  group marks    upd
2     C     3     up
3     C     4   down
6     C     7     up
1     T     2     up
4     T     5     up
5     T     6   down
7     T     1 middle
8     T     0   down

I used unique_groups=unique(a$group) to get a list of unique group names to be sent into a function.

Now, I need a function that gives a subset of this dataset where it takes each group from unique_groups and checks if 'upd' is in ('down','middle') and gives me a dataset having only those records having upd ('down','middle') of that respective group.

I tried to use do.call function to get the dataset that I want as-

do.call(rbind,sapply(unique_groups,FUN=function(y)
  {
  h=a[a$group==x & a$upd %in% c('down','middle'),]
  h
}
, simplify=F))

I wasnt getting the dataset that I want and it was giving this warning message:

In ==.default(a$group, x) :
  longer object length is not a multiple of shorter object length

This question seemed to be more suitable for general stackoverflow since it's a R question.

I'm using data.table. The syntax is a little bit different from data.frame, but the time spent on learning data.table is well worth the effort. See more tutorials on wiki, and this good sheet.

library(data.table)
s = "rowname    group   marks   upd
2   C   3   up
3   C   4   down
6   C   7   up
1   T   2   up
4   T   5   up
5   T   6   down
7   T   1   middle
8   T   0   down"
# fread is the fastest csv reader
a = fread(s)
# the i in data.table[i,j] subset rows, empty j select all columns
upd_subset = a[upd %in% c('down','middle'),]

> upd_subset
   rowname group marks    upd
1:       3     C     4   down
2:       5     T     6   down
3:       7     T     1 middle
4:       8     T     0   down

> upd_subset_bygroups = split(upd_subset, upd_subset$group)
> upd_subset_bygroups
$C
   rowname group marks  upd
1:       3     C     4 down

$T
   rowname group marks    upd
1:       5     T     6   down
2:       7     T     1 middle
3:       8     T     0   down


Using Predictions: Data does not have the same target as predictors As per the Orange video tutorial: youtu.be/D6zd7m2aYqU?list=PLmNPvQr9Tf-ZSDLwOzxpvY-HrE0yv-8Fy

I'm trying to make predictions on a new dataset based upon one that's already been 'coded' - both data sets are formatted exactly the same, where the 'training' set has all entries marked as either 1 or 0, see below:

whereas the second set contains no values, as I hope the 'predictions' feature will try to guess the status:

Trouble is, when I look at the predictions widget, it does nothing, displaying 'Data does not have the same target as predictors'

I've only been using Orange for two days, so I imagine it's a really simple error on my part.

Any help would be greatly appreciated.

John

This is because we have to exclude the target column from the second document (testing data-set).

In your example remove the column 'Status' from second file.

Example:

In below example, in training dataset, 'Loan_Status' column is present which is the target field, whereas, in testing dataset, 'Loan_Status' column is not present, and will be computed by the 'Prediction' as shown below:

Prediction:


Visualizing Latent Features In performing ALS and getting an item matrix of latent features, what would be the best method for inferring the possible "meaning" of each latent factor in the item space? And as a corollary, are the any methods for successfully visualizing this data to gain some intuition or understanding?

Depending on the item domain You can expect different results. I would try to cluster data using cosinus similarity over item latent features. If number of items is large try DIMSUM (it's already implemented in Spark).

For the visualization You can try one of the available implementation of t-SNE. Quick links: wiki and YT - Google Talk about t-SNE Visualization. But here I cannot give You any more advice, because I have never tried to visualize it. I looking forward You will share Yours results.


How to tackle a multilabel classification problem I am trying to build a LSTM model for a multiclass classification problem on textual data. Until now, I have only built a model when one input belongs to one of the categories. What do I do when one input can belong to more than one class (i.e.: one entry of data can belong to 2-3 categories)? Can anyone help me with some blogs or resources to build an intuition for making such a model?

Thanks.

LSTMs, like any other neural net, implicitly support multi-label classification.  You should ensure that your output layer has $n$ neurons, one for each class, and you should use logistic activation rather than softmax activation (which is typically used for the final layer in multi-class problems).
Each neuron $N_i$ in the final layer will output a value between 0 and 1.  If the output of $N_i$ is greater than 0.5, then the example is a member of label $i$.
Here's a pretty good article that might help.


How can we define missing rating in recommender system? I was reading about collaborative filtering where we need to pass (user, item and rating) in case of matrix factorisation (SVD). Now, my question is given data of following form

User |   Item   | Rating
A    |   X1     | 1
A    |   X2     | 3
B    |   X2     | 4
C    |   X1     | 3
C    |   X3     | 2

we need to convert the it into

U/I       | X1  | X2  | X3
A         |  1  | 3   | -
B         |  -  | 4   | -
C         |  3  | -   | 3

So, we need to replace all the - with the predicted value prior to applying svd on it. Now I would like to understand what are most practical or mostly adopted way to predict such missing rating:
I am aware about following crude ways, but they are not prediction of value but they are mere replacement of missing value

Replace all missing values by neutral rating . (Say either 2 or 3 if rating are from 1-5).
Replace with mean rating of the movie.
Replace with average rating of that user.
etc...

For the Collaborative Filtering while using Matrix Factorization you don't necessary use a vanilla SVD over a dense matrix. As for the most cases the input dataset of ratings is vary sparse and your U/I rating matrix will have most of values set to 0.0. So, when SVD is done for input date the final recommendation will suffer of that.

And this was the main breakthrough in the publication of using SGD for MF in Netflix Prize contest in 2006. All is described by the author here. In short - you calculate your lost function only over the seen ratings set, not over the all matrix (as all the zeros or other value actually represent unknown/unobserved cases). Thus, the natural choice is to use sparse matrix for your input data.


Machine learning for state-based transforms? If I provide:

A list of possible transforms, and,
A list of input states, and,
A corresponding list of output states for each input state, and,
A fitness function to score each output state

Which subset of machine learning can direct me towards an optimization algorithm that can map each input state to a dictionary of input states, and, failing to find a match, apply the necessary transforms to get me to the closest-related output state?

An example involving polygon legalization:

Any given "window" can contain N different polygons, where each polygon has lower-left and upper-right co-ordinates, as well as a polygon "type".
The input state of the polygons may or may not be "illegal".
A list of transforms includes: move, copy, rotate, resize
If the input state maps directly to any output state, the input state is decided to be legal. Nothing more to be done; move on the next window.
If the input state matches any previously seen input state, transform to the matching (known-legal) output state. Nothing more to be done; move on the next window.
Attempt transforms in different sequences until a state is reached that satisfies a fitness function. Store this input:output state combination. Move on to the next window.

Would this imply some combination of neural networking (for classification) and genetic/evolutionary algorithms? Or, does the presence of a fitness function negate the need to store combinations of input:output states?

If i get it correctly:

You have an input polygon
As a first step you want to "match" that against a list of previously seen templates. If this is successful, you pick it's corresponding output and move on.
If not, you wish to find some optimal transformation, in order for it to satisfy some constraints that you have (your "objective function"). Then add the original+transformed shape to the templates list and move on.

Is this correct? I'll risk an answer anyways:

For the first part, I believe that there is a slew of literature out there. It's not my expertise, but first thing that comes to mind is measuring the distance in feature space between your shape and each template, and picking the closest one, it the distance is below a threshold that you set. "Feature" here would be either some low-level polygon property, e.g. x and y coordinates of vertices, or an abstraction, e.g. perimeter, area, no. vertices, mean side length/side length variance, etc.

For the second part, it really depends on the nature of your constraints/objective functions. Are they convex? Uni- or multi-modal? Single or multi-objective? Do you want to incorporate some domain knowledge (i.e. knowledge about what "good" transformations would be?)? One can really not tell without further details. Evolutionary algorithms are quite versatile but expensive methods (although some argue on that). If you can spare the possibly large amount of function evaluations, you could try EAs as a first step, and then refine your approach.

Finally, while not exactly related to what you describe in your process, I believe you may benefit by taking a look into auto-associative networks (and models in general); these are models that are able to perform constraint-satisfaction on their input, effectively enforcing learned relationships on input values. I could see this being used in your case by inputing a shape, and having a transformed shape as an output, which would be "legal", i.e. satisfying the constraints learned by the auto associative model. Thus, you would eliminate the need for a template matching + optimization altogether.


Training object detection models from the scratch without using pre-trained models and weights I have a data set of 25 images. I wish to run Faster RCNN or yolov3 object detection models on this images.I want to create my custom trained model and get weights after running say 10 epochs. Later I can save these weights and use that for prediction. Build a model, train on my image data set and get weights.
Is it possible?

Maybe you could first train the model on a large sized flower dataset, that way the conv layer will be optimised to detect flowers and then do transfer learning on that model using your custom dataset freezing all but the last layer.
I am not sure though, 25 images seem way too less. Try data augmentation on those images like horizontal and vertical flips, noise, shear etc.


How to select regression algorithm for noisy (scattered) data? I am going to do regression analysis with multiple variables. In my data I have n = 23 features and m = 13000 training examples. Here is the plot of my training data (area of houses against price):

There are 13000 training examples on the plot. As you can see it is relatively noisy data. My question is which regression algorithm is more appropriate and reasonable to use in my case. I mean is it more logical to use simple linear regression or some nonlinear regression algorithm.

To be more clear I provide some examples.
Here is some unrelated example of linear regression fit:

And some unrelated example of nonlinear regression fit:

And now I provide some hypothetic regression lines for my data:

AFAIK primitive linear regression for my data will generate very high error cost because it is very noisy and scattered data. On the other hand, there is no apparent nonlinear pattern (for example sinusoidal). What regression algorithm will be more reasonable to use in my case (house prices data) in order to get more or less appropriate houses' price prediction and why this algorithm (linear or nonlinear) is more reasonable?

The model I would use is the one that minimizes the accumulated quadratic error. Both models you are using, linear and quadratic, looks good. You can compute which one has the lowest error.

If you want to use an advanced method you can use RANSAC. It is an iterative method for regression that assumes that there are outliers and remove them from the optimization. So your model should be more accurate that just using the first approach I told you.


Can I use SVC() as a base_estimtor for ensemble methods? I am currently testing out a few different ensemble methods on my dataset. I've heard that you can also use support vector machines as base learners in boosting and bagging methods but I am not sure which methods allow it or not. In particular, e.g. for XGB i tried out trees and SVMs as base learners and got the exact same result for 5 different performance metrics which made me question the results and/or that the option can only take trees as base learners. I didn't find much info in the documentation or at least not in all of the documentations. I would be interested about AdaBoostClassifier(), BaggingClassifier() and XGBClassifier(). Does anybody know the details and whether or not I can use SVMs here as base learners?

In short: Yes.

Conceptually, bagging and boosting are model-agnostic techniques, meaning that they work regardless of the learner.

Bagging essentially is the following:

create multiple predictors (they can even be hard-coded!)
gather predictions from the learners and come up with a prediction

Boosting can be seen as:

train a predictor
find where the predictor makes mistakes
put more emphasis on these mistakes
repeat until satisfactory

Regarding the specific Sklearn implementations, here are the base learners that you can use:

AdaBoostClassifier()

The documentation says Support for sample weighting is required, as well as proper classes_ and n_classes_ attributes.

This means that you can use all models that can give weight to your samples as part of the learning process (KNN, SVM, etc.)

BaggingClassifier()

This is a simple bagging strategy, so all estimators can be used here.

GradientBoostingClassifier()

This requires that your learners are differentiable so that the gradient can be computed. Generally, this technique is specific for tree learning.


Can this problem be solved using deep learning? I want to predict price of used cars. I have data like this:

Is this problem suitable for deeplearning or Should I use XGBOOST, RandomForest etc.?

I used one hot approach for nominal features and scaled numeric values between 0-1. Is there anything I can do to improve my data?

I used deeplearning4j on java and predict prices with 10-15% error but I want to predict prices with 1-2% error

The problem is a case for "deep learning", I think, and I would start with boosting. I have good experiance with catboost (in Python). It can handle factors very well ("catfeatures", encoded as dummies) and it is relatively fast. Also lightgbm might be a good option. Also xgboost is an option.

I think data-wise one hot encoding and scaling is the first thing to do. Things beyond that is a bit try and error. The most important part might be that you tune your hyperparameter well (during boosting).

Catboost (Python): https://catboost.ai/docs/concepts/python-usages-examples.html
LightGBM (Python): https://lightgbm.readthedocs.io/en/latest/Python-Intro.html

Happy coding!


Regular expression in python - I want to extract the values of the below text

Paﬁent Name : Thomas Joseph MRNO : DQ026151?
Doctor : Haneef M An : 513! Gandar : Male
Admission Data : 19-Feb-2V'3‘¥T12:2'$ PM Bill No : IDOGIII.-H-17
Discharge Date : 22-Feb-20$? 1D:5‘F AM Bill Dale : E2-Feb-2017

extract only the values of the field names for example,

Thomas Joseph from the field name Pateint name, similarly for others field names and save the output to excel

Python code for the above

My attempt -

text = pt.image_to_string(img1)
print(text)
s = re.findall(r'\s:\s(\w+)', text)
print (s)

As @spacedman correctly mentioned, this will answered quicker at StackOverflow. But you can use this to create a dictionary like this. There might be a better way but this is a quick work around.

# -*- coding: utf-8 -*-
import re
st = '''Pafient Name : Thomas Joseph MRNO : DQ026151?
Doctor : Haneef M An : 513! Gandar : Male
Admission Data : 19-Feb-2V'3‘¥T12:2'$ PM Bill No : IDOGIII.-H-17
Discharge Date : 22-Feb-20$? 1D:5‘F AM Bill Dale : E2-Feb-2017'''
st = st.decode('utf-8').replace('\n','')+'<eof>'
words = ['Pafient Name','MRNO','Doctor','An','Gandar','Admission Data','PM Bill No','Discharge Date','Bill Dale','<eof>']
print {words[i]:st[st.index(words[i])+len(words[i]):st.index(words[i+1])].replace(':','').strip() for i in range(len(words)-1)}


How to Set the Same Categorical Codes to Train and Test data? Python-Pandas NOTE:
If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that:
You encode categorical values as INTEGERES only if you're dealing with Ordinal Classes, i.e. College degree, Customer Satisfaction Surveys as an example.
Otherwise if you're dealing with Nominal Classes like, gender, colors or names, you MUST convert them with other methods since they do not specific any numerical order, most known are One-hot Encoding or Dummy variables.
I encorage you to read more about them and hope this has been useful.
Check the link below to see a nice explanation:
https://www.youtube.com/watch?v=9yl6-HEY7_s
This may be a simple question but I think it can be useful for beginners.
I need to run a prediction model on a test dataset, so to convert the categorical variables into categorical codes that can be handled by the random forests model I use these lines with all of them:
Train:

data_['Col1_CAT'] = data_['Col1'].astype('category')
data_['Col1_CAT'] = data_['Col1_CAT'].cat.codes

So, before running the model I have to apply the same procedure to both, the Train and Test data.
And since both datasets have the same categorical variables/columns, I think it will be useful to apply the same categorical codes to each column respectively.
However, although I'm handling the same variables on each dataset I get different codes everytime I use these two lines.
So, my question is, how can I do to get the same codes everytime I convert the same categoricals on each dataset?
Thanks for your insights and feedback.

First, note that Random Forests can handle categorical variables (moreover, if you have too much categories, reducing this number is a good practice).
If you want to apply a filter to your data, I'd suggest you using sklearn transformers (like OneHot Encoder, Label Encoding, ... pick the one you need according to what you want to do).
In this case, you have to fit the encoder in your train dataset, and then apply it in your test. If you want to apply this in a real case, you have to save your trained encoders alongside your trained model, so you can apply the encoder directly to the new data before predicting on it, so it has the same pattern.
Here is an example with Label Encoder
from sklearn import preprocessing
train, test = ... # SEPARATE YOUR DATA AS YOU WANT
le = preprocessing.LabelEncoder()
trained_le = le.fit(train)
train = trained_le.transform(train)
test = trained_le.transform(test)


Text classification and predictive model I have collected a lot of data that I would like to analyse and classified. Unfortunately, they are not already labelled, so I am going to do manually.
The dataset consists of texts in Italian and I have not found a lot of models that I could use as a training model for labelling them and classifying them between True and False.

Suppose that I have 30000 of texts, which percentage could be enough to build a model for predicting the rest from that?
Do you have any model that I could build/use once labelled them?

As a person trying to do text classification myself,Let me try to help you to get started here. Please don't bother reading ahead if your question is about finding models which are able to fit the "italian" language features and find co-relation and you already know how to do text classification in general. If not then-

It is recommended to formulate your problem first properly i.e. what exactly are you trying to solve here. "..........classifying them between True and False." It doesn't give a lot of clarity. This should be clear for you.
What kind of data you are dealing with, saying "30000 of texts" does that mean 30000 lines of text? documents? books?
Now comes the part where we can do EDA which might include pre processing of data also. try to plot your data with respect to your labels and get some insights on how to go about the problem, if there are trends , patterns you can utilize.
There are many techniques to get the features out of text data. there is feature selection, feature extraction and there are various methods to do that.
Then you can decide on which classification models will fit the data, i.e.model selection, null hypothesis if I might suggest. You can try various models to fit the data and compare its performances and choose the one which gives you the best results.there is cross validation for example do help you decide on this.There is model evaluation which shall help you decide if the model is able to generalize well on unseen data.

Here is the point about percentage of data used for training and testing. Read about cross validation and hold out method to help you understand it. Generally it s a good idea to 70/30 ratio to train and test the model and also training data can be split into training and validation set.

However, here comes the famous bias-variance problem w.r.t. training size, generally more data leads to better generalize which avoids over-fitting. With less data there are more chances of over-fitting. Hence you should try run the models and plot the graph to understand if its over-fitting or under fitting(if you think you have way low data for any model to fit and predict).
https://scikit-learn.org/stable/modules/learning_curve.html

With the best selected model you can start predicting the results.
After you have got the best fit model you can tune its hyper parameters to get the better performance.

Now these are just rough list for you to go on. Please try below link to read about text classification with sk-learn which might be a good starting point if you haven't done this before-

https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html
https://www.nltk.org/book/ch06.html


Cyclic dependency between feature and predictor class I have a feature which has specific categorical values ex(Technology, Hardware, Software, Marketing, Evnts etc). Based on this and some other features, I am trying to classify the dataset into 2 categories IsSoftwareSystem or NotSoftwareSystem. In this case is this cause a reduce in accuracy because i am feeding the category itself in the data and trying to predict the same.
Using Random Forest/XGB.

You have two problems:

Technical problem: As 10xAI said in a comment, if the target also belongs to the features then the model should very easily predict every instance correctly. So you should obtain perfect performance on the test set. The ML model doesn't care about "cyclic dependency", it uses any good indicator it receives as feature. Since you mention a reduction in accuracy, it means that there is an error somewhere in the process.
Semantic problem: which problem are you trying to solve with this ML setup? If the goal is to distinguish software vs. non-software and you have as input a category "software" then it's pointless to train a ML model. You can obtain the same result much more efficiently with a simple condition:

for each instance:
  if instance.category == "software":
     instance.answer="software"
  else:
     instance.answer="non-software"


What is exactly psi and segZ function in "segment" function in R I am using "segment" package in R for fitting piecewise linear segments to a sample parabolic curve in R. I am not able to able to interpret the functions
"psi" and "segZ" from the R documentation. I have two variables X and Y .
If we consider X as dependent and Y as independent variable then,

  reg<- lm(Y~X, data = curved_data)

I am creating a parabola from the sample data given below-

  x<- c(-37, -36, -35, -34, -33, -32, -30, -28, -26, -25, -24, -23, -22, -21, -20, -19, -18, -17, -16, -15, -14, -12, -10, -8, -7, -5, -4, -2, -1, 0, 1, 2, 4, 5, 7, 8, 10, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 30, 32, 33, 34, 35, 36, 37)

y=4x^2

That is I am constructing a parabola with the help of the above equation in R.

So in this case y is independent and x is the dependent variable.

I want to have a piece wise linearity fit for this parabolala.

I have tried using two different codes of R i.e.

  seg_model<- segmented(reg, seg.Z = ~curved_data$X, psi = c(-35, 4, 30), control = seg.control(display = TRUE))

-35, 4, 30 are the 3 breakpoints that are chosen by me. Are they correctly chosen? How should they be chosen?

I get the error

    Error in segmented(reg, seg.Z = ~curved_data$X, psi = c(-35, 4, 30),  :
      psi' should be a list with more than one covariate inseg.Z'

What does this error means?

Also I have tried using code by writing psi = NA

I have got the below error

    Error in segmented(reg, seg.Z = ~curved_data$X, psi = NA, control = seg.control(display = TRUE)) :
      psi' should be a list with more than one covariate inseg.Z'

I am not able to understand psi and segZ from the R documentation. Please can I get an elaborate functioning of these two functions?
What is the relation between psi and segZ? What should I do to correct this error? Does number of points in vector of psi means number of breakpoints OR what is it?

You must attach the data frame curved_data before running the commands. Then, run your command like this:

seg_model <- segmented(reg, seg.Z = ~X, psi = c(-35, 4, 30),
                       control = seg.control(display = TRUE))


Fast checking of set membership on large data Let's say that I am an email provider like Gmail. Let's assume that I have two categories of email-address: spammers and non-spammers. When my servers receive a mail I need to quickly check if that email-ID is in the spammers set and if yes I take some action.

The problem is that each email-Id could be several bytes ( say 10 bytes each) and I might have 1 billion spammers so I need 10GB of RAM to just store the emails in main memory.

Let's say I want to use only 1 GB RAM. To do this I'm now ready to accept an approximate answer. In particular I'm ok with a non-spam email mistakenly flagged as belonging to the spam  set but not vice versa. How will I do it?

Somebody already tagged this with "bloom-filter".  I agree with them.

A Bloom Filter is a hash-based technique that never results in false negatives, but trades off probability of false positives for saved space.  It meets your specific performance requirements - "fewer than 10 bits per element are required for a 1% false positive probability, independent of the size or number of elements in the set (Bonomi et al. (2006))." (from the linked article)

The main reason you wouldn't use a Bloom filter for this is that removing spammers is computationally expensive, requiring you to rebuild the filter or track your list of removed spammers separately.


Sentimental Analysis on Twitter Data What are best ways to perform sentimental analysis on Twitter Data which I dont have labels for?

You should look at literature on unsupervised sentiment analysis. The paper by Peter Turney could be a good starting point.
Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews Turner 2002

You can also check this if you use R https://datascienceplus.com/unsupervised-learning-and-text-mining-of-emotion-terms-using-r/


How to build Generative Model when we have more than one variable I have a data-frame which has looks similar to this:
A   B   C
1   2   2
2   4   3
4   8   5
9   16  7
16  32  11
22  43  14
28  55  17
34  67  20
40  79  23

A,B and C can be assumed to be the features in machine learning literature.
I have read maximum likelihood estimation for 1 variable assuming Gaussian distribution.
The equation is something like, where xi's are each data-point:

Where x1,x2....xn are n data points each having dimension 3. If we assume p(x) to be gaussian, then we can use the Gaussian Normal distribution equation as:

This is well understandable if we have only 1 feature.
How can I generalise the above normal distribution equation when we have more than 1 features, here we have 3 features? Can someone help me to write the maximum likelihood for the above data-frame?
Do we learn mu and sigma for each features A,B and C that is total of 6 learnable parameters?
If we have 3 different distribution say Normal, exponential and so on for columns A,B and C then how does the MLE equation looks like over entire-data-frame?
If we do argmax of equation 1, we don't require the ground truth for it. right? We are just maximising the equation?

First, I'd like to clarify. The maximum likelihood function you gave there is NOT with respect to "one feature". Or at the very least, it's not meaningful to think of this as a "feature" because in your example, you would actually evaluate the loglikelihood function at observed values of y not x. You aren't using your variables in x to explain anything about y. The resulting maximum likelihood estimate you would get for $\mu$ would be $\bar x$, the sample mean of your target variable y.
Now, suppose you actually do want to use x (your features) to predict y as in the case of supervised learning. Then, as I alluded to in your comments you need to specify two things - the predictor or model function, denoted typically as $\hat f(X)$ and the "link function", denoted as $g$.
$\hat f(X)$ is a function of your predictor variables such that:
$$g(E[Y|X]) = \hat f(X)$$
In your case, $E[Y | X] = \mu $ since you have a Normal distribution. Hence,
$$g(\mu) = \hat f(X) \rightarrow \mu = g^{-1}(\hat f(X))$$
Now, in terms of choosing $\hat f(X) $, this depends on your goals and how complicated you wish to go. Regardless, this is a function that can take on any real number. In the standard case (say in linear regression) you set $\hat f(X) = B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}.$ There are other examples where writing out $\hat f(X)$ is impossible or tiresome, for example in the case of gradient boosting trees or deep neural networks. Other algorithms may set $\hat f(X) = B_{0} + h_1(x_{1}) + h_2(x_{2}) + h_3(x_{3})$ where $h_{i}$ are smooth functions. It really depends on again how complicated you wish to get and how interpretable you need your models to be.
With respect to g, the "link function", this is almost always chosen depending on the range of the response variable or the range of the parameter that you are linking with ($\mu$ in your case). In your case, since the normal distribution can take on any real number, most of the time $g$ is chosen as the identity function since $\hat f(X)$ naturally can take on any real number already. This leads to:
$$\mu = B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}.$$
Finally, when dealing with your likelihood function:
$$p(Y) = constant * \prod_{i = 1}^{n} e^{\frac{-1}{2}\frac{(y_{i} - (B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}))^2}{\sigma^2}} $$
Solving this equation will provide you with the ordinary least squares estimates which I am sure you have seen before. Of course, choosing a different $\hat f(X)$ or a different $g$ will likely change everything and often you may not even get nice-looking closed-form solutions that come out with ordinary least squares. This motivates different numerical optimization methods. However, the "ingredients" are the same.
I hope this helps.


Keras mnist.load_data() unshuffled? I want to do some data augmentation on MNIST and therefore must manually label the Y set. This can be automated if MNIST is unshuffled, however mnist.load_data() appears to be returned in a shuffled state. How do I load this with Keras or TensorFlow in an unshuffled state?

UPDATE: It's trivial to unshuffle manually with:

perms=np.where(Y==0)
x[perms] #all X that are 0

Nevertheless would be nice if Keras had a way to import MNIST directly without shuffle.

I do not believe that Keras returns the MNIST data shuffled. You can see that it is not the case below.

First I will define a function which we will use to plot the MNIST data nicely.

import matplotlib.pyplot as plt
%matplotlib inline

# utility function for showing images
def show_imgs(x_test, decoded_imgs=None, n=10):
    plt.figure(figsize=(20, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(x_test[i].reshape(28,28))
        plt.gray()
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

        if decoded_imgs is not None:
            ax = plt.subplot(2, n, i+ 1 +n)
            plt.imshow(decoded_imgs[i].reshape(28,28))
            plt.gray()
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)
    plt.show()

Then let's reload and plot the MNIST dataset 4 separate times. We will see that the order of these is unchanged

for i in range(4):
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    show_imgs(x_train, x_test)
    print('Training labels: ', y_train[0:10])
    print('Testing labels : ', y_test[0:10])

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]


Analyze performance Poisson regression model on a time series(count forecasting) I have tried to build a model to forecast the count of a particular variable.The model that was used for the purpose was poisson .Unfortunately ,i don't have enough stat knowledge to analyze the model performance .If somebody can provide some insights as of how the model is performing,as well as some tweaks to improve the model performance will be greatly helpful. I am also willing to try out other models if it performs better.

I am using python with the statmodels package to build the model.

  Attaching a graph which shows the fitted and the actual
  values(Green shows the actual values and Blue shows the fitted
  values)

Also,providing the summary() output of the model

                 Generalized Linear Model Regression Results
==============================================================================
Dep. Variable:         Work_Item_Type   No. Observations:                  581
Model:                            GLM   Df Residuals:                      574
Model Family:                 Poisson   Df Model:                            6
Link Function:                    log   Scale:                             1.0
Method:                          IRLS   Log-Likelihood:                -16752.
Date:                Mon, 22 Feb 2016   Deviance:                       31268.
Time:                        21:59:12   Pearson chi2:                 1.05e+05
No. Iterations:                     9
===============================================================================
                  coef    std err          z      P>|z|      [95.0% Conf. Int.]
-------------------------------------------------------------------------------
Intercept       2.8492      0.051     55.426      0.000         2.748     2.950
Weekday        -0.2066      0.032     -6.446      0.000        -0.269    -0.144
day_of_week    -0.0926      0.007    -13.367      0.000        -0.106    -0.079
wom             0.1122      0.007     16.996      0.000         0.099     0.125
week           -0.0411      0.001    -53.597      0.000        -0.043    -0.040
TimeDelta       0.0001    5.1e-05      2.933      0.003      4.96e-05     0.000
month_of_yr     0.2192      0.004     60.981      0.000         0.212     0.226
===============================================================================

Also attaching a sample of the dataset used

clear_date  Count_Work_Item_Type
7/7/2014    1
7/10/2014   1
7/11/2014   5
7/17/2014   2
7/22/2014   1
7/24/2014   1
7/29/2014   3
7/30/2014   4
8/13/2014   1

Since i had only the date and the variable to be forecast i created a bunch of other variables like

Weekday(binomial)?
Day of week
Week of Month
Week
Time Delta (Starts from 0 increment by one until end)
Month of Year

Also,i haven't done any kind of transformation on the variables.
Please do comment if you need additional information:
Thhanks

I'm not sure what you mean by "performance", but if what you mean is fit the answer is clear. You need to be using the log-likelihood to differentiate between different models. Basically, when you are fitting the model you are trying to maximize the log-likelihood. Thus the log-likelihood is giving you some sense of how well the parameters of your model are doing at fitting the data.

In your case, you want to get log-likelihood to be as close to zero as possible. Now this is kind of terrible advice, because if you were clever enough to come up with a feature for every observation, you could get a perfect fit. That's bad because your model would be completely useless. There are functions that take your log-likelihood as an input and transform it to penalize you for adding more variables, etc. We won't worry about those right now. Just keep in mind not to mindlessly lower the log-likelihood.

Once you have a model that you can live with you should run some sort of cross-validation, and/or hold out set. Then you can use any number of metrics to validate the predictive performance of your model. I think that is the more important of the two issues. You could calculate the mean square error on your hold out set. $$MSE=\sum_{i=1}^n(\hat y_i - y_i)^2$$
This would give you a really basic metric to assess how well your model is predictive of the output.


An outlier detection for this data I have a vector and want to detect outliers in it.

I need an outlier detection method (a non-parametric method) which can detect red points as outliers.

Edit:
I have a lot of vectors like this. The pattern is similar to this but the values are different. I mean we can see the height since the values are not constant but overall the patterns of outliers against the normal data are like this. This means in one place they beak the continuity of the blues points and go in different directions. Here is another figure.

Thanks in advance.

Not sure if this is time series data, but it looks like it might be.

For any given narrow window of time there is a distribution that looks like it's centered somewhere a bit higher than y=200 and has a reasonably stable spread. So whatever that distribution is, that's your model.

Then monitor the recent window, and treat that as a sample. Perform a goodness of fit test against the model. If it fails then the points in the window are anomalous.

UPDATE: Based on what you say about the data representing roughly a minute of time, it looks like you could use a window that's about 1 second wide. Run a hypothesis test to see if the data in the window comes from the typical distribution, and if not, just mark all the points in that window as being anomalous. If you need finer resolution than 1s, simply shrink the window.


Choropleth Map with plotly.express doesn't show map, just legend I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In addition, the 'box', 'pan', 'zoom' options show but don't work. I have the latest plotly version, installed via pip. any ideas? Thanks.

You executed the code either inside a cell of a jupyter notebook or in jupyter lab. That explains why no graph was shown.
If you would ran the same code as a pure python script it shows the graph properly.
Plotly provides two different solutions for getting it running inside jupyter:
Plotting Inline
Please search for Plotting Inline on that page to find the paragraph that explains the solution en detail.
Proper running example purePython and Jupyter Notebook
I just let it ran as it is also as purePython and as Jupyter Notebook cell:
import plotly.express as px

df = px.data.election()
geojson = px.data.election_geojson()

fig = px.choropleth(df, geojson=geojson, color="Bergeron",
                    locations="district", featureidkey="properties.district",
                    projection="mercator"
                   )
fig.update_geos(fitbounds="locations", visible=False)
fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0})
fig.show()


Why is hierarchical clustering quadratic and k-means linear? According to the internet, k-means clustering is linear in the number of data objects i.e. O(n), where n is the number of data objects. The time complexity of most of the hierarchical clustering algorithms is quadratic i.e. O(n2).

I am struggling to intuitively understand what is the difference between the two clustering approaches that causes this.

Question: What causes the difference in time complexity?

I think these time complexities are optimistic cases, but apart from that I think the reason is that in hierarchical clustering you consider the distances between many, if not all pairs of data points. The number of pairs scales quadratically with the number of points.

For k-means you somewhat cheat your way around considering all pairs by looking at the distances between each data point and the k means only. This scales linearly in both k and the number of data points.

So I think the speed up is caused by using the distance to the means as proxy for the distances between all points.


what is the main difference between GAN and autoencoder? what is the main difference between GAN and other older generative models? what were the characteristics of GAN that made it more successful than other generative models?

The main differences are the philosophy that drives the loss metric, and consequently the architecture (the latter goes without saying, obviously).

Autoencoders

The job of an autoencoder is to simultaneously learn an encoding network and decoding network. This means an input (e.g. an image) is given to the encoder, which attempts to reduce the input to a strongly compressed encoded form, which is then fed to the decoder.

The network learns this encoding/decoding because the loss metric increases with the difference between the input and output image - every iteration, the encoder gets a little bit better at finding an efficient compressed form of the input information, and the decoder gets a little bit better at reconstructing the input from the encoded form.

Summarised: An autoencoder learns to represent some input information very efficiently, and subsequently how to reconstruct the input from it's compressed form.

Generative Adversarial Networks

Here, we have a "generator" whose job is to take some noise signal and transform it to some target space (again, images is a popular example). The other component (the adversary) is the "discriminator", whose job is to distinguish real images drawn from the desired target space from the fake images created by the generator. In this case, the network is trained in two alternating phases, each with a different loss:

Firstly, the discriminator is given labelled samples from the real set of images, and fake images generated by the generator (of course, at the start, these images are just noise). Loss is calculated according to some binary classification loss metric (e.g. crossentropy). The discriminator thus begins to learn the difference between a fake image and a real image.

Before it can learn too much, though, we switch over to the generator. To train the generator, we once again feed in some noise and check the output of the discriminator. This time, we want the discriminator to detect that the image (generated by the generator) is real - this means that we have successfully fooled the discriminator, and therefore the generator has begun to learn how to make an image that resembles a "real" image from the training set.

Summarised: A GAN uses an adversarial feedback loop to learn how to generate some information that "seems real" (i.e. looks the same/sounds the same/is otherwise indistinguishable from some real data)

The difference

Hopefully that shows that there is a very fundamental structural difference between the two networks, and the goals that each of them are trying to achieve during the learning process. I don't really know if GANs are necessarily "more successful", but for certain generative problems they have been observed to give more "realistic" outputs. Why? My gut says that a GAN probably learns more about "how can I make an image look real in general" rather than "how can I memorise this particular set of images with the greatest accuracy/efficiency". But there are certainly similarities, in particular between the generator (of the GAN) and the decoder (of the autoencoder).


How to automatically mount my Google Drive to Google Colab I have recently discovered Google Colab and I am wondering if there is an option to permanently authorize Google Colab to access and mount my Google Drive.

from google.colab import drive
drive.mount('/content/drive')
Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=xxx
Enter your authorization code:
··········
Mounted at /content/drive

Right now, I need to all the time insert manually the authorization code and I want when I start a new Google Colab to have my drive mounted.

Is there an option to do that or I should execute this all the time manually?

For some reason, Colab allows me to automatically mount if I'm the creator of the notebook. If that's not the case, such was when I create a copy, the from google.colab import drive code block automatically gets added, and I have to authorize again.

File > New notebook

Then navigate to the file browser. Click on "Mount Drive".

From then on, every time I connect, it automatically mounts Drive during the Initializing phase.


Books on Reinforcement Learning I have been trying to understand reinforcement learning for quite sometime, but somehow I am not able to visualize how to write a program for reinforcement learning to solve a grid world problem. Can you suggest me some text books which would help me build a clear conception of Reinforcement Learning?

Here you have some good references on Reinforcement Learning:
Classic
Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p.
The draft for the second edition is available for free: Reinforcement Learning: An Introduction
Russell/Norvig Chapter 21:
Russell SJ, Norvig P, Davis E. Artificial intelligence: a modern approach. Upper Saddle River, NJ: Prentice Hall; 2010.
More technical
Szepesvári C. Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 2010;4(1):1–103.  Algorithms of Reinforcement Learning | Csaba Szepesvári
Bertsekas DP. Dynamic Programming and Optimal Control. 3rd edition. Belmont, Mass.: Athena Scientific; 2007. 1270 p.
Chapter 6, vol 2 is available for free: Dynamic Programming and Optimal Control
3rd Edition, Volume II | Massachusetts Institute of Technology
For more recent developments
Wiering M, van Otterlo M, editors. Reinforcement Learning. Berlin, Heidelberg: Springer Berlin Heidelberg; 2012 Available from: Reinforcement Learning | SpringerLink
Kochenderfer MJ, Amato C, Chowdhary G, How JP, Reynolds HJD, Thornton JR, et al. Decision Making Under Uncertainty: Theory and Application. 1 edition. Cambridge, Massachusetts: The MIT Press; 2015. 352 p.
Multi-agent reinforcement learning
Buşoniu L, Babuška R, Schutter BD. Multi-agent Reinforcement Learning: An Overview. In: Srinivasan D, Jain LC, editors. Innovations in Multi-Agent Systems and Applications - 1 . Springer Berlin Heidelberg; 2010 p. 183–221. Available from: Multi-agent Reinforcement Learning: An Overview
Schwartz HM. Multi-agent machine learning : a reinforcement approach. Hoboken, New Jersey: Wiley; 2014.
Videos / Courses
I would also suggest David Silver course in YouTube: RL Course by David Silver


Using sklearn DictVectorizer in real-time systems Any binary one-hot encoding is aware of only values seen in training, so features not encountered during fitting will be silently ignored. For real time, where you have millions of records in a second, and features have very high cardinality, you need to keep your hasher/mapper updated with the data.

How can we do an incremental update to the hasher (rather calculating the entire fit() every time we incounter a new feature-value pair)? What is the suggested approach here the tackle this?

If you're doing anything at the millions of records per second level, you're going to need something more high-performance than Python. Spark/PySpark is ideal for this since it interfaces with your existing Python knowledge and supports HDFS/distributed technologies that are made to take on the millions per second sized tasks.

As for sklearn: you might be out of luck here. The fit_transform() or fit() methods necessarily have to consider every record that you pass to them to keep the hash up to date, so if you have millions of data points coming in you're going to have to do millions of computations to keep moving, whether you do them one at a time or in a batch. The source code for the transform() method should convince you that doing either is equivalent.

One caveat is that if you have predictable features, e.g. maybe you're receiving counts of foo's bar's and baz's, like in the first link:

D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 1}]

Then you don't need to update the hash every time, all you have to do is transform D into the vectorized format and add it to your collection. As long as you don't need to fit new features this can be done relatively fast. However, if you're doing something less structured like NLP this won't give you hardly any useful information.


number of neurons for mnist dataset using mlp? I am trying to find out what is optimum number of neurons that can be used in MNIST dataset(60,000 training and 10,000 testing data). I build a single hidden layer model using keras,with relu activation but when I increase the neurons in hidden layer the accuracy improves slightly nearly .4% per 100 neurons ,even if I use 2000 neurons instead of 784 neurons it doesn't has any considerable change.I thought that increasing neurons above certain limit will decrease the classification accuracy because of overfitting.but It doesn't seem to look happening I even tried using 2000 neurons but the accuracy doesn't decrease on the test data.exceeding this limit I think is not possible for me because it can crash my computer.

Is hit and trial the only way to find the optimum number of neurons in a MLP hidden layer. How can I further tweak the parameters , can it be some other hyper parameters that I can play with like intialzing hidden layer weights initially.or adding more layers.(but adding more layer doesn't help much) I tried it.

If you train long enough and have "too many" hidden layer units, then you will eventually have over-training. Usually, the goal is to find the smallest number of hidden units that can do reasonably well, as that will generally give you good generalization. In fact, there is the "bottle neck theory" that attempts to show this effect (https://openreview.net/forum?id=ry_WPG-A-)

For more on comparisons between flat hidden layer networks versus convolutional networks, see:

http://conx.readthedocs.io/en/latest/MNIST.html#A-multi-layer-perceptron-network-for-MNIST-classification


How to interpret my neural network with high accuracy but low probability in test results I have built a classical ANN using keras which provides probability (using sigmoid function) of the outcomes (0 or 1). While the accuracy of the model is high when the model is fit ~90%, the outcome probability of the test set results is very poor. How can I interpret this?

Build ANN

classifier = Sequential()
classifier.add(Dense(activation="relu",input_dim=7,kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="relu",kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="sigmoid", kernel_initializer="uniform", units = 1))
classifier.compile(optimizer="adam", loss="binary_crossentropy",metrics=['accuracy'])
classifier.fit(X_train,y_train, batch_size=10,epochs=100)

Predict results:

y_pred = classifier.predict(X_test)

I have attached the test set results where result probabilities can be seen with respect to frequency. blue shows probabilities for 1 and orange for 0

Is this a case of overfitting? How can I tweak an ann to avoid overfitting?

First possibility:

How well is your dataset balanced? Are there mostly 0s? If I understand your graph well, it would seem that is the case.

If you in fact have 90% 0 and 10% 1 in your dataset, then your results would indicate the model is actually just performing random selection. This means it hasn't really learny anything about the data.

Second possibility:

If your training accuracy is high, but your test accuracy is low, this generally does point to overfitting, as you suspect.

Your model is essentially memorising the training data set, making it able to simply recall the correct result quite reliably, but without learning the underlying function that describes the dataset.

Here are a few things that might be behind this:

training for too many epochs
using a model that too big and complex
not using enough regularisation

I would recommend first checking the class balances of your dataset. If it highly imbalanced, you could look into performing stratified sampling to create your train/test split. This ensures that each batch has roughly the same number of 0 and 1 samples.

Secondly, you might consider using a slightly larger network. In the first instance, this could be as simple as increasing the number of neurons in each of your Dense layers. I would suggest something like (100, 50, 1) to begin with. That will immediately tell you if the size of the model is restricting learning.


R : Counting the number of observations per category I'm currently starting out in R and wondering how to count the number of observations per day, per node, per replicate from the below dataset, and store in a different data set.
The original dataset looks like this:

Would like the resulting dataset to look like this:

Can someone help me find out how I could do this in R?
Thanks

You can also do this with the dplyr package. The dplyr package has the functions group_by to group your data by one or more variables and summarise to do some aggregation function. The dplyr package also supports the 'pipe' notation %>%. This notation means the output of the previous function is the first argument of the next function. Here is what it might look like for one of your variables. The dplyr package is also nice in that its function do not require column names to be quoted or in character vectors.
library(dplyr)
my_summary_data <- mydata %>%
    group_by(Replicate) %>%
    summarise(Count = n())
# The last line creates a new column named Count with a value calculated by n(),
# which counts observations (rows) per group.

The output looks something like:
my_summary_data
Replicate  Count
1          8
2          7

The group_by function can group by multiple columns, so
my_summary_data <- mydata %>%
    group_by(Replicate, Node) %>%
    summarise(Count = n())

will produce:
Replicate  Node  Count
1          1     5
1          2     3
2          1     7


Unsupervised Learning::Satellite Images::Single Bands Has anyone has success with building models using KMeans for classification? I have images that only have one band and it continues to fail. My guess is that the issue is with both size of the image as well as the single band.

For example:

from osgeo import gdal,gdal_array
import numpy as np

src = '/Path/ImgA.TIF'
img_A = gdal.Open(src)

#Getting bands (count)
bands_n = img_A.RasterCount  #returns 1

band = img_A.GetRasterBand(1)

#read as array
band_arr = band.ReadAsArray()

band_sh = band.shape

#eg. output#
300,300

//This is where I am getting stuck. If I pass only two inputs (rows/cols) to KMeans, it fails as it requires a 2D array not 1D. It also fails when I manually set the band://

#Attempt 1
rows, cols = band_sh

#Attempt 2
rows, cols, band = band_sh, 1

X = band.reshape(rows*cols,band) // X = band.reshape(rows*cols,1)

from scikit.cluster import KMeans

kmeans = KMeans(n_classes = 2, random_state=2).fit(X)

Any ideas? This works just fine with RGB but fails each time when dealing with rasters that are single band.

it fails as it requires a 2D array not 1D

The X in .fit(X) is 2D, where each row is one training point and the columns are the features of that training point.

It should be a list of lists:

X = [
  [feature_1, feature_2,....feature_n], # training point 1
  [feature_1, feature_2,....feature_n]  # training point 2
  .
  .
  .
  [feature_1, feature_2,....feature_n]  # training point 123
]

The K-Means will train with all the examples in X.

However, you are only giving 1 training example, namely img_A. So your X is like this at the moment:

X = [feature_1, feature_2,....feature_n] # img_A

So, to .fit() K-Means, you need more image examples.


How to compare Timeseries Sequences? I have multiple time series sequences and I want for each new time series to find the most alike old one.

I found that I can use the sum of errors between points. Is this a good approach?
Is there a way to be able to compare sequences with different lengths (maybe a sequence look like a subsequence of another sequence)?
Will scaling the data before comparing make difference?

The answer to your questions depend a lot on the nature of the data represented in the time series. You should ask yourself some questions to better understand what might or might not work, e.g.:

Are the time sequences perfectly aligned?
Are two slightly shifted time series considered similar or not?
Are two time series with the same shape but different scale considered similar?

Normally, the answers to those questions are that series are not perfectly aligned and that variations in scale are also fine as long as the shape is similar. For these scenarios, the classical measure is Dynamic Time Warping (DTW). There are lower bounds for DTW that are very efficient computationally. The research of Professor Keogh might be interesting if you need theoretical foundation for it.

Also, normally euclidean distance and Manhattan distance are not very appropriate for time series due to their sensitivity to signal transformations (e.g. shifts), but actually they are often used in practice.


Which ANN structure to use? Let $\mathcal{S}$ be the training input data set where each input $u^i \in \mathcal{S}$  has $d$ features.
I want to design a ANN so that the cost function below is minimized (the sum of
square of pairwise differences between model outputs)  and the given constraint is satisfied, where $w$ is ANN model parameter vector.

Question: what kind of ANN is suitable for this purpose?[

A Siamese network (a network with multiple outputs) will work for such a case.


Hyperparameter tuning and cross validation I have some confusion about proper usage of cross-validation to

tune hyperparameters and
evaluate estimator performance and generalizeability.

As I understand it, this would be the process you would follow:

Split your full dataset into a training and test set (Python's train_test_split)
Use cross-validation to build a model and tune hyperparameters on the training set (GridSearchCV)
Evaluate the best estimator and assess generalizeability using cross-validation on the test set (cross_val_score)

I've looked through sklearn's cross-validation documentation, and it recommends still having a test set for final evaluation.

A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.

sklearn's grid-search information recommends:

When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process: it is recommended to split the data into a development set (to be fed to the GridSearchCV instance) and an evaluation set to compute performance metrics.
This can be done by using the train_test_split utility function.

My issue is that I often see conflicting work (for example, just cross_val_score on the entire dataset, only GridSearchCV on the entire dataset, or just a train_test_split variant), and I am hoping to understand what are best practices and clarify where I may be wrong.
Edit: This Stack Overflow answer seems to answer my question.

Some of the popular ways of splitting of data that the user can validate a model:

Train-Test (Most popular)
Train-Test-Validation
Train-Test-Development
Train-Test-Dev-Val

Every way has their own pros and cons. There is no one-size-fits-all approach for getting a perfect model. Choice is typically made by the developer considering following factors:

Size of data
Diversity of data
Computation budget
Efficiency
Necessity

But I would recommend K-fold CV is the best way to go with the basic train-test split model.

Thank you.


During a regression task, I am getting low R^2 values, but elementwise difference between test set and prediction values is huge I am doing a random forest regression on my dataset (which has abut 15 input features and 1 target feature). I am getting a decently low R^2 of <1 for both the train and test sets (please do let me know if <1 is not a good-enough R^2 score).

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

# load dataset
df = pd.read_csv('Dataset.csv')

# split into input (X) and output (Y) variables
X = df.drop(['ID_COLUMN', 'TARGET_COLUMN'], axis=1)
Y = df.TARGET_COLUMN

# Split the data into 67% for training and 33% for testing
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33)

# Fitting the regression model to the dataset
regressor = RandomForestRegressor(n_estimators = 100, random_state = 50)
regressor.fit(X_train, Y_train.ravel()) # Using ravel() to avoid getting 'DataConversionWarning' warning message

print("Predicting Values:")
y_pred = regressor.predict(X_test)

print("Getting Model Performance...")

# Get regression scores
print("R^2 train = ", regressor.score(X_train, Y_train))
print("R^2 test = ", regressor.score(X_test, Y_test))

This outputs the following:

Predicting Values:
Getting Model Performance...
R^2 train =  0.9791000275450427
R^2 test = 0.8577464692386905

Then, I checked the difference between the actual target column values in the test dataset versus the predicted values, like so:

diff = []
for i in range(len(y_pred)):
    if Y_test.values[i]!=0: # a few values were 0 which was causing the corresponding diff value to become inf
        diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i]) # element-wise percentage error

I found that the majority of the element-wise differences were between 40-60% and their mean was almost 50%!

np.mean(diff)
>>> 49.07580695857447

So, which one is correct? Is the regression score correct and my model is good for this data, or is the element-wise error I calculated correct and the model didn't do well for this data? If its the latter, please advise on how to increase the prediction accuracy.

I also checked the rmse score:

import math
rmse = math.sqrt(np.mean((np.array(Y_test) - y_pred)**2))
rmse
>>> 3.67328471827293

This seems quite high for the model to have done a good job, but please correct me if I'm wrong.

And I also checked the R^2 scores for different number of estimators:

import matplotlib.pyplot as plt
model = RandomForestRegressor(n_jobs=-1)
# Try different numbers of n_estimators
estimators = np.arange(10, 200, 10)
scores = []
for n in estimators:
    model.set_params(n_estimators=n)
    model.fit(X_train, Y_train)
    scores.append(model.score(X_test, Y_test))
plt.title("Effect of n_estimators")
plt.xlabel("n_estimator")
plt.ylabel("score")
plt.plot(estimators, scores)

Please advise.

I tried using linear regression first, and got a very high MSE (which is why I was trying out random forest):

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)

# The coefficients
print('Coefficients: \n', lr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(y_test, y_pred))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % r2_score(y_test, y_pred))

Coefficients:
 [ 1.93829229e-01 -4.68738825e-01  2.01635420e-01  6.35902010e-01
  6.57354434e-03  5.13180293e-03  2.84015810e-01 -1.31469084e-06
  1.95335035e+00]
Mean squared error: 86.92
Variance score: 0.08

This line looks wrong to me:

diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i])

Shouldn't the abs be around the entire calculation?

diff.append(100*np.abs((y_pred[i]-Y_test.values[i])/Y_test.values[i]))

That aside, the RMSE calculation looks accurate and is in the scale of the error, and the $R^2$ is great, so all things being equal, I would lean towards looking for something you did wrong in assessing the errors.  That's why I was focused on your calculation.

One other thought, have you checked for outliers?  This could affect some measures and not others as drastically.


AWS : Workflow for deep learning I am using my company computer since I don't have another one or linux. Therefore, I am starting to use cloud resources to perform some tasks.

I have a very simple question: Since most cloud resources don't have a GUI, how can I perform simple checks e.g. visualizing the bounding boxes my algorithm has found on a picture?

How do people performing such tasks usually accomplish this? Is there an easy fix or do I have to go through the installation of GUI on the cloud instance? Or, do you usually just download the results locally and view them locally?

My Simple suggestion is to install python , Anaconda on the Linux machine from command line.
As Jupyter Notebook gets installed with Anaconda package.
Just give command
jupyter notebook
Now we can connect to ipython notebook in the Virtual Machine from the token generated from above command from any browser in any other machine locally.


Extract person's academic degree from text I have a corpus of free form text (emails) and am trying to extract the highest degree (eg. High School, Bachelor, Master's, phD) from each of them. Is stemming the way to go? Or lemmatization? Note that the degree may not be mentioned at all.

Some example emails that I'll need to go through:

  Hi. My name is XXXXX. I have bachelor's in Math from UPenn and a Master's of Education from Stony Brook. Would love to work with XXXX!

  Dear XXXX, my name is XXXX and am interested in your company. Please email me for CV.

I would have started with a regexp to find all degrees in the email and then compare them to take the highest one (you can order them) ex. Masters > Bachelor. This might be a little bit too easy but worth try.


train Neural Network with SGD and see that it overfits data. Suppose you train Neural Network with SGD and see that it overfits data. Which of the following actions can help you to regularize model?

Change optimization method to Adam.

Insert (or increase rate of) Dropout layers inside NN.

Add (or increase) Weight Decay.

Reduce number of parameters (e.g. remove some layers)

Add more layers.

Of the options you listed, I would try adding a dropout layer first.

Another option you did not mention, but may, or may not be feasible, is to get more training data.  Often overfitting is a result of not having enough training data.


How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python) Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly there are not nearly as many older people, etc..).

How could I draw a sample from the old data set with respect to that shifted variable so that the distributions would mimick the new data distributions?

Let's assume you want a sample of size $N$ where variable $V$ follows its distribution in the new dataset:

Draw $N$ instances from the new dataset. Let $A= [v_1,..,v_N ]$ the list of $N$  values of $V$ corresponding to these instances.
For every distinct value $v \in A$:

Select the subset $S$ of instances in the old dataset which have $v$ as a value for $V$,
let $\#v$ be the frequency of $v$ in $A$: draw $\#v$ instances from the subset $S$

At the end of this process you have obtained $N$ instance from the old dataset which follow the distribution of $V$ in the new dataset.


What does "exaggeration" mean in the context of Boosting? I am learning boosting, the machine learning ensemble meta-algorithm. The professor is grouping 3 weak classifiers into an ensemble and said that before this time point it is easy to understand. Take a dataset, train a simple model, find the smallest error rate, something like this. This idea is easy to implement, for instance, gradient descent would take logistic regression to the smallest error rate. Then, the professor talked about the data with an exaggeration of classifier errors.

My question: What does that mean? Can anyone give a sample for this operation based on an open dataset in Python or R?

Not sure, but "exaggeration" could be an other way to talk about "overfitting".

Boosting are sequential model: each time you build a new tree, it will use the results of the precedent ones, and focus on residuals (where the precedent trees dont perform well). For this, the model exagerate the weights of the precedent mistakes.

If you build too much trees, the last ones will learn on noise, because
residuals will contains much noise than information. You will have an excellent error rate ("exaggerated error rates") for your train dataset, but if you apply your model on a new dataset, the error will be lower.


Practical way to convert jupyter notebook to MS Word document? What would be a practical way to convert a Jupyter Notebook to a Word document (.doc) ? I am asking this in a professional context, so I'd like to avoid manual solutions, do it in an efficient way (fast), avoid third parties... etc. Something that works like Rmarkdown to produce .doc would be very welcome.

The easiest way is probably using a method similarly to what is described in this answer, that is, convert the notebook to markdown and then use any of the tools available (such as Pandoc) to convert the markdown to a Word document.


Sliding window leads to overfitting in LSTM? Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs?
For a simplified example, assume that we have to predict the sequence of characters:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Is it bad (or better?) if I keep training my LSTM with the following minibatches:
A B C D E F G H I J K L M N, backprop, erase the cell

B C D E F G H I J K L M N O, backprop, erase the cell

 .... and so on, shifting by 1 every time?

Previously, I always trained it as:
A B C D E F G H I J K L M N,  backprop, erase the cell

O P Q R S T U V W X Y Z,  backprop, erase the cell

Instead of shifting by one, would it be better to slide the window by 2 entries instead, etc? What would that mean (in terms of precision/overfitting)?

Also, if I were to do the sliding-window approach in a Feed-forward network, would it result in overfitting? I would assume yes, because the network is exposed to the same information regions for a very long time. For example, it is exposed to E F G H I J K for a long time.

Edit:
Please remember that cell state is erased between training batches, so the LSTM will have a "hammer to head" at these times. It's unable to remember what was before OPQRSTUVWXYZ. This means that the LSTM is unable to ever learn that "O" follows the "M".
So, I thought (thus my entire question), why not to give it intermediate (overlapping) batch in between...and in that case why not use multiple overlapping minibatches - to me this would provide a smoother training? Ultimately, that would mean a sliding window for an LSTM.

Some useful info I've found after answer was accepted:
from here

The first word of the English translation is probably highly correlated with the first word of the source sentence. But that means decoder has to consider information from 50 steps ago, and that information needs to be somehow encoded in the vector. Recurrent Neural Networks are known to have problems dealing with such long-range dependencies. In theory, architectures like LSTMs should be able to deal with this, but in practice long-range dependencies are still problematic.
For example, researchers have found that reversing the source sequence
(feeding it backwards into the encoder) produces significantly better
results because it shortens the path from the decoder to the relevant
parts of the encoder. Similarly, feeding an input sequence twice also
seems to help a network to better memorize things. For example, if one training example is "John went home", you would give "John went home John went home" to the network as one input.

Edit after accepting the answer:
Several months after, I am more inclined to use the sliding window approach, as it uses the data better. But in that case you probably don't want to train BCDEFGHIJKLMNO right after ABCDEFGHIJKLMNO.  Instead, shuffle your examples, to gradually and uniformly "brush-in" all of the information into your LSTM.  Give it HIJKLMNOPQRSTU after ABCDEFGHIJKLMNO etc.  That's directly related to Catastrophic Forgetting. As always, monitor the Validation and Test set closely, and stop as soon as you see their errors steadily increasing
Also, the "hammer to head" issue can be improved, by using Synthetic Gradients. See its benefit here: (linked answer discusses its benefit of long sequences) https://datascience.stackexchange.com/a/32425/43077

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for framing forecasting as a supervised learning problem.

If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of parameters in an LSTM (in fact, there are $4(mn + n^2 + n)$ parameters, where $m$ is the input length and $n$ is the output length, according to this post).

Since LSTMs do not require fixed size input, they can find the optimal lookback number on their own. However, if you've done a prior autoregressive analysis and decided that, for example, the current time step is most correlated with the 10th previous time step, and not correlated with the 11th or any time steps further in the past, then you could perhaps save yourself some training time by feeding in fixed-length sequences. However, that kind of defeats the purpose of an LSTM.

If your data are not rich enough for an LSTM, I would recommend trying something much simpler, like an autoregressive model, and working your way up.

EDIT (responding to a comment):

Overlapping sequences are used as input, especially when the sequence is very long (although, of course, 'long' is relative). Although LSTMs are better than a vanilla RNN for long sequences, they can still have some trouble remembering time steps from the beginning of a sequence if the sequence is very long. That led to things like the bidirectional LSTM, which reads the sequence forwards and backwards, improving the exposure of the network to the beginning and end of each input sequence. The principle is the same with overlapping sequences, although I would argue that overlapping sequences is more intuitive.


Clusering based on categorical variables? I am working on a project and currently experimenting cluster analysis. The dataset is mainly categorical variables and discrete numbers. Please pardon my poor programming skills as I am not very familiar with MathJax, but I will try to summarize the data set in words in the following three examples.

imagine column 1 is participants names of course, from column2 - 5, each column's value range from 1 (least important) - 5 (most important). so in this case, column 2-8 only have discrete data.
column 6 for example, since this is a multiple choice question. row 1 chose "nice" as an answer, however, row 2 chose "poor". In this case, we have one column that contains multiple categorical answers.
for column 7-9, this is another type of multiple choice questions. However this time, each column represents only one answer. column 7 only allows string values "true", column 8 only allows string "somewhat". So in this case, we have multiple columns that represent multiple answers of the same question.

Any ideas how to solve this? Thankful to any input!

A2       A3       A4       A5       A6       A7         A8         A9
1        4        5        4        nice     true       somewhate  false
2        4        3        1        poor     true                  false
1        5        2        1        nice                somewhate
3        2        1        5        nice     true                  false

I assume your data set is sth like

A0 A1 A2 B0 B1 B2
1  0  0  1  1  0
0  1  0  0  0  1
0  1  0  0  1  1
      ...

where A0 refer to the answer of question A, choice 0 and question A is a single choice question,
B is a question allowing multiple answers.
Each row represents one record.

For this data format, each column can be considered as a dimension (if some columns can be quantified to a value, you may group them into one column, ex "Like = 1, Neutral = 0, Dislike = -1") Then applying clustering algorithms, such as K-mean to cluster it.

Please tell if your data set format is different from my assumption.

Updated on Jun 30:

The idea is to quantify the variables,
For column 2-5, the values are already quantified to [1, 5]
For column 5, it allows ("poor"/ "nice"). Although they are strings, they are representing the preference of different magnitude. They can also be converted to "poor" = 0, "nice" = 1.
For column 7-9, because it is a binary option, they can also be easily converted to ex. "true" = 0, "" = 1.

Then, you will have a matrix:

1    4    5    4    1    1    1    1
2    4    3    1    0    1    0    1
1    5    2    1    1    0    1    1
3    2    1    5    1    1    0    1

Remark1: For binary options, such as column 9, only allows "False" or "null", choosing "False" = 1 or 0 does not matter. In classification, only the distance matters.

Remark2: If you have a column allows different strings, such as "apple", "banana", "orange", you can convert it to an one hot vector.
Ex.

A
apple
orange
orange
banana

can be converted to

A0   A1   A2
1    0    0
0    0    1
0    0    1
0    1    0

where A0, A1, A2 represent "apple", "banana" and "orange".

Remark3: if there is a column allow storing multi answers, ex.

A
apple
apple orange
orange
orange banana

can be converted to
    A0   A1   A2
    1    0    0
    1    0    1
    0    0    1
    0    1    1

Remark4: before applying some classification algorithms such as K-mean, it is better to normalize the magnitude of each dimension. For example, value of column 9 is [0, 1] can be normalized by times 5 to [0, 5]. The factor/ the range of the dimension reflects the importance of that factor.

I hope my answer can help.


Why is my MLP with 2 features is doing worse than MLP with 1 feature where the one feature is a combination of feature1*feature2? I have programmed a MLP for a dataset (~500 rows) containing the length (L) and width (W) of an organism and the output of biomass (the organisms weight in pounds, B).

            mlp = MLPRegressor((5, 5), max_iter=1000)

I have trained the model with features

# Model 1
# Input = Feature 1: Length, Feature 2: Width. Output = Biomass
df = {'length': [60.1, 59.2, 59.4, 58.5], 'width': [15.4, 16.2, 14.9, 15.7], 'weight': [8.34, 7,65, 7.89, 7.14]}

# Model 2
# Input =  Feature 1: Length * Width^2. Output = Biomass
df = {'length*height^2': [60.1, 59.2, 59.4, 58.5], 'weight': [14253.31, 15536.44, 13187.39, 14419.66]}

The overall accuracy of my model with one feature is over 95%, however the accuracy with the features separated is about 85%.

My understanding of an MLP is that Model 1 should do better than model 2 as it will basically find the best combination of length and height to biomass, However my 1 feature model is doing significantly better. I have also tried standardizing the dataset with a scaler with no luck.

scaler = StandardScaler()

$h$ - height
$w$ - width
$p$ - weight

In real world we know that weight is volume times density. And volume is product of height, width and depth. If you assume constant depth and density the weight is just some constant multiplied by width and height i.e. $p = c \times hw$. If you have $hw$ as a feature the NN quickly learns $c$. This gives good performance.

Lets say your NN had identity as activation function i.e. no activation function. All it can learn is linear combination of input features.

So it will just learn $p \approx h w_h + w w_w + b$ where $w_h, w_w$ are weights and $b$ is bias.

If you have activation function like ReLU/Sigmoid it still learn linear combinations of outputs of ReLU/Sigmoid. Inputs to those is just linear combination of $h$ and $w$. In all that there is never a simple $hw$ term. This makes model complicated without having the important feature that is volume(area).

I think that is why the performance of $hw$ model is better.

If you have a NN with 2 neurons in the hidden layer with sigmoid activation, and 1 output neuron with no activation, then this is how $hw$ term shows up.

$\frac{1}{1+\exp(hw_{11} +w w_{12} + b_1)} + \frac{1}{1+exp(hw_{21} +w w_{22} + b_2)} = \frac{\ldots}{\ldots + \exp(w_{11} w_{22} hw + w_{12}w_{21}hw) } $

Thats very tricky place to show up if you just want to learn a constant multiplier.


Multiclass classification of textual data I have a problem statement in which I have to classify the text data into various classes, but the training data is very less (250-300 data points for 4 classes). I am confused about what approach to use? Also is there any way that deep learning could be used for this problem with such a small tarining data?

300 data points is enough to give good/average results with the traditional machine learning algorithms. You should test SVMs as they are considered good for text classification.

However, deep learning is very often used when you have a lot of data, which is not the case here. So I would not recommend it for your task.


SMOTE for multilabel classification I have a dataset with 77 different labels. Each sample has one or more of these labels.

I did some data analysis and found out that the dataset is highly imbalanced - there are a large number of examples that have a particular label, whereas the other labels don't occur so frequently across the data samples.

I'm trying to use SMOTE to synthesize new data samples for the minority labels but apparently, imblearn's SMOTE doesn't support multi-label data. Is there an alternative to SMOTE that I can use for multilabel classification, or should I treat my problem as 77 different binary classification problems, and apply SMOTE on each iteration separately?

If you have only one example for certain classes SMOTE won't work. Most of the Machine Learning algorithms won't work either.

There is a technique called One Shot Learning (it is normally used in computer vision) that  "Whereas most machine learning-based object categorization algorithms require training on hundreds or thousands of samples/images and very large datasets, one-shot learning aims to learn information about object categories from one, or only a few, training samples/images."

Maybe you could try with one OSL to help you with the classification but normal ML algorithms require more samples to be able to generalize.


What is the difference between the training and testdata set? What is the difference between the test and training data sets? As per blogs and papers I studied, what I understood is that we will have 100% data set that is divided into 2 sets (test data set is 30% and reaming 70% is training data set). I would like to know more points and use of differentiating the 100% data set to test and training data sets.

In Machine Learning, we basically try to create a model to predict on the test data. So, we use the training data to fit the model and testing data to test it. The models generated are to predict the results unknown which is named as the test set. As you pointed out, the dataset is divided into train and test set in order to check accuracies, precisions by training and testing it on it.

The proportion to be divided is completely up to you and the task you face. It is not essential that 70% of the data has to be for training and rest for testing. It completely depends on the dataset being used and the task to be accomplished. For example, a simple dataset like reuters. So, assume that we trained it on 50% data and tested it on rest 50%, the precision will be different from training it on 90% or so. This is mostly because in Machine Learning, the bigger the dataset to train is better. You can refer this paper, which tells the precision values based on the dataset size. It now depends on you, what precision or accuracy you need to achieve based on your task.

This said so, how would you predict the results for which you do not have the answer? (The model is ultimately being trained to predict results for which we do not have the answer). I would like to add on about validation dataset here.

Sets:

Training Set: Here, you have the complete training dataset. You can extract features and train to fit a model and so on.
Validation Set: This is crucial to choose the right parameters for your estimator. We can divide the training set into train set and validation set. Based on the validation test results, the model can be trained(for instance, changing parameters, classifiers). This will help us get the most optimized model.
Testing Set: Here, once the model is obtained, you can predict using the model obtained on the training set.

Refer this for more info.


Validation accuracy greater than training accuracy in cnn I've splitted my training set in the ratio 80:20 and have developed cnn model with a dropout of 0.5. I'm getting an accuracy of 98%. But the validation accuracy stays greater than training accuracy. Is there any problem with this?Does it leads to over fitting? If yes, then why am I getting an accuracy of 98%?The graph is given below.The red line is the validation accuracy and blue is training accuracy.

As @Suren has eluded to, it could be the case that your validation accuracy is higher than your training accuracy due to the distribution of the classes over the training and validation sets. If the training set contains a higher proportion of a particular class and the validation class contains examples of that particular class as well, then of course you will see validation accuracy being high.

The advice would be to balance out the classes over the training and validation and sets. You can do this data augmentation, in that you artificially increase the number of examples based not he data you already have.
Have a look at this paper which evaluates the effect of different methods of class balancing using data augmentation on generalisation performance of CNNs:
https://arxiv.org/pdf/1710.05381.pdf


Why does my neural net always converge on a trivial solution? As part of a more elaborate project, I am trying to get a rather simple network to distinguish between 5 very distinct classes of 1D signals (the members of each class is just the same bit of signal copied over 1000 times with a slight bit of noise added). And failing.

Below is the network setup, and the output. As far as I can tell (through matplotlib), I haven't done anything really silly with data (such as flip the dimensions, for instance). Does anyone have a suggestion for what could be going wrong?

I apologize for not generating dummy data within the script, I'm still much more comfortable in matlab, so it was easier to just port some dummy data from there.

model = Sequential()

model.add(Flatten(input_shape=(6000,1),name='S2'))
model.add(Dense(40,activation='relu',name='F1'))
model.add(Dense(10,activation='relu',name='F2'))
model.add(Dense(5,activation='relu',name='output'))
model.add(Activation(activation='softmax',name='softmax'))

model.summary()

# load fake data
earEEG=scipy.io.loadmat('forPython_dummy.mat')
data=earEEG['data'][:,:]
data=np.transpose(data)
data=data[:,:,None]

labels=earEEG['labels']
labels= keras.utils.to_categorical(labels)

sgd=keras.optimizers.SGD()
model.compile(loss='categorical_crossentropy',
              optimizer=sgd,
              metrics=['accuracy'])

hist=model.fit(data, labels, epochs=10, batch_size=50,class_weight='auto',shuffle='True')

And output:

Layer (type)                 Output Shape              Param #
=================================================================
S2 (Flatten)                 (None, 6000)              0
_________________________________________________________________
F1 (Dense)                   (None, 40)                240040
_________________________________________________________________
F2 (Dense)                   (None, 10)                410
_________________________________________________________________
output (Dense)               (None, 5)                 55
_________________________________________________________________
softmax (Activation)         (None, 5)                 0
=================================================================
Total params: 240,505.0
Trainable params: 240,505.0
Non-trainable params: 0.0
_________________________________________________________________
Epoch 1/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 2/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 3/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 4/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 5/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 6/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000

Clearly, the algorithm is just giving the same label to everything (this is also apparent when outputting the actual labels). The input is a 5000 x 6000 matrix, and 5000 labels. As the algorithm does not complain about dimensional mismatch, I feel pretty confident that the data is not being transposed or similar. As an easy test, I tried feeding the same dataset to a pca+decision tree in matlab, and got a perfect score.

Classes are perfectly balanced, by the way.

The problem turned out to be that the input had not been properly scaled - making sure all epochs had standard deviation = 1 solved it.