What are the best ways to tune multiple parameters? When building a model in Machine Learning, it's more than common to have several "parameters" (I'm thinking of real parameter like the step of gradient descent, or things like features) to tune. We validate these parameters on a validating set.

My question is: what is the best way of tuning these multiple parameters? For example, let say we have 3 parameters A, B and C that take 3 values each:

A = [ A1, A2, A3 ]
B = [ B1, B2, B3 ]
C = [ C1, C2, C3 ]

Two methods come to my mind.

Method 1:

Vary all the parameters at the same time and test different combinations randomly, such as:

Test1 = [A1,B1,C1]
Test2 = [A2,B2,C2]
Test3 = [A3,B3,C3]
Test4 = [A1,B2,C3]
Test5= [A3,B1,C2]
etc..

Method 2:

Fix all the parameters except one:
 - TestA1 = [A1,B1,C1]
 - TestA2 = [A2,B1,C1]
 - TestA3 = [A3,B1,C1]
In that way, we can find the best value for parameter A, then we fix this value and use it to find the best value for B, and finally the best for C.

It seems more logical to me to use the method 2 which seems more organized. But may be we will miss a combination which can be found only in method 1 that doesn’t appear in method 2, such as [A1,B2,C3] for example.

Which method is the best? Is there another method more accurate for tuning multiple parameters?

Thanks in advance.

Regards.

Generally people perform a grid search, which in its simplest "exhaustive" form is similar to Method 1.  However there are also more 'intelligent' ways to choose what to explore, which optimize in parameter space in a fashion similar to how each individual model is optimized. It can be tricky to do greedy optimization in this space, as it is often strongly non-convex.

This page describes the basics of optimizing model parameters.


How should I configure my Neural Network to accept a column of numpy.ndarrays as input? I have a dataframe with 10k rows and the following columns:
      array                                     target
[1,5,6,1,3,etc...]                                5
[3,3,1,0,5,etc...]                                10
[0,0,1,1,7,etc...]                                3
        .                                         .
        .                                         .
        .                                         .

Each array has 33222 elements, so I have 10,000 rows each with a 33,222 long numpy.array that I want to input into a Neural Network to predict the target variable.
Here is how the NN is configured:
x = df.loc[:, 'array']
y = df.loc[:, 'target']

model = Sequential()
model.add(Dense(12, activation='linear'))
model.add(Dense(1, activation='linear'))

model.compile(loss='mse',
                optimizer='adam',
                metrics=['accuracy', 'mse', 'mae'])

model.fit(x, y, epochs=10, batch_size=1, verbose=1)

I'm getting
"ValueError: Please provide as model inputs either a single array or a list of arrays"

I haven't attempted to use arrays as input for a NN before, so I would also appreciate any advice on the optimal layer choice and configuration for this kind of problem.

You should set the input_shape parameter in the first dense layer. Like this:
model = Sequential()
model.add(Dense(12, activation='linear', input_shape=(33222,)))
model.add(Dense(1, activation='linear'))


Categorical Variable and Target Variable Though a similar question is answered here , but I wanted to take a different approach. Assuming that I have a binary target variable 1/0 and a categorical variable Gender M/F. From this, I can have a proportion p1 for M with target 1 and p2 for F with target 1. N1 and N2 would be total number of M and F respectively.
Is it prudent to run a test for H0: p1=p2 ? If H0 cannot be rejected, doesn't it indicate that Gender doesn't have any correlation with Target

Reading from your comments, it appears you can use more that one predictor for your target variable.

If you want to understand whether Gender (M/F) has a significant association, you should run a logistic regression using it together with other predictors. This would let you control for the impact of Gender all things equal, i.e. after you controlled for the effects of other variables.
Logistic regression will return significant scores and standard errors for each variables.
Alternatively, you could look at relative importance scores from tree-based models such as Random Forest or XGBoost. (However, Logistic regression is better for hypothesis testing.)

BUT: If you still want to stick with bivariate statistics, i.e. using just one predictor at a time, you can use simpler tools such as T-test or one-way ANOVA to understand whether belonging to one gender or the other makes a difference (statistically speaking).

Hope this helps, otherwise let me know.


How to extract Question/s from document with NLTK? How to extract Only Question/s from document with NLTK ?

Can we categorise this Question into Y/N and details type answerable ?

Note: I am one week old in NLTK ;-)

Check out chapter 6 section 2.2 of the NLTK book.

EDIT: since apparently the community wants me to copy/paste stuff, here ya go:

2.2   Identifying Dialogue Act Types

When processing dialogue, it can be useful to think of utterances as a type of action performed by the speaker. This interpretation is most straightforward for performative statements such as "I forgive you" or "I bet you can't climb that hill." But greetings, questions, answers, assertions, and clarifications can all be thought of as types of speech-based actions. Recognizing the dialogue acts underlying the utterances in a dialogue can be an important first step in understanding the conversation.

The NPS Chat Corpus, which was demonstrated in 1, consists of over 10,000 posts from instant messaging sessions. These posts have all been labeled with one of 15 dialogue act types, such as "Statement," "Emotion," "ynQuestion", and "Continuer." We can therefore use this data to build a classifier that can identify the dialogue act types for new instant messaging posts. The first step is to extract the basic messaging data. We will call  xml_posts() to get a data structure representing the XML annotation for each post:

>>> posts = nltk.corpus.nps_chat.xml_posts()[:10000]

Next, we'll define a simple feature extractor that checks what words the post contains:

>>> def dialogue_act_features(post):
...     features = {}
...     for word in nltk.word_tokenize(post):
...         features['contains({})'.format(word.lower())] = True
...     return features

Finally, we construct the training and testing data by applying the feature extractor to each post (using  post.get('class') to get a post's dialogue act type), and create a new classifier:

>>> featuresets = [(dialogue_act_features(post.text), post.get('class'))
...                for post in posts]
>>> size = int(len(featuresets) * 0.1)
>>> train_set, test_set = featuresets[size:], featuresets[:size]
>>> classifier = nltk.NaiveBayesClassifier.train(train_set)
>>> print(nltk.classify.accuracy(classifier, test_set))
0.67


Can Adagrad be used to optimize non-differentiable functions? I am reading a book (TensorFlow For Dummies, Matthew Scarpino), and here it says:

  Adagrad methods compute subgradients instead of gradients. A subgradient is a generalization of a gradient that applies to nondifferentiable functions. This means AdaGrad methods can optimize both differentiable and nondifferentiable functions.

But I haven’t seen such a claim anywhere else. So I want to know:

Can we really optimize non-differentiable loss functions just because we are using Adagrad?

The book has a misunderstanding (but it's understandable where it came from).

If you can compute subgradients, you can use gradient descent.  You don't have to use AdaGrad -- you can use any gradient method you like.  Basically, you just use a subgradient in place of the gradient in the update step.  See, e.g., https://en.wikipedia.org/wiki/Subgradient_method.

AdaGrad is an adjustment to gradient descent that adjusts the update step.  I believe AdaGrad is orthogonal to whether you use subgradients or gradients in the update step.

I can see where the misunderstanding comes from.  The original paper on AdaGrad talks about subgradient methods.  However, if you read the introduction carefully, you will recognize what's going on there.  Subgradient methods are a broader category, that includes both ordinary gradient descent and subgradient descent.  So, the paper is just trying to be as general as possible.  Their method applies both to ordinary gradient descent and also to subgradient descent.


Why do we need convolutions over volume in convolutional neural networks for image recognition? In convolutional neural networks, we make convolutions of three channels (red, green, blue) with a filter of dimensions $k\times k\times 3$, like in the picture:

Each filter consists of adjustable weights, and can learn to detect primitive features, like edges. The filter can be different for each channel: one for R, another for G, yet another for B.

My question is: Why do we need separate filters for each channel?

If there's an edge, it will appear on each channel at the same place. The only case when this is not true is if we have lines that are purely red, green or blue which is possible only in synthetic images but not in real images.

For real images these three filters will contain the same information which means that we have three times more weights, so we need more data to train the network.

Is this construction redundant or I am missing something?

Isn't it more practical to detect features on a gray-scale image where we just need a 2-dimensional convolution? I mean, somewhere in the pre-processing step, the image can be separated into a gray-scale part and some other construction containing only a color information. Then each part is processed by a different network. Will this be more efficient? Do such networks exist already?

I'm not sure if the information in the colours is as redundant for real images as you say. I could imagine a scenario where the colours around an edge are relevant, e.g. training a network to tell brown bears from polar bears. In terms of edges and shapes they are very similar, the colour however is not. So while the edges show up in all channels, I would guess that they show up with different contrast, thus propagating colour-information into the classifier.

If you check out this blog where the author visualized some of the learned features, it seems that some features are fairly similar in terms of shape, but not in colour.

There might be a work-around, as you describe it, where you separate shapes and colours, but I think that would involve a lot of prior knowledge and it might be easier to just let the network figure this out during training. This is true even more so if you go to multi-spectral images as they come from EO-satellites for example. There you definitely have features popping up only in certain channels, but not so much in others.


Why isn't the neural network updated after every example in the dataset Why the neural network is updating only after the whole batch passes?

Weights get updated based on the number of examples you feed in a batch. This is because, a full pass(forward and backward) of matrix computations has to be completed in order for the weights to be updated, after back-propagation and proceeding with next epoch, with batch type you had chose.
Moreover, If you use stochastic gradient descent, where each example will be processed at a time, your weights will updated after having processed every example.


How would knowing spammers email address improve spam detection algorithms? I am new into Artificial Intelligence field and I am working on the classic example of Spam detection using classification. I am using Naive Bayes algorithm as well as SVM.

While working on them it came to my mind that having the email addresses of who has sent the emails as an extra feature would improve the accuracy of my algoithm as we will have some extra info besides the content of the email itself. Would it be true? And how could I use this extra feature? Should I just add it to my email content?

Please accept my apology as it is not really a data science answer.

A lot of spammers are using spoofed emails address as sender, so that field alone might not be very helpful as a dimension. The number of possible values also could make it impractical to be used (i.e., 1 hot-encoding) as a feature unless you preprocess it somehow. Sender domain, sender server reputation scores, etc. would probably be more helpful than the email address by itself.

How spammers spoof your email


Small data set in machine learning I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business.

I have a data set that will be around 100k units, how to I determine the best predictive or factor model to use when approaching this small'ish data set? I want to avoid overfitting etc.

Any common pitfalls that I should avoid and/or tricks I should use to get more out of this data?

In general, Machine Learning algorithms handle volumes data. This doesn't mean that you cannot extract information from "small" data.

Keep in mind:

Overfitting. With only a few data, the risk to overfit your model is far higher.
Outliers. Those become nastier. If you have millions of data, a couple of outliers will not be a problem. But with only a few, they will definitely skew your results.

Models to use

To avoid overfitting, you need to avoid complexity. As a result, you need a model that has as less of parameters as you can. For example, linear/logistic regression could be what you need. Naive Bayes and SVM might work well, but this is not a generalization.

Cleaning up your data

Together with the outliers, you need to be careful with the rest of the cleaning process. Noise can skew a lot your data.

This is a nice blog post about "small data" and what you can do when you are working with them. Highly recommended.

What to do with “small” data? (7 min read)

About Outliers

I recommend the answer from this Data Science Stack Exchange question. Short story. You need to examine the outliers one by one. There isn't any rule written in stone. But you need to identify if those outliers are just wrong input data or data that makes sense to be there, even as an outlier. For example, if you have the sales of an eshop per day and in your data you have a specific day where the sales are 10x more, you need to understand if that was an error or something happened that day to increase the sales (maybe a discount campaign or email advertisement).

Also, it depends on what kind of problem you have. If your problem is an anomaly detection, then you don't want to remove the outliers.

Another suggestion (I don't have any source to link) is to update the values of those outliers with the mean, mode or another metric from your data. If you have only a few rows of them, removing a dozen will make it more difficult.

In general, this should be part of your model process. You try once with outliers, a few times without or with updated values and try to understand if it is underperformed or overfitted.


Error with negative bins while melting dataframe I have a dataframe 'd' like this:

   breaks counts.x counts.y
1     -20        1        1
2     -15        0        1
3     -10        0        5
4      -5        4       18
5       0       13       27
6       5       18       25
7      10        9       12
8      15        2        1
9      20        1        7
10     25       NA        0
11     30       NA        1

When I try to melt using 'breaks'  as id variable it is giving me the following error:

 d=melt(d,id=breaks)
Error in varnames[id.vars] :
  only 0's may be mixed with negative subscripts

How can I resolve this?
I have to draw a ggplot using breaks on x axis and counts on y axis.

I am sorry, is this the kind of ggplot you want to obtain ?

If yes this is the code behind it:

library(reshape)
library(ggplot2)

d <- data.frame(breaks = c(-20, -15, -10, -5,  0, 5, 10, 15, 20, 25, 30),
                counts.x  = c(1, 0, 0 ,4 , 13, 18, 9, 2, 1, NA, NA),
                counts.y = c(1, 1, 5, 18, 27, 25, 12, 1, 7 , 0, 1))

d <- melt(d, id = "breaks")

ggplot(d, aes(x = breaks, y = value)) + geom_point(aes(colour = variable)) +
       labs(title = "Bins vs. Counts", x = "Bins", y = "Counts") +
       theme(plot.title = element_text(face = "bold"))


Facebook Prophet add Regressor I've been searching for a long time to answer my question, but I haven't found anything. So I hope you can help.
I'm searching for an opportunity to add a regressor to my prophet model in python. I know the 'model.add_regressor' method but they don't solve my problem. Because I will add data I don't know in the future, rather only in the past, as a regressor.
I want to predict 6 time steps (monthly data, so 6 months into the future).
So, know anybody an opportunity to solve this?

Perhaps your repressor is for one time step behind. For e.g. if you are predicting tomorrow's stock price, then the repressor say market volume can be from today.
The other thing you can do is build a separate model that estimates the regressor for the next day and then use that in prophet. Be aware that your regressor is then becomes imperfect since it represents an estimate and not the true value. Depending on your use case, this may or may mot be a problem.


Tweet Classification into topics- What to do with data Good evening,
First of all, I want to apologize if the title is misleading.
I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into topics. I am working on topic modelling with LDA getting the right number of topics (I guess) thanks to this R package, which calculates the value of three metrics("CaoJuan2009", "Arun2010", "Deveaud2014"). Since I am very new to this, I just thought about a few questions that might be obvious for some of you, but I can't find online.

I have removed, before cleaning the data (removing mentions, stopwords, weird characters, numbers etc), all duplicate instances (having all three columns in common), in order to avoid them influencing the results of topic modelling. Is this right?

Should I, for the same reason mentioned before, remove also all retweets?

Until now, I thought about classifing using the "per-document-per-topic" probability. If I get rid of so many instances, do I have to classify them based on the "per-word-per-topic" probability?

Do I have to divide the dataset into testing and training? I thought that is a thing only in supervised training, since I cannot really use the testing dataset to measure quality of classification.

Antoher goal would be to classify twitterers based the topic they most are passionate about. Do you have any idea about how to implement this?

Thank you all very much in advance.

As far as I'm aware there is no correct/standard way to apply topic modelling, most decisions depend on the specifics of the case. So below I just give my opinion about these points:

I have removed, before cleaning the data (removing mentions, stopwords, weird characters, numbers etc), all duplicate instances (having all three columns in common), in order to avoid them influencing the results of topic modelling. Is this right?
Should I, for the same reason mentioned before, remove also all retweets?

In general there is no strict need to deduplicate the data, doing it or not would depend on the goal. Duplicate documents would affect the proportion of the words which appear in these documents, and in turn the probability of the topic these documents are assigned to. If you want the model to integrate the notion of popularity/prominence of tweets/words/topics, it would probably make sense not to deduplicate and keep retweets. However if there is large amount of duplicates/retweets the imbalance might cause less frequent tweets/words to be less visible, possibly causing less diverse topics (the smallest topics might get merged together for instance).

Until now, I thought about classifing using the "per-document-per-topic" probability. If I get rid of so many instances, do I have to classify them based on the "per-word-per-topic" probability?

I'm not sure what is called the "per-document-per-topic" probability in this package. The typical way to use LDA in order to cluster the documents is to use the posterior probability of topic given document (this might be the same thing, I'm not sure): for any document $d$, the model can provide the conditional probability of every topic $t$ given $d$. The sum of this value across topics sums to 1 (it's a distribution over topics for $d$), and for classification purposes one can just select the topic which has the highest probability given $d$.

Do I have to divide the dataset into testing and training? I thought that is a thing only in supervised training, since I cannot really use the testing dataset to measure quality of classification.

You're right, you don't need to split into training and test set since this is unsupervised learning.

Antoher goal would be to classify twitterers based the topic they most are passionate about. Do you have any idea about how to implement this?

The model gives you the posterior probability distribution over topics for every tweet. From these values I think you can obtain a similar distribution over topics for every tweeter, simply by marginalizing over the tweets by this author $a$: if I'm not mistaken, this probability $p(t|a)$ can be obtained simply by calculating the mean of $p(t|d)$ across all the documents/tweets $d$ by author $a$.


I need some guidance how to develop myself in deep learning Few months ago I've taken an Andrew Ng course at coursera and started my data science/machine learning journey.

During those months, I've take another courses from the internet - about Deep Learning, statistics - read articles from towardsdatascience website, now I'm reading a book "Hands on Machine Learning".

But I feel I'm stuck, I have over a dozen pages of courses at coursera, few books, and I don't know what to do next.
I feel like I'm doing the same things all the time.

I'll be very thankful for some guidance :)

I am not an expert in the field, but I'd like to share with you my personal opinion. Taking online courses and reading books is great, but the problem is you don't get to practice much on your own. I know there might be assignments and projects included, but they're mostly guided and pretty straightforward, or related to a certain topic of the chapter/lecture.

However, real world problems are a bit different. They usually encompass several smaller sub-problems that might not be clear at the beginning, and there are a plenty of choices that you have to make on your own. Usually you'd have to try several solutions before you're confident what suits your problem best. I believe it's something you learn mostly by practice. So I think given that you have enough knowledge now, it's about time to use your knowledge and build interesting stuff, stuff that would speak for you. Kaggle is also a great way to move forward, since there are tons of competitions and data sets that could inspire you, and help you know where you stand among others. With every project you try to do, you discover something new or learn to appreciate something you already know more. Plus you can always learn from others who publish their kernels.

Whatever you choose to do, I wish you all the best :)


Predicting next adverse event based on previous events I have a clinical trial dataset where the patient details are recorded at certain time intervals(visits) over period of time. Each visit will contain data recordings of all the adverse events experienced prior to the visit, any medicines taken and the dosage of experimental drug in no particular order. for example a visit could have recordings of 5 adverse events (nausea, headache, hypertension etc.) experienced in the week prior to the visit, dosage of experimental drug and some medicines given to avoid the adverse events.

Now i need to predict the occurrence of adverse events for a patient before his next visit and also need to know which medicines cause some of these adverse effects.

I hope I've clearly explained the problem. What Machine learning/Statistical methods would best solve this?

What you're describing is known as a multilabel classification. You want to predict some output labels (adverse effects) in a given set of possible output labels  (all possible adverse effects), using a given a set of input indicators (previous adverse effects, drugs prescribed, dosage levels).

For those final two features, I'd combine them into one feature where if the drug is not prescribed obviously dosage is 0, and higher otherwise.

In R, you can use the mlR package for this prediction stage. You may also be able to use the generateFilterValues() function in this package to extract feature importances, however I'm not sure if this will work with multilabel and additionally this will only tell you how much your classifier believes your input features contribute to the output labels relative to all the other input features. It won't give you other valuable information for example at which dosage levels of a particular drug does the probability of negative symptoms increase most.

Again, I'm unsure how this would work with multilabel classifications as I don't have any experience with them, but take a look at plotting partial dependency plots to get this last bit of information on how each input feature contributes to your output labels.


Why does TensorFlow convert my decoded image to float32 instead of uint8/16? I am using the following code to import a bunch of .png images and decode them using TensorFlow:

from __future__ import absolute_import, division, print_function
import tensorflow as tf
import numpy as np
import os

tf.enable_eager_execution()

NUM_TRAINING_SAMPLES = 333
NUM_CLASSES = 3
BATCH_SIZE = 5
NUM_EPOCHS = 6
INPUT_SIZE = (256, 256, 3)

random_indices = np.random.choice(range(13000), NUM_TRAINING_SAMPLES)
directory = "/home/local/CYCLOMEDIA001/ebos/Downloads/SYNTHIA_RAND_CVPR16"
directory_images = "/home/Downloads/SYNTHIA_RAND_CVPR16/RGB"
directory_labels = "/home/Downloads/SYNTHIA_RAND_CVPR16/GT"
train_images = np.array(os.listdir(directory_images))
train_labels = np.array(os.listdir(directory_images))
train_images = train_images[random_indices]
train_labels = train_labels[random_indices]
train_images = [tf.read_file(os.path.join(directory_images, img)) for img in train_images]
train_labels = [tf.read_file(os.path.join(directory_labels, img)) for img in train_labels]
train_images = [tf.io.decode_image(img, channels=3) for img in train_images]
train_labels = [tf.io.decode_image(img, channels=3) for img in train_labels]
train_images = tf.image.resize_images(train_images, INPUT_SIZE[:2])
train_labels = tf.image.resize_images(train_labels, INPUT_SIZE[:2])

train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
train_dataset = train_dataset.batch(3)
print(train_dataset.output_types)

This returns:

(tf.float32, tf.float32)

However, according to the documentation it should return a tensor of uint8's or uint16's. Why and where does the conversion take place?

I checked all intermediate steps with print statements, which doesn't tell me much as most intermediate lists are of class 'tensorflow.python.framework.ops.EagerTensor'.

Types are changing to float due to tf.image.resize_images.

Convert them back to uint as follows:

train_images = tf.cast(train_images, dtype=tf.uint8)
train_labels = tf.cast(train_labels, dtype=tf.uint8)

Output:

(tf.uint8, tf.uint8)

Versions of my code:

tensorflow version: 1.14.1-dev20190330
numpy version: 1.16.2


CNN for classification giving extreme result probabilities I'm having issues with my CNN, using Keras with Theano backend.
Basically, I need to classify 340x340 grayscale images into 6 categories. The problem is my CNN gives too "hard" probabilities, for instance it will rarely give predictions with some uncertainty, and always tries to push for a 90%+ for one class. The problem is that for my coursework, the penalty used is very harsh for complete miss classification, and uncertainty is much preferred. ( so having a prediction like [0.6, 0.3, 0.2, ...] is much better than having [0.9,0.03,0.02,..].

I'm unsure why this is happening. My dataset consists of 2400 images,
which are from different CCTV, and task is about recognising possible objects. Only 800 of the samples are actually from the data, the other 1600 have been generated through data augmentation.
Note that it is therefore extremely likely a that some pictures are either identical, or extremely similar (e.g. the same scene, one second later)

model = Sequential()
#1
# Few filter to take big stuff out
# Also, first layer is not conv so that I can reuse that layer separately
model.add(Dropout(0.1, input_shape=(1,340,340)))
model.add(Convolution2D(64, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#2
model.add(Dropout(0.1))
model.add(Convolution2D(128, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#3
model.add(Dropout(0.1))
model.add(Convolution2D(256, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#4
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
#5
model.add(Dense(512))
model.add(Activation('relu'))
#6
model.add(Dense(6))
model.add(Activation('softmax'))

opt = SGD(lr=0.001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

print "Training.."

filepath = "log/weights-improvement-{epoch:02d}---{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
model.fit(X_t, y_t, validation_split=0.1, nb_epoch=500, batch_size=32, callbacks=callbacks_list)

How do you suggest I fix this?
Thank you in advance!

There are a few different factors involved here. It is difficult to tell, without getting heavily involved, which could be the most important. I will put them in the order I think worth looking at first.

Your data is images from CCTV, so you likely have more than one image from each camera. From your results (reasonable training and CV scores, but bad training), it looks like you are over-fitting. But your CV approach is not spotting this. So I think that the test set is likely to be from a different set of cameras to the training set. In order to properly measure CV therefore, you have to split train/cv by camera - you cannot just use the 0.1 split, because a random split will include images which are correlated with training data and will give you too high an estimate, allowing you to overfit without noticing.

It occurs to me it might just be your data augmentation causing a problem for you here. If you augment first then randomly split to train/CV, then your CV set will contain images very similar to training set, and will get too high a score. You can more easily check and fix this than split by camera, so give it a try.

800 original images is not much to work with. You need to do something about that. Here are some ideas:

Scale down the images. You probably don't need 340x340. Depending on the target object, maybe just 78x78 will do. You can assess this easily enough - scale down and check if you can still differentiate the classes easily by eye.
You don't have enough data to get best quality filters in the lower layers, which will limit the capabilities of the CNN. You might bootstrap from a pre-trained image model. Take a publicly available pre-trained CNN, such as VGG-19, use its weights in convolutional layers as a starting point, put different classifier layers on top and fine-tune your classifier starting from this. This might change the ideal image scale too - you want something that fits the pre-trained CNN.
Augmenting the data, which you have started. You could go a lot further. Take random patches from training examples, possibly flipped horizontally (if this maintains the object class). However, don't augment data used for cross validation, unless your model used for testing also includes augmentation - e.g. if it takes 8 random augmented variants of the test image and returns an average of predictions, then you can do similar for cross-validation.

A 0.1 train/CV split run once on this much data is not going to give you an accurate assessment of the model. You need to run k-fold cross-validation. This is annoying because NNs take a long time to train, but if you want some confidence that you have really found some good parameters, you will need to do this. Remember to split by camera if you can.
The scoring appears to be strongly related to categorical cross-entropy, so you have the right loss function. You should optimise, using cross-validation, to find the model with the lowest loss, not the best accuracy.


How does back propagation works through layers like maxpooling and padding? I know back propagation takes derivatives (changing one quantity wrt other). But how this is applied when there is maxpooling layer in between two Conv2D layers? How it gains its original shape when there is padding added?

Max pooling will cancel the effect of not pooled values to the gradients.
Padded values either have no effect.

Nice thing about convolution is, that it is basically reducable to a matrix multiplication and the backpropagation is simply the transposed of it. So you have already your backward pass stored in the foward pass.


sklearn's cross_validate does not work with catboost I would like to use cross validation with catboost. Since I do not just want to use catboost but also sampling I am using a pipeline and hence cannot use catboost's own cross validation (which works if I just use catboost and not a pipeline). So I want to use sklearn's cross validation, which works fine if I use just numerical variables but as soon as I also include the categorical variables (cat_features) and use catboost's encoding, cross_validate doesn't work anymore. Even if I don't use a pipeline but just catboost alone I get a KeyError: 0 message with cross_validate. But I don't understand why.
This is part of my code that doesn't work:

from sklearn.model_selection import cross_validate
model = cb.CatBoostClassifier(**params, cat_features=cat_features)
cv_score = cross_validate(model, X_train, y_train, scoring='roc_auc', cv=5, return_train_score=True)

I found out that adding xtrain.astype('O') works.

Apparently catboost doesn't work with pandas Categorical dtypes yet: https://github.com/catboost/catboost/issues/814


Kaggle Titanic Survival Table an example of Naive Bayes? is the survival table classification method on the Kaggle Titanic dataset an example of an implementation of Naive Bayes ? I am asking because I am reading up on Naive Bayes and the basic idea is as follows:
"Find out the probability of the previously unseen instance
belonging to each class, then simply pick the most probable class"

The survival table
(http://www.markhneedham.com/blog/tag/kaggle/)
seems like an evaluation of the possibilities of survival given possible combinations of values of the chosen features and I'm wondering if it could be an example of Naive Bayes in another name. Can someone shed light on this ?

Naive Bayes is just one of the several approaches that you may apply in order to solve the Titanic's problem. The aim of the Kaggle's Titanic problem is to build a classification system that is able to predict one outcome (whether one person survived or not) given some input data. The survival table is a training dataset, that is, a table containing a set of examples to train your system with.

As I mentioned before, you could apply Naive Bayes to build your classification system to solve the Titanic problem. Naive Bayes is one of the simplest classification algorithms out there. It assumes that the data in your dataset has a very specific structure. Sometimes Naive Bayes can provide you with results that are good enough. Even if that is not the case, Naive Bayes may be useful as a first step; the information you obtain by analyzing Naive Bayes' results, and by further data analysis, will help you to choose which classification algorithm you could try next. Other examples of classification methods are k-nearest neighbours, neural networks, and logistic regression, but this is just a short list.

If you are new to Machine Learning, I recommend you to take a look to this course from Stanford: https://www.coursera.org/course/ml


ValueError: Found input variables with inconsistent numbers of samples: [13366, 1735 import numpy as np
import tensorflow as tf
import pandas as pd

from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import OrdinalEncoder
from sklearn.model_selection import train_test_split

def import_data():
    pulstars = pd.read_csv("pulsar_stars.csv", index_col=0)
    x = pulstars[["Standard deviation of the integrated profile", "Excess kurtosis of the integrated profile","Skewness of the integrated profile","Mean of the DM-SNR curve","Standard deviation of the DM-SNR curve","Excess kurtosis of the DM-SNR curve","Skewness of the DM-SNR curve"]]
    y = pulstars[["target_class"]]

    return x,y

inp , out = import_data()

out = out.dropna()

#normalize data
scaler = MinMaxScaler()
inp = scaler.fit_transform(inp)
out = scaler.fit_transform(out)

classConverter = OneHotEncoder(sparse=False)
out = classConverter.fit_transform(out)

layer = {
    "input": 8,
    "hidden": 8,
    "output": 2
}

i_h = {
    "weight": tf.Variable(tf.random_normal( [ layer["input"] , layer["hidden"] ] )),
    "bias": tf.Variable(tf.random_normal([layer["hidden"]]))
}

# weight = {
#     "i_h":tf.Variable(tf.random_normal( [ layer["input"] ], [ layer["hidden"] ] )),
#     "h_o":tf.Variable(tf.random_normal([[layer["hidden"]],[layer["output"]]]))
# }

# bias = {
#     "i_h":tf.Variable(tf.random_normal([layer["hidden"]])),
#     "h_o":tf.Variable(tf.random_normal([layer["output"]]))
# }

h_o = {
    "weight": tf.Variable(tf.random_normal([layer["hidden"],layer["output"]])),
    "bias": tf.Variable(tf.random_normal([layer["output"]]))
}

inp_dataset = tf.placeholder(tf.float32, [None, layer["input"]] )
target = tf.placeholder(tf.float32, [None, layer["output"]])

def feed_forward(inp_dataset):
    x1 = tf.matmul(inp_dataset,i_h["weight"]) + i_h["bias"]
    y1 = tf.nn.sigmoid(x1)

    x2 = tf.matmul(y1,h_o["weight"]) + h_o["bias"]
    y2 = tf.nn.sigmoid(x2)

    return y2 #output

output = feed_forward(inp_dataset)

error = tf.reduce_mean(0.5 * (target - output) ** 2)

#back propagation (update weight & bias)
update = tf.train.GradientDescentOptimizer(0.2).minimize(error)

#split dataset
x_train, x_test, y_train, y_test = train_test_split(inp, out, test_size = 0.2)

and this happen

ValueError                                Traceback (most recent call last)
<ipython-input-12-48df591e9215> in <module>()
     79
     80 #split dataset
---> 81 x_train, x_test, y_train, y_test = train_test_split(inp, out, test_size = 0.2)
     82
     83

2 frames
/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py in check_consistent_length(*arrays)
    210     if len(uniques) > 1:
    211         raise ValueError("Found input variables with inconsistent numbers of"
--> 212                          " samples: %r" % [int(l) for l in lengths])
    213
    214

ValueError: Found input variables with inconsistent numbers of samples: [13366, 1735]

After you apply dropna on your out, some rows are removed but the same is not done for your input. Hence, they have different number of rows, leading to your error.


What should I do with a test and train set not having same classes? Assume we have a train dataset with 5 classes a,b,c,d,e but the test dataset have only d and e with extra class f not present in train set. If i want to do machine learning with this datset in weka or python what should I do with this dataset? should I transform them and how should I do it? Should I remove classes form train set which are not in test set?
(it's a multi-class classification, and the letters a-f are the target classes)

Well, what you should do depends highly on what your goal here is.
Do you - for whatever reason - have to use the given test and training sets as such?
If not, a valid approach would be to shuffle the available data and determine a new test and training set, so that they each contain all classes.
Do you want a classifier only for d and e, and have to use the given test and training sets?
Then nothing really speaks against just training on the training set with all classes and testing only the ones you are interested in/ have test data for. Of course, then you can not make statements about the accuracy for other classes.
You could also try removing the other classes from the training set, if they won't appear in your application, which may or may not make the classifier perform better on the test data.
Essentially, it's not easy to provide a ceratin answer without having more context about why you have this kind of data, and what your goal is.
It would be great if you can provide a few more details, so we can work on a more fitting answer.


Python: ggplot geom_histogram() binwidth or bin parameter missing? I am unable to change number of bins when using geom_histogram() function. There is nothing in the documention, except some examples where binwidth=X is used (which is how R:ggplot2 uses it). However, this parameter seems to have no effect on the actual output.

Version: ggplot 0.10.4, Python 3.5 64bit

So for example:

import matplotlib.pyplot as plt
plt.figure()
dataframe['column'].plot.hist(bins=30)
plt.show()

works as expected and draws 30 bins. Meanwhile:

from ggplot import *
gg = ggplot(dataframe, aes(x='column')) + geom_histogram(binwidth=300)
print(gg)

does not change number of bins (I understand that binwidth sets interval for one bin), and have tried multiple values and/or datasets

bins=10 parameter is actually added in latest version (ggplot 0.11) https://github.com/yhat/ggplot/issues/513


Recommendation/personalization algorithm conflict I'm trying to build a recommendation engine for an e-commerce site. By using the common recommendation approach, I'm assuming that each product I recommend has the same value, so all I need to do is optimize the conversion rate probably using a common recommendation algorithm, but when the product's price varies a lot, what I really need to optimize is the following formula for each user:

Value of recomendation = (probability to convert) * (product price)

The bigger problem than choosing the right algorithm and approach is choosing the right metric, so I could compare the different algorithms. For example, if you would like to only optimize the conversion rate, I would use the precision and recall or false/positive metrics.

What metrics and approaches/algorithms are recommended in this case?

Thanks

This is actually slightly similar to the problem that insurance companies face except that it seems like your loss costs are known.  Insurers have some probability of loss and then, given loss, the magnitude of the loss follows some distribution.  The cost to the insurer is dependent on both and they tend to be inversely related (lower losses are more likely than higher ones.)

In your case, the value is known so you don't need to predict it the way insurers need to predict losses so you could simply:

Model the probability (phat)
Multiply the predicted probability by the known value (score = phat * value)
Recommend based on the resulting score

Insurance companies typically do the same thing in calculating premiums except that they also need to model value.  They sometimes model the two components jointly but typically they have separate models for frequency and severity and then just multiply them together to determine how much premium they should charge somebody.


What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes? I am building a multiclass classifier to predict the "Intent" of a question.
There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/questions varying from 3 % to 40 %.

Questions

What would be a good evaluation metric for this classifier?
What is the best proxy loss function for optimizing the suggested evaluation metric?

EDIT: This is not a ranking problem. The model should predict only 1 target class.
The cost of misclassification is the same for each target class.

You could look at sensitivity and specificity. They can be combined effectively to either provide a basis for a correct classification, or the basis for an exclusionary classification within each class.  That said, if you are looking for a measure for the model as a whole, you'll probably want to look at some sort of weighted average of the sensitivity scores.

sensitivity is the same calculation as precision, but specificity is not the same as recall.

That said, you may need to have several models that specialize in different areas or, perhaps as worst case scenario, you may need to produce a one-vs-the-rest ensemble of models to get the best results. This might help if, as anymous.asker suggested, perhaps you can introduce some cost around positive or negative outcomes.

HTH


Why might trees work so much better than boosting classifiers? I am predicting 10 classes label encoded using scikit-learn with 6 factors, 1.2M cases.
DecisionTreeClassifier
RandomForestClassifier
ExtraTreesClassifier
give accuracies (and precision and recall) of 0.9

AdaBoostClassifier
GradientBoostingClassifier
give accuracies of 0.2

Any pointers on the huge discrepancy?

(I am doing gridsearchcv).
Code:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

def output_metrics():
    from sklearn.metrics import accuracy_score, precision_score, recall_score
    print("Accuracy:",accuracy_score(y_test, y_pred))
    print('Precision', precision_score(y_test, y_pred, average=None).mean())
    print('Recall', recall_score(y_test, y_pred, average=None).mean())

from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import GridSearchCV

tree_para =  { 'n_estimators': [16, 32] }
clf = GridSearchCV(AdaBoostClassifier(), tree_para, cv=5)
model= clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)

output_metrics()

As a disclaimer, I would like to point out that the performance in this specific case might still be due to the specific dataset used.

A likely explanation lies in the essence of what trees and boosting algorithms do.

As @akvall pointed out in the comments, Boosting algorithms can often overfit since this is what they are designed to do! As a reminder, regardless of how fancy a boosting algorithm works, it follows the following logic:

train on the training set
evaluate to see which mistakes were made
retrain and focus more on the previous mistakes
repeat until satisfactory results

Trees do not work the same way and are therefore less prone to overfitting. A Random Forest will simply compute independent trees and use majority voting to make a prediction.

Every boosting algorithm will "boost" for a certain amount of iterations, it is probably wise to look at how many times your algorithms did "boost".


difference betwen predicting seen and unseen data I tried to test my model with seen and unseen data (seen data are data that i used to learn the model). I figure out that as much as i increase the number of features seen data can be properly predicted, while when using a feature selection technique unseen data  can be properly predicted. Is there any explanation about this.

Thanks in advance

What you want to do in building predictive models , in general , is closing the gap between the loss on seen data ( training data ) and unseen data ( test data ). Increasing the numbers of features will naturally make the hypothesis ( the function that is meant to learn from your training and will receive unseen data ( X ) to predict your target value ( Y ) ) more complex and can fit your training data perfectly at a certain point, BUT there is a thresh-hold ( which you went above according to your results by increasing the number of features) where your model starts giving very good results on your training set and much poorer results on the test set, and that is called overfitting which is a problem caused by high variance in your model which basically means you are allowing your function to be complex enough to fit your training data perfectly, and be less able to predict unseen data ( your test data ).

Go with the feature selection technique if it reduces the gap between your training loss and test loss!

Notice here in this example that increasing the number of features gives lesser test loss but, at a certain point, the loss on test starts going up again.


What is PAC learning? I have seen here but I really cannot realize that.

  In this framework, the learner receives samples and must select a generalization function (called the hypothesis) from a certain class of possible functions. The goal is that, with high probability (the "probably" part), the selected function will have low generalization error.

Actually we do that in every machine learning situations and we do latter part for avoiding over-fitting. Why do we call it PAC-learning?
I also have not get the meaning of the math too. Is there anyone that can help?

PAC stands for Probably Approximately Correct.
It was a very common research area in computer science looking for proof of learnability of certain hypothesis sets.

The usual hypothesis sets were not distributions (like in statistics) but more logical formulas like DNF, CNF, DFA, etc.

In order to prove learnability, you need to show that for every sample distribution, for every concept $h$ in the hypothesis class $H$,  for every $\lambda$ (the probability part) and for every $\epsilon$ (the approximately part) you can find an hypothesis $h^*$ such that with probability at least $1- \lambda$ the disagreement between $h$ and $h^*$ will be smaller than $\epsilon$.

This bar for learning is very high, and some of the research proved negative results about many simple hypothesis sets.

When you try to learn to differentiate between cats and dogs, you don't have a mathematical definition of hypothesis set. Most publications run an algorithm on a dataset and show their result. Almost nobody proves that the algorithms are correct (even given some assumptions). I find the lack of proofs quite sad.

The math behind PAC is actually probability. Let's say that you have two cubes (classifier, concept) and you want to know how correlated they are. So, you toss them together $m$ times and you check the level of disagreement, epsilon. The higher the $m$ and the lower the $\epsilon$, the more confident you are that this is due to correlation and not due to error, $\lambda$. Given this intuition, PAC learning usually works in the other direction. Once you state the desired lambda and epsilon, the algorithm provides the number of samples $m$ needed to reach them.


How to test/validate unlabeled data in association rules in R? I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that these association rules are true? How can I validate them? What are common practice to test it?

I thought about cross validation (with training data and test data) as I read that it's not impossible to use it on unsupervised learning methods..but I'm not sure if it makes sense since I don't use labeled data.

If someone has a clue, even if it's not specifically about association rules (but testing other unsupervised learning methods), that would also be helpful to me.

I uploaded an example of the data that I use here in case it's relevant: https://www.mediafire.com/?4b1zqpkbjf15iuy

You may want to consider using your own APparameter object to put "significance" constraints on the rules learned by Apriori. See page 13 of the arules documentation. This could reduce the number of uninteresting rules returned in your run.

In lieu of gold standard data for your domain, consider bootstrap resampling as a form of validation, as described in this article.


finding similarity of a new datapoint I have built a recommendation engine using cosine similarity. When I want to find all the records similar to a given record that is already present in the dataset it works. Consider a case, a user enters a new record which does not exist in the similarity matrix, to find and recommend similar records I am adding the new data point into the dataset built the similarity matrix and then pick the similar records, Is there a better way to handle this problem?
Adding more details
data=["document1", "document2", "document", ... "documentN"]
count_matrix = count.fit_transform(data)
cosine_sim = cosine_similarity(count_matrix, count_matrix)

Instead of transforming, and calculating the whole cosine similarity matrix every time you receive a new data point, store your existent data, and whenever you get a new sample calculate the cosine similarity of a new sample against the existent data. Then append and store it.
That is, you need a two function:

First function - Calculate the distance matrix among N existent documents, then store it to the [N x N matrix], also you need to store transformed data, count_matrix.

Second function - When you get a new data point, calculate the cosine similarity of a new datapoint to all existent datapoints in the count_matrix, and append the new [1 x N+1] list to the existent cosine similarity matrix. Also, update the count_matrix. Afterward, store both.

Simply, instead of creating [N+1 x N+1] matrix, store the data and when you have new data, calculate [1 x N+1] and combine it with the existent matrix and store it.


unsupervised learning in medical systems and intelligent systems? I have a dataset which belongs to a hospital. It contains data about patients and healthy people. The problem is separating healthy ones from patients. I add some new features to dataset to solve this problem. When I reduce the dimensions of data including the new features and visualize the data, the patient and healthy individuals are distinguishable(visually separable). Now if one asks what is the relation between the used approach (feature extraction, visualization, using the human ability, unsupervised methods) and expert system, what can I say? How can I use this method to build an expert system?

Thank you in advance.

Expert systems rely on knowledge "extracted" from human subject matter experts (e.g. a team of physicians). They then use this knowledge to make prediction about the data (e.g. if blood pressure > x then patient).

Your approach used properties of the dataset you worked on. The knowledge was in fact extracted from the data themselves.

The second approach may or may not be compatible with expert advice. If your algorithm found that blood pressure was a good feature to split healthy people and patient, you will probably find experts agreeing. However, if your main feature comes from a combination of many variables (e.g. PCA) you may have trouble finding an expert validating your approach.

So the way you could use your approach would be to team up with subject matter experts and make sure that you use only features that have a meaning for them (i.e. don't over-engineer data).


What are the techniques for anomaly detection of Unsupervised learning problem I have sufficient and properly formatted data in millions without labels.
I have to find out the anomalies.

Heard Isolation forest, Mahalanobis distance about identifying anomalies in unsupervised learning.
Are these ok to try?

Are their any other techniques we can try?

Thanks

You can try these techniques and many more.- All anomaly detection techniques
As discussed in article, these are outlier detection techniques.Are you looking for outliers? better to get some known abnormalities and build a classification.
If supervised not possible, try to fit one of these approaches-
ABOD for identifying abnormalities in high dimensional data
Should clustering be based on distance or density to find outliers(abnormalities)
Connectivity based outlier detection technique
There are other techniques like PCA based, regression based, auto-encoder, knn, weighted Knn and even self organizing map (SOM) . let me know if you need some more information.
Imp-  Know your abnormalities better before jumping to machine learning, I have experienced that even qq plot or just data points 3sd away might give better anomaly detection.


Faster R-CNN wrapper for the number of RPNs in the layer dimensions? When I intialize Faster R-CNN in the deployment phase, the number of samples per image (parameter from config file: TEST.RPN_POST_NMS_TOP_N) is set to 300, that's the number of predicted bounding boxes to keep after non-max suppression. However, the network is initialized with the number set to 1:

('rpn/output', (1, 512, 14, 14))
('rpn/output_rpn_relu/3x3_0_split_0', (1, 512, 14, 14))
('rpn/output_rpn_relu/3x3_0_split_1', (1, 512, 14, 14))
('rpn_cls_score', (1, 18, 14, 14))
('rpn_bbox_pred', (1, 36, 14, 14))
('rpn_cls_score_reshape', (1, 2, 126, 14))
('rpn_cls_prob', (1, 2, 126, 14))
('rpn_cls_prob_reshape', (1, 18, 14, 14))
('rois', (1, 5))
('pool5', (1, 512, 7, 7))
('fc6', (1, 4096))
('fc7', (1, 4096))
('fc7_relu7_0_split_0', (1, 4096))
('fc7_relu7_0_split_1', (1, 4096))
('cls_score', (1, 21))
('bbox_pred', (1, 84))
('cls_prob', (1, 21))

The ones I'm particularly interested are fc6/7, bbox_pred and cls_prob. After net.forward(**kwargs) is run, the first dimension of these layers is changed to 300: (300,4096), (300, 84), (300,21) to match the number of RoIs. The rois output is reshaped in the TargetLayer class, but rest are a bit of a problem:

Caffe doesn't implement this out of the box, there should be some wrapper for this, but I can't find it. Any suggestions on where to look? I want to implement something similar for my algorithm.

It is a bit confusing because all these four layer (fc6/7, bbox_pred, cls_prob) are just fully connected layers defined in the config, nothing fancy.

PS Also I don't think (100,4096) means 100 times more weights, it would cetrainly be undoable for such layer size, so the weights are shared. But how?

OK so the trick was in the RoI Pooling Layer: it is initialized with dimensions (1,512,7,7) and outputs the tensor (#RoIs,512,7,7), which in turn is an input in a fully-connected layer fc6 that treats the input as batch size #RoIs and hence processes them in the normal way, like it would have processed any batch.


Meaning of subscript in min max value function This possibly is a very stupid question, but i have not been able to find the answer on the internet and have got no clue which keywords to use while searching.

What's the meaning of $\mathbb{E}_{x \sim p_{data}(h)} [...]$
Where ... is some function.

in context like

$\mathbb{E}_{x \sim p_{data}}[...] + \mathbb{E}_{x \sim p_{z}} [...]$

and

$\mathbb{E}_{x \sim p_{data}(h)} [...] + \mathbb{E}_{x \sim p_{z}(z)} [...]$

This is written in a lot of papers, but what does it mean?

I'm supposing you are reading the paper on GAN where $[...]$ is the loss function. $\mathbb{E}_{x \sim p_{data}(h)} [...]$ now represents the expected loss where the data is taken from the distribution $p_{data}(h)$.

Edit:
$\mathbb{E}[ ]$ is an operator which gives the expected value of the function on which it is acted upon. When specifically acted on the loss function it is called risk or empirical loss. And L(z) that you are asking is the loss.

For more on its definition and explanation, see the below questions:

Risk function or expected loss function

why do we calculate risk when we already have loss functions?


Score remains same during hyper parameter tuning My model:

    model = Sequential()
    model.add(Dense(128, activation='relu', input_dim=n_input_1))
    model.add(Dense(64, activation='relu'))
    #model.add(Dense(32, activation='relu'))
    #model.add(Dense(16, activation='relu'))
    model.add(Dense(1))
    model.compile(optimizer='adam', loss='mse',metrics=['mse'])

Now, I am doing hyper parameter tuning, but it is showing the same for every possible result:

Best: -61101.514139 using {'batch_size': 10, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 15}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 15}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 15}

This is the first time I have done hyper parameter tuning before and this has stumped me. I can provide additional details if needed. What is a possible reason for this behavior?

I am doing time series forecasting using MLP. I have used 'neg_mean_absolute_error as the scoring function in gridsearchCV.

Edit: this is what Im running:

from sklearn.model_selection import GridSearchCV
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# define the grid search parameters
model = KerasClassifier(build_fn=create_model, verbose=1)
batch_size = [10,20,2000]
epochs = [2,4,5,10, 25]
param_grid = dict(batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3,scoring='neg_mean_squared_error')
grid_result = grid.fit(scaled_train,scaled_train_y)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

I would suggest the following changes,

Try KerasRegressor instead of KerasClassifier.
Use other activation functions than relu (Eg: tanh, or just linear)
Normalize both target and regressors.


Recover a integer valued function with *-learning I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps
$$
 f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+}
$$
where $\mathbb{P} = \{1,\ldots,50\} \subset\mathbb{N}$. $n$ defines the amount of input and $p$ is a parameter of the process, which has a continuous influence on the runtime. We are interested in the value of $p$ such that $f(n,p)$ for a given $n$. At the run the experiment  some $n$, like 500, 1000, 5000, and for all possible values of $p$ and then select the optimal $p$ by hand. Like for everything smaller than 750 use the optimal $p$ of the test run with $n=500$ and so on. Typically for a fixed $n$ the runtime over $p$ has the following style but with noise added:

For increasing $n$ the bottom moves to the right and due to noise the bottom is not flat. So in the example the optimal $p$ is somewhere between $20$ and $30$.
We are now searching for a function $g$:
$$
g: \mathbb{N} \to \mathbb{P}
$$
which is defined by
$$
 g(n) = \operatorname{argmin}\limits_{p\in\mathbb{P}} f(n,p)
$$
Since sampling $f$ for fixed $n$ is not a problem and I already have a large set of samples I think this will be a good problem to start with for looking into the machine learning area for myself.
My basic question is where to start? Search for machine learning tutorial I mostly end up with the image recognition examples and did see any good connections between those and my problem.
Any suggestions about stuff to read for this problem? Any idea which software to use? And finally how to extract $g$ such that we can provide this function to the users or implement somewhere? At the moment I know that $g$ is non-decreasing for increasing $n$.
Edit
The sampling results can also look like this
Thereby the spikes in the "mininum" point area are no noise. They are caused by technical internals of the regarded process. So while processing the data and removing the noise these points need to kept. And they make finding $g$ so hard because in one sampling run the minimal $p$ lies left, in the middle or right of them. For this reason I though this would be a good point to use machine learning to get some insights.

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.
From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every  point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.
I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful  to know what it looks like, in particular to choose an appropriate regression method.
[edit] About dealing with noise.
The standard statistical way to deal with noise is to do nothing at all :)
How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.
I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.
[edit 2] Another interesting visualization: you can represent your full data as a heatmap with $n$/$p$ on the X/Y axis and the colour based on the value of $f(n,p$)$.


Minimization algorithm that can consider gradient close to solution I want to minimize a function which has sharp gradients close to each local minimum. Due to process tolerances, I want to find solutions which meet some minimum criterion (e.g. lower than x), but have a shallow gradient near to the found minimum.

My question is: is this simply a case of me assigning a penalty factor to minima with sharp nearby gradients, or is there a class of algorithm that can handle this sort of constraint as part of its optimization routine?

Well, that’s a tricky question. Do the gradients turn large somehow because they are discontinuous? (gradient descent will likely not work then) Is it maybe because the variables are in too different scales (Newton & related would help)? Is there some sort of barrier that would limit the domain (e.g. log(x))? Does the function have some flat uniform area, e.g. __/? Are these saddle points? Is your problem constrained and these are solutions at the boundaries? In some of these cases, switching to something different like subgradients might help you, but this highly depends on the reason why your problem has these optima that you want to avoid.

Generally speaking, gradient-based techniques converge to whatever local optimum they find first, and if you are not happy with that, you’ll have to use other metaheuristics (e.g. add restarts, incorporate penalties at known optima, use derivative-free methods). Perhaps other techniques that carry momentum like ADAM might somehow allow you to dodge these too.


Notation unclear - Radial Basis Funtion In Chapter 6 (Deep Forward Networks) on Page 193 of Deep Learning they talk about the design of Hidden Units.
The Radial Basis Function is introduced as follows:
$$
h_i = exp\big{(}-\frac{1}{\sigma_i^2}||W_{:,i} − x||^2\big{)}
$$
What does the colon as Index for $W$ mean?

According to the notation page, it means the $i$-th column.

$W$ is a template matrix, where the closer $x$ is to $i$-column of $W$, the bigger the value of $h_i$.


How to handle different categorical embedding sizes in hold out data set I have a pytorch tabular dataset with zip code as a categorical embedding.  I'm getting great results on the test set.  When I go to run my hold out sample through, it errors out because I have more zip codes in the hold out then what the model was trained on.

How do I handle this?  In production, the likelihood of seeing a new zip code is high so I need to learn something I can transfer into production.

Thank you.

One possibility would be to represent the zip codes using some transformation that could be applied to new (unseen) zip codes as well. For example, could you re-represent zip codes as latitude + longitude?


Finding Outliers in Resource Forecast Data Unsure if this is the correct place to place this, please close if so.

I'm a workforce analyst at a large retail company, I own and maintain all the forecasting for our retail stores. This is based on Product Sales Forecast which I run through a model which has various tasks (such as changing items on a shelve or selling a car alarm for example) binned into timed categories, e.g

Item A takes 30 seconds, so 500 product forecasts will allow you (0.5 * 500) or 4.1 hours which are then further binned into Weekly allocations.

Now, as, for each iteration of this model I've run, there are always outliers (read mistakes), such as Store Y has no product volume to Product Group X in Week 22, which is a mistake from the finance team.

My senior has said this has always been the case for the 8 or so years he's been with the company.

Now, in my head, I assume I would be able to find outliers based on historical data using some sort of statistical method, however, I have no idea where to start, my data is as follows and is around 10 million rows of data.

import pandas as pd
import numpy as np

data =  [19,21,24,18,3]
pg = ['PG','ZF','AA','GG','ZF']
location = ['AA_1','AA_1','AA_2','AA_2','AA_2']
weeks = [1,1,2,2,2]

df = pd.DataFrame({'Location' : location,
            'productGroup' : pg,
            'Week' : weeks,
            'productVolumes' : data })
print(df)

  Location  productGroup    Week    productVolumes
0   AA_1    PG  1   19
1   AA_1    ZF  1   21
2   AA_2    AA  2   24
3   AA_2    GG  2   18
4   AA_2    ZF  2   3

in your expert opinions, what would be the best method to use?

There is not one way of detecting outliers. Most statistical methods assume an underlying normal distribution. If this is the case you could apply for example an Thompson Tau test. You can also define some own criteria. In your use case I think an important question is: what will happen to the quality of the model if you drop or modify missing data. This gives you an idea how to handle these missing data points.


images from training set are different from images of test set I am doing image classification with CNN and I have a training set and a test set with different distributions. To try to overcome this problem I am thinking about doing a standardization using Imagegenerator, but I am encoutering some problems. Here is the part of the code I am working on:

trainingset = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

batch_size = 32
train_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255,\
    zoom_range=0.1,\
    rotation_range=10,\
    width_shift_range=0.1,\
    height_shift_range=0.1,\
    horizontal_flip=True,\
    vertical_flip=False)

train_datagen.fit(trainingset);

train_generator = train_datagen.flow_from_directory(
    directory=trainingset,
    #target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=True
)

test_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255)

test_datagen.fit(testset);

test_generator = test_datagen.flow_from_directory(
    directory=testset,
    #target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

num_samples = train_generator.n
num_classes = train_generator.num_classes
input_shape = train_generator.image_shape

classnames = [k for k,v in train_generator.class_indices.items()]

print("Image input %s" %str(input_shape))
print("Classes: %r" %classnames)

print('Loaded %d training samples from %d classes.' %
(num_samples,num_classes))
print('Loaded %d test samples from %d classes.' %
(test_generator.n,test_generator.num_classes))

so, what I am trying to do is using in the Imag  genereator the fields featurewise_center=True and featurewise_std_normalization=True to do standardization, but if I try to fit the generator to the trainingset by doing train_datagen.fit(trainingset); I get the following error:

    ValueError                                Traceback (most recent call last)
<ipython-input-16-28e4ebb819be> in <module>()
     23     vertical_flip=False)
     24
---> 25 train_datagen.fit(trainingset);
     26
     27 train_generator = train_datagen.flow_from_directory(

1 frames
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a,
dtype, order)
     83
     84     """
---> 85     return array(a, dtype, copy=False, order=order)
     86
     87

ValueError: could not convert string to float: '/content/drive/My Drive/Colab
Notebooks/Train'

Can somebody please help me? Thanks in advance.

[EDIT] I am trying to adapt what is written here to my problem.

[EDIT_2] I think the problem is that .fit() takes as parameter a numpy array, while I am trying to pass to it a string, which is the path for the images.

But I don't understand now how to do, because I should transform this to a numpy array in order to do the fit.

The way you use ImageDataGenerator is wrong. The .fit() method is trying to read in the directory path which is a string. To be able to run your code, you should remove the train_datagen.fit(training_set) and your script should work fine and all the images should have been preprocessed the way you want them. Here's a link on how to use the ImageDataGen.

The .flow_from_directory() function is used for raw images. The .fit() method is used on numericals.


Identifying templates from SMS text I am building an app where I identify information from the SMS, something similar to expense management apps. I have a parser which reads all the SMS of user, identifies SMS of interest and parses useful information from it. This bit is working fine, however the problem is that I want to also score completeness and accuracy of the parser.
Completeness - Out of 100 SMS of interest, how many of them parser is able to identify
Accuracy - Out of SMS identified as relevant, for how many SMS parser is able to extract right information
I have a manual process to identify these. I take a random sample from the corpus along with parsing results and manually tag which ones are correct or wrong. I am able to identify gaps in parser and fix them by this process. However more gaps I fix more it becomes difficult to identify further gaps with this process. Also since sample is random, it has many SMS with same structure. Parsing success/failure for similar SMS is similar. Completeness and accuracy from the random sample does not correctly represents correctness and accuracy of whole corpus as well.
I was thinking that since financial SMS of interest have templates associated with them, if I can tag template of the SMS, I can keep track the templates which are known to be parsed correctly and only do manual checks on templates which are not yet tagged manually for correctness. This would also help me build tools to identify correctness, accuracy and unknown status for the whole corpus. It can also help me identify new templates as they are added to the corpus.
There are 1000s of templates. Examples of templates:
Template1 - Thanks for spending INR 100 at Amazon on 1st Jan, your available credit limit is 1,000
Template2 - You have spent INR 100 on your credit card ending in 1234 at Amazon on 1st Jan 2020
Template3 - You txn of INR 100 is credited/reversed by Amazon on 2nd Jan 2020, Avl Bal - 1,100
Template4 - You account xxxx1234 has been debited with INR 500 and bene A/C xxxx4321 has been credit on 3rd Jan

I have searched a lot on google and many results which seemed relevant were from this forum. Similarity based on n-gram seemed quite interesting, but I was not able to make much of how to apply it to my problem. One of the questions which seemed very similar: Identifying templates with parameters in text fragments. However, it differs slightly in the sense that it identifies sentences with similar templates out of given corpus. I want to tag template of a sentence without looking at the whole corpus. I am okay with training a model to tag the template, but I would want the model to identify new templates which are not part of original corpus and tag them as well.
I am a software engineer and have no prior experience with data science. Please pardon me if I am not making any sense or if this forum is not the right place for this question. I would really appreciate any help in identifying the right approach and some references to understand the concepts involved.

I'm not sure I understand your problem very well but let's see.
First let me try to formalize the task as a ML problem:

Identifying the SMS of interest is a binary classification task. Your "completeness" score seems to correspond to the standard recall measure. It is usually a good idea to also look at precision, i.e. out of the SMS identified as relevant how many are truly relevant.
You don't really explain the second part of extracting information out of SMS of interest. In general this part might be some kind of sequence labelling problem (similar to Named Entity Recognition).

It's important to distinguish these two parts if you're going to use ML.
Now what you are currently doing is a rule-based system: you manually identify the patterns one by one, then the system just applies the list of patterns. rule-based systems are considered the ancestor of ML systems: as you noticed, it might be possible to automatize the part of identifying the patterns. Given that you already have labelled examples (training data), you could train a model to recognize the general patterns by itself. The main difficulty is to provide the model with the right features so that it can do the job as correctly as possible.
Note that in general it doesn't really work as a list of templates in the same way as what you do manually.
The question of identifying cases without looking at the whole corpus might be related to ''semi-supervised learning'', where a system iteratively learns to annotate unlabelled data from an small initial set of labelled data.


Interactive labeling/annotating of time series data I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use for supervised ML.

E.g. the input data is a csv-file and the output is another csv-file of the format timestamp,label.

Therefore I need something like this:

to visualize data
to select a specific area
output the labels with timestamps

As an example:

Building such a tool in python will not take too long, however I was just wondering how other people solve this problem and maybe there are already nice OS tools for doing this. Thank you!

Update: we have updated TRAINSET to include the ability to upload multiple series as well as apply multiple labels! See demo in GIF below.
We had this same problem again and again at Geocene, so we came up with this open-source web app called TRAINSET. You can use TRAINSET to brush labels onto time series data. You import data in a defined CSV format, then label the data, and export a labeled CSV. You can also import a pre-labeled CSV if you're really just trying to refine labels. You can use the hosted version of TRAINSET at https://trainset.geocene.com or you can deploy it yourself by following the readme at https://github.com/geocene/trainset


Dealing with Skewed data using Boxcox transformation by adding 1 I was trying to reduce the skewness of data using boxcox transformation. But was facing an error:

  ValueError: Data must be positive.

I figured out why it was throwing an error, my data ranged from .0 to 1 so I added one and it worked.

Although is it the right way to deal with it?

Boxcox you need to tune the lambda parameter, in order to transform the data to normal one (and than its implicitly non-skewed) however, I would suggest to rather use log transform, its faster.


Error: building keras model using LSTM I am trying to build a simple LSTM based model but I am getting "can't set attribute error" on the line to add LSTM layer to the model. I am unable to figure the reason as to why this error is appearing. This is the code I am using.

left = Sequential()
left.add(LSTM(64,activation='sigmoid',batch_input_shape=(10,look_back,dim)))
left.add(Dense(dim, activation='linear'))
left.compile(loss='mean_squared_error', optimizer='rmsprop')

I do not think there is an issue with code. The version of Keras is 2.0.7 with tensorflow backend. It is difficult to trace the cause of error.

Install the latest Keras version: https://github.com/keras-team/keras/issues/7736#issuecomment-324989522


What is the novelty in AlphaGo, Google Deepmind's Go playing system? Recently researchers at Google DeepMind published a paper, where they described a Go playing system that beat the best current computer programs and the human European champion.

I had a quick look at that paper, and it seems it is using many interesting ideas from previous papers. What did they do differently that allowed them to achieve this spectacular improvement?

The existing programs before AlphaGo were based on:

1) Convolutional neural networks (CNN), trained on a database of existing games.

OR

2) Monte Carlo tree search (MCTS)

AlphaGo is based on a combination of:

A) reinforcement learning: train networks by letting versions of CNN's (see above) play against eachother.

AND

B) MCTS using moves generated by step A)

On top of that, the performance was improved even further by using distributed computing with large amounts of CPU's and GPU's.

So the novelty was the combination of the above techniques A) and B).


Brain Connectogram or Connectivity Visualization Can anybody please guide me how can I draw the following figure in my windows 10 environment, I mean is there some built-in online tool or software that I can use for this purpose?
In my case I have two average correlation matrices , one for healthy subjects and one for ADHD subjects and each matrix is 90 x 90 as I am using AAL brain tempalte.

You might find this link useful if you are R user.
https://www.r-graph-gallery.com/chord-diagram.html


Is deep learning a must in a Data Science MSc programme? I am reading the programme outline of this two-year MSc in Data Science and I found that it has no deep learning content (as in many other european ones). I am no expert but as far as I've seen I think that DL is going to be a heavy weight of ML algorithms for a long time.

Do you think it is a bad idea to take a strong focus on classical models (e.g. bayesian) instead of teaching DL in a Data Science/ML MSc?

No, it's not problematic. Most data scientists do not need or use deep learning. Deep learning is very popular right now, but that does not mean it's widely used. Deep learning can lead to substantial overfitting on small to medium datasets (I'm arbitrarily going to say that means less than 2 GB), which are the sizes that most people have. Deep learning is primarily used for object recognition in images, or text/speech models. If you're not doing either of these two things, you probably don't need to use DL.


Right way to Fine Tune - Train a fully connected layer as a separate step I'm using Fine Tuning with caffenet and it works really well but then I read this in Keras blog entry on Fine Tuning (They use a trained VGG16 model):

"in order to perform fine-tuning, all layers should start with properly trained weights:
 for instance you should not slap a randomly initialized fully-connected network on top of a pre-trained convolutional base.
This is because the large gradient updates triggered by the randomly initialized weights would wreck the learned weights in the convolutional base.
 In our case this is why we first train the top-level classifier, and only then start fine-tuning convolutional weights alongside it."

So as a separate step in Fine tuning they save the output of the last layer before the fully connected layer (the "bottleneck features") and then they train a "small fully-connected model" on those features and only then they put the newly trained fully connected layer on top of the whole net and train the "last convolutional block".

Is this always the right way to do Fine Tuning?

thanks

First of all, you can do this in automatic way by setting the layers to be trainable or not. By selecting this hyper parameter to be false you freez the layer. we know that the first few layers are features extraction and the assumption when you perform fine tuning that the original problem and your problem are correlated, thus the features should be also the same.
What is the right thing to do?
It is very difficult to find a direct and absolute answer, it depends mainly on the original problem and the new one. In fact, I prefer to fine tune the whole network using different learning rates.
You can use smaller learning rate for the convolution layers and larger rate with the fully connected layers.


How to prepare data for LSTM time series prediction I have a binary classification task for time series data.
Every 14 rows in my CSV is relevant to one time slot. How should I prepare this data to be used in LSTM? In other word how to feed the model with this data?

I hope that dataset also consist of meta data, which means you also need to have a one to one mapping of those tuples, eg. dog > good, cat > bad, kittens > bad, puppies > good, etc.

Separate the data into X:training_data, Y:label. Then use a vectorizer and train using X, Y. If you're able to do above steps then use methods like test_train set  , cross_folds etc.

Friendly suggestion: Try seq2seq layers before LSTM (they require more resources).


Machine learning in audio? What are recommended books or online courses that are mostly helpful to someone wanting to delve into machine learning specifically in audio?

Or is the learning path mostly the same as for any other machine learning field?

I think that resources about speech processing would be a good introduction to audio files manipulation for machine learning in general.

For a broader introduction to audio,
Machine Learning for Audio, Image and Video Analysis: Theory and Applications, Reviews
Book by Alessandro Vinciarelli and Francesco Camastra could be a good read.

Also, check out some blogs like this one: https://www.analyticsvidhya.com/blog/2018/01/10-audio-processing-projects-applications/ (as you mention deep learning in the tags) that can give a good introduction with some hands-on experience.


which open-source frameworks exist for fact-extraction probably question is silly, but still: what tools one might use to transform an English text to a set of "facts"

for example a sentence "Julius Caesar crossed the Rubicon in 49 BC" would be parsed as something like:

{
"subject": "Julius Caesar (person)",
"object": "Rubicon (river)",
"verb": "to cross",
"verb aspects": ["time": "49BC"]
}

or anything remotely close to this

I don't know many examples but I'm aware of at least one such tool, specialized for the medical domain: SemRep

  SemRep is a UMLS-based program that extracts three-part propositions,
  called semantic predications, from sentences in biomedical text.
  Predications consist of a subject argument, an object argument, and
  the relation that binds them. For example, from the sentence in (1),
  SemRep extracts the predications in (2).

  1.We used hemofiltration to treat a patient with digoxin overdose that was complicated by refractory hyperkalemia.

  Hemofiltration-TREATS-Patients,
  Digoxin overdose-PROCESS_OF-Patients,
  hyperkalemia-COMPLICATES-Digoxin overdose,
  Hemofiltration-TREATS(INFER)-Digoxin overdose

In general, this is closely related to the problem of semantic role labeling:

  semantic role labeling (also called shallow semantic parsing) is the process that assigns labels to words or phrases in a sentence that indicate their semantic role in the sentence, such as that of an agent, goal, or result.

Apparently there are some implementations available: https://framenet.icsi.berkeley.edu/fndrupal/ASRL


Whether Interaction terms should be included in Linear Regression analysis? I am working on a linear model with 6 independent variables and when thinking about including an interaction I got lost.
An interaction exists if the level of one independent variable is affected by another independent variable. Doesn't that therefore mean that if an interaction exists there may also be collinearity problems? Similarly, if the correlation is low between the two variables, then that should imply there is no interaction?
I hope my question makes sense and that someone can help me clear up this confusion. I am now confused how we can ever get to having an interaction between two independent variables ?

Just to expand on @Ankita Talwar answer and give some slightly more formal intuition you can write a linear model with to regressors and their interaction as follows:
$$ y = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1 x_2$$
where $x_1 x_2$ is the interaction term.
Now refactoring you can see that the interaction can be absorbed into the coefficient for $x1$ making it depend on $x_2$: $ y = w_0 + v_1(x_2) x_1 + w_2 x_2$, where now $v_1(x_2) = w_1 + w_2 x_2$ is a function that depends on $x_2$ (alternatively you can view the interaction as modifying $x_2$ coefficient).
So, when you add an interaction term you allow the coefficient value of one variable to vary depending on the value of the other variable. This operation only touches the coefficients not the variables themselves (so it doesn't say anything about them being collinear).
I guess your question might come from the term $x_1 x_2$ in the linear model, noticing that this term depends on $x_1$ (or $x_2$). If that is the case, please notice that $x_1 x_2$ would be collinear with, let say, $x_1$ if $x_2$ were a constant and not a variable (and viceversa). When both are variables (and of course provided the original variables are not linearly related) the interaction term is not collinear with any of the two variables.
However you might encounter some collinearity if one variable has a much smaller scale than the other one, in which case the interaction is mainly driven by the variable with the larger scale. This problem can be solved by standardizing your variables (which is anyway a good idea in regression problems).
Hope this clarifies a bit.


R - How do I remove a varying number of digits from a date-vector I want to remove a varying number of digits from a date vector. My date vectors looks like this:

I want to convert this vector into a date vector, but first I have to get rid of the number in front of it. This is where I struggle since this number can be (here from 88-106) varys in my data.frame from 1-50 000. Does anyone know how to remove these number without destroying the date? I tried the following, but it didn't work out:

I would really appreciate your help! Thanks!

assuming your strings are of the form [NUMBER][WHITESPACE][STUFF-TO-KEEP]

you can use regex, for example with the stringr package.

# construct strings
strings <- paste(c(1,11,111,1111), "KEEP")
strings
[1] "1 KEEP"    "11 KEEP"   "111 KEEP"  "1111 KEEP"

stringr::str_remove(strings, "[0-9].* (?=[a-zA-Z])")
[1] "KEEP" "KEEP" "KEEP" "KEEP"

the regex "[0-9].* (?=[a-zA-Z])" expained:

'[0-9].* ' # (including the whitespace!)
# matches any number of digigs from 0 to 9 followed by a whitespace

(?=[a-zA-Z])
# this is a lookahead group, so an additional requirement is,
# that a match is only used when the above match is followed
# by a letter in the alphabet, small(a-z) or large (A-Z)

of course, str_remove removes the matches string.


How should I improve my Vectorized Gradient descent linear regression model? I wrote a vectorized Gradient descent implementation of the linear regression model. The Dataset looks something like:

It's Not Working properly as I am getting negative R Squared error I don't understand why ?? Should I decrease Alpha or No. of Iterations or is there any problem in my implementation what should I do?

My Regression plot looks something like below I don't know why I am getting such a line.

My Cost Function Error plot with respect to the number of iterations in Gradient descent looks something like below

R Squared Error is: -3.744682246118262

My Code Snippet:

import numpy as np
from sklearn.model_selection import train_test_split
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

def CostFunction(Theta,DataMatrix):
    Size = DataMatrix.shape[0]
    Error = 0
    for i in range(0,Size):
        Feature = np.vstack(([1],np.array(DataMatrix[i][:-1]).reshape(-1,1)))
        Error += (np.transpose(Theta).dot(Feature) - DataMatrix[i][-1]) ** 2
    return (1/(2*Size))*Error

def GradientDescent(Theta,Alpha,DataMatrix,Iterations):
    Progress = []
    Iterate = 0
    Size = DataMatrix.shape[0]
    Error = np.zeros((DataMatrix.shape[1],1))
    while(Iterations):
        for i in range(0,Size):
            Feature = np.vstack(([1],np.array(DataMatrix[i][:-1]).reshape(-1,1)))        #Last Entry is Label Thats Why
            Error += (np.transpose(Theta).dot(Feature) - DataMatrix[i][-1])*Feature
        Theta -= Alpha*(1/Size)*Error
        if(Iterations % 10 == 0):
            Progress.append([Iterate,CostFunction(Theta,DataMatrix)])
            Iterate += 10
        Iterations -= 1
    return [Theta,Progress]

def ProgressCurve(Progress):
    Progress = [[i[0],i[1].ravel()[0]] for i in Progress]
    sns.lineplot(x = np.array(Progress)[:,0],y =  np.array(Progress)[:,1],marker = '*')
    plt.show()

def Prediction(Theta,Test):
    Predicted = []
    for i in range(0,Test.size):
        Feature = np.vstack(([1],np.array(Test[i]).reshape(-1,1)))
        Predicted.append(np.transpose(Theta).dot(Feature))
    return Predicted

def Error_Metric(Actual,Predicted):
    Actual = np.array(Actual,dtype = 'float64').reshape(-1,1)
    Predicted = np.array(Predicted,dtype = 'float64').reshape(-1,1)
    Error = (Actual - Predicted) ** 2
    Variance = (Actual - np.mean(Actual)*np.ones((Actual.shape[0],1))) ** 2
    return (1 - np.sum(Error)/np.sum(Variance))

def RegressionLine(X,Y,Orig_X,Orig_Y):
    Y = [i[0].ravel()[0] for i in Y]
    sns.scatterplot(x = Orig_X,y = Orig_Y,color = "blue")
    sns.lineplot(x = X,y = Y,color = "red")
    plt.show()

X = 2*np.random.rand(1000)
Y = 4 + 3*X + np.random.randn(1000)
X_Train,X_Test,Y_Train,Y_Test = train_test_split(X,Y,test_size = 0.3,random_state = 0)
DataFrame = pd.DataFrame()
DataFrame['X'] = X_Train
DataFrame['Y'] = Y_Train
DataMatrix = DataFrame.as_matrix()
ThetaParams = np.random.randn(2,1)
Theta,Progress = GradientDescent(ThetaParams,0.001,DataMatrix,50)
Prediction_Out = Prediction(Theta,np.array(X_Test))
Error = Error_Metric(Y_Test,Prediction_Out)
ProgressCurve(Progress)
RegressionLine(X_Test,Prediction_Out,X,Y)
print(Error)

It seems like number of iterations is very small compare to learning rate. (50 vs 0.001). The optimizer can't converge when we look at the fitted line plot.

Try to increase number of iterations, it will help you. You can check this beautiful work and my another answer about the problem.


how does xgboost handle inf or -inf values? I am using xgboost for binary classfication. I have infs and -infs in my data due to the fact I am calculating ratios from one col and and another e.g. df[col1]/df[col2]. Since I have zeros and nans in these columns, it leads to me gettings infs, -infs, and nans.
I know xgboost can handle nan values, but if I replace the infs with a very large number e.g. 99999 or -99999 for -inf, is this an issue for xgboost? my understanding is that tree based methods for classification are unaffected by 'outliers'/inf values.
What would be best method to handle these?

Since (most) tree-based methods only care about the ordering of values in each feature, replacing your infinite values with very large values (larger than any finite value of the feature) is fine.  Of course, you'll have to do some thinking of whether df[col1]/df[col2] should actually be treated as $\pm$inf when col2 is zero, and nans can just be left in for xgboost.


learning rate very low 1e-5 for Adam optimizer good practice? I am used to of using learning rates 0.1 to 0.001 or something, now i was working on a siamese net work with sonar images. Was training too fast, overfitting after just 2 epochs. I tried to slow the learning rate lower and lower and I can report that the network still trains with Adam optimizer with learning rate 1e-5 and decay 1e-6. But I am curious if this is a good practice to use the learning rates so low?
I am using keras

I think that for the most part, the ends justify the means when it comes to learning rates. If the network is training well and you're confident that you're evaluating its generalization properly, use what works.

With that said, overfitting isn't usually caused by high learning rate.  It's hard to say with the information you've supplied here, but I suspect that your regularization isn't set up well enough, so using a high learning rate overfits the poorly regularized network rapidly, while training with a low learning rate doesn't get close to a local optimum because it can't move fast enough, but it's still heading towards an overfit optimum.

I'd check in on your regularization. What are you using?