Is there a possibility that there is no relationship between some inputs and outputs? I have a general question that comes to my mind, I'm doing machine learning projects and I took a look at many datasets and worked with, mostly there are already famous datasets that everyone uses.

Now the question comes to my mind, let's say I decided to make my own dataset, is there a possibility that my data are so random so that no relationship exist between my inputs and outputs ?? this is interesting because if this is possible, then no machine learning model will achieve to find an inputs outputs relationship in the data and will fail to solve the regression or classification problem.

Moreover, is it mathematically possible that some values have absolutely no relationship between them? in other words, there is no function (linear or nonlinear) that can map those inputs to the outputs.

Now I thought about this problem and concluded that if there is a possibility for this, then it will likely happen in regression because maybe the target outputs are in the same range and the same features values can correspond to the same output values and that will confuse the machine learning model.

What are your thoughts on this? did you had this problem in your daily life as a machine learning engineer, data scientist or hobbyist?

Yes and no

The YES

Let's consider at two distributions $F_X(x)$ and $G_Y(y)$ whose joint distribution is simply the product of the the two distributions: $H_{X,Y}(x,y) = F_X(x)G_Y(y)$. This means that the distributions are 100% independent: not just linearly (no correlation), but that there is absolutely no dependence between the two distributions. (Copula, Sklar's theorem, etc.)

The NO

For the above scenario to happen, there has to be absolutely no dependence between the two distributions. At the population level, this is calculated by Hoeffding's test of independence.

$$\theta =\int \bigg{[}H_{X,Y}(x,y) - F_X(x)G_Y(y)\bigg{]}^2 dH_{X,Y}(x,y) = 0$$

For empirical data, this would be tested with the hypothesis test $H_0: \theta=0, H_a: \theta\ne 0$.

Remember how we never accept a null hypothesis of equality because we know that it not quite true?

The YES (again)

It can be the case that, for all practical purposes, the distributions are unrelated. Yes, maybe if a butterfly flaps its wings in Ecuador it will be more likely that there will be an earthquake in Madagascar...but probably not much of an effect. This corresponds to $\theta\ne0$ but $\theta\approx0$ (equal for all practical purposes).


YOLO Pre-trained weights using Open Images Dataset V4 I want to use an object detection model for some use case.

I started with the YOLOv3, because I need to be able to perform detection on multiple images in less than few seconds (Even if it is not for real-time / video).

So, I successfully loaded and generated predictions using a keras model with pretrained weights.

The problem is that the pre-trained weights for this model have been generated with the COCO dataset, which contains very few classes (80).

What really surprises me is that all the pre-trained weights I can found for this type of algorithms use the COCO dataset, and none of them use the Open Images Dataset V4 (which contains 600 classes).

Is there a reason for that ? And if there is not, is there a place where I can find such pre-trained weights ?

Darknet has a yolo implementation using Open Images check this page (scroll down up to the end, just before the citation)

You can simply use
wget https://pjreddie.com/media/files/yolov3-openimages.weights


First two principal components explain 100% variance of data set with 300 features I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans.
My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and lemmatized
I create a data set of about 1.2 million tweet vectors (300 features each) by taking the averaged word vectors multiplied by their tfidf scores, like so:
# trained with same corpus as tfidf
# size=300, epochs=5, and min_count=10
tweet_w2v = Word2Vec.load('./models/tweet2vec_lemmatized_trained.model')

tweet_tfidf = TfidfVectorizer()
with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    tweet_tfidf.fit(infile)

tweet_tfidf_dict = dict(zip(tweet_tfidf.get_feature_names(), list(tweet_tfidf.idf_)))

tfidf_tweet_vectors = []

with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    for line in infile:
        word_vecs = []

        words = line.replace('\n', '').split(' ')

        if len(words) == 0:
            continue

        for word in words:
            try:
                word_vec = tweet_w2v.wv[word]
                word_weight = tweet_tfidf_dict[word]
                word_vecs.append(word_vec * word_weight)
            except KeyError:
                continue

        if len(word_vecs) != 0:
            tweet_vec = np.average(np.array(word_vecs), axis=0)
        else:
            continue
        tfidf_tweet_vectors.append(tweet_vec)

I also tried the above code with just average tweet vectors (no tfidf), and my problem still ended up happening.
I am starting to think that maybe my data set just isn't big enough or I am not training my word2vec model properly? I have somewhere around 100 million tweets I can use, but after filtering out retweets and only getting english language, it comes to around 1.3 million.
I'm not sure what's happening and what step I should take next. Any explanation is appreciated.
# Load in the data
df = pd.read_csv('./models/tfidf_weighted_tweet_vectors.csv')
df.drop(df.columns[0], axis=1, inplace=True)

# Standardize the data to have a mean of ~0 and a variance of 1
X_std = StandardScaler().fit_transform(df)

# Create a PCA instance: pca
pca = PCA(n_components=20)
principalComponents = pca.fit_transform(X_std)

# Plot the explained variances
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_, color='black')
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)

So the question is asking why the first two principal components of your encoded text data is encapsulating all of the variation in the data.
One potential issue could be the averaging over word vectors.
Suppose for a particular feature across word vectors for a particular post f, there could be an array of positive and negative values. When we then apply an average over f we could zero out the dimension and thus cause greater data sparsity, which could explain for what you are seeing (this zero value will exist regardless of whether you multiply this average with the td-idf or not). It could be the case that this sort of thing is happening across multiple dimensions in your text embeddings / feature vectors.
With this, you might need to think of another way of deriving a text embedding, maybe used Doc2Vec instead, which follows the same principles as Word2Vec, but instead derives document embeddings, which encapsulates the meaning of a section of text instead of word embeddings, which encapsulates the meaning of an individual word within a section of text.


Recommender system that connect users with each other , should I go for content based or collaborative filtering? I am trying to build a system where user come on the platform and he chooses a topic(predefined few topics) and then we connect him with any random online user who chooses the same topic. Then they can do conversation.

Now, I am trying to connect them smartly based on user's historical data (users with whom he had match earlier along with time duration of their conversation, and raing after the conversation etc). and his basic profile data.

How can I use collaborative filtering here, because I don't have any product here and their are very few users available online(10-15) at any time so I have to connect any one of them.

Thanks in advance!

You can use collaborative filtering with implicit features. However, I would first start with an even simpler approach. Maybe you could start using a distance metric, such as cosine similarity, or search for nearest neighbours using KNN.


Does MLPClassifier (sklearn) support different activations for different layers? According to the documentation, it says the 'activation' argument specifies:

  "Activation function for the hidden layer"

Does that mean that you cannot use a different activation function in different layers?

One can see from the code (look at uses of self.activation) that the same function is used for all the hidden layers.
You might want to consider the comments to this question for alternative approaches, generally being a move away from sklearn and towards a deep learning framework.


Setting batch size: training requires twice as much memory as validating I am using Keras with a Tensorflow backend to train an Image Classification model on a GPU. I have read somewhere that training uses roughly twice (both forward and back props) the GPU memory of validating, so therefore the training batch size should be the half of the validation batch size.

However, on many blogs and tutorials, I see that people use the same batch size for training and validating.

Is it true that training uses twice the GPU memory, because of the forward and backward pass, or is this false?

Aside: Tensorflow-GPU currently grabs 100% of GPU memory by design -- so that may may change the challenge a bit.

To add to @hssay on point answer -- with images, it is helpful to look at the entire pipeline.  Optimize the input pipeline, then you will use less memory.  May want to start here: TensorFlow

Test, test test.  You may find that your results do not really change much with smaller image files (less resolution, color, etc).  In some cases, results may even get better (the overtraining/undertraining balance).

[this is really should be a comment, but my rep does not transfer here :( ]


Does BERT has any advantage over GPT3? I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). So am curious to know whether BERT scores better than GPT-3 in any particular area of NLP?
It's quite interesting to note that OpenAI's GPT-3 is not open-sourced whereas tech behemoth Google's BERT is open-sourced. I felt OpenAI's stance and the hefty price tag for GPT-3 api is in stark contrast to its mission statement(OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity).
https://analyticsindiamag.com/gpt-3-vs-bert-for-nlp-tasks/
https://thenextweb.com/neural/2020/07/23/openais-new-gpt-3-language-explained-in-under-3-minutes-syndication/
https://medium.com/towards-artificial-intelligence/gpt-3-from-openai-is-here-and-its-a-monster-f0ab164ea2f8

This article on Medium introduces GPT-3 makes some comparisons with BERT.
Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still holds the edge. It’ s trained-on challenges which are better able to capture the latent relationship between text in different problem contexts."
Also, in section 6 from the article, author lists areas where GPT-3 struggles. It may be that BERT and other bi-directional encoder/transformers may do better, although I have no data/references to support this yet.


What are the applications of Solr in Big Data? Why is solr used in Big Data?

What is it's purpose?

Solr is a highly scalable, fault tolerant search server.

So, you can store files in the form of JSON, XML, CSV or binary over HTTP.
And you can query it via GET and receive JSON, XML, CSV and binary results. It also has a distributed architecture for the purpose, which is the primary and most important concept in Big Data and it's handling.

So, as it is highly scalable and fault tolerant for high traffic and huge documents, it is a choice for a reliable search engine for huge documents, or in short Big Data.


How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)? I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per Iteration:
y_hat = X.dot(W) + b

dW = - (2 * (X^T ).dot(Y - Y_hat)) / m # how does the minus and matrix multiplications are used instead of Summation?
db = - (2 * np.sum(Y - Y_hat)) / m  # np is numpy

W = W - lr * dW  # update weights
b = b - lr * db

I know from the code is that dW is derivative of the weight matrix per iteration, X^T is the Transpose of the X features, Y is original values of Y , Y_hat are predicted values using the formula X.dot(W)+b.
What I want to know is that dW = - (2 * (X^T ).dot(Y - Y_hat)) / m. Even with the MSE loss, as given in this link in the equation 1.4, it should be something else.
Can someone please elaborate how are the values of dW,db are calculated here?
Whole Python Code is giveb here for Linear Regression

dW and db are simply the derivative of the loss function with regards to the weights and biases. Given the loss function
$J = \frac{1}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))^2$
the derivatives of the loss to the weights (dW) and bias are equal to
$\frac{\partial}{\partial W} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i)) * x_i$
$\frac{\partial}{\partial b} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))$
As you can see, these equations align with the code you provided. Equation 1.4 from the link you provided is for $\theta_0$, i.e. the bias, which is the same as the code for db (with the only difference being the minus sign, causing by the fact that $y_i$ and $h(x_i)$ are swapped in the loss function between the first and second article).


What is the best way to calculate score based on input features? Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score should reflect whether it is in shortage or not. Shortage is when there is high demand (Sales) and low inventory/supply. Score of 100 means it is in shortage. Score of 1 means it is not in shortage.
Right now i use a random forest model and use the probability from the model as the score. However this leads to a lot of products having score between 0-10 and 90-100. Very few products have score between 10-90.
Is there a better way to do this so that the scores are more distributed?

I assume that currently you're training a binary classification model, right?
You could try training a regression model which predicts the score between 0 and 100. It would be better to have some examples in your training data which are between 10 and 90, because that would make the model learn the distribution of the scores. But even if you don't have this kind of instances, it's possible (not sure though) that the model will predict instances in this range in some cases.


Choosing an optimizer to perfectly fit a neural networks to training data In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network if she wants perfect fitting (mae < 1e-04) on the training data?

Details: In my particular case, the input of the function is 60-dimensional and the output is 1-dimensional (the training data set is prepared by solving a forward model). The input and output are normalized and the sample mean is converted to 0. My neuron's activation function is tanh (this gave me a better result than ReLu and Sigmoid). I have so far used Keras's "adam" and "sgd" as optimizers and I have tried with various learning rates.

In addition, I have tried with increasing the number of neurons in each hidden layer and increasing the number of hidden layers as well. At some point, the total number of my trainable parameters was more than 100 million.

However, even for trying different batch sizes and 10,000 epochs, my best mean absolute error (mae) was never below 0.07. The only part I haven't yet experimented much is to customize/use advance optimizers. I am completely clueless why the neural network can't find a set of trainable parameters where, at least, it can over-fit the training data?

Any suggestions from the experts? Thanks in advance for your time and patience. I really appreciate your support.

An example code:

input_size = ND
nodes = 10000
inp = Input(shape=(input_size,),name='Input')
l0 = Dense(nodes, activation='tanh',name='Level0')(inp)
l1 = Dense(nodes, activation='tanh',name='Level1')(l0)

lo = Dense(1, activation='tanh',name='Level_out')(l1)

merged = Model(inputs=inp,outputs=l10)
opt = optimizers.adam(lr = 1e-5)
merged.compile(optimizer=opt,
              loss='mean_squared_error',
              metrics=['mae'])

There's no free lunch, only bunch of tips. Your search space always depends on the problem you're tackling, and there's no one that fits them all. You need to run experiments to check this. Additionally, your net's architecture is one of the parameters, so it's hard to tell if your net will even be of sufficient capacity, without knowing the exact functions, you want to approximate. At first glance, optimizer is last thing you should worry about.


What is new population in genetic algorithm? Here is my (mis?)understanding of genetic algorithm:

Create n individuals. This is initial population
Calculate fitness of each individual in this population
for i in range(n):
    select two individuals randomly with replacement from population with
    probability of selection of each individual proportional to its
    fitness value
    cross over the two selected individuals to create two offsprings
    introduce mutation
Calculate fitness value for each offspring

Now you have an initial population of n and n offsprings.

What comes now?

Is my new population the n offsprings or the n fittest individuals of the 2n (n population + n offsprings) available?

In the simplest implementations, you simply discard the old population and maintain a population of size $n$ in each generation.

There is lots of variation on this though, any of which could be useful in practice. As a start to show the range of possibilities, you can do some or all of the following:

Keep a top number (say $m$) of the original population, and only generate $n-m$ new offspring in each generation.
Generate offspring in small groups and assess them. They must beat a member of the current population on the fitness measure in order to find a place (this can work nicely in round-robin tournaments, where the fitness measure is winner of a game).
Keep snapshots every so many generations (e.g. every 10 generations) of the whole population or just the best performing individuals. You may use these snapshots to assess long-term improvement or as a "gene pool" to keep good performers around in some form for a while longer than a single generation.

Variations like these are hyperparameters for your GA. They might help in certain circumstances, or be a hindrance in others. So start with the simple "everything is replaced" is recommended initially, especially if you are learning genetic algorithms.


How to learn to score new documents based on a existing set of scored documents? I have a 50 000 documents of 1000 words or more ranked between 0 and 2000. They all deal with a similar topic. I'd like to create an algorithm that can learn to score new documents.

What approach do you think I should take?

I am a newbie in the field of machine learninig so if you could point me to introduction material about the particular solution you think of. I will be glad.

First come up with some features for the document. Stuff like frequency of some popular words associated with that topic might work. In this way get the features for all the documents and then apply some algorithm. Some ways you could apply them are:

1) k means - cluster the documents on the basis of the features. Each cluster should be predominantly associated with a particular score value. Then see which cluster a new document will be assigned to.

2) Supervised learning - use neural networks, multiclass SVMs etc to classify the new document to a particular class (score) using the model you would have generated.

All of these are examples of classification to a discrete score value. However, since you are dealing with a large score range (0-2000), you could also try something like regression which will give you a continuous value, but could be rounded off to the nearest discrete one.

Check out the Coursera course on Machine Learning for a great introduction!


How does "sameModel" mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs? I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline.  From the example, the model is fit using a pipeline (val model), then the pipeline is saved to a directory.  The next line is

  val sameModel =
  PipelineModel.load("/tmp/spark-logistic-regression-model")

I don't see how/where sameModel is being used.  I see model being used again on test data.  I would've expected the example to use sameModel to show how to invoke the loaded pipeline back into the process. Does sameModel automatically update model?

Any insight would be appreciated as I am interested in saving a pipeline following a fit then load it at a later point, but am having issues with:

Finding good examples saving & loading Spark ML pipelines, then invoking said loaded pipeline
Not fully understanding the examples I see online

Any assistance in clarifying what is going on and how to move forward would be greatly appreciated.  Thanks!

It appears that "sameModel" is just an example of loading the model back without any influence on "model" from the docs.


Taking Neural Network's false positives as the recommendation system result? I am creating a recommendation system and considering two parallel ways of formalizing the problem. One classical, using proximity (recommend the product to the customer if a majority vote of 2k+1 customers closest by has the product), and another one that I have trouble understanding but seems valid to some extent.

The approach I'm thinking about is:

1) Fit a highly regularized neural network (to make sure it doesn't overfit the training set) for a classification task that can predict if the person does or doesn't have given product

2) Make sure test accuracy is as close to train accuracy as possible

3) Take false positives (customers who don't have the product originally but the NN predicted that they have it) predicted on the whole dataset (the training set as well) as the result - the people I should recommend the product to

Now, I am aware of why in general one wouldn't want to take that approach but I also can't exactly explain why it wouldn't return people 'close by' to each other that 'should' have given product in a similar sense like the KNN-based approach. I'm not sure how to analyse this problem exactly to validate, modify or reject the idea altogether.

1) Is regression, not classification.

You should try to think of it from a decision tree perspective instead of NN. Your attributes for a customer are all products of this customer and all additional information about that customar.

Since the task is regression, all leafs of the decision tree will have the label 0 (if the decision path leads to the statement that the customer doesn't have the product) or the label 1 if the decision path leads to the statement that the customer has the product.

When building the decision tree, we start at the root where we have to decide which attribute we will use as the first node from where all the customers will take the corresponding branch depending on their value of the chosen attribute. This attribute is choosen by using entropy to calculate the attribute with the highest information-gain value. This simply means that we choose the attribute, that divides the customers into sets where each set has ideally the same amount of customers with 0 and 1.

This is repeated for each new node that branch from the parent node as long as there is an attribute left with information-gain above a threshold value or when the tree becomes to deep (since it is NP-complete).
Now we count at each leaf the amount of zeros and ones and label the leaf as max(0,1).
The conclusion of this: if a customer landed at a 1-leaf although he was 0 means, the highest information-gain attributes have the same values for this customer as they much more often do for customers who are 1.


Adding dense layer to CNN causes to stop learning I have 6 layer CNN model that (kind of) works with "GlobalAveragePooling2D()" in the end but instead if I flatten and add dense layers like below, it just does not learn. After 20 epochs loss and accuracy curves still doesn't change at all.

Do you think anything wrong the way I flatten or add dense layers in the end? I also printed out the shapes incase it helps.

model.add(Ativation('relu'))
model.add(BatchNormalization())
print("Conv6 after shape",model.output_shape)

model.add(Convolution2D(filters=64, kernel_size=(4,4),
                    strides=(2,2),
                    padding="same",
                    kernel_regularizer=l2(0.0001),
                    kernel_initializer="normal"))
model.add(BatchNormalization())
model.add(Activation('relu'))

print("before flat shape",model.output_shape)
model.add(Flatten())
model.add(Dropout(0.5))

print("after flat shape",model.output_shape)
model.add(Dense(200, activation='relu'))
print("after dense shape",model.output_shape)
model.add(Dense(2, activation='softmax'))

And here is output of full model:

0 shape (None, 1, 44100, 40)
1 shape (None, 1, 44100, 40)
2 shape (None, 1, 160, 40)
Conv3 after shape (None, 40, 160, 24)
Conv4 after shape (None, 40, 160, 24)
Conv5 after shape (None, 20, 80, 48)
Conv6 after shape (None, 10, 40, 48)
before flat shape (None, 5, 20, 64)
after flat shape (None, 6400)
after dense shape (None, 200)

Having dropout after convolution is rather unusual. Try to put it between the dense layers.

Explanation:

Dropout forces neuron to work with a random subset of the preceding neurons' activations. By doing so it breaks the co-adaptation of neurons and improves the generalization of the network. However, for convolutional layer co-adaptation is not an issue, as neighboring pixels of a feature map share their weights and guarantee to have different activations given different inputs in their receptive fields. It may only occur if two layers will learn the same kernel weights, but even then the problem is not so significant. Little number of weights in a convolution layer is a pledge of good generalization.

Here is a short article showing what happens when you use batchnorn/dropout between convolutions: Don’t Use Dropout in Convolutional Networks


Plots of data using DBSCAN algorithm not making sense I am using clustering for my data. Since the DBSCAN algorithm will also tell me an estimate of clusters that I can use, I have used DBSCAN. I have tried for the eps=[0.123,1,2] and min_smaples=[2,10,...60]. The print satatement in the below code prints 714 which is equal to the number of data-samples(rows). The code looks like:

dbscan = DBSCAN(eps=1, min_samples = 4)
clusters = dbscan.fit_predict(df)
print(len(clusters))
plt.scatter(df.iloc[:, 0], df.iloc[:, 1], c=clusters, cmap="plasma")
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")

Sample plots for some of the different min_samples is shown below:

The parameters for the above plots are given below in the same pattern.

EPS=1, MIN_SAMPLES=2, EPS=1, MIN_SAMPLES=10
EPS=1, MIN_SAMPLES=20, EPS=1, MIN_SAMPLES=40

None of these plots are making sense to me from a clustering perspective. From this, I am forced to conclude that I cannot use clustering for the given data or I am doing it wrong. So, I need help with insights into the weird appearance of the above plots.

Any help is appreciated

Dont try to visually confirm it.

You are plotting your clustering resutls in ONLY two dimensions and you expect that all of the information is in these two dimesnions. That is very unlikely. If you plot 3 dimensions you will see even  more seperability and it will make a bit more sense. In any case you need a metric for example Silhouette that tells you how well you clustered. Visualisation is just a sanity check if you know your features already.


How to deal with a machine learning model which affects future ground truth data? I wonder if a binary classification problem which will steer a business process really is binary. As you see steering means affecting the outcome of the business process. As such ground truth data will be different than it would have been without the model.

What strategies are there to deal with this problem (besides a hold out group).
Recently I read http://ieeexplore.ieee.org/document/7280527/ wich additionally incorporates early feedback.

Is it suitable to assume that after initial launch of such a model which predicts 0 and 1 class labels future ground truth data wich is returned with a delay of up  to 30 days will in reality be up to 6 classes:

0 prediction of label 0
1 prediction of label 1
feedback that label was actually

0 for a prediction of 0 (regular outcome)
1 for a prediction of 1 (regular /desired outcome)
0 for a prediction of 1 (model did falsely classify, groundtruth affected)
1 for a prediction of 0 (model did not detect classification, suboptimal and lost some money but ground-truth considered not affected)

How could the feedback be incorporated? Would you suggest to expand the binary classification to a multi-class classification?

One option is reinforcement learning (RL).
RL frames the problem as sequential decision-making under uncertainty. There is an agent making decisions and after the decision is made, the agent collects new data. The next round of decisions uses the new data.
What you are describing is a scenario where the distribution of "ground-truth" changes over time. Many statistical and machine learning models assume a stationary process where the underlying distribution does not change over time. Some types of RL does not assume a stationary process.


Can you apply PCA to part of your dataset? I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them are weakly correlated with the response variable but highly correlated with each other. I am trying to apply PCA to this part of the data and glue them back together as columns with the categorical variables. Is it ok to do so? Or should I one-hot-encoding / label encoding the categorical variables and do pca to the entire dataset?

I plotted the heatmap for the 14 continuous variables and found that most of them are weakly correlated with the response variable but highly correlated with each other

You absolutely can select specific columns [continous data] from your original data and apply PCA on them, PCA1, PCA2 eigenvectors will show you the amount of correlation between each feature. However, you should use all of the data points or rows when applying PCA as PCA calculates the maximal variance between data points and its best to use all of them for accurate results.

So in short, you should select column[feature] wise but not row[data points] wise.

  Is it ok to do so? Or should I one-hot-encoding / label encoding the categorical variables and do PCA to the entire dataset?

No need to do this, and it doesn't make sense in case of PCA as it only works on continuous data points.


Loading models from external source I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments.

I thought of using another repository that contains all the models and then fetch them in running time.

Does anybody know a clean approach or alternative?

In most cases, you would use a file-storage solution such as Amazon S3 or Google Cloud and many others, which provide designated solutions for large object storage and retrieval.

You would then ideally want to update your code to retrieve the model directly from the file storage. Whether this download needs be done on every run or only once (storing the model locally for future runs), should be decided based on your specific needs.


clustering with k means I have dataset with two label class (good and bad), I want to apply K Means on my dataset using python, should I use that label dataset or I have to delete the label class column ?

Delete the label column.

Assuming that you want to compare the clusters to the labels later, then the labels must not be part of the data passed to k-means.

And k-means only works well on continuous variables anyway.


Why neural networks models do not allow for multiplication of inputs? In a neural network, each neuron value is multiplied by the weight of the connection. Then, each neuron's input is the sum of all those values, on which we apply the activation function (sigmoid, relu, tanh, etc.).

Why can't the input be a vector of all those values, so that I can decide to multiply them if I want, or apply another function from the vector space to the reals, and then apply the activation function on that real value? Any implementation I have seen does not allow this, and if I wanted to do it, it seem I would have to implement my whole neural network architecture by myself without the possibility of using Theano, Keras or other libraries. Is there a reason that escapes me of why this would not work? And if there isn't, is there any library out there I could use that already does this without having to create it from scratch?

Using a low-level library, such as Theano or TensorFlow, it is likely that you can construct new schemes for reducing tensors (maybe via some learnable weight vector etc). In TensorFlow, you also get automated gradient calculations, so you should be able to still define a cost function and use existing optimisers, without needing to analyse the new design yourself or re-write back propagation formulae.

The universal approximation theorem essentially states that in order to have a network that could learn a specific function, you don't need anything more than one hidden layer using the standard matrix multiplying
plus non-linear activation function. So, invention and use of new architectures needs some reason. The reason is usually that these additions improve the speed or scope of what can be learned, they generalise better from training data, or they model something from a problem domain really well.

No doubt there have been explorations of all sorts of variations to the standard NN model over time. The ones that have become popular have all proven themselves on some task or other, and often have associated papers demonstrating their usefulness.

For example Convolutional Neural Networks have proven themselves good at image-based tasks. The design of a 2-dimensional CNN layer has a logical match to how pixels in an image relate to each other locally - defining edges, textures etc, so the architecture in the model nicely matches to some problem domains.

If you can find a good match from your vector multiplication model to a specific problem domain, then that's an indication that it may be worth implementing in order to test that theory. If you just want to explore other NN structures to see whether they work, you can still do so, but without a specific target you will be searching for a problem to solve (unless by chance you stumble upon something generally useful that has been overlooked before).

To address the question in the title:

  Why neural networks models do not allow for multiplication of inputs?

Some low-level ones do (e.g. TensorFlow). However, this is not an architecture that has proven itself useful, so authors of higher-level libraries (e.g. Keras) have no reason to implement or support it. There is the possibility that this situation is an oversight, and the idea is generally useful. In which case, once someone can show this you will find it would get support across actively-developed libraries pretty quickly since it seems straightforward to implement.


Multivariate time series forecast with VAR confusion I am new to time-series forecasting. I am working now on a task in which I have a data set, containing samples of approx. 15 variables for every hour for several years. Then, I have a test data set (continues at the next time step where training data ended) containing values for all the variables except one. My task is to build a model using training data that can predict that one variable in the test data set.
From reading online, I understood I could use vector autoregression (VAR). I have read many tutorials such as this one. I understand most of it except one thing. When it comes to predicting, they (in the tutorials) predict all the variables. However, I would like to do something different: I would like to predict just the one target variable. And of course take into account values of the other variables in the test data set.
To illustrate this, let's say Var Z is my target variable and this is my training set:
       Var X    Var Y     Var Z
 Day 1     11       20       30
 Day 2     22       40       60
 Day 3     33       60       90

Then this is my test set for which I want to predict Var Z:
       Var X    Var Y     Var Z
 Day 4     44       80       ??
 Day 5     55       84       ??
 Day 6     66       88       ??

But in the tutorials I have seen so far, they always predict all variables!
Question: How to specify I want to forecast only a single variable for certain timestamps and take into account values of other variables at those timestamps? Is VAR not the right tool to use?
I would be most grateful if someone could point me in the right direction. I use Python.

In essence, you want to both incorporate the past historical values of the target timeseries and the (past and) current historical values of other timeseries to predict the current value of the target timeseries.
In the tutorial you gave, they define the VAR[1] model with two time series ($Y_{1, t}$, $Y_{2, t}$) as:
$$ Y_{1,t} = \alpha_{1} + \beta_{11,1}Y_{1,t-1} + \beta_{12,1}Y_{2,t-1} +\epsilon_{1,t}$$
$$ Y_{2,t} = \alpha_{2} + \beta_{21,1}Y_{1,t-1} + \beta_{22,1}Y_{2,t-1} +\epsilon_{2,t}$$
where the author defined $\alpha$ as the intercept, $\beta$ the lag coefficients and $\epsilon$ as the error term.
Let us consider $Y_{1,t}$ as the target time series. It seems that it could be modified such that it incorporates the past and current values of $Y_{2,t}$ as follows:
$$ Y_{1,t} = \alpha_{1} + \beta_{11,1}Y_{1,t-1} + \beta_{12,1}Y_{2,t-1} + \beta_{12,0}Y_{2, t} + \epsilon_{1,t}$$
Here, the term $\beta_{12,0}Y_{2, t}$ has been added, which signifies the dependence on the current value of the other time series. In fact, this equation is the only relevant one when determining the $\beta$ parameters, as $Y_{2,t}$ is known in the test set.
Generalizing this to an arbitrary number of time series $n$ and lag $l$, the model for your target timeseries becomes:
$$ Y_{1,t} = \text{VAR}[l] + \sum^{n}_{i=2}\beta_{1i,0}Y_{i, t}$$
with VAR$[l]$ indicating the original VAR model from the article.
EDIT:
Worked example based on sample data you provided. We have time series $X,Y,Z$. Let us model our time series with lag = 1. Then our model for $Z_{t}$ becomes:
$$Z_{t} = (\alpha + \beta_{ZZ, 1}Z_{t-1} + \beta_{ZY,1}Y_{t-1} + \beta_{ZX,1}X_{t-1} + \epsilon_{t}) + \beta_{ZY,0}Y_{t} + \beta_{ZX,0}X_{t}$$
The values between brackets is standard vector autoregression. We can then use a simple linear regression procedure. Note that we only have two samples to use for fitting, because at Day 1 there are no previous timepoints. Thus we have the following equations:
$$Z_{2} = 60 = (\alpha + \beta_{ZZ, 1}Z_{1}+ \beta_{ZY,1}Y_{1} + \beta_{ZX,1}X_{1}) + \beta_{ZY,0}Y_{2} + \beta_{ZX,0}X_{2}) = (\alpha + \beta_{ZZ, 1}30 + \beta_{ZY,1}20 + \beta_{ZX,1}11) + \beta_{ZY,0}40 + \beta_{ZX,0}22)$$
$$Z_{3} = 90 = (\alpha + \beta_{ZZ, 1}Z_{2}+ \beta_{ZY,1}Y_{2} + \beta_{ZX,1}X_{2}) + \beta_{ZY,0}Y_{3} + \beta_{ZX,0}X_{3}) = (\alpha + \beta_{ZZ, 1}60 + \beta_{ZY,1}40 + \beta_{ZX,1}22) + \beta_{ZY,0}60 + \beta_{ZX,0}33)$$
Using python we can solve this as follows:
from sklearn.linear_model import LinearRegression

#Z-vals
Z_1 = 30
Z_2 = 60
Z_3 = 90

#Y-vals
Y_1 = 20
Y_2 = 40
Y_3 = 60

#X-vals
X_1 = 11
X_2 = 22
X_3 = 33

LR = LinearRegression()

result = LR.fit([[Z_1, Y_1, X_1, Y_2, X_2],
                [Z_2, Y_2, X_2, Y_3, X_3]],[Z_2, Z_3])

print(f'Alfa is {result.intercept_}')
print(f'Parameters are {result.coef_}')

which yields
Alfa is 21.95159629248201
Parameters are [0.46343975 0.30895984 0.16992791 0.30895984 0.16992791]

And thus our predictive model for Z has become (after rounding):
$$Z_{t} = 21.95 + 0.46Z_{t-1} + 0.31Y_{t-1} + 0.17X_{t-1} + 0.31Y_{t} + 0.17X_{t}$$


What is parts of speech technique in sentiment analysis? In an article, I saw Sentiment Analysis using Parts Of Speech(POS) technique. When I searched I got some paper on POS but I couldn't understand what POS basically is. Though I am new to sentiment analysis please help me to understand POS.

Parts of Speech (POS)

This is what it is called when you label each of the words (often called tokens) of a sentence or many sentences. Usually they are labelled with grammatical descriptions, such as Noun, Adjective, Adverb. They can often get quite specific, also distinguishing e.g. between types of nouns (proper nouns etc).

You can then use these descriptions of the tokens as input to a model or to filter the tokens to extract only the parts you are interested in.

POS are usually parts of the output when we parse a block of text using an NLP toolkit, such as spaCy. Have a look here for their available POS.

Here is a snippet of  parse tree of the sentence: Apple is looking at buying a UK startup for $1 billion.

Apple has been recognised as a proper noun (NNP) as well as being the subject of the first verb (shown by the arrow labelled nsubj).

For a nice introduction to POS among many other terms within NLP, check out this article..

Sentiment Analysis Perspective

There are many many reasons to include POS in a sentiment model (some examples below), but they really all boil down to one overarching reason: polysemy. The definition of which is:

  the coexistence of many possible meanings for a word or phrase.

So essentially saying, that words in different contexts can have different meanings. This is of course a massive gain in information that we can pass to a model!

The word duck can be a noun (the bird) or a verb (the motion, to crouch down). If we can tell a model which one of these it is in a given sentence, the model can learn to make a lot more sense out of the sentence.

Beyond distinguishing between meanings of single words, we can also simply uses them on their usage, or placement. One example use would be to use the adverb: however.

If our parser is good enough to tell us that it used in a particular sentence as a contrasting conjunction (which technically, would be grammatically incorrect!). An example sentence could be:

  I really love muffins, however, I hate strawberries.

We have two clauses: a positive one before however and one after. The first clause is positive, the latter negative. If we have a scale of -5 ro +5 for sentiment for each clause (perhaps the mean of each word in that clause) we could imagine scores such as +3 for the positive clause and -3 for the negative.

This is where I have seen some models (Vader, SentiStrength, etc.) using POS to scale those base scores. In our example, perhaps however would be used to increase the magnitude of the negative clause's score by 10%, giving it a final score of -3.3. Whether or not that makes sense depends on the use case, the data and probably the developers general experiences.

Summary

There are many uses for POS, you can imagine quite a few, whether to hand-tailor a sentiment model of just to produce more features. In any case, it is a process that extracts more information from the original raw text, applying langage models (like grammar!) that have been tested and are known to be robust for any official form of writing.


Can I forecast with discontinued data using ARIMA? I have data for sales on monthly basis, but a few months' information is not in the CSV file or data file. Can I forecast or fill that missing month with other calculated values from present records?

Part of the code I am using:

AIC = []
SARIMAX_model = []
for param in pdq:
    for param_seasonal in seasonal_pdq:
        try:
            mod = sm.tsa.statespace.SARIMAX(train_data,
                                            order=param,
                                            seasonal_order=param_seasonal,
                                            enforce_stationarity=False,
                                            enforce_invertibility=False)

            results = mod.fit()

            print('SARIMAX{}x{} - AIC:{}'.format(param, param_seasonal, results.aic), end='\r')
            AIC.append(results.aic)
            SARIMAX_model.append([param, param_seasonal])
        except:
            continue
print('The smallest AIC is {} for model SARIMAX{}x{}'.format(min(AIC), SARIMAX_model[AIC.index(min(AIC))][0],SARIMAX_model[AIC.index(min(AIC))][1]))

# Let's fit this model
mod = sm.tsa.statespace.SARIMAX(train_data,
                                order=SARIMAX_model[AIC.index(min(AIC))][0],
                                seasonal_order=SARIMAX_model[AIC.index(min(AIC))][1],
                                enforce_stationarity=False,
                                enforce_invertibility=False)

I ran into a similar problem and this worked for me:
#First I resampled the data to have all the months
train_data = df['sum of received quantity'].resample('MS').mean()
#Then I filled the Nan values
train_data = train_data.fillna(train_data.bfill())


Bounds for MLP hyperparameter search I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not exploring (e.g. I might be getting trapped in a local minima for hyperparams).

Essentially, I'd like some standard bounds for hyperparameter search. Ideally, if a person were to see the breadth of explored parameters, they might be reasonably certain to try another class of machine learning models. In particular, I am looking for concrete advice on at least

What size neurons to include in the search

The number of hidden layers, and the width of the layers
The
optimizer and learning rates
Activation functions and loss functions

Thank you so much!

In designing an MLP architecture, we can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume recommended ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers (see ipython notebook for comparison). Your loss function will depend on the problem: cross-entropy for classification and MSE for regression.

In order to avoid local optima in parameter search, it's better to use random search or bayesian optimization. For additional answers, see this post.


Using LeakyRelu as activation function in CNN and best alpha for it Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write:

model.add(Conv2D(32, kernel_size=(3, 3),
           input_shape=(380,380,1))
model.add(LeakyReLU(alpha=0.01))

I mean now by the written lines, the activation function for Conv2D layer is set as LeakyRelu or not?

Further, I want to know what is the best alpha? I couldn't find any resources analyzing it.

The code,

model.add(Conv2D(32, kernel_size=(3, 3), input_shape=(380,380,1))
model.add(LeakyReLU(alpha=0.01))

will definitely transform the outputs from the Conv2D using the LeakyReLU activation given parameter alpha ( negative slope of ReLU ).

  Further, I want to know what is the best alpha? I couldn't find any
  resources analyzing it.

In ReLU, we simply set the activation to 0 for negative values. This causes the dying ReLU problem which leds to overfitting. Hence, we return $x \alpha$ instead of 0, so that the unit does not become non-functional.

If we look at TensorFlow's tf.nn.leaky_relu method, we will find that the alpha is 0.2.

Whereas in Keras' layers.LeakyReLU class, you will find the alpha is 0.3.

So you can clearly get an idea of what the parameter's value should be. It's basically a hyperparameter which you need to adjust through trial-error observations.


Bayesian optimisation in deeplearning Has anyone tried using Bayesian optimisation to get best learning rates, and other hyperparameters for deeplearning.

How to change the parameters between the training. Any examples on callbacks?

Can I see some codes to implement them?

Currently I am fiddling around with Ax/BoTorch to optimize a ResNet50 for Object classification (and later on detection via SSD). I found it quite hard to apply the samples from the docs to my own model, so this is a very first prototype. Just have a look at this GIST...

https://gist.github.com/pinkerltm/74f98485d1408849476fb02686d0c228

So far I can tell, that it is somehow optimizing the learning rate into the right direction, but I think I have to wire some more Hyperparameters properly and perhaps optimize longer to get a result which is compareable to simply defining the best parameters manually so far, as Accuracy with the optimized Learning rate only gets around 67% after several 100 epochs while at least 71% should be easily possible with this model.

I am also very interested in sharing experiences and findings on this. I will edit my answer here accordingly so that it should provide at least an simple example of optimizing a CNN Classification Network with Ax/BoTorch.


Does a matrix factorization recommendation engine use user/item related features? All the tutorials I can find about matrix factorization recommendation systems start with importing users, items, and user-item-ratings, but then only use the rating matrix to train the recommender (not features of the users or items themselves like "age").

Example

Tutorial: Matrix Factorization for Movie Recommendations in Python

URL: https://beckernick.github.io/matrix-factorization-recommender/

Users (userID, gender, ageGroup, occupationGroup, zipCode)

Items (movieID, title, genres)

Ratings  (userID, movieID, rating, timestamp)

Then we do some cleaning and build a matrix of users-to-items with ratings as the values.

Then we proceed to to only use the matrix built from ratings (not other features of the user like age which I might guess has some predictive value).

# normalize data
import numpy as np
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis=1)
R_demeaned = R - user_ratings_mean.reshape(-1,1)

# import numpy as np
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis=1)
R_demeaned = R - user_ratings_mean.reshape(-1,1)

# SVD
import scipy
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k=2)

# convert to diagonal matrix
sigma2 = np.diag(sigma)

# build predication matrix
all_user_predicted_ratings = np.dot(np.dot(U, sigma2), Vt) + user_ratings_mean.reshape(-1, 1)
preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns, dtype='float')

Theres more in the tutorial but at this point our predication matrix is already set.

I wondered if this is just how matrix factorization recommenders work (aka. not use user/item features) or if online tutorials are just too simple?
If I wanted to incorporate a user feature like age, would it be possible with matrix factorization?

If I wanted to incorporate a user feature like age, would it be possible with matrix factorization?

Yes, it's possible. It's commonly called a hybrid recommendation system.


How can I predict student enrolment in September based on independent data available earlier in the year? I'd like to predict how many students will enroll in a college in September, based on independent variables which are known earlier in the annual student recruitment cycle.

For example, one independent variable would be the number of accepted offers; I know how many students have accepted an offer from the college for each week from January to August for the last five years, and I know how many of these students actually enrolled in September (generally about 80% of students who accept an offer actually enroll, but this varies year-on-year).

Other independent variables might include: how many students have already paid a deposit on their tuition fee, or even the current exchange rate for students travelling from abroad.

So far, I've used historical conversion rates to predict enrollment, but I am wondering whether I can do something more sophisticated which would allow me to add in other independent variables.

Any ideas of which approaches I should investigate would be very welcome. I'd be particularly interested to hear of any Python libraries which might be relevant.

I would try out regression in Python's scikit-learn library to predict the September headcount given all those other variables you have. Here is a basic example using a linear model. Once you have that working, you could try a more sophisticated algorithm.

http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html


Binary classification problem with imbalanced dataset, how to compare to random classifier We have a very imbalanced dataset (2% of class 1). To the best of our knowledge, there is no baseline in the literature to the problem we want to solve - so we thought of comparing our performance to a random classifier. We evaluate our model as a combination of precision and recall - we vary the threshold at which data points are classified as 1 and compute the rolling threshold and recall. We could use F1-score as well.

What would be an acceptable way to define a random predictor that we can compare to our model such that the comparison is as fair as possible?

Since you are interested in different decision thresholds, your random model should produce scores.  In that case, a reasonable base-line model assigns a score uniformly at random in $[0,1]$.  Such a model will, at threshold $t$, have

$$\begin{align*}
\operatorname{precision} &= \frac{2\%\cdot N\cdot (1-t)}{N(1-t)} = 0.02,\\[1em]
\operatorname{recall} &= \frac{2\%\cdot N\cdot (1-t)}{2\%\cdot N} = 1-t.
\end{align*}
$$

(Perhaps a very simple model will serve as a better baseline.)


Is there any way to calculate the true,false positives and negatives for a regression problem I am trying to predict the glucose values of the patients for example values like 45,256,115 etc. based on some features. Currently I am calculating the accuracy in means of RMSE,MSE,R². Is there any way to calculate the accuracy in means of a confusion matrix by setting a threshold value like ±10 for the value predicted. For example if the actual value is 110 and my prediction is 100 then we can tell that's it's kind of accurate since 100+10=110 where 10 is the threshold value. Any suggestions and some code to elaborate it will be very helpful.

I believe that there's no actual function in sklearn library that does that. However, that concept is relatively easy to implement: you can loop and compare each y_prediction with the respective y_true_value and increment TN, TP, FN, and FP accordingly to a certain threshold. For example:

if abs(y_pred[i] - y_true[i]) <= threshold:
     TP += 1
 elif abs(y_pred[i] - y_true[i]) > threshold:
     TN += 1
 ...

Don't forget to reverse transform the scaling on y_true and y_pred values first.


How to train a Model which can check whether image is from existing classes or not I have given with a image dataset of 1000 classes , each class has 100 images. Now My requirement is to train a model which will take a image as input, and it should answer whether the image is present or not, there is no need to specify which class it is.. It just has to say yes or no.

I thought of using a CNN with a sigmoid function at last layer, but not able to connect the dots.

There is an example of classification in this repository:
DogVsCat


Dynamic Regression Prediction with Flask(or sth. similar) as a self learner Data Scientist I am working on deploying my model in order to get regular predictions. And I need your guidance about deployment section, in which I know nothing about and there is nobody around to guide me. There is not any line of code etc, so I am just asking for your theoretical opinions.
My process will go on like this:
1- Every 2 hours, system will run a query and save a csv file to a specific location. It will overwrite of course. And this task is on me, I am not asking about it. This csv file is the prediction dataset actually, which will be retrieved by ML model periodically to make predictions on it.
2- I have to make predictions, lets say, every 2 hour. My service should retrieve this csv file automatically from the local path I will define.
3- Then it should make predictions based on the model I will define for it in pickle form.
4- Finally it should download the results or final dataframe in csv format again into the same path as predictions.csv for example. It should overwrite of course.
For this process, I am making my research on Flask Web Services. But I must ask you, is this process possible with flask applications? As you can see, there is not any manual trigger in this process, everything is automatic, no one needs to visit a web page or hit a run button etc. If you can suggest me any resources or web pages that you think may be beneficial for me, it is highly welcome.
Have a nice day

This seems more like an ETL job than an API one. Of course you can have both, but in your specific case I would design it as a series of tasks where each task you extract, transform and load data.
Airflow is an amazing tool for that: https://github.com/apache/airflow
In airflow, each ETL is defined as a DAG (directed acyclic graph) with a series of tasks that are customized by you.
So, for your specific case, you would create a DAG with following tasks:

Retrieve data, prep if necessary.
Make predictions
Save new dataset predictions.csv.

You can schedule it like a cron job to run at each 2 hours. I highly recommend you to look at.


Algorithm for sequences classification I want to ask wich algorithm can I use to do a sequences classification , knowing that I have two classes (positive /negative), but training is done using data from one class only (positive).

Thank you

With only one class in your training set, this sounds more like an unsupervised learning problem.  Maybe look into anomaly/outlier detection algorithms rather than more traditional binary classifiers.


Predict the best time of call I have a dataset including a set of customers in different cities of California, time of calling for each customer, and the status of call (True if customer answers the call and False if customer does not answer).

I have to find an appropriate time of calling for future customers such that the probability of answering the call is high. So, what is the best strategy for this problem? Should I consider it as a classification problem which the hours (0,1,2,... 23) are the classes? Or should I consider it as a regression task which the time is a continuous variable? How can I make sure that the probability of answering the call will be high?

Any help would be appreciated. It also would be great if you refer me to similar problems.

Below is a snapshot of the data.

You might actually encounter problems if you model this as a regression problem without a suitable transformation. For instance, we know that most calls are probably answered during daytime and less during night time and early morning. A linear regression would have difficulty because the relationship is likely curvilinear, not linear. For the same reason, treating this as a classification task with logistic regression would also be problematic.

As suggested by other respondents, reclassifying your data into time periods will help, and I'd suggest you try something like a decision tree or random forest first.

That all said, this might be a case for simple descriptive statistics. If you plot the proportion answered calls by time of day (split by city or any other demographic), is there a clear best time?  If so, why complicate things with a model?


Clustering (unsupervised learning) for uneven classes I am looking for an unsupervised method that can see also the points that start to look different from the majority. Which clustering techniques (I use python) can be used for such data sets?

I have tried k-means but as I was expecting it has failed considerably to see such peaks.

You can try with DBSCAN. It's a clustering algorithm that is meant to isolate outliers. You can read more about it here. It's already available in sklearn (documentation here).

If you tweak its parameters correctly, those "extreme" observations should be identified and separated from actual clusters.


Connecting Authors with Published Papers I'm specifically interested in tying doctors to their published papers. The key issue is that using name alone will result many collisions. I'm wondering what set of features I would need to reliably connect a doctor with a given published paper? Aside from weak features such as specialty, is there any database that has a link between doctor NPI and papers published?

I've seen this on linking NPIs to authors in PubMed but it seems rather unreliable.

It's not an easy question to answer even for groups with a lot of leverage such as Research Gate (RG). RG has it's own (I assume proprietary) author matching algorithm which has caused problems in the past.

They use the name of the author (in different combinations) to suggest authorship to RG users (so has you said, it does causes a lot of problems). Every once in a while users that are not the author accept the suggestions and, from the portal point of view, gain the equivalent reputation. It's a serious business that requires quite a bit of R&D before making a decision.

That being said I can't answer with certainties only with reasonable possibilities. A few questions I would make (and hope to answer with a bit of data analysis):

What is the probability that an author with a confirmed publication in an identified journal will publish in that journal again? - Journal Name
What is the probability that an author that has partnered with other authors will repeat the same co-authoring combination? Co-author Names
What is the expected time story for publishing for each author? Authors rarely publish articles 20 years apart. Typically they publish more and more, or less and less. Time frame
How frequently do authors change the institution they belong to? Institution Name
What are the preferred keywords for a given author? Key Names
What are the preferred citations made by a given author? Bibliography

All of the questions above require quite a bit of Text Mining and String Matching, as well as solid dataset to start your mining.

Some publishers have their own API although I can't say much about permissiveness (never tried it myself). RG has been promising one for years but, as far as I know, it still does not exist.

An unlikely thing I remember now is the inspiring story of Aaron Swartz. This activist, along with other persons, successfully managed to create large open archives for books and articles. Should that information still exist it might be worth your time to take a look there.

Also if you have a list of the journals you are considering (you've mentioned only "doctors" which is a bit vague) you can try and see with the publisher if they have any way of accessing their database.


Classifying multilabel images with TensorFlow The dataset that I am categorizing with TensorFlow ML library contains multiple labels per image. The contents are real estate images photographed from outside that are analyzed for various image features.

The question is how to assign the labels, the most straightforward way is to use either many long labels, or few short labels.

Categories:

Building type ("house", "apartment building", "condominium")
Build year ("old", "new")
Garage (boolean) - only for house
Floors (1, 2) - only for house
Construction ("standalone", "row house") - only for house

Many long labels (softmax_classifier):

house_old_nogarage_onefloor_standalone
house_old_nogarage_onefloor_row
house_old_nogarage_twofloor_standalone
...43 more
apartment_old
apartment_new

Few short labels (something like a Multiclass Support Vector Machine):

house
apartment
condominium
new
old
...7 more

Alternative to this would be to use a multi-label classifier by replacing the default softmax_classifier of the Inception Model v3 by something like the sequence_classifier.

I want to represent the data accurately, but also I don't mind doing simple categorization if the accuracy is acceptable.

Which of the proposed 3 labeling solutions suits the problem?

I suggest you try building multiple networks: one network for building type (outputs "house", "apartment", or "condominum"), another network for build year, another for garage (yes vs no), and so on.  This keeps the number of classes small for each network, and allows each network to tune itself for the specific task it is focusing on.

If you want to avoid multiple networks:

Conceptually, share the first n-1 layers, and have a separate nth layer for each of these tasks.  In other words, the last layer is made wider, with elements for building type, elements for building year, and so on.  You end with multiple softmax elements.  A simple example would be one softmax for building type outputs ("house", "apartment", or "condominum") and one softmax for build year ("new" or "old").  Thus, you'd have 5 output wires.

You can of course experiment with other splits, say, where the first n-3 layers are shared and then for the last 3 layers you have multiple different networks in parallel.


How to create a multiple layer perceptron with layers of specific sizes in keras? I am trying to recreate a model based on its description. The model is described as a "10 layers of size 500 ... 10". From my understanding, the size refers to the number of hidden units, which I could customize using model.add(dense)

My code looks like this:

model = Sequential()
model.add(Conv2D(32, (3,3), activation='relu', input_shape=input_shape)
model.add(Dense(500, activation = 'relu'))
...
model.add(Dense(10, activation = 'softmax'))

However, it is yielding a very low accuracy rate (about 20% lower than what I was supposed to get).

Am I doing something wrong?

There may be quite a lot of reasons for this. As I understand you try to repeat someone's results, and all you know is the architecture. Your keras architecture seems correct, but you should also take into account:

Data - did the authors used data augmentation?
Regularizations, such as dropout, L1, L2, batch normalization - that can significantly influence your results.
Other hyper-parameters, like optimizer, batch size etc. Did you do a grid/random search of those?

If you took all this into account, provide us with more complete training code, so we don't have to guess.


Reading accuracy in Keras from the saved model Yesterday I trained the model in Keras all night, but when I tried to see the results, my computer was not working. When I turned it ON, then I saw 'model.h5' file built already, but I couldn't see the results of accuracy of my training and testing. I saved the model with 'model.save', but how can I now see the accuracy of the model only from that saved model?

According to the official Keras website, you have to use:

keras.models.load_model(filepath)

Example:

model = load_model('my_model.h5')

This will load your saved H5 model to 'model' and then you can try:

model.evaluate(x,y)

To return the loss value & metrics values for the model in test mode.


How an input data flows through an lstm layer cells? I make this sec2sec NN model for the purposes of learning:
model = Sequential()
model.add(LSTM(100, activation="relu", input_shape=(3,1)))
model.add(RepeatVector(2))
model.add(LSTM(100, activation="relu", return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer="adam", loss="mse")

It was trained on this sequence:
[10, 20, 30, 40, 50, 60, 70, 80, 90]
for using like this
x_test = array([70, 80, 90])
x_test = x_test.reshape(1, 3, 1)
model.predict(x_test, verbose=0)

I can not understand whats happen with input data inside all 100 LSTM cells when I invoke model.predict().
Explain me please what all this 100 units do if x_test array goes into one unit input.

The test data is a 3 step sequence data i.e. 70,80,90.
Following will be the flow -

70 will reach the 1st LSTM layer i.e. to the all 100 in parallel
It will also come back by the Recurrent weight to be multiplied with the next seq.
80 will reach and add up with the Recurrent of 70.
Similarly, 90 will pass through the 1st LSTM layer.
All the seq_step is held at 1st layer because return_sequences=Flase
After the last seq_step, the output will reach the next layer
RepeatVector will duplicate the vector, so the single output will become 2 step input for the next LSTM layer.
When the first of the duplicate will reach the LSTM, it will behave exactly like the previous LSTM layer. In addition, it will pass the output to the next layer too because return_sequences=True here.
Same thing will repeat for the 2nd duplicate vector.
The role of TimeDistributedlayer is to wrap the two incoming data as two steps and hence you will receive a (2,) output.

How vectors look across the Recurrent Layers

This is intuitive for a single Neuron/Single Feature i.e. as depicted below

$\hspace{5cm}$

For a multi-feature, multiple Neuron, it goes like below
 Your case has a single feature for each step and 100 Neurons. The below depiction has 3 features and 4 Neurons. Features behave the way it does in a simple feedforward network.
The only new information here is that recurrent of each Neuron goes back to each of the 100.

$\hspace{7cm}$ Image credit - SO Answer


Story Tag Prediction - Optional Labels I'm currently working on a prediction for fiction. I have a database with fiction, which are each described with different story tags. My idea is to use a neural network that can tell you by processing a new story which tags are relevant.
The problem is, that the original data wasn't generated but added by users. A story in the woods could be tagged with trees, nature etc. Another story that also takes place in the woods might not be tagged with nature, even though the tag applies. This might confuse the neural network. Is there a way to prevent this form happening?

Thank you!

Welcome to our community!

If I understood correctly, you don't trust the labels on your training dataset.

  The problem is, that the original data wasn't generated but added by users.

That is not properly a problem, actually almost every dataset was created by human labeling. Your problem is that you don't trust the judgement of those users and you think that data might be incomplete.

It is true that this might affect your model's performance. But, you should try either way. Sometimes the model learns to label even better than the training dataset:

Recently I looked the master's dissertation of a friend (Wesley L. Passos,M.Sc. by UFRJ) which used deep learning to detect tires on drone images (for Aedes Aegyptis prevention procedures). The dataset was created by our group my manually annotating images with bounding boxes and while we missed some of the tires that were pretty well hidden the D-CNN model was capable of detecting those hard subjects.

Note: This dissertation was recently accepted and isn't available online yet. Once it does, I will update this answer with the proper reference.

Since wrangling with data is our everyday job this is a good opportunity for you to put in practice what you have learned:

Clean your data by either removing incomplete samples or filling missing values. This is a common part of our jobs as data scientists.
If you feel that the amount of work is  a bit overwhelming you can try clustering instead or semi-supervised methods to speed up the cleaning process.
Also you can try posting your data online for further contributing.

Curiosity Note:

Crowds are usually more intelligent than individual, so with problem statistical treatment data can actually be better annotated by crowds, using mean answers or by voting. Check for The Wisdom of Crowds (Not the TV Show).


Classification accuracy of a Random Multi-label Classifier What is the exact accuracy of a random classifier which has n labels (say 1000) where k labels (say 50) are true?
Can I say the accuracy of a random classifier has an upper bound of k/n?

-Edit-

I am interested in a numerical figure or an upper bound for a random classifier (for example the accuracy of a classifier randomly picking 50 classes out of 1000 where some 50 classes are correct).
Sorry if my previous question is not fully understood.

In the context of a multi-label classifier (that is, having potentially more than one label per data point) we can have a prediction which is either fully correct, partially correct or fully incorrect. To account for this, the following accuracy metric has been proposed:
$$\text{Accuracy} = \frac{1}{n}\sum_{i=1}^n\frac{|T_i \cap P_i|}{|T_i\cup P_i|},$$
where $n$ is the sample size, $|\cdot|$ denotes the number of elements of a set, $T_i$ is the set of true labels and $P_i$ is the set of predicted labels for an item of the sample.

For instance, let's say the $i$-th item has actual labels $A, B, C$ and $D$ (that is $T_i = \{A, B, C, D\}$, but your classifier predicts labels $B, D$ and $E$ (that is $P_i =\{B, D, E\}$), then you'd get
$$\frac{|T_i\cap P_i|}{|T_i\cup P_i|}=\frac{|\{B,D\}|}{|\{A,B,C,D,E\}|}=\frac{2}{5}.$$
To get the accuracy for the entire training set, you calculate this for each instance $i=1,\dots, n$ and you average it out.

You can also have a look at the evaluation metrics section of the article in Wikipedia about multi-label classifiers: https://en.wikipedia.org/wiki/Multi-label_classification#Statistics_and_evaluation_metrics


Training Object Detection model on just 10 images I am trying to train an object detection model using Mask-RCNN with Resnet50 as backbone. I am using the pre-trained models from PyTorch's Torchvision library. I have only 10 images that I can use to train. Of the same 10 images, I am using 3 images for validation. For the evaluation, I am using the evaluation method used in COCO dataset which is also provided as .py scripts in the TorchVision's github repository.
To have enough samples for training, I am oversampling the same 10 images by a factor of 100 i.e. I end up with 1000 images that I can use to train my model. Similarly, I end up having 300 images that I can use for validation.
Now, the problem is that I am getting 0% mAP after train and 0% recall. I have two questions:
Q1. Why would it return 0% mAP?
If it has something to do with the fact that I am oversampling to a large extent, then my next question is
Q2.  Shouldn't the oversampling just cause the model to Overfit and instead provide a higher training as well as validation accuracy for my case (since I have picked the validation data from the training data itself?

Consider using data Augmentation first, you'll have a couple of hundred new images for you to work with, then add some noisy images since to avoid over-fitting. This way you'll have better results.


Graph-Document-Recommendations I want to build a Recommendation System to recommend products to users.
This is for research purposes.
The context-system the engine will be integrated in is also not build yet.

So right now I am starting the project, building kind of a E-Commerce & Social-Network-Platform for research purposes.

To realize the recommendation system, I want to integrate Neo4j and Elasticsearch with each other. For the content based filtering part Elasticsearch should do its job nicely. For the collaborative filtering part I want to use the Graph in Neo4j.

I would like to ask you for some experience or suggestions about the following questions regarding this topic:

Should I use another database as the main system storage and just use Neo4j to store recommendation data? Or is it a suggestive way to just store everything in the same graph?
How would you determine which part of the recommendation computation should happen online in Realtime and which offline as a precomputation?
Does anyone here have some experience with this kind of realization? What did your architecture look like?

Sorry, if this is a little vague described here and there. I am also new to this and want to expand my horizon.

Thanks alot for your help. I would be really happy to get some input here?

Cheers TJ

Why not using a Multi-Model DBMS like OrientDB (I'm the author) or any others? In this way, with just one system you can store documents and graphs together.

It would be:
- much faster (because you avoid the double call to 2 systems where most of the times the 2nd call needs the output of the 1st call, so you pay a lot of latency)
- easier to manage: one DBMS, one set of skills, in the case of commercial support, only one vendor

Following your use case, with a Multi-Model DBMS like OrientDB, you could start with a FULL-TEXT query (the underlying technology is Lucene, the same of Elastic Search) and then navigate the resulting graph.

Look at the Social Travel Agency example about how you can mix multiple models at the same time: https://orientdb.com/docs/3.0.x/gettingstarted/demodb/DemoDB-DataModel.html.


Relation between an underlying function and the underlying probability distribition function of data I heard and read a lot of times the following statements and got a lot of confusion over time.

Statement 1: The goal of machine learning is to get a function from
the given data
Statement 2: The goal of machine learning is to find the underlying
distribution function of the given data

From the above two statements, I generally interpret that the underlying function is a probability distribution function of the given data.
But I did not understand the relation between the probability distribution function and the function we want to get for a particular task.
Let us consider the following example.
A random experiment $E$ has a sample space $\Omega$ and I defined two random vectors $X_1, X_2$ on $\Omega$. I am using a neural network for my task. The domain of the neural network is the range of $X_1$ and I am expecting the range of the neural network to be the range of $X_2$ with correct mapping that satisfies the data. Let $f$ be the actual function that we want the neural network to become that maps from the range of $X_1$ to the range of $X_2$. Assume that I have the joint probability distribution $P(X_1, X_2)$ for my dataset.
Now, what is the relation between the actual function $f$ we are approximating using the neural network that transforms the range of $X_1$ into a range of $X_2$ and the joint distribution $P$?

The function you are thinking about (both statement 1 and statement 2) is the same function.  Allow me to use your framework to describe it.

The domain of the neural network is the range of $X_1$ and there exists some distribution function $F_{X1}(x)$ that describes how the values of $X_1$ are generated.  The same is true for $X_2$ and $F_{X2}(x)$.  Then there exists $f$ which maps:

$$
f: \mathbf{range}(X_1) \rightarrow \mathbf{range}(X_2)
$$

Note that this is not $\dot{f}: X_1 \rightarrow X_2$, (not sure if we can this easily define a function whose domain and range are spaces with different densities, an interesting idea to explore but off-topic for us here).

Now.  Here I'll diverge from your framework.  We will assume that we in fact constructed and trained a neural network which estimates (with good confidence) $f: \mathbf{range}(X_1) \rightarrow \mathbf{range}(X_2)$.  What we can say about this neural network and about $f$, is:

$f$ can be used to predict, given a value from the range of $X_1$ a corresponding value in the range of $X_2$.  This is by our understanding that our model is a good predictor.
The model models $F_{X2}(x)$.  In other words $f$ is $F_{X2}(x)$; and it models the distribution of $P(X_2)$.

Why is that?

In order for $f$ to be a model of $F_{X2}(x)$ then we must have:

$$
f: \mathbf{domain}(X_2) \rightarrow \mathbf{range}(X_2) \Longleftrightarrow
f: \mathbf{range}(X_1) \rightarrow \mathbf{range}(X_2)
$$

Which is true by the way how we designed our neural network.  We said that the network receives the range of $X_1$ as its input, which is the domain of the neural network.  We also said that the output of the network is the range of $X_2$, therefore the network itself if the model of $P(X_2)$.

Common Sense Reasoning

The neural network receives values taken from $F_{X1}(x)$ as the input but compares its output against values in $F_{X2}(x)$, and it can only change (train) itself against values it can compare something to (its output).  In other words, the network does not attempt to understand the distribution of its input.  If we have the input as just a single sample from $F_{X1}(x)$ we should still be able to evaluate $f$ correctly.

Reality may sometimes seem different though, and the distribution of inputs actually may affect the training of a neural network in practice.  For example, classes/regions with very little support would impact the score of the network (and consequently its estimate of $P(X_2)$.  Yet, the idea of little support is not relevant for us since we are thinking of $X_1$, which is a random variable and its expected value would be from a big enough number of samples.

Extra Note

The constrain that $X_2$ must be from the same sample space ($\Omega$) as $X_1$ is not needed, although possible.  This constrain would affect how the network is trained (it is a constrain after all), yet it will be the same constrain on how $P(X_2)$ looks.


Linear Regression to predict a growing variable with time Can we use Multiple Linear Regression to predict a dependent variable that is growing exponentially with time?

When you have an exponential shape in $x$, you can transform $x$ by taking logs.

So while a model $y=\beta x + u$ will likely perform badly, a model such as $y=\beta \log(x)+u$ will likely perform better.

However, based on your minimal description of the problem, you cannot expect a really detailed answer.


Why does Bagging or Boosting algorithm give better accuracy than basic Algorithms in small datasets? I was working with a small dataset, with 392 values, and it was kind of an imbalanced dataset, with 262 values belonging to class 1 and rest 130 to class 0. So I did the upsampling technique, importing sklearn.resampling module.

However, the total dataset was now around 520 values. I applied basic, algorithms first like Logistic Regression and SVM Classifier, and since we all know that precision is not a good accuracy metric for imbalanced dataset, I use the f1-score and recall score. In logistic Regression I found out, it was giving 78% f1-score for class 1 and 80% for class 0 , and almost 99% f1-score for class 0 in SVM and 72% for class 1, which shows that it is overfitting.

But to my surprise I found out that Random Forest gave me a better accuracy, with having around 83% f1-score for class 0 and 82% for class 1 . But till now everywhere I have seen that for bagging and boosting algorithms to work well, we need a lot of data, which is not the case in this scenario.

I've searched google a lot, but unfortunately I haven't been able to get any specific answer, and I need to know the fundamentals, why does this happen?

Logistic Regression:

                precision    recall  f1-score   support

          0       0.80      0.80      0.80        91
          1       0.78      0.78      0.78        82

avg / total       0.79      0.79      0.79       173

[[73 18]
 [18 64]] (confusion matrix)

SVM with rbf-kernel:

                precision    recall  f1-score   support

          0       0.80      0.99      0.88        91 (kind of overfitting for class 0)
          1       0.98      0.72      0.83        82

avg / total       0.89      0.86      0.86       173

[[90  1]
 [23 59]]

Random Forest Classifier:

                precision    recall  f1-score   support

          0       0.82      0.86      0.84        87
          1       0.85      0.81      0.83        86

avg / total       0.84      0.84      0.84       173

[[75 12]
 [16 70]]

SVMs finds a boundary to divide the data between the two classes, and the rule of thumb is to use SVMs for binary classification problems. Typically, when you have linear dependencies SVMs performs better than random forest since there is a distance to measure. But random forest performs better with categorical values / non-linear dependencies. Random forest does benefit from more data, but nature of data even in small data-sets matters.

Logistic regression follows similar logic, if your underlying relation is linear in nature then it probably perform better than Random Forest.


What is the difference of R-squared and adjusted R-squared? I have in mind that R-squared is the explained variance of the response by the predictors. But i'd like to know how the adjusted value is computed ? and if the concept has any change from the original.

A google search for r-squared adjusted yielded several easy to follow explanations.  I am going to paste a few directly from such results.

Meaning of Adjusted R2
Both R2 and the adjusted R2 give you an idea of how many data points fall within the line of the regression equation. However, there is one main difference between R2 and the adjusted R2: R2 assumes that every single variable explains the variation in the dependent variable. The adjusted R2 tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable.

What Is the Adjusted R-squared?
The adjusted R-squared compares the explanatory power of regression models that contain different numbers of predictors.
Suppose you compare a five-predictor model with a higher R-squared to a one-predictor model. Does the five predictor model have a higher R-squared because it’s better? Or is the R-squared higher because it has more predictors? Simply compare the adjusted R-squared values to find out!
The adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance. The adjusted R-squared can be negative, but it’s usually not.  It is always lower than the R-squared.


Regression model for continuous dependent variable and count independent variables I am currently learning R and I am relatively inexperienced in the field.  Hope I can get some advice from you guys!

I am working on a project where I have to estimate the average processing time of different work items (tasks).

I have the following panel data:

My sample size is n=2000 individual workers, and T=10 (each time interval is a four week period)

Independent variables: 51 different work items.  I have count data for each work item (# of times they are performed by each worker over a four week period)
Dependent variable: Total Working Hour of the worker (over a 4 week period)

The goal of my analysis is to find the regression coefficents (which are estimâtes of the average completion time of each work item).  I may also include other regressors (other than #of work items) such as experience, age... into my model.

y= Bo + B1*X1 +...+Bk*Xk + e

y: total working hours
X: # of each work items type

Issues:

Right now, I finished cleaning and processing the data and I performed some exploratory data analysis.

Some work items have a lot of zeros (the work item is only performed once or twice by several workers in the time period).
From VIF, I can see that there are imperfect multicollinearity in the independent variables.  Some independent variables have VIF of 5 to 6.

Questions:

Any advice on how I should specify my model?

I look at boxplots and eliminate outliers of each regressor,  I see that some regressors are highly skewed (due to lots of zéros).

I also plot each regressors against the total complétion time to see if there is any linear relation. So do, other looks more like a quadratic relation.

Any way to deal with the multicollinearity aside from eliminating the regressors that have high VIF?  This is because I need to estimate the coefficent of each of the work item.
Should I set the intercept to 0?  I know for sure that when ALL the regressors are 0 (# of work items are all 0, I should have zero total working hours).

I would also welcome any other advice for this problem. Thanks!

As you would like to retain all the predictors, you should try implementing ridge regression, which is a regularization technique popularly used for multi-collinearity problems like yours, by means of coefficient shrinkage.


How to create a general plot for this problem I would like to have some ideas about each customer depending on how many products he sells each month.
We have created a new feature X based on some other features.
We have noticed that, in most case, when Feature X decrease the amount of product sells decrease as it is illustrated with the figure bellow

For each customer, I have a similar plot (i.e in the most cases when the feature decrease buying product decrease).
How can I generalize this analysis for getting one global plot?

I think your data has three variables: customer_ID, number_of_products_sold, and feature_x. Time is not, as far as I can tell from your description, relevant.
To get all three relevant variables plotted in one figure, start with the data for a single customer and show number_of_products_sold versus feature_x as a scatter plot (removing the time aspect). This scatter plot with, say, circles for each data point, could be extended to multiple customer_IDs by overlaying scatter plots with different marker types.


Predictions of a Deep Learning Model I’ve built a model that classifies digits from 0-9. My dataset is tf.keras.datasets.mnist. I use softmax as the activation function for the output layer.

Q1:

The output layer should consist of 10 neurons. Each representing a digit from 0-9. But even when I change the number of output layer neurons to 20, the predictions are accurate. Seems as if even though I only need to predict 10 digits, I can have more than 10 neurons in the output layer. Why’s that?

Q2:

Also, the prediction is a list of lists right. Each list containing probabilities corresponding to 10 neurons and we take the neuron having the highest probability. My question is, suppose the 6th neuron has the highest probability, how do I know what digit(label) is assigned to it?

I’m a complete newbie to deep learning, so please dumb it down for me.

Q1: I do not really understand how the situation in your Q1 is possible - I would expect an error to be thrown about as a mismatch in shape. For example, when I change the number of classes in the final dense layer, I do indeed get an error.

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
# adjust the number of layers from 10 to 20 - which throws an error
model.add(Dense(20, activation='softmax'))
model.compile(loss= 'categorical_crossentropy',
          optimizer = 'rmsprop',
          metrics=['accuracy'])

Output:

  ValueError: A target array with shape (60000, 10) was passed for an
  output of shape (None, 20) while using as loss
  categorical_crossentropy. This loss expects targets to have the same
  shape as the output.

Perhaps you can post your code when you build in the layers.

Q2: The list of probabilities that are output by the model will match the order of the classes on which you trained it. For example, if the order of the classes for the digits 0-9 is [0, 1, 2, 3, ... 9] and the model outputs [0.91, 0.01... 0.01] for a certain image, then the model thinks there is a 91% chance that the image is a '0'


Is there a difference between Vectorization and Featurization? I have come across articles that mention Featurization and Vectorization interchangeably. Is there a minute difference between each other or are they exactly referring to the same thing?

It's hard to say for sure without having the article in hand.  At risk of putting words in the author's mouth, I would guess that the two terms do mean the same thing if they are being used interchangeably.  Featurization is a common-ish term for feature engineering or feature extraction.  In that context,  you typically start with some low-level, noisy representation of the data, then pass it through some algorithm to transform it into a "feature vector" (a high-level representation of the data).  So I guess it might be tempting to call that process "vectorization" as well.  Once again, hard to say for sure without taking a look at the article.
Outside of a given article, I would not want to use them interchangeably.  Usually "vectorization" refers to modifying an operation so that it works with vectors rather than scalars.  (this is what numpy.ndarray is all about).


How to compare paired count data? I am working with a machine learning approach that counts cars in images. I have a predicted dataset, which is the predicted output from the machine learning approach and a paired "true" dataset, which is the result of a human going through each image and counting the number of cars.

The following is a sample of what the datasets look like (note that the actual dataset has 2500 paired samples):

import pandas as pd

d = {'true': [0,0,0,1,1,0,1,0,0,0,0,0,0,0,4,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],
     'predicted': [0,0,0,0,0,0,1,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1]}
df = pd.DataFrame(data=d)

    true  predicted
0      0          0
1      0          0
2      0          0
3      1          0
4      1          0
5      0          0
6      1          1
7      0          0
8      0          0
9      0          0
10     0          0
11     0          0
12     0          0
13     0          0
14     4          2
15     2          2
16     0          0
17     0          0
18     0          0
19     0          0
20     0          0
21     0          0
22     0          0
23     0          0
24     0          1
25     0          0
26     0          0
27     0          0
28     0          0
29     0          0
30     0          0
31     0          0
32     1          1

I am looking for a way to present the predicted approach to an audience so that they see if the predictions are statistically the same as the true observations and visualize any trends in the data (e.g. the predicted approach has a tendency to over or under predict). If these were categorical data, I would use a confusion matrix, however, I am not sure how to deal with these paired, discrete datasets that are heavily weighted with 0's.

What approach can I take to statistically compare the predicted vs true datasets?

For visualizing paired 1D data, i.e. true vs predicted counts, you may use something like ggpaired.

You may also visualize the distribution of differences, where each sample is the difference between the true count and its predicted value; excluding the zero differences could better accentuate any deviation.

For a statistical test, you may use Wilcoxon signed rank test (python), or alternatively Sign test if the distribution of differences (true count - predicted count) is not symmetric around the mean.

It is worth noting that a statistical test may lead to "not the same" (rejection of "the same" null hypothesis) for an arbitrarily small difference when a large enough sample is provided, so be careful about the interpretation of "significant". For example, the statement "predicted counts under-estimate the true counts" could be statistically significant but with mean difference 0.05. Although this difference is statistically significant, however, depending on the task, it might be negligible. Therefore, for example, reporting the "mean ± std" of difference along side the statistical test would help interpreting the "significant" better. Also, take a look at this post on effect size.


Is it possible that a CNN has better accuracy than RNN in word classification? So I found something strange once I compared the accuracy of the prediction of a class for a question between a CNN and an RNN (GRU).

The CNN achieved 0.87 accuracy over the RNN (GRU) with 0.7520 accuracy. In terms of question classification task, would it makes sense if CNN outperforms RNN or is it a rare case?

CNN considers only the current input while RNN considers the current input and also the previously received inputs. It can memorize previous inputs due to its internal memory

Here's a comparative study on RNN vs CNN on different problems.

As you can see RNN-GRU outperforms CNN on most of the cases